@kirill.konshin/lint 0.6.6 → 0.6.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/package.json +1 -1
- package/rules/github.md +3 -1
- package/rules/gitlab.md +2 -5
- package/rules/nx.md +1 -0
- package/skills/lint-github/SKILL.md +214 -42
- package/skills/lint-gitlab/SKILL.md +24 -0
- package/skills/spec-tools/SKILL.md +1 -1
- package/skills/spec-tools/references/contract.md +34 -10
- package/skills/spec-tools/references/gitlab.md +4 -3
- package/skills/spec-tools/references/rules/common.md +20 -10
- package/skills/spec-tools/references/rules/spec-verify-judge.md +9 -8
- package/skills/spec-tools/references/rules/spec-verify.md +7 -3
- package/skills/spec-tools/scripts/cli.js +418 -257
- package/skills/spec-verify/SKILL.md +10 -10
package/README.md
CHANGED
|
@@ -371,7 +371,7 @@ Where each runs:
|
|
|
371
371
|
|
|
372
372
|
Specifications change only by hand, or when the steward skill applies the owner's accepted review answers; `spec-tools steward check --fix` makes the only mechanical repairs. No audit edits code or specifications.
|
|
373
373
|
|
|
374
|
-
The audits share one set of judging rules, `skills/spec-tools/references/rules/`:
|
|
374
|
+
The audits share one set of judging rules, `skills/spec-tools/references/rules/`: each finding gets a severity and a confidence (0–100) from its reader and, when it is severe and sure enough, from a judge; the merge fuses the two readings and its thresholds give the tier. An ERROR is a demonstrated defect with a production effect (a broken flow, a contract or invariant violation, a sensitive-data leak, a performance problem): severity 70 or more at a confidence above 70%, by default. The thresholds and the judge gate are `AUDIT_SEVERITY_*`, `AUDIT_CONFIDENCE_*` and `AUDIT_JUDGE_*` variables; the affected set of a merge request or a push is tuned by `AUDIT_AFFECTED_*` (the `spec-tools` contract). `spec-verify`'s checks 2–5 are spec-steward's items 10, 9, 11 and 5, worded identically — steward asks whether the rule should change, `spec-verify` whether the code or the rule has diverged. Every file the tools write is under `.spec-audit/`, as YAML; Markdown only as the final report.
|
|
375
375
|
|
|
376
376
|
Setup and the GitLab pipeline are in the `spec-tools` skill.
|
|
377
377
|
|
package/package.json
CHANGED
package/rules/github.md
CHANGED
|
@@ -8,4 +8,6 @@ paths:
|
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
- Agentic review MUST verify this effective CI order: checkout -> enable Corepack -> setup Node with Yarn cache -> restore build caches -> immutable install -> prepare -> verification (lint/test/build) -> publish; publishing MUST depend on successful verification.
|
|
11
|
-
-
|
|
11
|
+
- Key build caches per commit when jobs run one after another: key `${{ runner.os }}-<tool>-${{ github.sha }}`, restore key `${{ runner.os }}-<tool>-`. A later job gets an exact hit and does not save again.
|
|
12
|
+
- Key build caches per job and commit when jobs run in parallel: key `${{ runner.os }}-<tool>-${{ github.job }}-${{ github.sha }}`, restore key `${{ runner.os }}-<tool>-${{ github.job }}-`. Only the first job that saves a key keeps its outputs. A downstream job restores the producing job's key and does not save.
|
|
13
|
+
- When caching Nx locally, cache its DB as `.nx/workspace-data/default-v*.db` next to `.nx/cache`: rename it to the runner's machine ID before install and back to `default-v*.db` before an explicit `actions/cache/save`, otherwise Nx ignores the restored `.nx/cache` (see `lint-github` skill).
|
package/rules/gitlab.md
CHANGED
|
@@ -10,8 +10,5 @@ paths:
|
|
|
10
10
|
- Always collect coverage from tests
|
|
11
11
|
- Always publish important build stats as artifacts
|
|
12
12
|
- Agentic review MUST verify this effective CI order: checkout and cache restore -> enable Corepack -> immutable install -> prepare -> verification (lint/test/build) -> publish; publishing MUST depend on successful verification.
|
|
13
|
-
- Keep local build-cache keys job-specific in parallel pipelines;
|
|
14
|
-
|
|
15
|
-
# Example
|
|
16
|
-
|
|
17
|
-
- Use Nx Cloud or Turbo Remote Cache when build results must be shared safely across parallel jobs.
|
|
13
|
+
- Keep local build-cache keys job-specific in parallel pipelines; a remote Nx or Turbo cache is needed only to share results between different jobs.
|
|
14
|
+
- When caching Nx locally, cache its DB as `.nx/workspace-data/default-v*.db` next to `.nx/cache`: rename it to the runner's machine ID in `before_script` (before install) and back to `default-v*.db` in `after_script`, otherwise Nx ignores the restored `.nx/cache` (see `lint-gitlab` skill).
|
package/rules/nx.md
CHANGED
|
@@ -10,6 +10,7 @@ paths:
|
|
|
10
10
|
# NX
|
|
11
11
|
|
|
12
12
|
- If NX is used root `package.json` must have `nx show projects > /dev/null` in the `prepare`/`postinstall` script, so the project graph cache exists for tools that read it outside `nx` — e.g. `@nx/eslint-plugin` rules in a bare `eslint` run silently skip without it. With `prepare`, Yarn 2+ skips it on install — run `yarn prepare` explicitly (after cloning + as a CI step); with `postinstall` (private root) Yarn runs it on install, so no extra step
|
|
13
|
+
- Nx 20+ indexes `.nx/cache` in a SQLite DB in `.nx/workspace-data`, named after the machine ID. A CI cache must carry both, with the DB under the fixed name `default-v*.db`: rename it to the machine ID before install and back before saving (see `github.md` / `gitlab.md`). One cache entry holds one DB, so restoring several entries does not merge their outputs.
|
|
13
14
|
- NX Release (v23+) with conventional commits silently demotes bumps for `0.x` packages (breaking → minor, `feat` → patch) while the log still claims `Applied semver relative bump "minor"`;
|
|
14
15
|
- set `release.version.adjustSemverBumpsForZeroMajorVersion: false` in `nx.json` if `feat:` must bump minor before `1.0.0`
|
|
15
16
|
- Root NX task-script convention: bare name = `nx run-many -t <target>` over all projects (`build`/`test`/`start`/`serve`), with concurrency flags set once here;
|
|
@@ -3,6 +3,147 @@ name: lint-github
|
|
|
3
3
|
description: Describes how to work with Github & example workflow as a reference which essential jobs and concepts usual Github Workflow should have
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
+
Reference CI: two local composite actions and a workflow. Checks run as parallel jobs; publishing needs all of them.
|
|
7
|
+
|
|
8
|
+
`.github/actions/nx-cache/action.yml` — Nx 20+ ignores a restored `.nx/cache` without its DB:
|
|
9
|
+
|
|
10
|
+
```yml
|
|
11
|
+
name: Nx cache
|
|
12
|
+
description: Caches .nx/cache with the Nx DB under the fixed name default-v*.db. Restore runs before install (./.github/actions/setup does it); save is the last step of a job that keeps its Nx outputs.
|
|
13
|
+
|
|
14
|
+
inputs:
|
|
15
|
+
mode:
|
|
16
|
+
description: restore | save
|
|
17
|
+
required: false
|
|
18
|
+
default: restore
|
|
19
|
+
job:
|
|
20
|
+
description: Job whose cache to use (defaults to this job); save only your own job's cache, parallel jobs must not share a key
|
|
21
|
+
required: false
|
|
22
|
+
default: ''
|
|
23
|
+
|
|
24
|
+
runs:
|
|
25
|
+
using: composite
|
|
26
|
+
steps:
|
|
27
|
+
- name: Check mode
|
|
28
|
+
if: inputs.mode != 'restore' && inputs.mode != 'save'
|
|
29
|
+
shell: bash
|
|
30
|
+
run: echo "::error::nx-cache mode must be restore or save, got '${{ inputs.mode }}'" && exit 1
|
|
31
|
+
|
|
32
|
+
# Nx indexes .nx/cache in a SQLite DB, cached next to it as default-v*.db.
|
|
33
|
+
# One key per job and commit: parallel jobs never race for a key.
|
|
34
|
+
- name: Restore Nx cache
|
|
35
|
+
if: inputs.mode == 'restore'
|
|
36
|
+
uses: actions/cache/restore@v6
|
|
37
|
+
with:
|
|
38
|
+
path: |
|
|
39
|
+
.nx/cache
|
|
40
|
+
.nx/workspace-data/default-v*.db
|
|
41
|
+
key: ${{ runner.os }}-nx-${{ inputs.job || github.job }}-${{ github.sha }}
|
|
42
|
+
restore-keys: |
|
|
43
|
+
${{ runner.os }}-nx-${{ inputs.job || github.job }}-
|
|
44
|
+
${{ runner.os }}-nx-
|
|
45
|
+
|
|
46
|
+
# Nx names the DB after the machine ID and ignores outputs it has no DB records for.
|
|
47
|
+
# Give the DB this runner's ID before install (postinstall runs Nx).
|
|
48
|
+
- name: Adopt Nx cache DB
|
|
49
|
+
if: inputs.mode == 'restore'
|
|
50
|
+
shell: bash
|
|
51
|
+
run: |
|
|
52
|
+
d=.nx/workspace-data
|
|
53
|
+
id=$(cat /var/lib/dbus/machine-id 2>/dev/null || cat /etc/machine-id 2>/dev/null || echo machine)
|
|
54
|
+
db=$(ls $d/default-v*.db 2>/dev/null | head -n 1 || true)
|
|
55
|
+
if [ -z "$db" ]; then
|
|
56
|
+
rm -rf .nx/cache # outputs without their DB are unusable
|
|
57
|
+
else
|
|
58
|
+
to=$d/$id-v${db##*-v}
|
|
59
|
+
rm -f "$to" "$to-wal" "$to-shm"
|
|
60
|
+
mv "$db" "$to"
|
|
61
|
+
fi
|
|
62
|
+
echo "Nx cache DB for machine ID $id: $(ls $d/*.db 2>/dev/null || echo none)"
|
|
63
|
+
|
|
64
|
+
# After the last Nx step: store the DB under the fixed name the cache holds
|
|
65
|
+
- name: Store Nx cache DB
|
|
66
|
+
if: inputs.mode == 'save'
|
|
67
|
+
shell: bash
|
|
68
|
+
run: |
|
|
69
|
+
d=.nx/workspace-data
|
|
70
|
+
db=$(ls -t $d/*-v*.db 2>/dev/null | head -n 1 || true)
|
|
71
|
+
to=$d/default-v${db##*-v}
|
|
72
|
+
[ -z "$db" ] || [ "$db" = "$to" ] || mv -f "$db" "$to"
|
|
73
|
+
echo "Nx cache DB to save: $(ls $d/default-v*.db 2>/dev/null || echo none)"
|
|
74
|
+
|
|
75
|
+
- name: Save Nx cache
|
|
76
|
+
if: inputs.mode == 'save'
|
|
77
|
+
uses: actions/cache/save@v6
|
|
78
|
+
with:
|
|
79
|
+
path: |
|
|
80
|
+
.nx/cache
|
|
81
|
+
.nx/workspace-data/default-v*.db
|
|
82
|
+
key: ${{ runner.os }}-nx-${{ inputs.job || github.job }}-${{ github.sha }}
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
`.github/actions/setup/action.yml`:
|
|
86
|
+
|
|
87
|
+
```yml
|
|
88
|
+
name: Setup
|
|
89
|
+
description: Node.js + Corepack + build cache restore + Yarn install. The repo must already be checked out (a local action cannot run before checkout). Jobs that keep their Nx outputs end with ./.github/actions/nx-cache (mode save).
|
|
90
|
+
|
|
91
|
+
inputs:
|
|
92
|
+
registry-url:
|
|
93
|
+
description: npm registry URL — set only on jobs that publish (configures .npmrc auth)
|
|
94
|
+
required: false
|
|
95
|
+
default: ''
|
|
96
|
+
nx-cache:
|
|
97
|
+
description: Job whose Nx cache to restore (defaults to this job); a job that does not save reuses another job's cache
|
|
98
|
+
required: false
|
|
99
|
+
default: ''
|
|
100
|
+
|
|
101
|
+
runs:
|
|
102
|
+
using: composite
|
|
103
|
+
steps:
|
|
104
|
+
- name: Enable Corepack
|
|
105
|
+
run: corepack enable
|
|
106
|
+
shell: bash
|
|
107
|
+
|
|
108
|
+
- name: Setup Node.js
|
|
109
|
+
uses: actions/setup-node@v6
|
|
110
|
+
with:
|
|
111
|
+
node-version: 24
|
|
112
|
+
registry-url: ${{ inputs.registry-url }}
|
|
113
|
+
cache: yarn
|
|
114
|
+
cache-dependency-path: yarn.lock
|
|
115
|
+
|
|
116
|
+
# If project uses Turbo
|
|
117
|
+
#TODO https://turbo.build/repo/docs/guides/ci-vendors/github-actions#remote-caching
|
|
118
|
+
#TODO https://turborepo.dev/docs/guides/ci-vendors/github-actions#remote-caching-with-github-actionscache
|
|
119
|
+
# One entry per commit when jobs run one after another; for parallel jobs add ${{ github.job }} to key and restore key (only the first job that saves a key keeps its outputs)
|
|
120
|
+
- name: Cache Turbo
|
|
121
|
+
uses: actions/cache@v6
|
|
122
|
+
with:
|
|
123
|
+
path: .turbo
|
|
124
|
+
key: ${{ runner.os }}-turbo-${{ github.job }}-${{ github.sha }}
|
|
125
|
+
restore-keys: |
|
|
126
|
+
${{ runner.os }}-turbo-${{ github.job }}-
|
|
127
|
+
${{ runner.os }}-turbo-
|
|
128
|
+
|
|
129
|
+
# If project uses Nx
|
|
130
|
+
- name: Restore Nx cache
|
|
131
|
+
uses: ./.github/actions/nx-cache
|
|
132
|
+
with:
|
|
133
|
+
job: ${{ inputs.nx-cache }}
|
|
134
|
+
|
|
135
|
+
- name: Install dependencies
|
|
136
|
+
run: yarn install --immutable
|
|
137
|
+
shell: bash
|
|
138
|
+
|
|
139
|
+
# Add this if Yarn 2+ is used and package is NOT private, otherwise postinstall should be configured, and this block skipped
|
|
140
|
+
- name: Prepare
|
|
141
|
+
run: yarn prepare
|
|
142
|
+
shell: bash
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
`.github/workflows/release.yml`:
|
|
146
|
+
|
|
6
147
|
```yml
|
|
7
148
|
name: Release
|
|
8
149
|
|
|
@@ -11,20 +152,76 @@ on:
|
|
|
11
152
|
branches:
|
|
12
153
|
- main
|
|
13
154
|
|
|
14
|
-
# If project publishes to NPM
|
|
15
155
|
permissions:
|
|
16
|
-
id-token: write # Required for OIDC
|
|
17
156
|
contents: read
|
|
18
157
|
|
|
19
158
|
concurrency: ${{ github.workflow }}-${{ github.ref }}
|
|
20
159
|
|
|
160
|
+
env:
|
|
161
|
+
TURBO_CACHE_DIR: .turbo # if project uses Turbo
|
|
162
|
+
YARN_ENABLE_GLOBAL_CACHE: false # so that .yarn/cache is written
|
|
163
|
+
|
|
164
|
+
# Checks run as parallel jobs; release needs all of them. Only jobs with their own Nx cache save it.
|
|
21
165
|
jobs:
|
|
166
|
+
lint:
|
|
167
|
+
name: Lint
|
|
168
|
+
runs-on: ubuntu-latest
|
|
169
|
+
|
|
170
|
+
steps:
|
|
171
|
+
- name: Checkout Repo
|
|
172
|
+
uses: actions/checkout@v7
|
|
173
|
+
|
|
174
|
+
- name: Setup
|
|
175
|
+
uses: ./.github/actions/setup
|
|
176
|
+
with:
|
|
177
|
+
nx-cache: build # reuses the build job's cache; this job does not save
|
|
178
|
+
|
|
179
|
+
- name: Lint
|
|
180
|
+
run: yarn lint
|
|
181
|
+
|
|
182
|
+
test:
|
|
183
|
+
name: Test
|
|
184
|
+
runs-on: ubuntu-latest
|
|
185
|
+
|
|
186
|
+
steps:
|
|
187
|
+
- name: Checkout Repo
|
|
188
|
+
uses: actions/checkout@v7
|
|
189
|
+
|
|
190
|
+
- name: Setup
|
|
191
|
+
uses: ./.github/actions/setup
|
|
192
|
+
|
|
193
|
+
- name: Test
|
|
194
|
+
run: yarn test
|
|
195
|
+
|
|
196
|
+
- name: Save Nx cache
|
|
197
|
+
uses: ./.github/actions/nx-cache
|
|
198
|
+
with:
|
|
199
|
+
mode: save
|
|
200
|
+
|
|
201
|
+
build:
|
|
202
|
+
name: Build
|
|
203
|
+
runs-on: ubuntu-latest
|
|
204
|
+
|
|
205
|
+
steps:
|
|
206
|
+
- name: Checkout Repo
|
|
207
|
+
uses: actions/checkout@v7
|
|
208
|
+
|
|
209
|
+
- name: Setup
|
|
210
|
+
uses: ./.github/actions/setup
|
|
211
|
+
|
|
212
|
+
- name: Build
|
|
213
|
+
run: yarn build
|
|
214
|
+
|
|
215
|
+
- name: Save Nx cache
|
|
216
|
+
uses: ./.github/actions/nx-cache
|
|
217
|
+
with:
|
|
218
|
+
mode: save
|
|
219
|
+
|
|
22
220
|
release:
|
|
23
221
|
name: Release
|
|
24
222
|
runs-on: ubuntu-latest
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
YARN_ENABLE_GLOBAL_CACHE: false # so that .yarn/cache is written
|
|
223
|
+
|
|
224
|
+
needs: [lint, test, build]
|
|
28
225
|
|
|
29
226
|
# If project publishes to NPM
|
|
30
227
|
# https://davistobias.com/articles/adding-changeset/#2.1.b-adding-changeset-to-github-workflows
|
|
@@ -36,43 +233,18 @@ jobs:
|
|
|
36
233
|
|
|
37
234
|
steps:
|
|
38
235
|
- name: Checkout Repo
|
|
39
|
-
uses: actions/checkout@
|
|
236
|
+
uses: actions/checkout@v7
|
|
40
237
|
|
|
41
|
-
- name:
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
- name: Setup Node.js
|
|
45
|
-
uses: actions/setup-node@v6
|
|
46
|
-
with:
|
|
47
|
-
node-version: 24
|
|
48
|
-
registry-url: 'https://registry.npmjs.org'
|
|
49
|
-
cache: yarn
|
|
50
|
-
cache-dependency-path: yarn.lock
|
|
51
|
-
|
|
52
|
-
#TODO https://turbo.build/repo/docs/guides/ci-vendors/github-actions#remote-caching
|
|
53
|
-
#TODO https://turborepo.dev/docs/guides/ci-vendors/github-actions#remote-caching-with-github-actionscache
|
|
54
|
-
- name: Cache Turbo
|
|
55
|
-
uses: actions/cache@v6
|
|
238
|
+
- name: Setup
|
|
239
|
+
uses: ./.github/actions/setup
|
|
56
240
|
with:
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
path: .nx/cache
|
|
67
|
-
key: ${{ runner.os }}-nx-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}
|
|
68
|
-
restore-keys: |
|
|
69
|
-
${{ runner.os }}-nx-${{ github.run_id }}-${{ github.run_attempt }}-
|
|
70
|
-
${{ runner.os }}-nx-
|
|
71
|
-
|
|
72
|
-
- name: Install dependencies
|
|
73
|
-
run: yarn install --immutable
|
|
74
|
-
|
|
75
|
-
# Add this if Yarn 2+ is used and package is NOT private, otherwise postinstall should be configured, and this block skipped
|
|
76
|
-
- name: Prepare
|
|
77
|
-
run: yarn prepare
|
|
241
|
+
registry-url: 'https://registry.npmjs.org' # publish job needs .npmrc auth config
|
|
242
|
+
nx-cache: build # exact key of the build job's cache; this job does not save
|
|
243
|
+
|
|
244
|
+
# Different runner: outputs come from the build job's Nx cache, so nothing should rebuild
|
|
245
|
+
- name: Build
|
|
246
|
+
run: yarn build
|
|
247
|
+
|
|
248
|
+
- name: Release
|
|
249
|
+
run: yarn release # the project's publish command
|
|
78
250
|
```
|
|
@@ -24,6 +24,7 @@ cache:
|
|
|
24
24
|
paths:
|
|
25
25
|
- .turbo
|
|
26
26
|
- .nx/cache
|
|
27
|
+
- .nx/workspace-data/default-v*.db # Nx indexes .nx/cache in a SQLite DB, cached under this fixed name
|
|
27
28
|
- '**/.tscache'
|
|
28
29
|
- '**/.tsbuildinfo'
|
|
29
30
|
- '**/.next/cache'
|
|
@@ -37,9 +38,32 @@ stages:
|
|
|
37
38
|
# separate so that cache can pre-populate if other steps would fail, it speeds things up
|
|
38
39
|
before_script:
|
|
39
40
|
- corepack enable
|
|
41
|
+
# Nx names the DB after the machine ID and ignores outputs it has no DB records for.
|
|
42
|
+
# Give the DB this runner's ID before install (postinstall may run Nx).
|
|
43
|
+
- |
|
|
44
|
+
d=.nx/workspace-data
|
|
45
|
+
id=$(cat /var/lib/dbus/machine-id 2>/dev/null || cat /etc/machine-id 2>/dev/null || echo machine)
|
|
46
|
+
db=$(ls $d/default-v*.db 2>/dev/null | head -n 1 || true)
|
|
47
|
+
if [ -z "$db" ]; then
|
|
48
|
+
rm -rf .nx/cache # outputs without their DB are unusable
|
|
49
|
+
else
|
|
50
|
+
to=$d/$id-v${db##*-v}
|
|
51
|
+
rm -f "$to" "$to-wal" "$to-shm"
|
|
52
|
+
mv "$db" "$to"
|
|
53
|
+
fi
|
|
54
|
+
echo "Nx cache DB for machine ID $id: $(ls $d/*.db 2>/dev/null || echo none)"
|
|
40
55
|
- yarn install --immutable
|
|
41
56
|
- yarn prepare # Add this if Yarn 2+ is used and package is NOT private, otherwise postinstall should be configured, and this line skipped
|
|
42
57
|
|
|
58
|
+
# GitLab saves the cache after after_script: store the DB under the fixed name the cache holds
|
|
59
|
+
after_script:
|
|
60
|
+
- |
|
|
61
|
+
d=.nx/workspace-data
|
|
62
|
+
db=$(ls -t $d/*-v*.db 2>/dev/null | head -n 1 || true)
|
|
63
|
+
to=$d/default-v${db##*-v}
|
|
64
|
+
[ -z "$db" ] || [ "$db" = "$to" ] || mv -f "$db" "$to"
|
|
65
|
+
echo "Nx cache DB to save: $(ls $d/default-v*.db 2>/dev/null || echo none)"
|
|
66
|
+
|
|
43
67
|
install:
|
|
44
68
|
stage: install
|
|
45
69
|
script:
|
|
@@ -45,7 +45,7 @@ Run them through the package manager (`yarn spec-tools gates`), the bin (`node_m
|
|
|
45
45
|
|
|
46
46
|
- `gates` — spec-steward's findings come first (the `spec-steward` skill says what each kind asks), then one line per change gate with what violates it and a hint.
|
|
47
47
|
- `evidence` — an open change carries deltas, or is synced but not archived: archive it before the branch merges.
|
|
48
|
-
- `report --gate` / `verdict` — exit 1 is `FAIL`, an ERROR above 80% confidence on a gating run; 77 is `ADVISORY` (ERRORs at 71–80%) or any `FAIL` on an advisory run; 3 is `INCOMPLETE`, a part with no valid findings file or requirements nobody judged (the report names them). An ERROR
|
|
48
|
+
- `report --gate` / `verdict` — exit 1 is `FAIL`, an ERROR above 80% confidence on a gating run; 77 is `ADVISORY` (ERRORs at 71–80%) or any `FAIL` on an advisory run; 3 is `INCOMPLETE`, a part with no valid findings file or requirements nobody judged (the report names them). An ERROR is severity 70 or more at a fused confidence above 70% — the reader's reading and the judge's, fused. The thresholds are variables (`AUDIT_SEVERITY_*`, `AUDIT_CONFIDENCE_*`, `AUDIT_JUDGE_*`); the report's second paragraph names the values a run used. A reader's own coverage claims are notes and change no verdict. Summarize `.spec-audit/spec-verify.md` as the `spec-verify` skill's _Interactive use_ says.
|
|
49
49
|
- `workers` exit 1 — a worker changed a file outside `.spec-audit/`; the paths are listed.
|
|
50
50
|
- `workers` log line `still invalid: …` — a worker's YAML file failed its schema after two corrections; the part counts as not read, and the completion pass reads it again.
|
|
51
51
|
|
|
@@ -45,7 +45,22 @@ Cases:
|
|
|
45
45
|
|
|
46
46
|
## Changed — `spec-tools changed`
|
|
47
47
|
|
|
48
|
-
The author's pre-hand-back check
|
|
48
|
+
The author's pre-hand-back check over the affected set (below) — each requirement with its bound tests, where its own terms occur in the comment-stripped sources, and its related requirements, quoted in full. Written to `.spec-audit/parts/changed.yaml` from spec-steward's evidence model (`.spec-audit/evidence.yaml`, written first).
|
|
49
|
+
|
|
50
|
+
### The affected set
|
|
51
|
+
|
|
52
|
+
Built in three stages, `judged = UNIT(EXPAND^HOPS(SEED_CONTEXT(diff)))`; each stage only adds, and a tighter value of its variable always gives a subset. `changed` and `scope` read the same three variables; an incorrect value exits 2, naming it, and both print the values they used.
|
|
53
|
+
|
|
54
|
+
- **SEED** (`AUDIT_AFFECTED_CONTEXT`, default `3`): the requirements a hunk of `openspec/specs/` overlaps (a deletion at its line included), the requirements an unsynced touched delta names, and the requirements a changed line reaches — a line within the context of a change, every line of a changed file with `file`, every line of an untracked file. A line reaches a requirement when it falls in the window of a binding or a citation of it (a test binding's window is its test block; any other's the lines from the citation on), or when it names one of the requirement's distinctive terms on a source line; a source line the diff removed counts too, comments stripped.
|
|
55
|
+
- **EXPAND** (`AUDIT_AFFECTED_HOPS`, default `1`): each hop adds the related requirements of the set.
|
|
56
|
+
- **UNIT** (`AUDIT_AFFECTED_UNIT`, default `edited`): `requirement` — the set; `edited` — the set, plus every requirement of each capability whose specification the diff edits; `capability` — every requirement of each capability in the set or edited.
|
|
57
|
+
|
|
58
|
+
Cases:
|
|
59
|
+
|
|
60
|
+
- **WHEN** a request changes one test of a suite **THEN** the requirements that test binds are in its affected set, and those only its siblings bind are not
|
|
61
|
+
- **WHEN** a request removes a source line that names a requirement's distinctive term **THEN** that requirement is in its affected set
|
|
62
|
+
- **WHEN** a request edits a capability's specification and `AUDIT_AFFECTED_UNIT` is `edited` **THEN** every requirement of that capability is in its affected set
|
|
63
|
+
- **WHEN** `AUDIT_AFFECTED_HOPS` is `0` **THEN** no related requirement is added, and each judged requirement still quotes its related ones
|
|
49
64
|
|
|
50
65
|
## The audit
|
|
51
66
|
|
|
@@ -65,27 +80,36 @@ Every run reads on the cheap model. `AUDIT_MODEL_CHEAP`, `AUDIT_EFFORT_CHEAP`, `
|
|
|
65
80
|
|
|
66
81
|
### Scope — `spec-tools scope`
|
|
67
82
|
|
|
68
|
-
The corpus on the nightly, or wherever there is no merge-request target; otherwise the affected set — what `changed` names — judged with cross-capability checks against the corpus its evidence quotes. The evidence is assembled mechanically first: per capability, every requirement verbatim, its bound tests at their real lines, where its terms occur, its related requirements; citations are not evidence and are left out. The scope is cut into `AUDIT_JOBS` × `AUDIT_SLOTS` parts — one round of workers across the pipeline's reading jobs (`AUDIT_JOBS`, its own count, default 1) — balanced by load, none larger than one cheap reader holds, a capability larger than that cut between its requirements; only a scope larger than a round holds opens more parts. The cut is a function of the evidence and those two numbers. Written to `.spec-audit/scope.yaml`, `.spec-audit/evidence.yaml` and `.spec-audit/parts/` (one YAML evidence file per capability or slice, and `units.yaml`).
|
|
83
|
+
The corpus on the nightly, or wherever there is no merge-request target; otherwise the affected set — what `changed` names — judged with cross-capability checks against the corpus its evidence quotes. The evidence is assembled mechanically first: per capability, every requirement in scope verbatim (a capability the affected set holds in part says so in its file), its bound tests at their real lines, where its terms occur, its related requirements; citations are not evidence and are left out. The scope is cut into `AUDIT_JOBS` × `AUDIT_SLOTS` parts — one round of workers across the pipeline's reading jobs (`AUDIT_JOBS`, its own count, default 1) — balanced by load, none larger than one cheap reader holds, a capability larger than that cut between its requirements; only a scope larger than a round holds opens more parts. The cut is a function of the evidence and those two numbers. Written to `.spec-audit/scope.yaml`, `.spec-audit/evidence.yaml` and `.spec-audit/parts/` (one YAML evidence file per capability or slice, and `units.yaml`).
|
|
69
84
|
|
|
70
85
|
Cases:
|
|
71
86
|
|
|
72
87
|
- **WHEN** one capability's evidence exceeds one reader **THEN** it is cut between its requirements, each in exactly one piece
|
|
73
|
-
- **WHEN** a request changes a source file that cites no requirement **THEN** the requirements whose
|
|
88
|
+
- **WHEN** a request changes lines of a source file that cites no requirement **THEN** the requirements whose distinctive terms those lines name are in its affected set, and those whose terms only other lines of the file name are not
|
|
74
89
|
|
|
75
90
|
### Workers — `spec-tools workers [--complete|--verify]`
|
|
76
91
|
|
|
77
|
-
One headless `claude --print` worker per part loads the `spec-verify` skill, reads the judging rules (`references/rules/`), and writes its own YAML findings file, `.spec-audit/parts/findings/part-<n>-<reader>.yaml`. When a worker stops, its file is parsed and checked against its schema; on errors the worker's own session is resumed (`claude --resume <session>`) with the error list, at most twice, and a file still invalid counts as not written — the reader as not run, the verdict as missing. In a parallel job (`CI_NODE_INDEX` of `CI_NODE_TOTAL`) it reads only its share: part p goes to job ((p − 1) mod total) + 1. `--complete` judges, as each part's next reader, exactly what the merge listed short; `--verify` puts
|
|
92
|
+
One headless `claude --print` worker per part loads the `spec-verify` skill, reads the judging rules (`references/rules/`), and writes its own YAML findings file, `.spec-audit/parts/findings/part-<n>-<reader>.yaml`. When a worker stops, its file is parsed and checked against its schema; on errors the worker's own session is resumed (`claude --resume <session>`) with the error list, at most twice, and a file still invalid counts as not written — the reader as not run, the verdict as missing. In a parallel job (`CI_NODE_INDEX` of `CI_NODE_TOTAL`) it reads only its share: part p goes to job ((p − 1) mod total) + 1. `--complete` judges, as each part's next reader, exactly what the merge listed short — mechanical gaps only: a part with no valid findings file, and requirement ids no reader's judged list names; a reader's claim that it skipped a source is a note in the report and is not read again, since a second reader meets the same limit; `--verify` puts each finding whose capped reader severity is `AUDIT_JUDGE_FROM` or more and whose reader confidence is `AUDIT_JUDGE_MIN_CONFIDENCE` or more to one judge, and logs how many each gate held back. The judge loads no skill: its brief inlines the common and judge rules, leaves out the reader's numbers unless `AUDIT_JUDGE_READER_NUMBERS=show`, and the judge answers with its own `judgeSeverity` and `judgeConfidence` (0–100 each), a reason, and a production scenario for an ERROR-band severity. How many run at once follows the container's memory (300 MB a worker), never more than `AUDIT_WORKERS` (default 2), and each pass logs the job's memory peak against its limit; a worker past `AUDIT_WORKER_TIMEOUT` (20m) is killed and its part judged by the completion pass. A pass that changed any file outside the audit's outputs exits 1, naming the paths; a missing input exits 2. A worker that exits non-zero within its first minute and wrote nothing failed to start (e.g. "Not logged in" when many sessions start at once); it is retried once after `AUDIT_START_RETRY_DELAY` (default 30s, plus up to a third more at random, so the sessions that failed together do not start together again), rather than left to the completion pass.
|
|
78
93
|
|
|
79
94
|
### Merge — `spec-tools report [--gate] [--worktree]`
|
|
80
95
|
|
|
81
|
-
Nothing is judged twice.
|
|
96
|
+
Nothing is judged twice. Each finding carries a reading — `readerSeverity` and `readerConfidence`, 0–100 each — and the judge adds a second. The merge, deterministically:
|
|
97
|
+
|
|
98
|
+
- **Caps** each finding's severity by rules no reading overrides: `spec-dup`, `untested` and an unknown kind stay below the WARN band; `conflict` and a finding against an ⚠️ Advisory requirement stay below the ERROR band; and an ERROR-class kind (`code-mismatch`, `undeclared-gap`) reaches the ERROR band only when two of its quotes are found at their `file:line` in the audited commit (`HEAD`, or the working tree with `--worktree`). Quotes from `.spec-audit/` and from installed dependency code (`node_modules/`) are not the repository's own and count for nothing, so a run in CI and a run on the working tree grade a finding alike. A cap that lowers a severity says why (`regraded`).
|
|
99
|
+
- **Fuses** a judged finding's two readings: each weighted by the inverse of its uncertainty (`100 − confidence`, at least 1), `severity = Σ(s/u) / Σ(1/u)`; `confidence = 100 − min(u) − max(0, |s_reader − s_judge| − AUDIT_CONFIDENCE_IGNORE_GAP) / 2` (default gap 20), clamped to 0–100 — never surer than the surer reading, less sure when they disagree beyond about one band's width. The cap applies again; a judge's ERROR-band severity with no production scenario caps the finding below the ERROR band. A finding the verification pass did not reach keeps the reader's reading, and says so.
|
|
100
|
+
- **Tiers** each finding by the thresholds: ERROR at `AUDIT_SEVERITY_ERROR` (default 70) or more with a confidence above `AUDIT_CONFIDENCE_SURE` (default 70); WARN at `AUDIT_SEVERITY_WARN` (default 40) or more, or severe but not sure enough; INFO otherwise. An incorrect value exits 2, naming the variable.
|
|
101
|
+
|
|
102
|
+
A requirement of a part that no reader's judged list names was not judged, whatever coverage the reader claimed. Written to `.spec-audit/spec-verify.yaml` (the data, its `verdict` field the verdict, its `thresholds` the values used) and `.spec-audit/spec-verify.md` (for people, its first line the verdict, then the thresholds); each finding shows its fused reading, each model's reading and the judge's production scenario.
|
|
82
103
|
|
|
83
|
-
The verdict: `FAIL` when an ERROR stands at a confidence above
|
|
104
|
+
The verdict: `FAIL` when an ERROR stands at a fused confidence above `AUDIT_CONFIDENCE_BLOCKING` (default 80); `ADVISORY` when ERRORs stand and none is above it; `INCOMPLETE` when none stands but a part fell short; `PASS` otherwise. A part falls short only by mechanical signals: no valid findings file (none written, still invalid after corrections, a killed worker), or a requirement id missing from every reader's judged list. What a reader says of its own coverage (`sampled: …`, `not-run: …`) is a claim no tool can prove — a path it names may not exist — so the report lists it under "Readers' coverage notes" and it changes no verdict. `--gate` exits 0 (`PASS`), 1 (`FAIL`), 77 (`ADVISORY`) or 3 (`INCOMPLETE`).
|
|
84
105
|
|
|
85
106
|
Cases:
|
|
86
107
|
|
|
87
|
-
- **WHEN** an ERROR's quote is not at the `file:line` it names **THEN** the report
|
|
108
|
+
- **WHEN** an ERROR-band finding's quote is not at the `file:line` it names **THEN** the report caps it below the ERROR band, naming the quote
|
|
109
|
+
- **WHEN** the judge reads a finding **THEN** its severity is the two readings fused, and the more confident reading weighs more
|
|
110
|
+
- **WHEN** a finding's reader confidence is below `AUDIT_JUDGE_MIN_CONFIDENCE` **THEN** no judge reads it, and it is no ERROR unless its confidence is above `AUDIT_CONFIDENCE_SURE`
|
|
88
111
|
- **WHEN** a part has no findings file and no ERROR stands **THEN** the verdict is `INCOMPLETE` and `--gate` exits 3
|
|
112
|
+
- **WHEN** a reader judged every requirement of its part but claims it skipped a source **THEN** the report lists the claim as a note, and the completion pass does not read the part again
|
|
89
113
|
- **WHEN** a reader claims it sampled a check **THEN** the claim is a note and the verdict does not change
|
|
90
114
|
- **WHEN** the only ERRORs stand at 80% or lower **THEN** the verdict is `ADVISORY` and `--gate` exits 77
|
|
91
115
|
- **WHEN** every check ran in full and no ERROR stands **THEN** the verdict is `PASS`
|
|
@@ -106,13 +130,13 @@ Cases:
|
|
|
106
130
|
|
|
107
131
|
## Review glue
|
|
108
132
|
|
|
109
|
-
- `spec-tools run <skill> [--verdict <report.yaml>] [--advisory]` — one skill-driven review, headless, on `AUDIT_MODEL`; the stream kept as `.spec-audit/claude.jsonl`, the final result as the job log (`.spec-audit/job-log-<job>.md`). The skill writes its report as YAML (`findings[
|
|
110
|
-
- `spec-tools verdict <file> [--advisory]` — the verdict as an exit code: 0 `PASS`, 1 `FAIL`, 77 `ADVISORY` or `FAIL` on an advisory run (`--advisory` or `AUDIT_GATING=advisory`), 3 `INCOMPLETE`, 1 no verdict or no report. For the spec-verify data it reads `verdict`; for a skill's YAML report it computes the verdict itself, never taking it from the model —
|
|
133
|
+
- `spec-tools run <skill> [--verdict <report.yaml>] [--advisory]` — one skill-driven review, headless, on `AUDIT_MODEL`; the stream kept as `.spec-audit/claude.jsonl`, the final result as the job log (`.spec-audit/job-log-<job>.md`). The skill writes its report as YAML (`findings[readerSeverity, where, detail, readerConfidence, fields]`, `coverage{complete, notes}`); the report is checked and, on errors, sent back to the same session at most twice. Exits with Claude's status, then gates `--verdict` if given.
|
|
134
|
+
- `spec-tools verdict <file> [--advisory]` — the verdict as an exit code: 0 `PASS`, 1 `FAIL`, 77 `ADVISORY` or `FAIL` on an advisory run (`--advisory` or `AUDIT_GATING=advisory`), 3 `INCOMPLETE`, 1 no verdict or no report. For the spec-verify data it reads `verdict`; for a skill's YAML report it computes the verdict itself, never taking it from the model — each finding's one reading gets its tier by the merge's thresholds, an ERROR is `FAIL` above `AUDIT_CONFIDENCE_BLOCKING` and `ADVISORY` otherwise, and `coverage.complete: false` is `INCOMPLETE` — and renders the Markdown report beside it (`<report>.md`, its first line the verdict).
|
|
111
135
|
- `spec-tools comment <name>=<report.md>...` — the merge-request comment: one row per review from its verdict line, links into the job's artifacts, the coverage report and the diff beneath; written to `.spec-audit/mr-comment.md`.
|
|
112
136
|
- `spec-tools html <file.md>...` — each report as self-contained HTML beside it.
|
|
113
137
|
|
|
114
138
|
Cases:
|
|
115
139
|
|
|
116
140
|
- **WHEN** a review's report says `FAIL` on an advisory run **THEN** `verdict` exits 77
|
|
117
|
-
- **WHEN** a skill's report holds one ERROR at 70% confidence and nothing else **THEN** `verdict` computes `PASS` and exits 0
|
|
141
|
+
- **WHEN** a skill's report holds one ERROR-band finding at 70% confidence and nothing else **THEN** `verdict` computes `PASS` and exits 0
|
|
118
142
|
- **WHEN** a review wrote no report **THEN** `verdict` exits 1
|
|
@@ -7,11 +7,12 @@ The jobs a GitLab repository runs around `spec-steward` and `spec-tools`. Names
|
|
|
7
7
|
- `openspec validate` and `spec-tools gates` are gates over repository state, not unit tests: each runs as a job of its own, never inside a unit suite. An error fails its job; a warning never does.
|
|
8
8
|
- The diff, the coverage report and the merge-request comment are reports: they never fail a pipeline.
|
|
9
9
|
- `workflow-evidence` runs on merge requests to the default branch, and on default-branch pushes, where a red holds the release.
|
|
10
|
-
- The audit jobs accept exit 3 (`INCOMPLETE`) and 77 (`ADVISORY`, ERRORs at 80% confidence or lower, or a `FAIL` on an advisory run) as allowed failures; a gating `FAIL` (exit 1, an ERROR above 80%) blocks.
|
|
10
|
+
- The audit jobs accept exit 3 (`INCOMPLETE`) and 77 (`ADVISORY`, ERRORs at 80% confidence or lower, or a `FAIL` on an advisory run) as allowed failures; a gating `FAIL` (exit 1, an ERROR above 80%) blocks. The severity and confidence thresholds and the judge gate are variables (`AUDIT_SEVERITY_WARN`, `AUDIT_SEVERITY_ERROR`, `AUDIT_CONFIDENCE_SURE`, `AUDIT_CONFIDENCE_BLOCKING`, `AUDIT_CONFIDENCE_IGNORE_GAP`, `AUDIT_JUDGE_FROM`, `AUDIT_JUDGE_MIN_CONFIDENCE`, `AUDIT_JUDGE_READER_NUMBERS` — [contract.md](contract.md)); set them once for all audit jobs, so the merge, the judge pass and the comment use the same values.
|
|
11
11
|
- Report jobs depend on the audits with `dependencies:` and `when: always`, never `needs:` — a failed audit is exactly when its report matters.
|
|
12
12
|
- The default-branch pipeline publishes the reports on Pages, a failing one included; a report a run did not produce is omitted.
|
|
13
13
|
- The scheduled nightly sets `SPEC_AUDIT_NIGHTLY=true`, so `spec-tools tier` audits the corpus there.
|
|
14
14
|
- Reading jobs × workers is the pipeline's choice: `AUDIT_JOBS` reading jobs (default 1; a `parallel:` job sets the same number), each running `AUDIT_SLOTS` workers at once — as many as `AUDIT_JOB_MEMORY` holds, never more than `AUDIT_WORKERS` — and the scope is cut for one round across them. The reading and the judging hand off through files, so they run in one job (below) or as parallel reading jobs and a judge job that `needs:` them; `spec-tools workers` reads only its share in a parallel job. Set `AUDIT_JOB_MEMORY` once and give the AI jobs the same `KUBERNETES_MEMORY_LIMIT` (a YAML anchor keeps them one value; the runner must allow the overwrite); each pass logs the memory it used, which is what `AUDIT_WORKERS` is tuned from. Parallel reading jobs pay when one job cannot hold the round — a runner's default 2 GiB holds three workers; with the memory raised one job holds it, and the job that judges is the job that read: one pod, no artifact hop.
|
|
15
|
+
- The affected set of a merge request or a push is tuned by `AUDIT_AFFECTED_CONTEXT`, `AUDIT_AFFECTED_HOPS` and `AUDIT_AFFECTED_UNIT` ([contract.md](contract.md#the-affected-set)); a job sets them in its `variables:` to widen or narrow what it judges, and the scope line in its log names the values it used.
|
|
15
16
|
- spec-steward's corpus audit (`--audit spec-steward`) has no job: it is heavy, and its review file is the owner's to answer, locally.
|
|
16
17
|
- Declare what binds beyond spec-steward's defaults once, in `package.json`: `"spec-steward": { "binds": ["scripts/checks/**", "**/*.Dockerfile", ".gitlab-ci.yml"] }`.
|
|
17
18
|
|
|
@@ -80,7 +81,7 @@ spec-coverage:
|
|
|
80
81
|
paths: [.spec-audit/, node_modules/@kirill.konshin/lint/skills/]
|
|
81
82
|
|
|
82
83
|
# The audit, in one job: read (a failed reading leaves its parts to the completion pass), merge, complete what the
|
|
83
|
-
# reading left short, judge
|
|
84
|
+
# reading left short, judge the severe and sure findings, gate.
|
|
84
85
|
spec-verify:
|
|
85
86
|
stage: review
|
|
86
87
|
image: <an image with node>
|
|
@@ -127,4 +128,4 @@ render-reviews:
|
|
|
127
128
|
|
|
128
129
|
A release job that must wait for the audit `needs:` the judge (`spec-verify`), so a gating `FAIL` on the default branch holds it.
|
|
129
130
|
|
|
130
|
-
A skill-driven review writes `.spec-audit/<review>.yaml` in the shape `spec-tools run` checks (`findings[
|
|
131
|
+
A skill-driven review writes `.spec-audit/<review>.yaml` in the shape `spec-tools run` checks (`findings[readerSeverity, where, detail, readerConfidence, fields]`, `coverage{complete, notes}`). `verdict` computes its state from that file, never from the model's words, and writes `.spec-audit/<review>.md` beside it for the comment and the HTML.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Judging rules: common
|
|
2
2
|
|
|
3
|
-
These rules tell what a finding is and
|
|
3
|
+
These rules tell what a finding is, how severe it is, and how sure you are of it. They apply to every audit that spec-tools runs: spec-verify, its judge, spec-steward and check-entity-hacks. The skill and the tool tell how a run works.
|
|
4
4
|
|
|
5
5
|
## Evidence
|
|
6
6
|
|
|
@@ -21,23 +21,33 @@ These rules tell what a finding is and which tier it gets. They apply to every a
|
|
|
21
21
|
- Code that does not run in production has no production effect. This includes tests, local-only and development-only modes, demo agents, and internal CI or developer scripts. The exception: such code lets a production defect through now, and you show an instance.
|
|
22
22
|
- A difference in words, names, structure or style has no production effect when the behaviour stays the same.
|
|
23
23
|
|
|
24
|
-
##
|
|
24
|
+
## Severity
|
|
25
25
|
|
|
26
|
-
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
26
|
+
- Give each finding a severity, an integer from 0 to 100, on this scale:
|
|
27
|
+
|
|
28
|
+
| Band | Severity | The finding is |
|
|
29
|
+
| --- | --- | --- |
|
|
30
|
+
| ERROR | 90–100 | data loss, a leak of sensitive data, a security hole, or a main flow that fails in production |
|
|
31
|
+
| ERROR | 70–89 | a demonstrated defect with a production effect: a flow fails, a contract or an invariant is broken, or users or operators see a performance problem |
|
|
32
|
+
| WARN | 40–69 | a suspected defect; a demonstrated defect with no production effect; a conflict between rules |
|
|
33
|
+
| INFO | 1–39 | a state or a risk for a reviewer: a duplicate rule, a test that asserts less than its scenario, a wording problem. It is never a failure. |
|
|
34
|
+
| — | 0 | not true |
|
|
35
|
+
|
|
36
|
+
- For a severity in the ERROR band, write the production scenario in one sentence: what occurs, to whom, and what breaks, leaks or becomes slow.
|
|
37
|
+
- The tool turns severity and confidence into ERROR, WARN or INFO with its thresholds. The bands above are its defaults; a run can move them. Give the severity that the finding has. Do not aim at a band.
|
|
38
|
+
- A finding against a requirement marked `**⚠️ Advisory:**` is in the WARN band or lower.
|
|
39
|
+
- Use the effect to find the severity. The size of the difference does not decide the severity.
|
|
40
|
+
- Find if the defect is on a critical path or is an edge case. A defect that is not on a critical path, or an edge case with a small effect, is in the WARN band. An edge case that can cause a real production defect can be in the ERROR band.
|
|
31
41
|
|
|
32
42
|
## Confidence
|
|
33
43
|
|
|
34
|
-
- Give each finding
|
|
44
|
+
- Give each finding a confidence, an integer from 0 to 100. It tells how sure you are that the defect exists as you write it, and that its severity is correct.
|
|
35
45
|
- Use a value below 50 when you could not read a side, or when the effect needs a condition that you could not confirm.
|
|
36
|
-
-
|
|
46
|
+
- Put your doubt in the confidence, not in the severity. An unsure ERROR has an ERROR-band severity and a low confidence.
|
|
37
47
|
|
|
38
48
|
## Scope
|
|
39
49
|
|
|
40
|
-
- Report each defect one time, at its final
|
|
50
|
+
- Report each defect one time, at its final severity.
|
|
41
51
|
- Do not report a thing that a deterministic tool reports: coverage, citations, gates, lint.
|
|
42
52
|
- Do not report style preferences.
|
|
43
53
|
|
|
@@ -4,14 +4,15 @@ Use the common rules first: [common.md](common.md).
|
|
|
4
4
|
|
|
5
5
|
Be aware of the reader rules: [spec-verify.md](spec-verify.md), but focus on the rules below in THIS file, you MUST follow them strictly.
|
|
6
6
|
|
|
7
|
-
You judge one finding.
|
|
7
|
+
You judge one finding. Give your own reading of it: `judgeSeverity` and `judgeConfidence`, on the scales of the common rules. The tool combines your reading with the reader's. Your severity can be lower or higher than the reader's.
|
|
8
8
|
|
|
9
|
-
- Read the requirement, the
|
|
10
|
-
-
|
|
9
|
+
- Read the requirement, the quoted sources, and the code near them.
|
|
10
|
+
- Be pragmatic. Judge the production effect that the repository shows. Give a severity in the ERROR band only to a defect that a real user or operator meets on a path that the code takes. A theoretical path, an edge case with no trigger in the code, or a difference with no effect has a severity in the WARN band or lower.
|
|
11
|
+
- Give a severity in the ERROR band only when all three statements are true:
|
|
11
12
|
- The quotes exist, and they say what the finding tells.
|
|
12
13
|
- The code does not satisfy the intent of the requirement. The difference is not in words only. Other code does not resolve it. It does not come from a dependency that the repository does not show.
|
|
13
|
-
- The defect has a production effect. Write the production scenario
|
|
14
|
-
-
|
|
15
|
-
- Give `judgeConfidence`, an integer from 0 to 100. It tells how sure you are that the
|
|
16
|
-
-
|
|
17
|
-
- The tool keeps
|
|
14
|
+
- The defect has a production effect. Write the production scenario in `scenario`. If you cannot write it, give a severity in the WARN band or lower.
|
|
15
|
+
- When a statement is not true, name it in `reason`. When the finding is not true, give severity 0.
|
|
16
|
+
- Give `judgeConfidence`, an integer from 0 to 100. It tells how sure you are that the defect exists as you read it, and that your severity is correct.
|
|
17
|
+
- When the brief shows the reader's numbers, do not copy them. Make your own judgement.
|
|
18
|
+
- The tool keeps an ERROR-band severity without a `scenario` below the ERROR band.
|