@journeykit/canon 0.14.0 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/skills/SKILL.src.md +15 -5
- package/dist/skills/reference/canon-gate.md +108 -14
- package/dist/skills/reference/canon-policy.md +84 -4
- package/dist/skills/reference/canon-review.md +125 -15
- package/dist/skills/reference/canon-storage.md +4 -1
- package/dist/skills/reference/canon-subject.md +23 -7
- package/package.json +4 -4
package/dist/skills/SKILL.src.md
CHANGED
|
@@ -17,8 +17,9 @@ a feature, a one-line request included. Two points stop for a human:
|
|
|
17
17
|
|
|
18
18
|
- **The brief** (step 1): the request, your assumptions, your open questions.
|
|
19
19
|
- **The policy**: `.canon/policy.yaml` (written by `canon init`) is what the
|
|
20
|
-
gate grades the work against. A human approves it
|
|
21
|
-
|
|
20
|
+
gate grades the work against. A human approves it with
|
|
21
|
+
`canon policy approve --by <who>` (and again after every change to it);
|
|
22
|
+
never edit it to make the gate pass, and never approve it yourself.
|
|
22
23
|
|
|
23
24
|
When no human can answer, write the question into the brief, proceed on the
|
|
24
25
|
assumption you state there, and say so in your final report.
|
|
@@ -43,18 +44,27 @@ assumption you state there, and say so in your final report.
|
|
|
43
44
|
so the ledger shows which unit did what. A delegated unit attests its own
|
|
44
45
|
work; never attest another unit's work under your id.
|
|
45
46
|
5. **Implement.** Write the code with tests whose names carry the scenario
|
|
46
|
-
id (`game.run.01: …`), so a test report binds to its scenario.
|
|
47
|
+
id (`game.run.01: …`), so a test report binds to its scenario. Record a
|
|
48
|
+
defect you find in your own work as a finding the moment you find it,
|
|
49
|
+
before review:
|
|
50
|
+
`canon finding add --change-id <slug> --round <n> --seq <n> --severity <s> --reviewer <me> --actor-id <me> --session-id <session> --summary "…"`.
|
|
47
51
|
6. **Evidence.** Per scenario:
|
|
48
52
|
`canon evidence add --project-id <root-id> --scenario-id <id> --kind test-run --role implementer --ref "<command>" --actor-id <unit> --session-id <session>`,
|
|
49
53
|
with `--report junit:<path>` or `--artifact <path>` when files prove it.
|
|
50
54
|
canon stores those bytes in `.canon/artifacts/sha256/` (commit it with the
|
|
51
55
|
ledger) and `canon gate check` re-checks them. Then `canon gate promote`.
|
|
56
|
+
An external evaluator's audit is recorded with `canon evidence add --kind
|
|
57
|
+
audit`; canon never generates one.
|
|
58
|
+
Evidence for a plan task (`--task <id>`, before `canon gate task`) must
|
|
59
|
+
bind the artifact or report that proves the task; a canon command
|
|
60
|
+
(`canon status`, `canon gate check`) is never that proof.
|
|
52
61
|
7. **Independent review.** A different session from the one that wrote the
|
|
53
|
-
code and evidence reviews the change.
|
|
54
|
-
|
|
62
|
+
code and evidence reviews the change. The reviewer records every defect
|
|
63
|
+
the review finds as a finding:
|
|
55
64
|
`canon finding add --change-id <slug> --round <n> --seq <n> --severity <blocker|should-fix|note> --reviewer <me> --actor-id <me> --session-id <session> --summary "…"`,
|
|
56
65
|
then `canon gate promote`; once fixed,
|
|
57
66
|
`canon finding close --change-id <slug> --round <n> --seq <n> --disposition fixed --resolution-sha <sha>`
|
|
67
|
+
(a short sha is fine; canon resolves it)
|
|
58
68
|
and `canon gate promote`. Every bug fix adds or extends a scenario. The
|
|
59
69
|
reviewer signs off each scenario:
|
|
60
70
|
`canon review add --project-id <root-id> --scenario-id <id> --reviewer <who> --actor-id <who> --role reviewer --session-id <review-session> --pin <sha> --upstream-ref <ref>`
|
|
@@ -15,14 +15,14 @@ release-scoped check (below) enforces it. A human-only `flagged` overlay
|
|
|
15
15
|
overrides the ladder — a flagged artifact is never green regardless of
|
|
16
16
|
evidence. (Author `policy.yaml`'s trust/staleness fields via `canon-policy`.)
|
|
17
17
|
|
|
18
|
-
## The
|
|
18
|
+
## The ten failure classes
|
|
19
19
|
|
|
20
20
|
Every violation carries one of these stable, grep-able strings:
|
|
21
21
|
|
|
22
22
|
| Class | Meaning |
|
|
23
23
|
|---|---|
|
|
24
24
|
| `uncovered-cell` | Either a policy-required evidence cell (role × artifact) with no matching record, or — with `spec_coverage` enabled — a spec scenario that is unimplemented or mismatched, or a feature surface missing a `require_cases` case. The detail string distinguishes them. Coverage means "a test exists", not "a test passed": even a `Divergent` verdict satisfies the role-cell form. |
|
|
25
|
-
| `unreviewed-promotion` | An artifact tagged `reviewed` has no matching ledger review record — or, with `spec_coverage.require_review` set, an in-scope scenario has no qualifying `Review
|
|
25
|
+
| `unreviewed-promotion` | An artifact tagged `reviewed` has no matching ledger review record — or, with `spec_coverage.require_review` set, an in-scope scenario has no qualifying `Review`: none at all, only self-reviews, only stale reviews (the spec or evidence it pinned changed since), or only `indeterminate`/`unavailable` ones. See `canon-review`. |
|
|
26
26
|
| `trust-below-required` | Achieved trust level is below `policy.yaml`'s `trust_required` for its class — RELEASE-scoped only (`canon gate check --release`); never fires on an ordinary run. |
|
|
27
27
|
| `stale-evidence` | A passing record degraded to stale: its declared surface changed since its `evidence_sha`, or HEAD moved past `staleness.max_commits_behind`. Only degrades an already-green record. Also: a file bound to a latest evidence record whose bytes are gone — no intact blob in `.canon/artifacts/sha256/` and no working-tree copy with the recorded digest (see Bound evidence stays provable below). |
|
|
28
28
|
| `malformed-evidence` | A candidate record doesn't parse, is misfiled, or carries an unparseable interim tag. Malformed evidence is no evidence. |
|
|
@@ -30,6 +30,7 @@ Every violation carries one of these stable, grep-able strings:
|
|
|
30
30
|
| `unevidenced-flip` | `canon gate task <task_id>` was asked to flip a checkbox with no matching, non-`Divergent` evidence record. |
|
|
31
31
|
| `fabricated-evidence` | An evidence note contains a blocklisted marker (`"would pass"`, `"TBD"`, `"n/a"`) or a bare `verified` claim with no attached command result. |
|
|
32
32
|
| `open-blocker` | With `spec_coverage.require_review` set (and `block_on_findings`, the default), a Finding whose latest version is severity `blocker`, disposition `open`, on a change an in-scope subject adopted. Subject string `<change_id>#<round>.<seq>`. See `canon-review`. |
|
|
33
|
+
| `false-success` | The latest audit record (`canon evidence add --kind audit`) of a scenario says `refuted`: an external evaluator found a counterexample to the claimed success. Always on, regardless of policy or sampling, and takes precedence over any declared faithful verdict; only a newer audit (`confirmed` or `indeterminate`) clears it. The detail names the evaluator, its version, its sha256 and the counterexample paths. |
|
|
33
34
|
|
|
34
35
|
## `canon gate check [--repo <dir>] [--release]`
|
|
35
36
|
|
|
@@ -50,6 +51,12 @@ Exit `0` clean, `1` gate-red (any violation), `2` usage/load failure
|
|
|
50
51
|
resolves through the nearest-ancestor `canon.yaml` walk, so it reads the
|
|
51
52
|
repo ROOT's `.canon/policy.yaml` and `.canon/ledger` from any subdirectory.
|
|
52
53
|
|
|
54
|
+
When `.canon/policy.yaml` exists but has no approval, or its bytes no
|
|
55
|
+
longer match the approved digest, the gate prints `advisory: policy not
|
|
56
|
+
approved: …` or `advisory: policy changed since approval: …` after its
|
|
57
|
+
result, naming `canon policy approve --by <who>`; the exit code is
|
|
58
|
+
unchanged (see `canon-policy`).
|
|
59
|
+
|
|
53
60
|
### The spec-derived worklist (`spec_coverage`, opt-in)
|
|
54
61
|
|
|
55
62
|
Coverage groups the EVIDENCE corpus, so it can only ever enumerate
|
|
@@ -111,14 +118,18 @@ A present section, even `require_evidence: false` or a malformed one
|
|
|
111
118
|
- **Independent review** (`require_review`) — scenarios whose subject is
|
|
112
119
|
in `require_review.scope` (default `[verifying, shipped]`; `[]` = every
|
|
113
120
|
scenario; independent of `scope` above) need a `Review` record
|
|
114
|
-
(`canon review add`)
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
(default `true`)
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
121
|
+
(`canon review add`) whose latest version per reviewer is `signed`,
|
|
122
|
+
still matches the scenario digest and latest evidence it pinned
|
|
123
|
+
(a review written before 0.15 has no pins and skips this check), and,
|
|
124
|
+
when `distinct_actor` (default `true`), is by a reviewer and actor
|
|
125
|
+
other than the scenario's evidence actor and from a session other than
|
|
126
|
+
any evidence session when both carry one (`--session-id`);
|
|
127
|
+
gaps are `unreviewed-promotion`, and the detail says whether the
|
|
128
|
+
review is missing, a self-review, stale (naming whether the spec, the
|
|
129
|
+
evidence or both changed), or not signed. With `block_on_findings`
|
|
130
|
+
(default `true`), an open `blocker` Finding on a change an in-scope
|
|
131
|
+
subject adopted is `open-blocker`. `exclude_lanes` also exempts
|
|
132
|
+
scenarios from review. Full contract: `canon-review`.
|
|
122
133
|
|
|
123
134
|
```yaml
|
|
124
135
|
spec_coverage:
|
|
@@ -129,12 +140,28 @@ A present section, even `require_evidence: false` or a malformed one
|
|
|
129
140
|
block_on_findings: true # default
|
|
130
141
|
```
|
|
131
142
|
|
|
143
|
+
- **Sampled audits** (`require_audit`) — a deterministic sample of
|
|
144
|
+
in-scope scenarios (`require_audit.scope`, default `[verifying,
|
|
145
|
+
shipped]`; `exclude_lanes` applies) must carry an external audit
|
|
146
|
+
(`canon evidence add --kind audit`, see Audit records below). A sampled
|
|
147
|
+
scenario whose latest audit is missing or `indeterminate` is
|
|
148
|
+
`uncovered-cell`:
|
|
149
|
+
|
|
150
|
+
```
|
|
151
|
+
uncovered-cell cart.add.01 — sampled for audit by spec_coverage.require_audit (rate 0.1, seed `s1`) and has no audit record — …
|
|
152
|
+
uncovered-cell cart.add.01 — sampled for audit by spec_coverage.require_audit (rate 0.1, seed `s1`), but its latest audit is indeterminate (evaluator `judge` 1.2) — …
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Keys, defaults and the exact sampling formula: `canon-policy`.
|
|
156
|
+
|
|
132
157
|
Two refusals rather than a silent pass, both surfacing as
|
|
133
158
|
`uncovered-cell` on the subject `spec_coverage`:
|
|
134
159
|
|
|
135
160
|
- A **malformed section** (unknown key, unknown `scope` status, or a
|
|
136
|
-
malformed `require_cases`/`require_review` — unknown
|
|
137
|
-
`null`, non-mapping
|
|
161
|
+
malformed `require_cases`/`require_review`/`require_audit` — unknown
|
|
162
|
+
key, bad status, `null`, non-mapping, a missing or out-of-range
|
|
163
|
+
`rate`) refuses at check time (`uncovered-cell spec_coverage — ...
|
|
164
|
+
unusable`). It is never treated as
|
|
138
165
|
absent — a typo must not be silently equivalent to not opting in.
|
|
139
166
|
- A **corpus kind routed off the gate's rung** (`scenario`,
|
|
140
167
|
`divergence`, or `subject` sent anywhere but `local`) refuses, because
|
|
@@ -144,6 +171,54 @@ Malformed rows in those three kinds do NOT surface as
|
|
|
144
171
|
`malformed-evidence`; they are kept off the evidence violation set on
|
|
145
172
|
purpose, so enabling nothing changes no existing verdict.
|
|
146
173
|
|
|
174
|
+
### Audit records (`--kind audit`)
|
|
175
|
+
|
|
176
|
+
canon never generates or runs an audit. An external evaluator does, and
|
|
177
|
+
canon records its result:
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
canon evidence add --kind audit --project-id root --scenario-id cart.add.01 \
|
|
181
|
+
--ref "judge run 42" --role reviewer \
|
|
182
|
+
--evaluator-id judge --evaluator-version 1.2 \
|
|
183
|
+
--evaluator-digest <64 lowercase hex sha256> \
|
|
184
|
+
--audit-result refuted --report junit:audit/counterexample.xml
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
- `--kind audit` requires `--evaluator-id`, `--evaluator-version`,
|
|
188
|
+
`--evaluator-digest` (sha256, 64 lowercase hex), `--audit-result
|
|
189
|
+
confirmed|refuted|indeterminate`, and `--scenario-id` with
|
|
190
|
+
`--project-id`. It refuses `--task`, `--verdict`, `--surface-ref` and
|
|
191
|
+
`--approval-*`; the four evaluator flags are refused on any other kind.
|
|
192
|
+
All refusals exit `2`.
|
|
193
|
+
- The record carries `audit: {evaluator_id, evaluator_version,
|
|
194
|
+
evaluator_digest, result}`; its verdict is derived (`confirmed` →
|
|
195
|
+
faithful, `refuted` → divergent, `indeterminate` → not-applicable).
|
|
196
|
+
- Optional `--report`/`--artifact` counterexamples are bound and stored
|
|
197
|
+
like any report; the latest audit's bindings are re-hashed by the
|
|
198
|
+
artifact check.
|
|
199
|
+
- Audits fold apart: per `(project_id, scenario_id)` there is a latest
|
|
200
|
+
declared record and a latest audit. An audit never becomes the
|
|
201
|
+
scenario's latest declared verdict.
|
|
202
|
+
- `canon subject status <id> shipped` refuses `false-success`, and an
|
|
203
|
+
owed sampled audit when `require_audit.scope` covers `shipped`.
|
|
204
|
+
- `canon status` shows an `audits` section (sampled / confirmed / owed /
|
|
205
|
+
refuted) and suggests `canon evidence add --kind audit …` in `next:`.
|
|
206
|
+
|
|
207
|
+
### Dropping a binding (`--drop-attachment`)
|
|
208
|
+
|
|
209
|
+
`canon evidence add` for a scenario refuses (exit `2`) when the
|
|
210
|
+
scenario's current latest record of the same fold (declared records, or
|
|
211
|
+
audit records for an audit) — committed ledger plus `_staging` — binds
|
|
212
|
+
files the new record does not. The refusal names the paths and two ways
|
|
213
|
+
out: re-bind them (`--artifact`/`--report`; new bytes at the same path
|
|
214
|
+
are fine), or drop them on purpose with `--drop-attachment <path>`
|
|
215
|
+
(repeatable) and one single-line `--drop-reason <text>`. The drop is
|
|
216
|
+
recorded as `attachment_drop: {paths, reason}`. Refused (`2`): a drop
|
|
217
|
+
without a reason, a reason without a drop, a drop path that is not
|
|
218
|
+
actually dropped, a multi-line reason, or a drop without
|
|
219
|
+
`--scenario-id` (task-only evidence has no drop guard). `canon status`
|
|
220
|
+
lists recorded drops (`dropped bindings (N):`; `drops[]` in `--json`).
|
|
221
|
+
|
|
147
222
|
### Review-waiver advisories
|
|
148
223
|
|
|
149
224
|
A subject moved into a review-scoped status with `canon subject status
|
|
@@ -160,6 +235,17 @@ review waivers: 1 advisory(ies) — not failing the gate:
|
|
|
160
235
|
|
|
161
236
|
The waiver lasts only until the subject's next status write.
|
|
162
237
|
|
|
238
|
+
### Reviews that predate pinning
|
|
239
|
+
|
|
240
|
+
With `require_review` on, the reviews in effect (latest per reviewer and
|
|
241
|
+
scenario) that carry no pins were written before 0.15. They keep
|
|
242
|
+
counting, and one advisory line after the violations counts them; the
|
|
243
|
+
exit code is unchanged:
|
|
244
|
+
|
|
245
|
+
```
|
|
246
|
+
advisory: 340 review(s) predate review pinning (0.15) and still count without a check that their scenario's spec and evidence are unchanged; a new `canon review add` pins what it saw
|
|
247
|
+
```
|
|
248
|
+
|
|
163
249
|
## Experimental: evidence binding (`experimental.evidence_binding`, off by default)
|
|
164
250
|
|
|
165
251
|
An evidence record is an attestation: canon never runs `--ref`, and it
|
|
@@ -313,6 +399,14 @@ The success line is `canon gate task: <task_id> flipped — written
|
|
|
313
399
|
directly; nothing to promote`. An already-`[x]` row is an idempotent no-op (exit `0`). An unknown
|
|
314
400
|
`task_id` is reported (exit `1`).
|
|
315
401
|
|
|
402
|
+
A flip on evidence that cannot prove the task still succeeds, but warns
|
|
403
|
+
on stderr: when none of the task's evidence records binds a file and one
|
|
404
|
+
of them cites a canon command as its `--ref` (`canon gate check`,
|
|
405
|
+
`canon status`), the warning says to bind the artifact or report that
|
|
406
|
+
proves the task (`canon evidence add --task <id> --artifact <path>` or
|
|
407
|
+
`--report <format>:<path>`). A canon command reports on canon's own
|
|
408
|
+
records, so it is never the proof of a task.
|
|
409
|
+
|
|
316
410
|
```bash
|
|
317
411
|
canon gate task my-change#5.2 --repo .
|
|
318
412
|
```
|
|
@@ -367,8 +461,8 @@ Prefer this over hand-editing `settings.json`/`hooks.json`.
|
|
|
367
461
|
|
|
368
462
|
## `canon gate selftest`
|
|
369
463
|
|
|
370
|
-
Runs the shipped fixture corpus — one fixture per failure class (
|
|
371
|
-
including `
|
|
464
|
+
Runs the shipped fixture corpus — one fixture per failure class (ten,
|
|
465
|
+
including `false-success`), each a
|
|
372
466
|
deliberately broken corpus proving that class fires and ONLY that class
|
|
373
467
|
(both under-detection and over-triggering fail the run). Takes no
|
|
374
468
|
`--repo`; self-contained. Run it before trusting any other `canon gate
|
|
@@ -37,10 +37,56 @@ spec_coverage:
|
|
|
37
37
|
|
|
38
38
|
The file carries comments explaining each field. It loads with zero
|
|
39
39
|
diagnostics. It is what `canon gate check` grades the work against, so a
|
|
40
|
-
human should review and approve it, and every later change to it
|
|
41
|
-
agent doing the work should not write its own grading
|
|
42
|
-
with `scope: [building, verifying]` to check evidence
|
|
43
|
-
in those statuses (see `canon-gate`).
|
|
40
|
+
human should review and approve it, and every later change to it (next
|
|
41
|
+
section). The agent doing the work should not write its own grading
|
|
42
|
+
policy. Narrow it with `scope: [building, verifying]` to check evidence
|
|
43
|
+
only for subjects in those statuses (see `canon-gate`).
|
|
44
|
+
|
|
45
|
+
## Approving the policy
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
canon policy approve --by <who> [--note <text>] [--repo <dir>]
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
A human runs this after reviewing `.canon/policy.yaml`. It writes
|
|
52
|
+
`.canon/policy.approval.json` directly (nothing to promote):
|
|
53
|
+
|
|
54
|
+
```json
|
|
55
|
+
{
|
|
56
|
+
"policy_digest": "sha256:<sha256 of the policy.yaml bytes>",
|
|
57
|
+
"approved_by": "<who>",
|
|
58
|
+
"approved_at": "2026-10-11T09:00:00Z",
|
|
59
|
+
"note": "<text, or null>"
|
|
60
|
+
}
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Commit it with the policy: the file is tracked, and neither canon's
|
|
64
|
+
`.gitignore` nor the one `canon init` writes ignores it. Running the
|
|
65
|
+
command again replaces the record. With no `.canon/policy.yaml` it is
|
|
66
|
+
refused (exit `2`), and so is an empty `--by`.
|
|
67
|
+
|
|
68
|
+
While a policy exists with no approval, `canon gate check` prints after
|
|
69
|
+
its result:
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
advisory: policy not approved: .canon/policy.yaml has no approval record (.canon/policy.approval.json); a human reviews it and runs `canon policy approve --by <who>`
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Once the policy's bytes no longer match the approved digest, it prints
|
|
76
|
+
this instead:
|
|
77
|
+
|
|
78
|
+
```
|
|
79
|
+
advisory: policy changed since approval: .canon/policy.yaml is sha256:<now>, but `<who>` approved sha256:<then> at <time>; a human reviews it and runs `canon policy approve --by <who>`
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
An approval file that does not parse reads
|
|
83
|
+
`advisory: policy approval unreadable: …`.
|
|
84
|
+
`canon status` prints the same text as a `warning:` line (and in
|
|
85
|
+
`--json` `warnings`). The advisory never changes an exit code: canon
|
|
86
|
+
cannot tell whether a human or an agent ran the command, so the record
|
|
87
|
+
names who approved which bytes, and the advisory says when that no
|
|
88
|
+
longer holds. An agent never runs `canon policy approve` on its own
|
|
89
|
+
behalf.
|
|
44
90
|
|
|
45
91
|
## The closed CEL profile
|
|
46
92
|
|
|
@@ -124,6 +170,40 @@ question rather than adding an expression field. Conditional logic about
|
|
|
124
170
|
what a record *means* lives in `policy.yaml`/`applies_when:`, evaluated
|
|
125
171
|
against the record's fields, never inside the record.
|
|
126
172
|
|
|
173
|
+
## Sampled audits: `spec_coverage.require_audit`
|
|
174
|
+
|
|
175
|
+
Optional; absent means no audit is ever owed (a recorded `refuted` audit
|
|
176
|
+
is still `false-success` regardless — see `canon-gate`).
|
|
177
|
+
|
|
178
|
+
```yaml
|
|
179
|
+
spec_coverage:
|
|
180
|
+
require_audit:
|
|
181
|
+
rate: 0.1 # required: a number in [0, 1]
|
|
182
|
+
seed: "2026-q4" # optional string; default ""
|
|
183
|
+
scope: [verifying, shipped] # optional subject statuses; this is the default; [] = every scenario
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
`spec_coverage.exclude_lanes` applies. A malformed `require_audit` —
|
|
187
|
+
unknown key, missing `rate`, `rate` out of range or a string, bad
|
|
188
|
+
`scope`, `null` or a non-mapping — poisons the whole `spec_coverage`
|
|
189
|
+
section: the gate reports `uncovered-cell spec_coverage — ... unusable`
|
|
190
|
+
rather than checking anything under it.
|
|
191
|
+
|
|
192
|
+
Sampling is deterministic. A scenario is in the sample iff the first 8
|
|
193
|
+
bytes of
|
|
194
|
+
|
|
195
|
+
```
|
|
196
|
+
sha256(seed || 0x00 || project_id || 0x00 || scenario_id)
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
read as a big-endian u64, are less than `rate · 2^64` — computed exactly,
|
|
200
|
+
as a comparison against `ceil(rate · 2^64)`, so `rate: 0` samples nothing
|
|
201
|
+
and `rate: 1` samples everything. A sampled in-scope scenario whose
|
|
202
|
+
latest audit is missing or `indeterminate` is `uncovered-cell`; with
|
|
203
|
+
`shipped` in `scope`, `canon subject status <id> shipped` also refuses
|
|
204
|
+
it. `canon context` and the `canon status` policy line show
|
|
205
|
+
`require_audit=(rate=… seed=`…` scope=…)`.
|
|
206
|
+
|
|
127
207
|
## Effect-aware risk tiers
|
|
128
208
|
|
|
129
209
|
The optional `risk_tiers` section is not CEL. Absent means no risk-approval
|
|
@@ -34,7 +34,8 @@ The order an agent follows to claim a subject done:
|
|
|
34
34
|
```
|
|
35
35
|
|
|
36
36
|
3. **One Review per scenario**, by someone other than the evidence actor,
|
|
37
|
-
from a session other than the one that attested the evidence
|
|
37
|
+
from a session other than the one that attested the evidence, after
|
|
38
|
+
the evidence is in (the review pins what it saw):
|
|
38
39
|
|
|
39
40
|
```bash
|
|
40
41
|
canon review add --project-id root --scenario-id world.demo.01 \
|
|
@@ -43,7 +44,11 @@ The order an agent follows to claim a subject done:
|
|
|
43
44
|
--pin "$(git rev-parse HEAD)" --upstream-ref "pr#41"
|
|
44
45
|
```
|
|
45
46
|
|
|
46
|
-
`review add` writes directly; there is nothing to promote.
|
|
47
|
+
`review add` writes directly; there is nothing to promote. If the
|
|
48
|
+
reviewer could not reach a verdict, record that instead of signing:
|
|
49
|
+
`--outcome indeterminate` (or `unavailable` when it could not run).
|
|
50
|
+
Changing the scenario or re-attesting it afterwards makes the review
|
|
51
|
+
stale; review it again.
|
|
47
52
|
|
|
48
53
|
4. **Move the subject**: `canon subject status <id> verifying`, later
|
|
49
54
|
`shipped`. See `canon-subject`.
|
|
@@ -73,14 +78,24 @@ appends a second version at the same natural key — the first stays. Its
|
|
|
73
78
|
`--actor-id` and `--session-id` name who authored the transition; the
|
|
74
79
|
raiser's stay on the first version.
|
|
75
80
|
|
|
81
|
+
Every sha flag (`--reviewed-sha`, `--resolution-sha`, `--introduced-by`)
|
|
82
|
+
takes a full 40-character sha or a short one (4 or more lowercase hex
|
|
83
|
+
characters). A short sha is resolved with `git rev-parse --verify
|
|
84
|
+
<sha>^{commit}` in the repo, and the record stores the full sha. It is
|
|
85
|
+
refused (exit `2`) when it names no commit, when more than one commit
|
|
86
|
+
starts with it (pass more characters), when it only resolves by peeling a
|
|
87
|
+
tag, or outside a git work tree.
|
|
88
|
+
|
|
76
89
|
### A defect the author found in its own work
|
|
77
90
|
|
|
78
|
-
Record it like any other finding,
|
|
79
|
-
|
|
80
|
-
the
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
91
|
+
Record it like any other finding, at the moment you find it and before
|
|
92
|
+
review, with `--reviewer` and `--actor-id` set to your own id; never ask
|
|
93
|
+
the reviewer to record it later under theirs. Nothing refuses a finding
|
|
94
|
+
whose reviewer is also the implementer, and an open `blocker` the author
|
|
95
|
+
raised blocks under `block_on_findings` exactly like a reviewer's.
|
|
96
|
+
`--introduced-by` names the commit that introduced the defect, not who
|
|
97
|
+
found it: `--introduced-by self` is a usage error (exit 2) that points at
|
|
98
|
+
`--reviewer`.
|
|
84
99
|
|
|
85
100
|
### Severities
|
|
86
101
|
|
|
@@ -113,19 +128,66 @@ fixed finding reopened with `--disposition open` counts as open again.
|
|
|
113
128
|
```bash
|
|
114
129
|
canon review add --project-id <root-id> --scenario-id <id> \
|
|
115
130
|
--reviewer <who> --pin <sha-or-ref> --role <role> \
|
|
116
|
-
(--upstream-ref <ref> | --original-spec-ref <ref>) [--actor-id <id>] [--session-id <id>]
|
|
131
|
+
(--upstream-ref <ref> | --original-spec-ref <ref>) [--actor-id <id>] [--session-id <id>] \
|
|
132
|
+
[--outcome signed|indeterminate|unavailable] \
|
|
133
|
+
[--reviewer-provider <p>] [--reviewer-model <m>] [--reviewer-harness <h>]
|
|
117
134
|
```
|
|
118
135
|
|
|
119
136
|
One attributed Review record for one scenario, joined on
|
|
120
137
|
`(project_id, scenario_id)` — both are required because two spec roots
|
|
121
138
|
may carry the same scenario id. Exactly one provenance ref is required.
|
|
122
139
|
`--actor-id` defaults to `canon`; `--session-id` fills `actor.session_id`.
|
|
123
|
-
A Review carries no verdict; the `verifying → shipped`
|
|
124
|
-
refuses missing or `Divergent` evidence verdicts on its
|
|
125
|
-
written directly to the ledger, with no staging step, and says
|
|
126
|
-
`canon review add: wrote <path> (new) — written directly; nothing to
|
|
140
|
+
A Review carries no verdict about the behavior; the `verifying → shipped`
|
|
141
|
+
ship gate still refuses missing or `Divergent` evidence verdicts on its
|
|
142
|
+
own. It is written directly to the ledger, with no staging step, and says
|
|
143
|
+
so: `canon review add: wrote <path> (new) — written directly; nothing to
|
|
127
144
|
promote`.
|
|
128
145
|
|
|
146
|
+
### What the review saw (pins)
|
|
147
|
+
|
|
148
|
+
`review add` reads the corpus and records `pins` on the review; the
|
|
149
|
+
reviewer never types them:
|
|
150
|
+
|
|
151
|
+
- `scenario_digest` — the scenario record's `scenario_digest`: a sha256
|
|
152
|
+
over that scenario alone, as `canon inventory sync` stores it. It
|
|
153
|
+
covers the scenario's tags and block (header, steps, doc strings,
|
|
154
|
+
tables, `Examples:`) and the context it inherits: the Feature's tags,
|
|
155
|
+
the Feature `Background:`, and an enclosing `Rule:` with its tags and
|
|
156
|
+
`Background:`. Layout does not count: comments, blank lines, line
|
|
157
|
+
indentation, tag order, and table alignment (cells are compared
|
|
158
|
+
trimmed and unescaped). Values do: any step, tag or cell change, and
|
|
159
|
+
whitespace inside a doc string after the delimiter's own indentation
|
|
160
|
+
is removed. Other scenarios never count. A record synced before 0.15 has no
|
|
161
|
+
`scenario_digest`; then the review pins its `source_digest` (the
|
|
162
|
+
sha256 of the whole `.feature` file) and records
|
|
163
|
+
`scenario_digest_scope: file`;
|
|
164
|
+
- `evidence` — the scenario's latest evidence record per role, each as
|
|
165
|
+
`{key, digest}`: the ledger file `kind=evidence_record/<key>__<digest>.json`.
|
|
166
|
+
A scenario with no evidence yet is pinned with an empty list.
|
|
167
|
+
|
|
168
|
+
It prints `canon review add: pinned <id> at scenario digest <12 hex> and
|
|
169
|
+
<n> latest evidence record(s); …` (`at source digest` for a file pin). A
|
|
170
|
+
scenario with no record (never synced) is refused with exit 2 and nothing
|
|
171
|
+
is written: there is nothing to pin.
|
|
172
|
+
|
|
173
|
+
The gate compares each pin like for like: a scenario pin with the
|
|
174
|
+
scenario's current `scenario_digest`, a file pin with its current
|
|
175
|
+
`source_digest`. Adding or editing another scenario in the file leaves a
|
|
176
|
+
scenario-pinned review current; editing this scenario, or a Background
|
|
177
|
+
or Rule it inherits, makes it stale. A file-pinned review stays a file
|
|
178
|
+
pin until it is reviewed again. Review after the spec and evidence are
|
|
179
|
+
final.
|
|
180
|
+
|
|
181
|
+
### The reviewer's outcome and runtime
|
|
182
|
+
|
|
183
|
+
`--outcome` records how the reviewer's run ended: `signed` (the default:
|
|
184
|
+
an approval), `indeterminate` (it ran and reached no verdict) or
|
|
185
|
+
`unavailable` (it could not run). Only `signed` counts; the other two
|
|
186
|
+
are recorded so a failed review is visible, never mistaken for a
|
|
187
|
+
sign-off or a rejection, and `review add` says so. `--reviewer-provider`,
|
|
188
|
+
`--reviewer-model` and `--reviewer-harness` record the runtime that
|
|
189
|
+
produced the review, as given. An empty or padded label is a usage error.
|
|
190
|
+
|
|
129
191
|
`--session-id` (on `evidence add`, `review add`, `finding add` and
|
|
130
192
|
`finding close`) takes one opaque token: non-empty, no surrounding
|
|
131
193
|
whitespace, no control characters. Anything else is a usage error
|
|
@@ -169,9 +231,41 @@ the starter policy `canon init` writes.
|
|
|
169
231
|
|
|
170
232
|
## How it meets the gate
|
|
171
233
|
|
|
234
|
+
### Which review counts
|
|
235
|
+
|
|
236
|
+
The reviews of a scenario are folded to the **latest per `reviewer`**
|
|
237
|
+
first (by `at`, then schema, then content digest — the fold every reader
|
|
238
|
+
uses). A reviewer's later review replaces their earlier one: a reviewer
|
|
239
|
+
who signed and later recorded `indeterminate` no longer counts as
|
|
240
|
+
signing, and a later `signed` review counts again. Reviews by different
|
|
241
|
+
reviewers never replace each other.
|
|
242
|
+
|
|
243
|
+
The scenario is reviewed when at least one of those latest reviews:
|
|
244
|
+
|
|
245
|
+
1. is `signed`;
|
|
246
|
+
2. still matches its pins — the scenario record's current
|
|
247
|
+
`source_digest`, and its current latest evidence record per role —
|
|
248
|
+
when it has pins;
|
|
249
|
+
3. passes `distinct_actor`, when that is on.
|
|
250
|
+
|
|
251
|
+
A review written before 0.15 has no pins. It keeps counting under rules
|
|
252
|
+
1 and 3, without the freshness check, and `canon gate check` prints one
|
|
253
|
+
advisory line counting such in-effect reviews (exit code unchanged):
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
advisory: 340 review(s) predate review pinning (0.15) and still count without a check that their scenario's spec and evidence are unchanged; a new `canon review add` pins what it saw
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
`canon status` prints the same line as a warning, and `canon subject
|
|
260
|
+
status` prints it on stderr for the subject's own scenarios. Reviewing
|
|
261
|
+
the scenario again pins it and drops it from the count.
|
|
262
|
+
|
|
172
263
|
### `unreviewed-promotion`
|
|
173
264
|
|
|
174
|
-
An in-scope scenario with no qualifying review
|
|
265
|
+
An in-scope scenario with no qualifying review. When several latest
|
|
266
|
+
reviews fail for different reasons, the detail names the one closest to
|
|
267
|
+
counting: a current signed review failing only `distinct_actor`, then a
|
|
268
|
+
stale review, then non-signed outcomes.
|
|
175
269
|
|
|
176
270
|
```
|
|
177
271
|
unreviewed-promotion world.demo.01 — no review record for this scenario; spec_coverage.require_review wants an independent review (`canon review add`) once its subject is claimed done
|
|
@@ -190,6 +284,22 @@ evidence:
|
|
|
190
284
|
unreviewed-promotion world.demo.01 — every review of this scenario is by an actor that also attested its evidence (`impl-agent`) or from a session that did (`sess-impl`); spec_coverage.require_review.distinct_actor wants a reviewer other than the evidence actor, in a session other than any evidence session
|
|
191
285
|
```
|
|
192
286
|
|
|
287
|
+
or, when the scenario's spec, its evidence, or both changed after the
|
|
288
|
+
review (`its spec changed`, `its evidence changed`, `its spec and its
|
|
289
|
+
evidence changed`):
|
|
290
|
+
|
|
291
|
+
```
|
|
292
|
+
unreviewed-promotion world.demo.01 — stale review by `reviewer-2`: its evidence changed (the latest evidence records are not the pinned ones) since the review; a pinned review counts only while what it saw is current — review it again (`canon review add`)
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
or, when every latest review recorded no sign-off:
|
|
296
|
+
|
|
297
|
+
```
|
|
298
|
+
unreviewed-promotion world.demo.01 — no signed review: the latest review recorded no sign-off (`reviewer-2` indeterminate); an indeterminate or unavailable review is a gap, never an approval — run the review again and record `--outcome signed`
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
A non-signed review is a gap even without `distinct_actor`.
|
|
302
|
+
|
|
193
303
|
### `open-blocker`
|
|
194
304
|
|
|
195
305
|
With `block_on_findings`, a Finding whose latest version is severity
|
|
@@ -222,7 +332,7 @@ prints on stderr which review checks it ran or skipped:
|
|
|
222
332
|
|
|
223
333
|
```
|
|
224
334
|
canon subject status: review guard for building → verifying (spec_coverage.require_review, scope: verifying, shipped)
|
|
225
|
-
ran unreviewed-promotion — every owned scenario needs a review by an actor, and from a session, other than its evidence's
|
|
335
|
+
ran unreviewed-promotion — every owned scenario needs a signed review, current with the spec and evidence it pinned, by an actor, and from a session, other than its evidence's
|
|
226
336
|
ran open-blocker — 1 adopted change(s) checked for open blocker findings
|
|
227
337
|
```
|
|
228
338
|
|
|
@@ -146,7 +146,10 @@ query:
|
|
|
146
146
|
```
|
|
147
147
|
|
|
148
148
|
Every query, export, and purge appends an actor-stamped record to
|
|
149
|
-
`.canon/audit/access.jsonl` (mode `0600` on creation)
|
|
149
|
+
`.canon/audit/access.jsonl` (mode `0600` on creation) in the repository
|
|
150
|
+
whose `canon.yaml` it read (through `--repo` or `--canon-yaml`), never in
|
|
151
|
+
the current directory; with no `canon.yaml` there is no audit and nothing
|
|
152
|
+
is written. `CANON_ACTOR` takes
|
|
150
153
|
precedence over the OS username. Audit rows contain operation, kind, scope,
|
|
151
154
|
sensitive flag, counts, and timestamp only—never query text, secrets, or
|
|
152
155
|
record bodies.
|
|
@@ -162,7 +162,10 @@ unreviewed-promotion — `specced` is not in require_review.scope`. A
|
|
|
162
162
|
transition INTO an in-scope status with gaps — an owned scenario with no
|
|
163
163
|
qualifying review (`unreviewed-promotion`), or an open blocker finding
|
|
164
164
|
on an adopted change (`open-blocker`) — prints the violations, exits 1,
|
|
165
|
-
leaves the record unchanged, and names `--override-reason`.
|
|
165
|
+
leaves the record unchanged, and names `--override-reason`. `shipped`
|
|
166
|
+
also refuses a `false-success` (a scenario whose latest audit is
|
|
167
|
+
`refuted`) and, when `require_audit.scope` covers `shipped`, an owed
|
|
168
|
+
sampled audit; see `canon-gate`.
|
|
166
169
|
|
|
167
170
|
```bash
|
|
168
171
|
canon subject status demo-subject verifying \
|
|
@@ -274,12 +277,25 @@ uses, so status and the gate cannot disagree about a gap:
|
|
|
274
277
|
scenarios it owns through `@subject:` (after `exclude_lanes`, which
|
|
275
278
|
are counted separately), how many carry a ledger verdict (the
|
|
276
279
|
`verifying → shipped` gate's reading; divergent ones are counted too),
|
|
277
|
-
how many carry a review that satisfies `require_review` (any
|
|
278
|
-
when it is off; with `distinct_actor`, a self-review does
|
|
279
|
-
open blocker findings on its adopted changes, and each
|
|
280
|
-
missing a `require_cases` case. `(N due)` marks reviews
|
|
281
|
-
transition will require
|
|
280
|
+
how many carry a review that satisfies `require_review` (any current
|
|
281
|
+
signed review when it is off; with `distinct_actor`, a self-review does
|
|
282
|
+
not count), open blocker findings on its adopted changes, and each
|
|
283
|
+
feature surface missing a `require_cases` case. `(N due)` marks reviews
|
|
284
|
+
the next transition will require; `N stale review(s)` and `N unsigned
|
|
285
|
+
review(s)` (printed when non-zero) count scenarios whose review went
|
|
286
|
+
stale or whose latest reviews are all `indeterminate`/`unavailable`
|
|
287
|
+
(see `canon-review`), and the `canon review add` step names both. With
|
|
288
|
+
`require_review` on, a `warning:` line counts the in-effect reviews that
|
|
289
|
+
predate review pinning (0.15).
|
|
282
290
|
- **Unowned** — scenarios with no `@subject:` tag.
|
|
291
|
+
- **Audits** — how many scenarios are sampled for
|
|
292
|
+
`spec_coverage.require_audit`, confirmed, owed (no audit or
|
|
293
|
+
`indeterminate`) and refuted (`false-success`). An owed or
|
|
294
|
+
refuted audit adds a `next:` step suggesting `canon evidence add
|
|
295
|
+
--kind audit …` (recorded from an external evaluator; canon never runs
|
|
296
|
+
one).
|
|
297
|
+
- **Dropped bindings** — `dropped bindings (N):` lists every recorded
|
|
298
|
+
`--drop-attachment` (scenario, paths, reason); `drops[]` in `--json`.
|
|
283
299
|
- **`next:`** — at most five commands, each with a one-line reason; the
|
|
284
300
|
rest are counted. Rules, in order: no `canon.yaml` → `canon init`; an
|
|
285
301
|
unreadable one → `canon init --check-config`; a scenario tagged with a
|
|
@@ -320,7 +336,7 @@ Status fails closed rather than suggest progress the gate would refuse:
|
|
|
320
336
|
"warnings": [],
|
|
321
337
|
"subjects": [
|
|
322
338
|
{ "id": "cart", "title": "Cart", "status": "building", "scenarios": 2, "excluded": 0, "evidenced": 2, "divergent": 0,
|
|
323
|
-
"reviewed": 1, "reviewDue": true, "openBlockers": 1, "missingCases": [] }
|
|
339
|
+
"reviewed": 1, "staleReviews": 0, "unsignedReviews": 0, "reviewDue": true, "openBlockers": 1, "missingCases": [] }
|
|
324
340
|
],
|
|
325
341
|
"unowned": ["cart.promo.01"],
|
|
326
342
|
"next": [
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@journeykit/canon",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.15.0",
|
|
4
4
|
"description": "canon — harness knowledge substrate. JS launcher: platform detection + native binary resolution.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -30,9 +30,9 @@
|
|
|
30
30
|
},
|
|
31
31
|
"dependencies": {},
|
|
32
32
|
"optionalDependencies": {
|
|
33
|
-
"@journeykit/canon-core-darwin-arm64": "0.
|
|
34
|
-
"@journeykit/canon-core-linux-x64": "0.
|
|
35
|
-
"@journeykit/canon-core-win32-x64": "0.
|
|
33
|
+
"@journeykit/canon-core-darwin-arm64": "0.15.0",
|
|
34
|
+
"@journeykit/canon-core-linux-x64": "0.15.0",
|
|
35
|
+
"@journeykit/canon-core-win32-x64": "0.15.0"
|
|
36
36
|
},
|
|
37
37
|
"devDependencies": {
|
|
38
38
|
"@types/node": "^20.0.0",
|