@agent-ix/quoin 0.11.0 → 0.12.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{base-D8vwPhgW.js → base-CVgcB7WY.js} +1 -1
- package/dist/cli.d.ts +15 -0
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +11 -7
- package/dist/commands/catalog/list.js +2 -2
- package/dist/commands/catalog/show.js +2 -2
- package/dist/commands/catalog/validate.js +2 -2
- package/dist/commands/config/doctor.js +1 -1
- package/dist/commands/config/edit.js +1 -1
- package/dist/commands/config/get.js +1 -1
- package/dist/commands/config/set.js +1 -1
- package/dist/commands/matrix.js +1 -1
- package/dist/commands/module/ensure-defaults.js +3 -3
- package/dist/commands/module/install.js +2 -2
- package/dist/commands/module/list.js +2 -2
- package/dist/commands/module/remove.js +2 -2
- package/dist/commands/review.js +1 -1
- package/dist/commands/to-plan.js +1 -1
- package/dist/commands/update.js +2 -2
- package/dist/commands/write.js +3 -3
- package/dist/{flow-command-x_KT1XeX.js → flow-command-C6aaG6Jf.js} +1 -1
- package/dist/hooks/command-not-found.d.ts +23 -0
- package/dist/hooks/command-not-found.d.ts.map +1 -0
- package/dist/hooks/command-not-found.js +8 -0
- package/dist/index.js +6 -6
- package/dist/{modules-DtVoj0FW.js → modules-BORtSWpQ.js} +2 -2
- package/dist/{plugins-CpD3ilZ5.js → plugins-SrgT7Vs-.js} +1 -1
- package/dist/{version-BdYjwrt7.js → version-BWpgJrPz.js} +1 -1
- package/dist/{write-DTRGUdVY.js → write-R0lAN4cR.js} +1 -1
- package/package.json +6 -2
- package/skills/spec-correctness/SKILL.md +98 -0
- package/skills/spec-correctness/assets/property-queue-template.md +78 -0
- package/skills/spec-correctness/assets/strategy-table.md +101 -0
- package/skills/spec-correctness/assets/templates/fast-check.ts.md +153 -0
- package/skills/spec-correctness/assets/templates/hypothesis.py.md +155 -0
- package/skills/spec-correctness/assets/templates/proptest.rs.md +144 -0
- package/skills/spec-correctness/references/step-0-scope-and-harness.md +94 -0
- package/skills/spec-correctness/references/step-1-census.md +53 -0
- package/skills/spec-correctness/references/step-2-clause-grounding.md +159 -0
- package/skills/spec-correctness/references/step-3-strategy-selection.md +83 -0
- package/skills/spec-correctness/references/step-4-generate-tests.md +99 -0
- package/skills/spec-correctness/references/step-5-second-pass.md +79 -0
- package/skills/spec-correctness/references/step-6-review-queue.md +64 -0
- package/skills/spec-correctness/references/step-7-report-and-handoff.md +63 -0
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
# Step 2: Clause Grounding
|
|
2
|
+
|
|
3
|
+
**Goal**: turn each criterion's prose into three executable pieces — a **generator domain**,
|
|
4
|
+
a **precondition**, and an **oracle** — each cited to a real location.
|
|
5
|
+
|
|
6
|
+
This is the step the engine cannot do. It reads one statement at a time; you can read the
|
|
7
|
+
whole FR and the code. Spans, when present, are a starting hint; ~96% of criteria have none.
|
|
8
|
+
|
|
9
|
+
## The grounding record
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
{
|
|
13
|
+
row_id: "FR-012-AC-1",
|
|
14
|
+
symbol: "detectDuplicates",
|
|
15
|
+
module_path: "src/catalog.ts",
|
|
16
|
+
domain_expr: "fc.array(entryArb, {maxLength: 32})",
|
|
17
|
+
precondition_expr: null,
|
|
18
|
+
oracle_expr: "d.modules === [...d.modules].sort()",
|
|
19
|
+
evidence: [
|
|
20
|
+
"spec/functional/FR-012-detect-duplicate-types.md:31", // Inputs
|
|
21
|
+
"spec/functional/FR-012-detect-duplicate-types.md:44", // Behavior SHALL
|
|
22
|
+
"src/catalog.ts:88" // signature
|
|
23
|
+
]
|
|
24
|
+
}
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
**Rule: every one of domain / precondition / oracle that the strategy needs must have at
|
|
28
|
+
least one `file:line` citation. No citation → no test.** A precondition of `null` is fine
|
|
29
|
+
when the criterion is unconditional; a fabricated one is not.
|
|
30
|
+
|
|
31
|
+
## Read in this order
|
|
32
|
+
|
|
33
|
+
1. **The criterion's own row.** Open `record.document` at `record.line`. Read the whole
|
|
34
|
+
table row, including the `Verification` column — it often names the test file.
|
|
35
|
+
|
|
36
|
+
2. **The FR's structured sections.**
|
|
37
|
+
- `## Inputs` → **domain**. The bullets name the parameters and their carrier types.
|
|
38
|
+
"the assembled catalog and a `catalog validate` invocation with an optional `--json`
|
|
39
|
+
flag" → a catalog arbitrary × an optional boolean.
|
|
40
|
+
- `## Behavior` → **precondition** and **oracle**. Find the `SHALL` clause whose subject
|
|
41
|
+
matches the criterion's subject. Its `when` / `where` / `if` tail is the precondition;
|
|
42
|
+
the `SHALL` predicate is the oracle.
|
|
43
|
+
- `## Outputs` → the **observable** the oracle must be written in (return value, exit
|
|
44
|
+
status, stderr payload). Prefer the observable form. An oracle you cannot observe is
|
|
45
|
+
not an oracle.
|
|
46
|
+
- `## Constraints` / `## Options` → generator bounds. A stated min/max becomes a range
|
|
47
|
+
and a boundary-biased generator; an option list becomes a permutation axis.
|
|
48
|
+
|
|
49
|
+
3. **Sibling ACs of the same FR.** They partition the domain — AC-1 covers the duplicate
|
|
50
|
+
case, AC-2 the unique case. Use them to (a) avoid a generator that only ever hits one
|
|
51
|
+
branch, and (b) find the paired negative domain when the criterion is an `error-case`.
|
|
52
|
+
|
|
53
|
+
4. **The test file named in `Verification`.** An existing example test hands you the real
|
|
54
|
+
symbol, its import path, the constructed argument shapes, and the assertion vocabulary
|
|
55
|
+
already in use. Reuse its fixtures rather than inventing parallel ones.
|
|
56
|
+
|
|
57
|
+
5. **The code.** Grep for the command, flag, type, or function named in the statement.
|
|
58
|
+
- signature → the generator's concrete types, replacing the prose domain;
|
|
59
|
+
- error enum / exception class → the `error-case` oracle. This is the **only** source
|
|
60
|
+
for it: the engine emits `span:refused-weak-boundary` on these, meaning it declined to
|
|
61
|
+
guess;
|
|
62
|
+
- return type / struct fields → the `invariant` predicate;
|
|
63
|
+
- state enum plus its transition `match` / `switch` → the `lifecycle` state machine.
|
|
64
|
+
|
|
65
|
+
6. **Upstream StR/US** via the FR's `relationships:` targets — only when the FR states its
|
|
66
|
+
oracle in terms it does not itself define.
|
|
67
|
+
|
|
68
|
+
## Using non-null spans
|
|
69
|
+
|
|
70
|
+
When `domain`, `precondition`, or `oracle` is present, `statement[start..end] == text` and
|
|
71
|
+
the spans are in-bounds, non-overlapping and ascending. Use `domain.text` to seed the
|
|
72
|
+
generator's subject, `precondition.text` as the filter, `oracle.text` as the assertion —
|
|
73
|
+
but still do read 5. A span gives you the clause, not an expression.
|
|
74
|
+
|
|
75
|
+
## Refuse to ground
|
|
76
|
+
|
|
77
|
+
Record the reason and stop. Never fabricate to fill a hole.
|
|
78
|
+
|
|
79
|
+
| Reason | Trigger |
|
|
80
|
+
| --- | --- |
|
|
81
|
+
| `symbol-not-found` | Nothing in the source matches the statement's subject |
|
|
82
|
+
| `ambiguous-symbol` | Two or more plausible symbols, no `Verification` column to disambiguate |
|
|
83
|
+
| `oracle-is-adjectival` | The predicate is a quality word — "actionable", "clear", "reasonable", "appropriate" — with no observable |
|
|
84
|
+
| `unimplemented` | The spec is ahead of the code; the symbol does not exist yet |
|
|
85
|
+
| `no-state-machine` | A `lifecycle` criterion whose states and transitions are not enumerable |
|
|
86
|
+
| `domain-unbounded` | No type, no constraint, and no example to bound the generator |
|
|
87
|
+
| `singleton-domain` | The determiner quantifies over a domain with one element |
|
|
88
|
+
| `label-from-mention` | The metamorphic label came from the criterion *naming* a shape, not exhibiting one |
|
|
89
|
+
| `criterion-describes-its-test` | The criterion is written as a description of the test that verifies it |
|
|
90
|
+
| `static-or-demonstration` | The oracle is a fact about the source tree or an end-to-end narrative, not a function of an input |
|
|
91
|
+
| `no-row-id` | `row_id` is null, so no test could be reconciled |
|
|
92
|
+
|
|
93
|
+
### `singleton-domain` deserves its own note
|
|
94
|
+
|
|
95
|
+
`property: universal` most often comes from the `universal:determiner` signal — a closed
|
|
96
|
+
determiner set (`a`, `an`, `any`, `every`, `each`, `all`, `no`) read at three bounded
|
|
97
|
+
sentence positions. English uses the same determiner for a universal and for a single
|
|
98
|
+
witness: *"A repeated module root is loaded only once"* quantifies; *"A bareword after
|
|
99
|
+
`write` is parsed as a positional"* names one case.
|
|
100
|
+
|
|
101
|
+
The engine cannot tell these apart without giving up determinism, and it is not supposed
|
|
102
|
+
to — that is what this step is for. Ground the domain: if it has one element, the criterion
|
|
103
|
+
is a witness. Route it to the queue as a `Unit` test (step 5's *witness* outcome), not as an
|
|
104
|
+
unattended property.
|
|
105
|
+
|
|
106
|
+
This is a **routing** decision, not a complaint about the criterion, and never a reason to
|
|
107
|
+
suggest rewording. Record it as `singleton-domain` in the queue.
|
|
108
|
+
|
|
109
|
+
### `label-from-mention`
|
|
110
|
+
|
|
111
|
+
A criterion that *talks about* a property shape picks up that shape's signal. Real example:
|
|
112
|
+
|
|
113
|
+
> "A criterion carrying a round-trip, idempotence, ordering or invariant signal classifies
|
|
114
|
+
> as that metamorphic shape."
|
|
115
|
+
|
|
116
|
+
This is labelled `round-trip` because the words are in it. Its actual oracle is a
|
|
117
|
+
classification assertion — `classify(x).property == expected` — which is a `universal`, not
|
|
118
|
+
a metamorphic law. The tell is that the oracle contains no *pair* of operations to compose.
|
|
119
|
+
|
|
120
|
+
Re-derive the strategy from the grounded oracle, not from the label, and record
|
|
121
|
+
`label-from-mention` so the reclassification is visible on review. Do not report this to
|
|
122
|
+
quire-rs as a miss: the classifier is doing exactly what it says on the tin, and the
|
|
123
|
+
criterion is a fine criterion.
|
|
124
|
+
|
|
125
|
+
### `criterion-describes-its-test`
|
|
126
|
+
|
|
127
|
+
Common in mature specs written alongside their suite:
|
|
128
|
+
|
|
129
|
+
> "A proptest fuzzes patches across all archetypes in the test corpus and confirms
|
|
130
|
+
> `apply_patch` returns a valid document."
|
|
131
|
+
> "A static audit (`rg` for `Mutex`/`RwLock`) confirms the parallel parse uses no locks."
|
|
132
|
+
|
|
133
|
+
The criterion names its own verification method. Generating a second test from it duplicates
|
|
134
|
+
work that exists. **Go find the existing test first** — grep the suite for the named symbol
|
|
135
|
+
or the described technique. If it exists, record the criterion as *already covered* and, if
|
|
136
|
+
the existing test carries no tracking tag, note that adding one is the cheaper fix than
|
|
137
|
+
generating anything.
|
|
138
|
+
|
|
139
|
+
### `static-or-demonstration`
|
|
140
|
+
|
|
141
|
+
Two shapes turn up regularly and are neither properties nor witnesses:
|
|
142
|
+
|
|
143
|
+
- **Static** — an assertion about the source tree ("No hand-rolled argument dispatcher
|
|
144
|
+
remains in `src/cli.ts`"). Verified by `Static` in the matrix vocabulary, not by a
|
|
145
|
+
generated test.
|
|
146
|
+
- **Demonstration** — an end-to-end narrative, common in StR validation criteria ("An
|
|
147
|
+
author installs a module, sees its types appear, requests them, and validates files
|
|
148
|
+
authored against them"). Verified by an eval or a demo, not by a property.
|
|
149
|
+
|
|
150
|
+
Both are correct criteria verified by another method. Record the reason and move on.
|
|
151
|
+
|
|
152
|
+
A refusal is not a finding about the spec. `oracle-is-adjectival` in particular is often a
|
|
153
|
+
correct NFR that is verified by inspection rather than by test — say so, and do not suggest
|
|
154
|
+
rewording it.
|
|
155
|
+
|
|
156
|
+
## Output of this step
|
|
157
|
+
|
|
158
|
+
A grounding record per criterion, or a refusal reason. Grounded records go to step 3;
|
|
159
|
+
refusals go to step 5.
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Step 3: Strategy Selection
|
|
2
|
+
|
|
3
|
+
**Goal**: pick the assertion skeleton and the routing lane. Two independent axes.
|
|
4
|
+
|
|
5
|
+
- **Strategy** comes from `property` — the 10-value taxonomy.
|
|
6
|
+
- **Routing** comes from `extraction` — `extractable` / `candidate` / `not-extractable`.
|
|
7
|
+
|
|
8
|
+
`shape` (`assertion | obligation | given-when-then | unstructured`) is the FR-047 grammar
|
|
9
|
+
axis. **It never selects a strategy.** At most it tells you where the clause boundaries sit
|
|
10
|
+
in the sentence.
|
|
11
|
+
|
|
12
|
+
## Routing
|
|
13
|
+
|
|
14
|
+
| | `extractable` | `candidate` | `not-extractable` |
|
|
15
|
+
| --- | --- | --- | --- |
|
|
16
|
+
| One of the 8 generatable properties | emit **unattended** | emit → **queue** | grounded → queue; refused → step 5 |
|
|
17
|
+
| `example` / `unclassified` | n/a | n/a | **step 5 only** |
|
|
18
|
+
|
|
19
|
+
**Grounding outranks the table.** The lanes above assume step 2 grounded the criterion as
|
|
20
|
+
the classification describes it. Where step 2 returned one of its reclassifying reasons, use
|
|
21
|
+
that instead — a record can be `extraction: extractable` and still not be a property:
|
|
22
|
+
|
|
23
|
+
| step-2 reason | lane |
|
|
24
|
+
| --- | --- |
|
|
25
|
+
| `singleton-domain` | **queue as a `Unit` witness**, never unattended |
|
|
26
|
+
| `label-from-mention` | re-derive the strategy from the grounded oracle; lane unchanged |
|
|
27
|
+
| `criterion-describes-its-test` | find the existing test; record *already covered*, emit nothing |
|
|
28
|
+
| `static-or-demonstration` | refuse; no matrix row |
|
|
29
|
+
| any other refusal | queue with the reason, or step 5 |
|
|
30
|
+
|
|
31
|
+
`singleton-domain` is the common one. On a real run it moved 31 of 52 `universal` criteria
|
|
32
|
+
out of the unattended lane. Routing on `extraction` alone would have emitted all 52.
|
|
33
|
+
|
|
34
|
+
`candidate` means the criterion carries a metamorphic label the structural pass did not
|
|
35
|
+
corroborate — the label came from a declared idiom alone, or the shape landed but no
|
|
36
|
+
predicate marker supplied an oracle. It is the only field a module declaration can move,
|
|
37
|
+
and it is safe precisely because it is review-gated. Nothing reaches the unattended set
|
|
38
|
+
through it.
|
|
39
|
+
|
|
40
|
+
## Strategy per property
|
|
41
|
+
|
|
42
|
+
Full table with generator notes: [`../assets/strategy-table.md`](../assets/strategy-table.md).
|
|
43
|
+
|
|
44
|
+
| property | generator domain | assertion skeleton |
|
|
45
|
+
| --- | --- | --- |
|
|
46
|
+
| `round-trip` | `x : Domain` (the inner type) | `dec(enc(x)) == x`; if lossy, `norm(dec(enc(x))) == norm(x)` |
|
|
47
|
+
| `idempotence` | `x`, seeded with `f`'s own outputs so the fixpoint branch is hit | `f(f(x)) == f(x)` |
|
|
48
|
+
| `ordering` | `xs : list<T>`, length 0..32 | pick by verb — *sorted* → `is_sorted_by(key, f(xs))`; *regardless of order* → `f(xs) == f(shuffle(xs))`; *reorders/preserves* → `multiset(f(xs)) == multiset(xs)` |
|
|
49
|
+
| `invariant` | `x : Domain` | `P(f(x))`; if stateful, `P(state)` after **every** op in a generated sequence |
|
|
50
|
+
| `error-case` | the **negative** domain — complement generator plus `prop_filter` / `fc.pre` / `assume` | fails with error class `E` **and** the stated payload shape; never panics, never returns success |
|
|
51
|
+
| `lifecycle` | an op sequence drawn from the state machine | per op `sut.apply(op).ok == model.allows(op)`, then the invariant; terminal state at the end |
|
|
52
|
+
| `concurrency` | interleaved or parallel op sequences | the concurrent result is *some* linearization of the sequential model; no invariant violated at any interleaving |
|
|
53
|
+
| `universal` | the clause subject | `oracle(x)` — a direct translation of the criterion |
|
|
54
|
+
| `example` | — | not a property. Step 5 may reclassify it, or emit a `Unit` witness test |
|
|
55
|
+
| `unclassified` | — | not a property. Step 5 only |
|
|
56
|
+
|
|
57
|
+
`universal` is the default and the largest bucket. `round-trip` needs **two** symbols — if
|
|
58
|
+
grounding found only one side of the pair, route to the queue rather than inventing the
|
|
59
|
+
inverse.
|
|
60
|
+
|
|
61
|
+
## Harness-specific downgrades
|
|
62
|
+
|
|
63
|
+
- **`concurrency` in Python** — hypothesis ships no interleaving scheduler. Emit a
|
|
64
|
+
`RuleBasedStateMachine` sketch over the async model, always queued, never unattended.
|
|
65
|
+
- **`lifecycle` anywhere** — generatable only if the states and transitions are enumerable
|
|
66
|
+
from the FR's `## Behavior`. Otherwise the refusal is `no-state-machine`.
|
|
67
|
+
- **`concurrency` in Rust** — prefer `loom` for exhaustive interleaving of a small model;
|
|
68
|
+
fall back to `proptest-state-machine`'s parallel mode when the op set is large.
|
|
69
|
+
|
|
70
|
+
## Weak assertions are a failure, not a fallback
|
|
71
|
+
|
|
72
|
+
If the strategy collapses to any of these, do not emit it — queue with the grounding
|
|
73
|
+
refusal instead:
|
|
74
|
+
|
|
75
|
+
- `assert result is not None` / `expect(x).toBeDefined()` / `assert!(r.is_ok())` alone
|
|
76
|
+
- a type check with no value check
|
|
77
|
+
- a test with no assertion at all
|
|
78
|
+
- an assertion that restates the implementation rather than the criterion
|
|
79
|
+
|
|
80
|
+
## Output of this step
|
|
81
|
+
|
|
82
|
+
Per record: `{row_id, strategy, lane, grounding}` — where `lane ∈ {unattended, queue,
|
|
83
|
+
second-pass}`.
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# Step 4: Generate Tests
|
|
2
|
+
|
|
3
|
+
**Goal**: write the test, tagged so `gap-analysis` can reconcile it.
|
|
4
|
+
|
|
5
|
+
Templates per harness:
|
|
6
|
+
[proptest](../assets/templates/proptest.rs.md) ·
|
|
7
|
+
[fast-check](../assets/templates/fast-check.ts.md) ·
|
|
8
|
+
[hypothesis](../assets/templates/hypothesis.py.md)
|
|
9
|
+
|
|
10
|
+
## The tag contract
|
|
11
|
+
|
|
12
|
+
`gap-analysis` step 3 greps for four forms. Every generated test carries **two independent
|
|
13
|
+
carriers**, so a single formatting change cannot break reconciliation:
|
|
14
|
+
|
|
15
|
+
1. `Trace: FR-027-AC-1` on its own line in the doc comment or docstring.
|
|
16
|
+
2. The test function or `describe` name containing the id —
|
|
17
|
+
`fr_027_ac_1_…` in Rust/Python, `"FR-027-AC-1 …"` in TypeScript.
|
|
18
|
+
|
|
19
|
+
Plus one provenance line, ignored by the reconciliation grep and read by this skill on
|
|
20
|
+
re-runs:
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
spec-correctness: row=FR-027-AC-1 property=universal extraction=extractable origin=regex review=none
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
`origin ∈ {regex, regex-candidate, llm-second-pass}`. `review ∈ {none, required}`.
|
|
27
|
+
For `origin=llm-second-pass`, add `confidence=<high|medium|low>`.
|
|
28
|
+
|
|
29
|
+
Hard rules:
|
|
30
|
+
|
|
31
|
+
- **Never emit a `row_id` that is not in the classification output.** Not a guessed one,
|
|
32
|
+
not a renumbered one, not one you inferred from a nearby row.
|
|
33
|
+
- **Never alter an existing tag** on a hand-written test. If a test already claims the
|
|
34
|
+
`row_id`, do not emit a second one — record it as already covered.
|
|
35
|
+
- Match the repo's dominant tag style from step 0 **in addition to** the two carriers.
|
|
36
|
+
|
|
37
|
+
## Idempotent re-runs
|
|
38
|
+
|
|
39
|
+
The provenance line is the key. On a re-run:
|
|
40
|
+
|
|
41
|
+
- Same `row_id`, same strategy, same grounding → leave the file alone.
|
|
42
|
+
- Same `row_id`, changed grounding → rewrite the body, keep the tag byte-identical.
|
|
43
|
+
- `row_id` no longer in the classification output (the criterion was deleted) → do not
|
|
44
|
+
delete the test silently; report it as orphaned and let the user decide.
|
|
45
|
+
- A test whose provenance line is absent is hand-written. **Never overwrite it.**
|
|
46
|
+
|
|
47
|
+
## File placement
|
|
48
|
+
|
|
49
|
+
| Harness | Unattended | Queued |
|
|
50
|
+
| --- | --- | --- |
|
|
51
|
+
| Rust | `tests/props_fr_NNN.rs` | `tests/props_review_fr_NNN.rs` |
|
|
52
|
+
| TypeScript | `tests/props/fr-NNN.prop.test.ts` | `tests/props/_review/fr-NNN.prop.test.ts` |
|
|
53
|
+
| Python | `tests/props/test_fr_NNN.py` | `tests/props/_review/test_fr_NNN.py` |
|
|
54
|
+
|
|
55
|
+
**Rust paths are flat on purpose.** Cargo auto-discovers integration tests only at
|
|
56
|
+
`tests/*.rs`; a file under `tests/props/` is treated as a helper module and is never
|
|
57
|
+
compiled as a test target. A nested layout would need a `[[test]]` entry per file in
|
|
58
|
+
`Cargo.toml`, and this skill does not edit build manifests. Flat names with a `props_`
|
|
59
|
+
prefix keep the grouping without the manifest edit.
|
|
60
|
+
|
|
61
|
+
For a Rust repo that keeps `#[cfg(test)]` unit tests inline and no `tests/` dir at all,
|
|
62
|
+
still write `tests/props_fr_NNN.rs` — an integration test only sees the public API, which is
|
|
63
|
+
the right boundary for a criterion-derived property anyway. If the symbol under test is
|
|
64
|
+
private, that is a grounding result, not a placement problem: record `symbol-not-public` and
|
|
65
|
+
queue it.
|
|
66
|
+
|
|
67
|
+
One file per FR, one test per `row_id`. Shared generators go in a sibling `arbitraries`
|
|
68
|
+
module rather than being duplicated per test.
|
|
69
|
+
|
|
70
|
+
## Inert markers for queued tests
|
|
71
|
+
|
|
72
|
+
A queued test must not be able to turn a matrix row green:
|
|
73
|
+
|
|
74
|
+
| Harness | Marker |
|
|
75
|
+
| --- | --- |
|
|
76
|
+
| Rust | `#[ignore = "spec-correctness review pending"]` |
|
|
77
|
+
| TypeScript | `it.skip(…)` / `describe.skip(…)` |
|
|
78
|
+
| Python | `@pytest.mark.skip(reason="spec-correctness review pending")` |
|
|
79
|
+
|
|
80
|
+
The tag stays byte-identical whether the test is queued or accepted — so
|
|
81
|
+
`gap-analysis` reports the same identifier before and after, and acceptance is a
|
|
82
|
+
one-line diff plus a move.
|
|
83
|
+
|
|
84
|
+
## Generator hygiene
|
|
85
|
+
|
|
86
|
+
- Bound every collection (`0..32` is a good default). An unbounded generator turns a fast
|
|
87
|
+
suite slow and hides shrinking.
|
|
88
|
+
- Reuse the repo's existing fixtures and factories before writing a new arbitrary.
|
|
89
|
+
- Seed generators so both branches of a partitioned domain are reachable — check the
|
|
90
|
+
sibling ACs from step 2.
|
|
91
|
+
- Do not set a fixed RNG seed. Do not raise the case count above the harness default
|
|
92
|
+
without a reason recorded in the queue.
|
|
93
|
+
|
|
94
|
+
## After writing
|
|
95
|
+
|
|
96
|
+
Run the suite. A failing **unattended** test is a grounding bug in step 2, not a spec bug
|
|
97
|
+
and not a code bug — fix the grounding or move the test to the queue with a reason. Never
|
|
98
|
+
weaken an assertion to make a generated test pass, and never edit the spec to match a
|
|
99
|
+
generated test.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Step 5: The LLM Second Pass
|
|
2
|
+
|
|
3
|
+
**Goal**: recover tests from the criteria the deterministic pass could not settle —
|
|
4
|
+
`property: example`, `property: unclassified`, and anything step 2 refused to ground.
|
|
5
|
+
|
|
6
|
+
**Ask before running this** when the residue is large (more than ~30 records). It is an
|
|
7
|
+
expensive judgement pass. Fan out one subagent per FR when you do run it.
|
|
8
|
+
|
|
9
|
+
## Why this pass exists here and not in the engine
|
|
10
|
+
|
|
11
|
+
quire-rs#45 is closed as answered: **31.3% recall at 93.3% precision is the engine's
|
|
12
|
+
deterministic ceiling.** Three widenings were measured factorially against a precision gate
|
|
13
|
+
fixed in advance at 85%; one shipped at 100%, two were deleted at 70% and 56.7%. Another
|
|
14
|
+
regex cannot get further without surrendering determinism or precision.
|
|
15
|
+
|
|
16
|
+
A generated test is reviewed before it lands. That makes a recall/precision tradeoff
|
|
17
|
+
affordable *here* that is not affordable in the engine. That is the whole argument — it is
|
|
18
|
+
not that the engine is wrong.
|
|
19
|
+
|
|
20
|
+
## Input per record
|
|
21
|
+
|
|
22
|
+
The full quire record, the step-2 grounding attempt (including its refusal reason), the FR
|
|
23
|
+
file, the sibling ACs, and the located symbol if there is one.
|
|
24
|
+
|
|
25
|
+
## It may propose exactly one of three things
|
|
26
|
+
|
|
27
|
+
1. **Reclassify** into one of the 8 generatable families, with a proposed domain,
|
|
28
|
+
precondition and oracle, each carrying at least one `file:line` citation. → a property
|
|
29
|
+
test into the queue.
|
|
30
|
+
|
|
31
|
+
Worth looking for: a criterion phrased about one concrete input that actually holds for
|
|
32
|
+
a whole class ("Repeated `--types` accumulate into an ordered list" is `ordering`, not an
|
|
33
|
+
example); an error message criterion that is really `error-case` over a negative domain.
|
|
34
|
+
|
|
35
|
+
2. **Witness** — the criterion genuinely describes a single case, or a small closed set of
|
|
36
|
+
them ("`version`, `--version`, and `-v` each print the package version" is a 3-case
|
|
37
|
+
witness). → an example-based test, `Type: Unit`, queued, and **explicitly not counted as
|
|
38
|
+
property coverage** in the report.
|
|
39
|
+
|
|
40
|
+
3. **Nothing** — record a note with the reason. A note, not a finding, not a severity.
|
|
41
|
+
|
|
42
|
+
## It must refuse to
|
|
43
|
+
|
|
44
|
+
- **Reword, split, or recommend editing a criterion**, or emit anything that reads as
|
|
45
|
+
pressure on the author. A concrete criterion is a legitimate criterion (CON-1).
|
|
46
|
+
- **Invent an oracle** absent from both the FR and the code. No tautologies, no
|
|
47
|
+
`assert x is not None`, no isinstance-only checks, no assertionless tests. The
|
|
48
|
+
weak-assertion heuristics in `gap-analysis/references/step-5-semantic-review.md` apply to
|
|
49
|
+
generated tests too.
|
|
50
|
+
- **Propose a new regex, lexicon entry, or idiom widening for quire-rs.** The ceiling is
|
|
51
|
+
accepted. Do not re-propose the two deleted widenings.
|
|
52
|
+
- **Write a framework name into any spec artifact** (CON-2).
|
|
53
|
+
- **Alter or synthesize a `row_id`.** Every emitted tag must be a `row_id` present in the
|
|
54
|
+
classification output.
|
|
55
|
+
- **Mark its own output unattended**, or promote a test out of the queue.
|
|
56
|
+
|
|
57
|
+
## Marking
|
|
58
|
+
|
|
59
|
+
Every product of this pass:
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
spec-correctness: row=FR-005-AC-2 property=example extraction=not-extractable origin=llm-second-pass review=required confidence=medium
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
filed under `tests/props/_review/`, carrying the harness's inert marker, and listed as one
|
|
66
|
+
row in the queue file.
|
|
67
|
+
|
|
68
|
+
`extraction: candidate` records use the identical mechanism with `origin=regex-candidate`
|
|
69
|
+
and no `confidence` key — they came from the deterministic pass, not from this one.
|
|
70
|
+
|
|
71
|
+
## Calibrating confidence
|
|
72
|
+
|
|
73
|
+
- `high` — the oracle is a direct quote of a `SHALL` predicate and the symbol was located.
|
|
74
|
+
- `medium` — the oracle was assembled from `## Behavior` plus the code, and reads as a fair
|
|
75
|
+
restatement.
|
|
76
|
+
- `low` — the oracle is a plausible reading among more than one. Say what the alternative
|
|
77
|
+
reading was, in the queue row.
|
|
78
|
+
|
|
79
|
+
Anything below `low` is proposal 3: nothing.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# Step 6: The Review Queue
|
|
2
|
+
|
|
3
|
+
**Goal**: everything not settled unattended lands somewhere a person can accept or reject
|
|
4
|
+
it, and lands **inert** so it cannot be mistaken for coverage.
|
|
5
|
+
|
|
6
|
+
## Where it lives
|
|
7
|
+
|
|
8
|
+
`tests/props/QUEUE.md`, in the test tree. Template:
|
|
9
|
+
[`../assets/property-queue-template.md`](../assets/property-queue-template.md).
|
|
10
|
+
|
|
11
|
+
It is **not** a `SpecReview`. `SpecReview.analysis` is a closed enum in
|
|
12
|
+
`spec-artifacts-process` with no `spec-correctness` value, so a file under `reviews/` would
|
|
13
|
+
fail `quire validate`. Keeping the queue in the test tree also keeps it out of every
|
|
14
|
+
`spec/**` validation glob. Adding the enum value upstream is a reasonable follow-up; it is
|
|
15
|
+
not a prerequisite for this skill.
|
|
16
|
+
|
|
17
|
+
## What goes in it
|
|
18
|
+
|
|
19
|
+
| Lane | Reason it is queued |
|
|
20
|
+
| --- | --- |
|
|
21
|
+
| `extraction: candidate` | Metamorphic label the structural pass did not corroborate |
|
|
22
|
+
| Second-pass reclassify | An LLM read, not a deterministic one |
|
|
23
|
+
| Second-pass witness | An example test, not property coverage |
|
|
24
|
+
| `singleton-domain` witness | Classified `extractable`, but grounding found a one-element domain — a `Unit` test, not property coverage |
|
|
25
|
+
| Grounded but harness-downgraded | e.g. Python `concurrency` |
|
|
26
|
+
| Refused | No test written; the reason is the row |
|
|
27
|
+
| Generator library missing | **No file written** — the proposal is a code block in the queue. An import of a missing library breaks collection even for a skipped test |
|
|
28
|
+
|
|
29
|
+
## Inert by construction
|
|
30
|
+
|
|
31
|
+
Every queued test carries its harness's skip marker (see step 4). Two consequences worth
|
|
32
|
+
being explicit about:
|
|
33
|
+
|
|
34
|
+
- **Its matrix row is `🚧`, never `✅`.** `gap-analysis` treats a `✅` row backed by a
|
|
35
|
+
skipped test as a *status lie* and raises a `high` finding — correctly. Queued rows are
|
|
36
|
+
in-progress rows.
|
|
37
|
+
- **A skipped test still carries its tag**, so `gap-analysis` sees the trace and reports it
|
|
38
|
+
as present-but-not-passing rather than as unbacked.
|
|
39
|
+
|
|
40
|
+
## Acceptance procedure
|
|
41
|
+
|
|
42
|
+
For each accepted row:
|
|
43
|
+
|
|
44
|
+
1. Delete the skip marker line.
|
|
45
|
+
2. Move the file out of the review location: `tests/props/_review/` → `tests/props/`
|
|
46
|
+
for TypeScript and Python, `tests/props_review_fr_NNN.rs` → `tests/props_fr_NNN.rs`
|
|
47
|
+
for Rust, whose paths are flat (step 4).
|
|
48
|
+
3. Change the provenance line's `review=required` to `review=accepted`, leaving `row=` and
|
|
49
|
+
`origin=` untouched.
|
|
50
|
+
4. Run the suite.
|
|
51
|
+
5. Flip the matrix row from `🚧` to `✅`.
|
|
52
|
+
|
|
53
|
+
The tracking tag is **byte-identical** through all of this — that is the point. Nothing
|
|
54
|
+
about reconciliation changes when a test is accepted.
|
|
55
|
+
|
|
56
|
+
Rejection is simpler: delete the file, and leave the queue row with `status: rejected` and
|
|
57
|
+
a one-line reason so the next run does not re-propose it. A re-run reads existing
|
|
58
|
+
`rejected` rows and skips those `row_id`s unless the criterion's `statement` has changed.
|
|
59
|
+
|
|
60
|
+
## What the queue must not contain
|
|
61
|
+
|
|
62
|
+
- A verdict, a score, or a completion percentage framed as a target.
|
|
63
|
+
- A recommendation to edit any spec file.
|
|
64
|
+
- A `row_id` absent from the classification output.
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# Step 7: Report and Handoff
|
|
2
|
+
|
|
3
|
+
**Goal**: close the loop — the matrix knows about the new tests, and `gap-analysis` can
|
|
4
|
+
reconcile every tag this run emitted.
|
|
5
|
+
|
|
6
|
+
## The run report
|
|
7
|
+
|
|
8
|
+
Prepended to `tests/props/QUEUE.md`. Counts only:
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
spec-correctness — <repo> — <YYYY-MM-DD>
|
|
12
|
+
quire-cli <version> · harness <name> · <N> criteria
|
|
13
|
+
|
|
14
|
+
emitted unattended 37 (extractable, grounded)
|
|
15
|
+
queued 22 candidate 0 · second-pass 18 · downgraded 1 · dep-missing 3
|
|
16
|
+
refused 14 symbol-not-found 6 · oracle-is-adjectival 5 · unimplemented 3
|
|
17
|
+
already covered 9 hand-written tests already carry the row_id
|
|
18
|
+
witnesses 5 Unit tests, not property coverage
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
`emitted unattended + queued + refused + already covered` must equal the number of records
|
|
22
|
+
with a `row_id`. If it does not, a record was dropped — find it before reporting.
|
|
23
|
+
|
|
24
|
+
No thresholds, no grades, no rewording suggestions. Same rule as step 1.
|
|
25
|
+
|
|
26
|
+
## Handoff to `spec-matrix`
|
|
27
|
+
|
|
28
|
+
Emit rows for the `spec-matrix` Test Case Summary — do not write them yourself if the repo
|
|
29
|
+
already runs the matrix workflow; hand them over.
|
|
30
|
+
|
|
31
|
+
`Test ID | Title | Type | Priority | Traces To | Status`
|
|
32
|
+
|
|
33
|
+
- `Type` — `Property` for the 8 generatable families; `Unit` for witnesses, whether they
|
|
34
|
+
came from the second pass or from a `singleton-domain` grounding result.
|
|
35
|
+
The vocabulary is owned by `spec_artifacts_process/manifest.yaml`, not by this skill; if
|
|
36
|
+
it disagrees with what you write, the manifest wins.
|
|
37
|
+
- `Traces To` — the `row_id`, exactly (`FR-027-AC-1`). Never a range you invented, never
|
|
38
|
+
another TC.
|
|
39
|
+
- `Status` — `✅` only for an unattended test that passes. `🚧` for anything queued.
|
|
40
|
+
Refused criteria get **no row**; they are reported in the queue, not claimed in the
|
|
41
|
+
matrix.
|
|
42
|
+
|
|
43
|
+
## Reconciliation check
|
|
44
|
+
|
|
45
|
+
Before finishing, prove the tags are greppable by the tool that will read them:
|
|
46
|
+
|
|
47
|
+
```
|
|
48
|
+
quire validate --scope <repo> 'spec/**/*.md'
|
|
49
|
+
```
|
|
50
|
+
then run `gap-analysis` step 3's index over the test tree and confirm:
|
|
51
|
+
|
|
52
|
+
- every `row_id` this run emitted appears in the tag index;
|
|
53
|
+
- no matrix row this run added is `✅` while its test is skipped;
|
|
54
|
+
- no emitted tag names a `row_id` absent from the `quire properties` output.
|
|
55
|
+
|
|
56
|
+
A mismatch here is a bug in this run, not in `gap-analysis`.
|
|
57
|
+
|
|
58
|
+
## Finish by saying
|
|
59
|
+
|
|
60
|
+
- What now has tests, in concrete terms: which FRs, how many criteria.
|
|
61
|
+
- What is waiting in the queue and roughly how long a review pass would take.
|
|
62
|
+
- The one next action — usually "review `tests/props/QUEUE.md`" or "add `<lib>` to
|
|
63
|
+
dev-dependencies so the queued tests can run".
|