@pilotspace/add 1.17.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +144 -0
- package/GETTING-STARTED.md +20 -64
- package/README.md +128 -56
- package/agents/add.md +68 -0
- package/bin/cli.js +86 -21
- package/package.json +2 -3
- package/personas-teacher/VENDOR.md +6 -0
- package/skill/add/SKILL.md +100 -134
- package/skill/add/adopt.md +28 -0
- package/skill/add/beyond.md +36 -0
- package/skill/add/deltas.md +25 -2
- package/skill/add/design.md +56 -74
- package/skill/add/{report-template.md → gate-udd.md} +28 -38
- package/skill/add/intake.md +56 -25
- package/skill/add/loop.md +32 -19
- package/skill/add/phases/build.md +78 -0
- package/skill/add/phases/direction.md +280 -0
- package/skill/add/phases/verify.md +160 -0
- package/skill/add/run.md +11 -18
- package/skill/add/terms.md +18 -0
- package/tooling/add.py +2383 -3261
- package/tooling/add_engine/autonomy.py +2 -37
- package/tooling/add_engine/components.py +4 -87
- package/tooling/add_engine/constants.py +145 -91
- package/tooling/add_engine/guidelines.py +9 -11
- package/tooling/add_engine/io_state.py +69 -2
- package/tooling/add_engine/predicates.py +90 -2
- package/tooling/add_engine/search.py +6 -6
- package/tooling/add_engine/taskdoc.py +15 -10
- package/tooling/templates/CONVENTIONS.md.tmpl +1 -1
- package/tooling/templates/GLOSSARY.md.tmpl +10 -4
- package/tooling/templates/MILESTONE.md.tmpl +20 -16
- package/tooling/templates/PLAN.md.tmpl +128 -0
- package/tooling/templates/PROJECT.md.tmpl +6 -1
- package/tooling/templates/PROMPT.persona.md.tmpl +17 -21
- package/tooling/templates/personas/_template.md.tmpl +33 -29
- package/tooling/templates/personas/build-engineer.md.tmpl +45 -0
- package/tooling/templates/personas/data-steward.md.tmpl +45 -0
- package/tooling/templates/personas/evidence-verifier.md.tmpl +45 -0
- package/tooling/templates/personas/platform-engineer.md.tmpl +45 -0
- package/tooling/templates/personas/product-lead.md.tmpl +51 -0
- package/tooling/templates/personas/quality-auditor.md.tmpl +47 -0
- package/tooling/templates/personas/release-manager.md.tmpl +52 -0
- package/tooling/templates/personas/security-gatekeeper.md.tmpl +45 -0
- package/tooling/templates/personas/software-architect.md.tmpl +45 -0
- package/tooling/templates/personas/stream-orchestrator.md.tmpl +44 -0
- package/tooling/templates/personas/technical-writer.md.tmpl +49 -0
- package/tooling/templates/personas/ux-experience-lead.md.tmpl +48 -0
- package/tooling/templates/specs/SPEC.md.tmpl +18 -0
- package/tooling/templates/udd-wireframe.md +1 -1
- package/agents/add-advisor.md +0 -31
- package/agents/add-build.md +0 -29
- package/agents/add-design.md +0 -32
- package/agents/add-persona.md +0 -33
- package/agents/add-verify.md +0 -36
- package/docs/00-introduction.md +0 -46
- package/docs/01-principles.md +0 -71
- package/docs/02-the-flow.md +0 -113
- package/docs/03-step-1-specify.md +0 -123
- package/docs/04-step-2-scenarios.md +0 -90
- package/docs/05-step-3-contract.md +0 -80
- package/docs/06-step-4-tests.md +0 -85
- package/docs/07-step-5-build.md +0 -82
- package/docs/08-step-6-verify.md +0 -105
- package/docs/09-the-loop.md +0 -75
- package/docs/10-setup-and-stages.md +0 -118
- package/docs/11-governance.md +0 -91
- package/docs/12-roles.md +0 -99
- package/docs/13-adoption.md +0 -67
- package/docs/14-foundation.md +0 -142
- package/docs/15-foundations-and-lineage.md +0 -106
- package/docs/16-releasing.md +0 -182
- package/docs/17-components.md +0 -143
- package/docs/18-personas.md +0 -80
- package/docs/README.md +0 -77
- package/docs/add-competencies.png +0 -0
- package/docs/add-flow.png +0 -0
- package/docs/add-foundation.png +0 -0
- package/docs/add-hierarchy.png +0 -0
- package/docs/add-milestone-task-lifecycle.png +0 -0
- package/docs/add-task-growth-wheel.png +0 -0
- package/docs/appendix-a-templates.md +0 -88
- package/docs/appendix-b-prompts.md +0 -154
- package/docs/appendix-c-glossary.md +0 -176
- package/docs/appendix-d-worked-example.md +0 -231
- package/docs/appendix-e-checklists.md +0 -92
- package/docs/appendix-f-requirements-matrix.md +0 -171
- package/docs/appendix-g-references.md +0 -106
- package/skill/add/advisor.md +0 -81
- package/skill/add/compact-foundation.md +0 -40
- package/skill/add/components.md +0 -52
- package/skill/add/confidence.md +0 -32
- package/skill/add/fold.md +0 -63
- package/skill/add/graduate.md +0 -39
- package/skill/add/phases/0-ground.md +0 -64
- package/skill/add/phases/0-setup.md +0 -103
- package/skill/add/phases/1-specify.md +0 -61
- package/skill/add/phases/2-scenarios.md +0 -48
- package/skill/add/phases/3-contract.md +0 -60
- package/skill/add/phases/4-tests.md +0 -61
- package/skill/add/phases/5-build.md +0 -62
- package/skill/add/phases/6-verify.md +0 -69
- package/skill/add/phases/7-observe.md +0 -45
- package/skill/add/phases/fast-lane.md +0 -34
- package/skill/add/release.md +0 -61
- package/skill/add/scope.md +0 -72
- package/skill/add/self-improve.md +0 -20
- package/skill/add/sensitivity.md +0 -51
- package/skill/add/setup-review.md +0 -48
- package/skill/add/soul.md +0 -58
- package/skill/add/streams.md +0 -269
- package/tooling/add_engine/release.py +0 -86
- package/tooling/templates/TASK.fast.md.tmpl +0 -78
- package/tooling/templates/TASK.md.tmpl +0 -181
|
@@ -1,123 +0,0 @@
|
|
|
1
|
-
# 03 · Step 1 — Specify
|
|
2
|
-
|
|
3
|
-
[← 02 The flow](./02-the-flow.md) · [Contents](./README.md) · Next: [04 Step 2 Scenarios →](./04-step-2-scenarios.md)
|
|
4
|
-
|
|
5
|
-
> **Purpose:** state, in plain language, what the feature must do and what it must reject, with no ambiguity left for the AI to resolve by guessing.
|
|
6
|
-
> **Produces:** the **§1 SPECIFY** block of the task's `TASK.md` (one file per task — what this chapter is the *why* behind; historically a separate `SPEC.md`).
|
|
7
|
-
> **How it works — co-specification:** AI and human **brainstorm the shape together**; the AI drafts; the **human validates, with the AI's advice.** The decisive advice is a *lowest-confidence flag* — the AI names the one or two things most likely to be wrong, so the human's attention lands where it matters. The human owns the decision; the AI owns surfacing what it does not yet know.
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## Why this step is first
|
|
12
|
-
|
|
13
|
-
The specification is the description the AI will build from. Every other artifact descends from it. Anything vague here does not stay vague — it becomes a concrete wrong guess in the code, discovered late. The cheapest moment to remove an ambiguity is now, in a sentence, before anything depends on it.
|
|
14
|
-
|
|
15
|
-
There is also a diagnostic value: **if you cannot write the spec, you do not yet understand the feature well enough to build it.** The inability to specify is information, not an obstacle to push past.
|
|
16
|
-
|
|
17
|
-
## Co-specification — how the spec gets made
|
|
18
|
-
|
|
19
|
-
A specification is not dictated by one side. It is made in three moves:
|
|
20
|
-
|
|
21
|
-
1. **Diverge — brainstorm by both.** Before drafting, the AI surfaces the *decision space*: the two or three genuine ways to frame the feature, and the open questions it would otherwise resolve by guessing. You react — add, kill, redirect. This is the brainstorm, and it lives in the conversation, not in a new document.
|
|
22
|
-
2. **Converge — the AI drafts, and ranks its own uncertainty.** The AI writes the spec below, then ranks where its confidence is lowest. It does not hand you a flat list of equal-looking assumptions to nod through; it tells you *where it is most likely wrong, and what that would cost.*
|
|
23
|
-
3. **Validate — you decide, with the AI's advice.** You read the ranked uncertainty first, then confirm, correct, or send it back. Your approval is real because your attention was aimed.
|
|
24
|
-
|
|
25
|
-
The brainstorm leaves a *light trace, not a document.* What you chose becomes a rule; what you weighed and dropped becomes a one-line **`Framings weighed:`** note; what stayed genuinely uncertain becomes a **lowest-confidence flag**. Nothing new to maintain — the residue lands in the spec you were writing anyway.
|
|
26
|
-
|
|
27
|
-
## What a good specification contains
|
|
28
|
-
|
|
29
|
-
Four parts, kept short:
|
|
30
|
-
|
|
31
|
-
1. **Must** — the behaviors the feature is required to perform.
|
|
32
|
-
2. **Reject** — the inputs or situations it must refuse, each paired with a named error.
|
|
33
|
-
3. **After** — the state that is true once it succeeds (what changed).
|
|
34
|
-
4. **Assumptions — lowest-confidence first** — the things you are taking for granted, **ranked so the most-likely-wrong come first.** The top one or two carry a `⚠` flag with *why it is uncertain* and *what it costs if wrong*; the rest are the low-stakes tail. A spec with genuinely nothing uncertain still names its single biggest risk, however small — the AI never claims a blank mind.
|
|
35
|
-
|
|
36
|
-
Naming the errors matters. "Reject bad amounts" is an instruction to guess; `amount <= 0 -> "amount_invalid"` is a rule that produces a testable scenario and a defined contract response.
|
|
37
|
-
|
|
38
|
-
### Rule IDs (optional — opt-in by usage)
|
|
39
|
-
|
|
40
|
-
Give each Must a stable `M<n>:` ID; a Reject's own error code already IS its ID (referenced later as `R:<error_code>`). Once a task uses a tag anywhere in §2 SCENARIOS or §4 TESTS, `add.py check` starts asking that every Must/Reject be covered by a scenario tag or a test's `covers:` line — a task that never tags anything is never retro-flagged. See the template's own inline example for the exact grammar.
|
|
41
|
-
|
|
42
|
-
## Template
|
|
43
|
-
|
|
44
|
-
```
|
|
45
|
-
# SPEC.md
|
|
46
|
-
Feature: <name>
|
|
47
|
-
Framings weighed: <chosen> (chosen) · <alternative> · <alternative>
|
|
48
|
-
Must:
|
|
49
|
-
- <required behavior>
|
|
50
|
-
Reject:
|
|
51
|
-
- <bad input / situation> -> "<error_code>"
|
|
52
|
-
After:
|
|
53
|
-
- <what is true once it succeeds>
|
|
54
|
-
Assumptions — lowest-confidence first:
|
|
55
|
-
⚠ <most-likely-wrong assumption> — lowest confidence because <why>; if wrong: <cost>
|
|
56
|
-
- [x] <confirmed / low-stakes assumption> — <one line>
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
## ▶ Example
|
|
60
|
-
|
|
61
|
-
```
|
|
62
|
-
Feature: Transfer money between my own accounts
|
|
63
|
-
Framings weighed: synchronous single-currency transfer (chosen) · queued transfer · multi-currency with FX
|
|
64
|
-
Must:
|
|
65
|
-
- move an amount from one of my accounts to another of mine
|
|
66
|
-
- amount > 0
|
|
67
|
-
- source and destination are different accounts
|
|
68
|
-
- source has enough balance
|
|
69
|
-
After:
|
|
70
|
-
- source balance -= amount, destination balance += amount
|
|
71
|
-
Reject:
|
|
72
|
-
- amount <= 0 -> "amount_invalid"
|
|
73
|
-
- source == destination -> "same_account"
|
|
74
|
-
- balance < amount -> "insufficient_funds"
|
|
75
|
-
- account not mine -> "forbidden"
|
|
76
|
-
Assumptions — lowest-confidence first:
|
|
77
|
-
⚠ same currency only (no FX) in v1 — lowest confidence because the ticket never said; if wrong: the whole amount/rounding model changes and this contract is wrong
|
|
78
|
-
- [x] no daily limit in v1 — confirmed: out of scope for v1
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
The `Framings weighed:` line shows what was considered and dropped, so the chosen shape is a *decision*, not a default. The `⚠` line is the one the stakeholder reads first: the assumption most likely to be wrong and most expensive to get wrong. The flat `[x]` line is real but low-stakes. A reviewer can now spend their attention where it pays.
|
|
82
|
-
|
|
83
|
-
## The AI's role here
|
|
84
|
-
|
|
85
|
-
Use the AI to **open the space and then narrow it honestly.** First it brainstorms the genuine framings with you (diverge). Then it drafts the spec from whatever raw material you have — a ticket, an interview, a contract document — listing every assumption it had to make, **ranked lowest-confidence first**, and flagging the one or two it is least confident in with *why* and *what it costs if wrong*. Its instinct is to fill gaps silently and present a confident wall; the method forces those gaps into the open, and forces the confident wall to declare its own soft spots. See `playbook/1_specify.md` in [Appendix B](./appendix-b-prompts.md).
|
|
86
|
-
|
|
87
|
-
The defining instruction: *if a requirement is unclear, ask — do not resolve it by guessing — and of the things you must assume, say plainly where your confidence is lowest.*
|
|
88
|
-
|
|
89
|
-
## Common mistakes
|
|
90
|
-
|
|
91
|
-
- **Stating only the happy path.** The "Reject" list is where most real complexity lives; an empty one usually means it has not been thought through.
|
|
92
|
-
- **Free-text errors.** Errors must be named codes, not sentences, so they can become scenarios and contract responses.
|
|
93
|
-
- **Hidden assumptions.** If an assumption is not written down, it is not confirmed — it is a future bug with a delay timer.
|
|
94
|
-
- **A flat list of "confirmed" assumptions.** Eight equal-looking ticks invite a reflex approval. Rank them; flag the one or two that are load-bearing. An unranked list hides the risk inside the noise.
|
|
95
|
-
- **"Existing behavior" claims without a citation.** An assumption row that asserts "this is how X works today" is describing intent, not code. Any wiring claim or assumption that depends on the current state of an existing path must carry a grep/line citation (e.g. `file.rs:203`) — otherwise it is a future bug in disguise.
|
|
96
|
-
- **Wiring claims that name a symbol, not a caller chain.** Verifying that a function exists is not the same as verifying it is reachable. A wiring claim is only valid when it names the production caller chain from an actual entry point — not just the symbol's location in a file. A function that nothing calls is dead, not wired.
|
|
97
|
-
|
|
98
|
-
## Exit check
|
|
99
|
-
|
|
100
|
-
A spec is done when:
|
|
101
|
-
|
|
102
|
-
- [ ] Every required behavior is stated explicitly.
|
|
103
|
-
- [ ] Every rejection has a named error code.
|
|
104
|
-
- [ ] The success state-change is described.
|
|
105
|
-
- [ ] The assumptions are ordered lowest-confidence first, and the one or two `⚠` flags carry *why* + *cost* — or, for genuinely trivial scope, an honest "none material" that still names the single biggest risk.
|
|
106
|
-
|
|
107
|
-
The shift from older practice: you no longer pre-confirm every assumption to advance. You confirm that the AI has *ranked* its uncertainty and that you have *engaged the top of the rank.* Stated honestly: the flag makes a genuine review cheap and a lazy one visibly negligent — it cannot force the read. That is the most a lightweight check can buy.
|
|
108
|
-
|
|
109
|
-
## If the check fails
|
|
110
|
-
|
|
111
|
-
If you cannot state a rule clearly, the feature is not ready to build. Stop, take the question to whoever owns the requirement, and resolve it. Do not let the AI proceed on an unresolved point — that is the exact failure the whole method exists to prevent.
|
|
112
|
-
|
|
113
|
-
---
|
|
114
|
-
|
|
115
|
-
## The one approval, and where the flag really lands
|
|
116
|
-
|
|
117
|
-
In the one-approval flow, you do not approve the spec alone — you approve the whole frozen bundle (spec, scenarios, contract, tests) once, at the contract freeze. So the lowest-confidence flag is **bundle-wide**: at that single decision point the AI leads with *"of everything I'm asking you to freeze, these one or two points are most likely wrong"* — and a flag may point at an uncovered scenario or the contract shape, not only a spec assumption. The ranking you do here in Specify is the first input into that one gate. See [05 Contract](./05-step-3-contract.md) and the `add` skill's `run.md`.
|
|
118
|
-
|
|
119
|
-
---
|
|
120
|
-
|
|
121
|
-
## When the feature has a user interface
|
|
122
|
-
|
|
123
|
-
For anything with a UI, extend this step with a quick design: the **user flows** (the happy path and the main alternatives) and **every screen state** — loading, empty, error, and success. Correct logic behind a confusing or incomplete interface is still a poor product, and undesigned states are exactly where an AI will improvise something ugly. In the early **Prototype** stage, this design work is the main event and the code is throwaway (see [10 Stages](./10-setup-and-stages.md)).
|
|
@@ -1,90 +0,0 @@
|
|
|
1
|
-
# 04 · Step 2 — Scenarios
|
|
2
|
-
|
|
3
|
-
[← 03 Step 1 Specify](./03-step-1-specify.md) · [Contents](./README.md) · Next: [05 Step 3 Contract →](./05-step-3-contract.md)
|
|
4
|
-
|
|
5
|
-
> **Purpose:** rewrite each rule from the spec as a concrete, pass-or-fail scenario.
|
|
6
|
-
> **Produces:** the **§2 SCENARIOS** block of the task's `TASK.md` (one file per task — what this chapter is the *why* behind; historically a separate `features/<name>.feature`).
|
|
7
|
-
> **Person's job:** decide what "correct" looks like in concrete situations. **AI's job:** draft the scenarios.
|
|
8
|
-
|
|
9
|
-
> **Part of the specification bundle (v7).** In the default flow these scenarios are drafted by the AI alongside the spec, contract, and failing tests as **one bundle**, approved by a person **once** (the one approval), at the contract freeze — not signed off step by step. This chapter is how to get the scenarios *right*; [05 Contract](./05-step-3-contract.md) is where the bundle is frozen. See [11 Governance](./11-governance.md).
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
## Why turn rules into scenarios
|
|
14
|
-
|
|
15
|
-
A plain rule is still open to interpretation. "Source must have enough balance" leaves open: enough for what, exactly? What happens to the balances when it is *not* enough? A scenario removes the interpretation by pinning a specific situation to a specific expected result.
|
|
16
|
-
|
|
17
|
-
Scenarios occupy a unique position: they are **readable by people and checkable by machines at the same time.** A product owner can confirm a scenario is what they meant; a test can be generated directly from it. This makes them the bridge between the human-led half of the flow and the machine-led back. They are the single most leverage-bearing artifact in the method, because everything downstream — the tests, and through them the build's definition of success — is generated from them.
|
|
18
|
-
|
|
19
|
-
## The form
|
|
20
|
-
|
|
21
|
-
Each scenario has three parts:
|
|
22
|
-
|
|
23
|
-
- **Given** — the starting situation.
|
|
24
|
-
- **When** — the action taken.
|
|
25
|
-
- **Then** — the result that must follow.
|
|
26
|
-
|
|
27
|
-
Where a rule also constrains what must *not* change, add an **And** clause to state it. Unwanted side effects are caught by what you assert stays the same, not only by what you assert changes.
|
|
28
|
-
|
|
29
|
-
## Template
|
|
30
|
-
|
|
31
|
-
```
|
|
32
|
-
Scenario: <short name>
|
|
33
|
-
Given <starting situation>
|
|
34
|
-
When <action>
|
|
35
|
-
Then <expected result>
|
|
36
|
-
And <what must remain unchanged> # when relevant
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
## ▶ Example
|
|
40
|
-
|
|
41
|
-
```
|
|
42
|
-
Scenario: successful transfer
|
|
43
|
-
Given A has 100 and B has 0, both mine
|
|
44
|
-
When I transfer 30 from A to B
|
|
45
|
-
Then A has 70 and B has 30
|
|
46
|
-
|
|
47
|
-
Scenario: insufficient funds
|
|
48
|
-
Given A has 20, mine
|
|
49
|
-
When I transfer 50 from A to B
|
|
50
|
-
Then it is rejected "insufficient_funds"
|
|
51
|
-
And no balance changes
|
|
52
|
-
|
|
53
|
-
Scenario: not my account
|
|
54
|
-
Given account C is not mine
|
|
55
|
-
When I transfer 10 from C to B
|
|
56
|
-
Then it is rejected "forbidden"
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
The `And no balance changes` line is doing real work: it specifies that a rejected transfer must leave the world untouched — a property the AI could easily violate by deducting before checking.
|
|
60
|
-
|
|
61
|
-
### Tagging back to a rule ID (optional — opt-in by usage)
|
|
62
|
-
|
|
63
|
-
If §1's Musts and Rejects carry stable IDs (`M1:`, and a Reject's own error code as `R:<error_code>`), tag each `Scenario:` line with the ID(s) it covers, e.g. `# M1, R:amount_invalid`. Once a task tags even one scenario this way, `add.py check` starts confirming every §1 ID is covered by a tag here or a `covers:` line in §4 TESTS — a task that never tags anything is left alone. See the template's own inline example for the exact grammar.
|
|
64
|
-
|
|
65
|
-
## Cover the edge cases
|
|
66
|
-
|
|
67
|
-
The transfer above is one domain; the same gaps recur in every domain — an HR leave request, a marketing campaign send, a checkout. Beyond the spec's "Reject" rules, sweep the recurring gaps and add a scenario for each that applies (or rule it out on purpose): boundary, duplicate/idempotent, ownership, stale/out-of-order, partial failure, concurrency, malformed input, limits/volume.
|
|
68
|
-
|
|
69
|
-
## The AI's role here
|
|
70
|
-
|
|
71
|
-
Hand the AI the spec and have it draft a scenario for each rule, including the rejection rules. Then read them as the person who owns the requirement: do they describe what you actually meant? Correct any that drift. See `playbook/2_scenarios.md` in [Appendix B](./appendix-b-prompts.md).
|
|
72
|
-
|
|
73
|
-
## Common mistakes
|
|
74
|
-
|
|
75
|
-
- **Only happy-path scenarios.** Every "Reject" rule in the spec needs its own scenario, or that rule will never be verified.
|
|
76
|
-
- **Edge cases left to the build.** A boundary, a duplicate, or a partial failure with no scenario becomes whatever the AI happens to code. Sweep the categories above against the task's domain.
|
|
77
|
-
- **Vague results.** "Then it works" is not checkable. The result must be a specific, observable fact ("A has 70").
|
|
78
|
-
- **Forgetting the unchanged state.** For any rejection, assert that nothing changed; otherwise a partial, corrupting failure can pass.
|
|
79
|
-
|
|
80
|
-
## Exit check
|
|
81
|
-
|
|
82
|
-
- [ ] Every "Must" rule has at least one scenario.
|
|
83
|
-
- [ ] Every "Reject" rule has at least one scenario.
|
|
84
|
-
- [ ] The edge-case categories that apply to this task's domain have a scenario (or are ruled out on purpose).
|
|
85
|
-
- [ ] Each scenario's result is a specific, observable fact.
|
|
86
|
-
- [ ] Rejections assert what must stay unchanged.
|
|
87
|
-
|
|
88
|
-
## If the check fails
|
|
89
|
-
|
|
90
|
-
A rule with no scenario will never be tested, and therefore will never be verified — it is a rule in name only. Either write the missing scenario or remove the rule from the spec. Do not carry an unscenarioed rule into the contract.
|
|
@@ -1,80 +0,0 @@
|
|
|
1
|
-
# 05 · Step 3 — Contract
|
|
2
|
-
|
|
3
|
-
[← 04 Step 2 Scenarios](./04-step-2-scenarios.md) · [Contents](./README.md) · Next: [06 Step 4 Tests →](./06-step-4-tests.md)
|
|
4
|
-
|
|
5
|
-
> **Purpose:** fix the external shape of the feature — interfaces, data structures, names, and error cases — and freeze it.
|
|
6
|
-
> **Produces:** `contracts/<name>.md` (plus a mock and contract tests).
|
|
7
|
-
> **Person's job:** approve and freeze the shape. **AI's job:** generate the first draft, the mock, and the contract tests.
|
|
8
|
-
|
|
9
|
-
> **The one approval lands here (v7).** In the default flow the AI drafts spec, scenarios, this contract, and the failing tests as **one specification bundle**, and a person gives a **single approval at this freeze**. Freezing the contract is the one human gate of the bundle, not the third of three sign-offs; reject any part and the whole bundle returns to draft (backward correction, not failure). See [11 Governance](./11-governance.md).
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
## The decision point of the whole method
|
|
14
|
-
|
|
15
|
-
This step is the decision point between the human-led and machine-led halves of the flow, and it is what makes everything after it safe.
|
|
16
|
-
|
|
17
|
-
The reasoning is simple. The AI is allowed to write and rewrite code quickly. That is only safe if there is a stable surface that the rest of the system depends on and that the AI is not allowed to disturb. The frozen contract is that surface. Below it, the code is disposable and can be regenerated freely; above it, nothing breaks, because the shape it depends on does not move.
|
|
18
|
-
|
|
19
|
-
Freezing the contract is therefore not bureaucracy — it is the precondition for granting the AI real autonomy in the build step. Without it, every regeneration risks silently changing an interface that another part of the system relies on.
|
|
20
|
-
|
|
21
|
-
## What the contract contains
|
|
22
|
-
|
|
23
|
-
- **Interfaces** — the endpoints, functions, or messages, with their inputs and outputs.
|
|
24
|
-
- **Data structures** — the request and response shapes, and the persistent schema.
|
|
25
|
-
- **Names** — drawn from the project glossary, so the same concept has the same name everywhere.
|
|
26
|
-
- **Error cases** — the defined failures, using the error codes from the spec.
|
|
27
|
-
|
|
28
|
-
## Template
|
|
29
|
-
|
|
30
|
-
```
|
|
31
|
-
# contracts/<name>.md
|
|
32
|
-
<METHOD> <path> body: { <fields> }
|
|
33
|
-
200 -> { <success fields> }
|
|
34
|
-
4xx -> { error: "<code>" | "<code>" }
|
|
35
|
-
Schema: <tables/fields touched, and access pattern>
|
|
36
|
-
Status: FROZEN @ v<n>
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
## ▶ Example
|
|
40
|
-
|
|
41
|
-
```
|
|
42
|
-
POST /transfers body: { fromAccountId, toAccountId, amount }
|
|
43
|
-
200 -> { transferId, fromBalance, toBalance }
|
|
44
|
-
400 -> { error: "amount_invalid" | "same_account" | "insufficient_funds" }
|
|
45
|
-
403 -> { error: "forbidden" }
|
|
46
|
-
Schema: accounts.balance (read + write, must be transactional)
|
|
47
|
-
Status: FROZEN @ v1
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
Every error code traces back to a rejection rule in the spec, and the schema note (`must be transactional`) flags the one place where correctness depends on more than shape — a hint the verification step will follow up.
|
|
51
|
-
|
|
52
|
-
## The AI's role here
|
|
53
|
-
|
|
54
|
-
The AI generates the contract from the spec and design, and additionally produces two things that make the contract enforceable: a **mock server** that returns the contracted shapes, and **contract tests** that pin those shapes. With the mock in place, work that depends on this feature can proceed before the real code exists. See `playbook/3_contract.md` in [Appendix B](./appendix-b-prompts.md).
|
|
55
|
-
|
|
56
|
-
## The change-request rule
|
|
57
|
-
|
|
58
|
-
Once frozen, a contract does not change casually. A needed change is a **change request**: you return to [Step 1](./03-step-1-specify.md), adjust the spec, re-freeze at a new version, and come forward again. The AI never alters a frozen contract on its own initiative.
|
|
59
|
-
|
|
60
|
-
This rule is what keeps the contract trustworthy as a foundation. If it could drift, nothing built on it would be safe.
|
|
61
|
-
|
|
62
|
-
> **Do:** version and freeze the contract before any implementation.
|
|
63
|
-
> **Don't:** let the build step quietly change an interface to make code easier — that breaks everything depending on it.
|
|
64
|
-
|
|
65
|
-
## Common mistakes
|
|
66
|
-
|
|
67
|
-
- **Inconsistent names.** If the contract calls it `fromAccountId` and the schema calls it `src_acct`, the AI will produce subtle mismatches. Use the glossary everywhere.
|
|
68
|
-
- **Undefined errors.** Every failure the spec rejects must have a contracted response, or callers cannot handle it.
|
|
69
|
-
- **Freezing too early or too late.** Freeze once the spec and design are stable — not before they are agreed, and not after code has already been written against an unfrozen shape.
|
|
70
|
-
|
|
71
|
-
## Exit check
|
|
72
|
-
|
|
73
|
-
- [ ] Contract is versioned and marked `FROZEN`.
|
|
74
|
-
- [ ] Contract tests pass against the mock.
|
|
75
|
-
- [ ] Every name matches the project glossary.
|
|
76
|
-
- [ ] Every spec rejection has a contracted error response.
|
|
77
|
-
|
|
78
|
-
## If the check fails
|
|
79
|
-
|
|
80
|
-
If the contract is not yet stable enough to freeze, the upstream artifacts are not settled — return to the spec or scenarios and resolve what is still open. If a frozen contract later needs to change, treat it as a change request rather than an edit; the discipline is the point.
|
package/docs/06-step-4-tests.md
DELETED
|
@@ -1,85 +0,0 @@
|
|
|
1
|
-
# 06 · Step 4 — Tests
|
|
2
|
-
|
|
3
|
-
[← 05 Step 3 Contract](./05-step-3-contract.md) · [Contents](./README.md) · Next: [07 Step 5 Build →](./07-step-5-build.md)
|
|
4
|
-
|
|
5
|
-
> **Purpose:** turn the scenarios and contract into automated tests, and confirm they fail before any code exists.
|
|
6
|
-
> **Produces:** a failing (red) automated test suite.
|
|
7
|
-
> **Person's job:** set the targets and coverage. **AI's job:** generate the tests.
|
|
8
|
-
|
|
9
|
-
> **Part of the specification bundle (v7).** In the default flow these tests are drafted by the AI as part of the specification **bundle** (spec · scenarios · contract · tests) and approved by a person **once**, at the contract freeze — the tests are part of what that one approval covers. They still must be **red before the build**. See [11 Governance](./11-governance.md).
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
## Why tests come before code
|
|
14
|
-
|
|
15
|
-
This is the step that operationalizes the second principle — *trust through evidence, not inspection.* The tests written here are how you will judge the AI's code in [Step 5](./07-step-5-build.md). For that judgment to be honest, the tests must exist *before* the code.
|
|
16
|
-
|
|
17
|
-
The reason is mechanical. If code is written first and tests after, the tests are unconsciously shaped to match whatever the code happens to do — including its mistakes. Tests written first, from the scenarios, are shaped only by the agreed definition of correct. They are an independent standard the code must rise to meet, not a description of what the code already does.
|
|
18
|
-
|
|
19
|
-
## The must-fail principle
|
|
20
|
-
|
|
21
|
-
After generating the tests, you run them — and they must **fail**, because no implementation exists yet. This sounds trivial and is not. A test that passes before any code is written is testing nothing; it is a false reassurance that will later wave bad code through. Confirming the suite is "red for the right reason" (a missing implementation, not a broken test) is what makes it genuinely protective.
|
|
22
|
-
|
|
23
|
-
## What to test
|
|
24
|
-
|
|
25
|
-
- **One test per scenario** — every scenario from [Step 2](./04-step-2-scenarios.md) becomes an executable test.
|
|
26
|
-
- **Contract conformance** — tests that pin the shapes and error responses from [Step 3](./05-step-3-contract.md).
|
|
27
|
-
- **Edge cases from the spec** — the boundary values implied by the "Reject" rules.
|
|
28
|
-
- **Behavior, not internals** — tests assert what the feature does (the observable result), never how it is implemented, so the code can be regenerated freely beneath them.
|
|
29
|
-
|
|
30
|
-
## ▶ Example
|
|
31
|
-
|
|
32
|
-
```python
|
|
33
|
-
def test_successful_transfer():
|
|
34
|
-
a = account(balance=100, owner=me); b = account(balance=0, owner=me)
|
|
35
|
-
r = transfer(a.id, b.id, 30)
|
|
36
|
-
assert r.status == 200
|
|
37
|
-
assert a.balance == 70 and b.balance == 30
|
|
38
|
-
|
|
39
|
-
def test_insufficient_funds():
|
|
40
|
-
a = account(balance=20, owner=me); b = account(balance=0, owner=me)
|
|
41
|
-
r = transfer(a.id, b.id, 50)
|
|
42
|
-
assert r.status == 400 and r.error == "insufficient_funds"
|
|
43
|
-
assert a.balance == 20 # unchanged — the side-effect assertion
|
|
44
|
-
|
|
45
|
-
def test_not_my_account():
|
|
46
|
-
c = account(balance=100, owner=someone_else); b = account(balance=0, owner=me)
|
|
47
|
-
r = transfer(c.id, b.id, 10)
|
|
48
|
-
assert r.status == 403 and r.error == "forbidden"
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
Run this now, with no implementation: all three fail. That is the correct, honest starting point for the build.
|
|
52
|
-
|
|
53
|
-
### Declaring which rule a test covers (optional — opt-in by usage)
|
|
54
|
-
|
|
55
|
-
If §1's Musts and Rejects carry stable IDs, add a trailing `covers: M1, R:amount_invalid` to a test-plan line to declare which ID(s) that test satisfies. Once any test in a task declares a `covers:` line (or any §2 scenario carries a tag), `add.py check` confirms every §1 ID is covered by a tag or a `covers:` line somewhere — a task that never uses either is left alone. See the template's own inline example for the exact grammar.
|
|
56
|
-
|
|
57
|
-
## The AI's role here
|
|
58
|
-
|
|
59
|
-
The AI generates the test suite from the scenarios and contract. Your job is to confirm two things it cannot judge for itself: that each test asserts *behavior* rather than internal detail, and that none of them pass by accident before code exists. See `playbook/4_tests.md` in [Appendix B](./appendix-b-prompts.md).
|
|
60
|
-
|
|
61
|
-
## Common mistakes
|
|
62
|
-
|
|
63
|
-
- **Tests that test the implementation.** Asserting on private internals couples the test to one version of the code and defeats disposability.
|
|
64
|
-
- **A green suite before the build.** Means the tests are not actually exercising the missing feature — fix them now.
|
|
65
|
-
- **Skipping the side-effect assertions.** Without `assert a.balance == 20` on the rejection path, a corrupting partial failure passes silently.
|
|
66
|
-
- **No coverage target.** Without a recorded target, coverage can quietly erode during the build.
|
|
67
|
-
- **`should_panic` as a red test.** Marking a test `#[should_panic(expected = "implement in green wave")]` (or the equivalent in any language) passes immediately and stays green while red — it is a lying red. Declare unimplemented paths with `todo!()` (or `unimplemented!()`) so the test actually fails. If a test is intentionally designed to flip from red to green during the build, say so with a comment: `// flip authorized at green wave`.
|
|
68
|
-
- **Collateral tests named by category, not by exact name.** When a spec adds a slash command, a new CLI subcommand, or any other globally-enumerated thing, there is a fixed collateral set of tests that count or enumerate it (e.g. a command-registry count test, a help-text snapshot, an autocomplete positional assert). Pre-list these tests by their **exact test names** in §4 — not categories — so the build agent's edits to those "pre-existing" tests are expected and the count is right. Naming only the category means the agent finds the wrong test or misses one.
|
|
69
|
-
- **Arithmetic not checked against frozen constants.** Before freezing, check that the red suite can reach green: a fixture with N bytes fails a hard-coded M-byte budget if N > M — the suite can never pass. Run the numbers before freeze, and add an additive override (e.g. `set_budget`) when the scenario implies a limit the production constant cannot satisfy in test.
|
|
70
|
-
- **Non-hermetic tests that read real user state.** Tests that call a loader with `None` (defaulting to `~/.helios/settings.json` or the real home dir) become torn-read flakes under a parallel suite and assert nothing useful. Red tests that create or read production paths must redirect them to a temp dir; grep new tests for `home_dir`, `~/.config`, real-path defaults before freeze.
|
|
71
|
-
- **Tests that share a per-machine singleton without isolation.** Background services (embedded servers, filesystem watchers) bind to fixed ports or paths. Tests that start such a service must tear it down, or they collide with a parallel run or an already-running dev instance. If the singleton cannot be isolated, gate those tests as serial (one thread, no parallel execution) and document it.
|
|
72
|
-
|
|
73
|
-
## Exit check
|
|
74
|
-
|
|
75
|
-
- [ ] One test exists per scenario.
|
|
76
|
-
- [ ] The suite runs in the pipeline and is **red for the right reason**.
|
|
77
|
-
- [ ] Tests assert observable behavior, not internals.
|
|
78
|
-
- [ ] A coverage target is recorded.
|
|
79
|
-
- [ ] No `should_panic` lying reds — unimplemented paths use `todo!()` or equivalent so they actually fail.
|
|
80
|
-
- [ ] Collateral tests for globally-enumerated things (command counts, help snapshots) are listed by exact name.
|
|
81
|
-
- [ ] Arithmetic checked: the red fixtures can reach green against the frozen constants.
|
|
82
|
-
|
|
83
|
-
## If the check fails
|
|
84
|
-
|
|
85
|
-
If a test passes before any implementation, it is a fake test — repair it before continuing, because it is your only independent check on the AI. If the suite is red for the wrong reason (a syntax or harness error), fix the harness first; a build cannot be judged against a broken net.
|
package/docs/07-step-5-build.md
DELETED
|
@@ -1,82 +0,0 @@
|
|
|
1
|
-
# 07 · Step 5 — Build
|
|
2
|
-
|
|
3
|
-
[← 06 Step 4 Tests](./06-step-4-tests.md) · [Contents](./README.md) · Next: [08 Step 6 Verify →](./08-step-6-verify.md)
|
|
4
|
-
|
|
5
|
-
> **Purpose:** have the AI implement the feature so that every failing test passes.
|
|
6
|
-
> **Produces:** working code, plus the evidence that the tests now pass.
|
|
7
|
-
> **Person's job:** direct, in small batches. **AI's job:** implement.
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## The only step the AI leads
|
|
12
|
-
|
|
13
|
-
This is the step the AI is genuinely good at, and the only one where it should be doing the heavy lifting. It works precisely because the previous four steps removed all the ambiguity: the AI is no longer guessing what to build: it has a spec, a set of scenarios, a frozen contract, and a suite of failing tests that define "done" exactly. Its task is narrow and checkable — turn the suite green.
|
|
14
|
-
|
|
15
|
-
This is the difference between AIDD and vague-prompt coding. The same agent that produces confident nonsense from "build me a transfer feature" produces correct, bounded code from "make these specific failing tests pass without changing them." The agent did not change; the direction did.
|
|
16
|
-
|
|
17
|
-
## The build prompt
|
|
18
|
-
|
|
19
|
-
The instruction is explicit about constraints, because the constraints are what keep the speed safe.
|
|
20
|
-
|
|
21
|
-
```
|
|
22
|
-
Read SPEC.md, contracts/<name>.md, and tests/<name>_test.py.
|
|
23
|
-
Implement the feature so that EVERY test passes.
|
|
24
|
-
Constraints:
|
|
25
|
-
- Do NOT change any test.
|
|
26
|
-
- Do NOT change the contract.
|
|
27
|
-
- <feature-specific safety rule>.
|
|
28
|
-
- Stop and ask if any requirement is unclear — do not guess.
|
|
29
|
-
- Use only packages listed in dependencies.allowlist.
|
|
30
|
-
Report which tests pass and exactly what you changed.
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
For the running example, the feature-specific safety rule is *"make the balance update atomic — debit and credit occur in a single transaction."* This is the one correctness property the tests alone may not force, so it is named directly to the builder.
|
|
34
|
-
|
|
35
|
-
See `playbook/5_build.md` in [Appendix B](./appendix-b-prompts.md).
|
|
36
|
-
|
|
37
|
-
## Work in small batches
|
|
38
|
-
|
|
39
|
-
Direct the AI one task at a time, and keep each task small enough that its result can be reviewed in full. This is a direct application of the principle *you cannot move faster than you can verify.* A single enormous change that turns the whole suite green at once is not a triumph — it is an unreviewable blob. Small batches keep the verification step (next chapter) tractable and keep a human genuinely in the loop.
|
|
40
|
-
|
|
41
|
-
## The iteration loop
|
|
42
|
-
|
|
43
|
-
```
|
|
44
|
-
AI writes code → pipeline runs the tests → some still fail
|
|
45
|
-
→ AI iterates → ... → all green → hand to Verify
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
The loop is tight and largely autonomous within a task: the AI runs the tests, sees what fails, and adjusts. Your attention is needed at the boundaries — defining the task going in, and reviewing the result coming out — not on each internal iteration.
|
|
49
|
-
|
|
50
|
-
## The cardinal rule: never change the test to pass
|
|
51
|
-
|
|
52
|
-
An AI under pressure to make a suite green has an available shortcut: weaken or delete the failing test. This must be forbidden explicitly and caught reliably. A test changed to fit the code inverts the entire method — the code is now judging itself. If you find a test was altered during the build, reject the change outright and re-prompt with the constraint restated.
|
|
53
|
-
|
|
54
|
-
The same applies to the contract: the build implements *against* the frozen contract and may not edit it. A genuine need to change either is a change request that returns to an earlier step.
|
|
55
|
-
|
|
56
|
-
## How much autonomy
|
|
57
|
-
|
|
58
|
-
The autonomy granted in this step should match the evidence and your review capacity (see [11 Governance](./11-governance.md)):
|
|
59
|
-
|
|
60
|
-
- Where the area is new or risky, the AI proposes and a person reviews every change.
|
|
61
|
-
- Where the contract and tests are solid, the AI generates freely and a person reviews each batch.
|
|
62
|
-
- Only in narrow, well-tested areas, with a full evidence bundle attached, may the AI integrate its own work.
|
|
63
|
-
|
|
64
|
-
## Common mistakes
|
|
65
|
-
|
|
66
|
-
- **Batches too large to review.** Shrinks verification to approving without reading.
|
|
67
|
-
- **Letting the AI add unknown dependencies.** The allow-list check in the pipeline should block this automatically; if it does not, the supply-chain risk is real (an AI may invent a plausible package name that an attacker has registered).
|
|
68
|
-
- **Accepting "all tests pass" without reading the change.** Passing tests are necessary, not sufficient — the next step exists for exactly this reason.
|
|
69
|
-
|
|
70
|
-
## Exit check
|
|
71
|
-
|
|
72
|
-
- [ ] All tests pass.
|
|
73
|
-
- [ ] Test coverage did not decrease.
|
|
74
|
-
- [ ] No test and no contract was modified by the AI.
|
|
75
|
-
- [ ] No dependency outside the allow-list was added.
|
|
76
|
-
- [ ] The change is small enough to review in full.
|
|
77
|
-
|
|
78
|
-
## If the check fails
|
|
79
|
-
|
|
80
|
-
If the AI weakened a test, reject and re-prompt. If it added an out-of-allow-list package, the pipeline blocks it; have the AI find an approved alternative or raise the package for human approval. If the batch is too large to review, ask the AI to split the work and resubmit. Only once the exit check passes does the change proceed to verification.
|
|
81
|
-
|
|
82
|
-
And in the other direction: if the *verify* gate later finds a confirmed cheat — a tamper, or a build that gamed the green (overfit to the fixtures, vacuous asserts, stubbed-away logic) — the task returns *here* for an honest redo. That return is the **bounded self-heal loop** (see the run chapter): revert the tampered file or de-overfit the code, then advance again. It is capped — after the cap a confirmed cheat HARD-STOPs to the human rather than looping forever, and a gamed green is never auto-passed.
|
package/docs/08-step-6-verify.md
DELETED
|
@@ -1,105 +0,0 @@
|
|
|
1
|
-
# 08 · Step 6 — Verify
|
|
2
|
-
|
|
3
|
-
[← 07 Step 5 Build](./07-step-5-build.md) · [Contents](./README.md) · Next: [09 The loop →](./09-the-loop.md)
|
|
4
|
-
|
|
5
|
-
> **Purpose:** confirm the result is correct and safe to release.
|
|
6
|
-
> **Produces:** a reviewed change with a recorded outcome, ready to release.
|
|
7
|
-
> **Who resolves it:** set per task by the `autonomy:` header. Under `autonomy: auto` (the default) the run resolves the gate on evidence; under `conservative`, or for any residue, it is the human's check. **Security always escalates to a human.**
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## Where trust is actually established
|
|
12
|
-
|
|
13
|
-
The build produced passing tests. That is necessary but not sufficient. Verification is where a person establishes trust — and the principle governing it is *trust through evidence, not inspection.*
|
|
14
|
-
|
|
15
|
-
This needs care, because it is easy to misread. "Not by inspection" does not mean "do not look at the code." It means the *basis* of trust is the passing evidence plus a deliberate check of the specific things tests cannot easily catch — not a general impression that the code reads plausibly. Plausibility is exactly the trap: AI code is frequently plausible and wrong. So verification has two parts: confirm the evidence, then check the known non-functional risks.
|
|
16
|
-
|
|
17
|
-
## Who resolves Verify — the automated quality gate
|
|
18
|
-
|
|
19
|
-
Verify can be resolved two ways, set per task by the `autonomy:` header (see [governance](./11-governance.md) and the autonomy level):
|
|
20
|
-
|
|
21
|
-
- **Auto (the default).** When `autonomy: auto`, the run resolves the gate on **evidence** rather than waiting for a person — but only when *all* of these hold — the **same canonical list run.md owns**: every test green, coverage not decreased, no test weakened and no contract edited, the convergence loops dry, the completeness-critic clean, the deep check filled, the earned-green refute-read verdict recorded (the §6 `### Refute-read verdict` field, under auto), and **no residue** (security, concurrency, or architecture). It records `PASS` as *auto-resolved*, naming the run as the accountable owner — an explicit pass, not a skip. This is principle 7: a gate may be resolved by evidence when that evidence is sufficient and the result is logged.
|
|
22
|
-
- **Human.** When `autonomy: conservative`, or whenever the run finds residue it cannot judge, the gate stops for a person; the two parts below are theirs.
|
|
23
|
-
|
|
24
|
-
**Security is always a `HARD-STOP` and is never auto-passed, at any autonomy level.** The two parts that follow — confirm the evidence, then check the non-functional risks — are what *either* resolver works through; the only question is whether a person or the recorded run signs the outcome.
|
|
25
|
-
|
|
26
|
-
## Part one — confirm the evidence
|
|
27
|
-
|
|
28
|
-
Before the build, the task pre-declares its **build expectations** — the observable outcomes a correct build must produce, read off §2's scenarios and §3's contract. Confirm each one here against evidence you can see, not a restated test name: this is what makes the gate check the build is *right*, not merely that the suite is green.
|
|
29
|
-
|
|
30
|
-
- [ ] All tests pass.
|
|
31
|
-
- [ ] Coverage did not decrease.
|
|
32
|
-
- [ ] No test or contract was altered during the build.
|
|
33
|
-
- [ ] Every pre-declared build expectation is confirmed by real evidence (not merely a green test).
|
|
34
|
-
- [ ] Every §1 Must/Reject rule still traces to a §2 scenario tag or a §4 `covers:` line.
|
|
35
|
-
- [ ] Every symbol §3's contract cites still resolves in the *current* tree, not just at Ground SHA.
|
|
36
|
-
|
|
37
|
-
If any of these is false, stop here and return to the build; there is nothing to verify yet.
|
|
38
|
-
|
|
39
|
-
A green suite proves the tests you wrote pass — it says nothing about a rule you never wired a scenario or test to at all. That is a different failure than the ones Part four covers: not a cheated pass, but a rule with no trace of ever being checked. `add.py check` scans every task project-wide for exactly this — a §1 Must or Reject with no §2 scenario tag and no §4 `covers:` line — but that sweep is easy to forget to run. `add.py audit`'s `rule_coverage_gap` notice surfaces the SAME check for the task actually at this gate, the moment it reaches Verify (a task that never adopted the tag convention at all is silently grandfathered, never retro-flagged). Neither blocks the gate; both make an already-real gap impossible to miss.
|
|
40
|
-
|
|
41
|
-
The second check guards against a quieter kind of drift. §0's Ground SHA anchors the symbols §3's contract cites to the commit they were read at — but code moves *during* the build that follows. Before this gate, re-resolve every one of those symbols against the tree as it stands now, not the Ground SHA snapshot, and record it in §6's **Live-verify evidence** block. A moved or renamed anchor caught here is a one-line note; the same drift left silent is a future reader chasing a line number that no longer means what it did at Ground.
|
|
42
|
-
|
|
43
|
-
## Part two — check what tests miss
|
|
44
|
-
|
|
45
|
-
Automated tests are excellent at behavior on defined inputs and poor at a few specific things. Check those by hand, every time:
|
|
46
|
-
|
|
47
|
-
- **Concurrency and timing.** Is the operation correct when two of them happen at once? Tests usually run serially and miss races.
|
|
48
|
-
- ▶ *Example: the balance update must be one atomic transaction. Confirm that two simultaneous transfers from the same account cannot both pass the balance check and overdraw it.* This is the single most important check for this feature, and it is the reason the build prompt named atomicity explicitly.
|
|
49
|
-
- **Security.** Are there exposed secrets, injection openings, or unexpected dependencies? AI-generated code is known to hardcode secrets and to pull in packages by plausible-but-wrong names. A security finding is always a `HARD-STOP`, never a waiver — and it escalates only if it is *written down*: `add.py audit`'s `unescalated_security_note` check fires on a **marked** note that slipped through as an auto-gate PASS, but a finding you **never marked** is invisible to the engine, escalated to no one. That is the honest limit of the auto-gate: under `auto`, a human **spot-audit** (reading the diff) is the only backstop for a *missed* security finding — the same backstop named for an unrecorded earned-green refute-read.
|
|
50
|
-
- **Architecture conformance.** Does the change respect the layering and dependency rules in `CONVENTIONS.md`? Speed with no architectural check produces a fast-growing tangle that becomes unmaintainable within months.
|
|
51
|
-
|
|
52
|
-
## Part three — the deep check (do not skim)
|
|
53
|
-
|
|
54
|
-
Two failures slip straight past green tests. The first is code that is never *wired in* — a new function that nothing calls, an endpoint no route reaches: the tests for it pass in isolation while the feature is, in practice, absent. The second is the opposite — code left *dead* behind a path nothing exercises, quietly rotting. And for a change that produced prose rather than code, the equivalent failure is signing off on a claim you never actually read in full. Plausibility hides all three. So verification carries one explicit requirement beyond the non-functional review:
|
|
55
|
-
|
|
56
|
-
> Deep check — do not skim. If the task produced code, record that every new symbol is referenced (wiring) and that no new dead/unused code was introduced. If it produced prose or non-code, record a semantic read — what you read in full and what it confirmed. Which path applies is the resolver's judgement; the engine never classifies.
|
|
57
|
-
|
|
58
|
-
This is *evidence*, not impression: a reference search showing where each new symbol is called, a scan confirming nothing new is orphaned, or — for prose — a note of exactly what was read and what it confirmed. An unfilled deep check is a **shallow verify**, not a pass. The engine cannot judge wiring, dead code, or whether prose was truly read; the resolver records the evidence, and a person (under `conservative`) or the recorded run (under `auto`) signs it.
|
|
59
|
-
|
|
60
|
-
**The wiring trace is a named step, not a free-form note.** For every new hook, closure, or middleware registered in this task: trace from the process entry point to the call site and record it explicitly — symbol, file, line. A symbol that is only reachable via a test helper or `make_config` but not via the production entry point (e.g. `build_harness_with_dispatcher`, `interactive_mode`) is not wired. This is the third repeated class in production: "runtime-activation-order/silent-noop" — the code exists and the unit tests pass, but the feature is absent in the running program. The wiring trace is how you catch it before a user does.
|
|
61
|
-
|
|
62
|
-
## Part four — was the green earned?
|
|
63
|
-
|
|
64
|
-
Passing tests say the code satisfies the cases you wrote down. They do not say it earned that pass honestly — and the mechanical tamper tripwire (Step 6's floor) only catches an *edited* test or contract, not a build that gamed the *unchanged* suite. The same rubric the phase guide carries names what the tripwire cannot see:
|
|
65
|
-
|
|
66
|
-
A green suite proves the tests pass — not that the build EARNED them. Three judgment cheats pass the unchanged suite without earning it: src overfit to the test fixtures (special-cased to the literal inputs, not the general behavior §1 asked for), vacuous asserts (tautological — green even against an empty implementation), and real logic stubbed away (the function returns a constant the tests happen to accept). These cheats are invisible to the mechanical tamper tripwire, which only sees edited files. Score them with an adversarial refute-read: an independent reviewer — a subagent under `autonomy: auto` is recommended, the engine never spawns one — prompted to argue the green was NOT earned from outside the build context. This is the verify-gate, whole-suite specialization of run.md's adversarial verify (see run.md), not a new discipline. A confirmed earned-green failure is HARD-STOP-class: never auto-passed, never RISK-ACCEPTED — but a first cheat is a chance to redo: a confirmed cheat (mechanical tamper or a reported earned-green failure) enters the bounded self-heal loop — it returns to build for an honest redo, and only after the loop's cap does it HARD-STOP to the human (the loop lives in run.md).
|
|
67
|
-
|
|
68
|
-
Under `autonomy: auto` there is no human at the gate to confirm the refute-read happened, so the verdict is **recorded** — a `### Refute-read verdict` field in §6 (`Verdict: EARNED | NOT-EARNED`, plus who read it and what they adversarially probed). This closes an honesty gap the auto-gate would otherwise leave open: the method *says* an auto-PASS rests on an earned-green check, but nothing made that check visible. Now `add.py audit` surfaces a verify-or-later task whose verdict block is still a placeholder as `refute_unrecorded` — a **measure, not a gate**: it never blocks the run (a hard gate on this universal completion path would break far too much to justify it, and the engine cannot read whether a self-graded `EARNED` is *true* anyway — only whether it was written). The honest backstop is the same one named for a missed security finding: a human **spot-audit** reading the diff. A `NOT-EARNED` verdict is not a dead end — it routes through `add.py heal` for an honest redo, never an auto-PASS. The discipline this records is simple: under auto, you are the accountable owner of the gate, so you leave the trace of the check you owe.
|
|
69
|
-
|
|
70
|
-
## Recording the outcome
|
|
71
|
-
|
|
72
|
-
Every verification ends with exactly one recorded outcome, with an accountable owner — never a silent pass:
|
|
73
|
-
|
|
74
|
-
| Outcome | Meaning | Allowed when |
|
|
75
|
-
|---------|---------|--------------|
|
|
76
|
-
| `PASS` | all checks met | the normal path |
|
|
77
|
-
| `RISK-ACCEPTED` | proceed with a signed waiver: named owner, linked ticket, expiry date | a non-security gap only |
|
|
78
|
-
| `HARD-STOP` | cannot proceed | any failing test or any security finding |
|
|
79
|
-
|
|
80
|
-
A security finding is always a `HARD-STOP`; it is never waved through with a waiver. A `RISK-ACCEPTED` outcome is a deliberate, documented decision to ship a known, non-security limitation — not a way to skip the check.
|
|
81
|
-
|
|
82
|
-
## The verification checklist
|
|
83
|
-
|
|
84
|
-
- [ ] All tests pass (the evidence).
|
|
85
|
-
- [ ] Every pre-declared build expectation is confirmed by observable evidence.
|
|
86
|
-
- [ ] Every §1 rule traces to §2/§4; every §3-cited symbol still resolves in the current tree.
|
|
87
|
-
- [ ] Concurrency/timing of the risky operation is safe.
|
|
88
|
-
- [ ] No exposed secrets, injection openings, or unexpected dependencies.
|
|
89
|
-
- [ ] Layering and dependencies follow `CONVENTIONS.md`.
|
|
90
|
-
- [ ] Deep check (do not skim): for code, every new symbol is referenced (wiring) and no new dead/unused code was introduced; for prose/non-code, a semantic read is recorded.
|
|
91
|
-
- [ ] The change is approved — by a person, **or** (under `autonomy: auto`, no residue) auto-resolved by the run as the recorded accountable owner.
|
|
92
|
-
- [ ] An outcome is recorded (`PASS` / `RISK-ACCEPTED` / `HARD-STOP`).
|
|
93
|
-
|
|
94
|
-
## Common mistakes
|
|
95
|
-
|
|
96
|
-
- **Shipping on plausibility.** Reading the diff, finding it reasonable, and approving — without the evidence and the non-functional review — is the precise failure the method exists to prevent.
|
|
97
|
-
- **Treating a security gap as acceptable risk.** It is a `HARD-STOP`, not a waiver.
|
|
98
|
-
- **Skipping the concurrency check** because the tests are green. Tests rarely exercise simultaneity; this is a manual check by design.
|
|
99
|
-
- **Trusting the green agent's self-reported test count.** A build agent running a filtered suite (e.g. `-E 'test(theme)'`) only sees tests inside the filter. Collateral failures outside the filter — a stale count in `all_commands_in_registry`, an e2e snapshot the agent did not touch — are invisible. The orchestrator's **full-suite rerun is load-bearing**; never skip it on the grounds that the scoped run was green.
|
|
100
|
-
- **User-observable-only failures escalate to the human before exhausting discriminating probes.** When a symptom is only observable by a person (a TCC dialog, a visual flicker, an OS-level prompt), do not respond by running the suite again. Instead, design two or three targeted probes that let the user distinguish cause A from cause B in one interaction each. Three AskUser probes resolve what three blind reruns cannot.
|
|
101
|
-
- **Background-process hangs misdiagnosed as test failures.** A test that never exits is not a failure in the test logic — it is a hang. The diagnosis recipe: background the test process, run `pgrep` to find it, use the platform profiler (`sample <pid>` on macOS, `perf` on Linux) to sample the stack, then `lsof -p <pid>` to see open files. Run an isolation experiment (suspect line on/off, 3×3) before reading any code. Entry-count caps do not bound wall time — a single huge directory or a blocking syscall inside a `spawn_blocking` call can hang indefinitely even when the entry cap is satisfied.
|
|
102
|
-
|
|
103
|
-
## If the check fails
|
|
104
|
-
|
|
105
|
-
A failing test or a security finding returns the change to the build step ([Step 5](./07-step-5-build.md)). A non-security limitation may proceed only with a signed `RISK-ACCEPTED` record carrying an owner and an expiry — so the team can find and close it later. Nothing proceeds on an unrecorded decision.
|