@aperant/framework 0.10.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +236 -1
- package/agents/apt-planner.md +15 -0
- package/agents/apt-verifier.md +15 -10
- package/bin/apt-proof-video.mjs +256 -76
- package/dist/cli/commands/apply-recipe.d.mts +6 -0
- package/dist/cli/commands/apply-recipe.d.mts.map +1 -0
- package/dist/cli/commands/apply-recipe.mjs +98 -0
- package/dist/cli/commands/apply-recipe.mjs.map +1 -0
- package/dist/cli/commands/catch-up.d.mts +9 -0
- package/dist/cli/commands/catch-up.d.mts.map +1 -0
- package/dist/cli/commands/catch-up.mjs +64 -0
- package/dist/cli/commands/catch-up.mjs.map +1 -0
- package/dist/cli/commands/mobile-prepare.d.mts +53 -0
- package/dist/cli/commands/mobile-prepare.d.mts.map +1 -0
- package/dist/cli/commands/mobile-prepare.mjs +139 -0
- package/dist/cli/commands/mobile-prepare.mjs.map +1 -0
- package/dist/cli/commands/pr-review.d.mts +16 -4
- package/dist/cli/commands/pr-review.d.mts.map +1 -1
- package/dist/cli/commands/pr-review.mjs +30 -11
- package/dist/cli/commands/pr-review.mjs.map +1 -1
- package/dist/cli/commands/recipe-diff.d.mts +6 -0
- package/dist/cli/commands/recipe-diff.d.mts.map +1 -0
- package/dist/cli/commands/recipe-diff.mjs +75 -0
- package/dist/cli/commands/recipe-diff.mjs.map +1 -0
- package/dist/cli/commands/roadmap.d.mts.map +1 -1
- package/dist/cli/commands/roadmap.mjs +53 -0
- package/dist/cli/commands/roadmap.mjs.map +1 -1
- package/dist/cli/commands/route.d.mts.map +1 -1
- package/dist/cli/commands/route.mjs +92 -2
- package/dist/cli/commands/route.mjs.map +1 -1
- package/dist/cli/commands/verify-sandbox.d.mts +6 -0
- package/dist/cli/commands/verify-sandbox.d.mts.map +1 -0
- package/dist/cli/commands/verify-sandbox.mjs +200 -0
- package/dist/cli/commands/verify-sandbox.mjs.map +1 -0
- package/dist/cli/config/share-policy.d.mts.map +1 -1
- package/dist/cli/config/share-policy.mjs +1 -0
- package/dist/cli/config/share-policy.mjs.map +1 -1
- package/dist/cli/config/upgrade-gitignore.d.mts.map +1 -1
- package/dist/cli/config/upgrade-gitignore.mjs +1 -0
- package/dist/cli/config/upgrade-gitignore.mjs.map +1 -1
- package/dist/cli/consistency/parse-qa.d.mts.map +1 -1
- package/dist/cli/consistency/parse-qa.mjs +18 -4
- package/dist/cli/consistency/parse-qa.mjs.map +1 -1
- package/dist/cli/consistency/rules/r5-verdict-consistency.d.mts.map +1 -1
- package/dist/cli/consistency/rules/r5-verdict-consistency.mjs +11 -10
- package/dist/cli/consistency/rules/r5-verdict-consistency.mjs.map +1 -1
- package/dist/cli/coordination/catch-up.d.mts +26 -0
- package/dist/cli/coordination/catch-up.d.mts.map +1 -0
- package/dist/cli/coordination/catch-up.mjs +239 -0
- package/dist/cli/coordination/catch-up.mjs.map +1 -0
- package/dist/cli/coordination/last-seen.d.mts +45 -0
- package/dist/cli/coordination/last-seen.d.mts.map +1 -0
- package/dist/cli/coordination/last-seen.mjs +128 -0
- package/dist/cli/coordination/last-seen.mjs.map +1 -0
- package/dist/cli/coordination/store.d.mts +15 -0
- package/dist/cli/coordination/store.d.mts.map +1 -1
- package/dist/cli/coordination/store.mjs +16 -0
- package/dist/cli/coordination/store.mjs.map +1 -1
- package/dist/cli/coverage-check/user-outcomes.d.mts +7 -2
- package/dist/cli/coverage-check/user-outcomes.d.mts.map +1 -1
- package/dist/cli/coverage-check/user-outcomes.mjs +50 -17
- package/dist/cli/coverage-check/user-outcomes.mjs.map +1 -1
- package/dist/cli/dispatch.d.mts.map +1 -1
- package/dist/cli/dispatch.mjs +30 -0
- package/dist/cli/dispatch.mjs.map +1 -1
- package/dist/cli/gate/gates/verify-approved.d.mts +0 -48
- package/dist/cli/gate/gates/verify-approved.d.mts.map +1 -1
- package/dist/cli/gate/gates/verify-approved.mjs +23 -80
- package/dist/cli/gate/gates/verify-approved.mjs.map +1 -1
- package/dist/cli/git/default-branch.d.mts +51 -0
- package/dist/cli/git/default-branch.d.mts.map +1 -0
- package/dist/cli/git/default-branch.mjs +234 -0
- package/dist/cli/git/default-branch.mjs.map +1 -0
- package/dist/cli/git/identity.d.mts +3 -5
- package/dist/cli/git/identity.d.mts.map +1 -1
- package/dist/cli/git/identity.mjs +10 -4
- package/dist/cli/git/identity.mjs.map +1 -1
- package/dist/cli/install/legacy-paths.d.mts.map +1 -1
- package/dist/cli/install/legacy-paths.mjs +2 -0
- package/dist/cli/install/legacy-paths.mjs.map +1 -1
- package/dist/cli/install/mcp-server-specs.d.mts +6 -5
- package/dist/cli/install/mcp-server-specs.d.mts.map +1 -1
- package/dist/cli/install/mcp-server-specs.mjs +22 -4
- package/dist/cli/install/mcp-server-specs.mjs.map +1 -1
- package/dist/cli/roadmap/backend-adapter.d.mts +43 -0
- package/dist/cli/roadmap/backend-adapter.d.mts.map +1 -0
- package/dist/cli/roadmap/backend-adapter.mjs +222 -0
- package/dist/cli/roadmap/backend-adapter.mjs.map +1 -0
- package/dist/cli/util/runtime-capabilities.d.mts +7 -0
- package/dist/cli/util/runtime-capabilities.d.mts.map +1 -1
- package/dist/cli/util/runtime-capabilities.mjs +65 -0
- package/dist/cli/util/runtime-capabilities.mjs.map +1 -1
- package/dist/cli/verify-proof/manifest-validator.mjs +13 -0
- package/dist/cli/verify-proof/manifest-validator.mjs.map +1 -1
- package/dist/cli/verify-proof/resolver.d.mts +10 -7
- package/dist/cli/verify-proof/resolver.d.mts.map +1 -1
- package/dist/cli/verify-proof/resolver.mjs +28 -7
- package/dist/cli/verify-proof/resolver.mjs.map +1 -1
- package/dist/cli/verify-proof/sandbox/apply.d.mts +64 -0
- package/dist/cli/verify-proof/sandbox/apply.d.mts.map +1 -0
- package/dist/cli/verify-proof/sandbox/apply.mjs +379 -0
- package/dist/cli/verify-proof/sandbox/apply.mjs.map +1 -0
- package/dist/cli/verify-proof/sandbox/gate-predicate.d.mts +63 -0
- package/dist/cli/verify-proof/sandbox/gate-predicate.d.mts.map +1 -0
- package/dist/cli/verify-proof/sandbox/gate-predicate.mjs +115 -0
- package/dist/cli/verify-proof/sandbox/gate-predicate.mjs.map +1 -0
- package/dist/cli/verify-proof/sandbox/pattern-matcher.d.mts +57 -0
- package/dist/cli/verify-proof/sandbox/pattern-matcher.d.mts.map +1 -0
- package/dist/cli/verify-proof/sandbox/pattern-matcher.mjs +124 -0
- package/dist/cli/verify-proof/sandbox/pattern-matcher.mjs.map +1 -0
- package/dist/cli/verify-proof/sandbox/recipe-diff.d.mts +28 -0
- package/dist/cli/verify-proof/sandbox/recipe-diff.d.mts.map +1 -0
- package/dist/cli/verify-proof/sandbox/recipe-diff.mjs +190 -0
- package/dist/cli/verify-proof/sandbox/recipe-diff.mjs.map +1 -0
- package/dist/cli/verify-proof/sandbox/sandbox-config.d.mts +25 -0
- package/dist/cli/verify-proof/sandbox/sandbox-config.d.mts.map +1 -0
- package/dist/cli/verify-proof/sandbox/sandbox-config.mjs +85 -0
- package/dist/cli/verify-proof/sandbox/sandbox-config.mjs.map +1 -0
- package/dist/cli/verify-proof/sandbox/self-test.d.mts +39 -0
- package/dist/cli/verify-proof/sandbox/self-test.d.mts.map +1 -0
- package/dist/cli/verify-proof/sandbox/self-test.mjs +248 -0
- package/dist/cli/verify-proof/sandbox/self-test.mjs.map +1 -0
- package/dist/plugin/.claude-plugin/plugin.json +2 -1
- package/dist/plugin/agents/apt-planner.md +15 -0
- package/dist/plugin/agents/apt-verifier.md +15 -10
- package/dist/plugin/skills/apt/SKILL.md +29 -0
- package/dist/plugin/skills/apt-catch-up/SKILL.md +79 -0
- package/dist/plugin/skills/apt-plan/SKILL.md +23 -0
- package/dist/plugin/skills/apt-pr-review/SKILL.md +5 -4
- package/dist/plugin/skills/apt-run/SKILL.md +1 -1
- package/dist/plugin/skills/apt-setup/SKILL.md +132 -0
- package/dist/plugin/skills/apt-verify/SKILL.md +20 -15
- package/dist/plugin/skills/apt-verify-proof/SKILL.md +146 -11
- package/dist/proof-report.d.ts.map +1 -1
- package/dist/proof-report.js +4 -1
- package/dist/proof-report.js.map +1 -1
- package/dist/types/config.d.ts +42 -4
- package/dist/types/config.d.ts.map +1 -1
- package/dist/types/qa-scoring.d.ts +45 -9
- package/dist/types/qa-scoring.d.ts.map +1 -1
- package/dist/types/qa-scoring.js +34 -13
- package/dist/types/qa-scoring.js.map +1 -1
- package/dist/types/state.d.ts +7 -2
- package/dist/types/state.d.ts.map +1 -1
- package/dist/types/task-record.d.ts +10 -2
- package/dist/types/task-record.d.ts.map +1 -1
- package/drivers/mobile/README.md +40 -0
- package/drivers/mobile/driver.mjs +106 -0
- package/drivers/mobile/manifest.json +49 -0
- package/package.json +1 -1
- package/prompts/inbox-clarification.md +11 -0
- package/prompts/inbox-triage.md +20 -0
- package/prompts/qa_orchestrator_agentic.md +18 -17
- package/prompts/qa_reviewer.md +17 -15
- package/skills/apt/SKILL.md +29 -0
- package/skills/apt-catch-up/SKILL.md +79 -0
- package/skills/apt-plan/SKILL.md +23 -0
- package/skills/apt-pr-review/SKILL.md +5 -4
- package/skills/apt-run/SKILL.md +1 -1
- package/skills/apt-setup/SKILL.md +132 -0
- package/skills/apt-verifier.md +7 -4
- package/skills/apt-verify/SKILL.md +20 -15
- package/skills/apt-verify-proof/SKILL.md +146 -11
- package/src/cli/commands/apply-recipe.mjs +105 -0
- package/src/cli/commands/catch-up.mjs +67 -0
- package/src/cli/commands/mobile-prepare.mjs +151 -0
- package/src/cli/commands/pr-review.mjs +32 -11
- package/src/cli/commands/recipe-diff.mjs +81 -0
- package/src/cli/commands/roadmap.mjs +54 -0
- package/src/cli/commands/route.mjs +92 -1
- package/src/cli/commands/verify-sandbox.mjs +231 -0
- package/src/cli/config/share-policy.mjs +1 -0
- package/src/cli/config/upgrade-gitignore.mjs +1 -0
- package/src/cli/consistency/parse-qa.mjs +20 -4
- package/src/cli/consistency/rules/r5-verdict-consistency.mjs +11 -13
- package/src/cli/coordination/catch-up.mjs +231 -0
- package/src/cli/coordination/last-seen.mjs +131 -0
- package/src/cli/coordination/store.mjs +18 -0
- package/src/cli/coverage-check/user-outcomes.mjs +52 -17
- package/src/cli/dispatch.mjs +29 -0
- package/src/cli/gate/gates/verify-approved.mjs +22 -81
- package/src/cli/git/default-branch.mjs +250 -0
- package/src/cli/git/identity.mjs +9 -3
- package/src/cli/install/legacy-paths.mjs +2 -0
- package/src/cli/install/mcp-server-specs.mjs +24 -4
- package/src/cli/roadmap/backend-adapter.mjs +231 -0
- package/src/cli/util/runtime-capabilities.mjs +67 -0
- package/src/cli/verify-proof/manifest-validator.mjs +15 -0
- package/src/cli/verify-proof/resolver.mjs +27 -7
- package/src/cli/verify-proof/sandbox/apply.mjs +401 -0
- package/src/cli/verify-proof/sandbox/gate-predicate.mjs +126 -0
- package/src/cli/verify-proof/sandbox/pattern-matcher.mjs +127 -0
- package/src/cli/verify-proof/sandbox/recipe-diff.mjs +208 -0
- package/src/cli/verify-proof/sandbox/sandbox-config.mjs +82 -0
- package/src/cli/verify-proof/sandbox/sandbox-schema.json +33 -0
- package/src/cli/verify-proof/sandbox/self-test.mjs +265 -0
- package/templates/config.json +21 -1
- package/templates/proof-verification.md +27 -8
- package/workflows/verify-proof.md +376 -302
|
@@ -1,10 +1,14 @@
|
|
|
1
1
|
<purpose>
|
|
2
2
|
Visual proof-of-work verification with structured evidence capture. Creates verification.json
|
|
3
|
-
that tracks every
|
|
3
|
+
that tracks every outcome with action → evidence → verdict. Produces HTML proof report.
|
|
4
4
|
|
|
5
5
|
This workflow is adversarial — it assumes things are broken until proven working.
|
|
6
6
|
Every claim requires evidence. Every screenshot follows an interaction.
|
|
7
7
|
|
|
8
|
+
v0.9.0 — **outcome-walker**: iterate `spec.md ## User Outcomes` parent bullets
|
|
9
|
+
one at a time, reload ONLY that outcome card into working memory, drive the
|
|
10
|
+
stated bullet AND each inferred child, fail-loop on red, advance on green.
|
|
11
|
+
|
|
8
12
|
OUTPUT FORMAT: verification.json (NOT VERIFICATION.md). JSON is the source of truth.
|
|
9
13
|
</purpose>
|
|
10
14
|
|
|
@@ -40,7 +44,6 @@ Read spec and plan:
|
|
|
40
44
|
```bash
|
|
41
45
|
cat spec.md 2>/dev/null
|
|
42
46
|
cat implementation_plan.json 2>/dev/null
|
|
43
|
-
cat testable_features.json 2>/dev/null
|
|
44
47
|
```
|
|
45
48
|
|
|
46
49
|
Generate run ID and create proof directory. v0.8.6 — when the spec
|
|
@@ -65,12 +68,41 @@ screenshot for that surface.
|
|
|
65
68
|
</step>
|
|
66
69
|
|
|
67
70
|
<step name="pre_check">
|
|
68
|
-
## 2. Pre-Check + Driver Selection
|
|
71
|
+
## 2. Pre-Check + Driver Selection + Outcome Loader
|
|
69
72
|
|
|
70
73
|
If `implementation_plan.json` exists, verify all subtasks are completed.
|
|
71
74
|
If any are pending: report and exit.
|
|
72
75
|
|
|
73
|
-
### 2a.
|
|
76
|
+
### 2a. Load `## User Outcomes` from spec.md (HARD REQUIREMENT — v0.9.0)
|
|
77
|
+
|
|
78
|
+
`## User Outcomes` is the verify-proof input contract. Parse the
|
|
79
|
+
section via `parseUserOutcomes`:
|
|
80
|
+
|
|
81
|
+
```js
|
|
82
|
+
import { parseUserOutcomes } from '.aperant/deps/node_modules/@aperant/framework/src/cli/coverage-check/user-outcomes.mjs'
|
|
83
|
+
const parsed = parseUserOutcomes(readFileSync('spec.md', 'utf-8'))
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
If `parsed.body_kind !== 'outcomes'` AND `parsed.body_kind !== 'empty-with-note'`,
|
|
87
|
+
**ABORT** with a clear error: `"spec.md missing ## User Outcomes — verify-proof
|
|
88
|
+
requires the outcome contract. Run /apt:plan to author it."`
|
|
89
|
+
|
|
90
|
+
**Do NOT fall back to feature-registry-derived tests.** The legacy
|
|
91
|
+
"discover features from the registry" path is removed (A4 escape hatch).
|
|
92
|
+
The feature registry is informational only — it does NOT seed test
|
|
93
|
+
generation. Every test entry must trace back to a parent or inferred
|
|
94
|
+
outcome from `## User Outcomes`.
|
|
95
|
+
|
|
96
|
+
For each parent outcome, `parsed.outcomes[i]` carries:
|
|
97
|
+
|
|
98
|
+
- `id` — positional ID (`O1`, `O2`, ...)
|
|
99
|
+
- `surface` — closed set `{electron, web, cli, api, electron+web}`
|
|
100
|
+
- `text` — outcome prose (the parent bullet)
|
|
101
|
+
- `inferred` — string[] of indented child bullets (the planning LLM's
|
|
102
|
+
derived sub-checks; empty `[]` when the author wrote none)
|
|
103
|
+
- `requires` — optional dependency on a prior outcome's ID
|
|
104
|
+
|
|
105
|
+
### 2b. Resolve the verify-proof driver
|
|
74
106
|
|
|
75
107
|
The workflow is **runtime-agnostic** — it refers to IDL verbs
|
|
76
108
|
(`driver.click(target)`, `driver.screenshot(path)`), never to concrete
|
|
@@ -95,9 +127,8 @@ If no driver satisfies a REQUIRED capability, the resolver throws
|
|
|
95
127
|
`unsatisfied_required_capability` — fail-closed, NOT a silent
|
|
96
128
|
`blocked_by_platform` skip (per ID-02).
|
|
97
129
|
|
|
98
|
-
**v0.8.6 — per-outcome dispatch.**
|
|
99
|
-
|
|
100
|
-
SURFACE via `resolveDriversForOutcomes()` instead of a single
|
|
130
|
+
**v0.8.6 — per-outcome dispatch.** The workflow runner picks a driver
|
|
131
|
+
PER OUTCOME SURFACE via `resolveDriversForOutcomes()` instead of a single
|
|
101
132
|
per-run driver pick. Each outcome's `[surface]` tag maps to the
|
|
102
133
|
runtime-capability flag a driver must carry; `[electron+web]`
|
|
103
134
|
outcomes dispatch to TWO drivers (one per surface) sharing the same
|
|
@@ -107,7 +138,7 @@ still works for the per-run "what driver would be picked?" preview
|
|
|
107
138
|
the workflow runner imports `resolveDriversForOutcomes` directly
|
|
108
139
|
from the resolver module in Step 7c.
|
|
109
140
|
|
|
110
|
-
###
|
|
141
|
+
### 2c. Probe optional evidence-capture tools
|
|
111
142
|
|
|
112
143
|
```bash
|
|
113
144
|
# CDP video capture (optional)
|
|
@@ -120,223 +151,185 @@ sandbox_profile}` and `meta.fallback_attempts: [{driver_id, reason}]`
|
|
|
120
151
|
(see ST11's schema extension in `templates/proof-verification.md`).
|
|
121
152
|
</step>
|
|
122
153
|
|
|
123
|
-
<step name="
|
|
124
|
-
##
|
|
154
|
+
<step name="load_deferred_tools">
|
|
155
|
+
## 2.5. Load Deferred Driver Tools (Claude Code / OpenCode hosts)
|
|
125
156
|
|
|
126
|
-
|
|
157
|
+
> **Precondition.** When `--parallel` is requested, the sandbox gate at apt-verify-proof/SKILL.md Step 2c runs BEFORE this step. An unverified sandbox exits with `{status:"error", code:"E_SANDBOX_UNVERIFIED", verdict:"needs_human", reason:"sandbox_unverified"}` without loading any MCP driver tools or entering the outcome-walker loop. See `docs/verify-proof-sandbox-patterns.md` for setup.
|
|
127
158
|
|
|
128
|
-
|
|
159
|
+
On Claude Code and OpenCode hosts, the MCP driver tools listed in
|
|
160
|
+
`drivers/<surface>/manifest.json :: transport.mcp_server_id` are
|
|
161
|
+
**deferred** — they exist in the host's MCP server registry but their
|
|
162
|
+
schemas are NOT loaded by default. Before driving ANY UI-surface outcome
|
|
163
|
+
you MUST load them via `ToolSearch(query='select:...')`.
|
|
129
164
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
```
|
|
165
|
+
**Worked example.** For the bundled `browser` driver (whose manifest
|
|
166
|
+
field `transport.mcp_server_id = "puppeteer"` exposes
|
|
167
|
+
`mcp__puppeteer__*` tools):
|
|
134
168
|
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
pnpm typecheck 2>&1 | head -100
|
|
169
|
+
```
|
|
170
|
+
ToolSearch(query='select:mcp__puppeteer__browser_navigate,mcp__puppeteer__browser_click,mcp__puppeteer__browser_screenshot,mcp__puppeteer__browser_type')
|
|
138
171
|
```
|
|
139
172
|
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
173
|
+
For the bundled `electron` driver (whose manifest field
|
|
174
|
+
`transport.mcp_server_id = "electron"` exposes `mcp__electron__*` tools):
|
|
175
|
+
|
|
176
|
+
```
|
|
177
|
+
ToolSearch(query='select:mcp__electron__app_launch,mcp__electron__app_click,mcp__electron__app_screenshot,mcp__electron__app_type')
|
|
143
178
|
```
|
|
144
179
|
|
|
145
|
-
|
|
180
|
+
After the `ToolSearch` call, list the now-loaded tool names back to
|
|
181
|
+
confirm they appear in your tool surface. If any required driver tool
|
|
182
|
+
fails to load, emit `result: "needs_human_with_transport"` on every
|
|
183
|
+
outcome targeting that surface and proceed — do NOT fall back to
|
|
184
|
+
file-evidence (the legacy "commit hash as proof" pattern is rejected
|
|
185
|
+
by the proof_gate).
|
|
186
|
+
|
|
187
|
+
The manifest field is `transport.mcp_server_id` (and
|
|
188
|
+
`transport.skill_id` where applicable). The names
|
|
189
|
+
`mcp_transport.tool_prefix` and `transport.tool_prefix` are NOT real
|
|
190
|
+
manifest fields — do not search for them.
|
|
191
|
+
|
|
192
|
+
This step is a no-op on host CLIs whose entire MCP surface is loaded
|
|
193
|
+
eagerly (e.g. some Codex builds). Run it unconditionally; if the
|
|
194
|
+
selected tools are already in the surface, `ToolSearch` is idempotent.
|
|
146
195
|
</step>
|
|
147
196
|
|
|
148
|
-
<step name="
|
|
149
|
-
##
|
|
197
|
+
<step name="automated_checks">
|
|
198
|
+
## 3. Run Automated Checks (HARD STOP gate — no "enforced at commit" dodge)
|
|
150
199
|
|
|
151
|
-
|
|
200
|
+
Run each of `pnpm lint`, `pnpm test`, `pnpm typecheck` (or the
|
|
201
|
+
configured equivalents) IN THIS RUN and capture the full terminal output.
|
|
152
202
|
|
|
153
|
-
|
|
203
|
+
**Closed set for status.** Each check's `status` MUST be one of
|
|
204
|
+
`{pass, fail}`. The legacy string `"enforced_at_commit"` is no longer
|
|
205
|
+
valid — if you find yourself about to write it, STOP. You must actually
|
|
206
|
+
execute the check in this run.
|
|
154
207
|
|
|
155
208
|
```bash
|
|
156
|
-
|
|
157
|
-
|
|
209
|
+
# Tests
|
|
210
|
+
pnpm test 2>&1 | tee /tmp/proof-tests.log
|
|
158
211
|
```
|
|
159
212
|
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
If no feature registry exists:
|
|
166
|
-
- Suggest: "No feature registry found. Consider running `/apt:scan {area}` first for exhaustive coverage."
|
|
167
|
-
- Proceed with manual discovery below, but warn that coverage may be incomplete
|
|
168
|
-
|
|
169
|
-
**v0.8.6 — read `## User Outcomes` from spec.md FIRST when present.**
|
|
170
|
-
The PRD-shaped spec.md (STANDARD/DEEP tracks) ships a
|
|
171
|
-
`## User Outcomes` section that is the verify-proof input contract:
|
|
172
|
-
`**Epic:**` headline + N atomic outcomes tagged by `[surface]`. When
|
|
173
|
-
present, each outcome maps 1:1 to a test entry — the workflow runner
|
|
174
|
-
does NOT have to invent test units from the feature registry. Fall
|
|
175
|
-
back to the feature-registry-based discovery below ONLY when the spec
|
|
176
|
-
lacks `## User Outcomes` (the v0.8.x warn-not-block migration window
|
|
177
|
-
per spec ID-06; the section becomes a hard block in v0.9.0).
|
|
178
|
-
|
|
179
|
-
### 4b. Read the source code
|
|
180
|
-
|
|
181
|
-
You MUST read the actual source code for the features under test. Do not guess.
|
|
182
|
-
For example, if testing a terminal feature:
|
|
183
|
-
- Read the component files to find every prop, every button, every interaction
|
|
184
|
-
- Read settings/config components to find every slider, dropdown, toggle
|
|
185
|
-
- Read toolbar components to find every button and its handler
|
|
186
|
-
- Read any test files for hints about expected behavior
|
|
213
|
+
```bash
|
|
214
|
+
# Type check
|
|
215
|
+
pnpm typecheck 2>&1 | tee /tmp/proof-typecheck.log
|
|
216
|
+
```
|
|
187
217
|
|
|
188
|
-
|
|
218
|
+
```bash
|
|
219
|
+
# Lint
|
|
220
|
+
pnpm lint 2>&1 | tee /tmp/proof-lint.log
|
|
221
|
+
```
|
|
189
222
|
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
- Every input/slider → what range does it accept? What does it change?
|
|
193
|
-
- Every dropdown → what are the options? What does each option do?
|
|
194
|
-
- Every toggle → what does it enable/disable?
|
|
195
|
-
- Every keyboard shortcut → what does it trigger?
|
|
223
|
+
Record results in `verification.json :: automated_checks` with full
|
|
224
|
+
output. Each entry shape:
|
|
196
225
|
|
|
197
|
-
|
|
226
|
+
```json
|
|
227
|
+
{
|
|
228
|
+
"automated_checks": {
|
|
229
|
+
"tests": { "status": "pass | fail", "output": "...", "count": 42 },
|
|
230
|
+
"typecheck": { "status": "pass | fail", "output": "..." },
|
|
231
|
+
"lint": { "status": "pass | fail", "output": "..." }
|
|
232
|
+
}
|
|
233
|
+
}
|
|
234
|
+
```
|
|
198
235
|
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
236
|
+
**Hard STOP gate.** If any of the three checks has a `status` outside
|
|
237
|
+
`{pass, fail}` (e.g. `enforced_at_commit`, `skipped`, `unknown`), the
|
|
238
|
+
run aborts immediately with `automation_gate: fail` and overall
|
|
239
|
+
verdict `rejected`. The renderer / mapper rejects any input
|
|
240
|
+
verification.json carrying a non-conforming status.
|
|
203
241
|
|
|
204
|
-
|
|
242
|
+
**Verdict roll-up.** `automation_gate = pass` iff ALL THREE of tests,
|
|
243
|
+
typecheck, lint are `pass`. Any single `fail` → `automation_gate: fail`
|
|
244
|
+
→ overall verdict `rejected` regardless of `proof_gate`.
|
|
245
|
+
</step>
|
|
205
246
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
- **observable**: can see in screenshot but cannot interact (e.g., xterm.js internal rendering)
|
|
209
|
-
- **blocked**: cannot test or observe via available tools
|
|
247
|
+
<step name="execute_outcomes">
|
|
248
|
+
## 4. Execute Outcomes (outcome-walker loop)
|
|
210
249
|
|
|
211
|
-
|
|
250
|
+
**This step replaces the legacy "Deep Feature Discovery" + "Generate
|
|
251
|
+
Tests" + "Execute Tests" sequence.** The workflow runner iterates the
|
|
252
|
+
parent outcomes from Step 2a (`parsed.outcomes`) ONE AT A TIME, in
|
|
253
|
+
document order. For each parent outcome, drive the stated bullet AND
|
|
254
|
+
each inferred child, capture surface-appropriate evidence per child,
|
|
255
|
+
fix-loop on red, advance on green.
|
|
212
256
|
|
|
213
|
-
###
|
|
257
|
+
### 4a. Reload ONLY the current outcome card
|
|
214
258
|
|
|
215
|
-
|
|
259
|
+
Before driving outcome `Oi`, reload ONLY that outcome's card into
|
|
260
|
+
working memory:
|
|
216
261
|
|
|
217
262
|
```
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
coverage = tested_count / registered_count
|
|
263
|
+
const outcome = parsed.outcomes[i]
|
|
264
|
+
// outcome = { id, surface, text, inferred[], requires? }
|
|
221
265
|
```
|
|
222
266
|
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
- If `coverage >= 80%`: Proceed normally.
|
|
267
|
+
Do NOT keep the prior outcome's evidence, screenshots, or driver state
|
|
268
|
+
in active memory — this defends against context bloat on long runs
|
|
269
|
+
(30+ outcomes). Persist progress to `verification.json :: tests[]`
|
|
270
|
+
between outcomes; that file is the durable source of truth.
|
|
228
271
|
|
|
229
|
-
|
|
230
|
-
Valid skip reasons:
|
|
231
|
-
- `time_constraint` — would take too long relative to value
|
|
232
|
-
- `unsatisfied_optional_capability` — the selected driver lacks an OPTIONAL IDL verb needed by this test (e.g. `assert_network` not implemented). Loud, deliberate skip.
|
|
233
|
-
- `depends_on_other` — requires a feature not yet implemented
|
|
234
|
-
- `not_applicable` — feature is deprecated or not relevant to this verification
|
|
272
|
+
### 4b. Drive the stated bullet (IDL-verb dispatch, MUST-invoke directives)
|
|
235
273
|
|
|
236
|
-
|
|
274
|
+
Use the per-outcome driver from `resolveDriversForOutcomes()`:
|
|
237
275
|
|
|
238
|
-
|
|
276
|
+
```js
|
|
277
|
+
import { resolveDriversForOutcomes } from '.aperant/deps/node_modules/@aperant/framework/src/cli/verify-proof/resolver.mjs'
|
|
239
278
|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
279
|
+
const entries = resolveDriversForOutcomes({
|
|
280
|
+
drivers, runtime,
|
|
281
|
+
outcomes: parsed.outcomes,
|
|
282
|
+
})
|
|
283
|
+
// → [{outcome_id: "O1", surface: "electron", driver: {...}}, ...]
|
|
284
|
+
// For [electron+web] outcomes, two entries share outcome_id.
|
|
285
|
+
```
|
|
243
286
|
|
|
244
|
-
|
|
245
|
-
## 5. Generate Test Cases
|
|
246
|
-
|
|
247
|
-
For EVERY sub-feature discovered in step 4, generate an **action-based test**.
|
|
248
|
-
|
|
249
|
-
Each test MUST have:
|
|
250
|
-
- **id**: sequential test ID (e.g., "TEST-01")
|
|
251
|
-
- **registry_id**: the sub-feature ID from `.aperant/features/{area}.json` (e.g., "terminals-grid-auto-tile")
|
|
252
|
-
- **name**: descriptive test name
|
|
253
|
-
- **precondition**: what must be true before testing (e.g., "app is running, terminal tab is active")
|
|
254
|
-
- **action**: a specific thing to DO (e.g., "Type 'echo hello' and press Enter")
|
|
255
|
-
- **expected**: a specific observable result (e.g., "Terminal displays 'hello' on the next line")
|
|
256
|
-
|
|
257
|
-
**Rules for writing tests:**
|
|
258
|
-
1. Every test involves an interaction (type, click, submit, navigate) OR a visual quality check
|
|
259
|
-
2. "It renders" is NOT a valid test — "it renders X with properties Y when you do Z" IS
|
|
260
|
-
3. If a feature has a text input, you must type into it
|
|
261
|
-
4. If a feature has a button, you must click it
|
|
262
|
-
5. If a feature has a slider, you must change it and verify the change took effect
|
|
263
|
-
6. If a feature creates something, verify the creation AND its quality
|
|
264
|
-
7. **Visual quality tests are valid** — "terminal shows clean prompt with path and git branch" IS a valid test
|
|
265
|
-
8. **Artifact detection is valid** — "no rendering artifacts, stray characters, or broken formatting" IS a valid test
|
|
266
|
-
9. Test sub-features exhaustively — if settings has 5 sliders, write 5 tests, not 1
|
|
267
|
-
10. **Every test must reference a `registry_id`** from the feature registry. This links tests to sub-features and makes coverage tracking automatic.
|
|
268
|
-
|
|
269
|
-
Write ALL tests into verification.json `tests` array before starting execution.
|
|
270
|
-
|
|
271
|
-
**Gate:** Do not proceed to step 6 until ALL tests are written AND the coverage gate from step 4f passes.
|
|
272
|
-
</step>
|
|
287
|
+
For each `entry` whose `outcome_id` matches `outcome.id`:
|
|
273
288
|
|
|
274
|
-
|
|
275
|
-
|
|
289
|
+
1. Compose an action sequence from `outcome.text` (the parent prose
|
|
290
|
+
describes the user-facing capability — translate into IDL verbs):
|
|
291
|
+
- If it says "type X" → `driver.type({ target, text: "X" })`
|
|
292
|
+
- If it says "click X" → `driver.click({ target })`
|
|
293
|
+
- If it says "navigate to X" → `driver.navigate({ url: "X" })`
|
|
294
|
+
- If it says "press key X" → `driver.key({ key: "X" })`
|
|
295
|
+
2. Invoke IDL verbs on `entry.driver`.
|
|
296
|
+
3. Capture surface-appropriate evidence per Step 4d.
|
|
297
|
+
4. Record the test entry in `verification.json :: tests[]` with
|
|
298
|
+
`outcome_id = outcome.id` and `surface = entry.surface`.
|
|
276
299
|
|
|
277
|
-
|
|
300
|
+
**Transport dispatch directive (MUST honor — TD-08).** When `driver.transport.skill_id` is set AND the host CLI exposes the Skill tool AND the outcome surface is `web`, the workflow runner MUST invoke `Skill({skill: driver.transport.skill_id, args: ...})` (passing the IDL verb args) and MUST record `transport: "skill"` on the resulting test entry in verification.json. When the host CLI does NOT expose the Skill tool, fall back to the MCP transport via the existing puppeteer MCP server and record `transport: "mcp"`. The driver's `chooseTransport({has_skill_tool})` helper computes the choice; the workflow runner is the site that knows `has_skill_tool`. This directive is LLM-honored (the v1 verify-proof runner is the host LLM following this workflow as prose); a text-level grep guard (TD-08) pins the `MUST` language so future edits cannot silently soften it to `SHOULD`.
|
|
278
301
|
|
|
279
|
-
|
|
280
|
-
apt-proof-video capture --port 9222 --duration {estimated_total_seconds} --fps 2 --output ${PROOF_DIR}
|
|
281
|
-
```
|
|
302
|
+
### 4c. Drive each inferred child
|
|
282
303
|
|
|
283
|
-
|
|
284
|
-
the same interactions that produce screenshots. This ensures video and screenshots
|
|
285
|
-
are from the SAME session.
|
|
304
|
+
For each `child` in `outcome.inferred`:
|
|
286
305
|
|
|
287
|
-
|
|
288
|
-
-
|
|
289
|
-
-
|
|
290
|
-
|
|
306
|
+
1. Compose an action sequence from the child prose (these are the
|
|
307
|
+
sub-checks the planning LLM derived from the parent — e.g. "X-button
|
|
308
|
+
closes", "outside-click closes", "Esc closes + returns focus").
|
|
309
|
+
2. Invoke IDL verbs.
|
|
310
|
+
3. Capture evidence per Step 4d.
|
|
311
|
+
4. Record a SEPARATE test entry in `verification.json :: tests[]`
|
|
312
|
+
carrying:
|
|
313
|
+
- `outcome_id` = `outcome.id` (same as parent)
|
|
314
|
+
- `inferred_index` = index of the child in `outcome.inferred[]`
|
|
315
|
+
- `surface` = `entry.surface`
|
|
316
|
+
- the standard `action / expected / actual / evidence / result` fields.
|
|
291
317
|
|
|
292
|
-
|
|
318
|
+
### 4d. Capture surface-appropriate evidence (fail-closed)
|
|
293
319
|
|
|
294
|
-
|
|
295
|
-
</step>
|
|
320
|
+
Surface evidence requirements:
|
|
296
321
|
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
### 7a. Perform the Action
|
|
303
|
-
Actually do what the test says — invoke the IDL verb on the selected driver:
|
|
304
|
-
- If it says "type X" → `driver.type({ target, text: "X" })`
|
|
305
|
-
- If it says "click X" → `driver.click({ target })`
|
|
306
|
-
- If it says "navigate to X" → `driver.navigate({ url: "X" })`
|
|
307
|
-
- If it says "press key X" → `driver.key({ key: "X" })`
|
|
308
|
-
- If it says "run command X" → `driver.execute_js({ script: "X" })`
|
|
309
|
-
|
|
310
|
-
The driver layer translates IDL verbs to its transport (an MCP server,
|
|
311
|
-
a local binary, or a built-in HTTP request). Workflow prose names ONLY
|
|
312
|
-
IDL verbs per ID-01 — concrete tool names live in
|
|
313
|
-
`packages/framework/drivers/<driver>/driver.mjs`.
|
|
314
|
-
|
|
315
|
-
**v0.8.6 — transport dispatch directive (MUST honor).** When
|
|
316
|
-
`driver.transport.skill_id` is set AND the host CLI exposes the
|
|
317
|
-
Skill tool AND the outcome surface is `web`, the workflow runner
|
|
318
|
-
MUST invoke `Skill({skill: driver.transport.skill_id, args: ...})`
|
|
319
|
-
(passing the IDL verb args) and MUST record `transport: "skill"` on
|
|
320
|
-
the resulting test entry in verification.json. When the host CLI
|
|
321
|
-
does NOT expose the Skill tool, fall back to the MCP transport via
|
|
322
|
-
the existing puppeteer MCP server and record `transport: "mcp"`.
|
|
323
|
-
The driver's `chooseTransport({has_skill_tool})` helper computes
|
|
324
|
-
the choice; the workflow runner is the site that knows
|
|
325
|
-
`has_skill_tool`. This directive is LLM-honored (the v1 verify-proof
|
|
326
|
-
runner is the host LLM following this workflow as prose); a
|
|
327
|
-
text-level grep guard (TD-08) pins the `MUST` language so future
|
|
328
|
-
edits cannot silently soften it to `SHOULD`.
|
|
329
|
-
|
|
330
|
-
### 7b. Wait for Result
|
|
331
|
-
Wait for the action to complete. If it involves loading, wait for the load.
|
|
332
|
-
Add a short delay (1-2 seconds) so the CDP capture also captures this state.
|
|
333
|
-
|
|
334
|
-
### 7c. Capture Evidence
|
|
335
|
-
Take a screenshot AFTER the action completes. v0.8.6 — screenshots
|
|
336
|
-
land under PER-SURFACE subdirectories, with the per-outcome driver
|
|
337
|
-
returned by `resolveDriversForOutcomes()`:
|
|
322
|
+
| Surface | Acceptable evidence |
|
|
323
|
+
|---|---|
|
|
324
|
+
| `electron`, `web`, `electron+web` | Screenshot from `driver.screenshot()` under `${PROOF_DIR}/${surface}/screenshots/...` |
|
|
325
|
+
| `cli` | Captured command output (stdout/stderr) under `${PROOF_DIR}/cli/output/...` |
|
|
326
|
+
| `api` | HTTP request/response capture under `${PROOF_DIR}/api/captures/...` |
|
|
338
327
|
|
|
339
|
-
|
|
328
|
+
For UI-surface outcomes, take a screenshot AFTER each action completes.
|
|
329
|
+
Screenshots land under PER-SURFACE subdirectories — the per-outcome
|
|
330
|
+
driver returned by `resolveDriversForOutcomes()` carries the surface:
|
|
331
|
+
|
|
332
|
+
```js
|
|
340
333
|
// Pick a driver per outcome surface (NOT per run). `entries[]` is
|
|
341
334
|
// the resolver's per-(outcome, surface) dispatch — for an
|
|
342
335
|
// [electron+web] outcome you get TWO entries sharing outcome_id
|
|
@@ -351,130 +344,217 @@ for (const entry of entries) {
|
|
|
351
344
|
|
|
352
345
|
Single-surface outcomes (`[electron]`, `[web]`, `[cli]`, `[api]`)
|
|
353
346
|
produce one screenshot under `${PROOF_DIR}/${surface}/screenshots/`;
|
|
354
|
-
cross-surface (`[electron+web]`) produces two — one per surface tree
|
|
355
|
-
|
|
356
|
-
same `
|
|
357
|
-
|
|
358
|
-
### 7d. Analyze Evidence
|
|
359
|
-
**MANDATORY and CRITICAL.** Every screenshot must be analyzed in detail.
|
|
360
|
-
|
|
361
|
-
You MUST describe what you see — not what you expect to see. Be adversarial:
|
|
362
|
-
- Are there rendering artifacts? Stray characters? Broken formatting?
|
|
363
|
-
- Does the UI look production-quality or does it look broken?
|
|
364
|
-
- Compare against what a real user would expect to see
|
|
365
|
-
- If testing a terminal: is the prompt clean? Is the path correct? Are there garbage characters?
|
|
366
|
-
- If testing settings: did the value actually change? Is the control responsive?
|
|
347
|
+
cross-surface (`[electron+web]`) produces two — one per surface tree —
|
|
348
|
+
with both test entries in `verification.json :: tests[]` carrying the
|
|
349
|
+
same `outcome_id` but distinct `surface` and `evidence` fields.
|
|
367
350
|
|
|
368
|
-
|
|
369
|
-
"The terminal shows a cursor" is NOT analysis.
|
|
370
|
-
"The terminal shows a blinking cursor at position 0,0 with no shell prompt, path, or git branch — this indicates the PTY session is not connected" IS analysis.
|
|
371
|
-
|
|
372
|
-
If Gemini API is available, also run:
|
|
373
|
-
```bash
|
|
374
|
-
apt-proof-video analyze "${PROOF_DIR}/screenshots/{feature-id}-test-{n}.png" \
|
|
375
|
-
--model default \
|
|
376
|
-
--spec "{expected result from this test}"
|
|
377
|
-
```
|
|
378
|
-
|
|
379
|
-
### 7e. Record Result to verification.json
|
|
380
|
-
After each test, update the test entry in verification.json `tests` array.
|
|
381
|
-
v0.8.6 adds `surface` and `transport` fields per test entry — the workflow
|
|
382
|
-
runner populates them from the per-outcome dispatch directive:
|
|
351
|
+
Example test entry shape (electron surface, MCP transport):
|
|
383
352
|
|
|
384
353
|
```json
|
|
385
354
|
{
|
|
386
355
|
"id": "TEST-01",
|
|
387
|
-
"registry_id": "terminals-grid-auto-tile",
|
|
388
|
-
"name": "Three Terminal Grid Auto-Tile",
|
|
389
356
|
"outcome_id": "O3",
|
|
390
357
|
"surface": "electron",
|
|
391
358
|
"transport": "mcp",
|
|
392
|
-
"precondition": "2 terminals active",
|
|
393
359
|
"action": "Press Cmd+T",
|
|
394
360
|
"expected": "Grid reflows to 2+1 layout",
|
|
395
|
-
"
|
|
396
|
-
"
|
|
397
|
-
"analysis": "Detailed adversarial analysis of what is actually visible in the screenshot...",
|
|
398
|
-
"result": "pass",
|
|
399
|
-
"severity": null
|
|
361
|
+
"evidence": "electron/screenshots/O3.png",
|
|
362
|
+
"result": "pass"
|
|
400
363
|
}
|
|
401
364
|
```
|
|
402
365
|
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
366
|
+
**`file-evidence: <commit-hash>` is NEVER sufficient.** The proof_gate
|
|
367
|
+
rejects that pattern for ALL surfaces — UI surfaces require screenshots,
|
|
368
|
+
CLI requires actual command output, API requires actual HTTP traces.
|
|
369
|
+
A commit hash is not proof a feature works.
|
|
370
|
+
|
|
371
|
+
**If a UI-surface outcome cannot capture screenshot evidence because
|
|
372
|
+
the host lacks the MCP driver transport (deferred tools didn't load,
|
|
373
|
+
puppeteer/electron-mcp-server unavailable):** write
|
|
374
|
+
`result: "needs_human_with_transport"` on that test entry, include
|
|
375
|
+
a `to_unblock` field explaining what the human needs to install /
|
|
376
|
+
load, and proceed. Do NOT fall back to file-evidence.
|
|
377
|
+
|
|
378
|
+
### 4e. Analyze evidence (adversarial)
|
|
379
|
+
|
|
380
|
+
For each evidence artifact, write the `analysis` field with what you
|
|
381
|
+
**see**, not what you **expect to see**. Be adversarial:
|
|
382
|
+
|
|
383
|
+
- UI: Are there rendering artifacts? Stray characters? Broken layout?
|
|
384
|
+
Does the actual screenshot match the outcome's stated expectation?
|
|
385
|
+
- CLI: Did the command actually emit the expected output, or just exit 0?
|
|
386
|
+
- API: Did the response payload match the expected shape, or just 200?
|
|
387
|
+
|
|
388
|
+
"The terminal shows a cursor" is NOT analysis. "The terminal shows a
|
|
389
|
+
blinking cursor at position 0,0 with no shell prompt — the PTY session
|
|
390
|
+
is not connected" IS analysis.
|
|
391
|
+
|
|
392
|
+
### 4f. Fail-loop on red, advance on green
|
|
393
|
+
|
|
394
|
+
After processing the parent + all inferred children for outcome `Oi`:
|
|
395
|
+
|
|
396
|
+
- If ALL test entries for `Oi` have `result ∈ {pass}` → advance to `Oi+1`.
|
|
397
|
+
- If ANY test entry has `result: "fail"` → fix the code, RE-RUN ONLY
|
|
398
|
+
THE FAILING ENTRIES for `Oi`, capture fresh evidence, write back to
|
|
399
|
+
`verification.json`. Loop up to `verification.max_iterations` times
|
|
400
|
+
(default 3 from config). After the iteration cap, leave `Oi` with
|
|
401
|
+
`result: "fail"` on the unresolved entries and continue — overall
|
|
402
|
+
verdict will be `rejected`.
|
|
403
|
+
- If ANY test entry has `result: "needs_human_with_transport"` →
|
|
404
|
+
advance to `Oi+1` (the human will install the transport and re-run).
|
|
405
|
+
Overall verdict will be `needs_human`.
|
|
406
|
+
|
|
407
|
+
### 4g. Write verification.json after each outcome
|
|
408
|
+
|
|
409
|
+
Persist `verification.json :: tests[]` to disk after EVERY outcome
|
|
410
|
+
completes (success, fail, or needs_human_with_transport). This ensures
|
|
411
|
+
progress survives session interruption — `parsed.outcomes[i].id`
|
|
412
|
+
becomes the resume point.
|
|
413
|
+
|
|
414
|
+
### 4h. Skip-reason taxonomy (legacy compatibility)
|
|
415
|
+
|
|
416
|
+
When an outcome cannot be driven because the selected driver lacks an
|
|
417
|
+
**OPTIONAL** IDL verb (e.g. `assert_network` not implemented), record
|
|
418
|
+
the test entry with `result: "skip"` and `reason: "unsatisfied_optional_capability"`.
|
|
419
|
+
This is the deliberate, loud skip path — log it in `skipped[]` so
|
|
420
|
+
coverage loss is auditable.
|
|
421
|
+
|
|
422
|
+
> **Legacy alias.** `blocked_by_platform` (pre-driver-architecture) is
|
|
423
|
+
> read by the schema reader as an alias for `unsatisfied_optional_capability`
|
|
424
|
+
> and emits a deprecation warning. New writes MUST use the new reason.
|
|
425
|
+
|
|
426
|
+
> **Fail-closed, not a skip:** if a REQUIRED IDL capability has no
|
|
427
|
+
> implementing driver in the resolver chain, the resolver throws
|
|
428
|
+
> `unsatisfied_required_capability` (per ID-02). This aborts the run —
|
|
429
|
+
> it does NOT silently land in `skipped[]`.
|
|
430
|
+
|
|
431
|
+
**Gate:** Do not proceed to Step 5 until every parent outcome has been
|
|
432
|
+
processed (success, fail, or needs_human_with_transport) AND
|
|
433
|
+
`verification.json :: tests[]` carries entries for every parent +
|
|
434
|
+
inferred-child combination.
|
|
435
|
+
</step>
|
|
408
436
|
|
|
409
|
-
|
|
410
|
-
|
|
437
|
+
<step name="start_video">
|
|
438
|
+
## 5. Start Video Capture (optional)
|
|
411
439
|
|
|
412
|
-
|
|
413
|
-
Write the updated verification.json to disk after each test completes.
|
|
414
|
-
This ensures progress is saved even if the session is interrupted.
|
|
440
|
+
If CDP capture is available, start recording BEFORE executing outcomes:
|
|
415
441
|
|
|
416
|
-
|
|
442
|
+
```bash
|
|
443
|
+
apt-proof-video capture --port 9222 --duration {estimated_total_seconds} --fps 2 --output ${PROOF_DIR}
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
Run this in the background. The video will capture your ACTUAL outcome
|
|
447
|
+
walker session — the same interactions that produce screenshots. This
|
|
448
|
+
ensures video and screenshots are from the SAME session.
|
|
449
|
+
|
|
450
|
+
**Important:** Estimate duration generously. Better to have extra video
|
|
451
|
+
than to miss outcomes.
|
|
452
|
+
- ~15 seconds per outcome (parent + inferred children) is a reasonable estimate
|
|
453
|
+
- Add 30 seconds for navigation overhead
|
|
454
|
+
|
|
455
|
+
**Note:** Captured frames are auto-stitched into an MP4 file by the tool.
|
|
456
|
+
|
|
457
|
+
If CDP is NOT available, note it and proceed without video.
|
|
458
|
+
|
|
459
|
+
(In practice this step is interleaved with Step 4 — kick off the
|
|
460
|
+
capture before the first outcome and stop it after the last.)
|
|
417
461
|
</step>
|
|
418
462
|
|
|
419
463
|
<step name="score_dimensions">
|
|
420
|
-
##
|
|
464
|
+
## 6. Score QA Dimensions (informational rationale)
|
|
421
465
|
|
|
422
466
|
Based on the evidence collected (not assumed), score each dimension.
|
|
423
|
-
Write scores to verification.json `dimensions` object
|
|
467
|
+
Write scores to verification.json `dimensions` object.
|
|
468
|
+
|
|
469
|
+
**v0.9.0 — dimensions are INFORMATIONAL rationale only.** The verdict
|
|
470
|
+
no longer derives from per-dimension minimums; it derives from the
|
|
471
|
+
two-gate envelope (Step 7).
|
|
424
472
|
|
|
425
473
|
```json
|
|
426
474
|
{
|
|
427
475
|
"dimensions": {
|
|
428
|
-
"completeness": {
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
}
|
|
432
|
-
"correctness": {
|
|
433
|
-
"score": 8,
|
|
434
|
-
"rationale": "All tested features behave as expected. No blockers or majors."
|
|
435
|
-
},
|
|
436
|
-
"code_quality": {
|
|
437
|
-
"score": 9,
|
|
438
|
-
"rationale": "Zero lint errors, zero type errors."
|
|
439
|
-
},
|
|
440
|
-
"test_coverage": {
|
|
441
|
-
"score": 6,
|
|
442
|
-
"rationale": "Unit tests cover core logic but no integration tests for new grid layout."
|
|
443
|
-
}
|
|
476
|
+
"completeness": { "score": 7, "rationale": "All 12 outcomes processed; 2 needed transport, 10 driven." },
|
|
477
|
+
"correctness": { "score": 8, "rationale": "All driven outcomes matched expected; no regressions." },
|
|
478
|
+
"code_quality": { "score": 9, "rationale": "Zero lint errors, zero type errors." },
|
|
479
|
+
"test_coverage":{ "score": 6, "rationale": "Unit tests cover core logic; integration tests for the new outcome walker live in the framework package." }
|
|
444
480
|
}
|
|
445
481
|
}
|
|
446
482
|
```
|
|
447
483
|
|
|
448
|
-
**Scoring rules
|
|
484
|
+
**Scoring rules** (all 1-10):
|
|
449
485
|
|
|
450
|
-
**Completeness (
|
|
451
|
-
|
|
452
|
-
-
|
|
453
|
-
|
|
454
|
-
-
|
|
486
|
+
- **Completeness**: count outcomes processed vs. authored (`tests` per
|
|
487
|
+
parent + inferred / total expected).
|
|
488
|
+
- **Correctness**: per-outcome verdict roll-up — any blocker = max 3,
|
|
489
|
+
any major = max 5.
|
|
490
|
+
- **Code Quality**: lint + typecheck results.
|
|
491
|
+
- **Test Coverage**: unit + integration tests landed for the feature.
|
|
455
492
|
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
493
|
+
Dimensions surface in the HTML report as rationale chips alongside the
|
|
494
|
+
two gate badges. They do NOT change the verdict.
|
|
495
|
+
</step>
|
|
496
|
+
|
|
497
|
+
<step name="derive_verdict">
|
|
498
|
+
## 7. Derive Two-Gate Verdict
|
|
499
|
+
|
|
500
|
+
The verdict is the closed set `{approved, rejected, needs_human}`.
|
|
501
|
+
**`approved_with_notes` is DELETED — do not emit it.**
|
|
502
|
+
|
|
503
|
+
### 7a. Compute `automation_gate`
|
|
504
|
+
|
|
505
|
+
```
|
|
506
|
+
automation_gate = (
|
|
507
|
+
automated_checks.tests.status === 'pass' AND
|
|
508
|
+
automated_checks.typecheck.status === 'pass' AND
|
|
509
|
+
automated_checks.lint.status === 'pass'
|
|
510
|
+
) ? 'pass' : 'fail'
|
|
511
|
+
```
|
|
512
|
+
|
|
513
|
+
### 7b. Compute `proof_gate`
|
|
514
|
+
|
|
515
|
+
For each test entry in `verification.json :: tests[]`:
|
|
461
516
|
|
|
462
|
-
|
|
463
|
-
|
|
517
|
+
- UI-surface (`electron`, `web`, `electron+web`) with
|
|
518
|
+
`evidence` matching `^file-evidence:` → that test's effective result
|
|
519
|
+
is `needs_human_with_transport` regardless of the written `result`.
|
|
520
|
+
- Any test whose effective result is `fail` → contributes to
|
|
521
|
+
`proof_gate: fail`.
|
|
522
|
+
- Any test whose effective result is `needs_human_with_transport` →
|
|
523
|
+
contributes to overall verdict `needs_human` (does NOT fail `proof_gate`
|
|
524
|
+
on its own — the gate "passes" with the caveat that a human needs to
|
|
525
|
+
install the transport).
|
|
464
526
|
|
|
465
|
-
|
|
466
|
-
|
|
527
|
+
```
|
|
528
|
+
proof_gate = (
|
|
529
|
+
no test has effective result 'fail' AND
|
|
530
|
+
every outcome has at least one evidence entry per surface
|
|
531
|
+
) ? 'pass' : 'fail'
|
|
532
|
+
```
|
|
467
533
|
|
|
468
|
-
|
|
469
|
-
- ALL dimensions >= 7: `"approved"`
|
|
470
|
-
- ALL dimensions >= 5: `"approved_with_notes"`
|
|
471
|
-
- ANY dimension < 5: `"rejected"`
|
|
534
|
+
### 7c. Derive overall verdict
|
|
472
535
|
|
|
473
|
-
|
|
536
|
+
```
|
|
537
|
+
if (automation_gate === 'fail') return 'rejected'
|
|
538
|
+
if (any test has effective result 'needs_human_with_transport') return 'needs_human'
|
|
539
|
+
if (proof_gate === 'fail') return 'rejected'
|
|
540
|
+
return 'approved'
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
Write `verification.json :: meta.verdict` and `verification.json :: gates`:
|
|
544
|
+
|
|
545
|
+
```json
|
|
546
|
+
{
|
|
547
|
+
"meta": { "verdict": "approved | rejected | needs_human" },
|
|
548
|
+
"gates": {
|
|
549
|
+
"automation_gate": "pass | fail",
|
|
550
|
+
"proof_gate": "pass | fail"
|
|
551
|
+
}
|
|
552
|
+
}
|
|
553
|
+
```
|
|
474
554
|
</step>
|
|
475
555
|
|
|
476
556
|
<step name="multi_model" optional="true">
|
|
477
|
-
##
|
|
557
|
+
## 8. Multi-Model Verification (if enabled in config)
|
|
478
558
|
|
|
479
559
|
Only run if `.aperant/config.json` has `multi_model.enabled: true`.
|
|
480
560
|
|
|
@@ -485,7 +565,7 @@ Record each model's verdict in verification.json `model_verdicts` array.
|
|
|
485
565
|
</step>
|
|
486
566
|
|
|
487
567
|
<step name="generate_report">
|
|
488
|
-
##
|
|
568
|
+
## 9. Generate HTML Proof Report
|
|
489
569
|
|
|
490
570
|
Use the `apt-proof-video report` command to generate the HTML report from verification.json:
|
|
491
571
|
|
|
@@ -496,10 +576,12 @@ apt-proof-video report --proof-dir "${PROOF_DIR}" --title "{descriptive title}"
|
|
|
496
576
|
The report command:
|
|
497
577
|
1. Reads verification.json as the primary data source
|
|
498
578
|
2. Recursively scans for screenshots in subdirectories
|
|
499
|
-
3. Generates
|
|
500
|
-
|
|
579
|
+
3. Generates one card per `outcome_id` with stated prose + inferred
|
|
580
|
+
children + per-child evidence
|
|
581
|
+
4. Renders two gate badges (`automation_gate`, `proof_gate`) plus
|
|
582
|
+
the overall verdict badge in the header
|
|
501
583
|
5. Shows automated checks with expandable output
|
|
502
|
-
6. Shows QA dimension
|
|
584
|
+
6. Shows QA dimension rationale chips
|
|
503
585
|
7. Shows gaps with severity badges
|
|
504
586
|
8. Embeds video player (frames are auto-stitched to MP4)
|
|
505
587
|
|
|
@@ -510,7 +592,7 @@ ln -sf "${PROOF_DIR}/report.html" .aperant/proof-report.html
|
|
|
510
592
|
</step>
|
|
511
593
|
|
|
512
594
|
<step name="open_report">
|
|
513
|
-
##
|
|
595
|
+
## 10. Open Report
|
|
514
596
|
|
|
515
597
|
Read `verification.open_report` from the MERGED config (per-device override lives in `.aperant/config.local.json` per AUDIT-001; do not read raw `.aperant/config.json`). If true (default):
|
|
516
598
|
```bash
|
|
@@ -519,7 +601,7 @@ open "${PROOF_DIR}/report.html"
|
|
|
519
601
|
</step>
|
|
520
602
|
|
|
521
603
|
<step name="write_artifacts">
|
|
522
|
-
##
|
|
604
|
+
## 11. Write Artifacts
|
|
523
605
|
|
|
524
606
|
Write the following files:
|
|
525
607
|
- `${PROOF_DIR}/verification.json` — the structured verification record (source of truth)
|
|
@@ -536,40 +618,35 @@ apt-tools commit "qa: proof verification {verdict}" --files ${PROOF_DIR}/verific
|
|
|
536
618
|
</step>
|
|
537
619
|
|
|
538
620
|
<step name="report_output">
|
|
539
|
-
##
|
|
621
|
+
## 12. Final Output
|
|
540
622
|
|
|
541
623
|
```
|
|
542
624
|
Proof Verification: {VERDICT}
|
|
543
625
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
544
626
|
Outcome: {one-line summary}
|
|
545
627
|
|
|
628
|
+
Gates:
|
|
629
|
+
automation_gate: {pass/fail}
|
|
630
|
+
proof_gate: {pass/fail}
|
|
631
|
+
|
|
546
632
|
Checks:
|
|
547
633
|
Tests: {pass/fail} ({count} tests)
|
|
548
634
|
Types: {pass/fail}
|
|
549
635
|
Lint: {pass/fail}
|
|
550
636
|
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
Coverage:
|
|
558
|
-
Registry: {tested}/{total} features ({percent}%)
|
|
559
|
-
Skipped: {n} features (see skipped array)
|
|
560
|
-
|
|
561
|
-
Evidence:
|
|
562
|
-
Features: {n} tested ({passed} pass, {failed} fail, {blocked} blocked)
|
|
563
|
-
Screenshots: {n} captured, {n} analyzed
|
|
564
|
-
Video: {captured/not available}
|
|
565
|
-
Gaps: {n} issues ({blockers} blockers, {majors} major)
|
|
637
|
+
Outcomes:
|
|
638
|
+
Total: {n} parent outcomes (+ {m} inferred children)
|
|
639
|
+
Approved: {n}
|
|
640
|
+
Rejected: {n}
|
|
641
|
+
Needs Human: {n} (transport unavailable)
|
|
566
642
|
|
|
567
643
|
Proof: {PROOF_DIR}/
|
|
568
|
-
verification.json (structured
|
|
644
|
+
verification.json (structured outcome record)
|
|
569
645
|
report.html (opened in browser)
|
|
570
646
|
report.md (for PR embedding)
|
|
571
647
|
|
|
572
648
|
{If rejected:} Fix gaps and re-run /apt:verify-proof
|
|
649
|
+
{If needs_human:} Install the missing driver transports and re-run
|
|
573
650
|
{If approved:} Next: /apt:review (optional code review)
|
|
574
651
|
```
|
|
575
652
|
</step>
|
|
@@ -581,17 +658,14 @@ Proof: {PROOF_DIR}/
|
|
|
581
658
|
|
|
582
659
|
These gates MUST be respected — do not skip them:
|
|
583
660
|
|
|
584
|
-
1. **Pre-check gate** (
|
|
585
|
-
2. **
|
|
586
|
-
3. **
|
|
587
|
-
4. **
|
|
588
|
-
5. **
|
|
589
|
-
6. **
|
|
590
|
-
7. **
|
|
591
|
-
8. **
|
|
592
|
-
9. **Quality gate** (step 7d): Visual artifacts, broken rendering, and quality issues MUST be reported as failures
|
|
593
|
-
10. **Scoring gate** (step 8): Scores must be derived from evidence, not assumed
|
|
594
|
-
11. **Output format gate**: Write verification.json, NEVER VERIFICATION.md
|
|
661
|
+
1. **Pre-check gate** (Step 2): All subtasks must be completed before verification starts.
|
|
662
|
+
2. **User Outcomes gate** (Step 2a): spec.md MUST carry `## User Outcomes` with `body_kind ∈ {outcomes, empty-with-note}`. No fallback to feature-registry-derived tests.
|
|
663
|
+
3. **Deferred-tool-load gate** (Step 2.5): On Claude Code / OpenCode hosts, MCP driver tools listed under `drivers/<surface>/manifest.json :: transport.mcp_server_id` MUST be loaded via `ToolSearch(query='select:...')` BEFORE driving any UI-surface outcome.
|
|
664
|
+
4. **Automation gate** (Step 3): `automated_checks.{tests,typecheck,lint}.status ∈ {pass, fail}` (closed set). No `"enforced_at_commit"`. Any single `fail` → `automation_gate: fail`.
|
|
665
|
+
5. **Outcome-walker gate** (Step 4): every parent outcome AND every inferred child has its own test entry in `verification.json :: tests[]` with evidence appropriate to its surface OR `result: "needs_human_with_transport"`. File-evidence (`"file-evidence: ..."`) is NEVER sufficient for any surface.
|
|
666
|
+
6. **Proof gate** (Step 7b): every outcome has surface-appropriate evidence. UI outcomes carrying `evidence: "file-evidence: ..."` are rejected as `needs_human_with_transport`.
|
|
667
|
+
7. **Verdict gate** (Step 7c): the verdict is exactly one of `{approved, rejected, needs_human}`. The legacy `approved_with_notes` middle band is DELETED — emitting it is a hard error.
|
|
668
|
+
8. **Output format gate** (Step 11): Write verification.json, NEVER VERIFICATION.md.
|
|
595
669
|
</gates>
|
|
596
670
|
</process>
|
|
597
671
|
</output>
|