proof-harness 0.1.0-next.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/COMPATIBILITY.md +26 -0
- package/LICENSE +201 -0
- package/PROOF_SDK_CONTRACT.md +1255 -0
- package/README.md +60 -0
- package/cli/config.mjs +116 -0
- package/cli/engines/generate_proof_action_registry.mjs +271 -0
- package/cli/engines/parse_proof_schema.mjs +457 -0
- package/cli/engines/proof_coverage.mjs +871 -0
- package/cli/engines/proof_drift.mjs +980 -0
- package/cli/engines/proof_module_check.mjs +498 -0
- package/cli/engines/proof_mutation_check.mjs +1084 -0
- package/cli/engines/proof_mutation_inventory.mjs +508 -0
- package/cli/engines/proof_source_mutation.mjs +35 -0
- package/cli/engines/proof_verify.mjs +620 -0
- package/cli/engines/scan_module_meta.mjs +352 -0
- package/cli/engines/scan_proof_capabilities.mjs +359 -0
- package/cli/proof-harness.mjs +107 -0
- package/dist/action-types-C3OmHRyH.d.ts +26 -0
- package/dist/fixture-B-dmnG5G.d.ts +105 -0
- package/dist/mission-types-Dr5rBHYO.d.ts +299 -0
- package/dist/node.d.ts +79 -0
- package/dist/node.js +553 -0
- package/dist/node.js.map +1 -0
- package/dist/playwright.d.ts +433 -0
- package/dist/playwright.js +1824 -0
- package/dist/playwright.js.map +1 -0
- package/dist/portable-vocabulary.d.ts +86 -0
- package/dist/portable-vocabulary.js +43 -0
- package/dist/portable-vocabulary.js.map +1 -0
- package/dist/server.d.ts +55 -0
- package/dist/server.js +376 -0
- package/dist/server.js.map +1 -0
- package/dist/shared.d.ts +169 -0
- package/dist/shared.js +121 -0
- package/dist/shared.js.map +1 -0
- package/dist/trace-types-DkIzq-l9.d.ts +333 -0
- package/package.json +109 -0
- package/proof-consumer-fixtures/README.md +60 -0
- package/proof-consumer-fixtures/aggregates/M-CONFORMANCE-FAIL.json +112 -0
- package/proof-consumer-fixtures/aggregates/M-CONFORMANCE-PASS.json +101 -0
- package/proof-consumer-fixtures/baseline/failing/widgets-tenant-isolation.json +46 -0
- package/proof-consumer-fixtures/baseline/passing/widgets-tenant-isolation.json +45 -0
- package/proof-consumer-fixtures/corpus.json +102 -0
- package/proof-consumer-fixtures/duplicates/first/widgets-tenant-isolation.json +12 -0
- package/proof-consumer-fixtures/duplicates/second/widgets-tenant-isolation.json +12 -0
- package/proof-consumer-fixtures/inputs/capabilities.json +4 -0
- package/proof-consumer-fixtures/inputs/mission.json +20 -0
- package/proof-consumer-fixtures/inputs/schema.json +4 -0
- package/proof-consumer-fixtures/malformed/invalid-trace.json.invalid +4 -0
- package/proof-consumer-fixtures/malformed/wrong-manifest-version.json +11 -0
- package/proof-consumer-fixtures/malformed/wrong-trace-shape.json +8 -0
- package/proof-consumer-fixtures/mutations/MUT-WIDGETS-LEAK/mutation.json +17 -0
- package/proof-consumer-fixtures/mutations/MUT-WIDGETS-LEAK/widgets-tenant-isolation.json +50 -0
|
@@ -0,0 +1,1255 @@
|
|
|
1
|
+
# Proof Harness — Consumer Integration Contract
|
|
2
|
+
|
|
3
|
+
This document describes what an external consumer (an orchestrator, bespoke
|
|
4
|
+
automation, or a human developer in manual mode) must do to drive a product
|
|
5
|
+
repository through mission-based verification.
|
|
6
|
+
|
|
7
|
+
If you are building the template, see [`.cursor/rules/proof-authoring.mdc`](.cursor/rules/proof-authoring.mdc)
|
|
8
|
+
instead; this document is for anyone _consuming_ the template.
|
|
9
|
+
|
|
10
|
+
> **"Manifest" in this document always means the _mission_ manifest**
|
|
11
|
+
> (`.proof/current-mission.json`) — the file that pins what one PR must prove.
|
|
12
|
+
> It is unrelated to a **module descriptor** (`module.meta.ts` →
|
|
13
|
+
> `.proof/modules.json`), which describes what a module is for and what a buyer
|
|
14
|
+
> must decide before using it. See §3.1 below.
|
|
15
|
+
|
|
16
|
+
- **Version:** 1 (`schemaVersion` in every manifest and artifact)
|
|
17
|
+
- **Stability:** The shapes in this document are the contract. Additive
|
|
18
|
+
changes (new optional fields, new vocabulary entries) do not bump the
|
|
19
|
+
version; breaking changes bump `schemaVersion` and are documented here.
|
|
20
|
+
- **Last reconciled:** August 2026. The migration aggregate and the
|
|
21
|
+
`capabilities.json` / `schema.json` artifacts are **derived build artifacts,
|
|
22
|
+
regenerated from source in setup and CI, and are not committed** (see §2/§3).
|
|
23
|
+
Earlier revisions of this doc described them as committed and regenerated by a
|
|
24
|
+
pre-commit hook; that is no longer true.
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## 1. The loop
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
┌──────────────────────────────────────────────────────────────────────┐
|
|
32
|
+
│ │
|
|
33
|
+
│ 1. Consumer writes .proof/current-mission.json (manifest) │
|
|
34
|
+
│ 2. Coding agent implements feature + proof specs + migrations │
|
|
35
|
+
│ (migrations live in app/**/src/db/migrations/*.sql — source) │
|
|
36
|
+
│ 3. Consumer opens PR │
|
|
37
|
+
│ 4. GHA Proof job runs: │
|
|
38
|
+
│ ├─ pnpm aggregate-migrations --fresh │
|
|
39
|
+
│ │ (rebuilds the gitignored supabase/migrations aggregate) │
|
|
40
|
+
│ ├─ supabase start (local stack + migrations + seed users) │
|
|
41
|
+
│ ├─ playwright install │
|
|
42
|
+
│ └─ pnpm proof:check │
|
|
43
|
+
│ ├─ proof:migrations → diffs module SQL vs the PR base; │
|
|
44
|
+
│ │ shipped files are immutable, new files must not │
|
|
45
|
+
│ │ quietly loosen constraints │
|
|
46
|
+
│ ├─ proof:build → regenerates capabilities.json + schema │
|
|
47
|
+
│ │ .json and verifies action registry │
|
|
48
|
+
│ ├─ proof:drift → diffs the proof surfaces vs the PR │
|
|
49
|
+
│ │ base right after proof:build, so undeclared scope │
|
|
50
|
+
│ │ fails cheaply BEFORE the database proofs run │
|
|
51
|
+
│ ├─ proof:verify │
|
|
52
|
+
│ ├─ runs every *.proof.ts spec │
|
|
53
|
+
│ ├─ aggregates .proof/traces/<missionId>.json │
|
|
54
|
+
│ └─ validates manifest requirements against artifacts │
|
|
55
|
+
│ ├─ proof:coverage --strict │
|
|
56
|
+
│ └─ proof:inventory → requires mutation or acceptance │
|
|
57
|
+
│ then uploads .proof/traces/ as proof-traces and │
|
|
58
|
+
│ .proof/drift.json as proof-drift artifacts │
|
|
59
|
+
│ 5. On pass: PR mergeable; trace artifact is the durable evidence │
|
|
60
|
+
│ On fail: CI log contains [PROOF_FAIL] lines; agent iterates │
|
|
61
|
+
│ 6. Consumer downloads proof-traces artifact via gh run download│
|
|
62
|
+
│ and closes its ticket. │
|
|
63
|
+
│ │
|
|
64
|
+
└──────────────────────────────────────────────────────────────────────┘
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
> Runtime artifacts are not committed: the migration aggregate
|
|
68
|
+
> (`supabase/migrations/…_initial.sql`), `.proof/capabilities.json`, and
|
|
69
|
+
> `.proof/schema.json` are rebuilt on every setup and CI run. The generated
|
|
70
|
+
> TypeScript action registry is the deliberate exception: it is committed
|
|
71
|
+
> because ordinary fresh-checkout `build` and `typecheck` jobs import it without
|
|
72
|
+
> running `proof:build`. Run `pnpm proof:registry` after changing action
|
|
73
|
+
> requirements or probes; `proof:build` fails if the committed result is stale.
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## 2. Inputs the consumer owns
|
|
78
|
+
|
|
79
|
+
Everything the consumer writes into the product repo.
|
|
80
|
+
|
|
81
|
+
| Input | Location | Purpose |
|
|
82
|
+
| ---------------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
83
|
+
| Mission manifest | `.proof/current-mission.json` | Pins what this PR must prove (capabilities, schema, traces). |
|
|
84
|
+
| Feature code | `app/__business-logic/<module>/` | The implementation. Template conventions in `AGENTS.md`. |
|
|
85
|
+
| Proof specs | `e2e/proofs/<feature>.proof.ts` | Playwright tests that emit structured traces. |
|
|
86
|
+
| Migrations | `app/__business-logic/<module>/src/db/migrations/*.sql` | Schema + RLS policies. The **tracked source of truth**; the `supabase/migrations/` aggregate is rebuilt from these by `pnpm aggregate-migrations --fresh` and is gitignored. Files already on the PR base are immutable; `pnpm proof:migrations` rejects edits/deletes and constraint-loosening unless accepted in `.proof/migration-policy.json`. |
|
|
87
|
+
|
|
88
|
+
> **The template ships without `.proof/current-mission.json`.** That is
|
|
89
|
+
> deliberate: with no manifest present, `pnpm proof:verify` runs in
|
|
90
|
+
> no-manifest mode (see §7) and never gates on requirements the consumer
|
|
91
|
+
> hasn't written yet. The consumer writes this file per PR. A reference
|
|
92
|
+
> manifest is included at [`.proof/missions/M-TEST-001.json`](.proof/missions/M-TEST-001.json)
|
|
93
|
+
> as a copy-ready starting point.
|
|
94
|
+
>
|
|
95
|
+
> The opt-in [autonomous factory profile](AUTONOMOUS_FACTORY.md) is stricter:
|
|
96
|
+
> its external planner or human administrator establishes the active mission on
|
|
97
|
+
> the protected base before the executor branch starts. The required Keystone
|
|
98
|
+
> check then rejects executor changes to that mission, proof specs, Proof SDK,
|
|
99
|
+
> policies, workflows, and seed data. This profile is disabled by default.
|
|
100
|
+
|
|
101
|
+
### Mission manifest shape
|
|
102
|
+
|
|
103
|
+
See [`src/shared/mission-types.ts`](src/shared/mission-types.ts)
|
|
104
|
+
for the canonical TypeScript types and [`.proof/current-mission.example.json`](.proof/current-mission.example.json)
|
|
105
|
+
for a worked example. The full schema:
|
|
106
|
+
|
|
107
|
+
```jsonc
|
|
108
|
+
{
|
|
109
|
+
"schemaVersion": 1,
|
|
110
|
+
"missionId": "M-042", // stable ticket id; also the filename of the merged trace
|
|
111
|
+
"missionTitle": "Add issue tracking with assignees",
|
|
112
|
+
"productRepo": "example-org/acme-issues", // informational
|
|
113
|
+
"prBranch": "feature/M-042-issue-tracking", // informational
|
|
114
|
+
"createdAt": "2026-04-16T10:30:00Z", // ISO-8601
|
|
115
|
+
"requirements": {
|
|
116
|
+
"capabilities_must_exist": [
|
|
117
|
+
// server actions that must exist and declare the expected withProof
|
|
118
|
+
{
|
|
119
|
+
"name": "createIssue", // must equal createAction({ functionName })
|
|
120
|
+
"module": "issues", // leaf folder name under __business-logic/
|
|
121
|
+
"verb": "create", // one of PROOF_VERBS
|
|
122
|
+
"object": "issue", // free-form noun
|
|
123
|
+
"invariants": ["tenant_isolation"], // optional; strings that must appear in withProof({ invariants })
|
|
124
|
+
},
|
|
125
|
+
],
|
|
126
|
+
"schema_must_contain": [
|
|
127
|
+
// tables that must exist with a specific shape and RLS stance
|
|
128
|
+
{
|
|
129
|
+
"table": "issues",
|
|
130
|
+
"required_columns": [
|
|
131
|
+
"id",
|
|
132
|
+
"title",
|
|
133
|
+
"workspace_id",
|
|
134
|
+
"assignee_id",
|
|
135
|
+
"status",
|
|
136
|
+
],
|
|
137
|
+
"rls_classification": "workspace_scoped", // one of RLS_CLASSIFICATIONS
|
|
138
|
+
},
|
|
139
|
+
],
|
|
140
|
+
"policies_must_not_allow": [
|
|
141
|
+
// optional. Per-command claims about RLS policies, for the cases one
|
|
142
|
+
// rls_classification label cannot express. See Policy prohibitions.
|
|
143
|
+
{
|
|
144
|
+
"table": "audit_logs",
|
|
145
|
+
"role": "authenticated", // database role no policy may target
|
|
146
|
+
"commands": ["INSERT", "UPDATE", "DELETE"], // optional; omit to mean any command
|
|
147
|
+
"description": "members never write to the audit log; only the service role does",
|
|
148
|
+
},
|
|
149
|
+
],
|
|
150
|
+
"trace_must_prove": [
|
|
151
|
+
// assertion targets that must pass in the aggregated trace
|
|
152
|
+
{
|
|
153
|
+
"kind": "happy_path", // one of PROOF_KINDS
|
|
154
|
+
"target": "create_issue", // matches the `target` emitted by the assertion
|
|
155
|
+
"description": "A workspace member can create an issue in their workspace",
|
|
156
|
+
"role": "primary", // optional, defaults to "primary"; see Assertion roles
|
|
157
|
+
"proofId": "issues-create-happy-path", // optional; pin the requirement to one spec
|
|
158
|
+
"evidence": "helper", // optional; see Assertion provenance. Defaults to
|
|
159
|
+
// "helper" for tenant_isolation and "any" for every other kind.
|
|
160
|
+
},
|
|
161
|
+
],
|
|
162
|
+
},
|
|
163
|
+
"expectedChanges": {
|
|
164
|
+
// optional drift budget; see Proof-surface drift report. Omitting the
|
|
165
|
+
// block keeps proof:drift report-only.
|
|
166
|
+
"modules": ["issues"], // covers ONLY added actions in these modules
|
|
167
|
+
"actions": [
|
|
168
|
+
// exact refs; string = any change, object narrows the change kinds
|
|
169
|
+
{ "ref": "workspace:removeMember", "changes": ["modified"] },
|
|
170
|
+
],
|
|
171
|
+
"tables": [
|
|
172
|
+
"issues", // string = any change to this table
|
|
173
|
+
{ "name": "workspaces", "changes": ["columns_added"] }, // facet-narrowed
|
|
174
|
+
],
|
|
175
|
+
"dependencies": [], // runtime deps that may be added/removed/version-changed
|
|
176
|
+
"lockfile": false, // true to accept a lockfile-only resolution change
|
|
177
|
+
},
|
|
178
|
+
"acceptanceNotes": "free-form notes for reviewers and the coding agent",
|
|
179
|
+
}
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
### Rules the consumer's agent MUST follow
|
|
183
|
+
|
|
184
|
+
- Every `trace_must_prove` entry needs at least one proof spec whose step emits
|
|
185
|
+
a matching `{ kind, target, passed: true }` assertion via `assert.*` helpers.
|
|
186
|
+
Raw `expect(...)` calls do not count — only helper-driven assertions populate
|
|
187
|
+
the `assertions[]` array the validator reads.
|
|
188
|
+
- A requirement is satisfied only by an assertion of the same **role**
|
|
189
|
+
(see [Assertion roles](#assertion-roles)). A passing control assertion never
|
|
190
|
+
satisfies a `primary` requirement.
|
|
191
|
+
- A `tenant_isolation` requirement is satisfied only by an assertion whose
|
|
192
|
+
`emittedBy` identity is allowed for its target: `assert.tenantIsolation` for
|
|
193
|
+
tables, plus `assert.authorization` for action refs (see
|
|
194
|
+
[Assertion provenance](#assertion-provenance)). A hand-rolled
|
|
195
|
+
`recordAssertion` — or a caller-kind helper like `assert.httpResponse` —
|
|
196
|
+
fails as `trace_unverified`, and there is no `evidence: "any"` opt-out for
|
|
197
|
+
isolation claims.
|
|
198
|
+
- Set `proofId` on a requirement when the claim must come from one specific spec.
|
|
199
|
+
Without it, any spec in the run may satisfy it.
|
|
200
|
+
- `capabilities_must_exist[].name` must exactly match an action's
|
|
201
|
+
`createAction({ functionName: "..." })` argument.
|
|
202
|
+
- `schema_must_contain[].rls_classification` must be one of the closed values
|
|
203
|
+
in [`RLS_CLASSIFICATIONS`](src/shared/vocabulary.ts). The
|
|
204
|
+
migration parser only emits these values; a manifest that asks for anything
|
|
205
|
+
else fails at the shape-validation step.
|
|
206
|
+
- Do not commit or hand-edit `.proof/capabilities.json` or
|
|
207
|
+
`.proof/schema.json`. They are gitignored, derived build artifacts,
|
|
208
|
+
regenerated from source by `pnpm proof:build` (`proof:scan` + `proof:parse`)
|
|
209
|
+
during setup and in CI before `proof:verify` runs. Before schema parsing,
|
|
210
|
+
`proof:build` non-mutatingly verifies that every module migration, generated
|
|
211
|
+
SQL file, source-to-output mapping, and manifest hash agrees; stale aggregates
|
|
212
|
+
fail with an instruction to run `pnpm aggregate-migrations --fresh`.
|
|
213
|
+
`proof:build` also verifies that the committed generated action registry is current; use
|
|
214
|
+
`pnpm proof:registry` to update it. `schema.json` is parsed
|
|
215
|
+
from the gitignored migration aggregate, so a committed copy would encode a
|
|
216
|
+
stale, unverifiable `initial.sql` filename. If you need them locally, run
|
|
217
|
+
`pnpm proof:build` (or `pnpm aggregate-migrations --fresh && pnpm proof:build`).
|
|
218
|
+
|
|
219
|
+
### Capability map derived security fields
|
|
220
|
+
|
|
221
|
+
Each capability keeps the optional `withProof` fields and also carries scanner
|
|
222
|
+
facts used by schema-wide coverage:
|
|
223
|
+
|
|
224
|
+
```jsonc
|
|
225
|
+
{
|
|
226
|
+
"name": "updateWidget",
|
|
227
|
+
"module": "widgets",
|
|
228
|
+
"acceptsWorkspaceId": true,
|
|
229
|
+
"internalOnly": false,
|
|
230
|
+
"usesDirectUpdateTag": false,
|
|
231
|
+
"serviceRoleMutations": [{ "table": "widgets", "operation": "update" }],
|
|
232
|
+
"middleware": {
|
|
233
|
+
"auth": true,
|
|
234
|
+
"tenantIsolation": true,
|
|
235
|
+
"rbac": true,
|
|
236
|
+
},
|
|
237
|
+
}
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
`acceptsWorkspaceId` plus a non-empty `serviceRoleMutations` list derives a
|
|
241
|
+
mandatory action boundary unless `internalOnly` marks `_BOT`/`server-only`
|
|
242
|
+
plumbing. `middleware` is diagnostic context, not proof: coverage still demands
|
|
243
|
+
behavioral evidence if a generated action forgot one of those guards.
|
|
244
|
+
`usesDirectUpdateTag` blocks registry generation until the action switches to
|
|
245
|
+
`updateTagSafe`, because the proof endpoint invokes it through a Route Handler.
|
|
246
|
+
|
|
247
|
+
### Policy prohibitions
|
|
248
|
+
|
|
249
|
+
`rls_classification` gives a table one label. Postgres RLS is per-command, so a
|
|
250
|
+
table can be readable only by admins while being writable only by the service
|
|
251
|
+
role, and no single label says that. `policies_must_not_allow` covers the gap:
|
|
252
|
+
|
|
253
|
+
```jsonc
|
|
254
|
+
{
|
|
255
|
+
"table": "audit_logs",
|
|
256
|
+
"role": "authenticated",
|
|
257
|
+
"commands": ["INSERT", "UPDATE", "DELETE"], // omit for "any command"
|
|
258
|
+
"description": "members never write to the audit log; only the service role does",
|
|
259
|
+
}
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
The validator reads the `policies[]` array in `.proof/schema.json` and fails with
|
|
263
|
+
`schema_policy_permits`, naming every policy that grants the forbidden access.
|
|
264
|
+
|
|
265
|
+
**What it checks is reachability, not the predicate.** The claim is "no policy on
|
|
266
|
+
this table targets this role for these commands" — it does not read `USING` or
|
|
267
|
+
`WITH CHECK`. So `audit_logs` in this template, whose read policy is
|
|
268
|
+
`TO authenticated USING (is_super_admin())`, would _fail_ a prohibition on
|
|
269
|
+
`authenticated` + `SELECT` even though only admins can actually read a row. Write
|
|
270
|
+
prohibitions like the one above are the natural fit; "this role may act only under
|
|
271
|
+
condition X" is a behavioural claim and belongs in `trace_must_prove`, where
|
|
272
|
+
something exercises it. Three more rules worth knowing:
|
|
273
|
+
|
|
274
|
+
- A policy with no `TO` clause defaults to `PUBLIC` in Postgres, so it counts as
|
|
275
|
+
granting **every** role — including the one you forbade.
|
|
276
|
+
- A policy declared `FOR ALL` matches whatever commands you listed, because that
|
|
277
|
+
is what it does in the database.
|
|
278
|
+
- A prohibition naming a table that does not exist fails with `schema_missing`
|
|
279
|
+
rather than passing, since a claim about nothing is satisfied by accident.
|
|
280
|
+
|
|
281
|
+
There is deliberately **no** positive form ("a policy must exist"). A policy's
|
|
282
|
+
existence says a rule was written, not that it works; a permission that must hold
|
|
283
|
+
belongs in `trace_must_prove`, where something actually exercises it. A
|
|
284
|
+
prohibition is the asymmetric case: it is checkable from the migrations alone, and
|
|
285
|
+
it fails silently and dangerously when nobody checks.
|
|
286
|
+
|
|
287
|
+
### Assertion roles
|
|
288
|
+
|
|
289
|
+
Every assertion carries a `role` that says what it is evidence _of_:
|
|
290
|
+
|
|
291
|
+
| Role | Meaning |
|
|
292
|
+
| --------- | ------------------------------------------------------------------------------------- |
|
|
293
|
+
| `primary` | The claim itself — "the outsider could not read org A's rows". The default. |
|
|
294
|
+
| `control` | A sanity check that makes the primary claim meaningful — "the owner _can_ read them". |
|
|
295
|
+
|
|
296
|
+
The distinction exists because a security assertion can pass for the wrong
|
|
297
|
+
reason. "User B saw 0 rows" is satisfied both by working RLS and by a table that
|
|
298
|
+
is empty, missing, or denied to everyone. `assert.tenantIsolation` therefore also
|
|
299
|
+
proves the owner _can_ read their own rows and records that as a `control`.
|
|
300
|
+
|
|
301
|
+
Both roles land in the trace, so a reviewer can see the control was run — but only
|
|
302
|
+
`primary` assertions satisfy a `primary` requirement. Without the split, a mission
|
|
303
|
+
asking for tenant isolation could be signed off by a blanket `USING (false)`
|
|
304
|
+
policy that denies everybody, which is a bug, not isolation.
|
|
305
|
+
|
|
306
|
+
Security assertions also record the concrete `operation`: `select`, `insert`,
|
|
307
|
+
`update`, `delete`, `invoke`, or `request`. Mutation inventory keys a claim by
|
|
308
|
+
`{ kind, target, operation }`; proving that a SELECT check can turn red does not
|
|
309
|
+
silently cover an UPDATE or server-action invocation against the same target.
|
|
310
|
+
|
|
311
|
+
`"negative"` and `"positive_control"` are accepted as deprecated aliases for
|
|
312
|
+
`primary` and `control` respectively.
|
|
313
|
+
|
|
314
|
+
### Assertion status
|
|
315
|
+
|
|
316
|
+
`passed` stays the authoritative boolean — only `true` is evidence — but a
|
|
317
|
+
boolean cannot express "we did not find out", and that outcome is real. Every
|
|
318
|
+
assertion therefore also carries a `status`:
|
|
319
|
+
|
|
320
|
+
| Status | `passed` | Meaning |
|
|
321
|
+
| ------------ | -------- | --------------------------------------------------------------------------------------------- |
|
|
322
|
+
| `passed` | `true` | The claim held. |
|
|
323
|
+
| `failed` | `false` | The claim was tested and did not hold. |
|
|
324
|
+
| `incomplete` | `false` | The probe ran but could not reach a verdict — a precondition was missing. |
|
|
325
|
+
| `skipped` | `false` | The probe was deliberately not run, e.g. an isolation direction with no fixture on that side. |
|
|
326
|
+
|
|
327
|
+
Rounding a non-verdict to `true` is how a suite quietly stops proving anything;
|
|
328
|
+
rounding it to `false` makes a skip indistinguishable from a real failure. The
|
|
329
|
+
status keeps them apart while `passed: false` keeps them out of the evidence:
|
|
330
|
+
neither `incomplete` nor `skipped` can satisfy a requirement.
|
|
331
|
+
|
|
332
|
+
Where this shows up in practice: `assert.tenantIsolation` proves both directions
|
|
333
|
+
when its fixture populates both tenants, and records the reverse direction as
|
|
334
|
+
`skipped` when it does not. A factory that leaves the primary side empty records
|
|
335
|
+
`incomplete` and fails as `tenant_isolation_vacuous`; zero rows can never become
|
|
336
|
+
green evidence.
|
|
337
|
+
|
|
338
|
+
Per-proof tallies are in the aggregate under `proofs[].assertionStatus`, so a
|
|
339
|
+
consumer can spot a green run containing skips without walking every assertion.
|
|
340
|
+
|
|
341
|
+
`status` is absent on traces written before it existed; derive it from `passed`
|
|
342
|
+
(`passed ? "passed" : "failed"`), which is what the SDK does.
|
|
343
|
+
|
|
344
|
+
### Assertion provenance
|
|
345
|
+
|
|
346
|
+
Every assertion an `assert.*` helper emits carries `emittedBy` — the helper's
|
|
347
|
+
name, e.g. `"assert.tenantIsolation"`. The field is stamped by the SDK through
|
|
348
|
+
an internal channel: the public `recordAssertion(...)` strips any
|
|
349
|
+
caller-supplied value, so spec code cannot impersonate a helper. An assertion
|
|
350
|
+
without `emittedBy` was recorded directly by a spec.
|
|
351
|
+
|
|
352
|
+
The distinction is evidential, not cosmetic. A helper-emitted assertion went
|
|
353
|
+
through the helper's vacuity controls, positive controls, and service-role
|
|
354
|
+
ground-truth re-reads; a spec-recorded assertion asserts whatever the spec
|
|
355
|
+
chose to write into it. `trace_must_prove` entries therefore accept an
|
|
356
|
+
optional `evidence` field:
|
|
357
|
+
|
|
358
|
+
| Value | Meaning |
|
|
359
|
+
| ---------- | ---------------------------------------------------------------------- |
|
|
360
|
+
| `"helper"` | Only assertions with helper provenance satisfy this requirement. |
|
|
361
|
+
| `"any"` | Spec-recorded assertions count too (mutation inventory still applies). |
|
|
362
|
+
|
|
363
|
+
**Tenant isolation is always `"helper"` — there is no opt-out.** Declaring
|
|
364
|
+
`evidence: "any"` for a `tenant_isolation` requirement fails as
|
|
365
|
+
`manifest_shape`: strict coverage rejects raw isolation assertions
|
|
366
|
+
unconditionally, and the mission gate must never accept what coverage
|
|
367
|
+
refuses. Every other kind defaults to `"any"`, because bespoke probes (RPC
|
|
368
|
+
grants, column guards) are legitimate there and remain backstopped by the
|
|
369
|
+
mutation inventory.
|
|
370
|
+
|
|
371
|
+
**For tenant isolation the stamp is identity-checked, not merely
|
|
372
|
+
present-checked.** Helpers differ in who chooses the `kind`:
|
|
373
|
+
`assert.httpResponse` records its caller's kind verbatim, so its stamp on a
|
|
374
|
+
`tenant_isolation` assertion would prove only that an HTTP probe ran. The
|
|
375
|
+
allowed identities are:
|
|
376
|
+
|
|
377
|
+
| Target | Accepted `emittedBy` |
|
|
378
|
+
| -------------------------- | ------------------------------------------------ |
|
|
379
|
+
| A table in `schema.json` | `assert.tenantIsolation` only |
|
|
380
|
+
| Anything else (action ref) | `assert.tenantIsolation`, `assert.authorization` |
|
|
381
|
+
|
|
382
|
+
**User callbacks never inherit the stamp.** Provenance is suspended around
|
|
383
|
+
every executor-authored callback a helper invokes (`fixture.create`,
|
|
384
|
+
`setup`), so a hostile fixture calling `recordAssertion` mid-helper records an
|
|
385
|
+
unstamped assertion.
|
|
386
|
+
|
|
387
|
+
A matching passing assertion that fails only the provenance requirement is
|
|
388
|
+
reported as `trace_unverified`, not `trace_missing`, since the fix is
|
|
389
|
+
different: emit through the owning helper, or (for non-isolation kinds) set
|
|
390
|
+
`evidence: "any"` deliberately.
|
|
391
|
+
|
|
392
|
+
`pnpm proof:coverage` applies the same identity rules: a passing primary
|
|
393
|
+
`tenant_isolation` assertion with the wrong (or no) origin does not count
|
|
394
|
+
toward a table's invariant or a service-role action boundary, and is listed in
|
|
395
|
+
the report's `UNVERIFIED ORIGIN` section with its actual origin.
|
|
396
|
+
|
|
397
|
+
`emittedBy` is absent on traces written before it existed; treat absence as
|
|
398
|
+
"origin unknown" (which is why fresh verification, never old traces, feeds the
|
|
399
|
+
validator). Provenance is a validator-visible guarantee, not a hard sandbox: a
|
|
400
|
+
spec deep-importing SDK internals could reach the stamping channel. The
|
|
401
|
+
factory profile's Keystone protection of the SDK and specs closes that; in
|
|
402
|
+
tandem mode the raised bar plus mutation inventory is the honest claim.
|
|
403
|
+
|
|
404
|
+
### Proof-surface drift report
|
|
405
|
+
|
|
406
|
+
`pnpm proof:drift` (running directly after `proof:build`, so undeclared
|
|
407
|
+
scope is rejected cheaply before the database-backed proofs start) answers
|
|
408
|
+
the question the mission validator cannot: **what else did this PR do — on
|
|
409
|
+
the surfaces the proof system can see?** It regenerates `capabilities.json` and `schema.json`
|
|
410
|
+
from the PR's merge base (via `git archive` + the base checkout's own
|
|
411
|
+
generator scripts — no database needed), diffs them, `package.json`, and the
|
|
412
|
+
lockfile-changed signal against the head, and writes `.proof/drift.json`
|
|
413
|
+
(gitignored) plus a printed table of every delta:
|
|
414
|
+
|
|
415
|
+
| Severity | Deltas |
|
|
416
|
+
| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
417
|
+
| `high` | action added/removed/changed (incl. weakened middleware, new service-role mutations); table added/removed/reshaped (columns, RLS classification, policies incl. **role lists**); ANY runtime dependency delta (add/remove/version change); lockfile-only resolution change |
|
|
418
|
+
| `info` | action file moved; devDependency added; lockfile change accompanying package.json dependency changes |
|
|
419
|
+
|
|
420
|
+
**What drift does NOT see.** It grades exactly the scanner-derived surfaces,
|
|
421
|
+
and calling it more would be dishonest: RLS policy **predicates**
|
|
422
|
+
(`USING` / `WITH CHECK` bodies) are not parsed — only name, command, and role
|
|
423
|
+
list; column types, constraints, and defaults; SQL grants, triggers, and
|
|
424
|
+
function bodies; transitive dependency contents; and ordinary product
|
|
425
|
+
behavior are all invisible to it. "No drift" means "no change on the
|
|
426
|
+
proof-relevant derived surfaces", never "nothing else changed" — behavior
|
|
427
|
+
inside declared scope still merits review or a `trace_must_prove` entry.
|
|
428
|
+
|
|
429
|
+
Without an `expectedChanges` budget in the mission, the report is
|
|
430
|
+
informational. With one, every high-severity delta must fall inside the
|
|
431
|
+
declared budget or the check fails with `drift_undeclared`, naming the change
|
|
432
|
+
and the missing declaration. The budget is narrow by default:
|
|
433
|
+
|
|
434
|
+
- `modules` covers **only added actions** in the named module (the new-module
|
|
435
|
+
convenience). Modifying or removing an existing action is never authorized
|
|
436
|
+
by a module declaration.
|
|
437
|
+
- `actions` names exact `module:name` refs. A bare string allows any change;
|
|
438
|
+
`{ "ref": "...", "changes": [...] }` narrows to facets (`added` ·
|
|
439
|
+
`removed` · `middleware_changed` · `invariants_changed` ·
|
|
440
|
+
`service_role_mutations_changed` · `metadata_changed`). A modified action
|
|
441
|
+
is covered only when **every** facet of the change is declared — declaring
|
|
442
|
+
`invariants_changed` does not authorize an RBAC/auth middleware change on
|
|
443
|
+
the same action.
|
|
444
|
+
- `tables` names tables. A bare string allows any change;
|
|
445
|
+
`{ "name": "...", "changes": [...] }` narrows to facets (`added` ·
|
|
446
|
+
`removed` · `columns_added` · `columns_removed` ·
|
|
447
|
+
`rls_classification_changed` · `policies_changed`). A modified table is
|
|
448
|
+
covered only when **every** facet of the change is declared — declaring
|
|
449
|
+
`columns_added` does not authorize a policy edit on the same table.
|
|
450
|
+
- `dependencies` names packages; a declared name covers add, remove, and
|
|
451
|
+
version change alike.
|
|
452
|
+
- `lockfile: true` accepts a lockfile-only resolution change (the lockfile
|
|
453
|
+
moved while the `package.json` dependency surface did not) — high-severity
|
|
454
|
+
because it means what ships changed with no visible dependency edit.
|
|
455
|
+
|
|
456
|
+
Budget that saw no change is reported ("declared but unchanged"), not failed.
|
|
457
|
+
|
|
458
|
+
CI uploads `.proof/drift.json` as the `proof-drift` workflow artifact (30-day
|
|
459
|
+
retention, alongside `proof-traces`), so a consumer reads the structured
|
|
460
|
+
report the same way it reads the traces — `gh run download --name
|
|
461
|
+
proof-drift`.
|
|
462
|
+
|
|
463
|
+
**Enforced drift fails closed.** When the mission declares a budget and the
|
|
464
|
+
drift check cannot be assessed — base unresolvable, base artifacts
|
|
465
|
+
unbuildable — it fails with `drift_unassessed` instead of exiting 0; a gate
|
|
466
|
+
that skips on breakage is not a gate. A malformed `expectedChanges` block
|
|
467
|
+
(or an unparseable mission file) likewise fails `proof:drift` itself as
|
|
468
|
+
`manifest_shape`, even standalone, rather than downgrading to report-only. Report-only mode (no budget) stays
|
|
469
|
+
permissive so unusual local checkouts never break on this step. The base is
|
|
470
|
+
the PR base in CI (`PROOF_DRIFT_BASE`, set by the workflow) and the merge
|
|
471
|
+
base with `origin/main`/`main` locally; an explicitly supplied base that does
|
|
472
|
+
not resolve fails loudly in either mode.
|
|
473
|
+
|
|
474
|
+
### Pre-schema fixture contract
|
|
475
|
+
|
|
476
|
+
A proof may be authored before the migration that defines its valid row shape.
|
|
477
|
+
In that state, create `e2e/fixtures/<table>.ts` with
|
|
478
|
+
`pendingProofFixture("<table>", reason)` and pass it to
|
|
479
|
+
`assert.tenantIsolation({ table, fixture })`. The run records a primary
|
|
480
|
+
`incomplete` assertion and fails with `fixture_factory_required`; the unfinished
|
|
481
|
+
proof cannot satisfy a mission.
|
|
482
|
+
|
|
483
|
+
After the schema settles, the builder replaces the placeholder with
|
|
484
|
+
`defineProofFixture({ table, create })`. The factory receives the two seeded
|
|
485
|
+
workspaces, two seeded users, and the service-role client. Required foreign keys
|
|
486
|
+
must resolve to those handles or rows the factory explicitly creates, while
|
|
487
|
+
`CHECK`, domain, and enum values must come from the real product schema. A
|
|
488
|
+
factory declares its table at runtime, so accidentally importing a copied
|
|
489
|
+
factory for another target fails as `fixture_factory_mismatch` before insertion.
|
|
490
|
+
|
|
491
|
+
The planner-owned proof, not the executor-owned factory, defines which semantic
|
|
492
|
+
rows count as evidence. `assert.tenantIsolation` accepts an optional
|
|
493
|
+
`criterion: { description, where }`; the equality filter is combined with each
|
|
494
|
+
side's tenant scope for the service-role ground-truth count, the owner's positive
|
|
495
|
+
control, and the outsider probe. For example, a proof may require
|
|
496
|
+
`{ description: "published widgets", where: { status: "published" } }`.
|
|
497
|
+
|
|
498
|
+
Rows that satisfy tenant scope but not the criterion do not rescue an empty
|
|
499
|
+
proof: if the primary side has no matching row, the assertion is `incomplete`
|
|
500
|
+
and fails as `tenant_isolation_vacuous`; if the reverse side has none, that
|
|
501
|
+
direction is recorded as `skipped`. The criterion may not contain the resolved
|
|
502
|
+
scope column because the helper supplies a different scope value for each
|
|
503
|
+
direction. This keeps autonomous-factory handoff honest: an executor can build a
|
|
504
|
+
constraint-valid fixture, but cannot silently redefine the semantic subset
|
|
505
|
+
stated by the protected proof.
|
|
506
|
+
|
|
507
|
+
There is deliberately no adaptive `insertFixtureRow()` in v1. The derived schema
|
|
508
|
+
artifact retains table/column names and RLS classification, but not the types,
|
|
509
|
+
nullability, defaults, generated columns, foreign keys, checks, domains, or enum
|
|
510
|
+
members needed to synthesize a valid row without guessing. Iterating on database
|
|
511
|
+
errors would encode product decisions in a generic SDK and incentivize weakening
|
|
512
|
+
real constraints. Keep those decisions explicit in the table factory; see
|
|
513
|
+
[`e2e/fixtures/README.md`](e2e/fixtures/README.md).
|
|
514
|
+
|
|
515
|
+
### Closed vocabularies
|
|
516
|
+
|
|
517
|
+
Full definitions in [`src/shared/vocabulary.ts`](src/shared/vocabulary.ts).
|
|
518
|
+
|
|
519
|
+
| Vocabulary | Allowed values |
|
|
520
|
+
| --------------------- | ----------------------------------------------------------------------------------------------------- |
|
|
521
|
+
| `PROOF_VERBS` | `create` · `read` · `update` · `delete` · `invoke` · `transfer` |
|
|
522
|
+
| `PROOF_KINDS` | `happy_path` · `tenant_isolation` · `authorization` · `validation` · `error_handling` · `idempotency` |
|
|
523
|
+
| `AssertionRole` | `primary` · `control` (aliases: `negative` → `primary`, `positive_control` → `control`) |
|
|
524
|
+
| `AssertionStatus` | `passed` · `failed` · `incomplete` · `skipped` |
|
|
525
|
+
| `ProofOperation` | `select` · `insert` · `update` · `delete` · `invoke` · `request` |
|
|
526
|
+
| `RLS_CLASSIFICATIONS` | `workspace_scoped` · `user_scoped` · `admin_only` · `public_read` · `service_only` · `unclassified` |
|
|
527
|
+
|
|
528
|
+
---
|
|
529
|
+
|
|
530
|
+
## 3. Outputs the consumer reads
|
|
531
|
+
|
|
532
|
+
Everything the template produces that the consumer consumes.
|
|
533
|
+
|
|
534
|
+
| Output | Location | Emitted by | Cadence |
|
|
535
|
+
| --------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------- | -------------------------------------- |
|
|
536
|
+
| Capability map | `.proof/capabilities.json` (gitignored) | `pnpm proof:scan` (via `proof:build` in setup/CI) | Every setup + CI run |
|
|
537
|
+
| Module descriptors | `.proof/modules.json` (gitignored) | `pnpm proof:modules` (via `proof:build` in setup/CI) | Every setup + CI run |
|
|
538
|
+
| Schema + RLS classification | `.proof/schema.json` (gitignored) | `pnpm proof:parse` (via `proof:build` in setup/CI) | Every setup + CI run |
|
|
539
|
+
| Proof action allowlist | `app/__shared/proof-runtime/action-registry.generated.ts` | `pnpm proof:registry` | When action requirements/probes change |
|
|
540
|
+
| PR verification status | GitHub check `Proof` | `pnpm proof:check` in `.github/workflows/ci.yml` | Every PR push |
|
|
541
|
+
| Per-spec trace | `proof-traces` artifact → `<proofId>.json` | `trace.proof()` helper during spec run | Every PR push |
|
|
542
|
+
| Aggregated mission trace | `proof-traces` artifact → `<missionId>.json` | `pnpm proof:verify` aggregator | Every PR push |
|
|
543
|
+
| Mutation inventory | CI log / local exit code | `pnpm proof:inventory` (via `proof:check`) | Every proof check |
|
|
544
|
+
| Mutation detection traces | `proof-traces-mutated` artifact | `pnpm proof:mutate` (`proof:check:full` locally) | Every PR push |
|
|
545
|
+
| Schema-wide coverage report | stdout (`--json` for machine-readable) | `pnpm proof:coverage` (via `proof:check` in CI) | Every PR push |
|
|
546
|
+
| Proof-surface drift report | `proof-drift` artifact → `drift.json` (local: `.proof/drift.json`, gitignored) | `pnpm proof:drift` (via `proof:check`) | Every proof check |
|
|
547
|
+
| Migration history vs base | CI log / local exit code | `pnpm proof:migrations` (via `proof:check`) | Every proof check |
|
|
548
|
+
|
|
549
|
+
> `capabilities.json` and `schema.json` are **not committed** — they are
|
|
550
|
+
> regenerated from source on every setup and CI run. A consumer that wants to
|
|
551
|
+
> inspect them reads the copy produced in its own CI run (or runs
|
|
552
|
+
> `pnpm proof:build` locally), not a copy from the repo. The generated action
|
|
553
|
+
> registry is committed and freshness-checked because application compilation
|
|
554
|
+
> imports it before proof artifacts necessarily exist.
|
|
555
|
+
|
|
556
|
+
### Reading the trace artifact
|
|
557
|
+
|
|
558
|
+
After the `Proof` check finishes (pass or fail), the full
|
|
559
|
+
`.proof/traces/` directory is uploaded as a workflow artifact named
|
|
560
|
+
`proof-traces` with a 30-day retention window. Consumers fetch it with the
|
|
561
|
+
GitHub CLI:
|
|
562
|
+
|
|
563
|
+
```bash
|
|
564
|
+
RUN_ID=$(gh run list \
|
|
565
|
+
--workflow ci.yml \
|
|
566
|
+
--branch "$PR_BRANCH" \
|
|
567
|
+
--limit 1 \
|
|
568
|
+
--json databaseId -q '.[0].databaseId')
|
|
569
|
+
|
|
570
|
+
# gh run download refuses to overwrite; always extract into a fresh dir.
|
|
571
|
+
rm -rf /tmp/proof && mkdir -p /tmp/proof
|
|
572
|
+
gh run download "$RUN_ID" --name proof-traces --dir /tmp/proof
|
|
573
|
+
|
|
574
|
+
# Single source of truth for pass/fail:
|
|
575
|
+
jq '.passed' "/tmp/proof/${MISSION_ID}.json" # -> true | false
|
|
576
|
+
jq '.issues' "/tmp/proof/${MISSION_ID}.json" # -> [] when passed, [{category, message, ...}, ...] when failed
|
|
577
|
+
jq '.proofs[] | {proofId, passed}' "/tmp/proof/${MISSION_ID}.json" # per-spec verdicts
|
|
578
|
+
```
|
|
579
|
+
|
|
580
|
+
The aggregated `<missionId>.json` is the authoritative evidence for a
|
|
581
|
+
mission. It is **written on both pass and fail** — consumers should always
|
|
582
|
+
read `.passed`, never infer from file presence. Per-spec `<proofId>.json`
|
|
583
|
+
files sit alongside it for drill-down debugging.
|
|
584
|
+
|
|
585
|
+
The separate `proof-traces-mutated` artifact contains one directory per planted
|
|
586
|
+
defect under `<mutationId>/`. Every contained trace is an intentionally red (or,
|
|
587
|
+
when the harness finds a miss, unexpectedly green) run against that defect.
|
|
588
|
+
These files are detection evidence, not mission verdicts; never merge them into
|
|
589
|
+
the green `proof-traces` directory before validation.
|
|
590
|
+
|
|
591
|
+
### Canonical consumer conformance corpus
|
|
592
|
+
|
|
593
|
+
[`proof-consumer-fixtures/`](proof-consumer-fixtures/) is the executable
|
|
594
|
+
ingestion companion to this contract. Its [`corpus.json`](proof-consumer-fixtures/corpus.json)
|
|
595
|
+
indexes canonical passing, failing, mission-aggregate, mutation-directory,
|
|
596
|
+
duplicate, and malformed cases with their expected classification or rejection.
|
|
597
|
+
The template runs every case through focused tests, including the same mission
|
|
598
|
+
validator used by `pnpm proof:verify`.
|
|
599
|
+
|
|
600
|
+
An external orchestrator should run the whole corpus through its production
|
|
601
|
+
download and ingestion path. In particular, it must:
|
|
602
|
+
|
|
603
|
+
- classify aggregates before collecting per-spec traces, so the aggregate's
|
|
604
|
+
embedded `traces` are not counted a second time;
|
|
605
|
+
- keep mutation directories separate from baseline evidence even when both
|
|
606
|
+
contain `<proofId>.json` with the same filename;
|
|
607
|
+
- retain and cross-check the mutation ID from the directory, `mutation.json`,
|
|
608
|
+
and each trace's `mutation` provenance instead of dropping it during
|
|
609
|
+
normalization;
|
|
610
|
+
- reject duplicate baseline `proofId` values before a map, database upsert, or
|
|
611
|
+
extraction step silently overwrites one run; and
|
|
612
|
+
- preserve `requirementEvidence` and verify that each satisfied entry still
|
|
613
|
+
resolves to the named proof/spec and its passing assertion.
|
|
614
|
+
|
|
615
|
+
See the fixture [README](proof-consumer-fixtures/README.md) for the exact
|
|
616
|
+
ingestion expectations and the reference test command. GitHub permission
|
|
617
|
+
classification and repository-readiness polling remain downstream concerns;
|
|
618
|
+
they are integration-specific and deliberately absent from this template-core
|
|
619
|
+
corpus.
|
|
620
|
+
|
|
621
|
+
### Aggregated trace shape
|
|
622
|
+
|
|
623
|
+
```jsonc
|
|
624
|
+
{
|
|
625
|
+
"schemaVersion": 1,
|
|
626
|
+
"missionId": "M-042",
|
|
627
|
+
"passed": true, // top-level verdict; consumers read this single boolean
|
|
628
|
+
"aggregatedAt": "2026-04-16T11:00:00Z",
|
|
629
|
+
"manifestSummary": {
|
|
630
|
+
"missionTitle": "…",
|
|
631
|
+
"capabilities_must_exist": 2,
|
|
632
|
+
"schema_must_contain": 1,
|
|
633
|
+
"trace_must_prove": 3,
|
|
634
|
+
},
|
|
635
|
+
// Empty [] on pass. On fail, every [PROOF_FAIL] the validator emitted
|
|
636
|
+
// (same structure the logs print) so consumers never need to log-scrape.
|
|
637
|
+
"issues": [
|
|
638
|
+
// example failure entry, omitted when `passed: true`
|
|
639
|
+
// { "category": "capability", "message": "…", "expected": "…", "found": "…" }
|
|
640
|
+
],
|
|
641
|
+
// Which proof actually satisfied each requirement — attribution, so a
|
|
642
|
+
// consumer never has to guess where a verdict came from.
|
|
643
|
+
"requirementEvidence": [
|
|
644
|
+
{
|
|
645
|
+
"requirement": "trace:tenant_isolation:issues",
|
|
646
|
+
"satisfied": true,
|
|
647
|
+
"proofId": "issues-tenant-isolation",
|
|
648
|
+
"specFile": "e2e/proofs/issues-tenant-isolation.proof.ts",
|
|
649
|
+
},
|
|
650
|
+
],
|
|
651
|
+
// What produced this run. Answers "which code emitted this evidence?".
|
|
652
|
+
"provenance": {
|
|
653
|
+
"commit": "a1b2c3d…",
|
|
654
|
+
"branch": "feature/M-042-issue-tracking",
|
|
655
|
+
"dirty": false, // true when the working tree had uncommitted changes
|
|
656
|
+
"repository": "example-org/acme-issues",
|
|
657
|
+
"ci": true,
|
|
658
|
+
"command": "proof-harness verify",
|
|
659
|
+
"nodeVersion": "v24.19.0",
|
|
660
|
+
"platform": "linux",
|
|
661
|
+
},
|
|
662
|
+
"proofs": [
|
|
663
|
+
{
|
|
664
|
+
"proofId": "issues-create-happy-path",
|
|
665
|
+
"missionId": "M-042", // null if the spec did not declare one
|
|
666
|
+
"passed": true,
|
|
667
|
+
"durationMs": 1234,
|
|
668
|
+
"stepCount": 2,
|
|
669
|
+
"assertionCount": 1,
|
|
670
|
+
// Tally by status. `skipped` above 0 on a passing proof means a direction
|
|
671
|
+
// was never measured — see "Assertion status".
|
|
672
|
+
"assertionStatus": {
|
|
673
|
+
"passed": 1,
|
|
674
|
+
"failed": 0,
|
|
675
|
+
"incomplete": 0,
|
|
676
|
+
"skipped": 0,
|
|
677
|
+
},
|
|
678
|
+
},
|
|
679
|
+
],
|
|
680
|
+
"traces": [
|
|
681
|
+
/* the full TraceArtifact[] from .proof/traces/<proofId>.json for each spec */
|
|
682
|
+
],
|
|
683
|
+
}
|
|
684
|
+
```
|
|
685
|
+
|
|
686
|
+
Each per-spec `TraceArtifact` (`schemaVersion: 2`) additionally carries four
|
|
687
|
+
provenance fields, so a single `<proofId>.json` is self-contained evidence rather
|
|
688
|
+
than something that only means anything next to its aggregate:
|
|
689
|
+
|
|
690
|
+
| Field | Answers |
|
|
691
|
+
| ---------- | ------------------------------------------------------------- |
|
|
692
|
+
| `specFile` | Which spec made this claim |
|
|
693
|
+
| `specHash` | First 12 hex of SHA-256 of that spec's contents |
|
|
694
|
+
| `commit` | Which commit of the application it was made against |
|
|
695
|
+
| `dirty` | Whether the working tree had uncommitted changes at that time |
|
|
696
|
+
| `mutation` | Planted defect id and `planted: true`; absent on normal runs |
|
|
697
|
+
|
|
698
|
+
A trace whose `specHash` does not match the spec in the PR was produced by
|
|
699
|
+
different code. A trace whose `commit` does not match the PR head describes
|
|
700
|
+
different code — which is what lets a consumer say "red at X, green at Y" and
|
|
701
|
+
mean it. `dirty: true` marks evidence gathered against a working tree that exists
|
|
702
|
+
on exactly one machine; a consumer gating on proofs should refuse it.
|
|
703
|
+
|
|
704
|
+
`commit` and `dirty` are absent, never fabricated, outside a git checkout —
|
|
705
|
+
shallow clones and exported tarballs still have to produce usable artifacts. On a
|
|
706
|
+
CI `pull_request` run, `commit` is the ephemeral merge commit the checkout
|
|
707
|
+
created: accurate about what was tested, but not a commit you can return to, so
|
|
708
|
+
record the PR head alongside it if you need to navigate back.
|
|
709
|
+
|
|
710
|
+
`pnpm proof:verify` clears previous top-level JSON traces before every normal
|
|
711
|
+
run, then resolves provenance once and passes it to the specs so every trace
|
|
712
|
+
agrees. In `--no-run` mode it preserves the supplied evidence but rejects traces
|
|
713
|
+
from another commit or whose `specFile` no longer exists with `stale_trace`.
|
|
714
|
+
|
|
715
|
+
---
|
|
716
|
+
|
|
717
|
+
### 3.1 Module descriptors — what exists, and what must be decided
|
|
718
|
+
|
|
719
|
+
`.proof/modules.json` answers a different question from everything else here.
|
|
720
|
+
The trace artifacts say _"this claim was proved"_; this one says _"this is what
|
|
721
|
+
exists, and this is what a buyer has to settle before it can be used"_. It is
|
|
722
|
+
what a planning agent reads instead of the template's source, which is the point:
|
|
723
|
+
without it the planner needs the codebase in its context, and that grows with
|
|
724
|
+
every module we write.
|
|
725
|
+
|
|
726
|
+
Regenerate it with `pnpm proof:modules`. Unlike `schema.json` it needs **no
|
|
727
|
+
Docker, database, or migration aggregate** — it reads the filesystem and
|
|
728
|
+
`capabilities.json`, and takes about a second.
|
|
729
|
+
|
|
730
|
+
Each entry has a declared half and a derived half:
|
|
731
|
+
|
|
732
|
+
```jsonc
|
|
733
|
+
{
|
|
734
|
+
"schemaVersion": 1,
|
|
735
|
+
"modules": [
|
|
736
|
+
{
|
|
737
|
+
// Declared in app/__core/<module>/module.meta.ts
|
|
738
|
+
"id": "user-auth",
|
|
739
|
+
"kind": "core", // core | extension | business | generated
|
|
740
|
+
"purpose": "Signs people in and proves who they are — …",
|
|
741
|
+
"env": [],
|
|
742
|
+
"decisions": [
|
|
743
|
+
{
|
|
744
|
+
"id": "mfa_enforcement", // snake_case, unique across all modules
|
|
745
|
+
"ask": "Must people use a second factor to sign in?",
|
|
746
|
+
"because": "Stricter is safer but adds a step for everyone.",
|
|
747
|
+
"options": [{ "value": "optional", "means": "…" }],
|
|
748
|
+
"applied_by": "config", // config | content | code
|
|
749
|
+
"applies": ["app/__core/user-auth/src/config.ts#MFA_POLICY"],
|
|
750
|
+
},
|
|
751
|
+
],
|
|
752
|
+
"inputs": [
|
|
753
|
+
/* values the buyer supplies rather than picks */
|
|
754
|
+
],
|
|
755
|
+
"defaults": [
|
|
756
|
+
/* what we chose for them, each with a seam */
|
|
757
|
+
],
|
|
758
|
+
|
|
759
|
+
// Derived from source — never written by hand
|
|
760
|
+
"path": "app/__core/user-auth",
|
|
761
|
+
"entities": [], // tables its own migrations create
|
|
762
|
+
"requires": ["config", "user-management"],
|
|
763
|
+
"uses_shared": ["ui", "db-supabase"],
|
|
764
|
+
"actions": [{ "name": "signin", "verb": null, "object": null }],
|
|
765
|
+
"proven": { "specs": 1, "mutations": 0 },
|
|
766
|
+
},
|
|
767
|
+
],
|
|
768
|
+
// Tables no descriptor claims — today, the ones owned by __shared modules.
|
|
769
|
+
"unowned_tables": [
|
|
770
|
+
{
|
|
771
|
+
"table": "audit_logs",
|
|
772
|
+
"module": "audit-logging",
|
|
773
|
+
"path": "app/__shared/audit-logging",
|
|
774
|
+
},
|
|
775
|
+
],
|
|
776
|
+
}
|
|
777
|
+
```
|
|
778
|
+
|
|
779
|
+
**Two audiences, one file.** A planner reads `purpose`, `decisions`, `inputs`
|
|
780
|
+
and `applied_by` — business language, no paths, roughly 80 tokens per decision.
|
|
781
|
+
The agent that _applies_ an answer additionally reads `applies`, which names
|
|
782
|
+
every place the answer lands, including files in other modules.
|
|
783
|
+
|
|
784
|
+
**Every claim is checked.** `pnpm proof:modules:check` fails when a module has
|
|
785
|
+
no descriptor, when an id does not match its directory, when a decision has
|
|
786
|
+
fewer than two options or a duplicate id, when a declared environment variable
|
|
787
|
+
is undocumented or an undeclared one is read, and — the one that matters most in
|
|
788
|
+
practice — **when a seam no longer exists**. A moved file fails CI instead of
|
|
789
|
+
quietly sending the next agent to the wrong place.
|
|
790
|
+
|
|
791
|
+
Circular dependencies in the derived graph are **reported, not failed**, while a
|
|
792
|
+
known inversion is being resolved.
|
|
793
|
+
|
|
794
|
+
## 4. Failure contract
|
|
795
|
+
|
|
796
|
+
When `pnpm proof:check` fails, the CI log contains one or more structured
|
|
797
|
+
lines the consumer can parse without regexing prose:
|
|
798
|
+
|
|
799
|
+
```
|
|
800
|
+
[PROOF_FAIL] <category>: <human-readable message including expected vs found>
|
|
801
|
+
```
|
|
802
|
+
|
|
803
|
+
The categories a consumer needs to handle are:
|
|
804
|
+
|
|
805
|
+
| Category | Meaning | Typical routing |
|
|
806
|
+
| ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
|
|
807
|
+
| `capability_missing` | Manifest required an action the scanner didn't find | Route to coding agent: implement the action |
|
|
808
|
+
| `capability_mismatch` | Action exists but its `withProof({ verb/object/invariants })` is wrong | Route to coding agent: update the `withProof` declaration |
|
|
809
|
+
| `schema_missing` | Manifest required a table that isn't in `schema.json` | Route to coding agent: add the CREATE TABLE migration |
|
|
810
|
+
| `schema_column_missing` | Table exists, required column(s) missing | Route to coding agent: update the migration |
|
|
811
|
+
| `schema_rls_mismatch` | Table's RLS classification doesn't match manifest | Route to coding agent: revise RLS policies |
|
|
812
|
+
| `schema_policy_permits` | A policy grants access a `policies_must_not_allow` entry forbids | Route to coding agent: drop or narrow the policy |
|
|
813
|
+
| `trace_missing` | No passing assertion with the required `kind + target` | Route to coding agent: write or fix the proof spec |
|
|
814
|
+
| `trace_unverified` | A matching passing assertion exists but its origin cannot vouch for it (spec-recorded, or wrong helper) | Route to coding agent: emit the claim through the owning helper, or (non-isolation kinds) set `evidence: "any"` deliberately |
|
|
815
|
+
| `happy_path` / `tenant_isolation` / `authorization` / `validation` / `error_handling` / `idempotency` | An assertion inside a proof ran and failed | Route to coding agent: fix the spec or the feature |
|
|
816
|
+
| `manifest_shape` | Manifest JSON itself is malformed or uses an unknown vocabulary value | Route to consumer: fix the manifest generator |
|
|
817
|
+
| `proof_run` | One or more Playwright proofs threw before reaching assertions | Route to coding agent: investigate test setup / infra |
|
|
818
|
+
| `proof_server_stale` | A process on the proof port has an old secret or is not this checkout's proof server | Route to coding agent: restart the dev server after loading the current `.env.local` |
|
|
819
|
+
| `no_traces` | The run produced no trace artifacts at all | Route to coding agent: no evidence was emitted |
|
|
820
|
+
| `stale_trace` | A trace describes another commit or a proof spec that no longer exists | Route to coding agent: run fresh verification or restore the referenced spec |
|
|
821
|
+
| `coverage_gap` | An invariant no proof asserts: implied by a table's classification, or declared by an action's `withProof` | Route to coding agent: add the proof, accept the gap (tables only), or drop the declaration |
|
|
822
|
+
| `service_role_action_gap` | A workspace-id-driven service-role mutation lacks a denial and/or allowed-path control | Route to coding agent: exercise both paths, then run `pnpm proof:registry` |
|
|
823
|
+
| `coverage_blind_spot` | A table's policies could not be classified and were never reviewed | Route to coding agent: review it in the coverage policy |
|
|
824
|
+
| `coverage_policy_stale` | An accepted gap or acknowledgement no longer applies | Route to coding agent: delete the stale entry |
|
|
825
|
+
| `drift_undeclared` | A high-severity action/table/dependency/lockfile change is outside the mission's `expectedChanges` budget | Route to coding agent: revert the unplanned change, or route to consumer: widen the mission's declared budget |
|
|
826
|
+
| `drift_unassessed` | The mission declares an `expectedChanges` budget but drift could not be assessed (base unavailable) | Route to consumer: fix the CI base resolution (fetch history, set `PROOF_DRIFT_BASE`); enforced drift never passes unassessed |
|
|
827
|
+
| `migration_edited` / `migration_deleted` | A module migration already on the PR base was modified or removed | Route to coding agent: restore the file and add a new migration that `ALTER`s |
|
|
828
|
+
| `migration_loosened` | A new migration drops NOT NULL / CONSTRAINT or `ALTER COLUMN … SET DEFAULT` without an accept-list entry | Route to coding agent: keep the constraint, or accept the file in `.proof/migration-policy.json` with a written reason |
|
|
829
|
+
| `migration_unassessed` | The migration history check could not see the PR base (shallow clone, missing `main`) | Route to consumer: fetch history (`fetch-depth: 0`) or set `PROOF_DRIFT_BASE`; this check never passes unassessed |
|
|
830
|
+
| `migration_policy` | `.proof/migration-policy.json` is missing, malformed, or names a new file that does not loosen | Route to coding agent: fix the accept-list |
|
|
831
|
+
|
|
832
|
+
Every `[PROOF_FAIL]` line is terminal — exit code is `1` regardless of how
|
|
833
|
+
many issues appeared.
|
|
834
|
+
|
|
835
|
+
### What cannot silently pass
|
|
836
|
+
|
|
837
|
+
A verification harness is only worth its exit code, so these are guarantees, each
|
|
838
|
+
covered by a test or a mutation:
|
|
839
|
+
|
|
840
|
+
- **A shipped migration cannot be rewritten.** `pnpm proof:migrations` diffs
|
|
841
|
+
module SQL against the PR base. Editing or deleting a file already on that
|
|
842
|
+
base fails as `migration_edited` / `migration_deleted`; a new file that
|
|
843
|
+
drops NOT NULL / CONSTRAINT or `ALTER COLUMN … SET DEFAULT` fails as
|
|
844
|
+
`migration_loosened` unless `.proof/migration-policy.json` accepts that
|
|
845
|
+
file. A missing base (shallow clone) fails as `migration_unassessed`
|
|
846
|
+
instead of passing.
|
|
847
|
+
- **A crashed run fails.** Playwright's exit code is propagated as `proof_run`,
|
|
848
|
+
even when the specs that did run emitted passing traces.
|
|
849
|
+
- **A stale dev server is never reused by address alone.** Normal and mutation
|
|
850
|
+
runners authenticate `/api/proof/health` with the current `API_SECRET_KEY`
|
|
851
|
+
before reusing a process and require the current health protocol plus the same
|
|
852
|
+
Supabase URL. A secret, protocol, or backend mismatch (or an unrelated app on
|
|
853
|
+
the port) fails as `proof_server_stale` with restart guidance instead of
|
|
854
|
+
turning every proof into an unexplained 403 or testing the wrong database.
|
|
855
|
+
- **An absent action cannot count as a refusal.** `actAsUser.invokeAction`
|
|
856
|
+
throws on every non-2xx harness response, including `action_not_registered`;
|
|
857
|
+
only a registered action's own `{ success: false }` can satisfy a denial
|
|
858
|
+
probe.
|
|
859
|
+
- **An action refusal can carry a positive control.** `assert.actionSucceeds`
|
|
860
|
+
exercises the allowed path and records `role: "control"` by default, while
|
|
861
|
+
action probes can record the `withProof` invariant they actually establish.
|
|
862
|
+
- **An assertion with nothing to observe fails.** `assert.tenantIsolation`
|
|
863
|
+
refuses to certify isolation when the seeded org has no rows
|
|
864
|
+
(`tenant_isolation_vacuous`), and requires the owner's positive control to pass
|
|
865
|
+
(`tenant_isolation_control`).
|
|
866
|
+
- **A write probe cannot mistake an API error for a denial.**
|
|
867
|
+
`assert.authorization` decides `insert` / `update` / `delete` verdicts by
|
|
868
|
+
re-reading the table with the service role, because PostgREST can return an
|
|
869
|
+
RLS error for the `RETURNING` clause while the row is committed anyway.
|
|
870
|
+
- **A read probe with nothing to read fails.** `assert.authorization`'s `select`
|
|
871
|
+
probe first counts matching rows with the service role, and reports
|
|
872
|
+
`authorization_vacuous` when the table is empty — reading zero rows from an
|
|
873
|
+
empty table is not evidence that RLS filtered anything.
|
|
874
|
+
- **A write probe whose actor cannot see the target fails.** PostgreSQL filters
|
|
875
|
+
the rows an UPDATE or DELETE reaches through its WHERE clause by the table's
|
|
876
|
+
SELECT policy, so an actor who cannot read a row cannot write to it whatever the
|
|
877
|
+
write policy and grants allow — and the service-role re-read then truthfully
|
|
878
|
+
reports the row unchanged. `assert.authorization` proves the actor can read at
|
|
879
|
+
least one target row first, records it as a `control`, and fails as
|
|
880
|
+
`authorization_blind` otherwise. On an admin-only table that means probing as the
|
|
881
|
+
admin: "not even the actor who can read every row may change one" is both
|
|
882
|
+
provable and the stronger claim. If the invariant is really "this actor cannot
|
|
883
|
+
SEE these rows", that is a `select` probe or `assert.tenantIsolation`.
|
|
884
|
+
- **A request that never reached the database is not a denial.** Local Supabase
|
|
885
|
+
sits behind a gateway that intermittently answers 502 under the load of a proof
|
|
886
|
+
run, and "the write errored and no row landed" is exactly what that looks like.
|
|
887
|
+
Every Supabase client the SDK hands out retries gateway statuses (502/503/504)
|
|
888
|
+
and transport failures at the transport layer, bounded and announced as
|
|
889
|
+
`[proof:retry]`; if one still gets through, the probe fails as
|
|
890
|
+
`authorization_setup` / `tenant_isolation_setup` rather than recording a pass.
|
|
891
|
+
A 4xx — and a 500, which PostgREST and route handlers use for real failures —
|
|
892
|
+
is an answer and is never retried.
|
|
893
|
+
- **Yesterday's evidence cannot vouch for today's code.** Normal verification
|
|
894
|
+
clears old traces before Playwright starts. `--no-run` preserves artifacts for
|
|
895
|
+
explicit inspection but rejects a different commit or a missing `specFile` as
|
|
896
|
+
`stale_trace`.
|
|
897
|
+
- **A control cannot stand in for the claim.** See
|
|
898
|
+
[Assertion roles](#assertion-roles).
|
|
899
|
+
- **A spec cannot impersonate a helper — and neither can a fixture.**
|
|
900
|
+
`recordAssertion` strips caller-supplied `emittedBy`; only code running
|
|
901
|
+
inside an `assert.*` helper gets the stamp, and provenance is suspended
|
|
902
|
+
around every user callback a helper invokes. For tenant isolation the stamp
|
|
903
|
+
is identity-checked, so a caller-kind helper (`assert.httpResponse`) cannot
|
|
904
|
+
vouch for it either. A hand-rolled or mislabelled isolation assertion fails
|
|
905
|
+
as `trace_unverified` and never counts for coverage. See
|
|
906
|
+
[Assertion provenance](#assertion-provenance).
|
|
907
|
+
- **An unplanned proof-surface change cannot ride along unnoticed.** When the
|
|
908
|
+
mission declares `expectedChanges`, `pnpm proof:drift` fails on any
|
|
909
|
+
undeclared high-severity action, table, dependency, or lockfile delta
|
|
910
|
+
against the PR base — and fails closed (`drift_unassessed`) when it cannot
|
|
911
|
+
assess the diff at all. See
|
|
912
|
+
[Proof-surface drift report](#proof-surface-drift-report).
|
|
913
|
+
- **Install cannot silently rewrite the gate's inputs.** The Proof CI job
|
|
914
|
+
verifies the tracked tree is git-clean immediately after `pnpm install`, so
|
|
915
|
+
a package lifecycle script cannot doctor specs, policies, or the mission
|
|
916
|
+
before verification runs. The factory profile additionally Keystone-protects
|
|
917
|
+
the root install lifecycle entries (`preinstall`/`install`/`postinstall`/
|
|
918
|
+
`prepare`), `.npmrc`, and `pnpm-workspace.yaml`.
|
|
919
|
+
- **A non-verdict cannot stand in for the claim.** `incomplete` and `skipped`
|
|
920
|
+
assertions carry `passed: false` and never satisfy a requirement. See
|
|
921
|
+
[Assertion status](#assertion-status).
|
|
922
|
+
- **A new scoped table cannot arrive unnoticed.** `pnpm proof:coverage --strict`
|
|
923
|
+
fails when a table's classification implies an invariant that no proof asserts.
|
|
924
|
+
See [Schema-wide coverage](#schema-wide-coverage).
|
|
925
|
+
- **A declared invariant cannot go unproven.** An action that claims an invariant
|
|
926
|
+
via `withProof` must have an assertion against its `<module>:<name>` ref, so a
|
|
927
|
+
declaration is evidence rather than decoration.
|
|
928
|
+
- **A claim nobody has seen fail is rejected.** `pnpm proof:check` includes the
|
|
929
|
+
mutation inventory and fails when a passing primary `{ kind, target,
|
|
930
|
+
operation }` assertion has neither a matching mutation nor a valid entry in
|
|
931
|
+
`.proof/mutation-policy.json`. Fresh passing primary `tenant_isolation`
|
|
932
|
+
assertions for classified scoped tables receive an implementation-independent
|
|
933
|
+
RLS-disable mutation automatically when no narrower mutation covers the claim.
|
|
934
|
+
- **A database mutation cannot silently stand in for an action mutation.**
|
|
935
|
+
`workspace:removeMember` is the reference source mutation: it removes the
|
|
936
|
+
action's manual role guard, requires the exact action assertion to turn red,
|
|
937
|
+
runs after database mutations, and restores the pre-run source bytes exactly
|
|
938
|
+
(including dirty-tree edits). A source mutation uses a fresh CI dev server;
|
|
939
|
+
an already-running compatible local server reloads the changed module.
|
|
940
|
+
- **Accepted mutation gaps expire.** Claim exceptions are committed in
|
|
941
|
+
`.proof/mutation-policy.json`, pinned to the current spec hash and a passing
|
|
942
|
+
positive control. A changed/deleted claim, changed spec, missing control, or
|
|
943
|
+
newly added mutation makes the acceptance stale and fails inventory.
|
|
944
|
+
|
|
945
|
+
`pnpm proof:mutate` is the regression test for these guarantees: it re-opens known
|
|
946
|
+
vulnerabilities in the local database one at a time and fails if the corresponding
|
|
947
|
+
proof stays green. Run it after changing anything in `assert.*`, the validator, or
|
|
948
|
+
the RLS migrations.
|
|
949
|
+
|
|
950
|
+
### Schema-wide coverage
|
|
951
|
+
|
|
952
|
+
Everything above is scoped to one mission: the manifest states what this PR must
|
|
953
|
+
prove, and the validator grades the PR against its own homework. That leaves a
|
|
954
|
+
specific blind spot — a mission adds a scoped table, never asks for tenant
|
|
955
|
+
isolation on it, CI goes green, and nothing in the pipeline is capable of
|
|
956
|
+
noticing.
|
|
957
|
+
|
|
958
|
+
`pnpm proof:coverage` grades the whole schema instead. It joins `schema.json`
|
|
959
|
+
(every table and its RLS classification), `capabilities.json` (declared
|
|
960
|
+
invariants) and `.proof/traces/` (assertions actually emitted), then reports, per
|
|
961
|
+
table, whether a proof exists for the invariant its classification implies:
|
|
962
|
+
|
|
963
|
+
| Classification | Required invariant | Why |
|
|
964
|
+
| ------------------ | ------------------ | ------------------------------------------------------- |
|
|
965
|
+
| `workspace_scoped` | `tenant_isolation` | Another workspace must not read it |
|
|
966
|
+
| `user_scoped` | `tenant_isolation` | Another user must not read it |
|
|
967
|
+
| `admin_only` | `authorization` | A normal user must be refused |
|
|
968
|
+
| `service_only` | `authorization` | An authenticated client must be refused |
|
|
969
|
+
| `public_read` | — | Readable by design |
|
|
970
|
+
| `unclassified` | — | Cannot be derived; must be reviewed by hand (see below) |
|
|
971
|
+
|
|
972
|
+
A gap means **unproven, not unprotected**: the policy exists — the
|
|
973
|
+
classification was derived from it — but nothing exercises it.
|
|
974
|
+
|
|
975
|
+
Only a **passing `primary`** assertion counts. Targets are matched to tables by
|
|
976
|
+
taking the segment before the first `.` and keeping it only if `schema.json` has
|
|
977
|
+
such a table, so `users.is_super_admin` resolves to `users` while `/secret-admin`
|
|
978
|
+
and `deduct_workspace_credits` do not. Unresolved targets are listed rather than
|
|
979
|
+
dropped. A typo resolves to nothing, which leaves the intended table showing as a
|
|
980
|
+
gap — wrong in the direction that nags rather than reassures.
|
|
981
|
+
|
|
982
|
+
Computed evidence is measured against authored intent in
|
|
983
|
+
`.proof/coverage-policy.json`, which **is** committed (unlike the derived
|
|
984
|
+
artifacts) because an accepted gap is a decision, and a decision is only worth
|
|
985
|
+
anything if it is reviewable in a diff:
|
|
986
|
+
|
|
987
|
+
```jsonc
|
|
988
|
+
{
|
|
989
|
+
"schemaVersion": 1,
|
|
990
|
+
// Invariants knowingly left unproven, each with a reason.
|
|
991
|
+
"acceptedGaps": [
|
|
992
|
+
{
|
|
993
|
+
"table": "widgets",
|
|
994
|
+
"kind": "tenant_isolation",
|
|
995
|
+
"since": "2026-08-15",
|
|
996
|
+
"reason": "…",
|
|
997
|
+
},
|
|
998
|
+
],
|
|
999
|
+
// Tables whose policies the parser cannot classify. No requirement can be
|
|
1000
|
+
// derived for them, so an unparseable policy would otherwise be the cheapest
|
|
1001
|
+
// way for a table to vanish from this report.
|
|
1002
|
+
"reviewedUnclassified": [
|
|
1003
|
+
{ "table": "audit_logs", "since": "2026-08-15", "note": "…" },
|
|
1004
|
+
],
|
|
1005
|
+
}
|
|
1006
|
+
```
|
|
1007
|
+
|
|
1008
|
+
`--strict` (which `pnpm proof:check` runs after `proof:verify`) fails on an
|
|
1009
|
+
unaccepted gap, an unreviewed unclassified table, or an entry that no longer
|
|
1010
|
+
applies. That makes the file a ratchet: adding a scoped table without a proof
|
|
1011
|
+
fails CI until someone either proves it or writes down why they chose not to.
|
|
1012
|
+
|
|
1013
|
+
The report is sectioned so it can be skimmed — `TABLE INVARIANTS`,
|
|
1014
|
+
`ACTION CLAIMS`, `SERVICE-ROLE ACTION BOUNDARIES`, `NOT ASSESSABLE`,
|
|
1015
|
+
`STALE POLICY ENTRIES`,
|
|
1016
|
+
`UNRESOLVED TARGETS` — and ends either on `[proof:coverage] PASS · …` or on a
|
|
1017
|
+
count of problems. **Only `--strict` emits `[PROOF_FAIL]` lines.** Without it the
|
|
1018
|
+
command reports and exits `0`, so a consumer can grep the output for the token
|
|
1019
|
+
and trust that finding it means the build failed.
|
|
1020
|
+
|
|
1021
|
+
#### Declared claims (the action half)
|
|
1022
|
+
|
|
1023
|
+
Table requirements and privileged action boundaries are derived, so they exist
|
|
1024
|
+
whether anyone wants them or not. `withProof` remains voluntary for other action
|
|
1025
|
+
claims. What coverage does with a declaration is make it cost something. Every
|
|
1026
|
+
invariant an action declares must have a passing `primary` assertion whose
|
|
1027
|
+
`kind` is that invariant and whose `target` is the `<module>:<name>` ref, or
|
|
1028
|
+
`--strict` fails with `coverage_gap`:
|
|
1029
|
+
|
|
1030
|
+
```
|
|
1031
|
+
ACTION CLAIMS
|
|
1032
|
+
─────────────────────────────────────────────────────────────────────────────
|
|
1033
|
+
Invariants declared via withProof, which must be backed by an assertion. 1
|
|
1034
|
+
of 51 capabilities declare one; 1 of 1 claim(s) proven.
|
|
1035
|
+
|
|
1036
|
+
action requires status
|
|
1037
|
+
---------------------- ------------- ------
|
|
1038
|
+
workspace:removeMember authorization proven
|
|
1039
|
+
```
|
|
1040
|
+
|
|
1041
|
+
There is deliberately **no acceptance path** for a declared claim. A table gap
|
|
1042
|
+
can be waived because the requirement is a fact about the schema; an unprovable
|
|
1043
|
+
claim is just a claim, so the honest fix is deleting it. `removeMember` is the
|
|
1044
|
+
template's worked example — note that it declares only `authorization` even
|
|
1045
|
+
though it also writes workspace-scoped data, because isolation of
|
|
1046
|
+
`workspace_members` is proven against the table, not against this action.
|
|
1047
|
+
|
|
1048
|
+
#### Derived service-role action boundaries
|
|
1049
|
+
|
|
1050
|
+
A user-facing action that accepts `workspaceId` and mutates through
|
|
1051
|
+
`createSupabaseServiceClient()` gets a requirement independently of `withProof`.
|
|
1052
|
+
Coverage needs two passing assertions against the exact `<module>:<name>` target:
|
|
1053
|
+
|
|
1054
|
+
- a `primary` `authorization` or `tenant_isolation` assertion for an
|
|
1055
|
+
unauthorized/cross-tenant call;
|
|
1056
|
+
- a `control` `happy_path`, `authorization`, or `tenant_isolation` assertion for
|
|
1057
|
+
an allowed call.
|
|
1058
|
+
|
|
1059
|
+
Both are required for a `proven` verdict. Existing migration debt may be recorded
|
|
1060
|
+
in `coverage-policy.json` as an `acceptedActionGaps` entry; new actions fail
|
|
1061
|
+
strict coverage unless their paired evidence is added or the debt is made
|
|
1062
|
+
explicit in review. Reads, user-only actions without a workspace input, and
|
|
1063
|
+
`_BOT`/`server-only` jobs or webhooks do not create this requirement.
|
|
1064
|
+
|
|
1065
|
+
The proof invocation allowlist is generated from these derived boundaries,
|
|
1066
|
+
explicit `withProof` claims, and literal action probes in `*.proof.ts`. Run
|
|
1067
|
+
`pnpm proof:registry` and commit `action-registry.generated.ts` when those inputs
|
|
1068
|
+
change. `pnpm proof:build` checks freshness before verification. The invoke
|
|
1069
|
+
route still fails unknown keys as `action_not_registered`; generation removes
|
|
1070
|
+
bookkeeping, not the allowlist boundary.
|
|
1071
|
+
|
|
1072
|
+
Coverage grades the traces on disk from the last run, which is why `proof:check`
|
|
1073
|
+
runs verification and coverage back to back. `pnpm proof:mutate` runs last in CI
|
|
1074
|
+
and immediately archives each planted run under `.proof/mutations/<mutationId>/`
|
|
1075
|
+
before the next mutation can overwrite the same proof id. CI uploads that tree
|
|
1076
|
+
separately as `proof-traces-mutated`.
|
|
1077
|
+
|
|
1078
|
+
---
|
|
1079
|
+
|
|
1080
|
+
## 5. Reference flow
|
|
1081
|
+
|
|
1082
|
+
```mermaid
|
|
1083
|
+
sequenceDiagram
|
|
1084
|
+
participant Kanban as Consumer Kanban
|
|
1085
|
+
participant Bot as Consumer Agent
|
|
1086
|
+
participant Fork as Product Repo (fork of template)
|
|
1087
|
+
participant GHA as GitHub Actions
|
|
1088
|
+
|
|
1089
|
+
Kanban->>Bot: ticket assigned (M-042)
|
|
1090
|
+
Bot->>Bot: derive manifest from ticket + schema intent
|
|
1091
|
+
Bot->>Fork: open branch feature/M-042-*
|
|
1092
|
+
Bot->>Fork: write .proof/current-mission.json
|
|
1093
|
+
Bot->>Fork: implement module + migration + proof specs
|
|
1094
|
+
Bot->>Fork: commit source + generated action registry (not runtime artifacts)
|
|
1095
|
+
Fork->>GHA: PR opened
|
|
1096
|
+
GHA->>GHA: Proof job: aggregate-migrations --fresh → supabase start → proof:check
|
|
1097
|
+
GHA->>GHA: upload proof-traces artifact (30d retention)
|
|
1098
|
+
GHA->>GHA: proof:mutate → upload proof-traces-mutated artifact
|
|
1099
|
+
GHA-->>Bot: check status + [PROOF_FAIL] lines
|
|
1100
|
+
alt pass
|
|
1101
|
+
Bot->>GHA: gh run download --name proof-traces
|
|
1102
|
+
GHA-->>Bot: .proof/traces/M-042.json
|
|
1103
|
+
Bot->>Fork: merge PR
|
|
1104
|
+
Bot->>Kanban: close ticket with trace link
|
|
1105
|
+
else fail
|
|
1106
|
+
GHA-->>Bot: structured failure categories
|
|
1107
|
+
Bot->>Fork: iterate on implementation
|
|
1108
|
+
end
|
|
1109
|
+
```
|
|
1110
|
+
|
|
1111
|
+
---
|
|
1112
|
+
|
|
1113
|
+
## 6. Worked example
|
|
1114
|
+
|
|
1115
|
+
**Manifest the consumer writes to `.proof/current-mission.json`:**
|
|
1116
|
+
|
|
1117
|
+
```json
|
|
1118
|
+
{
|
|
1119
|
+
"schemaVersion": 1,
|
|
1120
|
+
"missionId": "M-042",
|
|
1121
|
+
"missionTitle": "Add issue tracking with assignees",
|
|
1122
|
+
"productRepo": "example-org/acme-issues",
|
|
1123
|
+
"prBranch": "feature/M-042-issue-tracking",
|
|
1124
|
+
"createdAt": "2026-04-16T10:30:00Z",
|
|
1125
|
+
"requirements": {
|
|
1126
|
+
"capabilities_must_exist": [
|
|
1127
|
+
{
|
|
1128
|
+
"name": "createIssue",
|
|
1129
|
+
"module": "issues",
|
|
1130
|
+
"verb": "create",
|
|
1131
|
+
"object": "issue"
|
|
1132
|
+
},
|
|
1133
|
+
{
|
|
1134
|
+
"name": "assignIssue",
|
|
1135
|
+
"module": "issues",
|
|
1136
|
+
"verb": "update",
|
|
1137
|
+
"object": "issue"
|
|
1138
|
+
}
|
|
1139
|
+
],
|
|
1140
|
+
"schema_must_contain": [
|
|
1141
|
+
{
|
|
1142
|
+
"table": "issues",
|
|
1143
|
+
"required_columns": [
|
|
1144
|
+
"id",
|
|
1145
|
+
"title",
|
|
1146
|
+
"workspace_id",
|
|
1147
|
+
"assignee_id",
|
|
1148
|
+
"status"
|
|
1149
|
+
],
|
|
1150
|
+
"rls_classification": "workspace_scoped"
|
|
1151
|
+
}
|
|
1152
|
+
],
|
|
1153
|
+
"trace_must_prove": [
|
|
1154
|
+
{
|
|
1155
|
+
"kind": "happy_path",
|
|
1156
|
+
"target": "create_issue",
|
|
1157
|
+
"description": "A workspace member can create an issue in their workspace"
|
|
1158
|
+
},
|
|
1159
|
+
{
|
|
1160
|
+
"kind": "tenant_isolation",
|
|
1161
|
+
"target": "issues",
|
|
1162
|
+
"description": "Members of workspace A cannot read issues in workspace B"
|
|
1163
|
+
},
|
|
1164
|
+
{
|
|
1165
|
+
"kind": "authorization",
|
|
1166
|
+
"target": "assign_issue",
|
|
1167
|
+
"description": "Only workspace admins can reassign an issue created by another user"
|
|
1168
|
+
}
|
|
1169
|
+
]
|
|
1170
|
+
},
|
|
1171
|
+
"acceptanceNotes": "Assignee picker must show workspace members only. Status transitions: open -> in_progress -> closed."
|
|
1172
|
+
}
|
|
1173
|
+
```
|
|
1174
|
+
|
|
1175
|
+
**Artifacts the template produces:**
|
|
1176
|
+
|
|
1177
|
+
- `.proof/capabilities.json` — regenerated in the CI run (via `proof:build`) with entries for `createIssue` + `assignIssue`
|
|
1178
|
+
- `.proof/schema.json` — regenerated in the CI run with the `issues` table and `workspace_scoped` classification
|
|
1179
|
+
- On PR: the `Proof` check passes only if all three `trace_must_prove` entries resolve against emitted `assertions[]`
|
|
1180
|
+
- Every PR run uploads `proof-traces` with `.proof/traces/M-042.json` and
|
|
1181
|
+
each per-spec trace; downloadable via `gh run download` (see §3). This is
|
|
1182
|
+
the durable proof the consumer reads.
|
|
1183
|
+
|
|
1184
|
+
---
|
|
1185
|
+
|
|
1186
|
+
## 7. Local development without a consumer
|
|
1187
|
+
|
|
1188
|
+
Running `pnpm proof:verify` with no `.proof/current-mission.json` falls back
|
|
1189
|
+
to **no-manifest mode**: every proof spec runs, every assertion is recorded,
|
|
1190
|
+
but no manifest requirements are enforced. Use this during day-to-day
|
|
1191
|
+
development — you still get the traces for debugging, you just don't gate
|
|
1192
|
+
on a mission.
|
|
1193
|
+
|
|
1194
|
+
`proof:verify` requires `.proof/capabilities.json` and `.proof/schema.json` to
|
|
1195
|
+
exist. Because they are gitignored, a fresh clone must build them first:
|
|
1196
|
+
`scripts/setup.sh` does this automatically (`aggregate-migrations --fresh` +
|
|
1197
|
+
`proof:build`), or run `pnpm proof:check`, which regenerates them before
|
|
1198
|
+
verifying.
|
|
1199
|
+
|
|
1200
|
+
Add a manifest under `.proof/missions/local-*.json` to experiment with
|
|
1201
|
+
requirements locally without committing them; files matching that pattern
|
|
1202
|
+
are gitignored.
|
|
1203
|
+
|
|
1204
|
+
`pnpm proof:coverage` needs no mission at all — it reads the traces from the last
|
|
1205
|
+
run. Useful on its own to see what the template currently proves about its own
|
|
1206
|
+
schema, and what it does not.
|
|
1207
|
+
|
|
1208
|
+
`pnpm proof:mutate` verifies the suite still bites (see
|
|
1209
|
+
[What cannot silently pass](#what-cannot-silently-pass)). It writes to the local
|
|
1210
|
+
database and may temporarily patch explicitly listed action source. It refuses
|
|
1211
|
+
non-local databases and restores each database object or source file from an
|
|
1212
|
+
exact pre-run snapshot before continuing.
|
|
1213
|
+
Use `pnpm proof:check:full` to run the normal local gate followed by every
|
|
1214
|
+
planted mutation; ordinary `proof:check` performs the cheap inventory only.
|
|
1215
|
+
`--list` prints every planted defect, its mapped claim, and any claim deliberately
|
|
1216
|
+
accepted in `.proof/mutation-policy.json`.
|
|
1217
|
+
|
|
1218
|
+
---
|
|
1219
|
+
|
|
1220
|
+
## 8. Versioning
|
|
1221
|
+
|
|
1222
|
+
The contract is versioned via the top-level `schemaVersion` on:
|
|
1223
|
+
|
|
1224
|
+
- the mission manifest
|
|
1225
|
+
- `.proof/capabilities.json`
|
|
1226
|
+
- `.proof/schema.json`
|
|
1227
|
+
- aggregated `<missionId>.json` inside the `proof-traces` artifact
|
|
1228
|
+
|
|
1229
|
+
All four share the same version number. A consumer in v1 should assert
|
|
1230
|
+
`schemaVersion === 1` on every artifact it reads and fail loudly on a
|
|
1231
|
+
mismatch. Additive changes (new optional fields, new vocabulary entries)
|
|
1232
|
+
do not bump the version.
|
|
1233
|
+
|
|
1234
|
+
Per-spec trace artifacts version independently and are currently at
|
|
1235
|
+
`schemaVersion: 2` (v2 added `specFile` / `specHash` / `commit` / `dirty`, plus
|
|
1236
|
+
the additive `role` and `status` fields on assertions). Consumers that only read
|
|
1237
|
+
the aggregate can ignore this; a consumer reading `<proofId>.json` directly should
|
|
1238
|
+
treat a missing `specHash` as "pre-v2, provenance unknown", a missing `commit` or
|
|
1239
|
+
`dirty` as "unknown" (a non-git checkout cannot supply them), and a missing
|
|
1240
|
+
`status` as derivable from `passed`.
|
|
1241
|
+
|
|
1242
|
+
`policies_must_not_allow` is an optional manifest block and did not bump the
|
|
1243
|
+
manifest version: omitting it is valid and means "no policy-level claims".
|
|
1244
|
+
The same holds for `expectedChanges` (omitting it means "drift is
|
|
1245
|
+
report-only") and `trace_must_prove[].evidence` (omitting it applies the
|
|
1246
|
+
kind-based default). Note the August 2026 provenance work is a behaviour
|
|
1247
|
+
tightening within v1: a `tenant_isolation` requirement satisfied by a
|
|
1248
|
+
hand-rolled `recordAssertion` — or a helper that cannot vouch for the kind —
|
|
1249
|
+
now correctly fails as `trace_unverified`, `evidence: "any"` is rejected for
|
|
1250
|
+
isolation claims, and assertions gained the additive `emittedBy` field without
|
|
1251
|
+
a trace version bump.
|
|
1252
|
+
|
|
1253
|
+
`.proof/coverage-policy.json` carries its own `schemaVersion` (currently `1`).
|
|
1254
|
+
It is authored, not derived, and is read only by `pnpm proof:coverage`; consumers
|
|
1255
|
+
do not need to parse it.
|