shapeup-sdlc 1.6.2 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/AGENTS.md +36 -103
- package/README.md +60 -67
- package/SECURITY.md +31 -16
- package/bin/init.mjs +24 -69
- package/commands/build.md +19 -0
- package/commands/eval.md +5 -4
- package/commands/scopes.md +5 -4
- package/commands/shape.md +1 -1
- package/commands/ship.md +37 -4
- package/hooks/anti-rationalization.mjs +7 -13
- package/hooks/compact-snapshot.mjs +5 -5
- package/hooks/gate-deadline.mjs +12 -16
- package/hooks/gate-intake.mjs +13 -14
- package/hooks/gate-l2.mjs +3 -3
- package/hooks/gate-zerowork.mjs +97 -16
- package/hooks/lib/decision.mjs +21 -13
- package/hooks/safety-spine.mjs +1 -1
- package/hooks/sandbox-guard.mjs +83 -47
- package/hooks/session-rehydrate.mjs +8 -9
- package/hooks/slop-cleaner.mjs +1 -1
- package/oracles/_shared.mjs +1 -1
- package/oracles/http-oracle.mjs +1 -1
- package/oracles/index.mjs +1 -1
- package/oracles/process-oracle.mjs +1 -1
- package/oracles/snapshot-oracle.mjs +1 -1
- package/oracles/test-oracle.mjs +1 -1
- package/package.json +3 -7
- package/skills/ba-pitch-analyzer/SKILL.md +20 -15
- package/skills/ba-pitch-analyzer/references/doc-schemas.md +1 -1
- package/skills/ba-pitch-analyzer/references/task-generation.md +6 -6
- package/skills/ba-pitch-analyzer/references/test-surface.md +1 -1
- package/skills/ba-pitch-analyzer/scripts/board-derive.mjs +5 -5
- package/skills/ba-pitch-analyzer/scripts/spec-lint.mjs +2 -2
- package/skills/coach/SKILL.md +3 -3
- package/skills/orient/SKILL.md +2 -1
- package/skills/qa-edge-hunter/SKILL.md +9 -39
- package/skills/scope-architect/SKILL.md +17 -12
- package/skills/scope-hammer/SKILL.md +5 -5
- package/skills/shapeup/SKILL.md +3 -3
- package/skills/shapeup/resources/context-compaction.md +4 -3
- package/skills/solution-architect/SKILL.md +18 -11
- package/skills/spec-evaluator/SKILL.md +12 -5
- package/skills/spec-evaluator/references/dimensions/_registry.md +2 -2
- package/skills/spec-evaluator/references/dimensions/completeness.md +1 -1
- package/skills/spec-evaluator/references/dimensions/integration.md +98 -67
- package/skills/spec-evaluator/references/dimensions/spec-conformance.md +4 -4
- package/skills/spec-evaluator/references/dimensions/test-surface-conformance.md +1 -1
- package/skills/spec-evaluator/references/probing.md +42 -23
- package/skills/spec-evaluator/references/report-schema.md +2 -2
- package/skills/spec-evaluator/scripts/verdict-ledger.mjs +1 -1
- package/skills/task-executor/SKILL.md +25 -12
- package/skills/tech-lead/SKILL.md +115 -438
- package/skills/tech-lead/references/delegation.md +38 -31
- package/skills/tech-lead/references/gates.md +75 -33
- package/skills/tech-lead/references/hard-rules.md +34 -0
- package/skills/tech-lead/references/invocation.md +2 -2
- package/skills/tech-lead/references/ledger-schema.md +17 -18
- package/skills/tech-lead/references/round-protocol.md +77 -27
- package/skills/tech-lead/references/state-model.md +1 -1
- package/skills/tech-lead/schemas/domain.schema.json +320 -148
- package/skills/tech-lead/schemas/gate-answers.schema.json +2 -2
- package/skills/tech-lead/schemas/work-result.schema.json +55 -17
- package/skills/tech-lead/scripts/aegis-digest.mjs +1 -1
- package/skills/tech-lead/scripts/budget-check.mjs +7 -13
- package/skills/tech-lead/scripts/compile-order.mjs +69 -22
- package/skills/tech-lead/scripts/fit-check.mjs +10 -10
- package/skills/tech-lead/scripts/gate-answers.mjs +11 -12
- package/skills/tech-lead/scripts/hill-derive.mjs +133 -0
- package/skills/tech-lead/scripts/ingest-result.mjs +7 -19
- package/skills/tech-lead/scripts/init-run.mjs +60 -18
- package/skills/tech-lead/scripts/lib/argv.mjs +1 -1
- package/skills/tech-lead/scripts/lib/contract-md.mjs +10 -10
- package/skills/tech-lead/scripts/lib/is-main.mjs +11 -12
- package/skills/tech-lead/scripts/lib/paths.mjs +15 -13
- package/skills/tech-lead/scripts/resume-state.mjs +387 -0
- package/skills/tech-lead/scripts/run-snapshot.mjs +1 -6
- package/skills/tech-lead/scripts/run-workflow.mjs +381 -0
- package/skills/tech-lead/scripts/ship-report.mjs +33 -2
- package/skills/tech-lead/scripts/stats.mjs +9 -10
- package/skills/tech-lead/scripts/t0-verify.mjs +4 -4
- package/skills/tech-lead/scripts/trace-lint.mjs +2 -2
- package/skills/tech-lead/workflows/shapeup-run.js +911 -0
- package/skills/translator/SKILL.md +2 -2
- package/skills/advisor-protocol/SKILL.md +0 -171
|
@@ -9,9 +9,9 @@ command line overrides this table for a single run.
|
|
|
9
9
|
|----|---------|------|----------------------|-------|
|
|
10
10
|
| `spec-conformance` | ✅ true | `dimensions/spec-conformance.md` | all tasks | baseline correctness: AC + Done-when + contract shapes + non-go |
|
|
11
11
|
| `tdd-surface` | ✅ true | `dimensions/tdd-surface.md` | all tasks | suite green + companion test files for new code; TDD-1/2 critical, TDD-3 advisory |
|
|
12
|
-
| `integration` | ✅ true | `dimensions/integration.md` | `.be` and `.e2e` variants only | full-stack integration test +
|
|
12
|
+
| `integration` | ✅ true | `dimensions/integration.md` | `.be` and `.e2e` variants only | full-stack integration test + access boundary + enforcement-path integrity; stack read from the project profile, not hard-coded; runs only when variant matches |
|
|
13
13
|
| `completeness` | ⚙️ auto | `dimensions/completeness.md` | all tasks (lens lite/standard) | auto-ON when spec has UC `## Invariants` (v2.8+); no-op on older specs |
|
|
14
|
-
| `test-surface-conformance` | ⚙️ auto | `dimensions/test-surface-conformance.md` | all tasks (lens lite/standard) | auto-ON when spec has UC `## Test Surface` (v2.9+ or
|
|
14
|
+
| `test-surface-conformance` | ⚙️ auto | `dimensions/test-surface-conformance.md` | all tasks (lens lite/standard) | auto-ON when spec has UC `## Test Surface` (v2.9+ or a retrofit-surface order); no-op otherwise. Report must list every TS row probed — qa-edge-hunter's negative-space input |
|
|
15
15
|
| `security` | ⛔ false | `dimensions/security.md` | be / shared | stub — flip on when ready |
|
|
16
16
|
| `performance` | ⛔ false | `dimensions/performance.md` | be / web | stub — flip on when ready |
|
|
17
17
|
|
|
@@ -25,7 +25,7 @@ criteria?"* — it grades the tasks that exist. It cannot see the task that was
|
|
|
25
25
|
written. Shape Up's discovered-task philosophy says the real bulk of work is found while
|
|
26
26
|
building; an invariant declared on a UC but never backed by a regression task is exactly
|
|
27
27
|
that blind spot. This dimension converts "absence of a task" from invisible into a graded
|
|
28
|
-
**GAP** — the judge surfaces it; the planner (a ba-pitch-analyzer reconcile
|
|
28
|
+
**GAP** — the judge surfaces it; the planner (a ba-pitch-analyzer `reconcile` order) fills it.
|
|
29
29
|
Judge never fills it itself.
|
|
30
30
|
|
|
31
31
|
**Distinction held:** conformance = *the tasks that exist are correct*; completeness =
|
|
@@ -15,28 +15,27 @@ requires_browser: false
|
|
|
15
15
|
|
|
16
16
|
**Why this exists.** Unit tests verify isolated logic. `spec-conformance` drives the
|
|
17
17
|
running app and confirms AC at the surface. Neither catches the class of bug where each
|
|
18
|
-
layer works in isolation but the *seam* is broken: wrong auth header forwarded,
|
|
19
|
-
|
|
20
|
-
test database, or a contract mismatch between
|
|
21
|
-
This dimension fills that seam.
|
|
22
|
-
|
|
23
|
-
**What "system integration" means here
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
**
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
- Never trust a `userId` sent in the request body — derive it from the verified JWT.
|
|
18
|
+
layer works in isolation but the *seam* is broken: the wrong auth header forwarded, an
|
|
19
|
+
authorization policy never invoked because the call bypassed the layer that enforces it, a
|
|
20
|
+
schema migration not applied to the test database, or a contract mismatch between the client
|
|
21
|
+
and the handler it calls. This dimension fills that seam.
|
|
22
|
+
|
|
23
|
+
**What "system integration" means here:** the full request path — client → entry point →
|
|
24
|
+
application logic → persistence — exercised by tests that use the project's **real**
|
|
25
|
+
dependencies, not mocked layers. An integration test that mocks the database client or
|
|
26
|
+
replaces the repository with an in-memory stub is a unit test wearing integration clothing
|
|
27
|
+
and does not satisfy this dimension.
|
|
28
|
+
|
|
29
|
+
**Read the project's stack from the artifacts, not from this file.** The commands, file
|
|
30
|
+
patterns, and enforcement mechanisms differ per project: resolve them from
|
|
31
|
+
`project-profile.md` (`archetype`, `entry_point`), the scope contract's
|
|
32
|
+
`e2e_verification_fixtures`, and the spec's `contracts/`. This file states the invariant;
|
|
33
|
+
the "Probing notes" section carries worked examples of what it looks like in common stacks.
|
|
34
|
+
A project whose stack matches none of them still owes the invariant.
|
|
36
35
|
|
|
37
36
|
**`.e2e` variant:** the e2e suite inherently exercises the full stack. INT-1 (integration
|
|
38
37
|
test existence) maps to the e2e test file(s) added by the task. INT-2 and INT-3 apply if
|
|
39
|
-
the e2e task introduces any
|
|
38
|
+
the e2e task introduces any access-controlled operation.
|
|
40
39
|
|
|
41
40
|
---
|
|
42
41
|
|
|
@@ -44,47 +43,59 @@ the e2e task introduces any auth-scoped endpoint.
|
|
|
44
43
|
|
|
45
44
|
```yaml
|
|
46
45
|
- id: INT-1
|
|
47
|
-
statement: "At least one integration/e2e test exercises the feature's main flow end-to-end
|
|
46
|
+
statement: "At least one integration/e2e test exercises the feature's main flow end-to-end against the project's real dependencies — no mocked data layer."
|
|
48
47
|
probe: static + cmd
|
|
49
48
|
evidence_required: true
|
|
50
49
|
pass_rule: >
|
|
51
|
-
Locate integration test files for this task's package
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
50
|
+
Locate the integration test files for this task's package — resolve the pattern and the
|
|
51
|
+
runner command from the scope contract's `e2e_verification_fixtures`, else the project's
|
|
52
|
+
test scripts (a target named for integration/e2e rather than unit). Grep for test cases
|
|
53
|
+
that touch the happy-path scenario the task implements. Verify the test does NOT mock the
|
|
54
|
+
data layer: search the file for the project's mocking primitive and inspect what it
|
|
55
|
+
replaces. Mocking outbound third-party calls (HTTP clients, email, payment sandboxes) is
|
|
56
|
+
acceptable and often required; mocking the database client, the repository, or the
|
|
57
|
+
persistence connection defeats the dimension. Then run the integration suite and capture
|
|
58
|
+
exit code + test count. Zero integration test files → FAIL at `critical`. Any integration
|
|
59
|
+
test that mocks the data layer → FAIL at `critical` (evidence: the mock call and the
|
|
60
|
+
module path it replaces).
|
|
59
61
|
source: code
|
|
60
62
|
|
|
61
63
|
- id: INT-2
|
|
62
|
-
statement: "An unauthorized-access scenario is covered: a request from
|
|
64
|
+
statement: "An unauthorized-access scenario is covered: a request from an unentitled caller is rejected with the documented outcome, exercised by an integration test."
|
|
63
65
|
probe: cmd + data
|
|
64
66
|
evidence_required: true
|
|
65
67
|
pass_rule: >
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
68
|
+
FIRST establish whether the feature has an access boundary at all: does the spec
|
|
69
|
+
(`usecases/` Error Cases, `contracts/`, the Non-Go list) document any caller who must NOT
|
|
70
|
+
reach this operation — a different tenant/user/org, a lower-privilege role, an
|
|
71
|
+
unauthenticated caller? If the spec documents none and the operation is genuinely public,
|
|
72
|
+
record INT-2 as N/A with that citation and exclude it from the denominator; do not invent
|
|
73
|
+
an authorization requirement the spec does not state. Otherwise, find the integration test
|
|
74
|
+
exercising the unauthorized path and assert it checks the outcome the spec documents
|
|
75
|
+
(a rejection status, an empty result set, a filtered response). If the spec also
|
|
76
|
+
distinguishes read from write for a role, verify the read-only role cannot mutate.
|
|
77
|
+
Evidence = the test case text + the run output showing it passes. A documented boundary
|
|
78
|
+
with no such test → FAIL at `critical` — this is the gap that lets real authorization
|
|
79
|
+
bugs slip through conformance testing.
|
|
80
|
+
source: usecases
|
|
74
81
|
|
|
75
82
|
- id: INT-3
|
|
76
|
-
statement: "Integration tests
|
|
83
|
+
statement: "Integration tests exercise the same enforcement path production uses — the test setup does not bypass or elevate past the mechanism that enforces the boundary."
|
|
77
84
|
probe: static
|
|
78
85
|
evidence_required: true
|
|
79
86
|
pass_rule: >
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
the
|
|
86
|
-
|
|
87
|
-
|
|
87
|
+
Applies only when INT-2 is in scope AND the project enforces the boundary in a layer a
|
|
88
|
+
test could bypass (database row-level policies, a middleware/guard chain, a tenant-scoped
|
|
89
|
+
client, a signed-context requirement). Identify that mechanism from the project profile,
|
|
90
|
+
the contracts, or the production wiring — then inspect the integration test setup
|
|
91
|
+
(suite-level hooks, test helpers, test env config) and confirm it goes through the same
|
|
92
|
+
path: the same credential/claim injection, the same connection or client mode, the same
|
|
93
|
+
guard chain. A setup that connects as an administrative or unscoped principal, disables
|
|
94
|
+
the guard layer, or seeds state through a back door calls the system with privileges the
|
|
95
|
+
real caller never has, and every access assertion above it is a false positive → FAIL at
|
|
96
|
+
`major`. If the project enforces the boundary only in code the test already traverses
|
|
97
|
+
(no bypassable layer exists), record INT-3 as N/A with that reasoning. Report each
|
|
98
|
+
distinct bypass as its own finding.
|
|
88
99
|
source: code
|
|
89
100
|
```
|
|
90
101
|
|
|
@@ -93,12 +104,16 @@ the e2e task introduces any auth-scoped endpoint.
|
|
|
93
104
|
## Threshold
|
|
94
105
|
|
|
95
106
|
`no-critical` — INT-1 and INT-2 are `critical`: a task that cannot prove end-to-end
|
|
96
|
-
correctness and cannot prove
|
|
97
|
-
not done. INT-3 (
|
|
98
|
-
|
|
99
|
-
which is
|
|
107
|
+
correctness, and cannot prove a documented authorization boundary, is not integration-tested
|
|
108
|
+
and is not done. INT-3 (enforcement-path integrity) is `major`: a bypassed enforcement layer
|
|
109
|
+
produces false-positive passes — the test appears to work while the boundary is never
|
|
110
|
+
exercised — which is confidence-undermining, but it is one step removed from observable
|
|
100
111
|
breakage and is reported as major rather than critical.
|
|
101
112
|
|
|
113
|
+
A criterion recorded N/A under its own rule (INT-2 with no documented boundary, INT-3 with
|
|
114
|
+
no bypassable layer) leaves the denominator, exactly like `tdd-surface`'s TDD-2 on `.e2e`.
|
|
115
|
+
N/A requires the citation that justifies it; "no auth here, probably" is not a citation.
|
|
116
|
+
|
|
102
117
|
---
|
|
103
118
|
|
|
104
119
|
## Bug template
|
|
@@ -106,24 +121,40 @@ breakage and is reported as major rather than critical.
|
|
|
106
121
|
```
|
|
107
122
|
severity: [critical|major]
|
|
108
123
|
criterion: [INT-1|INT-2|INT-3]
|
|
109
|
-
location: <test file path |
|
|
110
|
-
repro: <e.g.
|
|
111
|
-
expected: <e.g. "integration test
|
|
112
|
-
actual: <e.g. "no
|
|
113
|
-
fix_hint: <e.g. "add
|
|
124
|
+
location: <test file path | test setup/helper | test env config | file:line>
|
|
125
|
+
repro: <the command or search that shows it — e.g. the integration suite command, or the grep that finds the mock>
|
|
126
|
+
expected: <e.g. "integration test covering <flow> against the real database" | "test setup injects the caller's own credentials">
|
|
127
|
+
actual: <e.g. "no integration test found for <flow>" | "repository module is mocked in the suite" | "setup connects as the admin principal">
|
|
128
|
+
fix_hint: <e.g. "add <flow> integration test using the real test database; route setup through the same guard chain production uses">
|
|
114
129
|
```
|
|
115
130
|
|
|
116
131
|
## Probing notes
|
|
117
132
|
|
|
118
|
-
-
|
|
119
|
-
`test:
|
|
120
|
-
|
|
121
|
-
is
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
the
|
|
128
|
-
|
|
129
|
-
returns
|
|
133
|
+
- **Find the suite before judging it.** Integration targets are usually separate from unit
|
|
134
|
+
tests (`test:e2e`, `test:int`, `test:integration`, a tagged subset, a separate config).
|
|
135
|
+
Check every script the project defines, not just the default `test` — "no integration
|
|
136
|
+
tests" is a FAIL, but "I only ran the unit target" is a probing error, not a finding.
|
|
137
|
+
- **Mock detection is about the target, not the call.** Find the project's mocking primitive
|
|
138
|
+
(`jest.mock`/`vi.mock`, a DI container override, a fake registered in a test module,
|
|
139
|
+
a stubbed client factory) and inspect *what it replaces*. Outbound third parties: fine.
|
|
140
|
+
The data layer: fatal to the dimension.
|
|
141
|
+
- **Worked example — row-level policies with claim injection.** Where the database enforces
|
|
142
|
+
access (e.g. Postgres RLS), the enforcement is a per-connection claim set inside a
|
|
143
|
+
transaction; a test that queries outside that transaction runs as the application role and
|
|
144
|
+
silently returns rows the real caller could never see. INT-3 there means checking the setup
|
|
145
|
+
performs the claim injection, and that the connection mode preserves it — a session-mode
|
|
146
|
+
connection can leak transaction-scoped settings across pooled connections where a
|
|
147
|
+
transaction-mode one does not. Both are INT-3 findings, reported separately.
|
|
148
|
+
- **Worked example — guard/middleware chains.** Where a framework enforces access in a guard,
|
|
149
|
+
interceptor, or middleware, the bypass is a test that constructs the handler directly, or
|
|
150
|
+
registers a test module with the guard omitted. INT-3 means the test goes through the
|
|
151
|
+
composed application, not around it.
|
|
152
|
+
- **Worked example — tenant-scoped clients.** Where scoping is a client constructed per
|
|
153
|
+
caller, the bypass is a shared admin/service client in the test setup. INT-3 means the test
|
|
154
|
+
builds its client the way a request does.
|
|
155
|
+
- **For `.e2e` tasks**, a test driving the running server over HTTP counts as an integration
|
|
156
|
+
test if it reaches a real database. Verify the *test server's* configuration, not the
|
|
157
|
+
production server's — they routinely differ, and that difference is where INT-3 hides.
|
|
158
|
+
- **Do not skip INT-3 because the test "seems to work."** An enforcement layer that is never
|
|
159
|
+
invoked returns the same green as one that passes. Absence of a bypass must be evidenced,
|
|
160
|
+
not assumed.
|
|
@@ -26,16 +26,16 @@ dimensions, off by default.
|
|
|
26
26
|
- id: SC-AC
|
|
27
27
|
statement: "Every UC Step, Error Case, and (when present) Invariant/Test-Surface row for the
|
|
28
28
|
use case(s) this task/feature implements is satisfied."
|
|
29
|
-
probe: cmd # most are cmd; [ui]/[data] criteria use those probes per
|
|
29
|
+
probe: cmd # most are cmd; [ui]/[data] criteria use those probes per the CONTRACT-phase classification
|
|
30
30
|
evidence_required: true
|
|
31
31
|
pass_rule: >
|
|
32
32
|
Each criterion is drawn from usecases/UC-*.md (numbered ## Steps, ## Error Cases rows,
|
|
33
33
|
## Invariants [INV-NN] entries, ## Test Surface derived rows when present) for every UC
|
|
34
|
-
named in this task's use_case_refs (or confirmed with the user at
|
|
35
|
-
file exists). Run each via its classified probe and PASS. Zero failing criterion.
|
|
34
|
+
named in this task's use_case_refs (or confirmed with the user at CONTRACT when no local
|
|
35
|
+
task file exists). Run each via its classified probe and PASS. Zero failing criterion.
|
|
36
36
|
A stubbed/display-only implementation that satisfies a local task file's own AC wording but
|
|
37
37
|
not the UC's actual behavior FAILS — the task's AC (if present) is a paraphrase read only
|
|
38
|
-
for traceability
|
|
38
|
+
for traceability, never the grading text itself.
|
|
39
39
|
source: usecases
|
|
40
40
|
|
|
41
41
|
- id: SC-DONE-WHEN
|
|
@@ -21,7 +21,7 @@ requires_browser: true
|
|
|
21
21
|
> Explicit `--dimensions ...,test-surface-conformance` always wins over the auto rule.
|
|
22
22
|
|
|
23
23
|
**Why this exists.** `spec-conformance` grades the AC that were written; it does not
|
|
24
|
-
*expand* them. The
|
|
24
|
+
*expand* them. The settled division of labor locates the gap: nobody systematically derives
|
|
25
25
|
the test matrix — boundary values, negative cases, error-code coverage, no-go breaches —
|
|
26
26
|
that the spec already implies. The BA now derives that matrix into each UC's
|
|
27
27
|
`## Test Surface` (D1 Invariants · D2 Error Cases · D3 Contract shapes · D4 No-gos,
|
|
@@ -43,7 +43,7 @@ The probe loop per [ui] criterion:
|
|
|
43
43
|
the source file:line where the handler/wiring fails
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
-
## Oracle dispatch (evaluation contract
|
|
46
|
+
## Oracle dispatch (evaluation contract)
|
|
47
47
|
|
|
48
48
|
Each criterion / Test-Surface row carries an `oracle` tag — the dispatch key that declares *how*
|
|
49
49
|
this criterion is verified. **Dispatch on it to choose the probe mechanism; default to `ui` when
|
|
@@ -124,34 +124,53 @@ This is a static read; no running required. Record whether each key AC has a mat
|
|
|
124
124
|
|
|
125
125
|
## Integration probing (for `integration` dimension)
|
|
126
126
|
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
`
|
|
127
|
+
The commands below are shaped for a JS/TS monorepo because that is the most common case, not
|
|
128
|
+
because the dimension assumes one. Resolve the real test pattern and runner from the scope
|
|
129
|
+
contract's `e2e_verification_fixtures` and the project's own test scripts; the criteria in
|
|
130
|
+
`dimensions/integration.md` state the invariant each probe is collecting evidence for.
|
|
130
131
|
|
|
131
|
-
**
|
|
132
|
+
**Finding integration test files:** look for a naming convention that separates integration
|
|
133
|
+
from unit tests (e.g. `*.integration.spec.ts`, `*.e2e-spec.ts`, a top-level `test/`
|
|
134
|
+
directory) and inspect the project's script targets for an integration/e2e entry —
|
|
135
|
+
`test:e2e`, `test:int`, `test:integration`, a tagged subset, or a separate runner config.
|
|
136
|
+
Check every target the project defines: "I only ran the unit suite" is a probing error, not
|
|
137
|
+
a finding.
|
|
138
|
+
|
|
139
|
+
**INT-1 (no mocked data layer):** after locating the integration tests for the task's
|
|
140
|
+
feature, find the project's mocking primitive and inspect **what it replaces**:
|
|
132
141
|
```bash
|
|
133
|
-
grep -rn "jest\.mock\|vi\.mock" <integration-test-file>
|
|
142
|
+
grep -rn "jest\.mock\|vi\.mock" <integration-test-file> # adjust to the project's primitive
|
|
134
143
|
```
|
|
135
|
-
Mocking HTTP clients, email senders,
|
|
136
|
-
|
|
137
|
-
Then run the integration
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
**INT-2 (
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
144
|
+
Mocking outbound third parties (HTTP clients, email senders, payment sandboxes) is
|
|
145
|
+
acceptable. Mocking the database client, a repository class, or the persistence connection
|
|
146
|
+
defeats integration testing. Then run the integration target and capture exit code and test
|
|
147
|
+
count.
|
|
148
|
+
|
|
149
|
+
**INT-2 (access boundary):** first confirm from `usecases/` Error Cases, `contracts/`, or the
|
|
150
|
+
Non-Go list that the spec documents a caller who must not reach this operation. If it
|
|
151
|
+
documents none, record N/A with that citation. Otherwise locate the test case exercising the
|
|
152
|
+
unauthorized path and inspect its setup: it should present an unentitled caller (different
|
|
153
|
+
tenant/user/org, lower-privilege role, or no credentials) and assert the outcome the spec
|
|
154
|
+
documents. Capture the test body and the run output as evidence.
|
|
155
|
+
|
|
156
|
+
**INT-3 (enforcement-path integrity):** identify the layer that actually enforces the
|
|
157
|
+
boundary, then read the test setup — suite-level hooks, shared helpers, test env config — and
|
|
158
|
+
confirm it traverses that layer rather than around it. What to grep for depends on the
|
|
159
|
+
mechanism:
|
|
145
160
|
```bash
|
|
146
|
-
#
|
|
147
|
-
grep -rn "set_config\|
|
|
161
|
+
# database row-level policies: is the caller's claim set injected, and in the right scope?
|
|
162
|
+
grep -rn "set_config\|jwt\.claims\|SET LOCAL" <test-setup-files>
|
|
163
|
+
|
|
164
|
+
# connection mode: transaction-scoped settings can leak across pooled connections in
|
|
165
|
+
# session mode, so the configured port/mode is part of the evidence
|
|
166
|
+
grep -rn "DATABASE_URL\|DIRECT_URL" <test env config>
|
|
148
167
|
|
|
149
|
-
#
|
|
150
|
-
grep "
|
|
168
|
+
# guard / middleware chains: does the suite build the composed app, or the handler alone?
|
|
169
|
+
grep -rn "createTestingModule\|overrideGuard\|overrideProvider" <test-setup-files>
|
|
151
170
|
```
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
171
|
+
A setup that connects as an administrative or unscoped principal, omits the guard layer, or
|
|
172
|
+
seeds state through a back door calls the system with privileges the real caller never has —
|
|
173
|
+
every access assertion above it is a false positive. Report each distinct bypass separately.
|
|
155
174
|
|
|
156
175
|
## By task variant
|
|
157
176
|
- `.be` — no browser (`--browser none`). Start the API; send contract-shaped requests;
|
|
@@ -17,12 +17,12 @@ bug_count: [N]
|
|
|
17
17
|
browser_mode: cli | mcp | none
|
|
18
18
|
evaluator: spec-evaluator v0.1
|
|
19
19
|
eval_at: [ISO date]
|
|
20
|
-
linked_docs: ["[[usecases/UC-Name]]", "[[scope-summary]]"] # + "[[tasks/TASK-NNN-slug]]" only when a local task file exists this run
|
|
20
|
+
linked_docs: ["[[usecases/UC-Name]]", "[[scope-summary]]"] # + "[[tasks/TASK-NNN-slug]]" only when a local task file exists this run
|
|
21
21
|
t0_citation: ~ | { scope_id: cart-creation, artifact: t0/verdicts/r2-a3.json, sha256: 1a2b3c… }
|
|
22
22
|
---
|
|
23
23
|
```
|
|
24
24
|
`t0_citation` is `~` (not applicable) only when the spec folder has no scope contracts
|
|
25
|
-
(pre-v0.3.0 spec). On any scoped spec it is required —
|
|
25
|
+
(pre-v0.3.0 spec). On any scoped spec it is required — SKILL.md's T0-citation rule — an absent
|
|
26
26
|
citation there means the report should not have been written yet.
|
|
27
27
|
|
|
28
28
|
## Body sections (in order)
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
// Verdict-ledger reference implementation
|
|
2
|
+
// Verdict-ledger reference implementation — spec-evaluator's own skill-local
|
|
3
3
|
// reference impl (ships beside SKILL.md; not invoked at runtime).
|
|
4
4
|
//
|
|
5
5
|
// The `spec-evaluator` skill performs verdict re-probe / confidence / flip-detection as a
|
|
@@ -26,6 +26,7 @@ rely on (anything absent = **unknown**; never invent it):
|
|
|
26
26
|
| `substrate.allowed` / `substrate.shared` | The ONLY globs you may write. A needed file outside them → ESCALATE, never a write (a sandbox hook blocks it anyway) |
|
|
27
27
|
| `payload.decisions[]` | Adjudicated answers from prior escalations — binding precedent, apply them |
|
|
28
28
|
| `payload.digested_errors[]` | `{file, line, core_message}` triples from the previous attempt's failed verification — your starting bug list |
|
|
29
|
+
| `payload.trial_history[]` | Up to 8 prior attempts on this scope, oldest first, CROSSING the round boundary: `{score, status, delta, digest}`. `status: "reverted"` is a change that was tried and made things WORSE — do not re-propose it. `status: "kept"` with a still-red score is the tree you are building ON, not a failure to undo. Absent on the first attempt |
|
|
29
30
|
| `payload.verify.test_cmd` | The command that verifies your work. No test_cmd → command-verifiable ACs still need *some* observable check; say what you used |
|
|
30
31
|
| `payload.kb_rules_path` | Team guidelines (read if the file exists) — steering, never spec; conflict → the AC wins, note it in `deviations` |
|
|
31
32
|
| `payload.constraints` | Non-Go items and freezes (e.g. `ui_layers.layer3_frozen`) |
|
|
@@ -104,28 +105,41 @@ no recorded evidence is reported `fail`, by your own hand.
|
|
|
104
105
|
|
|
105
106
|
---
|
|
106
107
|
|
|
107
|
-
## ESCALATE protocol —
|
|
108
|
+
## ESCALATE protocol — how to stop without guessing
|
|
108
109
|
|
|
109
110
|
When blocked on a decision that is not yours (design decision, spec ambiguity, substrate
|
|
110
111
|
expansion): stop work on **that AC only**, keep building every AC that doesn't depend on the
|
|
111
|
-
answer,
|
|
112
|
+
answer, then return `status: "escalated"` with the blocker as the **first** entry in
|
|
113
|
+
`deviations[]`, in this shape (one entry per blocked decision, blockers first):
|
|
112
114
|
|
|
113
|
-
```json
|
|
114
|
-
{ "kind": "design-decision | spec-ambiguity | substrate-expansion",
|
|
115
|
-
"question": "one checkable question",
|
|
116
|
-
"blocked_ac": "which AC waits on this",
|
|
117
|
-
"context": "the two interpretations / the file and why it's needed" }
|
|
118
115
|
```
|
|
116
|
+
ESCALATE <design-decision | spec-ambiguity | substrate-expansion> [<blocked AC>]
|
|
117
|
+
<one checkable question>
|
|
118
|
+
<the two interpretations, or the file you need and why>
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
`deviations[]` is the channel because there is **no `escalates[]` field** — the WorkResult
|
|
122
|
+
envelope carries no structured escalation, so a question written anywhere else does not reach a
|
|
123
|
+
human at all. `status: "escalated"` is what the run notices: the phase leaves its artifact
|
|
124
|
+
unwritten, the orchestrator's post-condition fails, and the run stops and names the phase
|
|
125
|
+
instead of looping. Your `deviations[]` text is the only record of *why*, so make it answerable
|
|
126
|
+
on its own — whoever reads it will not have your context.
|
|
119
127
|
|
|
120
128
|
Orchestrated (`mode: orchestrated`): never ask an ad hoc question — there is no session to
|
|
121
|
-
answer it
|
|
122
|
-
|
|
123
|
-
questions) instead of escalating.
|
|
129
|
+
answer it. Standalone (`mode: standalone`): you may ask the user directly (max 2 questions)
|
|
130
|
+
instead of escalating.
|
|
124
131
|
|
|
125
132
|
---
|
|
126
133
|
|
|
127
134
|
## Output contract — the WorkResult
|
|
128
135
|
|
|
136
|
+
**Escalation rule.** If you return `status: "escalated"`, the **first** entry in `deviations[]`
|
|
137
|
+
must be the blocker: one specific, answerable question plus the context needed to answer it.
|
|
138
|
+
Nothing else in the envelope carries it — there is no `escalates[]` field — so a vague entry, or
|
|
139
|
+
the question buried under other notes, reaches the human as "something went wrong" and costs a
|
|
140
|
+
round. Write it so someone without your context can answer it in one reply.
|
|
141
|
+
|
|
142
|
+
|
|
129
143
|
Write `.shapeup/<slug>/results/<order-suffix>.json` (mirror of the order path; slug and
|
|
130
144
|
suffix come from `order_id`) matching `work-result.schema.json`, and print its path:
|
|
131
145
|
|
|
@@ -142,7 +156,6 @@ suffix come from `order_id`) matching `work-result.schema.json`, and print its p
|
|
|
142
156
|
"evidence": "pnpm --filter api test → 14/14" } ] }
|
|
143
157
|
],
|
|
144
158
|
"files_touched": [ { "path": "apps/api/src/orders/create.ts", "change": "created" } ],
|
|
145
|
-
"escalates": [],
|
|
146
159
|
"discoveries": [ { "marker": "+", "line": "empty-cart edge case unhandled" } ],
|
|
147
160
|
"assumptions": [ "amount stored in minor units — inferred from schema" ],
|
|
148
161
|
"deviations": []
|
|
@@ -167,7 +180,7 @@ orchestrator's `ingest-result.mjs` does all of that from your envelope.
|
|
|
167
180
|
- [ ] Every interactive element built binds to a manifest `test_id` with `data-state`
|
|
168
181
|
- [ ] No unused imports/symbols left behind by your changes
|
|
169
182
|
- [ ] Assumptions and deviations are in the envelope, not in your head
|
|
170
|
-
- [ ] Blocked ACs
|
|
183
|
+
- [ ] Blocked ACs are named in a `deviations[]` ESCALATE entry with `status: "escalated"`; unrelated ACs were still built
|
|
171
184
|
- [ ] The WorkResult validates against `work-result.schema.json` and its path was printed
|
|
172
185
|
|
|
173
186
|
---
|