qfai 1.10.0 → 1.10.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +95 -25
- package/assets/init/.qfai/assistant/README.md +27 -0
- package/assets/init/.qfai/assistant/agents/completion-reviewer.md +17 -1
- package/assets/init/.qfai/assistant/agents/delivery-planner.md +8 -0
- package/assets/init/.qfai/assistant/agents/implementation-reviewer.md +12 -1
- package/assets/init/.qfai/assistant/agents/orchestrator.md +2 -2
- package/assets/init/.qfai/assistant/agents/product-experience-architect.md +2 -1
- package/assets/init/.qfai/assistant/agents/product-surface-reviewer.md +1 -1
- package/assets/init/.qfai/assistant/agents/qa-gatekeeper.md +99 -14
- package/assets/init/.qfai/assistant/agents/test-design-analyst.md +4 -1
- package/assets/init/.qfai/assistant/catalog/cli-ux-guidelines.md +3 -0
- package/assets/init/.qfai/assistant/catalog/test-layers-ci-lanes.md +60 -0
- package/assets/init/.qfai/assistant/catalog/test-layers.md +169 -87
- package/assets/init/.qfai/assistant/catalog/worklog-entry.schema.md +4 -3
- package/assets/init/.qfai/assistant/constitution/agent-selection.md +7 -1
- package/assets/init/.qfai/assistant/constitution/communication.md +1 -1
- package/assets/init/.qfai/assistant/constitution/constitution.md +8 -2
- package/assets/init/.qfai/assistant/constitution/drift-protocol.md +177 -14
- package/assets/init/.qfai/assistant/constitution/review-convergence.md +121 -0
- package/assets/init/.qfai/assistant/constitution/shared-skill-delegation-baseline.md +205 -89
- package/assets/init/.qfai/assistant/constitution/shared-skill-operating-baseline.md +59 -5
- package/assets/init/.qfai/assistant/manifest/agent-catalog.yml +145 -21
- package/assets/init/.qfai/assistant/manifest/agent-routing.yml +69 -3
- package/assets/init/.qfai/assistant/skills/qfai-atdd/SKILL.md +131 -74
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/credential-reuse.md +146 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/red-provenance.md +498 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/review-fix-rounds.md +128 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/scaffolding.md +71 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/shared-test-artifacts.md +124 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/stale-manifest.md +34 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/test-case-depth-checklist.md +38 -1
- package/assets/init/.qfai/assistant/skills/qfai-configure/SKILL.md +4 -3
- package/assets/init/.qfai/assistant/skills/qfai-discussion/SKILL.md +10 -8
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/discussion-completion-matrix.md +23 -2
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/review-cycle-playbook.md +16 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/ui-bearing-playbook.md +47 -11
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/ui_ux/trend_scan_playbook.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/ui_ux_best_practices.md +6 -4
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/01_Context.md +2 -2
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/03_Story-Workshop.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/04_Sources.md +40 -6
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/14_Review-Request.md +7 -7
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/review/Rxx_reviewer.md +3 -3
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/review/review_request.md +4 -3
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/review/summary.json +3 -0
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/uiux/00_index.md +14 -5
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/uiux/40_screen_contracts.md +17 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/uiux/50_review_input_bundle.md +13 -8
- package/assets/init/.qfai/assistant/skills/qfai-implement/SKILL.md +83 -84
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/checkpoint-verification.md +339 -39
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/cross-spec-ownership.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/evidence-revision.md +348 -15
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/execution-ledger.md +203 -32
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/final-checklist.md +159 -12
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/finding-classification.md +70 -5
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/ledger-preconditions.md +55 -14
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/parallelization-policy.md +92 -3
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/red-not-observable.md +48 -6
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/relevant-test-suite.md +13 -1
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/review-artifact-layout.md +71 -9
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/round-evidence.md +31 -7
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/upstream-artifact-ordering.md +33 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/volume-policy.md +19 -12
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/SKILL.md +76 -8
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/design-md-spec.md +20 -0
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/evidence-requirements.md +7 -4
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/generator-prompt.md +83 -22
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/reviewer-prompt.md +29 -4
- package/assets/init/.qfai/assistant/skills/qfai-sdd/SKILL.md +134 -20
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/contract-artifact-rules.md +31 -3
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/review-cycle-playbook.md +22 -1
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/sdd-execution-playbook.md +41 -5
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/sdd-phase-checklists.md +16 -4
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/sdd-triage.md +77 -7
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/spec-traceability-rules.md +60 -11
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/ui-design-contract-normalization.md +9 -3
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/change-request.md +14 -1
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/contracts/ui-contract.sample.yaml +13 -3
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/evidence/sdd-spec.md +5 -1
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/report/preflight_summary.md +3 -2
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/04_Business-Flow.md +4 -1
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/05_Contracts.md +12 -5
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/11_Slice-Policy.md +8 -9
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/02_User-stories.md +9 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/tdd/test-list.md +68 -23
- package/assets/init/.qfai/assistant/skills/qfai-verify/SKILL.md +4 -4
- package/assets/init/.qfai/assistant/skills/qfai-verify/references/articles.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-verify/references/validate-json-schema.md +79 -0
- package/assets/init/.qfai/assistant/skills/web-research/SKILL.md +3 -2
- package/assets/init/root/.github/workflows/qfai-tests.yml +318 -0
- package/assets/init/root/.github/workflows/qfai-validate.yml +327 -24
- package/assets/init/root/qfai.config.yaml +11 -2
- package/dist/cli/index.cjs +25083 -11488
- package/dist/cli/index.cjs.map +1 -1
- package/dist/cli/index.mjs +25216 -11589
- package/dist/cli/index.mjs.map +1 -1
- package/dist/index.cjs +16731 -8386
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +407 -17
- package/dist/index.d.ts +407 -17
- package/dist/index.mjs +16701 -8370
- package/dist/index.mjs.map +1 -1
- package/package.json +6 -1
|
@@ -53,6 +53,7 @@ When unsure, read inputs in this order:
|
|
|
53
53
|
- `.qfai/specs/<spec-id>/03_Acceptance-Criteria.md` (AC)
|
|
54
54
|
- `.qfai/specs/<spec-id>/05_Examples.md` (EX)
|
|
55
55
|
- `.qfai/specs/<spec-id>/06_Test-Cases.md` (TC)
|
|
56
|
+
- `.qfai/specs/<spec-id>/tdd/test-list.md` (the execution ledger — enumerate the `Layer = E2E` / `Layer = API` / `Layer = Integration` rows this run owes evidence for, with their `TDD-ID`, obligation column and `Selector`)
|
|
56
57
|
- `.qfai/contracts/api/**` (CON-API)
|
|
57
58
|
- `.qfai/contracts/db/**` (CON-DB)
|
|
58
59
|
- `.qfai/contracts/ui/**` and `.qfai/contracts/design/**` when the target spec is UI-bearing
|
|
@@ -70,6 +71,7 @@ Do not read discussion-pack UI/UX sidecars. UI-bearing acceptance tests consume
|
|
|
70
71
|
- `.qfai/specs/<spec-id>/03_Acceptance-Criteria.md`
|
|
71
72
|
- `.qfai/specs/<spec-id>/05_Examples.md`
|
|
72
73
|
- `.qfai/specs/<spec-id>/06_Test-Cases.md`
|
|
74
|
+
- `.qfai/specs/<spec-id>/tdd/test-list.md` — read, never written. A seeded row's `Test file` and `Selector` are still `-` there, because Phase 2b seeds them before any test exists: record the path and selector of the test **this run authored** as the handoff entry's row identity rather than copying that placeholder, since this stage is where they first exist and `/qfai-implement` Phase Red step 3b is the step that writes them into the ledger. A run that does not enumerate its `Layer = E2E` / `Layer = API` / `Layer = Integration` rows produces no `## Ledger rows advanced` entry for them, and `/qfai-implement` Phase Red step 3b then stops on a missing handoff.
|
|
73
75
|
- Escalation Mode:
|
|
74
76
|
- allowed only when `01_Spec.md` Escalation Hook signals ambiguity / conflict / missing constraint / trade-off
|
|
75
77
|
- read only `.qfai/specs/_policies/01_Objective.md` and `.qfai/specs/_policies/08_Decisions.md`
|
|
@@ -112,9 +114,9 @@ Use the shared schema.
|
|
|
112
114
|
- Follow `.qfai/assistant/constitution/shared-skill-delegation-baseline.md#reviewer-gate-baseline`.
|
|
113
115
|
- Final completion gate MUST be delegated to an independent `completion-reviewer`.
|
|
114
116
|
- ATDD-specific reviewer checks:
|
|
115
|
-
- coverage obligations met: E2E covers `US`, API covers `CON-API`, and every `TC` is covered from the directory
|
|
117
|
+
- coverage obligations met: E2E covers `US`, API covers `CON-API`, and every `TC` **that declares `L3`/`L4`/`L5` or no `Level`** is covered from the directory that `Level` routes to. `L1`/`Unit` and `L2`/`Component` owe nothing here (CRITICAL CONSTRAINTS): the ledger covers them. An existing L1/L2 annotation in `tests/integration/**` is not a violation — the validator declines to count it and declines to flag it — so do not require one to be added, and do not require an existing one to be removed;
|
|
116
118
|
- Coverage Depth Matrix is reviewed and no unjustified `X` cells remain;
|
|
117
|
-
- validation evidence exists and `npx qfai validate --profile atdd --fail-on error
|
|
119
|
+
- validation evidence exists and `npx qfai validate --profile atdd --fail-on error --spec <spec-id>` passes;
|
|
118
120
|
- Drift Protocol is enforced;
|
|
119
121
|
- test-layer policy is checked against `.qfai/assistant/catalog/test-layers.md`;
|
|
120
122
|
- coverage floors and ratios are signals, not gates;
|
|
@@ -149,28 +151,44 @@ Follow `.qfai/assistant/constitution/shared-skill-operating-baseline.md#delta-re
|
|
|
149
151
|
- Do NOT declare completion based on unit/component tests.
|
|
150
152
|
- `10_Plan.md` is the primary How SSOT for execution phases.
|
|
151
153
|
- If `10_Plan.md` is missing, stop and run owner planning flow before proceeding.
|
|
152
|
-
- Completion gate is validation with zero errors
|
|
154
|
+
- Completion gate is validation with zero errors **for this spec**: `npx qfai validate --profile atdd --fail-on error --spec <spec-id>`. The scope flag is not optional bookkeeping. This skill runs one spec per invocation, and unscoped it reports every other spec's `QFAI-ATDD-111` / `-112` obligations — findings this run cannot act on and must not be blocked by. A `--spec` run also writes `<report>/validate.spec-<id>.json` rather than the shared `validate.json`, so the JSON gate artifact is per spec, and an unknown or unparseable value fails the run (`QFAI-SCOPE-001` / `QFAI-SCOPE-002`) instead of silently widening back to the whole repository. **That is not the same as being parallel-safe.** `<report>/validate.log` and the run-log pointer are shared by every run, scoped or not, and nothing serializes them — so two stages running at once can leave that pointer naming the other one's run. Cite the per-run `<report>/run-*/` directory, or this spec's `validate.spec-<id>.json`, as the Validate Hard Gate evidence; do not cite `validate.log` from a run you shared with another stage.
|
|
155
|
+
- **`--spec` scopes the spec-owned rules only, and the gate still fails on the rest.** Every rule whose finding names a spec is scoped: `QFAI-ATDD-111` (US) and `QFAI-ATDD-112` (TC) by the specs they name, `QFAI-ATDD-101` / `-102` by the spec in the unknown token, `QFAI-ATDD-121` / `-122` / `-123` by the specs whose TCs are misplaced, and `D-SCAFFOLD-PLACEHOLDER` by the spec its skeleton belongs to. A scoped run reports all of those for the requested spec and drops a sibling's. What cannot be scoped **does** fail a scoped gate:
|
|
156
|
+
- `QFAI-ATDD-113` (`CON-API`) and `QFAI-ATDD-115` (`CON-DB`) are attributed to `.qfai/contracts/**`, which has no spec owner in the model;
|
|
157
|
+
- a reference, or a scaffold directory, naming a spec number no spec pack has, **and sitting where no spec owns it either**: `--spec` on that number is itself rejected, so nothing would report it otherwise. A file under the canonical `tests/<layer>/spec-NNNN/**` layout is owned by _that_ spec whatever its annotation says, so the same broken reference inside `tests/integration/spec-0002/**` belongs to `--spec 0002` and to no other run — reading it as a repo-wide blocker in a sibling run reports something that run cannot see;
|
|
158
|
+
- anything else reported against a repo-level path.
|
|
159
|
+
|
|
160
|
+
So a sibling spec's uncovered contract exits 1 on this spec's gate. That is a real limit, not a formality. When it happens: record the finding, its owning spec and why it is not this stage's work as a cross-spec obligation in this stage's evidence, and say so in the completion report — do **not** claim the gate passed, weaken the profile, lower `--fail-on`, or waive it. Closing them is the owning spec's next `/qfai-atdd` run. The repo-wide run belongs to `/qfai-verify`, at the end of the stage.
|
|
161
|
+
|
|
153
162
|
- Coverage obligations are mandatory:
|
|
154
|
-
- `tests/e2e/**` must cover all required `US-*`.
|
|
163
|
+
- `tests/e2e/**` must cover all required `US-*`. A story outside the current slice is deferred in `02_User-stories.md` with a `- x-qfai-status: planned` meta line in its own `US-XXXX` block (a `##`-or-deeper heading, or its catalog list entry) — the same token both contract kinds use — and is named at `info` by `QFAI-ATDD-118`. It is not left uncovered, and it is not covered by a test that asserts nothing. `exception` is not the alternative here: that branch belongs to a ledger row, and a `US-*` owns none (`references/red-provenance.md#a-spec-with-no-atdd-owned-rows`).
|
|
155
164
|
- Every `TC-*` must be covered from the directory its declared `Level` routes
|
|
156
165
|
to: `L3`/`Integration` -> `tests/integration/**`, `L4`/`API` ->
|
|
157
166
|
`tests/api/**`, `L5`/`E2E` -> `tests/e2e/**`. A TC with no declared `Level`
|
|
158
167
|
routes to `tests/integration/**`.
|
|
159
|
-
- `
|
|
168
|
+
- **`L1`/`Unit` and `L2`/`Component` owe nothing here** — out of this skill's
|
|
169
|
+
scope, excluded from `QFAI-ATDD-112`, gated by `tdd/test-list.md` under
|
|
170
|
+
`/qfai-implement`, and named on every run by `QFAI-ATDD-117` (`info`). Do
|
|
171
|
+
not duplicate an L1/L2 annotation into `tests/integration/**` to quiet a
|
|
172
|
+
gate: that is the all-integration collapse `catalog/test-layers.md` lists as
|
|
173
|
+
an anti-pattern.
|
|
174
|
+
- `tests/api/**` must cover all required `CON-API-*`. **An ID carried by a file that declares no test is not coverage**: the scan reads markdown too, and a `.test.ts` holding only the annotation is the same ledger renamed. `QFAI-ATDD-119` (`info`) names them, and coverage is `missing` **and** `coveredByCarrierOnly` in `summary.json`, never `missing` alone. This skill runs one spec, so gate on the narrowed `QFAI-ATDD-119` in `<report>/validate.spec-<id>.json`: `summary.json` stays repo-wide under every scope, and a sibling spec's placeholder would keep its arrays non-empty forever.
|
|
160
175
|
- Forbidden references (a TC annotation outside its declared home):
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
- `tests/e2e/**` must not contain `QFAI:SPEC-XXXX:TC-YYYY` unless that TC
|
|
164
|
-
declares `Level` `L5`/`E2E`.
|
|
176
|
+
`tests/api/**` and `tests/e2e/**` must not contain `QFAI:SPEC-XXXX:TC-YYYY`
|
|
177
|
+
unless that TC declares `Level` `L4`/`API` or `L5`/`E2E` respectively.
|
|
165
178
|
- Unknown references (`US/TC/CON-API` not declared) must be treated as errors.
|
|
179
|
+
- **The E2E/API ledger rows this stage feeds are bound by `/qfai-implement`'s lifecycle.** See "Execution Ledger" below: a row advanced on none of the three RED-provenance forms is a lifecycle violation.
|
|
166
180
|
- Floors/ratios are planning signals only, not gates.
|
|
167
181
|
- Legacy `scenario.feature` or coverage ledgers may exist but are not mandatory inputs for completion.
|
|
168
|
-
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
182
|
+
- The per-item evidence file `.qfai/evidence/atdd-<spec-id>.md` is required and
|
|
183
|
+
committed. Ledger `Evidence` cells point to its anchors, and validation must
|
|
184
|
+
resolve them on a fresh clone. The managed `.gitignore` block re-includes it,
|
|
185
|
+
alongside `.qfai/evidence/implement-<spec-id>.md`. **Governance records also
|
|
186
|
+
stay in version control**: Change Requests (`.qfai/decisions/CR-*.md`),
|
|
187
|
+
durable decision records (`.qfai/evidence/decisions/*.json`) and the
|
|
188
|
+
**Coverage Depth Matrix** (`.qfai/evidence/coverage-depth-<spec-id>.md`).
|
|
189
|
+
- **The matrix is a governance record, not a log**, so it is committed:
|
|
190
|
+
`.qfai/evidence/coverage-depth-<spec-id>.md`, one justification per `❌`
|
|
191
|
+
(`references/test-case-depth-checklist.md#where-the-matrix-lives`).
|
|
174
192
|
|
|
175
193
|
## Completion Contract (Shared)
|
|
176
194
|
|
|
@@ -181,10 +199,36 @@ Follow `.qfai/assistant/constitution/shared-skill-operating-baseline.md#gate-fai
|
|
|
181
199
|
|
|
182
200
|
Turn specs/contracts obligations (`US` / `TC` / `CON-API` / `CON-DB`) into runnable acceptance tests in this repository.
|
|
183
201
|
|
|
202
|
+
## Execution Ledger: the rows this skill feeds
|
|
203
|
+
|
|
204
|
+
`.qfai/specs/<spec-id>/tdd/test-list.md` is `/qfai-implement`'s execution
|
|
205
|
+
ledger, and `qfai-implement/SKILL.md` states the split: **`Layer = E2E`,
|
|
206
|
+
`Layer = API` and `Layer = Integration` rows are tracked there, but their tests
|
|
207
|
+
are authored here.** Integration is there because this skill's scope puts it
|
|
208
|
+
there: `QFAI-ATDD-112` covers every `L3` TC, and every TC with no declared
|
|
209
|
+
`Level`, from `tests/integration/**`, and P4 writes those tests. Self-owned,
|
|
210
|
+
they had `/qfai-implement` demand a fresh RED for a test already green here.
|
|
211
|
+
|
|
212
|
+
- **This skill does not write the ledger.** `/qfai-implement` owns the `Status` / `DR-ID` / `Evidence` cells of every row — one writer, as `constitution/drift-protocol.md` grants. This stage owes the **evidence those cells point at**, in `.qfai/evidence/atdd-<spec-id>.md`.
|
|
213
|
+
- **The lifecycle is `../qfai-implement/references/execution-ledger.md#allowed-transitions`**: forward-only from `todo`, and `todo -> red` requires an **admissible RED** observed before the code that makes it pass exists.
|
|
214
|
+
- **`/qfai-sdd` Phase 2b seeds one `Layer = E2E` row per active `US-*` and one `Layer = API` row per active `CON-API-*` the spec owns — the lowest-numbered spec naming that contract; this stage still cannot create them.** A spec with an active obligation **of its own** therefore normally arrives with rows here — enumerate them and build the handoff from them. Zero is legitimate when every obligation is exempt **and** every active `CON-API-*` the spec references is owned by another spec, and even then is not "nothing to do"; a row missing for an **active** obligation this spec owns is an incomplete Phase 2b — report it, never write it. Do not demand an API row for a contract another spec owns: that row must not exist twice. `references/red-provenance.md#a-spec-with-no-atdd-owned-rows`.
|
|
215
|
+
- **The stage order makes that a real question**: Work Orders build the surfaces a journey needs (P3, P4), so a journey written after them passes first run — an anomaly bound for `exception`, which then becomes the only reachable terminal state.
|
|
216
|
+
|
|
217
|
+
### RED provenance for an ATDD-owned row (MUST)
|
|
218
|
+
|
|
219
|
+
**Read `references/red-provenance.md` before advancing any row.** It defines
|
|
220
|
+
three branches, in order — observed RED (stage gate **P1b**, before P2-P4 build
|
|
221
|
+
any surface), falsifiability when the surface is already there, and `exception`
|
|
222
|
+
with a `DR-*` when neither is available. Take the first that applies, record
|
|
223
|
+
which one, and put the evidence in `.qfai/evidence/atdd-<spec-id>.md` under
|
|
224
|
+
`## Ledger rows advanced` — `/qfai-implement` consumes that entry rather than
|
|
225
|
+
re-observing a RED. Branch 3 is the last resort: a stage that routes every row
|
|
226
|
+
to `exception` has recorded that it did not try branches 1 and 2.
|
|
227
|
+
|
|
184
228
|
## Scope (ATDD only)
|
|
185
229
|
|
|
186
|
-
|
|
187
|
-
-
|
|
230
|
+
In scope: E2E, API, Integration. Out of scope: Unit and Component
|
|
231
|
+
(`/qfai-implement`). Tests that must sign in: `references/credential-reuse.md`.
|
|
188
232
|
|
|
189
233
|
## Non-goals
|
|
190
234
|
|
|
@@ -194,18 +238,16 @@ Turn specs/contracts obligations (`US` / `TC` / `CON-API` / `CON-DB`) into runna
|
|
|
194
238
|
## Mandatory Outputs
|
|
195
239
|
|
|
196
240
|
1. Test Volume Estimate (signal table with evidence)
|
|
197
|
-
2. **Coverage Depth Matrix
|
|
198
|
-
3. Coverage obligations checklist (`US` / `TC` / `CON-API`)
|
|
199
|
-
4.
|
|
200
|
-
5.
|
|
201
|
-
6. Evidence file: `.qfai/evidence/atdd-<spec-id>.md`
|
|
241
|
+
2. **Coverage Depth Matrix**, written to `.qfai/evidence/coverage-depth-<spec-id>.md` (per spec; template and scoring in `references/test-case-depth-checklist.md`). Committed — see CRITICAL CONSTRAINTS.
|
|
242
|
+
3. Coverage obligations checklist (`US` / `TC` / `CON-API`), and the implemented tests per layer (E2E/API/Integration)
|
|
243
|
+
4. Reviewer notes (`PASS` or concrete rework list)
|
|
244
|
+
5. Evidence file: `.qfai/evidence/atdd-<spec-id>.md`
|
|
202
245
|
|
|
203
246
|
## Volume Signals (mandatory, not gates)
|
|
204
247
|
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
- When signals are low/high, propose options and recommendation; do not fail solely on signal values.
|
|
248
|
+
E2E = required `US-*`, API = declared `CON-API-*`, Integration = required `TC-*`.
|
|
249
|
+
When a signal is low or high, propose options and a recommendation; never fail
|
|
250
|
+
on a signal value alone.
|
|
209
251
|
|
|
210
252
|
### Estimator output table (required)
|
|
211
253
|
|
|
@@ -217,55 +259,43 @@ Turn specs/contracts obligations (`US` / `TC` / `CON-API` / `CON-DB`) into runna
|
|
|
217
259
|
|
|
218
260
|
## Scaffolding
|
|
219
261
|
|
|
220
|
-
`npx qfai atdd scaffold --spec <spec-id>` bulk-emits one placeholder test per
|
|
221
|
-
coverage-target `TC-*`, each carrying its `QFAI:SPEC-XXXX:TC-YYYY` annotation.
|
|
222
|
-
Skeletons land in `tests/integration/<spec-id>/` — the directory
|
|
223
|
-
`QFAI-ATDD-112` scans — so a filled-in skeleton counts as coverage. It is
|
|
224
|
-
idempotent: existing files are left untouched.
|
|
225
|
-
|
|
226
|
-
A skeleton left in placeholder shape across repeated validate runs escalates
|
|
227
|
-
(`qfai.config.yaml#atdd.scaffoldEscalateCycles`), so scaffolding is a start, not
|
|
228
|
-
a discharge of the obligation.
|
|
262
|
+
`npx qfai atdd scaffold --spec <spec-id>` bulk-emits one placeholder test per `TC-*` **this skill owns**, each carrying its `QFAI:SPEC-XXXX:TC-YYYY` annotation, into `tests/integration/<spec-id>/` — the directory `QFAI-ATDD-112` scans. It is idempotent. `L1`/`L2` and `L4`/`L5` TCs are skipped and named on stderr, and a skeleton left in placeholder shape escalates: `references/scaffolding.md`.
|
|
229
263
|
|
|
230
264
|
## Annotation obligations (mandatory)
|
|
231
265
|
|
|
232
266
|
Every generated ATDD test MUST include QFAI annotations by layer:
|
|
233
267
|
|
|
234
|
-
- `tests/e2e/**`: `QFAI:SPEC-XXXX:US-YYYY` (plus `QFAI:SPEC-XXXX:TC-YYYY` for a
|
|
235
|
-
|
|
236
|
-
- `tests/
|
|
237
|
-
`Integration`, and TCs with no declared `Level`)
|
|
238
|
-
- `tests/api/**`: `QFAI:CON-API-XXXX` (plus `QFAI:SPEC-XXXX:TC-YYYY` for a TC
|
|
239
|
-
that declares `Level` `L4`/`API`)
|
|
268
|
+
- `tests/e2e/**`: `QFAI:SPEC-XXXX:US-YYYY` (plus `QFAI:SPEC-XXXX:TC-YYYY` for a TC that declares `Level` `L5`/`E2E`)
|
|
269
|
+
- `tests/integration/**`: `QFAI:SPEC-XXXX:TC-YYYY` (TCs declaring `L3`/`Integration`, and TCs with no declared `Level`)
|
|
270
|
+
- `tests/api/**`: `QFAI:CON-API-XXXX` (plus `QFAI:SPEC-XXXX:TC-YYYY` for a TC that declares `Level` `L4`/`API`)
|
|
240
271
|
- `tests/integration/**` also carries `QFAI:CON-DB-XXXX` for every declared DB
|
|
241
272
|
contract the slice exercises
|
|
242
273
|
|
|
243
274
|
Notes:
|
|
244
275
|
|
|
245
|
-
- A TC's annotation belongs in exactly one directory — the one its declared
|
|
246
|
-
`Level` routes to. Placing it elsewhere is both uncovered (`QFAI-ATDD-112`)
|
|
247
|
-
and forbidden (`QFAI-ATDD-121` / `QFAI-ATDD-122` / `QFAI-ATDD-123`). The rule
|
|
248
|
-
is symmetric: an annotation left behind in `tests/integration/**` after the
|
|
249
|
-
TC moved to `L4`/`L5` is rejected by `QFAI-ATDD-123` just as an early one in
|
|
250
|
-
`tests/api/**` is rejected by `QFAI-ATDD-121`.
|
|
276
|
+
- A TC's annotation belongs in exactly one directory — the one its declared `Level` routes to. Elsewhere it is both uncovered (`QFAI-ATDD-112`) and forbidden, symmetrically: one left behind in `tests/integration/**` after the TC moved to `L4`/`L5` is rejected by `QFAI-ATDD-123` just as an early one in `tests/api/**` is rejected by `QFAI-ATDD-121`.
|
|
251
277
|
- AC annotations are optional in code.
|
|
252
278
|
- `QFAI:CON-API-*` in E2E is not forbidden, but contract guarantee belongs to API tests.
|
|
253
279
|
|
|
254
280
|
## Success Criteria (Definition of Done)
|
|
255
281
|
|
|
256
|
-
- All required `US` are covered by E2E tests.
|
|
282
|
+
- All required `US` are covered by E2E tests (`QFAI-ATDD-111`); a story outside the current slice is deferred with `- x-qfai-status: planned` in its `02_User-stories.md` block and reported by `QFAI-ATDD-118` (`info`), not left uncovered.
|
|
257
283
|
- All required `TC` are covered from the directory their declared `Level` routes
|
|
258
284
|
to (`L3`/`Integration` -> `tests/integration/**`, `L4`/`API` ->
|
|
259
285
|
`tests/api/**`, `L5`/`E2E` -> `tests/e2e/**`, no declared `Level` ->
|
|
260
|
-
`tests/integration/**`).
|
|
261
|
-
|
|
286
|
+
`tests/integration/**`). **`L1`/`Unit` and `L2`/`Component` are outside this
|
|
287
|
+
obligation** — the ledger covers them — so a spec whose TCs are all L1/L2 is
|
|
288
|
+
done here with no ATDD annotation at all. Duplicating a TC into a second layer
|
|
289
|
+
is a not-done condition, not extra credit.
|
|
262
290
|
- All required `CON-API` are covered by API tests.
|
|
263
291
|
- All required `CON-DB` are covered by integration tests (`QFAI-ATDD-115`); a contract
|
|
264
292
|
outside the current slice is deferred with `-- x-qfai-status: planned`, not left uncovered.
|
|
265
|
-
- Validation passes: `npx qfai validate --profile atdd --fail-on error
|
|
293
|
+
- Validation passes for this spec: `npx qfai validate --profile atdd --fail-on error --spec <spec-id>`.
|
|
266
294
|
- Repository quality gates (format/lint/type/tests/pack) pass with evidence.
|
|
267
295
|
- Evidence file exists and includes work orders + reviewer notes.
|
|
296
|
+
- Every ledger row this cycle advanced carries one of the three RED-provenance forms — an observed RED pair with its `Oracle proof`, the `Satisfied-by` + falsifiability trio, or a `DR-*` recording why neither was available — and `qa-gatekeeper` has accepted it. The third form is a valid _branch_, and it is **not a completion**: `exception` is a blocking output and needs a user-approved `TDDLIST-001` waiver, or the row is parked and the spec stays open (`references/red-provenance.md#branch-3-does-not-close-a-spec-on-its-own`).
|
|
268
297
|
- Completion is approved by a reviewer who did not implement tests.
|
|
298
|
+
- **The P8 reviewer's `Audited evidence hash` is recomputed before completion is declared**, from the current stage evidence file and Coverage Depth Matrix, by the stage-review procedure that produced it (`.qfai/assistant/constitution/shared-skill-delegation-baseline.md#reviewer-response-template`); a mismatch means the evidence moved after the verdict. **Seal the P8 pack too**: when the last reviewer response lands, and before this stage writes its verdict, hash the pack this stage opened — `.qfai/review/review-<timestamp>/`, whole — by the same procedure, and record it **outside the pack** in the stage evidence file's `## Final status` as `Review pack:` (that path) and `Review pack seal:` (that hash). That section is the one part excluded from the P8 audit subject, so writing it there does not stale the verdict, and it exists even on a spec with no ATDD-owned rows — where there is no item evidence entry to hold the seal at all. At completion, recompute the seal over the recorded path and compare it with the **recorded** value — `../qfai-implement/references/evidence-revision.md` states that rule once and it applies here: `## Final status` is outside every audit subject and outside the working-tree revision, so an expected value read from the working tree could be rewritten in the same pass that edited the pack, and every recomputation would still agree, and check that `## Final status` says what that pack says. The recording and the recomputation must be two moments: a value computed from the pack at completion always matches itself whatever was edited in between, and the stage hash covers the evidence but not the verdict, so a `REVISE` edited to `PASS` in the response, the summary and the status together left every recomputation unchanged. On a spec with no ATDD-owned rows `/qfai-implement`'s gate item 10 never runs, so without this the stored hash was written by P8 and read by nobody — and the evidence tree is out of the working-tree revision, so a later edit moved nothing else either.
|
|
269
299
|
|
|
270
300
|
## Not-done criteria
|
|
271
301
|
|
|
@@ -274,6 +304,8 @@ Notes:
|
|
|
274
304
|
- Tests exist but were never executed.
|
|
275
305
|
- Validation evidence is missing or failing.
|
|
276
306
|
- Coverage Depth Matrix is missing or contains unjustified ❌ cells (normal-path-only coverage is incomplete).
|
|
307
|
+
- A ledger row was advanced past `todo` with none of the three forms — no observed RED, no falsifiability evidence, and no `DR-*`.
|
|
308
|
+
- A row was sent to `exception` without a `DR-*` recording why **both** branches were unavailable. "The surface was built earlier in this cycle" is not such a reason.
|
|
277
309
|
|
|
278
310
|
## Failure handling (mandatory)
|
|
279
311
|
|
|
@@ -284,19 +316,17 @@ Notes:
|
|
|
284
316
|
|
|
285
317
|
Create and update: `.qfai/evidence/atdd-<spec-id>.md`
|
|
286
318
|
|
|
287
|
-
Required sections:
|
|
319
|
+
Required sections: the template below is the list. Two of them carry a contract
|
|
320
|
+
the heading cannot:
|
|
288
321
|
|
|
289
|
-
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
-
|
|
293
|
-
-
|
|
294
|
-
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
- Execution logs
|
|
298
|
-
- Gaps / Open risks
|
|
299
|
-
- Final status (PASS/FAIL) + who confirmed
|
|
322
|
+
- **Ledger rows advanced** — an index table plus one `### TDD-NNNN` section per
|
|
323
|
+
row. Exactly one form per row, never both and never neither; the cell is an
|
|
324
|
+
anchor and the payload goes in the section
|
|
325
|
+
(`references/red-provenance.md#evidence-shape`).
|
|
326
|
+
- **Coverage Depth Matrix** — a link to
|
|
327
|
+
`.qfai/evidence/coverage-depth-<spec-id>.md` and the `✅`/`⚠️`/`❌` totals.
|
|
328
|
+
The matrix and its per-`❌` justifications live in that committed file;
|
|
329
|
+
restating them here would lose them.
|
|
300
330
|
|
|
301
331
|
Template:
|
|
302
332
|
|
|
@@ -317,6 +347,15 @@ Template:
|
|
|
317
347
|
|
|
318
348
|
## Coverage obligations checklist
|
|
319
349
|
|
|
350
|
+
## Ledger rows advanced
|
|
351
|
+
|
|
352
|
+
<!-- Index table + one `### TDD-NNNN` section per row:
|
|
353
|
+
`references/red-provenance.md#evidence-shape`. -->
|
|
354
|
+
|
|
355
|
+
## Coverage Depth Matrix
|
|
356
|
+
|
|
357
|
+
See `.qfai/evidence/coverage-depth-<spec-id>.md` (committed). Totals: ✅ N / ⚠️ N / ❌ N.
|
|
358
|
+
|
|
320
359
|
## Work Orders Summary
|
|
321
360
|
|
|
322
361
|
## Execution logs
|
|
@@ -330,9 +369,7 @@ Template:
|
|
|
330
369
|
|
|
331
370
|
- **Test Case Depth Analyst**: `test-design-analyst` evaluates test cases using `references/test-case-depth-checklist.md`, produces Coverage Depth Matrix, flags gaps in boundary/error/edge coverage.
|
|
332
371
|
- Test Volume Estimator: compute US/TC/CON signals with evidence.
|
|
333
|
-
- ATDD
|
|
334
|
-
- ATDD API Implementer: implement required `CON-API` coverage.
|
|
335
|
-
- ATDD Integration Implementer: implement required `TC` coverage.
|
|
372
|
+
- ATDD Implementers, one per layer: required `US` coverage in E2E, `CON-API` in API, `TC` in Integration.
|
|
336
373
|
- Reviewer: validate coverage obligations + gate results + Coverage Depth Matrix (non-edit).
|
|
337
374
|
- Runtime Gatekeeper: run suites and capture logs.
|
|
338
375
|
|
|
@@ -343,11 +380,29 @@ Template:
|
|
|
343
380
|
|
|
344
381
|
## Stage Gates (Do not skip)
|
|
345
382
|
|
|
346
|
-
- P0: Plan and obligations checklist prepared.
|
|
383
|
+
- P0: Plan and obligations checklist prepared. A project whose routing has no
|
|
384
|
+
`red` phase predates it: `references/red-provenance.md#a-project-without-the-red-phase`.
|
|
347
385
|
- P1: Layer assignment validated against `.qfai/assistant/catalog/test-layers.md#layer-derivation-procedure-normative`.
|
|
386
|
+
- P1b: **A branch is chosen for every row**, provisional until its handoff.
|
|
387
|
+
- P1c: **A branch 1 row is discharged in that loop** — write the test, take the
|
|
388
|
+
RED, `qa-gatekeeper` PASS, hand it to `/qfai-implement`, GREEN, checkpoint —
|
|
389
|
+
before the next branch-1 row's failing test is written, and before P2-P4 build
|
|
390
|
+
any surface. One loop per `TDD-ID`; the nested run is an item cycle, not a
|
|
391
|
+
completion gate (`references/red-provenance.md#what-the-nested-run-owes`).
|
|
392
|
+
- P1d: **Branch 3 rows are judged here, then handed over.** Route
|
|
393
|
+
`qa-gatekeeper` on the `DR-*` — the claim is that the obligation genuinely
|
|
394
|
+
cannot be observed — and hand the row over **with that PASS recorded**. P1b's
|
|
395
|
+
gatekeeper judges branch 1 only and `/qfai-implement`'s exception path writes
|
|
396
|
+
`todo -> exception` and stops, so without this a correct branch-3 row reached
|
|
397
|
+
a terminal status judged by nobody. `/qfai-implement` is the only writer of
|
|
398
|
+
`Status` / `DR-ID` / `Evidence`, so every branch needs a handoff; which goes
|
|
399
|
+
when: `references/red-provenance.md#which-stage-hands-a-row-over`.
|
|
348
400
|
- P2: E2E implementation completed.
|
|
349
401
|
- P3: API implementation completed.
|
|
350
402
|
- P4: Integration implementation completed.
|
|
403
|
+
- P4b: **Branch 2 rows are handed over**, after P4 and before P6 — their mutation
|
|
404
|
+
needs the surface P2-P4 build, the trio is the row's RED payload, and that nested
|
|
405
|
+
run is an item cycle like P1c's (`references/red-provenance.md#what-the-nested-run-owes`).
|
|
351
406
|
- P5: Validation gate passed.
|
|
352
407
|
- P6: Runtime evidence captured.
|
|
353
408
|
- P7: Repo quality gates passed.
|
|
@@ -361,9 +416,13 @@ Before declaring completion:
|
|
|
361
416
|
2. Run:
|
|
362
417
|
|
|
363
418
|
```bash
|
|
364
|
-
npx qfai validate --profile atdd --fail-on error
|
|
419
|
+
npx qfai validate --profile atdd --fail-on error --spec <spec-id>
|
|
365
420
|
```
|
|
366
421
|
|
|
422
|
+
`--spec` scopes the gate to the spec this invocation owns. Omitting it makes
|
|
423
|
+
the gate report obligations belonging to specs this run never touched, which
|
|
424
|
+
is how a spec with every obligation discharged still fails to close.
|
|
425
|
+
|
|
367
426
|
3. Run repository standard gates:
|
|
368
427
|
- format check
|
|
369
428
|
- lint
|
|
@@ -383,11 +442,9 @@ If commands cannot be run due to environment limits, request user execution and
|
|
|
383
442
|
|
|
384
443
|
## DONE Declaration (Mandatory Output)
|
|
385
444
|
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
- DR-IDs referenced (or "none")
|
|
390
|
-
- Confirmation that no rejected options were reintroduced (or RE-OPEN DR-IDs)
|
|
445
|
+
Include the referenced inputs (instructions/steering and spec delta), the DR-IDs
|
|
446
|
+
referenced (or "none"), and confirmation that no rejected options were
|
|
447
|
+
reintroduced (or the RE-OPEN DR-IDs).
|
|
391
448
|
|
|
392
449
|
## FINAL CHECKLIST (Check Last)
|
|
393
450
|
|
|
@@ -437,6 +494,6 @@ A skill MAY narrow the auto-decide bucket (drop entries) but MUST NOT widen it.
|
|
|
437
494
|
|
|
438
495
|
project_memory:
|
|
439
496
|
|
|
440
|
-
- Coverage obligations stay layer-pinned for US and CON-API: tests/e2e/** must cover all required US; tests/api/\*\* all required CON-API. Each TC is covered from the directory
|
|
497
|
+
- Coverage obligations stay layer-pinned for US and CON-API: tests/e2e/** must cover all required US; tests/api/\*\* all required CON-API. Each TC declaring L3/L4/L5, or no Level, is covered from the directory that Level routes to (L3/Integration -> tests/integration/**, L4/API -> tests/api/\*\*, L5/E2E -> tests/e2e/\*\*; no declared Level -> tests/integration/\*\*). L1/Unit and L2/Component owe no ATDD annotation — tdd/test-list.md covers them. An existing one in tests/integration/\*\* is neither counted nor flagged, so do not require adding or removing it.
|
|
441
498
|
- Forbidden references guard the test-layer policy: a TC annotation outside its declared home is rejected — tests/api/** must not carry QFAI:SPEC-XXXX:TC-YYYY unless that TC declares L4/API, and tests/e2e/** likewise unless it declares L5/E2E.
|
|
442
499
|
- Floor / ratio signals are planning hints, never gates; legacy scenario.feature / coverage ledger files remain optional inputs.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# Credential reuse across parallel workers
|
|
2
|
+
|
|
3
|
+
Acceptance tests that need an authenticated actor pay for the sign-in once per worker, not once
|
|
4
|
+
per test. This file is the rule set. It is prose guidance — it adds no validator, no finding code,
|
|
5
|
+
no test layer and no annotation token, and nothing here is checked by a tool.
|
|
6
|
+
|
|
7
|
+
## Scope
|
|
8
|
+
|
|
9
|
+
These rules oblige the **E2E, API and Integration** layers only, which is this skill's whole scope.
|
|
10
|
+
They introduce no unit or component obligation: those layers belong to `/qfai-implement`, and a
|
|
11
|
+
unit test that needs a credential is describing an integration.
|
|
12
|
+
|
|
13
|
+
This guidance names no test runner and no browser automation library. Every rule below is stated
|
|
14
|
+
about a _session cache_ and a _worker_, which every parallel runner has under some spelling, so
|
|
15
|
+
choosing the tool stays the adopter's decision.
|
|
16
|
+
|
|
17
|
+
## Not dogfooded here
|
|
18
|
+
|
|
19
|
+
**QFAI's own suite has zero credentials.** Nothing in this repository signs in, so none of these
|
|
20
|
+
rules is exercised by running the tests here. They are recorded because the rule set is the
|
|
21
|
+
transferable asset, not because they were verified by execution in this tree — read them as
|
|
22
|
+
guidance to adopt, not as a report of something that already runs.
|
|
23
|
+
|
|
24
|
+
## The seven session-reuse rules
|
|
25
|
+
|
|
26
|
+
Each is a separate obligation. A harness can satisfy six of them and still authenticate on every
|
|
27
|
+
test, or still leak one account across workers.
|
|
28
|
+
|
|
29
|
+
### 1. Never sign in per test
|
|
30
|
+
|
|
31
|
+
Authentication belongs to worker setup, not to a test body. A suite that signs in per test pays
|
|
32
|
+
the slowest operation it has once per case, and the cost grows with the suite instead of with the
|
|
33
|
+
worker count.
|
|
34
|
+
|
|
35
|
+
### 2. Never share one account across parallel workers
|
|
36
|
+
|
|
37
|
+
Two workers driving the same actor at the same time is a race with no test to blame it on: a
|
|
38
|
+
sign-out, a session invalidation or a state change made by one worker arrives inside the other's
|
|
39
|
+
assertions. One account per worker, always — a shared account is not an optimisation, it is a
|
|
40
|
+
cross-test dependency that the runner cannot see.
|
|
41
|
+
|
|
42
|
+
### 3. Key the cached session by the pair of worker index and actor
|
|
43
|
+
|
|
44
|
+
Neither half is sufficient on its own. Keyed by actor alone, two workers exercising the same role
|
|
45
|
+
collide — rule 2 again, arriving through the cache. Keyed by worker alone, a suite that needs an
|
|
46
|
+
administrator and an ordinary user hands the second one the first one's session, and the test
|
|
47
|
+
fails on an authorisation it never intended to exercise. The key is the **pair**.
|
|
48
|
+
|
|
49
|
+
### 4. Tear the cache down at worker exit
|
|
50
|
+
|
|
51
|
+
A session cache that outlives its worker is a credential on disk. It also becomes a correctness
|
|
52
|
+
problem the next run: a restored session from a previous run reflects a state the current run
|
|
53
|
+
never established, so a test can pass against data it did not create. Bind the cache's lifetime to
|
|
54
|
+
the worker's.
|
|
55
|
+
|
|
56
|
+
### 5. Re-authenticate and rewrite the cache when a restored session is rejected
|
|
57
|
+
|
|
58
|
+
A cached session expires, gets revoked, or stops being valid because the environment was reset
|
|
59
|
+
between runs. The harness must detect the rejection, sign in again, and **overwrite** the cache
|
|
60
|
+
with the fresh session. Two failure modes this closes: a suite that treats the rejection as a test
|
|
61
|
+
failure reports a red that says nothing about the product, and a suite that retries against the
|
|
62
|
+
stale cache loops.
|
|
63
|
+
|
|
64
|
+
### 6. A test that mutates its own account creates a dedicated one
|
|
65
|
+
|
|
66
|
+
Changing a password, disabling the account, altering a profile, consuming a one-time entitlement:
|
|
67
|
+
each of those breaks the cached session every _other_ test on that actor depends on. A test whose
|
|
68
|
+
subject is the account itself provisions its own actor, uses it, and does not put it back in the
|
|
69
|
+
pool.
|
|
70
|
+
|
|
71
|
+
### 7. Test-level parallelism costs more workers, not more sign-ins
|
|
72
|
+
|
|
73
|
+
Raising parallelism adds workers; each new worker pays one sign-in. If raising parallelism raises
|
|
74
|
+
the sign-in count faster than the worker count, one of the six rules above is not being followed —
|
|
75
|
+
most often rule 1, and most often because a helper was called from a test body rather than from
|
|
76
|
+
setup. The ratio is the diagnostic.
|
|
77
|
+
|
|
78
|
+
## The companion rule: a caller-injected environment is not the harness's to manage
|
|
79
|
+
|
|
80
|
+
When the caller injects an environment identifier — a target URL, a tenant, a named deployment,
|
|
81
|
+
any handle that says _use this one_ — the harness **must not provision that environment and must
|
|
82
|
+
not tear it down.** It signs in against it, runs, and leaves it exactly as it found it.
|
|
83
|
+
|
|
84
|
+
The reason is that the injection is what tells the harness the environment is owned elsewhere.
|
|
85
|
+
A harness that provisions when it was handed a target creates a second environment nobody asked
|
|
86
|
+
for; a harness that tears one down at the end of a run destroys a shared or long-lived environment
|
|
87
|
+
that other work depends on. Both are irreversible from inside the run.
|
|
88
|
+
|
|
89
|
+
The rule is about _ownership_, not about the shape of the value: a harness that creates its own
|
|
90
|
+
ephemeral environment when nothing was injected is following this rule, not breaking it.
|
|
91
|
+
|
|
92
|
+
## Script naming: adopter guidance
|
|
93
|
+
|
|
94
|
+
**A credential-free lane and a credentialed lane should be reachable under different script
|
|
95
|
+
names.** A lane that structurally cannot touch the network is a different thing from one that must
|
|
96
|
+
have a working account, and giving them one name makes that difference invisible to whoever runs
|
|
97
|
+
them: contributors without credentials cannot tell which lane they can run, and CI cannot express
|
|
98
|
+
"run the half that needs no secrets" without a name for it.
|
|
99
|
+
|
|
100
|
+
Two properties do the work, whatever the names are:
|
|
101
|
+
|
|
102
|
+
- the credential-free lane is runnable by someone who has no account at all, and
|
|
103
|
+
- reading the script name tells you which of the two you are about to run.
|
|
104
|
+
|
|
105
|
+
**This is adopter guidance and QFAI adopts none of it.** QFAI keeps its own script names unchanged;
|
|
106
|
+
because its suite has no credentials there is no credentialed lane here to name, and renaming a
|
|
107
|
+
lane to model a distinction this repository does not have would be a cost with no reader.
|
|
108
|
+
|
|
109
|
+
## A worked example, as one illustration among possible backends
|
|
110
|
+
|
|
111
|
+
The shape below is **one** way to satisfy the rules, sketched in no particular tool. Any runner
|
|
112
|
+
with per-worker setup and a place to persist a session can express it, and the rules — not this
|
|
113
|
+
sketch — are what has to survive the translation.
|
|
114
|
+
|
|
115
|
+
```text
|
|
116
|
+
worker setup:
|
|
117
|
+
key = (worker index, actor role)
|
|
118
|
+
cached = load(key) # rule 3: the pair is the key
|
|
119
|
+
if cached is absent:
|
|
120
|
+
session = sign_in(actor for key) # rule 1: once per worker
|
|
121
|
+
save(key, session)
|
|
122
|
+
else:
|
|
123
|
+
if not accepted(cached): # rule 5: a restored session can be rejected
|
|
124
|
+
session = sign_in(actor for key)
|
|
125
|
+
save(key, session) # ...and the cache is rewritten
|
|
126
|
+
|
|
127
|
+
each test:
|
|
128
|
+
attach the session for this worker's key # never sign_in here
|
|
129
|
+
|
|
130
|
+
a test that mutates its actor:
|
|
131
|
+
actor = create_dedicated_actor() # rule 6: not from the pool
|
|
132
|
+
...
|
|
133
|
+
(the pooled session is untouched)
|
|
134
|
+
|
|
135
|
+
worker teardown:
|
|
136
|
+
discard(key) # rule 4: lifetime is the worker's
|
|
137
|
+
|
|
138
|
+
environment:
|
|
139
|
+
if the caller injected a target:
|
|
140
|
+
use it; provision nothing; tear down nothing # the companion rule
|
|
141
|
+
else:
|
|
142
|
+
the harness may create and destroy its own
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Nothing in that sketch names a library, and it should not: the moment a rule is expressed as one
|
|
146
|
+
tool's API call, it stops being portable to the next one.
|