qfai 1.9.2 → 1.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +67 -6
- package/assets/init/.qfai/assistant/README.md +27 -0
- package/assets/init/.qfai/assistant/agents/acceptance-test-engineer.md +1 -0
- package/assets/init/.qfai/assistant/agents/backend-engineer.md +1 -0
- package/assets/init/.qfai/assistant/agents/completion-reviewer.md +29 -2
- package/assets/init/.qfai/assistant/agents/delivery-planner.md +17 -0
- package/assets/init/.qfai/assistant/agents/frontend-engineer.md +1 -0
- package/assets/init/.qfai/assistant/agents/implementation-reviewer.md +17 -0
- package/assets/init/.qfai/assistant/agents/orchestrator.md +2 -2
- package/assets/init/.qfai/assistant/agents/qa-gatekeeper.md +183 -5
- package/assets/init/.qfai/assistant/agents/test-design-analyst.md +22 -3
- package/assets/init/.qfai/assistant/catalog/cli-ux-guidelines.md +2 -2
- package/assets/init/.qfai/assistant/catalog/spec_required_files.json +2 -1
- package/assets/init/.qfai/assistant/catalog/test-layers-ci-lanes.md +60 -0
- package/assets/init/.qfai/assistant/catalog/test-layers.md +430 -13
- package/assets/init/.qfai/assistant/catalog/worklog-entry.schema.md +165 -0
- package/assets/init/.qfai/assistant/constitution/communication.md +1 -1
- package/assets/init/.qfai/assistant/constitution/constitution.md +1 -1
- package/assets/init/.qfai/assistant/constitution/drift-protocol.md +359 -10
- package/assets/init/.qfai/assistant/constitution/quality.md +35 -5
- package/assets/init/.qfai/assistant/constitution/requirements-decomposition.md +37 -0
- package/assets/init/.qfai/assistant/constitution/shared-skill-delegation-baseline.md +419 -8
- package/assets/init/.qfai/assistant/constitution/shared-skill-operating-baseline.md +122 -5
- package/assets/init/.qfai/assistant/constitution/workflow.md +53 -7
- package/assets/init/.qfai/assistant/manifest/agent-catalog.yml +439 -946
- package/assets/init/.qfai/assistant/manifest/agent-routing.yml +116 -4
- package/assets/init/.qfai/assistant/manifest/review-profiles.yml +9 -0
- package/assets/init/.qfai/assistant/process/migrations/v1.4.27-atdd-alignment.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-atdd/SKILL.md +150 -53
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/credential-reuse.md +146 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/red-provenance.md +456 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/review-fix-rounds.md +128 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/scaffolding.md +29 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/shared-test-artifacts.md +96 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/stale-manifest.md +34 -0
- package/assets/init/.qfai/assistant/skills/qfai-atdd/references/test-case-depth-checklist.md +52 -5
- package/assets/init/.qfai/assistant/skills/qfai-configure/SKILL.md +16 -8
- package/assets/init/.qfai/assistant/skills/qfai-discussion/SKILL.md +8 -4
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/design-md-brand-catalog.md +2 -2
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/discussion-completion-matrix.md +17 -7
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/rcp_footer.md +10 -4
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/review-cycle-playbook.md +5 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/ui-bearing-playbook.md +4 -4
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/ui_ux/review_audit_playbook.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/ui_ux/trend_scan_playbook.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/references/ui_ux_best_practices.md +17 -7
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/01_Context.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/02_Inception-Deck.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/03_Story-Workshop.md +11 -3
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/05_Scope.md +5 -2
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/07_NFR.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/09_Constraints.md +7 -4
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/10_Policy.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/11_OQ-Register.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/12_OQ-Resolution-Log.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/14_Review-Request.md +14 -7
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/99_delta.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/review/Rxx_reviewer.md +16 -7
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/review/review_request.md +9 -6
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/review/summary.json +2 -0
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/uiux/40_screen_contracts.md +3 -2
- package/assets/init/.qfai/assistant/skills/qfai-discussion/templates/uiux/50_review_input_bundle.md +4 -2
- package/assets/init/.qfai/assistant/skills/qfai-implement/SKILL.md +252 -130
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/change-request-reset.md +93 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/checkpoint-verification.md +210 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/cross-spec-ownership.md +73 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/evidence-revision.md +291 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/execution-ledger.md +389 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/final-checklist.md +28 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/finding-classification.md +49 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/ledger-preconditions.md +55 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/oracle-strength.md +81 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/parallelization-policy.md +235 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/red-admissibility.md +91 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/red-not-observable.md +112 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/relevant-test-suite.md +87 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/review-artifact-layout.md +44 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/round-evidence.md +118 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/selector-granularity.md +24 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/upstream-artifact-ordering.md +33 -0
- package/assets/init/.qfai/assistant/skills/qfai-implement/references/volume-policy.md +149 -0
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/SKILL.md +35 -18
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/evidence-requirements.md +1 -1
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/generator-prompt.md +106 -7
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/handoff.md +40 -11
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/references/iteration-loop.md +48 -1
- package/assets/init/.qfai/assistant/skills/qfai-prototyping/templates/DESIGN.md.sample +6 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/SKILL.md +87 -22
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/contract-artifact-rules.md +148 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/rcp_footer.md +10 -4
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/review-cycle-playbook.md +5 -1
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/sdd-execution-playbook.md +4 -2
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/sdd-phase-checklists.md +18 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/sdd-quality-gate.md +44 -3
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/sdd-triage.md +60 -7
- package/assets/init/.qfai/assistant/skills/qfai-sdd/references/spec-traceability-rules.md +171 -6
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/change-request.md +125 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/contracts/db-contract.sample.sql +6 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/evidence/sdd-spec.md +92 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/01_Objective.md +27 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/02_Initiative.md +30 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/05_Contracts.md +16 -6
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/06_Glossary.md +19 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/07_Constraints.md +25 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/08_Decisions.md +22 -2
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/_policies/11_Slice-Policy.md +37 -10
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/02_User-stories.md +20 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/03_Acceptance-Criteria.md +19 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/04_Business-Rules.md +32 -3
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/06_Test-Cases.md +56 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/07_Decisions.md +29 -2
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/10_Plan.md +41 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/16_Traceability-ledger.md +58 -0
- package/assets/init/.qfai/assistant/skills/qfai-sdd/templates/specs/spec/tdd/test-list.md +56 -0
- package/assets/init/.qfai/assistant/skills/qfai-verify/SKILL.md +57 -102
- package/assets/init/.qfai/assistant/skills/qfai-verify/references/articles.md +24 -0
- package/assets/init/.qfai/assistant/skills/qfai-verify/references/context-load.md +22 -0
- package/assets/init/.qfai/assistant/skills/qfai-verify/references/verify-output-contract.md +48 -0
- package/assets/init/.qfai/assistant/skills/qfai-verify/templates/verify-evidence.md +48 -0
- package/assets/init/.qfai/assistant/skills/web-research/SKILL.md +25 -7
- package/assets/init/.qfai/waivers.yml +11 -5
- package/assets/init/root/.github/workflows/qfai-tests.yml +318 -0
- package/assets/init/root/.github/workflows/qfai-validate.yml +327 -24
- package/assets/init/root/DESIGN.md +6 -0
- package/assets/init/root/qfai.config.yaml +15 -12
- package/dist/cli/index.cjs +20646 -13620
- package/dist/cli/index.cjs.map +1 -1
- package/dist/cli/index.mjs +22101 -15061
- package/dist/cli/index.mjs.map +1 -1
- package/dist/index.cjs +13954 -8829
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +423 -14
- package/dist/index.d.ts +423 -14
- package/dist/index.mjs +9330 -4224
- package/dist/index.mjs.map +1 -1
- package/package.json +22 -19
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Test layers to CI lanes
|
|
2
|
+
|
|
3
|
+
A crosswalk between the test layers in [`test-layers.md`](./test-layers.md) and the CI lanes
|
|
4
|
+
the shipped workflows run. It exists so that "which job runs my L3 tests" has one answer a
|
|
5
|
+
reader can find, instead of being rediscovered from a workflow file each time.
|
|
6
|
+
|
|
7
|
+
**The layer policy loader does not read this file.** (Written without a hyphen on purpose: the
|
|
8
|
+
loader extracts anything matching `layer-<word>` from the file it parses, and while it does not
|
|
9
|
+
parse this one, a catalog-directory file carrying such a token is a hazard waiting for the day
|
|
10
|
+
someone widens the loader to scan the directory.) It resolves `catalog/test-layers.md` by
|
|
11
|
+
exact path and reads nothing else in this directory, so nothing written here can widen or
|
|
12
|
+
narrow the layer vocabulary. That is deliberate: this document is a map, and a map that could
|
|
13
|
+
change the territory would be a second source of truth for something that already has one. If
|
|
14
|
+
you need to change what layers exist, change `test-layers.md` — this file follows.
|
|
15
|
+
|
|
16
|
+
## The mapping
|
|
17
|
+
|
|
18
|
+
| Layer | What it exercises | Shipped CI lane |
|
|
19
|
+
| -------------- | --------------------------------------------- | --------------- |
|
|
20
|
+
| L1 Unit | one module, no I/O | `unit` |
|
|
21
|
+
| L2 Component | one component and its immediate collaborators | `component` |
|
|
22
|
+
| L3 Integration | several modules across a real boundary | `integration` |
|
|
23
|
+
| L4 API | a running interface, contract-first | `api` |
|
|
24
|
+
| L5 E2E | the product as a user reaches it | `e2e` |
|
|
25
|
+
|
|
26
|
+
Two lanes in the shipped workflow are not layers and have no row above:
|
|
27
|
+
|
|
28
|
+
- **`detection`** decides which of the lanes above need to run for a given change, and
|
|
29
|
+
publishes that decision as job outputs. It is infrastructure, not a test level.
|
|
30
|
+
- **`verdict`** aggregates the lanes into the single status a branch-protection rule can name.
|
|
31
|
+
It runs unconditionally so that a run where every lane was skipped is still distinguishable
|
|
32
|
+
from a run where nothing was verified.
|
|
33
|
+
|
|
34
|
+
## Lane names are a project's choice
|
|
35
|
+
|
|
36
|
+
The names in the third column are the ones the shipped workflow uses. They are not a contract:
|
|
37
|
+
a project that calls its integration lane `service-tests` is not violating anything, and this
|
|
38
|
+
table is not a rename list. What matters is that each layer a project uses has some lane that
|
|
39
|
+
runs it, and that the lane's name stays put once branch protection refers to it — a check name
|
|
40
|
+
is a repository setting, and renaming one silently makes a required check unsatisfiable.
|
|
41
|
+
|
|
42
|
+
## Per-level routing is not enforced
|
|
43
|
+
|
|
44
|
+
`test-layers.md` marks per-level annotation routing as a target state and says plainly: not
|
|
45
|
+
enforced, do not follow yet. This document does not change that and does not activate it. The
|
|
46
|
+
live traceability gate reads one directory for annotations, and it is the gate — not this
|
|
47
|
+
table — that decides where an annotation counts.
|
|
48
|
+
|
|
49
|
+
So: read this file to find out which lane runs a layer. Do not read it as instructions about
|
|
50
|
+
where anything belongs in the tree. That question is settled by the gate's own scope, and by
|
|
51
|
+
`test-layers.md` for the vocabulary.
|
|
52
|
+
|
|
53
|
+
## Keeping this file honest
|
|
54
|
+
|
|
55
|
+
- Every layer code here also appears in `test-layers.md`'s crosswalk. If you add a layer there
|
|
56
|
+
and not here, this table is incomplete; if you add one here and not there, you have invented
|
|
57
|
+
a layer nothing enforces.
|
|
58
|
+
- The lane column describes the shipped workflow. If the shipped workflow's job set changes,
|
|
59
|
+
this table is stale, and stale is worse than absent for a document whose only job is to save
|
|
60
|
+
someone a lookup.
|
|
@@ -2,8 +2,96 @@
|
|
|
2
2
|
|
|
3
3
|
This document is the SSOT for ATDD test-layer semantics and completion gates.
|
|
4
4
|
|
|
5
|
+
For which CI lane runs which layer, see the sibling map
|
|
6
|
+
[`test-layers-ci-lanes.md`](./test-layers-ci-lanes.md). That file is a crosswalk only — the
|
|
7
|
+
policy loader reads this file and not that one, so nothing there can change the
|
|
8
|
+
vocabulary declared below.
|
|
9
|
+
|
|
10
|
+
## Layer vocabulary crosswalk (normative)
|
|
11
|
+
|
|
12
|
+
qfai spells the same layer four ways across shipped artifacts. This table is
|
|
13
|
+
the crosswalk; every artifact MUST use the spelling in its column.
|
|
14
|
+
|
|
15
|
+
| Code | Word | Tag | Test directory | `06_Test-Cases.md#Level` | `tdd/test-list.md#Layer` |
|
|
16
|
+
| ---- | ----------- | ------------------- | --------------------------- | ------------------------ | ------------------------ |
|
|
17
|
+
| L1 | Unit | `layer-unit` | project convention | `L1` | `Unit` |
|
|
18
|
+
| L2 | Component | `layer-component` | project convention | `L2` | `Component` |
|
|
19
|
+
| L3 | Integration | `layer-integration` | `<testsDir>/integration/**` | `L3` | `Integration` |
|
|
20
|
+
| L4 | API | `layer-api` | `<testsDir>/api/**` | — | `API` |
|
|
21
|
+
| L5 | E2E | `layer-e2e` | `<testsDir>/e2e/**` | — | `E2E` |
|
|
22
|
+
|
|
23
|
+
Rules:
|
|
24
|
+
|
|
25
|
+
- **One value per cell.** A `Level` cell and a `Layer` cell each hold exactly
|
|
26
|
+
one layer. An obligation spanning two layers is two rows, not one row with
|
|
27
|
+
two values.
|
|
28
|
+
- `06_Test-Cases.md` uses the **code** (`L1`…`L3`) in its `Level` column.
|
|
29
|
+
- `tdd/test-list.md` uses the **word** in its `Layer` column.
|
|
30
|
+
- Test-strategy tags in prompts and policy files use the **tag** form.
|
|
31
|
+
- **`<testsDir>` is `paths.testsDir` from `qfai.config.yaml`**, whose default
|
|
32
|
+
is `tests` — hence the shipped `tests/integration/**`, `tests/api/**` and
|
|
33
|
+
`tests/e2e/**`. A project that repoints `paths.testsDir` moves all three at
|
|
34
|
+
once; the ATDD traceability scan follows the configured value, so never
|
|
35
|
+
hard-code the literal `tests/` prefix in a project's own artifacts.
|
|
36
|
+
- L1 and L2 have no mandated directory: unit and component tests live wherever
|
|
37
|
+
the project's own convention puts them. Only L3-L5 are directory-pinned, and
|
|
38
|
+
only those directories are scanned by the ATDD traceability rules.
|
|
39
|
+
- **A `TC-*` row's `Level` is L1-L3.** Of those, only L3 owes an ATDD
|
|
40
|
+
annotation: L1 and L2 have no mandated directory, so `QFAI-ATDD-112` does not
|
|
41
|
+
apply to them and `QFAI-ATDD-117` (`info`) names them instead. Their gate is
|
|
42
|
+
`tdd/test-list.md` / `TDDLIST_TC_NOT_COVERED`, under `/qfai-implement`.
|
|
43
|
+
`US-*` is answered from `<testsDir>/e2e/**` (`QFAI-ATDD-111`), `CON-API-*`
|
|
44
|
+
from `<testsDir>/api/**` (`QFAI-ATDD-113`) and `CON-DB-*` from
|
|
45
|
+
`<testsDir>/integration/**` (`QFAI-ATDD-115`) — those three are fixed by the
|
|
46
|
+
ID type. A `TC-*` is answered from the directory **its own declared `Level`**
|
|
47
|
+
names, which for a correctly filed row is `<testsDir>/integration/**`; see
|
|
48
|
+
[Annotation routing](#annotation-routing) for the full table and the
|
|
49
|
+
misplacement rules.
|
|
50
|
+
L4's goal is `CON-API-*` and L5's is `US-*` (see the layer definitions
|
|
51
|
+
below), so an oracle that lands at L4 or L5 means the obligation is misfiled:
|
|
52
|
+
record it as `CON-API-*` or `US-*` rather than leaving a `TC-*` row at a
|
|
53
|
+
layer whose goal is another ID type.
|
|
54
|
+
- The two code-side word lists (`tddHelpers.ts#UNIT_COMPONENT_LAYERS` /
|
|
55
|
+
`#NON_COVERAGE_LAYERS`) accept both the code and the word form for the same
|
|
56
|
+
layer; they MUST stay in step with this table.
|
|
57
|
+
|
|
58
|
+
## How this file is consumed
|
|
59
|
+
|
|
60
|
+
The layer set below is read by `core/layerPolicy.ts` and is the SSOT for two
|
|
61
|
+
checks: `QFAI-EX-005` on the legacy spec-pack layout, and `QFAI-EX-105` on the
|
|
62
|
+
layered layout `npx qfai init` produces. Until both consumed it, the file was
|
|
63
|
+
read, reported on, and then ignored on every modern project.
|
|
64
|
+
|
|
65
|
+
- A file that yields no layers raises `QFAI-SPACK-090` (error) rather than
|
|
66
|
+
silently widening to the built-in set.
|
|
67
|
+
- A declared set that disagrees with the built-in set raises `QFAI-SPACK-091`
|
|
68
|
+
(warning). Without it the two could drift in either direction and this file
|
|
69
|
+
would not be an SSOT.
|
|
70
|
+
|
|
71
|
+
## Scaffolded tests
|
|
72
|
+
|
|
73
|
+
`npx qfai atdd scaffold` writes skeletons to `<testsDir>/integration/<spec-id>/`.
|
|
74
|
+
There is deliberately no `<testsDir>/atdd/**` row below: a fourth root would
|
|
75
|
+
need a rule for which layer such a file belongs to, and qfai has none. The
|
|
76
|
+
writer targets a declared layer instead.
|
|
77
|
+
|
|
5
78
|
## Layer definitions
|
|
6
79
|
|
|
80
|
+
### L1 Unit
|
|
81
|
+
|
|
82
|
+
- Scope: pure decision logic — a single module's inputs and return values, with
|
|
83
|
+
no port collaboration and no real infrastructure.
|
|
84
|
+
- Goal: verify `TC-*` obligations whose oracle observes inputs and outputs only.
|
|
85
|
+
- Convention: `tests/unit/**`. L1 has no mandated directory and owes no ATDD annotation — see the crosswalk and "Unit and Component owe no ATDD annotation".
|
|
86
|
+
|
|
87
|
+
### L2 Component
|
|
88
|
+
|
|
89
|
+
- Scope: collaboration with a port through a fixture adapter (fake / in-memory),
|
|
90
|
+
with no real infrastructure.
|
|
91
|
+
- Goal: verify `TC-*` obligations whose oracle observes the interaction with a
|
|
92
|
+
port rather than infrastructure state.
|
|
93
|
+
- Convention: `tests/component/**`. L2 has no mandated directory and owes no ATDD annotation — see the crosswalk and "Unit and Component owe no ATDD annotation".
|
|
94
|
+
|
|
7
95
|
### L3 Integration
|
|
8
96
|
|
|
9
97
|
- Scope: real infrastructure integration (for example DB/queue/filesystem) within service boundaries.
|
|
@@ -22,8 +110,147 @@ This document is the SSOT for ATDD test-layer semantics and completion gates.
|
|
|
22
110
|
- Goal: verify `US-*` obligations from specs.
|
|
23
111
|
- Location rule: `tests/e2e/**`.
|
|
24
112
|
|
|
113
|
+
## Layer derivation procedure (normative)
|
|
114
|
+
|
|
115
|
+
Deciding a `TC-*`'s layer is a per-TC judgement, not a constant. Use the
|
|
116
|
+
falsifying-oracle rule:
|
|
117
|
+
|
|
118
|
+
1. **Find the oracle.** Identify the single assertion whose removal would let a
|
|
119
|
+
wrong implementation pass. That assertion, not the test's setup, is what the
|
|
120
|
+
TC verifies.
|
|
121
|
+
2. **Restrict it to the parent BR's obligations.** Anything the oracle observes
|
|
122
|
+
that the parent business rule does not own is incidental and does not raise
|
|
123
|
+
the layer.
|
|
124
|
+
3. **Read the layer off what the oracle observes:**
|
|
125
|
+
- inputs and return values only → **L1 Unit**
|
|
126
|
+
- collaboration with a port through a fixture adapter (no real
|
|
127
|
+
infrastructure) → **L2 Component**
|
|
128
|
+
- real infrastructure state — DB rows, queue messages, files → **L3 Integration**
|
|
129
|
+
- values at the service boundary — status codes, response bodies, auth and
|
|
130
|
+
error contracts → **L4 API**
|
|
131
|
+
- a full-system journey across UI/API/data → **L5 E2E**
|
|
132
|
+
|
|
133
|
+
### Worked examples
|
|
134
|
+
|
|
135
|
+
| Oracle asserts | Layer |
|
|
136
|
+
| ----------------------------------------------------------------------------- | -------------- |
|
|
137
|
+
| `price(order) === 1250` for a given input | L1 Unit |
|
|
138
|
+
| the repository port was called with the normalized key, via a fixture adapter | L2 Component |
|
|
139
|
+
| the row is present in the database after commit | L3 Integration |
|
|
140
|
+
| `POST /orders` returns `422` with `code: "OUT_OF_AREA"` | L4 API |
|
|
141
|
+
| a user can register, order, and see the order in their history | L5 E2E |
|
|
142
|
+
|
|
143
|
+
### Obligation spanning more than one layer
|
|
144
|
+
|
|
145
|
+
**Split the row.** One TC = one oracle = one layer. If an obligation is
|
|
146
|
+
observable at two layers, it is two obligations: write one TC per oracle and
|
|
147
|
+
give each its own `Level`.
|
|
148
|
+
|
|
149
|
+
- A multi-valued `Level` cell (`L3/L5`, `L1/L2`, `L1, L3`) is **illegal**. It
|
|
150
|
+
matches no entry in the crosswalk, so no rule can read a layer out of it.
|
|
151
|
+
- **What a reader does with one: split the row.** One TC per oracle, each with
|
|
152
|
+
its own single `Level`. Nothing else is a fix — in particular, do not record
|
|
153
|
+
a normalization that "drops one half": no tool performs one, and a note
|
|
154
|
+
saying an `L3/L5` row "normalizes to `L3`" is a claim about a value that only
|
|
155
|
+
`06_Test-Cases.md` can make.
|
|
156
|
+
- **What the validators do with one, until it is split.** They neither guess
|
|
157
|
+
nor let it through:
|
|
158
|
+
- `QFAI-ATDD-112` routes the TC to the same place a TC with no declared
|
|
159
|
+
`Level` goes — `<testsDir>/integration/**` — and keeps the obligation.
|
|
160
|
+
Unreadable is deliberately not "excused": if a cell qfai cannot read
|
|
161
|
+
discharged the obligation, `L1/L2` would be a one-keystroke way to delete
|
|
162
|
+
any TC from the gate. That default is where the obligation is _reported_,
|
|
163
|
+
not where the obligation _belongs_.
|
|
164
|
+
- `TDDLIST_UNKNOWN_LEVEL` (`warning`) names the cell, and the TC stays a
|
|
165
|
+
coverage target, so `tdd/test-list.md` still owes it a row.
|
|
166
|
+
- If splitting is genuinely impossible, escalate through the Drift Protocol
|
|
167
|
+
rather than inventing a combined value.
|
|
168
|
+
|
|
169
|
+
### Direction of authority (anti-pattern)
|
|
170
|
+
|
|
171
|
+
The test driver follows the declared layer. **The layer is never inferred from
|
|
172
|
+
how a test happens to be driven.** A unit-level obligation — one whose oracle,
|
|
173
|
+
after step 2, observes only inputs and return values — exercised through an
|
|
174
|
+
HTTP client is still L1 badly implemented; it is not an L4 test.
|
|
175
|
+
|
|
176
|
+
**Step 2 outranks step 3.** Restriction to the parent BR's obligations runs
|
|
177
|
+
first, so a transport the parent BR does not own is incidental and never
|
|
178
|
+
raises the layer. A BR that owns only the price calculation, asserted as
|
|
179
|
+
`response.body.price` over HTTP, reads as L1: the price is the BR-owned value,
|
|
180
|
+
the response envelope is the incidental transport. Step 3's "response bodies →
|
|
181
|
+
L4" applies only when the service-boundary values are themselves what the
|
|
182
|
+
parent BR owns. Inverting this is what collapses a designed pyramid into an
|
|
183
|
+
all-integration suite.
|
|
184
|
+
|
|
185
|
+
### Annotation routing
|
|
186
|
+
|
|
187
|
+
The derived `Level` records which oracle owns the obligation, and the
|
|
188
|
+
[ATDD annotation hard gate](#atdd-annotation-hard-gate) routes each obligation
|
|
189
|
+
ID to exactly one directory. `US-*` is answered from `tests/e2e/**`
|
|
190
|
+
(`QFAI-ATDD-111`) and `CON-API-*` from `tests/api/**` (`QFAI-ATDD-113`); those
|
|
191
|
+
two are fixed by the ID type. A `TC-*` is answered from the directory **its own
|
|
192
|
+
declared `Level`** names (`QFAI-ATDD-112`):
|
|
193
|
+
|
|
194
|
+
| `Level` | Answered from |
|
|
195
|
+
| ----------------------------- | ------------------------------ |
|
|
196
|
+
| `L1`/`Unit` | no ATDD obligation |
|
|
197
|
+
| `L2`/`Component` | no ATDD obligation |
|
|
198
|
+
| `L3`/`Integration` | `tests/integration/**` |
|
|
199
|
+
| `L4`/`API` | `tests/api/**` (note) |
|
|
200
|
+
| `L5`/`E2E` | `tests/e2e/**` (note) |
|
|
201
|
+
| none declared | `tests/integration/**` |
|
|
202
|
+
| anything else — typo, `L3/L5` | `tests/integration/**` (note2) |
|
|
203
|
+
|
|
204
|
+
**(note)** A `TC-*` **should not be** at L4 or L5 — the first bullet below says
|
|
205
|
+
why and what to do instead. The gate routes it there rather than rejecting it so
|
|
206
|
+
a misfiled row is reported once, by the rule that names the real cause, instead
|
|
207
|
+
of twice as "uncovered in integration" and "forbidden in api".
|
|
208
|
+
|
|
209
|
+
**(note2)** A `Level` the crosswalk does not list — a typo, a project's own
|
|
210
|
+
word, or the illegal multi-valued cell — falls to the same default as an
|
|
211
|
+
undeclared one, and keeps its obligation. The default is the conservative
|
|
212
|
+
answer to a cell qfai cannot read, never a supported spelling: fix the cell
|
|
213
|
+
(see [Obligation spanning more than one layer](#obligation-spanning-more-than-one-layer)).
|
|
214
|
+
`TDDLIST_UNKNOWN_LEVEL` (`warning`) names such a cell on the ledger side.
|
|
215
|
+
|
|
216
|
+
Exactly one directory, never two: an annotation outside the one its `Level`
|
|
217
|
+
names is both uncovered and rejected (`QFAI-ATDD-121` / `QFAI-ATDD-122` /
|
|
218
|
+
`QFAI-ATDD-123`), and
|
|
219
|
+
the rejection is symmetric — an annotation left in `tests/integration/**` after
|
|
220
|
+
its TC moved to `L4`/`L5` is rejected the same way an early one in
|
|
221
|
+
`tests/api/**` is. Two consequences bind every `TC-*` row:
|
|
222
|
+
|
|
223
|
+
- **A `TC-*` row's `Level` stays within L1–L3.** L4's goal is `CON-API-*` and
|
|
224
|
+
L5's goal is `US-*` (see the layer definitions above), so an oracle that
|
|
225
|
+
derives to L4 or L5 means the obligation is misfiled, not that the TC is an
|
|
226
|
+
L4/L5 test. Re-file it as `CON-API-*` or `US-*`.
|
|
227
|
+
**Re-filing is an upstream change, never a bare row deletion.** By step 2 the
|
|
228
|
+
derivation reaches L4/L5 only when the parent `BR-*` itself owns the
|
|
229
|
+
service-boundary contract or the journey, so the `TC-*` row is removed only
|
|
230
|
+
together with the `EX-*` it verifies and the `BR-*`/`AC-*` that EX
|
|
231
|
+
concretizes. Dropping the row alone leaves the parent EX with no `EX-Ref`
|
|
232
|
+
and `npx qfai validate --profile sdd --fail-on error` reports `QFAI-COV-203`
|
|
233
|
+
(and `QFAI-COV-201` when that TC was the AC's only cover); dropping the EX
|
|
234
|
+
but keeping its BR reports `QFAI-COV-202`. Move the whole chain in one
|
|
235
|
+
change through the Drift Protocol so `04_Business-Rules.md`,
|
|
236
|
+
`05_Examples.md`, `06_Test-Cases.md` and the contract stay consistent. If the
|
|
237
|
+
parent BR keeps a spec-side obligation, split the row instead: keep a `TC-*`
|
|
238
|
+
for the part the BR owns — by step 2 that part derives to L1–L3 — and re-file
|
|
239
|
+
only the boundary assertion as `CON-API-*` / `US-*`.
|
|
240
|
+
- **An L1/L2 `Level` carries no `QFAI-ATDD-112` obligation, and the gate does
|
|
241
|
+
not change the `Level`.** The two are independent: the `Level` records what
|
|
242
|
+
the oracle observes and is derived before any test exists, while
|
|
243
|
+
`QFAI-ATDD-112` asks only about the layers ATDD owns. Never rewrite a derived
|
|
244
|
+
L1/L2 to L3 to make a gate quieter — that would make the recorded oracle
|
|
245
|
+
depend on implementation order and hide the unit/component work
|
|
246
|
+
`/qfai-implement` selects. An L1/L2 row's obligation is discharged through
|
|
247
|
+
`tdd/test-list.md` and `TDDLIST_TC_NOT_COVERED`, not through an annotation in
|
|
248
|
+
a directory ATDD scans.
|
|
249
|
+
|
|
25
250
|
## TestKind resolution (single source)
|
|
26
251
|
|
|
252
|
+
- `tests/unit/**` -> Unit
|
|
253
|
+
- `tests/component/**` -> Component
|
|
27
254
|
- `tests/e2e/**` -> E2E
|
|
28
255
|
- `tests/api/**` -> API
|
|
29
256
|
- `tests/integration/**` -> Integration
|
|
@@ -31,46 +258,236 @@ This document is the SSOT for ATDD test-layer semantics and completion gates.
|
|
|
31
258
|
## Annotation schema (code-side)
|
|
32
259
|
|
|
33
260
|
- Smallest trace unit is ID.
|
|
34
|
-
- Multiple IDs per test file are allowed
|
|
261
|
+
- Multiple IDs per test file are allowed — but this is a trace rule, not
|
|
262
|
+
licence to aggregate a whole spec into one module. See Test-file granularity
|
|
263
|
+
below.
|
|
35
264
|
- AC annotations are optional (indirect coverage through TC is acceptable).
|
|
36
265
|
- Allowed forms:
|
|
37
266
|
- `QFAI:SPEC-0001:US-0001`
|
|
38
267
|
- `QFAI:SPEC-0001:TC-0001`
|
|
39
268
|
- `QFAI:CON-API-0001`
|
|
269
|
+
- `QFAI:CON-DB-0001`
|
|
40
270
|
|
|
41
271
|
## ATDD annotation hard gate
|
|
42
272
|
|
|
43
273
|
- E2E obligations:
|
|
44
|
-
- Every `US-*` in
|
|
274
|
+
- Every `US-*` in a **user-facing** spec must be referenced at least once from
|
|
275
|
+
`tests/e2e/**`. "User-facing" is the same surface **union**
|
|
276
|
+
`/qfai-prototyping` enforces. Any one of these signals puts a spec in it:
|
|
277
|
+
- frontmatter `surface_type: ui-bearing` in `01_Spec.md`;
|
|
278
|
+
- a matching UI contract `.qfai/contracts/ui/<spec-id>*.yaml` — a project
|
|
279
|
+
that declares its surfaces only through contracts is still in scope;
|
|
280
|
+
- a legacy `# … prototyping …` heading in `01_Spec.md`;
|
|
281
|
+
- the spec pinned by `qfai.config.yaml#prototyping.primarySpecId`.
|
|
282
|
+
|
|
283
|
+
A spec that declares no user-facing surface by **any** of those signals
|
|
284
|
+
owes no E2E reference, and `QFAI-ATDD-111` does not fire for it.
|
|
285
|
+
|
|
286
|
+
- Scoping applies only when the project declares at least one UI-bearing
|
|
287
|
+
spec. A project that has never declared a surface has not opted into
|
|
288
|
+
surface typing, so the obligation stays project-wide for it.
|
|
289
|
+
- Do not create an E2E tree whose only purpose is to receive annotations.
|
|
290
|
+
That is the "convert all obligations into E2E" anti-pattern below.
|
|
45
291
|
- Use `QFAI:SPEC-XXXX:US-YYYY` annotations.
|
|
46
|
-
|
|
47
|
-
|
|
292
|
+
|
|
293
|
+
- Integration obligations (enforced today):
|
|
294
|
+
- Every `TC-*` in specs must be referenced at least once from the directory
|
|
295
|
+
its declared `Level` routes to: L3/Integration -> `tests/integration/**`,
|
|
296
|
+
L4/API -> `tests/api/**`, L5/E2E -> `tests/e2e/**`. A TC with no declared
|
|
297
|
+
`Level` defaults to `tests/integration/**`. **L1/Unit and L2/Component owe
|
|
298
|
+
no reference at all** — see "Unit and Component owe no ATDD annotation"
|
|
299
|
+
below. This is what `QFAI-ATDD-112` checks.
|
|
48
300
|
- Use `QFAI:SPEC-XXXX:TC-YYYY` annotations.
|
|
301
|
+
- A `TC-*` annotation outside the directory its declared `Level` names is
|
|
302
|
+
rejected (`QFAI-ATDD-121` / `QFAI-ATDD-122` / `QFAI-ATDD-123`). The rule is
|
|
303
|
+
`Level`-relative,
|
|
304
|
+
not a blanket ban: a `TC-*` in `tests/api/**` is rejected **unless** that TC
|
|
305
|
+
declares `L4`/`API`, and in `tests/e2e/**` unless it declares `L5`/`E2E` —
|
|
306
|
+
an annotation matching its own `Level` is what discharges the obligation
|
|
307
|
+
there. A `TC-*` should not be at L4 or L5 in the first place (see
|
|
308
|
+
"Annotation routing"): re-file that obligation as `CON-API-*` or `US-*`.
|
|
309
|
+
But while the row exists at that `Level`, its annotation belongs in the one
|
|
310
|
+
directory the `Level` names, and putting it anywhere else leaves the TC
|
|
311
|
+
uncovered as well as forbidden.
|
|
312
|
+
- Every declared `CON-DB-*` must be referenced at least once from
|
|
313
|
+
`tests/integration/**` (`QFAI-ATDD-115`). Use `QFAI:CON-DB-XXXX`
|
|
314
|
+
annotations. L3 owns this because a DB contract is only exercised against
|
|
315
|
+
real infrastructure, which is L3's declared scope; a `CON-DB` reference
|
|
316
|
+
from `tests/e2e/**` is not counted, or an end-to-end assertion that never
|
|
317
|
+
touches the schema could close the obligation. A contract outside the
|
|
318
|
+
current slice defers with a `-- x-qfai-status: planned` comment line,
|
|
319
|
+
reported at `info` by `QFAI-ATDD-116` so the deferral stays visible.
|
|
320
|
+
|
|
321
|
+
- **Unit and Component owe no ATDD annotation.** A `TC-*` whose declared
|
|
322
|
+
`Level` is L1 or L2 is outside `QFAI-ATDD-112` entirely: it is not required
|
|
323
|
+
in any directory, and an annotation for it inside a scanned directory is not
|
|
324
|
+
a violation either. `QFAI-ATDD-117` (`info`) names the excluded TCs on every
|
|
325
|
+
run so the exclusion is visible rather than silent.
|
|
326
|
+
- This is the only reading consistent with the rest of the package.
|
|
327
|
+
`qfai-atdd/SKILL.md` puts Unit and Component out of its scope, and the
|
|
328
|
+
crosswalk above gives L1/L2 no mandated directory — only L3-L5 are
|
|
329
|
+
directory-pinned and only those three roots are scanned.
|
|
330
|
+
- Previously L1/L2 fell through to `tests/integration/**` — the fallback for
|
|
331
|
+
a spec with no `Level` column at all — so every declared Unit and Component
|
|
332
|
+
TC was an `error` demanding an annotation in a directory this file says is
|
|
333
|
+
not its home. `QFAI-WAIVER-002` refuses waivers on `error` rules, so a
|
|
334
|
+
project that filed unit tests where L1's own entry says to had no exit, and
|
|
335
|
+
the only validator-clean path was duplicating every annotation into
|
|
336
|
+
`tests/integration/**` — the all-integration collapse named under
|
|
337
|
+
Anti-patterns below.
|
|
338
|
+
- **They are still gated, by the other stage.** Every coverage-target `TC-*`
|
|
339
|
+
owes a `tdd/test-list.md` row, and `TDDLIST_TC_NOT_COVERED` (`error`)
|
|
340
|
+
reports a missing one. L1/L2 belong to `/qfai-implement`, which is the
|
|
341
|
+
stage that writes unit and component tests.
|
|
49
342
|
- API obligations:
|
|
50
343
|
- Every declared `CON-API-*` must be referenced at least once from `tests/api/**`.
|
|
51
344
|
- Use `QFAI:CON-API-XXXX` annotations.
|
|
345
|
+
- **Deferral.** `/qfai-sdd` authors contracts in Phase 0 (Contracts-first) and
|
|
346
|
+
slices them in Phase 2, so a contract legitimately exists before its slice
|
|
347
|
+
ships. A contract that declares `x-qfai-status: planned` is excluded from
|
|
348
|
+
the `QFAI-ATDD-113` obligation and reported as `QFAI-ATDD-114` (`info`)
|
|
349
|
+
instead. Remove the marker when the slice is implemented — leaving it in
|
|
350
|
+
place on a shipped slice is a review finding, not a tool finding.
|
|
351
|
+
- The marker counts only at the **document root**. The same key nested under
|
|
352
|
+
an operation defers nothing: one path must not be able to drop the
|
|
353
|
+
API-test obligation for every other `CON-API-*` the file declares.
|
|
354
|
+
- Deferral removes the test obligation, not the declaration. A deferred
|
|
355
|
+
`CON-API-*` stays a known ID, so writing its API test ahead of the slice
|
|
356
|
+
is fine and never raises `QFAI-ATDD-103`.
|
|
357
|
+
- Deferred IDs are recorded in `report/atdd-traceability/summary.{json,md}`
|
|
358
|
+
under `deferred.conApi`, so an empty `missing.conApi` can be told apart
|
|
359
|
+
from a project where every contract is still planned.
|
|
52
360
|
- Forbidden references:
|
|
53
|
-
- `tests/api/**` must not include `QFAI:SPEC-XXXX:TC-YYYY
|
|
54
|
-
|
|
361
|
+
- `tests/api/**` must not include `QFAI:SPEC-XXXX:TC-YYYY` **for a TC whose
|
|
362
|
+
declared `Level` is not L4/API**. A TC that declares an API-level
|
|
363
|
+
obligation belongs in `tests/api/**`, and its annotation there counts as
|
|
364
|
+
coverage. The rule exists to stop obligations drifting into the wrong
|
|
365
|
+
layer, not to make the correct layer unusable.
|
|
366
|
+
- `tests/e2e/**` must not include `QFAI:SPEC-XXXX:TC-YYYY` **for a TC whose
|
|
367
|
+
declared `Level` is not L5/E2E**. Same reason as above: the routing rule
|
|
368
|
+
and the forbidden rule must agree, or the layer the routing selects
|
|
369
|
+
becomes unusable.
|
|
370
|
+
- `tests/integration/**` must not include `QFAI:SPEC-XXXX:TC-YYYY` **for a TC
|
|
371
|
+
whose declared `Level` is not L3/Integration** (`QFAI-ATDD-123`). The rule
|
|
372
|
+
is symmetric so "exactly one directory" holds in both directions: an
|
|
373
|
+
annotation left behind here after the TC moved to L4/L5 is as wrong as one
|
|
374
|
+
filed early into `tests/api/**`.
|
|
55
375
|
- Unknown references (`US/TC/CON-API` not declared) are errors.
|
|
376
|
+
- A `TC-*` annotation outside the directory its `Level` routes to is a
|
|
377
|
+
misplacement, whichever directory it lands in. **This applies to L3-L5 only.**
|
|
378
|
+
L1 and L2 route nowhere — they carry no ATDD annotation obligation at all — so
|
|
379
|
+
an annotation for one is neither required nor misplaced, in
|
|
380
|
+
`tests/unit/**`, `tests/component/**` or anywhere else.
|
|
56
381
|
- AC annotations are not required in code; AC coverage is treated as indirect through TC coverage.
|
|
57
382
|
- `QFAI:CON-API-*` in `tests/e2e/**` is not forbidden, but contract guarantee belongs to API tests.
|
|
58
383
|
|
|
384
|
+
## Test-file granularity
|
|
385
|
+
|
|
386
|
+
- Default: **one test module per `TC-*`**. A TDD ledger row's `Test file`
|
|
387
|
+
names that module.
|
|
388
|
+
- Grouping several `TC-*` into one module is allowed when they verify the same
|
|
389
|
+
BR and the module stays reviewable in one pass. Above that, split by BR, then
|
|
390
|
+
by AC.
|
|
391
|
+
- A single `Test file` value shared by every row of a spec is an anti-pattern:
|
|
392
|
+
it makes the per-item "relevant test suite" indistinguishable from the whole
|
|
393
|
+
spec suite, and it puts the whole spec's test code in front of every
|
|
394
|
+
in-context reviewer gate.
|
|
395
|
+
- `qfai-sdd` should emit a per-item `Test file` value, not a per-spec one.
|
|
396
|
+
|
|
59
397
|
## Volume policy
|
|
60
398
|
|
|
61
|
-
- Floors and ratios are signals, not completion gates.
|
|
399
|
+
- Floors and ratios are signals, not completion gates. This is the only
|
|
400
|
+
statement in this section: no volume observation blocks completion, and none
|
|
401
|
+
triggers a Change Request.
|
|
62
402
|
- Completion gate is validation pass with no errors:
|
|
63
|
-
- `qfai validate --fail-on error`
|
|
403
|
+
- `npx qfai validate --fail-on error`
|
|
64
404
|
|
|
65
|
-
If
|
|
405
|
+
If an observed layer distribution looks wrong:
|
|
66
406
|
|
|
67
|
-
1.
|
|
68
|
-
2.
|
|
69
|
-
|
|
70
|
-
|
|
407
|
+
1. Do not auto-adjust the distribution to make it look better.
|
|
408
|
+
2. Record the observed distribution and the rationale for it in the running
|
|
409
|
+
stage's evidence file under `.qfai/evidence/` — for example
|
|
410
|
+
`.qfai/evidence/atdd-<spec-id>.md` or
|
|
411
|
+
`.qfai/evidence/implement-<spec-id>.md`.
|
|
412
|
+
3. Continue. The stage is not blocked.
|
|
413
|
+
|
|
414
|
+
The record goes to evidence, not into the spec, on purpose.
|
|
415
|
+
`constitution/drift-protocol.md` lists `*_delta.md` and the per-spec Open
|
|
416
|
+
Questions file among the upstream SSOT a downstream stage must not edit without
|
|
417
|
+
explicit user approval, and whitelists `.qfai/evidence/**` append/update as an
|
|
418
|
+
allowed exception. Pointing this step at the spec would send ATDD and implement
|
|
419
|
+
straight back into the STOP-and-wait state the policy exists to avoid.
|
|
420
|
+
|
|
421
|
+
The owner phase (`/qfai-sdd`) is the one that may carry the note into the
|
|
422
|
+
spec's own Open Questions file on a later run: `08_Open-questions.md` in a
|
|
423
|
+
layered spec, `15_Open-questions.md` in a spec pack.
|
|
424
|
+
(`09_Open-questions.md` is the shared `_policies` file, not a per-spec one.)
|
|
425
|
+
|
|
426
|
+
A Change Request is reserved for `constitution/drift-protocol.md`-class events
|
|
427
|
+
— an actual conflict with an upstream SSOT decision. A volume observation is not
|
|
428
|
+
one, and it stays non-blocking either way. What differs is how much of it is
|
|
429
|
+
measurable:
|
|
430
|
+
|
|
431
|
+
- **No configured guardrail.** qfai ships no default floor, ratio or threshold,
|
|
432
|
+
and no validator emits a volume rule, so "unmet" is a judgement call with no
|
|
433
|
+
tool-checkable meaning. Record the observation and the reasoning.
|
|
434
|
+
- **A configured guardrail.** When a project sets
|
|
435
|
+
`validation.testStrategy.maxE2eScenarioRatio` or `maxE2eScenarioCount` to a
|
|
436
|
+
non-null value, `report.ts` measures it and reports `ratioExceeded` /
|
|
437
|
+
`countExceeded` with a warning. That is the project's own stated limit, not a
|
|
438
|
+
subjective read: record the configured value, the measured value and the
|
|
439
|
+
report warning in the evidence, so the breach is auditable rather than
|
|
440
|
+
paraphrased.
|
|
441
|
+
|
|
442
|
+
**What it counts is narrow.** These two knobs measure **Gherkin scenarios
|
|
443
|
+
parsed out of each spec's Examples file**, bucketed by their `@layer-*` tags.
|
|
444
|
+
They never inspect `<testsDir>/e2e/**` or any other code test. A project that
|
|
445
|
+
writes no Gherkin — the normal shape for a layered project whose E2E lives in
|
|
446
|
+
code — measures zero scenarios, so the guardrail stays silent however many
|
|
447
|
+
code E2E tests exist. Treat them as a guardrail over the **scenario**
|
|
448
|
+
distribution, not over the real test-layer distribution: when they are silent,
|
|
449
|
+
the observation is the judgement call above, and the evidence entry must say
|
|
450
|
+
how the distribution was counted so it is not read as a tool measurement.
|
|
451
|
+
|
|
452
|
+
Either way completion is not blocked and no Change Request is raised: a
|
|
453
|
+
user-blocking Change Request against a project's own tuning knob cannot conclude
|
|
454
|
+
anything actionable.
|
|
71
455
|
|
|
72
456
|
## Anti-patterns
|
|
73
457
|
|
|
74
458
|
- Do not treat `scenario.feature` or a coverage ledger as mandatory completion input.
|
|
75
459
|
- Do not convert all obligations into E2E.
|
|
76
460
|
- Do not inflate tests only to satisfy floor numbers.
|
|
461
|
+
- Do not over-concentrate obligations into a single layer, module, or selector. Collapsing a matrix-shaped `TC-*` into one integration module — or into one test function behind one `test-list.md` selector — is the same failure mode as converting everything into E2E, and it additionally destroys the RED observation: a test function fails once, so only the first failing assert is ever observed. Split per independently observable boundary (see `qfai-implement/references/execution-ledger.md#selector-granularity-must`).
|
|
462
|
+
- Do not re-label an existing obligation's declared layer to change how a
|
|
463
|
+
distribution reads. Re-labelling is the cheapest way to clear a signal and
|
|
464
|
+
the one that destroys the most information; the layer of an obligation is
|
|
465
|
+
determined by what it verifies, never by how the totals look.
|
|
466
|
+
|
|
467
|
+
### Concentration signals (non-gating)
|
|
468
|
+
|
|
469
|
+
Treat these as review signals in the same class as volume floors — worth a finding, never a hard gate:
|
|
470
|
+
|
|
471
|
+
- one test module holding a disproportionate share of a spec's `assert` statements
|
|
472
|
+
- a very low `test_` functions per file ratio in a module that carries many obligations
|
|
473
|
+
- a single selector whose recorded runtime grows monotonically across RED rounds
|
|
474
|
+
|
|
475
|
+
## Test stub detection (QFAI-TEST-001 / QFAI-TEST-002)
|
|
476
|
+
|
|
477
|
+
`QFAI-TEST-001` (error) reports the silent-placeholder construct of each
|
|
478
|
+
supported stack:
|
|
479
|
+
|
|
480
|
+
| Extensions | Construct |
|
|
481
|
+
| -------------------- | ------------------------------------------------------------------- |
|
|
482
|
+
| `.ts` / `.js` family | `it.todo(` / `test.todo(` / `describe.todo(` |
|
|
483
|
+
| `.py` | `pytest.skip(`, `@pytest.mark.skip/skipif/xfail`, `@unittest.skip*` |
|
|
484
|
+
| `.go` | `t.Skip*(` |
|
|
485
|
+
| `.java` / `.kt` | `@Disabled` / `@Ignore` |
|
|
486
|
+
| `.rs` | `#[ignore]` |
|
|
487
|
+
| `.rb` | a line starting `skip` / `pending` |
|
|
488
|
+
| `.cs` | `[Ignore` / `Skip = "` |
|
|
489
|
+
|
|
490
|
+
`QFAI-TEST-002` (info) names any extension the scan opened that has no dialect.
|
|
491
|
+
Without it a clean run on an unsupported stack is indistinguishable from a
|
|
492
|
+
checked one — the detector used to be JS-only while file selection was
|
|
493
|
+
stack-agnostic, so every other stack got a clean result that meant nothing.
|