universal-dev-standards 5.7.3 → 5.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/ai/language-packs/README.md +55 -0
- package/bundled/ai/language-packs/language-pack-php-to-csharp.ai.yaml +83 -0
- package/bundled/ai/standards/acceptance-criteria-traceability.ai.yaml +49 -3
- package/bundled/ai/standards/behavior-snapshot.ai.yaml +272 -0
- package/bundled/ai/standards/deployment-standards.ai.yaml +108 -3
- package/bundled/ai/standards/feature-discovery-standards.ai.yaml +459 -0
- package/bundled/ai/standards/feature-manifest-standard.ai.yaml +149 -0
- package/bundled/ai/standards/mock-boundary.ai.yaml +93 -2
- package/bundled/ai/standards/multi-environment-e2e-testing.ai.yaml +250 -0
- package/bundled/ai/standards/test-completeness-dimensions.ai.yaml +39 -7
- package/bundled/ai/standards/verification-evidence.ai.yaml +30 -4
- package/bundled/core/acceptance-criteria-traceability.md +41 -6
- package/bundled/core/behavior-snapshot.md +239 -0
- package/bundled/core/feature-manifest-standard.md +213 -0
- package/bundled/locales/zh-CN/CHANGELOG.md +37 -3
- package/bundled/locales/zh-CN/README.md +1 -1
- package/bundled/locales/zh-TW/CHANGELOG.md +37 -3
- package/bundled/locales/zh-TW/README.md +1 -1
- package/bundled/skills/brainstorm-assistant/SKILL.md +183 -30
- package/bundled/skills/brainstorm-assistant/guide.md +607 -117
- package/package.json +4 -4
- package/src/commands/update.js +4 -1
- package/standards-registry.json +53 -5
|
@@ -3,12 +3,86 @@
|
|
|
3
3
|
|
|
4
4
|
id: mock-boundary
|
|
5
5
|
meta:
|
|
6
|
-
version: "1.
|
|
7
|
-
updated: "2026-05-
|
|
6
|
+
version: "1.1.0"
|
|
7
|
+
updated: "2026-05-13"
|
|
8
8
|
source: core/mock-boundary.md
|
|
9
9
|
description: >
|
|
10
10
|
Rules defining what can and cannot be mocked to prevent hollow tests —
|
|
11
11
|
tests that pass while the real system is broken.
|
|
12
|
+
v1.1.0: Added Level 1/Level 2 mock layer distinction and external
|
|
13
|
+
dependency testability matrix template (XSPEC-204).
|
|
14
|
+
|
|
15
|
+
# ─────────────────────────────────────────────────────────
|
|
16
|
+
# Mock Levels (v1.1.0)
|
|
17
|
+
# ─────────────────────────────────────────────────────────
|
|
18
|
+
mock_levels:
|
|
19
|
+
description: >
|
|
20
|
+
Two distinct levels of test doubles with separate governance rules.
|
|
21
|
+
Conflating them leads to the "STUB blocked ∩ real service unavailable = untestable"
|
|
22
|
+
deadlock in UAT environments.
|
|
23
|
+
|
|
24
|
+
level_1:
|
|
25
|
+
name: Code-level Mock
|
|
26
|
+
description: >
|
|
27
|
+
Test doubles within test code — jest.mock(), Moq.Mock<T>(), vi.fn(), etc.
|
|
28
|
+
governance: "Regulated by STUB marker rules (full-coverage-testing)"
|
|
29
|
+
examples:
|
|
30
|
+
- "vi.mock('../../src/services/sms-service.js')"
|
|
31
|
+
- "new Mock<ISmsService>()"
|
|
32
|
+
- "httpClient.Setup(x => x.PostAsync(...)).ReturnsAsync(...)"
|
|
33
|
+
rules:
|
|
34
|
+
- "Subject to // WARNING: STUB deployment-blocking rules"
|
|
35
|
+
- "Not allowed in flow/integration/E2E tests (see forbidden section)"
|
|
36
|
+
- "STUB marker must be placed at usage site"
|
|
37
|
+
|
|
38
|
+
level_2:
|
|
39
|
+
name: Infrastructure-level Stub Server
|
|
40
|
+
description: >
|
|
41
|
+
Independently running test-double services — WireMock, MockSoap, stub-server, json-server, etc.
|
|
42
|
+
These run as separate processes alongside the application under test.
|
|
43
|
+
governance: "Regulated by environment stratification rules (multi-environment-e2e-testing)"
|
|
44
|
+
examples:
|
|
45
|
+
- "WireMock running on port 9999, app configured via env var to hit it"
|
|
46
|
+
- "MockSoap sidecar for SOAP gateway simulation"
|
|
47
|
+
- "json-server for external REST API simulation"
|
|
48
|
+
rules:
|
|
49
|
+
- "NOT subject to STUB deployment-blocking rules (they are infrastructure, not code)"
|
|
50
|
+
- "MAY be used in UAT environment when real external service is unavailable"
|
|
51
|
+
- "MUST be documented in the external-dependency-testability-matrix as ⚠️"
|
|
52
|
+
- "MUST list dimensions NOT verifiable through the stub"
|
|
53
|
+
- "MUST NOT be deployed to or accessible from PRD environment"
|
|
54
|
+
- "MUST be deployed as sidecar (not bundled in production artifact)"
|
|
55
|
+
|
|
56
|
+
level_2_rationale: >
|
|
57
|
+
When full-coverage-testing blocks STUB markers AND the real external service
|
|
58
|
+
is unavailable in UAT (e.g., SMS gateway requires billing setup), Level 2
|
|
59
|
+
stub servers are the correct solution. They allow the flow to execute in UAT
|
|
60
|
+
while clearly documenting which verification dimensions are deferred to PRD.
|
|
61
|
+
|
|
62
|
+
# ─────────────────────────────────────────────────────────
|
|
63
|
+
# External Dependency Testability Matrix Template
|
|
64
|
+
# ─────────────────────────────────────────────────────────
|
|
65
|
+
external_dependency_testability_matrix:
|
|
66
|
+
rule: >
|
|
67
|
+
Projects with external service dependencies MUST maintain this matrix.
|
|
68
|
+
It answers: "In this environment, can we fully verify this service integration?"
|
|
69
|
+
location: "docs/testing/dependency-testability-matrix.md"
|
|
70
|
+
when_required: "Any project with SMS, payment, IdP, messaging, or external SOAP/REST dependencies"
|
|
71
|
+
|
|
72
|
+
template: |
|
|
73
|
+
## External Dependency Testability Matrix
|
|
74
|
+
|
|
75
|
+
| External Service | local-dev | UAT | PRD | Unverifiable Dimensions in UAT |
|
|
76
|
+
|-----------------|:---------:|:---:|:---:|-------------------------------|
|
|
77
|
+
| SMS Gateway | ⚠️ L2-stub | ⚠️ L2-stub / ❌ | ✅ | Billing correctness, DR reporting, carrier delivery |
|
|
78
|
+
| Payment SOAP | ⚠️ L2-stub | ⚠️/❌ | ✅ | Real debit/credit, bank reconciliation, card validation |
|
|
79
|
+
| LDAP / AD | ⚠️ container | ⚠️ AD LDS | ✅ | Enterprise AD tree, group sync, GPO policies |
|
|
80
|
+
| Push Notification | ⚠️ mock | ⚠️ sandbox | ✅ | Production delivery rates, carrier-specific behavior |
|
|
81
|
+
|
|
82
|
+
Legend:
|
|
83
|
+
✅ Full verification | ⚠️ Flow passes via stub (real-world dimensions deferred) | ❌ Cannot test
|
|
84
|
+
|
|
85
|
+
note: "L2-stub = Level 2 infrastructure stub server. Refer to mock-boundary Level 2 rules."
|
|
12
86
|
|
|
13
87
|
# ─────────────────────────────────────────────────────────
|
|
14
88
|
# Core Problem
|
|
@@ -223,6 +297,23 @@ rules:
|
|
|
223
297
|
If any hollow test indicator triggers, rewrite the mocking strategy.
|
|
224
298
|
priority: required
|
|
225
299
|
|
|
300
|
+
- id: level-2-stub-server-rules
|
|
301
|
+
trigger: UAT environment lacks a real external service (SMS, payment, IdP)
|
|
302
|
+
instruction: >
|
|
303
|
+
Use a Level 2 infrastructure stub server (WireMock, MockSoap, etc.) deployed
|
|
304
|
+
as a sidecar — not Level 1 code mocks, which are blocked in UAT by STUB rules.
|
|
305
|
+
Document the stub server in the external-dependency-testability-matrix as ⚠️
|
|
306
|
+
and list which verification dimensions are deferred to PRD.
|
|
307
|
+
priority: required
|
|
308
|
+
|
|
309
|
+
- id: no-stub-server-in-prd
|
|
310
|
+
trigger: deploying to production
|
|
311
|
+
instruction: >
|
|
312
|
+
Production artifact MUST NOT include stub server code.
|
|
313
|
+
Stub servers MUST NOT be running or accessible in PRD environment.
|
|
314
|
+
See deployment-standards for CI/CD exclusion rules.
|
|
315
|
+
priority: required
|
|
316
|
+
|
|
226
317
|
# ─────────────────────────────────────────────────────────
|
|
227
318
|
# Quick Reference
|
|
228
319
|
# ─────────────────────────────────────────────────────────
|
|
@@ -0,0 +1,250 @@
|
|
|
1
|
+
# Multi-Environment E2E Testing Standards - AI Optimized
|
|
2
|
+
# Source: XSPEC-204 (UDS Issue #94, #95)
|
|
3
|
+
|
|
4
|
+
id: multi-environment-e2e-testing
|
|
5
|
+
meta:
|
|
6
|
+
version: "1.0.0"
|
|
7
|
+
updated: "2026-05-13"
|
|
8
|
+
description: >
|
|
9
|
+
Standards for E2E test configuration across multiple deployment targets.
|
|
10
|
+
Core principle: The run command IS the documentation.
|
|
11
|
+
Each environment has one entry-point script that self-checks prerequisites
|
|
12
|
+
and runs the correct test subset.
|
|
13
|
+
|
|
14
|
+
# ─────────────────────────────────────────────────────────
|
|
15
|
+
# Core Principle
|
|
16
|
+
# ─────────────────────────────────────────────────────────
|
|
17
|
+
core_principle:
|
|
18
|
+
name: Executable Environment Documentation
|
|
19
|
+
statement: "The run command IS the documentation."
|
|
20
|
+
rationale: >
|
|
21
|
+
Developers repeatedly ask "how do I run tests against environment X?"
|
|
22
|
+
because the answer is scattered across README, wiki, and verbal knowledge.
|
|
23
|
+
When the run script self-checks prerequisites and sets the correct BASE_URL,
|
|
24
|
+
the script itself becomes the authoritative environment setup guide.
|
|
25
|
+
anti_pattern: >
|
|
26
|
+
Manually changing BASE_URL in .env before test runs — breaks other
|
|
27
|
+
developers' configs and cannot be committed without affecting everyone.
|
|
28
|
+
|
|
29
|
+
# ─────────────────────────────────────────────────────────
|
|
30
|
+
# Test Framework Multi-Environment Config
|
|
31
|
+
# ─────────────────────────────────────────────────────────
|
|
32
|
+
test_framework_config:
|
|
33
|
+
rule: "BASE_URL is baked into test project config, not read from .env"
|
|
34
|
+
rationale: >
|
|
35
|
+
If BASE_URL comes from .env, developers must modify .env before running
|
|
36
|
+
against a specific environment, creating race conditions in teams and
|
|
37
|
+
accidental commits of wrong URLs.
|
|
38
|
+
|
|
39
|
+
playwright_pattern: |
|
|
40
|
+
// playwright.config.ts
|
|
41
|
+
const ENVS = {
|
|
42
|
+
'local-iis': 'http://localhost/corp',
|
|
43
|
+
'local-iis-express': 'http://localhost:18080/lotest',
|
|
44
|
+
'uat': 'http://portal_uat.example.com/app',
|
|
45
|
+
'prd': 'https://app.example.com',
|
|
46
|
+
} as const;
|
|
47
|
+
|
|
48
|
+
export default defineConfig({
|
|
49
|
+
projects: Object.entries(ENVS).map(([name, url]) => ({
|
|
50
|
+
name,
|
|
51
|
+
use: { browserName: 'chromium', baseURL: url },
|
|
52
|
+
})),
|
|
53
|
+
});
|
|
54
|
+
|
|
55
|
+
cypress_pattern: |
|
|
56
|
+
// cypress.config.ts
|
|
57
|
+
const ENVS = {
|
|
58
|
+
'local': { baseUrl: 'http://localhost:3000' },
|
|
59
|
+
'uat': { baseUrl: 'http://uat.example.com' },
|
|
60
|
+
'prd': { baseUrl: 'https://app.example.com' },
|
|
61
|
+
};
|
|
62
|
+
// Pass via --env target=uat; default to local
|
|
63
|
+
|
|
64
|
+
rules:
|
|
65
|
+
- "One test config file; environment distinction via project name or --project flag"
|
|
66
|
+
- "No environment-specific playwright/cypress.config.*.ts files"
|
|
67
|
+
- "BASE_URL never in .env for E2E test configs"
|
|
68
|
+
|
|
69
|
+
# ─────────────────────────────────────────────────────────
|
|
70
|
+
# Runner Script Pattern
|
|
71
|
+
# ─────────────────────────────────────────────────────────
|
|
72
|
+
runner_script_pattern:
|
|
73
|
+
rule: "Each environment has one entry-point script that self-checks prerequisites"
|
|
74
|
+
location: "scripts/run-tests-<env>.(ps1|sh)"
|
|
75
|
+
template_powershell: |
|
|
76
|
+
# scripts/run-tests-local-iis.ps1
|
|
77
|
+
# Self-check prerequisites before running tests
|
|
78
|
+
|
|
79
|
+
# 1. Check Docker
|
|
80
|
+
if (-not (Get-Process "com.docker.backend" -ErrorAction SilentlyContinue)) {
|
|
81
|
+
Write-Error "Docker is not running. Start Docker Desktop first."
|
|
82
|
+
exit 1
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
# 2. Check App is responding
|
|
86
|
+
try {
|
|
87
|
+
$resp = Invoke-WebRequest "http://localhost/corp/health" -UseBasicParsing -TimeoutSec 5
|
|
88
|
+
} catch {
|
|
89
|
+
Write-Error "App not responding at http://localhost/corp. Start IIS site first."
|
|
90
|
+
exit 1
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
# 3. Run tests
|
|
94
|
+
npx playwright test --project=local-iis @args
|
|
95
|
+
|
|
96
|
+
template_bash: |
|
|
97
|
+
#!/bin/bash
|
|
98
|
+
# scripts/run-tests-local.sh
|
|
99
|
+
|
|
100
|
+
# 1. Check Docker
|
|
101
|
+
if ! docker info > /dev/null 2>&1; then
|
|
102
|
+
echo "ERROR: Docker is not running." >&2
|
|
103
|
+
exit 1
|
|
104
|
+
fi
|
|
105
|
+
|
|
106
|
+
# 2. Check App
|
|
107
|
+
if ! curl -s -f http://localhost:3000/health > /dev/null 2>&1; then
|
|
108
|
+
echo "ERROR: App not responding at http://localhost:3000" >&2
|
|
109
|
+
exit 1
|
|
110
|
+
fi
|
|
111
|
+
|
|
112
|
+
npx playwright test --project=local "$@"
|
|
113
|
+
|
|
114
|
+
prerequisite_checks:
|
|
115
|
+
- "Docker / container runtime"
|
|
116
|
+
- "Application health endpoint"
|
|
117
|
+
- "Database connectivity (if separate from app)"
|
|
118
|
+
- "Required environment-specific services"
|
|
119
|
+
|
|
120
|
+
# ─────────────────────────────────────────────────────────
|
|
121
|
+
# Environment Capability Matrix
|
|
122
|
+
# ─────────────────────────────────────────────────────────
|
|
123
|
+
capability_matrix:
|
|
124
|
+
rule: >
|
|
125
|
+
Projects with external dependencies MUST maintain an environment capability
|
|
126
|
+
matrix committed to the repository (docs/testing/environment-capability-matrix.md
|
|
127
|
+
or inline in testing README).
|
|
128
|
+
when_required: "Any project with external HTTP services, IdP, payment, messaging"
|
|
129
|
+
|
|
130
|
+
template: |
|
|
131
|
+
## Environment Capability Matrix
|
|
132
|
+
|
|
133
|
+
| Service / Feature | local-dev | local-iis | UAT | PRD |
|
|
134
|
+
|-------------------|:---------:|:---------:|:---:|:---:|
|
|
135
|
+
| Auth / SAML | ⚠️ Keycloak stub | ✅ Keycloak local | ✅ Keycloak UAT | ✅ Enterprise IdP |
|
|
136
|
+
| SMS Gateway | ⚠️ stub-server | ⚠️ stub-server | ⚠️ stub-server / ❌ | ✅ Real Gateway + billing |
|
|
137
|
+
| Payment / Finance | ⚠️ stub-server | ⚠️ stub-server | ⚠️ partial | ✅ Real + reconciliation |
|
|
138
|
+
| Background Jobs | ✅ in-process | ✅ in-process | ✅ | ✅ |
|
|
139
|
+
| File Storage | ✅ local | ✅ local | ✅ blob | ✅ blob |
|
|
140
|
+
|
|
141
|
+
Legend:
|
|
142
|
+
✅ Full verification possible
|
|
143
|
+
⚠️ Flow passes but through stub (real-world dimensions NOT verified)
|
|
144
|
+
❌ Cannot test in this environment
|
|
145
|
+
|
|
146
|
+
### Dimensions NOT verifiable in UAT (must defer to PRD smoke)
|
|
147
|
+
- SMS: Billing correctness, carrier delivery confirmation, DR reporting
|
|
148
|
+
- Payment: Real debit/credit, bank reconciliation, card validation
|
|
149
|
+
|
|
150
|
+
when_to_update: "Update matrix when adding any new external service dependency"
|
|
151
|
+
|
|
152
|
+
# ─────────────────────────────────────────────────────────
|
|
153
|
+
# CI Gate Mapping
|
|
154
|
+
# ─────────────────────────────────────────────────────────
|
|
155
|
+
ci_gate_mapping:
|
|
156
|
+
rule: "Map environments to CI/CD stages; document which gate must pass before each deployment stage"
|
|
157
|
+
|
|
158
|
+
pattern: |
|
|
159
|
+
# .github/workflows/ci.yml or .gitlab-ci.yml
|
|
160
|
+
|
|
161
|
+
e2e-smoke-gate: # Must pass → any deployment
|
|
162
|
+
runs-on: ubuntu-latest
|
|
163
|
+
script: scripts/run-tests-local.sh --grep smoke
|
|
164
|
+
|
|
165
|
+
e2e-uat-gate: # Must pass → PRD deployment
|
|
166
|
+
environment: uat
|
|
167
|
+
script: scripts/run-tests-uat.sh
|
|
168
|
+
only: [tags, release-branches]
|
|
169
|
+
|
|
170
|
+
e2e-prd-smoke: # Must pass → mark release as stable
|
|
171
|
+
environment: prd
|
|
172
|
+
script: scripts/run-tests-prd-smoke.sh
|
|
173
|
+
only: [tags]
|
|
174
|
+
|
|
175
|
+
gate_requirements:
|
|
176
|
+
before_staging_deploy: ["unit tests", "integration tests", "e2e-smoke-gate"]
|
|
177
|
+
before_uat_deploy: ["all staging gates", "e2e-uat-gate (if UAT environment available)"]
|
|
178
|
+
before_prd_deploy: ["e2e-uat-gate", "sign-off from capability matrix review"]
|
|
179
|
+
after_prd_deploy: ["e2e-prd-smoke within 10 min of deploy"]
|
|
180
|
+
|
|
181
|
+
# ─────────────────────────────────────────────────────────
|
|
182
|
+
# Credential Handling
|
|
183
|
+
# ─────────────────────────────────────────────────────────
|
|
184
|
+
credential_handling:
|
|
185
|
+
rule: "Separate what goes in git from what stays gitignored"
|
|
186
|
+
|
|
187
|
+
commit_to_git:
|
|
188
|
+
- "Base URLs per environment (non-secret, team needs to share)"
|
|
189
|
+
- "Test usernames for non-PRD environments"
|
|
190
|
+
- "Feature flags and test configuration"
|
|
191
|
+
- "Self-check scripts (no credentials embedded)"
|
|
192
|
+
|
|
193
|
+
gitignore:
|
|
194
|
+
- "Passwords and secrets"
|
|
195
|
+
- "API keys and tokens"
|
|
196
|
+
- ".env.test.local (personal overrides)"
|
|
197
|
+
|
|
198
|
+
ci_secrets: "Pass PRD test passwords via CI secret variables (GitHub Secrets / GitLab CI Variables)"
|
|
199
|
+
|
|
200
|
+
example_gitignore: |
|
|
201
|
+
# Test credentials
|
|
202
|
+
.env.test.local
|
|
203
|
+
tests/fixtures/auth-secrets.json
|
|
204
|
+
# Base URLs and non-secrets are committed; see playwright.config.ts
|
|
205
|
+
|
|
206
|
+
# ─────────────────────────────────────────────────────────
|
|
207
|
+
# Rules
|
|
208
|
+
# ─────────────────────────────────────────────────────────
|
|
209
|
+
rules:
|
|
210
|
+
- id: base-url-in-config
|
|
211
|
+
trigger: setting up E2E tests for a project with multiple environments
|
|
212
|
+
instruction: >
|
|
213
|
+
Define all environment BASE_URLs in the test framework config (playwright.config.ts /
|
|
214
|
+
cypress.config.ts) as named projects. Do not rely on .env for BASE_URL.
|
|
215
|
+
priority: required
|
|
216
|
+
|
|
217
|
+
- id: one-runner-per-env
|
|
218
|
+
trigger: adding a new deployment target
|
|
219
|
+
instruction: >
|
|
220
|
+
Create scripts/run-tests-<env>.(ps1|sh) with self-checking prerequisite steps.
|
|
221
|
+
The script must verify all required services before invoking the test runner.
|
|
222
|
+
priority: required
|
|
223
|
+
|
|
224
|
+
- id: capability-matrix-required
|
|
225
|
+
trigger: feature has external service dependencies (SMS, payment, IdP, file storage)
|
|
226
|
+
instruction: >
|
|
227
|
+
Create or update the environment capability matrix in docs/testing/.
|
|
228
|
+
Clearly mark ✅/⚠️/❌ for each service × environment combination.
|
|
229
|
+
List dimensions NOT verifiable in UAT; mark these as "PRD-only smoke" items.
|
|
230
|
+
priority: required
|
|
231
|
+
|
|
232
|
+
- id: ci-gate-mapping
|
|
233
|
+
trigger: defining CI/CD pipeline stages
|
|
234
|
+
instruction: >
|
|
235
|
+
Map each CI gate to the environments it must pass.
|
|
236
|
+
E2E gate must specify which environment it targets.
|
|
237
|
+
priority: required
|
|
238
|
+
|
|
239
|
+
# ─────────────────────────────────────────────────────────
|
|
240
|
+
# Relationship to Other Standards
|
|
241
|
+
# ─────────────────────────────────────────────────────────
|
|
242
|
+
related_standards:
|
|
243
|
+
- id: deployment-standards
|
|
244
|
+
relationship: "Extends — adds environment dimension to CI gates and deployment readiness"
|
|
245
|
+
- id: test-completeness-dimensions
|
|
246
|
+
relationship: "Complements — adds dimension 11: Environment Verifiability"
|
|
247
|
+
- id: verification-evidence
|
|
248
|
+
relationship: "Complements — evidence must specify which environment it was collected from"
|
|
249
|
+
- id: mock-boundary
|
|
250
|
+
relationship: "Complements — capability matrix references Level 2 stub server usage"
|
|
@@ -3,10 +3,12 @@
|
|
|
3
3
|
|
|
4
4
|
id: test-completeness-dimensions
|
|
5
5
|
meta:
|
|
6
|
-
version: "1.
|
|
7
|
-
updated: "2026-05-
|
|
6
|
+
version: "1.3.0"
|
|
7
|
+
updated: "2026-05-13"
|
|
8
8
|
source: core/test-completeness-dimensions.md
|
|
9
|
-
description:
|
|
9
|
+
description: >
|
|
10
|
+
Framework for evaluating test completeness across 11 dimensions.
|
|
11
|
+
v1.3.0 adds Dimension 11: Environment Verifiability (XSPEC-204).
|
|
10
12
|
|
|
11
13
|
dimensions:
|
|
12
14
|
- id: 1
|
|
@@ -114,6 +116,21 @@ dimensions:
|
|
|
114
116
|
when_required: When flow has any conditional logic or decision points
|
|
115
117
|
note: Use decision_table_expansion from flow-based-testing.ai.yaml to enumerate scenarios
|
|
116
118
|
|
|
119
|
+
- id: 11
|
|
120
|
+
name: Environment Verifiability
|
|
121
|
+
description: >
|
|
122
|
+
For each AC with external service dependencies, document which environment layer
|
|
123
|
+
can fully verify it, and plan PRD-smoke coverage for items that cannot be verified in UAT.
|
|
124
|
+
test_items:
|
|
125
|
+
- Environment stratification responsibility matrix exists for features with external dependencies
|
|
126
|
+
- Each AC is tagged with its minimum verifiable environment layer (local / UAT / PRD)
|
|
127
|
+
- ACs that are PRD-only are explicitly tracked and have a PRD smoke test plan
|
|
128
|
+
- Evidence for externally-dependent ACs includes environment_layer field
|
|
129
|
+
when_required: >
|
|
130
|
+
Any feature with external service dependencies (SMS, payment, IdP, email,
|
|
131
|
+
SOAP integrations, external REST APIs)
|
|
132
|
+
note: See multi-environment-e2e-testing.ai.yaml and deployment-standards.ai.yaml for environment capability matrix templates
|
|
133
|
+
|
|
117
134
|
feature_type_mapping:
|
|
118
135
|
note: "Dimension 8 (AI Generation Quality) applies when tests are AI-generated"
|
|
119
136
|
types:
|
|
@@ -138,8 +155,9 @@ feature_type_mapping:
|
|
|
138
155
|
with_ai: [1, 3, 5, 8]
|
|
139
156
|
|
|
140
157
|
- type: External Integration
|
|
141
|
-
dimensions: [1, 3, 7]
|
|
142
|
-
with_ai: [1, 3, 7, 8]
|
|
158
|
+
dimensions: [1, 3, 7, 11]
|
|
159
|
+
with_ai: [1, 3, 7, 8, 11]
|
|
160
|
+
note: Dimension 11 required for any external service dependency
|
|
143
161
|
|
|
144
162
|
- type: Workflow / Multi-step Process
|
|
145
163
|
dimensions: [1, 3, 4, 5, 9, 10]
|
|
@@ -147,6 +165,11 @@ feature_type_mapping:
|
|
|
147
165
|
required_pattern: journey-chained-test
|
|
148
166
|
note: Apply flow-based-testing standard; use shared ctx; Each-Choice minimum branch coverage
|
|
149
167
|
|
|
168
|
+
- type: External-Dependent Workflow
|
|
169
|
+
dimensions: [1, 3, 4, 5, 9, 10, 11]
|
|
170
|
+
with_ai: [1, 3, 4, 5, 9, 10, 8, 11]
|
|
171
|
+
note: Combines Workflow + External Integration; Dimension 11 mandatory
|
|
172
|
+
|
|
150
173
|
error_code_coverage:
|
|
151
174
|
- code: 200
|
|
152
175
|
meaning: Success
|
|
@@ -198,7 +221,7 @@ anti_patterns:
|
|
|
198
221
|
rules:
|
|
199
222
|
- id: use-checklist
|
|
200
223
|
trigger: designing tests for a feature
|
|
201
|
-
instruction: Use the
|
|
224
|
+
instruction: Use the 11 dimensions checklist to ensure completeness (dim 8 for AI-generated tests; dim 9/10 for multi-step workflows; dim 11 for external dependencies)
|
|
202
225
|
priority: required
|
|
203
226
|
|
|
204
227
|
- id: authorization-matrix
|
|
@@ -278,6 +301,12 @@ checklist_template: |
|
|
|
278
301
|
□ Each distinct error branch has its own describe block
|
|
279
302
|
□ Critical flows (auth/payment/security) use All-Combinations
|
|
280
303
|
|
|
304
|
+
□ Environment Verifiability (for features with external dependencies)
|
|
305
|
+
□ Environment stratification matrix created/updated
|
|
306
|
+
□ Each AC tagged with minimum verifiable environment layer
|
|
307
|
+
□ PRD-only ACs documented with smoke test plan
|
|
308
|
+
□ Evidence includes environment_layer field for externally-dependent ACs
|
|
309
|
+
|
|
281
310
|
quick_reference:
|
|
282
311
|
dimensions:
|
|
283
312
|
columns: [ID, Dimension, Key Focus]
|
|
@@ -292,9 +321,10 @@ quick_reference:
|
|
|
292
321
|
- [8, AI Generation, AI-generated test quality]
|
|
293
322
|
- [9, Flow Completeness, Complete path to each terminal state]
|
|
294
323
|
- [10, Branch Coverage, All decision-point branches covered]
|
|
324
|
+
- [11, Environment Verifiability, External-dep ACs tagged with environment layer]
|
|
295
325
|
|
|
296
326
|
feature_dimensions:
|
|
297
|
-
note: "*Dimension 8 applies when tests are AI-generated; †Dimensions 9/10 apply to multi-step workflows"
|
|
327
|
+
note: "*Dimension 8 applies when tests are AI-generated; †Dimensions 9/10 apply to multi-step workflows; ‡Dimension 11 applies to features with external service dependencies"
|
|
298
328
|
columns: [Feature Type, Required Dimensions]
|
|
299
329
|
rows:
|
|
300
330
|
- [CRUD API, "1,2,3,4,6,7,8*"]
|
|
@@ -304,3 +334,5 @@ quick_reference:
|
|
|
304
334
|
- [Background Job, "1,3,5,8*"]
|
|
305
335
|
- [External Integration, "1,3,7,8*"]
|
|
306
336
|
- [Workflow / Multi-step Process, "1,3,4,5,9†,10†,8*"]
|
|
337
|
+
- [External Integration, "1,3,7,11‡,8*"]
|
|
338
|
+
- [External-Dependent Workflow, "1,3,4,5,9†,10†,11‡,8*"]
|
|
@@ -8,15 +8,19 @@ standard:
|
|
|
8
8
|
description: 驗證證據標準,強化 anti-hallucination
|
|
9
9
|
|
|
10
10
|
meta:
|
|
11
|
-
version: "1.
|
|
12
|
-
updated: "2026-
|
|
11
|
+
version: "1.1.0"
|
|
12
|
+
updated: "2026-05-13"
|
|
13
13
|
source: core/verification-evidence.md
|
|
14
|
-
description:
|
|
14
|
+
description: >
|
|
15
|
+
驗證證據標準 — Iron Law: 無驗證證據不可聲稱完成。
|
|
16
|
+
v1.1.0: Evidence must specify which environment layer it was collected from (XSPEC-204).
|
|
15
17
|
inspired_by: superpowers/verification-before-completion
|
|
16
18
|
|
|
17
19
|
guidelines:
|
|
18
20
|
- "Iron Law:無驗證證據 = 不可聲稱完成"
|
|
19
21
|
- "每次驗證必須記錄 command + exitCode + output + timestamp"
|
|
22
|
+
- "Iron Law(Environment):驗收前必須確認「此環境層次能驗證此流程的哪些維度」"
|
|
23
|
+
- "驗收證據必須標明收集自哪個環境層次(local / UAT / PRD)"
|
|
20
24
|
- "回歸測試必須展示 RED → GREEN 循環"
|
|
21
25
|
- "代理報告 success ≠ 實際 success,需獨立驗證"
|
|
22
26
|
- "驗證輸出截斷至合理長度(2000 字元)但保留關鍵資訊"
|
|
@@ -39,6 +43,13 @@ standard:
|
|
|
39
43
|
type: string
|
|
40
44
|
required: true
|
|
41
45
|
description: "執行時間(ISO 8601 格式)"
|
|
46
|
+
- name: environment_layer
|
|
47
|
+
type: string
|
|
48
|
+
required: false
|
|
49
|
+
description: >
|
|
50
|
+
收集證據的環境層次(local / uat / prd)。
|
|
51
|
+
對有外部服務依賴的功能為必填——缺少此欄位時,該 AC 的驗收等級視為 local-only。
|
|
52
|
+
example: "uat"
|
|
42
53
|
|
|
43
54
|
red_green_cycle:
|
|
44
55
|
description: "回歸測試的 RED → GREEN 驗證"
|
|
@@ -72,13 +83,28 @@ standard:
|
|
|
72
83
|
action: "截斷但保留錯誤訊息與摘要行"
|
|
73
84
|
priority: medium
|
|
74
85
|
|
|
86
|
+
environment_rules:
|
|
87
|
+
- id: VE-005
|
|
88
|
+
trigger: "AC 涉及外部服務依賴(SMS、金流、IdP)"
|
|
89
|
+
action: >
|
|
90
|
+
驗收證據必須標明 environment_layer。
|
|
91
|
+
local-only 證據對此類 AC 不足——需確認 UAT 或 PRD 層次的驗證計畫。
|
|
92
|
+
priority: required
|
|
93
|
+
- id: VE-006
|
|
94
|
+
trigger: "有外部服務依賴的 AC 標記為 done,但無 environment_layer"
|
|
95
|
+
action: >
|
|
96
|
+
降級為 done_with_concerns。
|
|
97
|
+
要求補充環境層次聲明,或在 environment-stratification-matrix 中標記為 ⚠️/❌。
|
|
98
|
+
priority: high
|
|
99
|
+
|
|
75
100
|
physical_spec:
|
|
76
101
|
type: checklist
|
|
77
102
|
validator:
|
|
78
103
|
type: ai_review
|
|
79
|
-
rule: "檢查任務完成是否附帶驗證證據(command + exit_code + output + timestamp)"
|
|
104
|
+
rule: "檢查任務完成是否附帶驗證證據(command + exit_code + output + timestamp + environment_layer)"
|
|
80
105
|
checks:
|
|
81
106
|
- "完成聲明是否附帶 verification_evidence"
|
|
82
107
|
- "evidence 是否包含所有必填欄位"
|
|
83
108
|
- "exit_code 是否為 0(成功)"
|
|
84
109
|
- "Bug fix 是否有 RED → GREEN 循環證據"
|
|
110
|
+
- "有外部服務依賴的 AC 是否標明 environment_layer"
|
|
@@ -83,28 +83,60 @@ Scenario: User login with valid credentials
|
|
|
83
83
|
| **Covered** | ✅ | AC is fully tested | All conditions in AC have corresponding test assertions |
|
|
84
84
|
| **Partial** | ⚠️ | AC is partially tested | Some conditions tested, but edge cases or paths missing |
|
|
85
85
|
| **Uncovered** | ❌ | AC has no tests | No test case references this AC |
|
|
86
|
+
| **Not Implemented** | 🚫 | AC has no corresponding implementation | Feature code does not exist (not a test gap — a code gap) |
|
|
87
|
+
|
|
88
|
+
### `not_implemented` vs `uncovered` Decision Tree
|
|
89
|
+
|
|
90
|
+
```
|
|
91
|
+
Q1: Does the corresponding code exist in src/?
|
|
92
|
+
No → 🚫 not_implemented
|
|
93
|
+
Yes → Q2: Does any test reference this AC?
|
|
94
|
+
No → ❌ uncovered
|
|
95
|
+
Yes → Q3: Are all conditions in the AC tested?
|
|
96
|
+
Yes → ✅ covered
|
|
97
|
+
No → ⚠️ partial
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Typical signals of `not_implemented`: `throw new NotImplementedException()`, empty stub body, `// FEATURE_STUB:` marker.
|
|
86
101
|
|
|
87
102
|
### Coverage Calculation
|
|
88
103
|
|
|
89
104
|
```
|
|
90
|
-
AC Coverage % = (covered_count / total_ac_count) × 100
|
|
105
|
+
AC Coverage % = (covered_count + partial_count × 0.5) / (total_ac_count - not_implemented_count) × 100
|
|
91
106
|
|
|
92
107
|
Where:
|
|
93
108
|
covered_count = count of AC with status "covered"
|
|
94
109
|
total_ac_count = total number of AC in specification
|
|
110
|
+
not_implemented_count = count of AC with status "not_implemented" (excluded from denominator)
|
|
95
111
|
partial counts as 0.5 for coverage calculation
|
|
96
112
|
```
|
|
97
113
|
|
|
98
114
|
### Example Calculation
|
|
99
115
|
|
|
100
116
|
```
|
|
101
|
-
SPEC-001:
|
|
102
|
-
-
|
|
103
|
-
-
|
|
104
|
-
-
|
|
117
|
+
SPEC-001: 20 AC total
|
|
118
|
+
- 15 covered (✅)
|
|
119
|
+
- 0 partial (⚠️)
|
|
120
|
+
- 2 uncovered (❌)
|
|
121
|
+
- 3 not_implemented (🚫)
|
|
122
|
+
|
|
123
|
+
Coverage = (15 + 0) / (20 - 3) × 100 = 88.2%
|
|
124
|
+
Status: BLOCKED by 3 not_implemented AC(s)
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
### CI Gate for `not_implemented`
|
|
128
|
+
|
|
129
|
+
`not_implemented` ACs trigger a **blocking** CI gate independent of the coverage percentage gate:
|
|
105
130
|
|
|
106
|
-
Coverage = (5 + 2×0.5) / 8 = 6/8 = 75%
|
|
107
131
|
```
|
|
132
|
+
[AC-NOT-IMPL] 3 AC(s) marked not_implemented:
|
|
133
|
+
🚫 AC-007 OrderCancellation
|
|
134
|
+
🚫 AC-012 RefundCalculation
|
|
135
|
+
🚫 AC-019 ExportToPDF
|
|
136
|
+
All not_implemented ACs must be resolved before UAT.
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
The gate clears only when all `not_implemented` ACs are updated to `uncovered`, `partial`, or `covered`.
|
|
108
140
|
|
|
109
141
|
---
|
|
110
142
|
|
|
@@ -273,6 +305,8 @@ Generated spec MUST include:
|
|
|
273
305
|
| Ignoring uncovered AC | Gaps in verification | Track and plan coverage for all AC |
|
|
274
306
|
| AC without testability | Cannot be verified | Ensure all AC are testable |
|
|
275
307
|
| Coverage without assertions | Tests run but verify nothing | Check for meaningful assertions |
|
|
308
|
+
| Using `uncovered` for missing code | Hides functional completeness gap | Use `not_implemented` when code doesn't exist |
|
|
309
|
+
| Including `not_implemented` in denominator | Inflates coverage metric | Exclude `not_implemented` from denominator |
|
|
276
310
|
|
|
277
311
|
---
|
|
278
312
|
|
|
@@ -291,3 +325,4 @@ Generated spec MUST include:
|
|
|
291
325
|
| Version | Date | Changes |
|
|
292
326
|
|---------|------|---------|
|
|
293
327
|
| 1.0.0 | 2026-03-18 | Initial version — traceability matrix, coverage calculation, spec generation rules |
|
|
328
|
+
| 1.1.0 | 2026-05-12 | Add `not_implemented` 4th status; update CI gate formula; add decision tree (XSPEC-199) |
|