universal-dev-standards 5.8.0 → 5.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/ai/language-packs/README.md +55 -0
- package/bundled/ai/language-packs/language-pack-php-to-csharp.ai.yaml +83 -0
- package/bundled/ai/standards/behavior-snapshot.ai.yaml +122 -18
- package/bundled/ai/standards/deployment-standards.ai.yaml +108 -3
- package/bundled/ai/standards/feature-discovery-standards.ai.yaml +459 -0
- package/bundled/ai/standards/feature-manifest-standard.ai.yaml +23 -20
- package/bundled/ai/standards/mock-boundary.ai.yaml +93 -2
- package/bundled/ai/standards/multi-environment-e2e-testing.ai.yaml +250 -0
- package/bundled/ai/standards/test-completeness-dimensions.ai.yaml +39 -7
- package/bundled/ai/standards/verification-evidence.ai.yaml +30 -4
- package/bundled/core/adversarial-test.md +3 -0
- package/bundled/core/behavior-snapshot.md +1 -1
- package/bundled/core/capability-declaration.md +3 -0
- package/bundled/core/cd-deployment-strategies.md +3 -0
- package/bundled/core/chaos-injection-tests.md +3 -0
- package/bundled/core/circuit-breaker.md +3 -0
- package/bundled/core/container-security.md +3 -0
- package/bundled/core/cost-budget-test.md +3 -0
- package/bundled/core/data-migration-testing.md +3 -0
- package/bundled/core/disaster-recovery-drill.md +3 -0
- package/bundled/core/dual-phase-output.md +3 -0
- package/bundled/core/failure-source-taxonomy.md +3 -0
- package/bundled/core/feature-manifest-standard.md +1 -1
- package/bundled/core/flaky-test-management.md +3 -0
- package/bundled/core/full-coverage-testing.md +3 -0
- package/bundled/core/health-check-standards.md +3 -0
- package/bundled/core/immutability-first.md +3 -0
- package/bundled/core/llm-output-validation.md +3 -0
- package/bundled/core/no-cicd-deployment.md +3 -0
- package/bundled/core/pipeline-security-gates.md +3 -0
- package/bundled/core/policy-as-code-testing.md +3 -0
- package/bundled/core/prompt-regression.md +3 -0
- package/bundled/core/property-based-testing.md +3 -0
- package/bundled/core/recovery-recipe-registry.md +3 -0
- package/bundled/core/release-quality-manifest.md +3 -0
- package/bundled/core/replay-test.md +3 -0
- package/bundled/core/retry-standards.md +3 -0
- package/bundled/core/rollback-standards.md +3 -0
- package/bundled/core/sast-advanced.md +1 -1
- package/bundled/core/secure-op.md +3 -0
- package/bundled/core/security-decision.md +3 -0
- package/bundled/core/server-ops-security.md +3 -0
- package/bundled/core/skill-standard-alignment-check.md +3 -0
- package/bundled/core/smoke-test.md +3 -0
- package/bundled/core/standard-admission-criteria.md +3 -0
- package/bundled/core/standard-lifecycle-management.md +3 -0
- package/bundled/core/supply-chain-attestation.md +3 -0
- package/bundled/core/timeout-standards.md +3 -0
- package/bundled/core/token-budget.md +3 -0
- package/bundled/locales/zh-CN/CHANGELOG.md +38 -3
- package/bundled/locales/zh-CN/CLAUDE.md +1 -1
- package/bundled/locales/zh-CN/README.md +3 -3
- package/bundled/locales/zh-CN/SECURITY.md +1 -2
- package/bundled/locales/zh-CN/docs/CHEATSHEET.md +127 -15
- package/bundled/locales/zh-CN/docs/FEATURE-REFERENCE.md +141 -25
- package/bundled/locales/zh-CN/skills/adr-assistant/SKILL.md +43 -0
- package/bundled/locales/zh-CN/skills/ai-collaboration-standards/SKILL.md +35 -0
- package/bundled/locales/zh-CN/skills/ai-friendly-architecture/SKILL.md +35 -0
- package/bundled/locales/zh-CN/skills/ai-instruction-standards/SKILL.md +35 -0
- package/bundled/locales/zh-CN/skills/contract-test-assistant/SKILL.md +24 -0
- package/bundled/locales/zh-CN/skills/documentation-guide/SKILL.md +37 -0
- package/bundled/locales/zh-CN/skills/error-code-guide/SKILL.md +33 -0
- package/bundled/locales/zh-CN/skills/git-workflow-guide/SKILL.md +34 -0
- package/bundled/locales/zh-CN/skills/logging-guide/SKILL.md +33 -0
- package/bundled/locales/zh-CN/skills/project-structure-guide/SKILL.md +19 -0
- package/bundled/locales/zh-CN/skills/retrospective-assistant/SKILL.md +38 -0
- package/bundled/locales/zh-CN/skills/testing-guide/SKILL.md +36 -0
- package/bundled/locales/zh-TW/CHANGELOG.md +38 -3
- package/bundled/locales/zh-TW/CLAUDE.md +1 -1
- package/bundled/locales/zh-TW/README.md +3 -3
- package/bundled/locales/zh-TW/SECURITY.md +1 -2
- package/bundled/locales/zh-TW/docs/CHEATSHEET.md +127 -15
- package/bundled/locales/zh-TW/docs/FEATURE-REFERENCE.md +141 -25
- package/bundled/locales/zh-TW/skills/adr-assistant/SKILL.md +43 -0
- package/bundled/locales/zh-TW/skills/ai-collaboration-standards/SKILL.md +35 -0
- package/bundled/locales/zh-TW/skills/ai-friendly-architecture/SKILL.md +35 -0
- package/bundled/locales/zh-TW/skills/ai-instruction-standards/SKILL.md +35 -0
- package/bundled/locales/zh-TW/skills/brainstorm-assistant/guide.md +661 -267
- package/bundled/locales/zh-TW/skills/contract-test-assistant/SKILL.md +24 -0
- package/bundled/locales/zh-TW/skills/documentation-guide/SKILL.md +37 -0
- package/bundled/locales/zh-TW/skills/error-code-guide/SKILL.md +33 -0
- package/bundled/locales/zh-TW/skills/git-workflow-guide/SKILL.md +34 -0
- package/bundled/locales/zh-TW/skills/logging-guide/SKILL.md +33 -0
- package/bundled/locales/zh-TW/skills/project-structure-guide/SKILL.md +19 -0
- package/bundled/locales/zh-TW/skills/retrospective-assistant/SKILL.md +38 -0
- package/bundled/locales/zh-TW/skills/testing-guide/SKILL.md +36 -0
- package/bundled/skills/README.md +1 -1
- package/bundled/skills/spec-driven-dev/SKILL.md +41 -0
- package/package.json +4 -4
- package/src/commands/update.js +39 -1
- package/src/config/ai-agent-paths.js +8 -0
- package/src/i18n/messages.js +3 -0
- package/src/utils/integration-generator.js +18 -2
- package/standards-registry.json +29 -5
- package/bundled/locales/zh-TW/docs/MIGRATION-V5.md +0 -79
|
@@ -0,0 +1,250 @@
|
|
|
1
|
+
# Multi-Environment E2E Testing Standards - AI Optimized
|
|
2
|
+
# Source: XSPEC-204 (UDS Issue #94, #95)
|
|
3
|
+
|
|
4
|
+
id: multi-environment-e2e-testing
|
|
5
|
+
meta:
|
|
6
|
+
version: "1.0.0"
|
|
7
|
+
updated: "2026-05-13"
|
|
8
|
+
description: >
|
|
9
|
+
Standards for E2E test configuration across multiple deployment targets.
|
|
10
|
+
Core principle: The run command IS the documentation.
|
|
11
|
+
Each environment has one entry-point script that self-checks prerequisites
|
|
12
|
+
and runs the correct test subset.
|
|
13
|
+
|
|
14
|
+
# ─────────────────────────────────────────────────────────
|
|
15
|
+
# Core Principle
|
|
16
|
+
# ─────────────────────────────────────────────────────────
|
|
17
|
+
core_principle:
|
|
18
|
+
name: Executable Environment Documentation
|
|
19
|
+
statement: "The run command IS the documentation."
|
|
20
|
+
rationale: >
|
|
21
|
+
Developers repeatedly ask "how do I run tests against environment X?"
|
|
22
|
+
because the answer is scattered across README, wiki, and verbal knowledge.
|
|
23
|
+
When the run script self-checks prerequisites and sets the correct BASE_URL,
|
|
24
|
+
the script itself becomes the authoritative environment setup guide.
|
|
25
|
+
anti_pattern: >
|
|
26
|
+
Manually changing BASE_URL in .env before test runs — breaks other
|
|
27
|
+
developers' configs and cannot be committed without affecting everyone.
|
|
28
|
+
|
|
29
|
+
# ─────────────────────────────────────────────────────────
|
|
30
|
+
# Test Framework Multi-Environment Config
|
|
31
|
+
# ─────────────────────────────────────────────────────────
|
|
32
|
+
test_framework_config:
|
|
33
|
+
rule: "BASE_URL is baked into test project config, not read from .env"
|
|
34
|
+
rationale: >
|
|
35
|
+
If BASE_URL comes from .env, developers must modify .env before running
|
|
36
|
+
against a specific environment, creating race conditions in teams and
|
|
37
|
+
accidental commits of wrong URLs.
|
|
38
|
+
|
|
39
|
+
playwright_pattern: |
|
|
40
|
+
// playwright.config.ts
|
|
41
|
+
const ENVS = {
|
|
42
|
+
'local-iis': 'http://localhost/corp',
|
|
43
|
+
'local-iis-express': 'http://localhost:18080/lotest',
|
|
44
|
+
'uat': 'http://portal_uat.example.com/app',
|
|
45
|
+
'prd': 'https://app.example.com',
|
|
46
|
+
} as const;
|
|
47
|
+
|
|
48
|
+
export default defineConfig({
|
|
49
|
+
projects: Object.entries(ENVS).map(([name, url]) => ({
|
|
50
|
+
name,
|
|
51
|
+
use: { browserName: 'chromium', baseURL: url },
|
|
52
|
+
})),
|
|
53
|
+
});
|
|
54
|
+
|
|
55
|
+
cypress_pattern: |
|
|
56
|
+
// cypress.config.ts
|
|
57
|
+
const ENVS = {
|
|
58
|
+
'local': { baseUrl: 'http://localhost:3000' },
|
|
59
|
+
'uat': { baseUrl: 'http://uat.example.com' },
|
|
60
|
+
'prd': { baseUrl: 'https://app.example.com' },
|
|
61
|
+
};
|
|
62
|
+
// Pass via --env target=uat; default to local
|
|
63
|
+
|
|
64
|
+
rules:
|
|
65
|
+
- "One test config file; environment distinction via project name or --project flag"
|
|
66
|
+
- "No environment-specific playwright/cypress.config.*.ts files"
|
|
67
|
+
- "BASE_URL never in .env for E2E test configs"
|
|
68
|
+
|
|
69
|
+
# ─────────────────────────────────────────────────────────
|
|
70
|
+
# Runner Script Pattern
|
|
71
|
+
# ─────────────────────────────────────────────────────────
|
|
72
|
+
runner_script_pattern:
|
|
73
|
+
rule: "Each environment has one entry-point script that self-checks prerequisites"
|
|
74
|
+
location: "scripts/run-tests-<env>.(ps1|sh)"
|
|
75
|
+
template_powershell: |
|
|
76
|
+
# scripts/run-tests-local-iis.ps1
|
|
77
|
+
# Self-check prerequisites before running tests
|
|
78
|
+
|
|
79
|
+
# 1. Check Docker
|
|
80
|
+
if (-not (Get-Process "com.docker.backend" -ErrorAction SilentlyContinue)) {
|
|
81
|
+
Write-Error "Docker is not running. Start Docker Desktop first."
|
|
82
|
+
exit 1
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
# 2. Check App is responding
|
|
86
|
+
try {
|
|
87
|
+
$resp = Invoke-WebRequest "http://localhost/corp/health" -UseBasicParsing -TimeoutSec 5
|
|
88
|
+
} catch {
|
|
89
|
+
Write-Error "App not responding at http://localhost/corp. Start IIS site first."
|
|
90
|
+
exit 1
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
# 3. Run tests
|
|
94
|
+
npx playwright test --project=local-iis @args
|
|
95
|
+
|
|
96
|
+
template_bash: |
|
|
97
|
+
#!/bin/bash
|
|
98
|
+
# scripts/run-tests-local.sh
|
|
99
|
+
|
|
100
|
+
# 1. Check Docker
|
|
101
|
+
if ! docker info > /dev/null 2>&1; then
|
|
102
|
+
echo "ERROR: Docker is not running." >&2
|
|
103
|
+
exit 1
|
|
104
|
+
fi
|
|
105
|
+
|
|
106
|
+
# 2. Check App
|
|
107
|
+
if ! curl -s -f http://localhost:3000/health > /dev/null 2>&1; then
|
|
108
|
+
echo "ERROR: App not responding at http://localhost:3000" >&2
|
|
109
|
+
exit 1
|
|
110
|
+
fi
|
|
111
|
+
|
|
112
|
+
npx playwright test --project=local "$@"
|
|
113
|
+
|
|
114
|
+
prerequisite_checks:
|
|
115
|
+
- "Docker / container runtime"
|
|
116
|
+
- "Application health endpoint"
|
|
117
|
+
- "Database connectivity (if separate from app)"
|
|
118
|
+
- "Required environment-specific services"
|
|
119
|
+
|
|
120
|
+
# ─────────────────────────────────────────────────────────
|
|
121
|
+
# Environment Capability Matrix
|
|
122
|
+
# ─────────────────────────────────────────────────────────
|
|
123
|
+
capability_matrix:
|
|
124
|
+
rule: >
|
|
125
|
+
Projects with external dependencies MUST maintain an environment capability
|
|
126
|
+
matrix committed to the repository (docs/testing/environment-capability-matrix.md
|
|
127
|
+
or inline in testing README).
|
|
128
|
+
when_required: "Any project with external HTTP services, IdP, payment, messaging"
|
|
129
|
+
|
|
130
|
+
template: |
|
|
131
|
+
## Environment Capability Matrix
|
|
132
|
+
|
|
133
|
+
| Service / Feature | local-dev | local-iis | UAT | PRD |
|
|
134
|
+
|-------------------|:---------:|:---------:|:---:|:---:|
|
|
135
|
+
| Auth / SAML | ⚠️ Keycloak stub | ✅ Keycloak local | ✅ Keycloak UAT | ✅ Enterprise IdP |
|
|
136
|
+
| SMS Gateway | ⚠️ stub-server | ⚠️ stub-server | ⚠️ stub-server / ❌ | ✅ Real Gateway + billing |
|
|
137
|
+
| Payment / Finance | ⚠️ stub-server | ⚠️ stub-server | ⚠️ partial | ✅ Real + reconciliation |
|
|
138
|
+
| Background Jobs | ✅ in-process | ✅ in-process | ✅ | ✅ |
|
|
139
|
+
| File Storage | ✅ local | ✅ local | ✅ blob | ✅ blob |
|
|
140
|
+
|
|
141
|
+
Legend:
|
|
142
|
+
✅ Full verification possible
|
|
143
|
+
⚠️ Flow passes but through stub (real-world dimensions NOT verified)
|
|
144
|
+
❌ Cannot test in this environment
|
|
145
|
+
|
|
146
|
+
### Dimensions NOT verifiable in UAT (must defer to PRD smoke)
|
|
147
|
+
- SMS: Billing correctness, carrier delivery confirmation, DR reporting
|
|
148
|
+
- Payment: Real debit/credit, bank reconciliation, card validation
|
|
149
|
+
|
|
150
|
+
when_to_update: "Update matrix when adding any new external service dependency"
|
|
151
|
+
|
|
152
|
+
# ─────────────────────────────────────────────────────────
|
|
153
|
+
# CI Gate Mapping
|
|
154
|
+
# ─────────────────────────────────────────────────────────
|
|
155
|
+
ci_gate_mapping:
|
|
156
|
+
rule: "Map environments to CI/CD stages; document which gate must pass before each deployment stage"
|
|
157
|
+
|
|
158
|
+
pattern: |
|
|
159
|
+
# .github/workflows/ci.yml or .gitlab-ci.yml
|
|
160
|
+
|
|
161
|
+
e2e-smoke-gate: # Must pass → any deployment
|
|
162
|
+
runs-on: ubuntu-latest
|
|
163
|
+
script: scripts/run-tests-local.sh --grep smoke
|
|
164
|
+
|
|
165
|
+
e2e-uat-gate: # Must pass → PRD deployment
|
|
166
|
+
environment: uat
|
|
167
|
+
script: scripts/run-tests-uat.sh
|
|
168
|
+
only: [tags, release-branches]
|
|
169
|
+
|
|
170
|
+
e2e-prd-smoke: # Must pass → mark release as stable
|
|
171
|
+
environment: prd
|
|
172
|
+
script: scripts/run-tests-prd-smoke.sh
|
|
173
|
+
only: [tags]
|
|
174
|
+
|
|
175
|
+
gate_requirements:
|
|
176
|
+
before_staging_deploy: ["unit tests", "integration tests", "e2e-smoke-gate"]
|
|
177
|
+
before_uat_deploy: ["all staging gates", "e2e-uat-gate (if UAT environment available)"]
|
|
178
|
+
before_prd_deploy: ["e2e-uat-gate", "sign-off from capability matrix review"]
|
|
179
|
+
after_prd_deploy: ["e2e-prd-smoke within 10 min of deploy"]
|
|
180
|
+
|
|
181
|
+
# ─────────────────────────────────────────────────────────
|
|
182
|
+
# Credential Handling
|
|
183
|
+
# ─────────────────────────────────────────────────────────
|
|
184
|
+
credential_handling:
|
|
185
|
+
rule: "Separate what goes in git from what stays gitignored"
|
|
186
|
+
|
|
187
|
+
commit_to_git:
|
|
188
|
+
- "Base URLs per environment (non-secret, team needs to share)"
|
|
189
|
+
- "Test usernames for non-PRD environments"
|
|
190
|
+
- "Feature flags and test configuration"
|
|
191
|
+
- "Self-check scripts (no credentials embedded)"
|
|
192
|
+
|
|
193
|
+
gitignore:
|
|
194
|
+
- "Passwords and secrets"
|
|
195
|
+
- "API keys and tokens"
|
|
196
|
+
- ".env.test.local (personal overrides)"
|
|
197
|
+
|
|
198
|
+
ci_secrets: "Pass PRD test passwords via CI secret variables (GitHub Secrets / GitLab CI Variables)"
|
|
199
|
+
|
|
200
|
+
example_gitignore: |
|
|
201
|
+
# Test credentials
|
|
202
|
+
.env.test.local
|
|
203
|
+
tests/fixtures/auth-secrets.json
|
|
204
|
+
# Base URLs and non-secrets are committed; see playwright.config.ts
|
|
205
|
+
|
|
206
|
+
# ─────────────────────────────────────────────────────────
|
|
207
|
+
# Rules
|
|
208
|
+
# ─────────────────────────────────────────────────────────
|
|
209
|
+
rules:
|
|
210
|
+
- id: base-url-in-config
|
|
211
|
+
trigger: setting up E2E tests for a project with multiple environments
|
|
212
|
+
instruction: >
|
|
213
|
+
Define all environment BASE_URLs in the test framework config (playwright.config.ts /
|
|
214
|
+
cypress.config.ts) as named projects. Do not rely on .env for BASE_URL.
|
|
215
|
+
priority: required
|
|
216
|
+
|
|
217
|
+
- id: one-runner-per-env
|
|
218
|
+
trigger: adding a new deployment target
|
|
219
|
+
instruction: >
|
|
220
|
+
Create scripts/run-tests-<env>.(ps1|sh) with self-checking prerequisite steps.
|
|
221
|
+
The script must verify all required services before invoking the test runner.
|
|
222
|
+
priority: required
|
|
223
|
+
|
|
224
|
+
- id: capability-matrix-required
|
|
225
|
+
trigger: feature has external service dependencies (SMS, payment, IdP, file storage)
|
|
226
|
+
instruction: >
|
|
227
|
+
Create or update the environment capability matrix in docs/testing/.
|
|
228
|
+
Clearly mark ✅/⚠️/❌ for each service × environment combination.
|
|
229
|
+
List dimensions NOT verifiable in UAT; mark these as "PRD-only smoke" items.
|
|
230
|
+
priority: required
|
|
231
|
+
|
|
232
|
+
- id: ci-gate-mapping
|
|
233
|
+
trigger: defining CI/CD pipeline stages
|
|
234
|
+
instruction: >
|
|
235
|
+
Map each CI gate to the environments it must pass.
|
|
236
|
+
E2E gate must specify which environment it targets.
|
|
237
|
+
priority: required
|
|
238
|
+
|
|
239
|
+
# ─────────────────────────────────────────────────────────
|
|
240
|
+
# Relationship to Other Standards
|
|
241
|
+
# ─────────────────────────────────────────────────────────
|
|
242
|
+
related_standards:
|
|
243
|
+
- id: deployment-standards
|
|
244
|
+
relationship: "Extends — adds environment dimension to CI gates and deployment readiness"
|
|
245
|
+
- id: test-completeness-dimensions
|
|
246
|
+
relationship: "Complements — adds dimension 11: Environment Verifiability"
|
|
247
|
+
- id: verification-evidence
|
|
248
|
+
relationship: "Complements — evidence must specify which environment it was collected from"
|
|
249
|
+
- id: mock-boundary
|
|
250
|
+
relationship: "Complements — capability matrix references Level 2 stub server usage"
|
|
@@ -3,10 +3,12 @@
|
|
|
3
3
|
|
|
4
4
|
id: test-completeness-dimensions
|
|
5
5
|
meta:
|
|
6
|
-
version: "1.
|
|
7
|
-
updated: "2026-05-
|
|
6
|
+
version: "1.3.0"
|
|
7
|
+
updated: "2026-05-13"
|
|
8
8
|
source: core/test-completeness-dimensions.md
|
|
9
|
-
description:
|
|
9
|
+
description: >
|
|
10
|
+
Framework for evaluating test completeness across 11 dimensions.
|
|
11
|
+
v1.3.0 adds Dimension 11: Environment Verifiability (XSPEC-204).
|
|
10
12
|
|
|
11
13
|
dimensions:
|
|
12
14
|
- id: 1
|
|
@@ -114,6 +116,21 @@ dimensions:
|
|
|
114
116
|
when_required: When flow has any conditional logic or decision points
|
|
115
117
|
note: Use decision_table_expansion from flow-based-testing.ai.yaml to enumerate scenarios
|
|
116
118
|
|
|
119
|
+
- id: 11
|
|
120
|
+
name: Environment Verifiability
|
|
121
|
+
description: >
|
|
122
|
+
For each AC with external service dependencies, document which environment layer
|
|
123
|
+
can fully verify it, and plan PRD-smoke coverage for items that cannot be verified in UAT.
|
|
124
|
+
test_items:
|
|
125
|
+
- Environment stratification responsibility matrix exists for features with external dependencies
|
|
126
|
+
- Each AC is tagged with its minimum verifiable environment layer (local / UAT / PRD)
|
|
127
|
+
- ACs that are PRD-only are explicitly tracked and have a PRD smoke test plan
|
|
128
|
+
- Evidence for externally-dependent ACs includes environment_layer field
|
|
129
|
+
when_required: >
|
|
130
|
+
Any feature with external service dependencies (SMS, payment, IdP, email,
|
|
131
|
+
SOAP integrations, external REST APIs)
|
|
132
|
+
note: See multi-environment-e2e-testing.ai.yaml and deployment-standards.ai.yaml for environment capability matrix templates
|
|
133
|
+
|
|
117
134
|
feature_type_mapping:
|
|
118
135
|
note: "Dimension 8 (AI Generation Quality) applies when tests are AI-generated"
|
|
119
136
|
types:
|
|
@@ -138,8 +155,9 @@ feature_type_mapping:
|
|
|
138
155
|
with_ai: [1, 3, 5, 8]
|
|
139
156
|
|
|
140
157
|
- type: External Integration
|
|
141
|
-
dimensions: [1, 3, 7]
|
|
142
|
-
with_ai: [1, 3, 7, 8]
|
|
158
|
+
dimensions: [1, 3, 7, 11]
|
|
159
|
+
with_ai: [1, 3, 7, 8, 11]
|
|
160
|
+
note: Dimension 11 required for any external service dependency
|
|
143
161
|
|
|
144
162
|
- type: Workflow / Multi-step Process
|
|
145
163
|
dimensions: [1, 3, 4, 5, 9, 10]
|
|
@@ -147,6 +165,11 @@ feature_type_mapping:
|
|
|
147
165
|
required_pattern: journey-chained-test
|
|
148
166
|
note: Apply flow-based-testing standard; use shared ctx; Each-Choice minimum branch coverage
|
|
149
167
|
|
|
168
|
+
- type: External-Dependent Workflow
|
|
169
|
+
dimensions: [1, 3, 4, 5, 9, 10, 11]
|
|
170
|
+
with_ai: [1, 3, 4, 5, 9, 10, 8, 11]
|
|
171
|
+
note: Combines Workflow + External Integration; Dimension 11 mandatory
|
|
172
|
+
|
|
150
173
|
error_code_coverage:
|
|
151
174
|
- code: 200
|
|
152
175
|
meaning: Success
|
|
@@ -198,7 +221,7 @@ anti_patterns:
|
|
|
198
221
|
rules:
|
|
199
222
|
- id: use-checklist
|
|
200
223
|
trigger: designing tests for a feature
|
|
201
|
-
instruction: Use the
|
|
224
|
+
instruction: Use the 11 dimensions checklist to ensure completeness (dim 8 for AI-generated tests; dim 9/10 for multi-step workflows; dim 11 for external dependencies)
|
|
202
225
|
priority: required
|
|
203
226
|
|
|
204
227
|
- id: authorization-matrix
|
|
@@ -278,6 +301,12 @@ checklist_template: |
|
|
|
278
301
|
□ Each distinct error branch has its own describe block
|
|
279
302
|
□ Critical flows (auth/payment/security) use All-Combinations
|
|
280
303
|
|
|
304
|
+
□ Environment Verifiability (for features with external dependencies)
|
|
305
|
+
□ Environment stratification matrix created/updated
|
|
306
|
+
□ Each AC tagged with minimum verifiable environment layer
|
|
307
|
+
□ PRD-only ACs documented with smoke test plan
|
|
308
|
+
□ Evidence includes environment_layer field for externally-dependent ACs
|
|
309
|
+
|
|
281
310
|
quick_reference:
|
|
282
311
|
dimensions:
|
|
283
312
|
columns: [ID, Dimension, Key Focus]
|
|
@@ -292,9 +321,10 @@ quick_reference:
|
|
|
292
321
|
- [8, AI Generation, AI-generated test quality]
|
|
293
322
|
- [9, Flow Completeness, Complete path to each terminal state]
|
|
294
323
|
- [10, Branch Coverage, All decision-point branches covered]
|
|
324
|
+
- [11, Environment Verifiability, External-dep ACs tagged with environment layer]
|
|
295
325
|
|
|
296
326
|
feature_dimensions:
|
|
297
|
-
note: "*Dimension 8 applies when tests are AI-generated; †Dimensions 9/10 apply to multi-step workflows"
|
|
327
|
+
note: "*Dimension 8 applies when tests are AI-generated; †Dimensions 9/10 apply to multi-step workflows; ‡Dimension 11 applies to features with external service dependencies"
|
|
298
328
|
columns: [Feature Type, Required Dimensions]
|
|
299
329
|
rows:
|
|
300
330
|
- [CRUD API, "1,2,3,4,6,7,8*"]
|
|
@@ -304,3 +334,5 @@ quick_reference:
|
|
|
304
334
|
- [Background Job, "1,3,5,8*"]
|
|
305
335
|
- [External Integration, "1,3,7,8*"]
|
|
306
336
|
- [Workflow / Multi-step Process, "1,3,4,5,9†,10†,8*"]
|
|
337
|
+
- [External Integration, "1,3,7,11‡,8*"]
|
|
338
|
+
- [External-Dependent Workflow, "1,3,4,5,9†,10†,11‡,8*"]
|
|
@@ -8,15 +8,19 @@ standard:
|
|
|
8
8
|
description: 驗證證據標準,強化 anti-hallucination
|
|
9
9
|
|
|
10
10
|
meta:
|
|
11
|
-
version: "1.
|
|
12
|
-
updated: "2026-
|
|
11
|
+
version: "1.1.0"
|
|
12
|
+
updated: "2026-05-13"
|
|
13
13
|
source: core/verification-evidence.md
|
|
14
|
-
description:
|
|
14
|
+
description: >
|
|
15
|
+
驗證證據標準 — Iron Law: 無驗證證據不可聲稱完成。
|
|
16
|
+
v1.1.0: Evidence must specify which environment layer it was collected from (XSPEC-204).
|
|
15
17
|
inspired_by: superpowers/verification-before-completion
|
|
16
18
|
|
|
17
19
|
guidelines:
|
|
18
20
|
- "Iron Law:無驗證證據 = 不可聲稱完成"
|
|
19
21
|
- "每次驗證必須記錄 command + exitCode + output + timestamp"
|
|
22
|
+
- "Iron Law(Environment):驗收前必須確認「此環境層次能驗證此流程的哪些維度」"
|
|
23
|
+
- "驗收證據必須標明收集自哪個環境層次(local / UAT / PRD)"
|
|
20
24
|
- "回歸測試必須展示 RED → GREEN 循環"
|
|
21
25
|
- "代理報告 success ≠ 實際 success,需獨立驗證"
|
|
22
26
|
- "驗證輸出截斷至合理長度(2000 字元)但保留關鍵資訊"
|
|
@@ -39,6 +43,13 @@ standard:
|
|
|
39
43
|
type: string
|
|
40
44
|
required: true
|
|
41
45
|
description: "執行時間(ISO 8601 格式)"
|
|
46
|
+
- name: environment_layer
|
|
47
|
+
type: string
|
|
48
|
+
required: false
|
|
49
|
+
description: >
|
|
50
|
+
收集證據的環境層次(local / uat / prd)。
|
|
51
|
+
對有外部服務依賴的功能為必填——缺少此欄位時,該 AC 的驗收等級視為 local-only。
|
|
52
|
+
example: "uat"
|
|
42
53
|
|
|
43
54
|
red_green_cycle:
|
|
44
55
|
description: "回歸測試的 RED → GREEN 驗證"
|
|
@@ -72,13 +83,28 @@ standard:
|
|
|
72
83
|
action: "截斷但保留錯誤訊息與摘要行"
|
|
73
84
|
priority: medium
|
|
74
85
|
|
|
86
|
+
environment_rules:
|
|
87
|
+
- id: VE-005
|
|
88
|
+
trigger: "AC 涉及外部服務依賴(SMS、金流、IdP)"
|
|
89
|
+
action: >
|
|
90
|
+
驗收證據必須標明 environment_layer。
|
|
91
|
+
local-only 證據對此類 AC 不足——需確認 UAT 或 PRD 層次的驗證計畫。
|
|
92
|
+
priority: required
|
|
93
|
+
- id: VE-006
|
|
94
|
+
trigger: "有外部服務依賴的 AC 標記為 done,但無 environment_layer"
|
|
95
|
+
action: >
|
|
96
|
+
降級為 done_with_concerns。
|
|
97
|
+
要求補充環境層次聲明,或在 environment-stratification-matrix 中標記為 ⚠️/❌。
|
|
98
|
+
priority: high
|
|
99
|
+
|
|
75
100
|
physical_spec:
|
|
76
101
|
type: checklist
|
|
77
102
|
validator:
|
|
78
103
|
type: ai_review
|
|
79
|
-
rule: "檢查任務完成是否附帶驗證證據(command + exit_code + output + timestamp)"
|
|
104
|
+
rule: "檢查任務完成是否附帶驗證證據(command + exit_code + output + timestamp + environment_layer)"
|
|
80
105
|
checks:
|
|
81
106
|
- "完成聲明是否附帶 verification_evidence"
|
|
82
107
|
- "evidence 是否包含所有必填欄位"
|
|
83
108
|
- "exit_code 是否為 0(成功)"
|
|
84
109
|
- "Bug fix 是否有 RED → GREEN 循環證據"
|
|
110
|
+
- "有外部服務依賴的 AC 是否標明 environment_layer"
|
|
@@ -57,3 +57,6 @@ const FAIL_CLOSED_DEFAULTS: CapabilityDeclaration = {
|
|
|
57
57
|
- AI-optimized: [ai/standards/capability-declaration.ai.yaml](../ai/standards/capability-declaration.ai.yaml)
|
|
58
58
|
- XSPEC-037: Cross-project specification
|
|
59
59
|
- Borrowed from: [claude-code-book](https://github.com/lintsinghua/claude-code-book) Ch.3 `buildTool` Fail-Closed factory
|
|
60
|
+
|
|
61
|
+
|
|
62
|
+
**Scope**: universal
|
|
@@ -119,3 +119,6 @@ Q4: 基礎設施預算有限?
|
|
|
119
119
|
- [rollback-standards.md](rollback-standards.md) — 回滾觸發條件矩陣
|
|
120
120
|
- [no-cicd-deployment.md](no-cicd-deployment.md) — 無 CI/CD 部署策略
|
|
121
121
|
- AI 格式:[../ai/standards/cd-deployment-strategies.ai.yaml](../ai/standards/cd-deployment-strategies.ai.yaml)
|
|
122
|
+
|
|
123
|
+
|
|
124
|
+
**Scope**: universal
|
|
@@ -114,3 +114,6 @@ it('pipeline continues when one agent throws', async () => {
|
|
|
114
114
|
- `testing.ai.yaml` — general test structure
|
|
115
115
|
- `secure-op.ai.yaml` — Fail-Closed principle for AI agents
|
|
116
116
|
- `security-standards.ai.yaml` — security invariants
|
|
117
|
+
|
|
118
|
+
|
|
119
|
+
**Scope**: universal
|
|
@@ -56,3 +56,6 @@ interface CircuitBreaker {
|
|
|
56
56
|
- AI-optimized: [ai/standards/circuit-breaker.ai.yaml](../ai/standards/circuit-breaker.ai.yaml)
|
|
57
57
|
- XSPEC-036: Cross-project specification
|
|
58
58
|
- Borrowed from: [claude-code-book](https://github.com/lintsinghua/claude-code-book) Ch.2 `MAX_CONSECUTIVE_AUTOCOMPACT_FAILURES`
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
**Scope**: universal
|
|
@@ -67,3 +67,6 @@ describe("PipelineBudgetConfig semantics", () => {
|
|
|
67
67
|
- [Mutation Testing Standards](mutation-testing.md) — constants without test coverage survive mutations
|
|
68
68
|
- [Testing Standards](testing.md) — overall test pyramid
|
|
69
69
|
- [LLM Output Validation](llm-output-validation.md) — output-layer budget constraints
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
**Scope**: universal
|
|
@@ -108,3 +108,6 @@ Use `testcontainers` to spin up a fresh PostgreSQL container per test suite. The
|
|
|
108
108
|
- `database-standards.ai.yaml` — schema design principles
|
|
109
109
|
- `testing.ai.yaml` — general test structure and pyramid
|
|
110
110
|
- `verification-evidence.ai.yaml` — audit evidence requirements
|
|
111
|
+
|
|
112
|
+
|
|
113
|
+
**Scope**: universal
|
|
@@ -71,3 +71,6 @@ See `docs/DR-RUNBOOK.md` for the full runbook template.
|
|
|
71
71
|
- [Deployment Standards](deployment-standards.md) — deployment pipeline
|
|
72
72
|
- [Chaos Engineering Standards](chaos-engineering-standards.md) — failure injection
|
|
73
73
|
- [Verification Evidence Standards](verification-evidence.md) — drill records
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
**Scope**: universal
|
|
@@ -54,3 +54,6 @@ Applications may add fields inside `<summary>` but must not remove core fields:
|
|
|
54
54
|
- AI-optimized: [ai/standards/dual-phase-output.ai.yaml](../ai/standards/dual-phase-output.ai.yaml)
|
|
55
55
|
- XSPEC-035: Cross-project specification
|
|
56
56
|
- Borrowed from: [claude-code-book](https://github.com/lintsinghua/claude-code-book) Ch.7 `formatCompactSummary`
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
**Scope**: universal
|
|
@@ -70,3 +70,6 @@ interface FailureDetail {
|
|
|
70
70
|
- XSPEC-045: Cross-project specification
|
|
71
71
|
- Depends on: Recovery Recipe Registry (XSPEC-046)
|
|
72
72
|
- Borrowed from: [ultraworkers/claw-code](https://github.com/ultraworkers/claw-code) ROADMAP Phase 2 Failure Taxonomy
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
**Scope**: universal
|
|
@@ -181,3 +181,6 @@ The ratchet starts at your current coverage. From that point on, it can only inc
|
|
|
181
181
|
- `integration-testing.ai.yaml` — Integration test patterns
|
|
182
182
|
- `deployment-standards.ai.yaml` — Deploy gate requirements
|
|
183
183
|
- XSPEC-178 — Full specification and implementation phases
|
|
184
|
+
|
|
185
|
+
|
|
186
|
+
**Scope**: universal
|
|
@@ -176,3 +176,6 @@ npx vitest run agents/__tests__/contract.test.ts
|
|
|
176
176
|
- ISO/IEC 42001:2023 — AI 管理系統
|
|
177
177
|
- [UDS `security-testing.ai.yaml`](./security-testing.md) — SAST + DAST 整合
|
|
178
178
|
- [UDS `adversarial-test.ai.yaml`](./adversarial-test.md) — Prompt injection 紅隊標準
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
**Scope**: universal
|
|
@@ -107,3 +107,6 @@
|
|
|
107
107
|
- [pipeline-integration-standards.md](pipeline-integration-standards.md) — CI 管線整合標準
|
|
108
108
|
- [deployment-standards.md](deployment-standards.md) — 部署基礎原則
|
|
109
109
|
- AI 格式:[../ai/standards/pipeline-security-gates.ai.yaml](../ai/standards/pipeline-security-gates.ai.yaml)
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
**Scope**: universal
|
|
@@ -186,3 +186,6 @@ policies/
|
|
|
186
186
|
- [UDS `secure-op.ai.yaml`](./secure-op.md) — AI Agent 安全操作六大支柱
|
|
187
187
|
- [UDS `adversarial-test.ai.yaml`](./adversarial-test.md) — 對抗性測試(OWASP LLM01)
|
|
188
188
|
- [UDS `container-security.ai.yaml`](./container-security.md) — 容器安全(OPA Sidecar 部署)
|
|
189
|
+
|
|
190
|
+
|
|
191
|
+
**Scope**: universal
|
|
@@ -70,3 +70,6 @@ The comment is mandatory. PRs that update checksums without explanatory comments
|
|
|
70
70
|
- [LLM Output Validation](llm-output-validation.md) — schema-level validation
|
|
71
71
|
- [Adversarial Test](adversarial-test.md) — red-team corpus
|
|
72
72
|
- [Testing Standards](testing.md) — overall testing pyramid
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
**Scope**: universal
|
|
@@ -71,3 +71,6 @@ fc.assert(property, { seed: 1234567890 })
|
|
|
71
71
|
- [Mutation Testing Standards](mutation-testing.md) — complement to PBT
|
|
72
72
|
- [Testing Standards](testing-standards.md) — overall test pyramid
|
|
73
73
|
- [Adversarial Test Standards](adversarial-test.md) — security-focused fuzzing
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
**Scope**: universal
|
|
@@ -67,3 +67,6 @@ escalation: # required
|
|
|
67
67
|
- XSPEC-046: Cross-project specification
|
|
68
68
|
- Depends on: Failure Source Taxonomy (XSPEC-045)
|
|
69
69
|
- Borrowed from: [ultraworkers/claw-code](https://github.com/ultraworkers/claw-code) ROADMAP Phase 3 Recovery Recipes
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
**Scope**: universal
|
|
@@ -191,3 +191,6 @@ Generate a Markdown table alongside the YAML for inclusion in release notes:
|
|
|
191
191
|
- `supply-chain-attestation.ai.yaml` — SBOM and provenance
|
|
192
192
|
- `testing.ai.yaml` — overall test strategy
|
|
193
193
|
- `deployment-standards.ai.yaml` — release gate integration
|
|
194
|
+
|
|
195
|
+
|
|
196
|
+
**Scope**: universal
|
|
@@ -84,3 +84,6 @@ describe("Guardian replay fixtures", () => {
|
|
|
84
84
|
- [Adversarial Test Standards](adversarial-test.md) — red-team corpus
|
|
85
85
|
- [Verification Evidence Standards](verification-evidence.md) — AC traceability
|
|
86
86
|
- [Testing Standards](testing.md) — overall test pyramid
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
**Scope**: universal
|
|
@@ -60,3 +60,6 @@ wait_ms = min(cap_ms, base_ms * 2^attempt) * (0.5 + random() * 0.5)
|
|
|
60
60
|
- DEC-043: UDS 覆蓋完整性路線圖(驅動來源)
|
|
61
61
|
- Related: `circuit-breaker`, `failure-source-taxonomy`, `timeout-standards`, `recovery-recipe-registry`
|
|
62
62
|
- Industry: Netflix Hystrix retry, Google SRE Book Ch.22, AWS Architecture Blog — exponential backoff and jitter
|
|
63
|
+
|
|
64
|
+
|
|
65
|
+
**Scope**: universal
|
|
@@ -102,3 +102,6 @@
|
|
|
102
102
|
- [deployment-standards.md](deployment-standards.md) — 部署基礎策略
|
|
103
103
|
- [cd-deployment-strategies.md](cd-deployment-strategies.md) — 部署策略選用矩陣
|
|
104
104
|
- AI 格式:[../ai/standards/rollback-standards.ai.yaml](../ai/standards/rollback-standards.ai.yaml)
|
|
105
|
+
|
|
106
|
+
|
|
107
|
+
**Scope**: universal
|