@metasession.co/devaudit-cli 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +13 -10
- package/dist/index.js +17 -5
- package/dist/index.js.map +1 -1
- package/package.json +9 -5
- package/scripts/upload-evidence.sh +225 -0
- package/sdlc/.claude/settings.local.json +11 -0
- package/sdlc/CLAUDE.md +73 -0
- package/sdlc/HOST_ADAPTER.md +127 -0
- package/sdlc/SKILLS.md +137 -0
- package/sdlc/STACK_ADAPTER.md +130 -0
- package/sdlc/ai-rules/INSTRUCTIONS-SDLC.md +172 -0
- package/sdlc/ai-rules/README.md +103 -0
- package/sdlc/ai-rules/SDLC_RULES.md +584 -0
- package/sdlc/ai-rules/claude/CLAUDE.md +192 -0
- package/sdlc/ai-rules/cursor/.cursorrules +167 -0
- package/sdlc/ai-rules/windsurf/.windsurfrules +167 -0
- package/sdlc/article.md +219 -0
- package/sdlc/files/_common/0-project-setup.md +410 -0
- package/sdlc/files/_common/1-plan-requirement.md +381 -0
- package/sdlc/files/_common/2-implement-and-test.md +276 -0
- package/sdlc/files/_common/3-compile-evidence.md +603 -0
- package/sdlc/files/_common/4-submit-for-review.md +362 -0
- package/sdlc/files/_common/5-deploy-main.md +251 -0
- package/sdlc/files/_common/Periodic_Security_Review_Schedule.md +169 -0
- package/sdlc/files/_common/README_TEMPLATE.md +441 -0
- package/sdlc/files/_common/Test_Architecture.md +461 -0
- package/sdlc/files/_common/Test_Plan_TEMPLATE.md +311 -0
- package/sdlc/files/_common/Test_Policy.md +277 -0
- package/sdlc/files/_common/Test_Strategy.md +359 -0
- package/sdlc/files/_common/github/ISSUE_TEMPLATE/bug.yml +75 -0
- package/sdlc/files/_common/github/ISSUE_TEMPLATE/config.yml +11 -0
- package/sdlc/files/_common/github/ISSUE_TEMPLATE/requirement.yml +75 -0
- package/sdlc/files/_common/github/ISSUE_TEMPLATE/task.yml +48 -0
- package/sdlc/files/_common/github/pull_request_template.md +69 -0
- package/sdlc/files/_common/implementing-an-sdlc-issue.md +413 -0
- package/sdlc/files/_common/scripts/derive-release-version.sh +40 -0
- package/sdlc/files/_common/scripts/derive-release-version.test.sh +98 -0
- package/sdlc/files/_common/scripts/submit-for-uat-review.sh +162 -0
- package/sdlc/files/_common/scripts/validate-commits.sh +83 -0
- package/sdlc/files/_common/scripts/validate-compliance-artifacts.sh +202 -0
- package/sdlc/files/_common/scripts/validate-compliance-artifacts.test.sh +202 -0
- package/sdlc/files/_common/skills/_schema/skill.schema.json +36 -0
- package/sdlc/files/_common/skills/e2e-test-engineer/SKILL.md +254 -0
- package/sdlc/files/_common/skills/e2e-test-engineer/references/bootstrap.md +244 -0
- package/sdlc/files/_common/skills/e2e-test-engineer/references/evidence.ts +40 -0
- package/sdlc/files/_common/skills/sdlc-implementer/SKILL.md +189 -0
- package/sdlc/files/_common/skills/sdlc-implementer/references/call-graph.md +64 -0
- package/sdlc/files/_common/skills/sdlc-implementer/references/change-request-loop.md +192 -0
- package/sdlc/files/_common/skills/sdlc-implementer/references/compliance-constraints.md +81 -0
- package/sdlc/files/ci/check-release-approval.yml.template +201 -0
- package/sdlc/files/ci/ci-status-fallback.yml.template +41 -0
- package/sdlc/files/ci/ci.yml.template +390 -0
- package/sdlc/files/ci/compliance-evidence.yml.template +161 -0
- package/sdlc/files/ci/compliance-validation.yml.template +34 -0
- package/sdlc/files/ci/post-deploy-prod.yml.template +159 -0
- package/sdlc/files/ci/python/ci.yml.template +335 -0
- package/sdlc/files/hosts/_schema/adapter.schema.json +103 -0
- package/sdlc/files/hosts/railway/adapter.json +32 -0
- package/sdlc/files/sdlc-config.example.json +74 -0
- package/sdlc/files/stacks/_schema/adapter.schema.json +151 -0
- package/sdlc/files/stacks/node/adapter.json +54 -0
- package/sdlc/files/stacks/node/hooks/.prettierrc.json +9 -0
- package/sdlc/files/stacks/node/hooks/commit-msg +7 -0
- package/sdlc/files/stacks/node/hooks/commitlint.config.mjs +64 -0
- package/sdlc/files/stacks/node/hooks/lint-staged.config.mjs +16 -0
- package/sdlc/files/stacks/node/hooks/pre-commit +13 -0
- package/sdlc/files/stacks/node/hooks/pre-push +15 -0
- package/sdlc/files/stacks/node/scripts/check-requirement-jsdoc.sh +54 -0
- package/sdlc/files/stacks/python/adapter.json +36 -0
- package/sdlc/files/stacks/python/hooks/.pre-commit-config.yaml +51 -0
|
@@ -0,0 +1,359 @@
|
|
|
1
|
+
# Test Strategy
|
|
2
|
+
|
|
3
|
+
**Document Type:** Strategy | **Version:** 3.0 | **Effective Date:** March 2026 | **Review Cycle:** Annual
|
|
4
|
+
|
|
5
|
+
**Owner:** QA Team / Test Engineers | **Approved By:** Engineering Leadership
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Purpose
|
|
10
|
+
|
|
11
|
+
This Test Strategy defines the methodological approach to testing across Metasession. It establishes how we plan, execute, and evaluate testing — the testing levels, security methodology, AI testing protocol, defect lifecycle, traceability approach, risk classification, and evidence requirements.
|
|
12
|
+
|
|
13
|
+
This document answers **"how we approach testing."** For why we test and what we commit to, see the Test Policy (`sdlc/files/Test_Policy.md` in DevAudit). For specific tools, patterns, and code standards, see the Test Architecture.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## Scope
|
|
18
|
+
|
|
19
|
+
This strategy applies to all testing activities across Metasession products, client engagements, and internal systems. It covers all testing levels and all code regardless of authorship.
|
|
20
|
+
|
|
21
|
+
Project-specific details (environment URLs, test counts, infrastructure) are defined in each project's Test Plan.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Testing Levels
|
|
26
|
+
|
|
27
|
+
Metasession implements a multi-level testing approach. Each level serves a distinct purpose in the verification and validation chain.
|
|
28
|
+
|
|
29
|
+
### Unit Testing
|
|
30
|
+
|
|
31
|
+
Tests individual functions, methods, and components in isolation. Executed by developers during development. External dependencies (APIs, databases, file system) are mocked. Automated in CI/CD. Coverage targets defined in the Test Architecture.
|
|
32
|
+
|
|
33
|
+
### Integration Testing
|
|
34
|
+
|
|
35
|
+
Validates interactions between system components — API contracts, service integrations, and data flows. Focuses on boundaries between modules and third-party services. Executed in dedicated integration environments.
|
|
36
|
+
|
|
37
|
+
### System Testing (E2E)
|
|
38
|
+
|
|
39
|
+
End-to-end validation of complete user workflows from UI to database. Primary responsibility of the QA team. Automated using BDD frameworks that map acceptance criteria to executable specifications. Covers 100% of critical user paths.
|
|
40
|
+
|
|
41
|
+
### Acceptance Testing
|
|
42
|
+
|
|
43
|
+
Validates that requirements and acceptance criteria are met from a business perspective. Conducted in staging environments mirroring production. Requires sign-off from Product Managers. May include formal UAT with stakeholders for regulated features.
|
|
44
|
+
|
|
45
|
+
### Security Testing
|
|
46
|
+
|
|
47
|
+
A layered approach combining per-commit automated scanning with periodic manual assessment. See Section: Security Testing Methodology for full details.
|
|
48
|
+
|
|
49
|
+
### Performance Testing
|
|
50
|
+
|
|
51
|
+
Load testing, stress testing, and scalability validation. Baselines established for all products. Executed before major releases or infrastructure changes.
|
|
52
|
+
|
|
53
|
+
### Verification and Validation (V&V)
|
|
54
|
+
|
|
55
|
+
**Verification** — "Did we build the system correctly?" Unit tests, integration tests, security scans, type checking. Confirms technical correctness. Automated and continuous.
|
|
56
|
+
|
|
57
|
+
**Validation** — "Did we build the correct system?" Acceptance testing against business requirements. End-to-end user workflow testing. Product Manager sign-off confirms the system meets intent. For regulated projects, may require formal UAT with documented evidence.
|
|
58
|
+
|
|
59
|
+
Both are required before any release. The project's Test Plan specifies which validation activities apply based on regulatory context.
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## Security Testing Methodology
|
|
64
|
+
|
|
65
|
+
### Per-Commit Gates (Mandatory for All Changes)
|
|
66
|
+
|
|
67
|
+
Every push to `develop` triggers the full CI pipeline. All gates run on both develop pushes and PRs to main. Gate results are auto-uploaded to DevAudit (environment=uat) on develop pushes. Every code change must pass these automated gates:
|
|
68
|
+
|
|
69
|
+
**Static Application Security Testing (SAST)** — Scans source code for vulnerability patterns including injection, insecure defaults, and improper error handling. Exit criteria: 0 high/critical findings. Medium findings require documented justification or remediation plan.
|
|
70
|
+
|
|
71
|
+
**Dependency and Supply Chain Scanning (SCA)** — Audits all direct and transitive dependencies for known CVEs. Flags outdated, unmaintained, or fabricated packages. Exit criteria: 0 high/critical vulnerabilities. Runs on every change modifying dependency manifests.
|
|
72
|
+
|
|
73
|
+
**Type Checking and Linting** — Strict mode compilation and linting catch type errors, unsafe patterns, and code quality issues. Exit criteria: 0 errors.
|
|
74
|
+
|
|
75
|
+
Specific tools implementing these gates are defined in the Test Architecture.
|
|
76
|
+
|
|
77
|
+
### Per-Release Security Activities
|
|
78
|
+
|
|
79
|
+
**Access Control Testing** — Verifies RBAC across all protected endpoints. Tests that unauthenticated requests are rejected and role boundaries enforced. Required for releases touching authentication, authorization, or API endpoints.
|
|
80
|
+
|
|
81
|
+
**Audit Log Verification** — Confirms auditable actions produce log entries with timestamp, user identity, action, and affected resource. Tests that logs cannot be modified by application users. Required for releases touching auditable functionality.
|
|
82
|
+
|
|
83
|
+
**Post-Deploy Security Verification** — Access control spot-check in production, security header verification, confirmation that no debug information is exposed to unauthenticated users.
|
|
84
|
+
|
|
85
|
+
### Periodic Security Activities
|
|
86
|
+
|
|
87
|
+
Defined in the Periodic Security Review Schedule:
|
|
88
|
+
|
|
89
|
+
| Activity | Frequency |
|
|
90
|
+
|---|---|
|
|
91
|
+
| Full codebase SAST review | Quarterly |
|
|
92
|
+
| Dependency deep audit | Quarterly |
|
|
93
|
+
| Access control review | Quarterly |
|
|
94
|
+
| Audit log integrity review | Quarterly |
|
|
95
|
+
| Penetration testing (third party) | Annually |
|
|
96
|
+
| Disaster recovery test | Annually |
|
|
97
|
+
| Third-party security assessment | Annually |
|
|
98
|
+
|
|
99
|
+
### Remediation SLAs
|
|
100
|
+
|
|
101
|
+
| Severity | Per-Commit Gate | Periodic Finding |
|
|
102
|
+
|---|---|---|
|
|
103
|
+
| Critical | Block merge, fix immediately | 7 days |
|
|
104
|
+
| High | Block merge, fix immediately | 30 days |
|
|
105
|
+
| Medium | Document and plan remediation | 90 days |
|
|
106
|
+
| Low | Track for next review | Next quarterly review |
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## AI-Assisted Development Testing Methodology
|
|
111
|
+
|
|
112
|
+
This section implements the AI governance commitments from the Test Policy with specific testing procedures.
|
|
113
|
+
|
|
114
|
+
### Risk Classification for AI-Generated Code
|
|
115
|
+
|
|
116
|
+
AI involvement is a factor in risk classification:
|
|
117
|
+
|
|
118
|
+
| Code Category | Base Risk | AI Adjustment |
|
|
119
|
+
|---|---|---|
|
|
120
|
+
| Internal tools, no regulated data | Low | Remains Low |
|
|
121
|
+
| User-facing features, API changes, PII handling | Medium | Remains Medium |
|
|
122
|
+
| Auth, payments, RBAC, crypto, data validation | High | Remains High |
|
|
123
|
+
| Any of the above with AI regeneration | Any | Raise by one level |
|
|
124
|
+
|
|
125
|
+
### Mandatory Human Review Process
|
|
126
|
+
|
|
127
|
+
**Review scope by risk level:**
|
|
128
|
+
|
|
129
|
+
| Risk Level | Reviewer | Focus |
|
|
130
|
+
|---|---|---|
|
|
131
|
+
| Low | Any team member with domain knowledge | Functional correctness, obvious security issues |
|
|
132
|
+
| Medium | Developer experienced in affected area | Above + security implications, dependency validation |
|
|
133
|
+
| High | Senior developer + security-aware review | Above + independent verification, threat modeling |
|
|
134
|
+
|
|
135
|
+
**Every reviewer checks AI-generated code for:**
|
|
136
|
+
|
|
137
|
+
1. **Correctness** — Does it do what it claims? AI produces plausible code with subtle logic errors.
|
|
138
|
+
2. **Security** — Injection vulnerabilities, insecure defaults, improper error handling?
|
|
139
|
+
3. **Dependencies** — All imports real, current, vulnerability-free? AI fabricates package names.
|
|
140
|
+
4. **Hardcoded values** — Test data, placeholder credentials, debug flags?
|
|
141
|
+
5. **Consistency** — Follows project conventions, or has AI introduced alien patterns?
|
|
142
|
+
|
|
143
|
+
The PR approval process serves as the formal review gate. Reviewer identity and timestamp are recorded immutably.
|
|
144
|
+
|
|
145
|
+
### Regeneration Protocol
|
|
146
|
+
|
|
147
|
+
When a developer regenerates a component from scratch (not incremental editing):
|
|
148
|
+
|
|
149
|
+
1. Document the regeneration — which component, when, why
|
|
150
|
+
2. Full retest of the regenerated component
|
|
151
|
+
3. Retest all dependent components
|
|
152
|
+
4. Do not assume equivalence — two AI outputs for the same prompt are not functionally equivalent until tests prove it
|
|
153
|
+
|
|
154
|
+
Incremental AI-assisted edits follow standard testing gates.
|
|
155
|
+
|
|
156
|
+
### AI Documentation Requirements
|
|
157
|
+
|
|
158
|
+
| Risk Level | Commit Tag | Evidence | Prompts |
|
|
159
|
+
|---|---|---|---|
|
|
160
|
+
| Low | `Co-Authored-By` tag | Not required | Not required |
|
|
161
|
+
| Medium | `Co-Authored-By` tag | Summary of generation | Summary of prompts |
|
|
162
|
+
| High | `Co-Authored-By` tag | Detailed AI record | Detailed prompts and outputs |
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
## Risk-Based Testing Approach
|
|
167
|
+
|
|
168
|
+
### Classification
|
|
169
|
+
|
|
170
|
+
Risk level is determined at planning time for each requirement:
|
|
171
|
+
|
|
172
|
+
**High Risk** — Sensitive data (PII, payments), authentication/authorization, encryption, regulatory compliance features, audit logging, core revenue capabilities, production infrastructure, AI-generated code in any of these.
|
|
173
|
+
|
|
174
|
+
**Medium Risk** — New features, architectural changes, third-party integrations, performance optimizations, AI-generated code for non-security features.
|
|
175
|
+
|
|
176
|
+
**Low Risk** — UI updates without functional changes, configuration, documentation, internal tools with limited impact.
|
|
177
|
+
|
|
178
|
+
### Testing Depth by Risk Level
|
|
179
|
+
|
|
180
|
+
| Activity | Low | Medium | High |
|
|
181
|
+
|---|---|---|---|
|
|
182
|
+
| Unit tests | Required | Required | Required |
|
|
183
|
+
| Integration tests | As applicable | Required | Required |
|
|
184
|
+
| E2E tests | Critical paths | Full coverage | Full coverage |
|
|
185
|
+
| SAST scan | Required | Required | Required |
|
|
186
|
+
| Dependency audit | Required | Required | Required |
|
|
187
|
+
| Access control testing | If applicable | Required | Required |
|
|
188
|
+
| Audit log testing | If applicable | Required | Required |
|
|
189
|
+
| Performance testing | Not required | If applicable | Required |
|
|
190
|
+
| Penetration testing | Not required | Not required | Consider |
|
|
191
|
+
| Independent review | Not required | Not required | Required |
|
|
192
|
+
|
|
193
|
+
---
|
|
194
|
+
|
|
195
|
+
## Defect Management
|
|
196
|
+
|
|
197
|
+
### Severity Classification
|
|
198
|
+
|
|
199
|
+
- **Critical** — System unusable, data loss, security breach. Fix immediately.
|
|
200
|
+
- **Major** — Major feature broken, no workaround. Fix within sprint.
|
|
201
|
+
- **Minor** — Minor issue, workaround exists. Prioritize in backlog.
|
|
202
|
+
|
|
203
|
+
### Lifecycle
|
|
204
|
+
|
|
205
|
+
1. **Detection** — Test failure or manual report
|
|
206
|
+
2. **Triage** — Severity and priority assigned
|
|
207
|
+
3. **Fix** — Developer creates branch linked to issue
|
|
208
|
+
4. **Verification** — QA validates fix
|
|
209
|
+
5. **Closure** — Evidence recorded, audit trail complete
|
|
210
|
+
6. **Traceability** — Commit references issue ID
|
|
211
|
+
|
|
212
|
+
---
|
|
213
|
+
|
|
214
|
+
## Requirements Traceability
|
|
215
|
+
|
|
216
|
+
### Approach
|
|
217
|
+
|
|
218
|
+
Every requirement must be traceable through the complete chain: Requirement → Test Cases → Test Results → Code Commits → PR Review → Deployment.
|
|
219
|
+
|
|
220
|
+
### Implementation
|
|
221
|
+
|
|
222
|
+
- Requirements tracked with unique IDs (format defined per project)
|
|
223
|
+
- Test specifications tagged with requirement IDs
|
|
224
|
+
- Commit messages reference requirement/issue IDs
|
|
225
|
+
- Pull requests link commits to requirements
|
|
226
|
+
- Test results linked to requirement IDs
|
|
227
|
+
- Traceability matrix maintained per project (format in project Test Plan)
|
|
228
|
+
|
|
229
|
+
### Audit Readiness
|
|
230
|
+
|
|
231
|
+
- Vertical slice capability: trace any code change to its requirement and risk assessment
|
|
232
|
+
- Complete linkage chain preserved permanently in Git history
|
|
233
|
+
- Historical evidence accessible for all past releases via tags and archives
|
|
234
|
+
|
|
235
|
+
---
|
|
236
|
+
|
|
237
|
+
## Git Workflow and Change Control
|
|
238
|
+
|
|
239
|
+
### Branching Strategy
|
|
240
|
+
|
|
241
|
+
Projects may use trunk-based with integration branch or direct trunk-based flow. The specific pattern is defined in each project's Test Plan.
|
|
242
|
+
|
|
243
|
+
**Universal requirements:**
|
|
244
|
+
|
|
245
|
+
- Production branch protected: requires PR approvals, all status checks must pass, no direct pushes
|
|
246
|
+
- Integration branch (if used): requires PR approval, all checks pass
|
|
247
|
+
- Merge strategy preserves audit trail (merge commits for compliance, squash for feature branches)
|
|
248
|
+
|
|
249
|
+
### Change Control Process
|
|
250
|
+
|
|
251
|
+
1. **Implement** — Code on working branch with requirement references
|
|
252
|
+
2. **Local gates** — Type checking, linting, security scan, tests
|
|
253
|
+
3. **Push to develop** — Triggers full CI pipeline (all gates), evidence auto-uploaded to DevAudit, auto-deploys to UAT
|
|
254
|
+
4. **UAT review** — Review evidence in DevAudit, approve UAT release
|
|
255
|
+
5. **PR to main** — CI re-runs gates independently, DevAudit UAT approval check verified
|
|
256
|
+
6. **Review** — Human reviews code, security, AI involvement, compliance artifacts
|
|
257
|
+
7. **Approve** — Immutable record of reviewer identity and timestamp
|
|
258
|
+
8. **Merge** — Triggers production deployment
|
|
259
|
+
9. **Post-deploy** — Smoke tests against production, evidence uploaded to DevAudit (environment=production)
|
|
260
|
+
10. **Prod review** — Approve production release in DevAudit
|
|
261
|
+
|
|
262
|
+
The four-eyes principle is enforced through mandatory PR reviews. The PR approval constitutes formal sign-off for compliance purposes.
|
|
263
|
+
|
|
264
|
+
### Commit Standards
|
|
265
|
+
|
|
266
|
+
Conventional Commits format: `type(scope): description`
|
|
267
|
+
|
|
268
|
+
Types: feat, fix, docs, style, refactor, test, chore, perf, ci, build, revert, compliance, security
|
|
269
|
+
|
|
270
|
+
AI-assisted commits include: `Co-Authored-By: [Tool] <noreply@provider.com>`
|
|
271
|
+
Tracked requirements include: `Ref: REQ-XXX`
|
|
272
|
+
|
|
273
|
+
---
|
|
274
|
+
|
|
275
|
+
## Evidence Requirements
|
|
276
|
+
|
|
277
|
+
### By Phase
|
|
278
|
+
|
|
279
|
+
**Planning** — Requirements with acceptance criteria, risk classification, AI use intent (if applicable).
|
|
280
|
+
|
|
281
|
+
**Development** — PR with code review (four-eyes principle), commit messages with traceability, `Co-Authored-By` tags, branch protection enforcement, SAST and dependency scan results.
|
|
282
|
+
|
|
283
|
+
**Testing** — Test execution logs with timestamps and results, security scan evidence, screenshot/video capture for failures, AI review documentation.
|
|
284
|
+
|
|
285
|
+
**Release** — Completion report with Go/No-Go recommendation, security evidence summary, sign-off from designated approver.
|
|
286
|
+
|
|
287
|
+
### Retention
|
|
288
|
+
|
|
289
|
+
- CI/CD artifacts: per platform defaults (typically 90 days)
|
|
290
|
+
- Long-term storage: minimum 3 years for compliance
|
|
291
|
+
- All evidence tagged with release version
|
|
292
|
+
- Audit-ready retrieval within 24 hours
|
|
293
|
+
|
|
294
|
+
### Required Artifact Types
|
|
295
|
+
|
|
296
|
+
| Category | Artifacts |
|
|
297
|
+
|---|---|
|
|
298
|
+
| Planning | Test Policy, Test Strategy, Project Test Plans |
|
|
299
|
+
| Specification | BDD feature files, test case specifications, security scenarios |
|
|
300
|
+
| Execution | Test logs, CI/CD logs, SAST/SCA reports, AI use records, defect reports |
|
|
301
|
+
| Reporting | Status reports, completion reports, security summaries |
|
|
302
|
+
|
|
303
|
+
---
|
|
304
|
+
|
|
305
|
+
## Agile Artifact Mapping
|
|
306
|
+
|
|
307
|
+
| ISO Artifact | Metasession Implementation |
|
|
308
|
+
|---|---|
|
|
309
|
+
| Test Policy | This document hierarchy (Policy + Strategy + Architecture) |
|
|
310
|
+
| Requirements Specification | Product backlog with acceptance criteria |
|
|
311
|
+
| Test Plan | Project-specific Test Plan + sprint planning |
|
|
312
|
+
| Test Case Specification | BDD feature files (Gherkin Given/When/Then) |
|
|
313
|
+
| Test Execution Log | CI/CD pipeline logs, test management tool records |
|
|
314
|
+
| Defect Reports | Issue tracker with severity labels and workflows |
|
|
315
|
+
| Traceability Matrix | Requirement-test-defect linkage (RTM or tool-based) |
|
|
316
|
+
| Security Evidence | SAST/SCA results, dependency audits, security summaries |
|
|
317
|
+
| AI Audit Trail | Co-Authored-By tags, evidence directory records, PR history |
|
|
318
|
+
|
|
319
|
+
---
|
|
320
|
+
|
|
321
|
+
## Reporting Cadence
|
|
322
|
+
|
|
323
|
+
- **Daily** — Test execution status during active phases
|
|
324
|
+
- **Weekly** — Progress and defect trends during sprints
|
|
325
|
+
- **Sprint end** — Completion report for sprint deliverables
|
|
326
|
+
- **Release** — Comprehensive completion report with security summary and Go/No-Go
|
|
327
|
+
- **Quarterly** — Strategy effectiveness review, periodic security results, compliance audit
|
|
328
|
+
|
|
329
|
+
---
|
|
330
|
+
|
|
331
|
+
## Continuous Improvement
|
|
332
|
+
|
|
333
|
+
This strategy is reviewed:
|
|
334
|
+
|
|
335
|
+
- Annually as part of regular cycle
|
|
336
|
+
- When significant process or tooling changes occur
|
|
337
|
+
- Following major audits or compliance assessments
|
|
338
|
+
- When AI tooling or regulatory guidance changes
|
|
339
|
+
- When gaps are identified
|
|
340
|
+
|
|
341
|
+
Sprint retrospectives, quarterly metric reviews, and incident lessons learned feed into strategy updates.
|
|
342
|
+
|
|
343
|
+
---
|
|
344
|
+
|
|
345
|
+
## Document Control
|
|
346
|
+
|
|
347
|
+
| Version | Date | Author | Changes |
|
|
348
|
+
|---|---|---|---|
|
|
349
|
+
| 1.0 | January 2026 | QA Team | Initial creation |
|
|
350
|
+
| 2.0 | March 2026 | QA Team | Added AI methodology, security methodology, V&V |
|
|
351
|
+
| 3.0 | March 2026 | QA Team | Clean boundary split — moved specific tools, code patterns, CI config to Test Architecture. Strategy now owns methodology only. |
|
|
352
|
+
|
|
353
|
+
**Next Review Date:** March 2027
|
|
354
|
+
|
|
355
|
+
**Related Documents:** Test Policy, Test Architecture, Periodic Security Review Schedule, Project Test Plans (in devaudit/sdlc/files/)
|
|
356
|
+
|
|
357
|
+
---
|
|
358
|
+
|
|
359
|
+
**Strategy Status:** Approved | **Effective Date:** March 2026
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
name: Bug
|
|
2
|
+
description: Defect in shipped behaviour. Full SDLC if it touches a high-risk surface; otherwise lighter.
|
|
3
|
+
title: "[BUG] <one-line symptom>"
|
|
4
|
+
labels: ["bug", "needs-triage"]
|
|
5
|
+
body:
|
|
6
|
+
- type: markdown
|
|
7
|
+
attributes:
|
|
8
|
+
value: |
|
|
9
|
+
Use this template for defects in shipped behaviour. Bug fixes follow the SDLC at the depth dictated by the affected surface — see Test_Policy.md §Risk-Based Testing and the trivial-change escape hatch in the [SDLC walkthrough](https://github.com/metasession-dev/devaudit/blob/main/docs/implementing-an-sdlc-issue.md#trivial-change-escape-hatch).
|
|
10
|
+
|
|
11
|
+
**Security vulnerabilities: do NOT open a public issue.** See `SECURITY.md` (if present in this repo) for the private disclosure path.
|
|
12
|
+
|
|
13
|
+
- type: textarea
|
|
14
|
+
id: summary
|
|
15
|
+
attributes:
|
|
16
|
+
label: What's broken
|
|
17
|
+
description: One paragraph. What happens vs what should happen.
|
|
18
|
+
validations:
|
|
19
|
+
required: true
|
|
20
|
+
|
|
21
|
+
- type: textarea
|
|
22
|
+
id: repro
|
|
23
|
+
attributes:
|
|
24
|
+
label: Reproduction
|
|
25
|
+
description: Step-by-step. Include URLs, commands, payloads. The more deterministic, the faster the fix.
|
|
26
|
+
validations:
|
|
27
|
+
required: true
|
|
28
|
+
|
|
29
|
+
- type: textarea
|
|
30
|
+
id: expected
|
|
31
|
+
attributes:
|
|
32
|
+
label: Expected vs actual
|
|
33
|
+
description: Explicit contrast helps triage and writes the test case.
|
|
34
|
+
validations:
|
|
35
|
+
required: true
|
|
36
|
+
|
|
37
|
+
- type: textarea
|
|
38
|
+
id: environment
|
|
39
|
+
attributes:
|
|
40
|
+
label: Environment
|
|
41
|
+
description: Browser/OS, runtime version, deployment environment, commit SHA if relevant.
|
|
42
|
+
validations:
|
|
43
|
+
required: true
|
|
44
|
+
|
|
45
|
+
- type: dropdown
|
|
46
|
+
id: surface_risk
|
|
47
|
+
attributes:
|
|
48
|
+
label: Affected surface — risk class
|
|
49
|
+
description: "Same scale as the Requirement template. Defects in HIGH/CRITICAL surfaces still need stage-1 risk classification and full evidence even if the fix is small."
|
|
50
|
+
options:
|
|
51
|
+
- "LOW — UI polish, low-traffic admin tooling, no security/data/payment surface"
|
|
52
|
+
- "MEDIUM — user-visible feature defect, no security/data risk"
|
|
53
|
+
- "HIGH — auth, payments, RBAC, evidence/audit storage, third-party data egress"
|
|
54
|
+
- "CRITICAL — regulated data, AI making decisions about people, production infra"
|
|
55
|
+
- "Unsure — defer to stage-1 plan to classify"
|
|
56
|
+
validations:
|
|
57
|
+
required: true
|
|
58
|
+
|
|
59
|
+
- type: textarea
|
|
60
|
+
id: workaround
|
|
61
|
+
attributes:
|
|
62
|
+
label: Known workaround (if any)
|
|
63
|
+
description: What lets the user keep working until this is fixed. "None" is a valid answer.
|
|
64
|
+
validations:
|
|
65
|
+
required: false
|
|
66
|
+
|
|
67
|
+
- type: checkboxes
|
|
68
|
+
id: sdlc-prep
|
|
69
|
+
attributes:
|
|
70
|
+
label: SDLC prep
|
|
71
|
+
options:
|
|
72
|
+
- label: I confirm this is NOT a security vulnerability. (If it is, close this and follow `SECURITY.md`.)
|
|
73
|
+
required: true
|
|
74
|
+
- label: I've searched existing issues and this isn't a duplicate.
|
|
75
|
+
required: true
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
blank_issues_enabled: false
|
|
2
|
+
contact_links:
|
|
3
|
+
- name: 📚 What is DevAudit
|
|
4
|
+
url: https://github.com/metasession-dev/devaudit/blob/main/docs/what-is-devaudit.md
|
|
5
|
+
about: The three-pillar overview — start here if you're new to the framework.
|
|
6
|
+
- name: 📋 Standards coverage
|
|
7
|
+
url: https://github.com/metasession-dev/devaudit/blob/main/docs/standards-coverage.md
|
|
8
|
+
about: Clause-by-clause mapping from SDLC artefacts to ISO 29119, ISO 27001, SOC 2, GDPR, EU AI Act.
|
|
9
|
+
- name: 🛠️ Implementing an SDLC issue
|
|
10
|
+
url: https://github.com/metasession-dev/devaudit/blob/main/docs/implementing-an-sdlc-issue.md
|
|
11
|
+
about: End-to-end walkthrough — GitHub issue → merged. Required reading before opening a Requirement.
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
name: Requirement
|
|
2
|
+
description: A new feature, enhancement, or significant change. Goes through full SDLC stages 1–5.
|
|
3
|
+
title: "[REQ] <one-line summary>"
|
|
4
|
+
labels: ["requirement", "needs-triage"]
|
|
5
|
+
body:
|
|
6
|
+
- type: markdown
|
|
7
|
+
attributes:
|
|
8
|
+
value: |
|
|
9
|
+
Use this template for **new features, enhancements, or significant changes** that need full SDLC traceability — anything an auditor or stakeholder might later ask "was this tested?".
|
|
10
|
+
|
|
11
|
+
For bug fixes that touch the same high-risk surfaces, use [Bug](./bug.yml).
|
|
12
|
+
For typo fixes, doc tweaks, dependency bumps, or other zero-risk chores, use [Task](./task.yml).
|
|
13
|
+
|
|
14
|
+
End-to-end walkthrough: [Implementing an SDLC issue](https://github.com/metasession-dev/devaudit/blob/main/docs/implementing-an-sdlc-issue.md) — stages 1–5, sample AI prompts, the skills inventory.
|
|
15
|
+
|
|
16
|
+
- type: textarea
|
|
17
|
+
id: context
|
|
18
|
+
attributes:
|
|
19
|
+
label: Context / why
|
|
20
|
+
description: What problem does this solve? Who is the user, and what are they trying to do? Why now?
|
|
21
|
+
validations:
|
|
22
|
+
required: true
|
|
23
|
+
|
|
24
|
+
- type: textarea
|
|
25
|
+
id: acceptance
|
|
26
|
+
attributes:
|
|
27
|
+
label: Acceptance criteria
|
|
28
|
+
description: Bulleted, testable conditions. "Done" means every box is checked.
|
|
29
|
+
placeholder: |
|
|
30
|
+
- [ ] ...
|
|
31
|
+
- [ ] ...
|
|
32
|
+
- [ ] ...
|
|
33
|
+
validations:
|
|
34
|
+
required: true
|
|
35
|
+
|
|
36
|
+
- type: dropdown
|
|
37
|
+
id: suspected_risk
|
|
38
|
+
attributes:
|
|
39
|
+
label: Suspected risk class
|
|
40
|
+
description: "Per Test_Policy.md §Risk-Based Testing. Final classification happens in stage 1; this is your first-cut signal."
|
|
41
|
+
options:
|
|
42
|
+
- "LOW — UI polish, low-traffic admin tooling, no security/data/payment surface"
|
|
43
|
+
- "MEDIUM — new feature, integration, refactor with user-visible effect"
|
|
44
|
+
- "HIGH — auth, payments, RBAC, evidence/audit storage, third-party data egress"
|
|
45
|
+
- "CRITICAL — regulated data, AI making decisions about people, production infra"
|
|
46
|
+
- "Unsure — defer to stage-1 plan to classify"
|
|
47
|
+
validations:
|
|
48
|
+
required: true
|
|
49
|
+
|
|
50
|
+
- type: textarea
|
|
51
|
+
id: dependencies
|
|
52
|
+
attributes:
|
|
53
|
+
label: Dependencies
|
|
54
|
+
description: Other issues, infra prerequisites, external-team blockers. "None" is a valid answer.
|
|
55
|
+
validations:
|
|
56
|
+
required: true
|
|
57
|
+
|
|
58
|
+
- type: textarea
|
|
59
|
+
id: out_of_scope
|
|
60
|
+
attributes:
|
|
61
|
+
label: Out of scope
|
|
62
|
+
description: What this issue deliberately doesn't do. Helps reviewers gauge size and prevents scope creep.
|
|
63
|
+
validations:
|
|
64
|
+
required: false
|
|
65
|
+
|
|
66
|
+
- type: checkboxes
|
|
67
|
+
id: sdlc-prep
|
|
68
|
+
attributes:
|
|
69
|
+
label: SDLC prep
|
|
70
|
+
description: Confirm before triage.
|
|
71
|
+
options:
|
|
72
|
+
- label: I've read the SDLC walkthrough (https://github.com/metasession-dev/devaudit/blob/main/docs/implementing-an-sdlc-issue.md) and understand which stages this requirement will pass through.
|
|
73
|
+
required: true
|
|
74
|
+
- label: I've checked the existing RTM (`compliance/RTM.md`) for any pre-existing REQ-XXX that covers this. (If yes, link it above and close this issue.)
|
|
75
|
+
required: true
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
name: Task
|
|
2
|
+
description: Chore, refactor, doc update, or dependency bump. Uses the trivial-change escape hatch — skips SDLC stages 1 + 3.
|
|
3
|
+
title: "[TASK] <one-line summary>"
|
|
4
|
+
labels: ["task", "chore"]
|
|
5
|
+
body:
|
|
6
|
+
- type: markdown
|
|
7
|
+
attributes:
|
|
8
|
+
value: |
|
|
9
|
+
Use this template for **zero-risk chores**: typo fixes, formatting, dependency bumps, doc updates, refactors that produce no behaviour change. These follow the [trivial-change escape hatch](https://github.com/metasession-dev/devaudit/blob/main/docs/implementing-an-sdlc-issue.md#trivial-change-escape-hatch) — skip SDLC stages 1 (plan) and 3 (evidence), but **still run all gates locally** before pushing.
|
|
10
|
+
|
|
11
|
+
If the chore turns out to be non-trivial in flight, convert this issue to a [Requirement](./requirement.yml) and follow full stages 1–5.
|
|
12
|
+
|
|
13
|
+
- type: textarea
|
|
14
|
+
id: what
|
|
15
|
+
attributes:
|
|
16
|
+
label: What
|
|
17
|
+
description: One sentence describing the change.
|
|
18
|
+
validations:
|
|
19
|
+
required: true
|
|
20
|
+
|
|
21
|
+
- type: textarea
|
|
22
|
+
id: why
|
|
23
|
+
attributes:
|
|
24
|
+
label: Why
|
|
25
|
+
description: The minimum context a reviewer needs to approve without further questions.
|
|
26
|
+
validations:
|
|
27
|
+
required: true
|
|
28
|
+
|
|
29
|
+
- type: textarea
|
|
30
|
+
id: done
|
|
31
|
+
attributes:
|
|
32
|
+
label: Definition of done
|
|
33
|
+
description: Bullet list. "Done" means every box is checked.
|
|
34
|
+
placeholder: |
|
|
35
|
+
- [ ] ...
|
|
36
|
+
- [ ] ...
|
|
37
|
+
validations:
|
|
38
|
+
required: true
|
|
39
|
+
|
|
40
|
+
- type: checkboxes
|
|
41
|
+
id: confirms
|
|
42
|
+
attributes:
|
|
43
|
+
label: Confirmations
|
|
44
|
+
options:
|
|
45
|
+
- label: This change has no user-visible behaviour change and no security/data/auth surface impact.
|
|
46
|
+
required: true
|
|
47
|
+
- label: I will run all local gates before pushing, even though stages 1 + 3 are skipped.
|
|
48
|
+
required: true
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
SDLC-compliant PR template. Fields marked (REQUIRED) are mandatory for Requirement/Bug PRs.
|
|
3
|
+
For chores using the trivial-change escape hatch (TASK issues), delete the REQ / Risk /
|
|
4
|
+
Evidence / Four-eyes / Rollback sections and keep only the Summary + Test plan.
|
|
5
|
+
|
|
6
|
+
Reference: https://github.com/metasession-dev/devaudit/blob/main/docs/implementing-an-sdlc-issue.md
|
|
7
|
+
-->
|
|
8
|
+
|
|
9
|
+
## Summary
|
|
10
|
+
|
|
11
|
+
<!-- 1–3 sentences. What changes, and why. -->
|
|
12
|
+
|
|
13
|
+
## Linked issue (REQUIRED)
|
|
14
|
+
|
|
15
|
+
Closes #
|
|
16
|
+
|
|
17
|
+
## REQ (REQUIRED for Requirement and high-risk Bug PRs; "n/a" for trivial Task PRs)
|
|
18
|
+
|
|
19
|
+
REQ-
|
|
20
|
+
|
|
21
|
+
## Risk class (REQUIRED for Requirement and Bug PRs)
|
|
22
|
+
|
|
23
|
+
<!-- LOW | MEDIUM | HIGH | CRITICAL — per the implementation plan -->
|
|
24
|
+
|
|
25
|
+
Risk:
|
|
26
|
+
|
|
27
|
+
## Evidence link (REQUIRED for Requirement and Bug PRs)
|
|
28
|
+
|
|
29
|
+
<!-- https://devaudit.metasession.co/projects/<slug>/requirements/REQ-XXX -->
|
|
30
|
+
|
|
31
|
+
Evidence:
|
|
32
|
+
|
|
33
|
+
## Four-eyes attestation (REQUIRED for HIGH and CRITICAL risk only)
|
|
34
|
+
|
|
35
|
+
<!-- @username — second reviewer who has signed off on the implementation plan. -->
|
|
36
|
+
|
|
37
|
+
Reviewer:
|
|
38
|
+
|
|
39
|
+
## Rollback plan (REQUIRED for HIGH and CRITICAL risk only)
|
|
40
|
+
|
|
41
|
+
<!-- One paragraph. What reverts the change if production smoke fails or an incident is declared.
|
|
42
|
+
Reference: compliance/plans/REQ-XXX/implementation-plan.md §Rollback -->
|
|
43
|
+
|
|
44
|
+
## Test plan
|
|
45
|
+
|
|
46
|
+
<!--
|
|
47
|
+
Bulleted checklist of tests / verifications. CI runs the gates automatically; this is the
|
|
48
|
+
narrative the human reviewer follows to confirm the change does what the issue asked.
|
|
49
|
+
-->
|
|
50
|
+
|
|
51
|
+
- [ ]
|
|
52
|
+
- [ ]
|
|
53
|
+
- [ ]
|
|
54
|
+
|
|
55
|
+
## SDLC checklist (REQUIRED)
|
|
56
|
+
|
|
57
|
+
<!-- All required boxes must be checked before the reviewer hits Approve. -->
|
|
58
|
+
|
|
59
|
+
- [ ] All quality gates pass locally (per the consumer project's stack adapter: type-check, lint, unit + e2e tests, SAST, dep audit)
|
|
60
|
+
- [ ] `compliance/RTM.md` updated (Requirement / Bug PRs)
|
|
61
|
+
- [ ] Evidence uploaded to the portal under REQ-XXX (Requirement / Bug PRs)
|
|
62
|
+
- [ ] Conventional Commit format with `Ref: REQ-XXX` trailer in every commit (Requirement / Bug PRs)
|
|
63
|
+
- [ ] `Co-Authored-By: …` trailer present on commits where an AI tool contributed substantively
|
|
64
|
+
- [ ] No bypass-the-gate patterns: no `--no-verify`, no `eslint-disable`, no `@ts-expect-error`, no `xfail`
|
|
65
|
+
|
|
66
|
+
## Notes for reviewer
|
|
67
|
+
|
|
68
|
+
<!-- Anything that helps the review go faster — known weirdness, deferred follow-ups,
|
|
69
|
+
intentional non-decisions to flag. -->
|