@metasession.co/devaudit-cli 0.1.54 → 0.1.56

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@metasession.co/devaudit-cli",
3
- "version": "0.1.54",
3
+ "version": "0.1.56",
4
4
  "description": "DevAudit CLI — installs, syncs, and operates the Metasession SDLC across consumer projects.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -33,7 +33,8 @@
33
33
  },
34
34
  "dependencies": {
35
35
  "@clack/prompts": "^0.8.2",
36
- "@metasession.co/devaudit-plugin-sdk": "^0.1.54",
36
+ "@metasession.co/devaudit-plugin-sdk": "^0.1.56",
37
+ "ajv": "^8.20.0",
37
38
  "commander": "^12.1.0",
38
39
  "consola": "^3.2.3",
39
40
  "env-paths": "^3.0.0",
package/sdlc/CLAUDE.md CHANGED
@@ -8,8 +8,7 @@ This is a compliance-grade SDLC template system — a set of 12 documents across
8
8
 
9
9
  ## Repository Structure
10
10
 
11
- - `article.md`Long-form article explaining the system's design and rationale
12
- - `files/` — The 12 template documents:
11
+ - `files/`The template documents:
13
12
  - **Tier 1 (universal, never project-specific):** `Test_Policy.md`, `Test_Strategy.md`, `Test_Architecture.md`, `Periodic_Security_Review_Schedule.md`
14
13
  - **Tier 2 (project-specific, customized per project):** `0-project-setup.md` through `5-deploy-main.md` (workflows), `Test_Plan_TEMPLATE.md`, `README_TEMPLATE.md`
15
14
 
@@ -57,7 +56,6 @@ All templates assume these gates: TypeScript (0 errors), SAST/Semgrep (0 high/cr
57
56
  - Maintain the separation between Tier 1 and Tier 2 — if content applies universally, it belongs in Tier 1; if project-specific, Tier 2.
58
57
  - Keep workflow documents as executable procedures with concrete commands, not abstract guidance.
59
58
  - Risk classification rules and AI governance controls are defined in Test_Policy.md and Test_Strategy.md — don't duplicate them in workflow files.
60
- - The article (`article.md`) should stay consistent with the templates; if you change a process in a template, check whether the article describes the old process.
61
59
 
62
60
  ## Syncing Templates to Consuming Projects
63
61
 
@@ -263,4 +263,4 @@ Per the [#119 review](https://github.com/metasession-dev/DevAudit-Installer/issu
263
263
  - `sdlc/files/_common/skills/requirements-aligner/SKILL.md` — sibling skill (same SoT-alignment family); see for the symmetric shape (Stage 1 + Stage 3 hooks; advisory-then-blocking enforcement).
264
264
  - Sibling skill (forthcoming): `risk-register-keeper` (DevAudit-Installer#121).
265
265
  - Meta-reviewer (META-COMPLY): `framework-registry-auditor` reviews the `architecture_decision` evidence type's clause mappings before the META-COMPLY sub-PR opens. Per the [#119 sequencing](https://github.com/metasession-dev/DevAudit-Installer/issues/119#issuecomment-4631840651).
266
- - Existing ADRs in the framework itself: `DevAudit-Installer/docs/ADR/ADR-001-polyglot-sdlc-architecture.md`, `docs/devaudit-cli/ADR-001-language-and-distribution.md` — the pattern this skill maintains.
266
+ - Existing ADRs in the framework itself: [`DevAudit-Installer/docs/ADR/ADR-001-polyglot-sdlc-architecture.md`](https://github.com/metasession-dev/DevAudit-Installer/blob/main/docs/ADR/ADR-001-polyglot-sdlc-architecture.md) — the pattern this skill maintains.
@@ -232,22 +232,41 @@ jobs:
232
232
  register-release:
233
233
  name: Register Release
234
234
  runs-on: {{RUNNER}}
235
- if: ${{ vars.DEVAUDIT_BASE_URL != '' }}
236
235
  outputs:
237
236
  version: ${{ steps.version.outputs.version }}
238
237
  env:
239
- DEVAUDIT_BASE_URL: ${{ vars.DEVAUDIT_BASE_URL }}
238
+ DEVAUDIT_BASE_URL_VAR: ${{ vars.DEVAUDIT_BASE_URL }}
240
239
  DEVAUDIT_API_KEY: ${{ secrets.DEVAUDIT_API_KEY }}
241
240
  steps:
242
241
  - uses: actions/checkout@v4
243
242
 
244
- - name: Validate DevAudit env
243
+ - name: Resolve DevAudit base URL
245
244
  run: |
246
- if [ -z "${DEVAUDIT_BASE_URL}" ] || [ -z "${DEVAUDIT_API_KEY}" ]; then
247
- echo "::error::DEVAUDIT_BASE_URL (variable) and DEVAUDIT_API_KEY (secret) must both be set."
245
+ # Prefer sdlc-config.json devaudit.base_url (PR-visible) over the
246
+ # deprecated repo Variable matching compliance-evidence.yml and
247
+ # check-release-approval.yml. DevAudit-Installer#156: gating this job
248
+ # on `vars.DEVAUDIT_BASE_URL` silently skipped release registration +
249
+ # evidence upload for consumers that moved base_url into
250
+ # sdlc-config.json (the v1.23.0 direction).
251
+ BASE=""
252
+ if [ -f sdlc-config.json ]; then
253
+ CONFIG_URL=$(jq -r '.devaudit.base_url // empty' sdlc-config.json 2>/dev/null || true)
254
+ [ -n "$CONFIG_URL" ] && BASE="$CONFIG_URL"
255
+ fi
256
+ if [ -n "$BASE" ]; then
257
+ echo "Using devaudit.base_url from sdlc-config.json: $BASE"
258
+ elif [ -n "$DEVAUDIT_BASE_URL_VAR" ]; then
259
+ BASE="$DEVAUDIT_BASE_URL_VAR"
260
+ echo "::warning::Using repo Variable DEVAUDIT_BASE_URL (deprecated in v1.23.0). Move base_url to sdlc-config.json devaudit.base_url for PR-visible config."
261
+ else
262
+ echo "::warning::No DevAudit base URL configured — skipping release registration. Set devaudit.base_url in sdlc-config.json."
263
+ fi
264
+ if [ -n "$BASE" ] && [ -z "${DEVAUDIT_API_KEY}" ]; then
265
+ echo "::error::DEVAUDIT_API_KEY (secret) must be set when a DevAudit base URL is configured."
248
266
  exit 1
249
267
  fi
250
- echo "BASE=${DEVAUDIT_BASE_URL%/}" >> "$GITHUB_ENV"
268
+ echo "BASE=${BASE%/}" >> "$GITHUB_ENV"
269
+ echo "DEVAUDIT_BASE_URL=${BASE%/}" >> "$GITHUB_ENV"
251
270
 
252
271
  - name: Determine release version
253
272
  id: version
@@ -263,6 +282,7 @@ jobs:
263
282
  echo "Release version: ${VERSION}"
264
283
 
265
284
  - name: Ensure release exists
285
+ if: env.DEVAUDIT_BASE_URL != ''
266
286
  run: |
267
287
  chmod +x scripts/upload-evidence.sh 2>/dev/null || true
268
288
  # Create the release in DevAudit (no evidence yet — just registration)
@@ -273,6 +293,7 @@ jobs:
273
293
  --git-sha ${{ github.sha }} --branch ${{ github.ref_name }} || true
274
294
 
275
295
  - name: Sync known requirements from RTM
296
+ if: env.DEVAUDIT_BASE_URL != ''
276
297
  env:
277
298
  GH_TOKEN: ${{ github.token }}
278
299
  run: |
@@ -343,13 +364,33 @@ jobs:
343
364
  # evidence — `status=failed` is itself the audit trail. `!cancelled()`
344
365
  # still guards against partial state on operator-cancel.
345
366
  # DevAudit-Installer#96.
346
- if: ${{ always() && !cancelled() && vars.DEVAUDIT_BASE_URL != '' && needs.register-release.result == 'success' }}
367
+ if: ${{ always() && !cancelled() && needs.register-release.result == 'success' }}
347
368
  env:
348
- DEVAUDIT_BASE_URL: ${{ vars.DEVAUDIT_BASE_URL }}
369
+ DEVAUDIT_BASE_URL_VAR: ${{ vars.DEVAUDIT_BASE_URL }}
349
370
  DEVAUDIT_API_KEY: ${{ secrets.DEVAUDIT_API_KEY }}
350
371
  steps:
351
372
  - uses: actions/checkout@v4
352
373
 
374
+ - name: Resolve DevAudit base URL
375
+ run: |
376
+ # Prefer sdlc-config.json devaudit.base_url over the deprecated repo
377
+ # Variable (DevAudit-Installer#156). When neither is set the upload
378
+ # step below no-ops via `if: env.DEVAUDIT_BASE_URL != ''`.
379
+ BASE=""
380
+ if [ -f sdlc-config.json ]; then
381
+ CONFIG_URL=$(jq -r '.devaudit.base_url // empty' sdlc-config.json 2>/dev/null || true)
382
+ [ -n "$CONFIG_URL" ] && BASE="$CONFIG_URL"
383
+ fi
384
+ if [ -n "$BASE" ]; then
385
+ echo "Using devaudit.base_url from sdlc-config.json: $BASE"
386
+ elif [ -n "$DEVAUDIT_BASE_URL_VAR" ]; then
387
+ BASE="$DEVAUDIT_BASE_URL_VAR"
388
+ echo "::warning::Using repo Variable DEVAUDIT_BASE_URL (deprecated in v1.23.0). Move base_url to sdlc-config.json devaudit.base_url for PR-visible config."
389
+ else
390
+ echo "::warning::No DevAudit base URL configured — skipping evidence upload. Set devaudit.base_url in sdlc-config.json."
391
+ fi
392
+ echo "DEVAUDIT_BASE_URL=${BASE%/}" >> "$GITHUB_ENV"
393
+
353
394
  - name: Download CI gate artifacts
354
395
  uses: actions/download-artifact@v4
355
396
  continue-on-error: true
@@ -352,15 +352,19 @@ jobs:
352
352
 
353
353
  mkdir -p {{WORKING_DIR_PREFIX}}ci-evidence
354
354
 
355
+ # Precise evidence_type=sast_report (matches the node template; the
356
+ # imprecise audit_log used pre-devaudit#370 made the portal's SAST and
357
+ # dependency panels show identical content — DevAudit-Installer#157).
355
358
  if [ -f {{WORKING_DIR_PREFIX}}ci-evidence/sast-results.json ]; then
356
359
  upload sast-results.json \
357
- {{PROJECT_SLUG}} _compliance-docs audit_log {{WORKING_DIR_PREFIX}}ci-evidence/sast-results.json \
360
+ {{PROJECT_SLUG}} _compliance-docs sast_report {{WORKING_DIR_PREFIX}}ci-evidence/sast-results.json \
358
361
  --category security_scan --gate-status "$STATUS_SAST" ${FLAGS}
359
362
  fi
360
363
 
364
+ # Precise evidence_type=dependency_audit (matches the node template).
361
365
  if [ -f {{WORKING_DIR_PREFIX}}ci-evidence/dependency-audit.json ]; then
362
366
  upload dependency-audit.json \
363
- {{PROJECT_SLUG}} _compliance-docs audit_log {{WORKING_DIR_PREFIX}}ci-evidence/dependency-audit.json \
367
+ {{PROJECT_SLUG}} _compliance-docs dependency_audit {{WORKING_DIR_PREFIX}}ci-evidence/dependency-audit.json \
364
368
  --category security_scan --gate-status "$STATUS_DEPAUDIT" ${FLAGS}
365
369
  fi
366
370
 
package/sdlc/article.md DELETED
@@ -1,219 +0,0 @@
1
- # Building a Compliance-Grade Software Development Pipeline for the Age of AI-Assisted Coding
2
-
3
- _How a small team built a tiered documentation system, automated security gates, and workflow-driven compliance pipeline that satisfies ISO 29119, ISO 27001, and emerging AI governance requirements — without drowning in paperwork._
4
-
5
- ---
6
-
7
- ## The Problem No One Has Solved Cleanly Yet
8
-
9
- Software compliance has always been about proving that what you shipped was built correctly, tested thoroughly, and reviewed by a human who understood what they were approving. The documentation burden is real but manageable: write a test plan, maintain a traceability matrix, keep your evidence, get someone to sign off.
10
-
11
- Then AI started writing the code.
12
-
13
- Not all of it. Not even most of it, in many teams. But enough that the fundamental assumptions behind compliance frameworks started to crack. When an AI generates a function, who is accountable for its correctness? When it suggests a dependency, how do you know the package actually exists? When you reprompt and get a different implementation of the same feature, how do you prove the second version is equivalent to the first?
14
-
15
- These aren't hypothetical concerns. Regulatory bodies are beginning to provide guidance, and the core message is blunt: you are responsible for the code, regardless of who or what wrote it. The EU AI Act is adding disclosure requirements for AI-built systems in high-risk categories. ISO 27001 auditors want to see that your SDLC explicitly addresses AI tooling. And the informal, iterative, low-documentation nature of how most developers use AI is the opposite of what compliance requires.
16
-
17
- The question isn't whether you can use AI and still be compliant. You can. The question is what the framework looks like that makes it work without turning every feature into a bureaucratic ordeal.
18
-
19
- This article describes one answer. It's not the only answer, but it's a complete, working system built from first principles and tested against real compliance requirements.
20
-
21
- ---
22
-
23
- ## What We Built
24
-
25
- Over a series of iterations, we developed a two-tier documentation system with a workflow-driven development pipeline that handles everything from requirement planning through production deployment, with AI governance built into every stage.
26
-
27
- The system consists of 12 documents across two tiers, a GitHub Actions CI pipeline for independent verification, and five operational workflows that a developer (or an AI assistant working alongside a developer) follows for every change.
28
-
29
- ### The Tiered Structure
30
-
31
- The first insight was that compliance documentation has two distinct layers that most teams accidentally merge into one: universal standards that apply regardless of what you're building, and project-specific details that change with every product.
32
-
33
- **Tier 1** contains four documents that never reference a specific project:
34
-
35
- The **Test Policy** defines why the organization tests, what it commits to, and who is accountable for what. This is where AI governance lives at the policy level — the permitted tools, the mandatory controls, the accountability chain. It states plainly that the human who commits AI-generated code is responsible for it, and that "the AI wrote it" is not an acceptable answer to an auditor.
36
-
37
- The **Test Strategy** defines how testing is approached methodically. Testing levels (unit through acceptance), security testing methodology (per-commit SAST and dependency scanning, per-release access control and audit log verification, periodic penetration testing), AI-assisted development testing protocol (risk classification, human review requirements, the regeneration retest rule), verification vs. validation, defect management, traceability, and evidence requirements. It references tools by category ("SAST tool") but never names a specific product.
38
-
39
- The **Test Architecture** defines what tests are built with. Specific tools and versions (Playwright, Jest, Semgrep, Snyk), directory structures, design patterns (Page Object Model, fixtures, factories), ESLint and Prettier configurations, CI pipeline stage specifications, artifact storage and retention policies, code style and naming conventions. This is the document an engineer opens when setting up test infrastructure.
40
-
41
- The **Periodic Security Review Schedule** defines the recurring security activities that happen on a calendar rather than per-feature: quarterly SAST reviews, dependency deep audits, access control reviews, and audit log integrity checks; annual penetration testing, disaster recovery testing, and third-party security assessments.
42
-
43
- The critical design decision was giving each document a single, non-overlapping responsibility. The Policy says "we will run SAST on every commit." The Strategy says "SAST is a mandatory gate that blocks merge on high/critical findings." The Architecture says "use Semgrep with auto config." No content appears in two places. When something changes, you update one document, not three.
44
-
45
- **Tier 2** contains the project-specific documents: a Test Plan with concrete environment details, test suite counts, and exit criteria; five operational workflow files; a project setup guide; and a deployment reference. When you start a new project, you copy Tier 2 and customize it. Tier 1 stays untouched.
46
-
47
- ---
48
-
49
- ## The Six Workflows
50
-
51
- The operational heart of the system is six documents that the developer follows sequentially. These aren't guidelines or reference material — they're executable procedures with specific commands, file paths, and decision points.
52
-
53
- ### Workflow 0: Project Setup (Run Once)
54
-
55
- Before any development work begins, this workflow configures the repository, branch protection, CI pipeline, compliance directories, and local tooling. The key output is the GitHub Actions workflow file that implements the independent verification gate.
56
-
57
- This solved a problem that emerged during development: the five per-feature workflows assume CI is already configured. Without a setup workflow, teams would either skip CI configuration (creating a compliance gap) or configure it inconsistently across projects. Making it workflow 0 — an explicit, documented, one-time step — ensures every project starts with the same infrastructure.
58
-
59
- The setup guide includes the complete GitHub Actions YAML with `# UPDATE` markers for project-specific values (Node version, source directory, database service, seed script, environment variables). A developer customizes the markers and has a working CI pipeline.
60
-
61
- ### Workflow 1: Plan Requirement
62
-
63
- Before writing any code, the developer defines the requirement in the RTM, classifies its risk level, and generates a test scope document.
64
-
65
- The risk classification is the decision that governs everything downstream. Low risk (internal tools, no regulated data) gets standard gates and minimal documentation. Medium risk (PII, user-facing features, API changes) adds targeted security testing and validation. High risk (authentication, payments, RBAC) requires full verification and validation, independent review, and penetration testing consideration.
66
-
67
- AI involvement raises risk by one level when the code touches Medium or High categories. This isn't punitive — it reflects the empirical reality that AI-generated code introduces injection vulnerabilities, insecure defaults, and hallucinated dependencies at higher rates than careful human development.
68
-
69
- The test scope document is generated proportionally to risk. For Low risk, it's a few lines: standard gates plus acceptance criteria. For Medium, it's half a page: standard gates plus specific testing items (which endpoints need access control verification, which actions need audit log testing), a validation approach, and acceptance criteria. For High, it's a full page with security testing detail, independent review planning, and business workflow validation.
70
-
71
- The critical compliance point: this document exists before implementation begins. It's evidence of planned testing, not retroactive documentation. An auditor can see that the testing approach was determined before the code was written.
72
-
73
- ### Workflow 2: Implement and Test
74
-
75
- The development loop: code, commit, run all gates, fix, repeat. Every commit on `develop` must leave all gates green.
76
-
77
- The mandatory gates are TypeScript compilation, SAST scanning (Semgrep), dependency auditing (`npm audit`), and the full E2E test suite. These run locally on the developer's machine during implementation, providing comprehensive evidence.
78
-
79
- For AI-assisted work, the developer logs significant prompts and outputs to the evidence directory (proportional to risk level), and flags any component regenerations. The regeneration protocol is one of the system's more nuanced rules: if you ask the AI to generate a component from scratch a second time (rather than incrementally editing the first output), that triggers a full retest of the component and its dependents. You can't assume two AI outputs for the same prompt are functionally equivalent.
80
-
81
- ### Workflow 3: Compile Evidence
82
-
83
- After implementation and testing, this workflow gathers all compliance artifacts: test results, SAST scan output, dependency audit results, security summary, AI use documentation, and the release ticket. It updates the RTM status and verifies that every item in the test scope (from workflow 1) has been addressed.
84
-
85
- The release ticket template includes sections for AI involvement (which tool, which files, who reviewed the AI code, any regenerations), dependency changes (with vulnerability status), and a reviewer checklist that goes beyond code quality into security and compliance verification.
86
-
87
- ### Workflow 4: Submit for Review
88
-
89
- Creating the PR triggers two things simultaneously: GitHub Actions runs the independent verification pipeline (TypeScript, SAST, dependency audit, unauthenticated E2E tests), and the human reviewer is notified.
90
-
91
- This is where the two-source evidence model comes together. The developer's local evidence (in the compliance directory) proves comprehensive testing was performed. The CI evidence (in GitHub Actions logs and artifacts) proves it independently — produced by GitHub's infrastructure, not by the developer. The human reviewer sees both before making their decision.
92
-
93
- The PR checklist explicitly includes test scope verification: the reviewer confirms the test scope document exists, the risk classification is appropriate, the testing depth matches the risk level, and all items in the scope were addressed. This catches the case where the AI under-classifies risk or under-specifies the test scope.
94
-
95
- Branch protection prevents merging until CI passes. This is enforced by GitHub, not by team discipline.
96
-
97
- ### Workflow 5: Deploy to Main
98
-
99
- After approval, the PR is merged (creating a merge commit that preserves the full audit trail), the hosting platform auto-deploys, and the developer performs post-deploy verification: health check, smoke test, and security verification (access control enforcement, security headers, no exposed stack traces).
100
-
101
- The compliance artifacts are finalized: the release ticket moves from `pending-releases/` to `approved-releases/`, the RTM is updated to `APPROVED - DEPLOYED`, and the audit trail records every step from requirement to production.
102
-
103
- ---
104
-
105
- ## The AI Governance Model
106
-
107
- The system's approach to AI governance is designed around a specific insight: AI-generated code doesn't need different testing — it needs more disciplined testing, applied consistently. The controls are:
108
-
109
- **Accountability is non-transferable.** The Test Policy states it plainly: the human who commits the code is responsible for it. The reviewer who approves it shares that responsibility. This isn't a legal novelty — it's the same principle that applies to code from any source. But making it explicit in the policy prevents the "the AI wrote it" excuse from entering the culture.
110
-
111
- **Human review is a formal compliance gate, not just good practice.** Every piece of AI-generated code must be reviewed by a qualified human before entering the test pipeline. "Qualified" means the reviewer understands the domain. The review is logged through the PR approval process — who reviewed it, when, and their decision.
112
-
113
- **Risk classification accounts for AI involvement.** AI-generated code touching security-sensitive areas raises the risk level by one tier. This increases the review depth, testing requirements, and documentation burden proportionally.
114
-
115
- **Regeneration triggers retest.** The non-determinism of AI output means you can't assume two generations of the same feature are equivalent. The system treats regeneration as a distinct event requiring full retesting, not an edit that follows standard gates.
116
-
117
- **Documentation scales with risk.** Low-risk AI-assisted work needs only a `Co-Authored-By` commit tag. Medium risk adds a summary in the evidence directory. High risk requires detailed prompt and output retention. This avoids the trap of requiring full AI transcripts for every trivial change while ensuring the audit trail is robust where it matters.
118
-
119
- **Permitted tools are explicit.** The Test Policy maintains a table of approved AI tools with their permitted uses and restrictions. Adding a new tool requires Engineering Leadership approval and policy updates. This prevents shadow AI usage from creating compliance blind spots.
120
-
121
- ---
122
-
123
- ## The Independent Verification Gate
124
-
125
- One of the system's most important features emerged from a practical question: if a single developer is writing, testing, and committing code, where is the independent verification?
126
-
127
- Running all tests locally means the developer produces all the evidence. An auditor has to trust that the tests ran against the actual code and the results weren't modified. That's a trust-based model.
128
-
129
- Adding CI changes it to an evidence-based model. GitHub Actions runs TypeScript compilation, SAST scanning, dependency auditing, and the unauthenticated E2E test subset automatically on every PR. These results are produced by GitHub's infrastructure and recorded in GitHub's logs. The developer can't modify them.
130
-
131
- The two evidence sources serve different purposes:
132
-
133
- **Local testing** (developer machine) is the comprehensive gate. All tests, all security scans, the full suite. This is where defects are found and fixed.
134
-
135
- **CI testing** (GitHub Actions) is the independent verification gate. A subset that provides tamper-resistant proof. This is what the auditor trusts.
136
-
137
- Branch protection ties them together: the PR cannot merge unless CI passes. The human reviewer sees both sources — local evidence in the compliance directory and CI evidence in the PR checks — and approves based on the combination.
138
-
139
- ---
140
-
141
- ## What an Auditor Sees
142
-
143
- When an auditor examines a change that went through this pipeline, they find a complete chain of evidence:
144
-
145
- The **RTM** shows the requirement was defined with a risk classification before implementation began. The **test scope** document, timestamped before the first implementation commit, proves testing was planned, not retroactive. The **commit history** shows incremental development with requirement references and `Co-Authored-By` tags for AI involvement. The **compliance evidence directory** contains SAST scan results (JSON), dependency audit results (JSON), E2E test results, a security summary, and AI use documentation.
146
-
147
- The **PR** shows the complete reviewer checklist, the CI check results (independent of the developer), the reviewer's approval with their identity and timestamp (recorded immutably by GitHub), and the compliance artifacts in the changed files. The **release ticket** in `approved-releases/` shows the full audit trail from requirement creation through deployment verification.
148
-
149
- For AI-assisted changes, the auditor additionally finds: which AI tool was used, which files it generated, who reviewed the AI output, whether any components were regenerated (and whether retesting was performed), and — for Medium/High risk — the prompts and outputs that produced the code.
150
-
151
- The chain is unbroken: requirement → planned test scope → implementation → comprehensive local testing → compiled evidence → independent CI verification → human review → deployment → post-deploy verification → finalization. Every link is documented and traceable.
152
-
153
- ---
154
-
155
- ## Compliance Framework Coverage
156
-
157
- The system maps to specific compliance framework requirements:
158
-
159
- **ISO/IEC/IEEE 29119-3** (Test Documentation): The tiered document structure directly implements the standard's artifact hierarchy. Test Policy, Test Strategy, Test Plan, Test Case Specifications (BDD feature files), Test Execution Logs (CI and local evidence), and Test Completion Reports (release tickets) are all present and correctly scoped.
160
-
161
- **ISO 27001** (Information Security): Change control and traceability through the PR-based approval gate. Vulnerability management through per-commit SAST and dependency scanning plus periodic penetration testing. Access control testing as a per-release and quarterly activity. Audit log verification on the same schedule. Evidence retention (minimum 3 years) documented and implemented through Git history plus artifact storage.
162
-
163
- **GDPR**: Test data privacy controls (no production PII in non-production environments, synthetic data generation). Data subject rights testing (access, rectification, erasure, portability, objection) included in test scope templates for relevant changes. Privacy by design testing built into the Medium/High risk test scope.
164
-
165
- **EU AI Act preparedness**: AI tool documentation, human oversight integrated into the development process, prompt and output retention providing the basis for conformity assessment if required. The Test Policy explicitly addresses high-risk classification and its implications.
166
-
167
- **SOC 2**: System monitoring (quarterly SAST + audit log review), logical access (quarterly access control review), change management (per-feature pipeline + quarterly reviews) map to CC7.1, CC6.1, and CC8.1 respectively.
168
-
169
- ---
170
-
171
- ## What Makes This Different
172
-
173
- Most compliance frameworks for AI-assisted development fall into one of two traps: they're either so lightweight they don't satisfy auditors, or so heavy they make AI assistance counterproductive.
174
-
175
- This system avoids both by making documentation proportional to risk. A Low-risk change (a UI tweak, a config update) goes through the same pipeline but produces minimal documentation — a commit tag, standard gate results, a brief PR. The overhead is negligible. A High-risk change (modifying authentication logic with AI assistance) produces a full test scope, detailed AI use documentation, independent security review, and comprehensive evidence. The overhead is significant but justified.
176
-
177
- The workflow-driven approach means compliance isn't a separate activity bolted onto development — it's the development process itself. You don't "do compliance" after you finish coding. The compliance artifacts are natural outputs of the workflow steps you're already following.
178
-
179
- The tiered document structure means you don't rewrite your compliance framework for every project. Tier 1 is written once and applies to everything. Tier 2 is customized per project from templates. Starting a new project means running the setup workflow, filling in placeholders, and executing the pipeline.
180
-
181
- And the independent verification gate (CI) solves the single-developer trust problem without requiring a full QA team. GitHub Actions provides tamper-resistant evidence that the code passed all gates, regardless of who wrote it or who committed it.
182
-
183
- ---
184
-
185
- ## The Honest Limitations
186
-
187
- This system doesn't solve everything.
188
-
189
- It depends on the developer (or AI) actually following the workflows. If someone skips workflow 1 and starts coding without a test scope, the reviewer should catch it at workflow 4 — but the control is detective, not preventive.
190
-
191
- The authenticated E2E tests (those requiring credentials) run only locally, not in CI. This means a subset of the test suite lacks independent verification. The mitigation is that the unauthenticated subset covers the majority of test cases, and the local evidence for the full suite is committed to the repository.
192
-
193
- Periodic activities (penetration testing, disaster recovery testing) are defined in the schedule but require manual execution. They're not triggered by the pipeline. Someone has to remember to do them on schedule.
194
-
195
- The AI regeneration protocol requires the developer to self-report when they regenerate a component. There's no automated detection of regeneration vs. incremental editing. The mitigation is the reviewer checklist, but this is a human control, not a technical one.
196
-
197
- And the system is built for a single-developer or small-team context. Larger teams would need additional controls: more granular RBAC on who can approve which risk levels, automated test scope validation, and possibly automated evidence integrity checking.
198
-
199
- ---
200
-
201
- ## Getting Started
202
-
203
- For teams wanting to implement this system:
204
-
205
- 1. **Adopt Tier 1 as-is** — The Test Policy, Test Strategy, Test Architecture, and Periodic Security Review Schedule are designed to be used without modification. They're organizational standards, not project-specific.
206
-
207
- 2. **Run the project setup workflow** — For each new project, execute `0-project-setup.md`. This creates the branch structure, CI pipeline, compliance directories, and project Test Plan from templates.
208
-
209
- 3. **Follow the pipeline** — Every feature goes through workflows 1-5. The discipline comes from the workflow structure, not from individual memory.
210
-
211
- 4. **Start with Low risk** — The first few requirements through the pipeline should be Low risk to validate that the process works end-to-end before the overhead of Medium/High risk documentation is introduced.
212
-
213
- 5. **Let the reviewer calibrate** — The test scope verification in the PR checklist is where risk classification mistakes get caught. The first few reviews will calibrate what "appropriate testing depth" means for your specific project.
214
-
215
- The complete template set — all 12 documents — is available and ready to be customized for any project using the placeholder markers in the Tier 2 templates.
216
-
217
- ---
218
-
219
- _This framework was developed iteratively through analysis of ISO 29119, ISO 27001, GDPR, SOC 2, and emerging AI governance guidance, then implemented as a practical workflow system designed for AI-assisted software development in a compliance context._