@metasession.co/devaudit-cli 0.1.1 → 0.1.3

package/sdlc/files/_common/Test_Plan_TEMPLATE.md

@@ -0,0 +1,311 @@
+# Test Plan — [PROJECT NAME]
+
+**Document Type:** Test Plan (Project-Specific) | **Version:** 1.0 | **Effective Date:** [DATE]
+
+**Project:** [PROJECT NAME] | **Repository:** `[org/repo-name]`
+
+**Parent Documents:** Test Policy, Test Strategy, Test Architecture (all Tier 1, in devaudit/sdlc/files/)
+
+---
+
+## Purpose
+
+This Test Plan defines project-specific testing details for [PROJECT NAME]. It implements the universal standards from Tier 1 with concrete environment details, test suites, entry/exit criteria, and compliance artifacts.
+
+For testing philosophy and governance: Test Policy (`sdlc/files/Test_Policy.md` in DevAudit).
+For testing methodology and approach: Test Strategy.
+For tools, patterns, and code standards: Test Architecture.
+
+---
+
+## Project Overview
+
+| Attribute | Value |
+|---|---|
+| Application | [PROJECT NAME — brief description] |
+| Stack | [e.g., TypeScript, Next.js, MongoDB, Socket.IO] |
+| Hosting | [e.g., Railway, Vercel, AWS] (auto-deploy from `main`) |
+| Production URL | [URL] |
+| Health Endpoint | [e.g., /api/health] |
+| Database | [e.g., MongoDB — connection details] |
+| Runtime | [e.g., node_modules/.bin/tsx server.ts] |
+| Build | [e.g., Multi-stage Dockerfile, 3-5 min] |
+
+---
+
+## Branching Strategy
+
+Trunk-based with develop branch (per Test Strategy branching patterns):
+
+- **`main`** — Production. Auto-deploys to hosting platform. PR approval + all checks required.
+- **`develop`** — Working branch. All implementation here. Permanent, never deleted.
+- Merge commits (not squash) for `develop` → `main` to preserve audit history.
+
+---
+
+## Test Suites
+
+### E2E Tests (Playwright)
+
+| Attribute | Value |
+|---|---|
+| Framework | Playwright (per Test Architecture standard) |
+| Test count | [TOTAL] tests across [N] projects |
+| Unauthenticated (CI) | [COUNT] (run in CI on PR to main) |
+| Authenticated (local only) | [COUNT] (require credentials) |
+| Browser | Chromium |
+| Prerequisite | [e.g., npx tsx scripts/seed-e2e-admins.ts] |
+
+### Unit Tests
+
+| Attribute | Value |
+|---|---|
+| Framework | [Jest / Vitest] |
+| Coverage target | 70% for critical modules |
+| Run command | [e.g., npx vitest run] |
+
+### TypeScript Compilation
+
+```bash
+npx tsc --noEmit   # 0 errors required
+```
+
+### SAST Scanning
+
+| Attribute | Value |
+|---|---|
+| Tool | Semgrep (per Test Architecture) |
+| Config | auto |
+| Scan scope | [e.g., src/] |
+
+```bash
+npx semgrep scan --config auto [SOURCE_DIR]/ --severity ERROR --severity WARNING
+```
+
+### Dependency Auditing
+
+```bash
+npm audit --audit-level=high
+```
+
+---
+
+## Entry and Exit Criteria
+
+### Entry
+
+- On `develop`, up to date with remote
+- Dev server starts
+- Database running locally
+- Playwright browsers installed
+- Test data seeded
+
+### Exit (Before PR)
+
+| Gate | Local | CI (PR) | Threshold |
+|---|---|---|---|
+| TypeScript | Yes | Yes | 0 errors |
+| SAST (high/critical) | Yes | Yes | 0 findings |
+| Dependencies (high/critical) | Yes | Yes | 0 vulnerabilities |
+| E2E tests | [TOTAL]/[TOTAL] | [UNAUTH]/[UNAUTH] | All pass |
+| Severity-1 defects | — | — | 0 open |
+| Human review | — | PR approved | Approved |
+
+Additional for Medium/High risk (per Test Strategy risk matrix):
+
+| Gate | Threshold |
+|---|---|
+| Access control tests | RBAC endpoints return correct 401/403 |
+| Audit log tests | Auditable actions produce log entries |
+
+---
+
+## CI/CD
+
+### Pipeline Configuration
+
+| Trigger | What Runs | Independent Evidence |
+|---|---|---|
+| Push to `develop` | TypeScript check + build | Compilation clean |
+| PR to `main` | TypeScript + SAST + dependency audit + E2E (unauthenticated) | All gates independently verified by GitHub |
+| Merge to `main` | Auto-deploy to hosting platform | Deployment triggered |
+
+CI workflow file: `.github/workflows/ci.yml` (created during project setup — see `0-project-setup.md`)
+
+CI prerequisites: [e.g., E2E tests require seeding even for unauthenticated subset]
+
+### PR Pipeline Execution Order
+
+1. TypeScript check — fast, no database
+2. SAST scan — fast, no database
+3. Dependency audit — fast, no database
+4. Seed test data — required before E2E
+5. E2E tests — slow, requires database
+
+Steps 1-3 can run in parallel. Step 4 must complete before step 5.
+
+### Evidence Model
+
+**Local evidence** (in `compliance/evidence/REQ-XXX/`) — comprehensive, developer-produced, covers all tests. Committed to repository.
+
+**CI evidence** (in GitHub Actions logs + artifacts) — independent, GitHub-produced, covers unauthenticated subset. Tamper-resistant.
+
+Both required. Local proves comprehensive testing. CI proves it independently.
+
+---
+
+## AI Use — Project Configuration
+
+Per Test Policy AI governance:
+
+| Tool | Permitted Use |
+|---|---|
+| [e.g., Claude (Anthropic) — Opus 4.6, Sonnet 4.6] | [e.g., Code generation, test generation, documentation, review] |
+
+Documentation requirements (per Test Strategy AI methodology):
+
+| Risk | Commit | Evidence | Prompts |
+|---|---|---|---|
+| Low | `Co-Authored-By` tag | Not required | Not required |
+| Medium | Same | Summary in evidence dir | Summary |
+| High | Same | Detailed record | Detailed |
+
+Elevated review required for: [list security-sensitive code categories for this project, e.g., authentication, payment processing, user data/PII, API security, database schema changes]
+
+---
+
+## Requirements Traceability
+
+**Format:** `compliance/RTM.md`, Part B:
+
+```markdown
+| REQ-XXX | #NNN | Risk | Evidence | Status | Approver | Date |
+```
+
+**Status lifecycle:** DRAFT → IN PROGRESS → TESTED - PENDING SIGN-OFF → APPROVED - DEPLOYED
+
+**JSDoc headers:** `@requirement REQ-XXX - Brief description` in modified source files.
+
+---
+
+## Compliance Artifacts
+
+```
+compliance/
+├── RTM.md
+├── test-plan.md                    # This document
+├── test-cases.md
+├── test-summary-report.md
+├── pending-releases/
+│   └── RELEASE-TICKET-REQ-XXX.md
+├── approved-releases/
+│   └── RELEASE-TICKET-REQ-XXX.md
+└── evidence/
+    ├── REQ-XXX/
+    │   ├── test-scope.md           # Test plan for this requirement (PLAN stage)
+    │   ├── e2e-results.json
+    │   ├── sast-results.json
+    │   ├── dependency-audit.json
+    │   ├── security-summary.md
+    │   ├── ai-use-note.md
+    │   └── ai-prompts.md
+    └── periodic/
+        ├── sast-quarterly/
+        ├── dependency-audit/
+        ├── access-control/
+        ├── audit-log/
+        ├── pentest/
+        ├── dr-test/
+        └── third-party/
+```
+
+### Per-Requirement Test Scope
+
+Every tracked requirement gets a `test-scope.md` created during the PLAN stage **before implementation**. Scope scales with risk:
+
+| Risk | Content |
+|---|---|
+| Low | Standard gates + acceptance criteria. A few lines. |
+| Medium | Above + targeted testing (access control, audit logging, dependency review), validation approach. Half a page. |
+| High | Above + security testing detail, independent review plan, pen test consideration, business validation, AI detail. One page. |
+
+Templates in workflow 1 (`1-plan-requirement.md`).
+
+---
+
+## Post-Deploy Verification
+
+```bash
+# Health check
+curl -s [PRODUCTION_URL]/[HEALTH_ENDPOINT]
+
+# Smoke test
+curl -s [PRODUCTION_URL]/[PUBLIC_ENDPOINT] | head -c 200
+curl -s -o /dev/null -w "%{http_code}" [PRODUCTION_URL]/
+
+# Security verification (per Test Strategy post-deploy requirements)
+curl -s -o /dev/null -w "%{http_code}" [PRODUCTION_URL]/[ADMIN_ENDPOINT]
+# Expected: 401 or 403
+
+curl -s -I [PRODUCTION_URL]/ | grep -iE 'x-frame-options|strict-transport'
+
+curl -s [PRODUCTION_URL]/[NONEXISTENT_ENDPOINT]
+# Expected: generic error, NOT stack trace
+```
+
+---
+
+## Disaster Recovery
+
+| Metric | Target |
+|---|---|
+| RTO | [e.g., 4 hours] |
+| RPO | [e.g., 24 hours] |
+
+Recovery procedures:
+1. **Application failure:** [e.g., Redeploy previous version from hosting dashboard]
+2. **Database recovery:** [e.g., Restore from backup]
+3. **Full rebuild:** [e.g., Deploy from main to fresh environment]
+
+---
+
+## Periodic Review — Project Procedures
+
+Per the Periodic Security Review Schedule (Tier 1):
+
+```bash
+# Quarterly SAST
+semgrep scan --config auto --config p/security-audit --config p/owasp-top-ten [SOURCE_DIR]/ --json > compliance/evidence/periodic/sast-quarterly/sast-full-$(date -I).json
+
+# Quarterly dependencies
+npm audit --json > compliance/evidence/periodic/dependency-audit/npm-audit-$(date -I).json
+npm outdated --json > compliance/evidence/periodic/dependency-audit/outdated-$(date -I).json
+
+# Quarterly access control
+npx playwright test --grep "access control\|unauthorized\|forbidden\|RBAC"
+```
+
+Annual pen test scope: [PRODUCTION_URL], API endpoints, auth mechanism, [database-specific injection testing], [other scope items]
+
+---
+
+## Workflow Files
+
+| # | File | Purpose |
+|---|---|---|
+| 0 | `0-project-setup.md` | One-time: repository, CI, compliance setup |
+| 1 | `1-plan-requirement.md` | Create REQ, classify risk, generate test scope |
+| 2 | `2-implement-and-test.md` | Code, commit, run all local gates |
+| 3 | `3-compile-evidence.md` | Gather test + security + AI evidence |
+| 4 | `4-submit-for-review.md` | Create PR (triggers CI independent verification) |
+| 5 | `5-deploy-main.md` | Merge, deploy, verify, finalize |
+
+---
+
+## Document Control
+
+| Version | Date | Author | Changes |
+|---|---|---|---|
+| 1.0 | [DATE] | [AUTHOR] | Initial plan |
+
+**Parent Documents:** Test Policy, Test Strategy, Test Architecture, Periodic Security Review Schedule (in devaudit/sdlc/files/)

package/sdlc/files/_common/Test_Policy.md

@@ -0,0 +1,277 @@
+# Test Policy
+
+**Document Type:** Policy | **Version:** 3.0 | **Effective Date:** March 2026 | **Review Cycle:** Annual
+
+**Owner:** Engineering Leadership | **Approved By:** Executive Team
+
+---
+
+## Purpose
+
+This Test Policy establishes Metasession's organizational commitment to quality assurance and testing excellence. It defines our philosophy, principles, accountability standards, and governance for all testing activities — including AI-assisted development.
+
+This policy answers **"why we test"** and **"what we commit to."** For how we approach testing methodically, see the Test Strategy (`sdlc/files/Test_Strategy.md` in DevAudit). For what tools and patterns we use, see the Test Architecture.
+
+---
+
+## Scope
+
+This policy applies to:
+
+- All Metasession products and client service delivery engagements
+- Internal systems and infrastructure
+- All team members involved in software development, quality assurance, and product delivery
+- All code regardless of authorship — human-written, AI-generated, or AI-assisted
+
+---
+
+## Testing Philosophy
+
+**Core Principle:** We test to ensure zero critical defects reach production and to maintain the trust of our customers, partners, and end users.
+
+Metasession believes that:
+
+- **Quality is non-negotiable** — Testing is not optional; it is an integral part of our definition of done
+- **Prevention over detection** — We invest in early testing, automation, and continuous quality practices to prevent defects rather than finding them late
+- **Risk-based approach** — Testing resources are allocated based on business risk, compliance requirements, and customer impact
+- **Continuous improvement** — We regularly review and enhance our testing practices based on metrics, retrospectives, and industry best practices
+- **Compliance-first mindset** — All testing activities support our ISO 27001, GDPR, and other regulatory compliance objectives
+- **Accountability regardless of authorship** — The human who commits code is responsible for it, whether they wrote it manually, generated it with AI, or received it from a third party
+
+---
+
+## Strategic Alignment
+
+This policy directly supports Metasession's strategic plan:
+
+**Operational Excellence & Compliance** — Testing provides documented evidence for ISO 27001 certification audits, demonstrates adherence to secure SDLC practices, and produces automated audit trails.
+
+**Service Market Development** — Our testing expertise differentiates Metasession as a premium QA partner and builds credibility with clients in regulated industries.
+
+**Product Portfolio Growth** — Consistent testing standards ensure quality across all products while automation enables rapid iteration without sacrificing reliability.
+
+---
+
+## Organizational Commitments
+
+### What Every Release Must Include
+
+1. **Test planning** — Documented scope, resources, entry/exit criteria, and risks
+2. **Test design** — Test cases with traceability to requirements
+3. **Test execution** — Recorded evidence with timestamps, results, and responsible parties
+4. **Security scanning** — Static analysis and dependency auditing on every change
+5. **Defect management** — All defects tracked with severity and closure verification
+6. **Test reporting** — Completion reports with pass/fail metrics and release recommendations
+
+### Compliance Framework
+
+Metasession's testing practices conform to:
+
+- **ISO/IEC 29119-3** — Software testing documentation
+- **ISO 27001** — Information security management
+- **GDPR** — Data protection and privacy
+- Industry best practices for agile testing and DevOps
+
+### Periodic Security Commitments
+
+Beyond per-change scanning, Metasession commits to:
+
+- **Quarterly:** Full codebase security review, dependency deep audit, access control review, audit log integrity check
+- **Annually:** Penetration testing by qualified third party, disaster recovery testing, third-party security assessment, AI use policy review
+
+Specific schedules and procedures are defined in the Periodic Security Review Schedule.
+
+---
+
+## AI-Assisted Development Governance
+
+### Policy Statement
+
+Metasession embraces AI as a development tool while recognizing that AI-generated code introduces specific compliance risks requiring explicit controls. AI assistance does not reduce testing requirements — it increases them.
+
+### The Risk Profile
+
+AI-generated code presents risks distinct from human-authored code:
+
+- **Confident incorrectness** — Plausible-looking code with subtle logic errors or security flaws
+- **Hallucinated dependencies** — Fabricated, outdated, or vulnerable packages
+- **Non-determinism** — The same prompt may produce different code on different runs
+- **Training data contamination** — Reproduction of insecure patterns
+- **No inherent audit trail** — No default record of what was asked or generated
+
+### Mandatory Controls
+
+1. **Human review as a formal compliance gate** — Every AI-generated piece of code must be reviewed by a qualified human before entering the test pipeline. The review must be logged with reviewer identity, date, and scope.
+
+2. **Automated security scanning on every commit** — AI-generated code must pass the same SAST and dependency scanning gates as human-authored code. These are mandatory, not optional.
+
+3. **Dependency and supply chain verification** — All dependencies in AI-assisted changes must be verified as real, current, and vulnerability-free.
+
+4. **Regeneration triggers full retest** — If a developer regenerates a component from scratch (not incremental editing), that triggers full retest of the component and dependents. Functional equivalence cannot be assumed.
+
+5. **Documentation proportional to risk** — Each project's Test Plan defines AI tool permissions, review requirements, and prompt/output retention based on risk level.
+
+### Permitted AI Tools
+
+| Tool | Permitted Use | Restrictions |
+|---|---|---|
+| Claude (Anthropic) | Code generation, test generation, documentation, review assistance | No deployment without human review |
+| GitHub Copilot | Inline code suggestions | Same review requirements as any AI-generated code |
+
+Adding a new tool requires Engineering Leadership approval and updates to this policy, the Test Strategy, and relevant project Test Plans.
+
+### What AI May and May Not Generate
+
+**Permitted (with mandatory human review):** Application logic, UI components, test code, database queries, migration scripts, documentation, configuration, utility functions.
+
+**Requires elevated review (senior developer + security):** Authentication/authorization logic, cryptographic operations, payment processing, data validation for user input, API security middleware, database schema changes affecting PII.
+
+**Prohibited (must be human-authored):** Security credentials or secrets, production environment configuration values, compliance policy documents.
+
+### Accountability
+
+The human who commits AI-generated code is accountable for its correctness, security, and compliance. The PR reviewer who approves it shares accountability for having verified its quality. "The AI wrote it" is not an acceptable explanation.
+
+### EU AI Act Preparedness
+
+Metasession monitors evolving AI regulation. For projects under high-risk classification (healthcare, employment, critical infrastructure), using AI to build the system may require disclosure and conformity assessment. Project Test Plans must address this where applicable.
+
+---
+
+## Risk-Based Testing
+
+Testing effort is prioritized by risk level, determined at planning time:
+
+**High Priority** — Sensitive data handling, security-critical functionality, regulatory compliance features, core revenue capabilities, production infrastructure changes, AI-generated code in any of these categories.
+
+**Medium Priority** — New feature development, significant refactoring, third-party integrations, performance optimizations, AI-generated code for non-security features.
+
+**Low Priority** — Minor UI updates, configuration changes, documentation, internal tools with limited impact.
+
+AI involvement in Medium or High categories raises risk by one level. The Test Strategy defines specific testing depth requirements per level.
+
+---
+
+## Roles & Responsibilities
+
+### Engineering Leadership
+- Approve and maintain this policy
+- Allocate testing resources
+- Review metrics and drive improvement
+- Approve AI tool additions
+
+### QA Team / Test Engineers
+- Design and execute strategies and plans
+- Develop and maintain automated suites
+- Report defects and verify fixes
+- Generate reports and maintain evidence repositories
+- Verify security scan results
+
+### Developers
+- Write unit tests for all changes
+- Execute local testing before committing (including security scans)
+- Fix identified defects
+- Review and take accountability for all AI-generated code committed
+- Document AI use per project requirements
+
+### Product Managers / Business Analysts
+- Define clear acceptance criteria
+- Participate in test planning and risk assessment
+- Review and approve completion reports
+- Sign off on release readiness
+
+---
+
+## Metrics & Continuous Improvement
+
+Metasession tracks:
+
+- **Test coverage** — Percentage of requirements with associated tests
+- **Automation rate** — Automated vs. manual tests
+- **Defect detection rate** — Defects found per testing phase
+- **Defect escape rate** — Production defects not caught during testing
+- **Test execution rate** — Planned tests executed
+- **Pass/fail trends** — Historical results over time
+- **Security findings** — SAST and dependency findings per release
+- **AI code review rate** — AI-generated code formally reviewed before merge
+
+Quarterly reviews assess trends and identify improvements. Findings feed into retrospectives and annual planning.
+
+---
+
+## Compliance & Audit Support
+
+All test artifacts must be:
+
+- **Traceable** — Linked to requirements, user stories, or risk assessments
+- **Timestamped** — With date, time, and responsible party
+- **Retained** — Minimum 3 years for ISO audits and compliance reviews
+- **Accessible** — Retrievable within 24 hours for audit requests
+
+AI use documentation is retained alongside other evidence for the same period.
+
+---
+
+## Policy Exceptions
+
+Exceptions require:
+
+1. Written justification with risk assessment
+2. Documented risk acceptance by Engineering Leadership
+3. Compensating controls to mitigate risk
+4. Time-bound with defined remediation date
+
+All exceptions logged and reviewed quarterly. Exceptions to AI governance controls are not permitted for High risk changes.
+
+---
+
+## Training & Awareness
+
+All team members receive:
+
+- **Onboarding** — Testing standards and tools
+- **Annual refresher** — Policy changes, new tools, compliance updates
+- **Role-specific** — Specialized training for QA, developers, PMs
+- **Compliance** — ISO 27001, GDPR, secure testing practices
+- **AI-assisted development** — Responsible use, review requirements, recognizing AI failure modes
+
+Training completion is tracked as ISO 27001 evidence.
+
+---
+
+## Document Hierarchy
+
+```
+Test Policy (this document)
+  → WHY we test, WHAT we commit to, WHO is responsible
+  
+Test Strategy
+  → HOW we approach testing methodically
+  
+Test Architecture
+  → WHAT we build tests with, HOW we structure the code
+  
+Periodic Security Review Schedule
+  → WHEN periodic security activities happen
+  
+Project Test Plans (per product)
+  → WHERE and WHEN for specific products
+```
+
+---
+
+## Document Control
+
+| Version | Date | Author | Changes |
+|---|---|---|---|
+| 1.0 | January 2026 | Engineering Leadership | Initial creation |
+| 2.0 | March 2026 | Engineering Leadership | Added AI governance, security commitments |
+| 3.0 | March 2026 | Engineering Leadership | Clean boundary split — removed content now owned by Test Strategy (methodology) and Test Architecture (tooling) |
+
+**Next Review Date:** March 2027
+
+**Related Documents:** Test Strategy, Test Architecture, Periodic Security Review Schedule, Project Test Plans (in devaudit/sdlc/files/)
+
+---
+
+**Policy Status:** Approved | **Effective Date:** March 2026