chati-dev 1.4.0 → 2.0.1

package/framework/tasks/qa-impl-test-execute.md

@@ -0,0 +1,344 @@
+---
+id: qa-impl-test-execute
+agent: qa-implementation
+trigger: dev
+phase: build
+requires_input: false
+parallelizable: false
+outputs: [test-results.yaml]
+handoff_to: qa-impl-sast-scan
+autonomous_gate: false
+criteria:
+  - All unit tests executed
+  - All integration tests executed
+  - Test results collected and analyzed
+  - Failures categorized by severity
+---
+
+# Test Suite Execution
+
+## Purpose
+Execute the complete test suite (unit and integration tests) for the implemented code, collect results, analyze failures, and provide actionable feedback for remediation.
+
+## Prerequisites
+- Implementation complete (dev agent marked tasks as done)
+- Test files exist in test/ or **/*.test.js
+- Test framework configured (package.json scripts)
+- Coverage tool configured
+- qa-plan.yaml with coverage targets
+
+## Steps
+
+1. **Verify Test Environment**
+   - Check that test framework is installed (npm list vitest / jest / mocha)
+   - Verify test scripts exist in package.json (test, test:unit, test:integration)
+   - Ensure coverage tool is configured (c8, nyc, or built-in)
+   - Validate test environment variables if required
+
+2. **Discover Test Files**
+   - Scan for unit test files: **/*.test.js, **/*.spec.js in unit/
+   - Scan for integration test files: **/*.test.js in integration/
+   - List all discovered test files with count
+   - Verify test files match implemented modules (coverage-plan.yaml reference)
+
+3. **Execute Unit Tests**
+   - Run: `npm run test:unit` or equivalent
+   - Capture exit code (0 = pass, non-zero = fail)
+   - Parse test output for results (passed, failed, skipped)
+   - Collect execution time and memory usage
+   - Save raw output to test-results/unit-output.log
+
+4. **Execute Integration Tests**
+   - Run: `npm run test:integration` or equivalent
+   - Capture exit code
+   - Parse test output for results
+   - Collect execution time (integration tests typically slower)
+   - Save raw output to test-results/integration-output.log
+
+5. **Generate Coverage Report**
+   - Run: `npm run test:coverage` or `c8 npm test`
+   - Parse coverage output for percentages (line, branch, function, statement)
+   - Generate HTML coverage report for detailed inspection
+   - Compare coverage to qa-plan.yaml targets (overall, per-module)
+   - Identify modules below target coverage
+
+6. **Analyze Test Failures**
+   - For each failed test:
+     - Extract test name, file, line number
+     - Capture error message and stack trace
+     - Categorize failure type (assertion, exception, timeout, setup error)
+     - Determine affected module/component
+   - Group failures by module for reporting
+
+7. **Categorize by Severity**
+   - **CRITICAL**: Failures in critical-risk modules (state management, mode governance, file operations)
+   - **HIGH**: Failures in high-risk modules (agent logic, orchestrator, configuration)
+   - **MEDIUM**: Failures in medium-risk modules (CLI, i18n, error handling)
+   - **LOW**: Failures in low-risk modules (formatting, documentation)
+   - Reference risk-matrix.yaml for severity mapping
+
+8. **Check for Test Quality Issues**
+   - **Flaky tests**: Tests that fail intermittently (run failed tests 3x to verify)
+   - **Skipped tests**: Tests marked as .skip or .todo (should be minimal)
+   - **Long-running tests**: Tests exceeding 5 seconds (may indicate inefficiency)
+   - **No assertions**: Tests that pass but have no assertions (false positives)
+
+9. **Calculate Test Metrics**
+   - **Pass rate**: (passed / total) * 100
+   - **Coverage delta**: actual coverage - target coverage
+   - **Failure density**: failures per module
+   - **Test execution time**: total time for full suite
+   - **Flakiness rate**: flaky tests / total tests
+
+10. **Generate Recommendations**
+    - For coverage gaps: Suggest specific areas needing tests
+    - For failures: Provide first-action recommendations (fix code vs fix test)
+    - For flaky tests: Recommend stabilization strategies
+    - For long-running tests: Suggest optimization approaches
+
+11. **Compile Test Results Document**
+    - Summarize pass/fail status
+    - Include coverage report
+    - List failures with severity and recommendations
+    - Document test quality issues
+    - Provide overall assessment (PASS, FAIL, CONDITIONAL)
+
+12. **Log Results and Next Steps**
+    - Save test-results.yaml to session
+    - Update session.yaml with test execution status
+    - If all tests pass and coverage meets targets: proceed to SAST scan
+    - If failures exist: flag for dev agent remediation, then re-run
+
+## Decision Points
+
+- **Coverage Below Target but Tests Pass**: If coverage is 2-5% below target but all tests pass, decide whether to proceed with CONDITIONAL status or block for more tests. Recommend: CONDITIONAL if critical modules meet targets, FAIL if critical modules below target.
+
+- **Flaky Test Handling**: If a test fails once but passes on retry, mark as flaky rather than failed. However, if >10% of tests are flaky, treat as a test quality issue and FAIL the build.
+
+- **Integration Test Failures Due to Environment**: If integration tests fail due to missing external dependencies (e.g., filesystem permissions, network), distinguish from code failures. Provide remediation steps for environment setup.
+
+## Error Handling
+
+**Test Script Not Found**
+- If package.json lacks test scripts, check for test framework binary (node_modules/.bin/vitest)
+- Attempt to run directly: `npx vitest run`
+- If no test framework found, FAIL with error: "Test framework not installed"
+- Escalate to dev agent for test setup
+
+**Test Execution Timeout**
+- If tests hang (no output for 5 minutes), kill process
+- Log timeout error with test context (last test executed)
+- Investigate long-running tests or infinite loops
+- Recommend: Add --testTimeout flag or investigate hanging code
+
+**Coverage Tool Failure**
+- If coverage generation fails, proceed with test results only
+- Log coverage tool error
+- Mark coverage as UNKNOWN in test-results.yaml
+- Recommend: Fix coverage configuration, re-run
+
+**Parse Failure (Unrecognized Output Format)**
+- If test output format is unrecognized, save raw output
+- Attempt to extract pass/fail from exit code only
+- Log parse warning
+- Recommend: Use standard test reporter (JSON, TAP)
+
+## Output Format
+
+```yaml
+# test-results.yaml
+version: 1.0.0
+created: YYYY-MM-DD
+agent: qa-implementation
+phase: build
+
+summary:
+  status: FAIL # PASS, FAIL, CONDITIONAL
+  total_tests: 247
+  passed: 235
+  failed: 12
+  skipped: 0
+  pass_rate: 95.1%
+  execution_time: 42.3s
+
+unit_tests:
+  total: 198
+  passed: 191
+  failed: 7
+  skipped: 0
+  pass_rate: 96.5%
+  execution_time: 18.7s
+  output_log: test-results/unit-output.log
+
+integration_tests:
+  total: 49
+  passed: 44
+  failed: 5
+  skipped: 0
+  pass_rate: 89.8%
+  execution_time: 23.6s
+  output_log: test-results/integration-output.log
+
+coverage:
+  overall: 76.3%
+  target: 75%
+  delta: +1.3%
+  status: PASS
+
+  by_metric:
+    line: 76.3%
+    branch: 72.1%
+    function: 81.2%
+    statement: 76.5%
+
+  by_module:
+    orchestrator:
+      actual: 87%
+      target: 85%
+      status: PASS
+
+    state_management:
+      actual: 91%
+      target: 90%
+      status: PASS
+
+    agents:
+      actual: 78%
+      target: 75%
+      status: PASS
+
+    file_operations:
+      actual: 73%
+      target: 80%
+      status: FAIL
+      gap: -7%
+
+    validators:
+      actual: 92%
+      target: 90%
+      status: PASS
+
+    cli:
+      actual: 68%
+      target: 70%
+      status: FAIL
+      gap: -2%
+
+  html_report: coverage/index.html
+
+failures:
+  critical:
+    - test: "state_management: concurrent write protection"
+      file: test/unit/state-manager.test.js
+      line: 142
+      module: state_management
+      error: "Expected lock to be acquired, but was not"
+      stack_trace: |
+        AssertionError: Expected lock to be acquired
+            at Context.<anonymous> (test/unit/state-manager.test.js:142:12)
+      recommendation: "Fix lock acquisition logic in state/session-manager.js"
+      severity: CRITICAL
+      risk_level: critical
+
+  high:
+    - test: "file_operations: permission error handling"
+      file: test/integration/file-ops.test.js
+      line: 87
+      module: file_operations
+      error: "Expected error to be thrown, but function succeeded"
+      stack_trace: |
+        AssertionError: Expected error to be thrown
+            at Context.<anonymous> (test/integration/file-ops.test.js:87:15)
+      recommendation: "Add permission check in utils/file-ops.js before write"
+      severity: HIGH
+      risk_level: high
+
+  medium:
+    - test: "cli: invalid argument handling"
+      file: test/unit/cli-parser.test.js
+      line: 203
+      module: cli
+      error: "TypeError: Cannot read property 'value' of undefined"
+      stack_trace: |
+        TypeError: Cannot read property 'value' of undefined
+            at parseArgs (cli/index.js:45:18)
+      recommendation: "Add null check for optional arguments"
+      severity: MEDIUM
+      risk_level: medium
+
+  low: []
+
+test_quality_issues:
+  flaky_tests:
+    count: 2
+    rate: 0.8%
+    tests:
+      - name: "agent: task execution with timeout"
+        file: test/integration/agent-executor.test.js
+        pass_rate: 2/3
+        recommendation: "Increase timeout or mock async operations"
+
+  skipped_tests:
+    count: 0
+    rate: 0%
+
+  long_running_tests:
+    count: 5
+    threshold: 5s
+    tests:
+      - name: "upgrade: full migration end-to-end"
+        file: test/e2e/upgrade.test.js
+        duration: 12.3s
+        recommendation: "Acceptable for e2e test, consider splitting if grows >20s"
+
+  no_assertions:
+    count: 0
+
+metrics:
+  pass_rate: 95.1%
+  coverage_delta: +1.3%
+  failure_density:
+    state_management: 1 failure / 45 tests (2.2%)
+    file_operations: 1 failure / 38 tests (2.6%)
+    cli: 1 failure / 42 tests (2.4%)
+  execution_time: 42.3s
+  flakiness_rate: 0.8%
+
+assessment:
+  status: FAIL
+  rationale: |
+    - 12 test failures across critical, high, and medium severity
+    - Coverage for file_operations below target (-7%)
+    - Critical failure in state management (concurrent write protection)
+
+  blocking_issues:
+    - CRITICAL: state_management concurrent write protection failure
+    - HIGH: file_operations permission error handling failure
+    - Coverage gap in file_operations module (-7%)
+
+  conditional_pass_criteria:
+    - Fix all CRITICAL and HIGH severity failures
+    - Increase file_operations coverage to 80% (add 7% more tests)
+    - Re-run test suite and verify PASS
+
+  recommendations:
+    - Priority 1: Fix state_management lock acquisition logic
+    - Priority 2: Add permission check in file_operations
+    - Priority 3: Add 10-12 tests for file_operations (edge cases, error handling)
+    - Priority 4: Fix medium severity CLI parsing issue
+    - Address flaky agent test (increase timeout or mock)
+
+next_steps:
+  - Return to dev agent for remediation of blocking issues
+  - Re-run qa-impl-test-execute after fixes
+  - If rerun passes, proceed to qa-impl-sast-scan
+  - Do NOT proceed to SAST until all tests pass and coverage targets met
+
+handoff:
+  to: dev
+  reason: Test failures and coverage gap
+  priority_fixes:
+    - state_management: concurrent write protection
+    - file_operations: permission error handling
+    - file_operations: add 10-12 tests for 7% coverage increase
+```

package/framework/tasks/qa-impl-verdict.md

@@ -0,0 +1,339 @@
+---
+id: qa-impl-verdict
+agent: qa-implementation
+trigger: qa-impl-performance-test
+phase: validate
+requires_input: false
+parallelizable: false
+outputs: [qa-verdict.yaml]
+handoff_to: qa-impl-consolidate
+autonomous_gate: true
+criteria:
+  - All tests passing
+  - No critical SAST findings
+  - No regressions
+  - Performance within thresholds
+  - Final verdict issued (PASS/FAIL/CONDITIONAL)
+---
+
+# QA Implementation Verdict
+
+## Purpose
+Issue the final QA verdict by synthesizing all QA implementation results (tests, SAST, regressions, performance), determining overall quality status, and deciding whether to proceed to deployment or return to development.
+
+## Prerequisites
+- test-results.yaml with final test execution results
+- sast-report.yaml with security analysis
+- regression-report.yaml with regression testing results
+- performance-report.yaml with benchmark results
+- quality-gates.yaml with pass/fail thresholds
+
+## Steps
+
+1. **Load All QA Implementation Reports**
+   - Read test-results.yaml and extract status, pass rate, coverage
+   - Read sast-report.yaml and extract vulnerability counts, severity levels
+   - Read regression-report.yaml and extract regression count, severity
+   - Read performance-report.yaml and extract performance status, degradation
+   - Validate all reports are present and complete
+
+2. **Evaluate Test Results Gate**
+   - **Pass Criteria**: 100% test pass rate, coverage >= target (75% overall)
+   - **Fail Criteria**: Any test failures, coverage < target
+   - **Status**: PASS, FAIL, or CONDITIONAL (minor coverage gaps)
+   - Document gate outcome with rationale
+
+3. **Evaluate SAST Gate**
+   - **Pass Criteria**: 0 critical vulnerabilities, < 3 high vulnerabilities
+   - **Fail Criteria**: >= 1 critical OR >= 3 high vulnerabilities (exploitable)
+   - **Status**: PASS, FAIL, or CONDITIONAL (high vulns with mitigations)
+   - Document gate outcome with rationale
+
+4. **Evaluate Regression Gate**
+   - **Pass Criteria**: 0 true regressions, all changes intentional
+   - **Fail Criteria**: >= 1 critical or high regression
+   - **Status**: PASS, FAIL, or CONDITIONAL (medium regressions only)
+   - Document gate outcome with rationale
+
+5. **Evaluate Performance Gate**
+   - **Pass Criteria**: All metrics within thresholds, < 10% degradation
+   - **Fail Criteria**: Critical metrics outside thresholds, > 20% degradation
+   - **Status**: PASS, FAIL, or CONDITIONAL (minor degradation with justification)
+   - Document gate outcome with rationale
+
+6. **Calculate Overall Quality Score**
+   - **Test Results**: 25 points (pass rate, coverage)
+   - **SAST**: 25 points (vulnerability severity, count)
+   - **Regressions**: 25 points (regression count, severity)
+   - **Performance**: 25 points (threshold compliance, degradation)
+   - Total: 100 points maximum
+
+7. **Apply Gate Logic**
+   - **ALL gates PASS**: Overall verdict = PASS
+   - **ANY gate FAIL**: Overall verdict = FAIL
+   - **ALL gates PASS or CONDITIONAL, >= 1 CONDITIONAL**: Overall verdict = CONDITIONAL
+   - Document gate combination and logic
+
+8. **Determine Verdict**
+   - **PASS**: All quality gates passed, ready for deployment
+   - **FAIL**: One or more critical issues, must return to development
+   - **CONDITIONAL**: Minor issues, can deploy with risk acceptance or quick fixes
+
+9. **Compile Blocking Issues (if FAIL or CONDITIONAL)**
+   - List all CRITICAL and HIGH severity issues
+   - For each issue:
+     - Source (test failure, SAST, regression, performance)
+     - Severity and impact
+     - Fix recommendation
+     - Estimated effort
+   - Prioritize issues by risk and user impact
+
+10. **Define Acceptance Criteria for PASS**
+    - If verdict is CONDITIONAL, define what must be done to achieve PASS:
+      - Specific issues to fix
+      - Acceptable risk mitigations
+      - Documentation requirements
+    - If verdict is FAIL, define complete remediation plan
+
+11. **Generate Handoff Recommendations**
+    - **PASS**: Handoff to devops agent for deployment
+    - **FAIL**: Handoff to dev agent for remediation, list priority fixes
+    - **CONDITIONAL**: Provide decision framework for stakeholder (deploy with risk vs fix)
+
+12. **Log Verdict and Update Session**
+    - Save qa-verdict.yaml with complete verdict
+    - Update session.yaml with QA implementation status
+    - If PASS (autonomous gate), trigger handoff to qa-impl-consolidate
+    - If FAIL, flag for orchestrator to return to dev
+    - If CONDITIONAL, prompt orchestrator for user decision
+
+## Decision Points
+
+- **CONDITIONAL Verdict Threshold**: If overall score is 90-95%, verdict is CONDITIONAL. Requires stakeholder decision: accept minor issues and deploy, or remediate first. Provide clear risk assessment.
+
+- **Autonomous Transition**: If verdict is PASS and overall score >= 95%, autonomous gate allows automatic transition to qa-impl-consolidate and then devops. No manual approval needed.
+
+- **Risk Acceptance for Deployment**: If verdict is CONDITIONAL due to minor performance degradation or low-severity SAST findings, assess whether issues are acceptable for production. Consider: user impact, workarounds, fix timeline.
+
+## Error Handling
+
+**Missing QA Report**
+- If any of the four prerequisite reports are missing, cannot issue verdict
+- Log error with specific missing file
+- Status = INCOMPLETE
+- Escalate to orchestrator for dependency resolution
+
+**Conflicting Gate Results**
+- If reports contain conflicting information (e.g., test-results says PASS but has failures listed), investigate
+- Trust most conservative interpretation (if ambiguous, treat as FAIL)
+- Log conflict and recommendation for report validation
+- Flag for manual review
+
+**Score Calculation Anomaly**
+- If overall score exceeds 100 or is negative, recalculate
+- Log calculation details for audit
+- If recalculation fails, default to FAIL verdict
+- Escalate for manual scoring review
+
+**Autonomous Gate Trigger Failure**
+- If autonomous gate should trigger but handoff fails, log error
+- Do not proceed to next agent automatically
+- Escalate to orchestrator for manual handoff
+- Document failure in session.yaml
+
+## Output Format
+
+```yaml
+# qa-verdict.yaml
+version: 1.0.0
+created: YYYY-MM-DD
+agent: qa-implementation
+phase: validate
+
+verdict: PASS # PASS, FAIL, CONDITIONAL
+
+overall_score: 96
+threshold_for_pass: 95
+threshold_for_conditional: 90
+
+gates:
+  test_results:
+    status: PASS
+    score: 24 / 25
+    rationale: "All tests passing (100% pass rate), coverage 76.3% exceeds target 75%"
+    details:
+      pass_rate: 100%
+      coverage: 76.3%
+      target_coverage: 75%
+    penalty: -1 (one module slightly below target, but overall exceeds)
+
+  sast:
+    status: PASS
+    score: 25 / 25
+    rationale: "0 critical, 0 high exploitable vulnerabilities after remediation"
+    details:
+      critical: 0
+      high: 0
+      medium: 6
+      low: 10
+    threshold:
+      critical: 0
+      high: "< 3"
+
+  regressions:
+    status: PASS
+    score: 24 / 25
+    rationale: "0 true regressions, all changes intentional and documented"
+    details:
+      true_regressions: 0
+      intentional_changes: 4
+      non_deterministic: 1
+    penalty: -1 (minor documentation completeness issue)
+
+  performance:
+    status: PASS
+    score: 23 / 25
+    rationale: "All metrics within thresholds, overall performance improved 5.2%"
+    details:
+      within_thresholds: true
+      degradation: -5.2% # negative = improvement
+      accepted_regressions: 2 # YAML parsing, validation (justified)
+    penalty: -2 (accepted performance trade-offs, minor)
+
+gate_logic:
+  all_pass: true
+  any_fail: false
+  any_conditional: false
+  result: PASS
+
+quality_assessment:
+  confidence: high
+  readiness: 96%
+
+  strengths:
+    - All tests passing with good coverage
+    - No critical or high security vulnerabilities
+    - No functional regressions
+    - Performance improved overall
+
+  areas_of_concern: []
+
+  minor_issues:
+    - One module (cli) coverage 2% below target (acceptable)
+    - YAML parsing 20% slower (justified by security fix)
+    - Consistency validation 12.5% slower (justified by reliability)
+
+  recommendation: "Proceed to deployment. All quality gates passed, system ready for release."
+
+blocking_issues: []
+
+conditional_pass_criteria: null # Not applicable for PASS verdict
+
+risk_acceptance: []
+
+handoff_decision:
+  next_agent: qa-impl-consolidate
+  autonomous: true
+  reason: "PASS verdict with score 96% >= 95% threshold, autonomous gate allows transition"
+
+deployment_readiness:
+  status: READY
+  confidence: high
+  prerequisites_met:
+    - All tests passing: true
+    - No critical vulnerabilities: true
+    - No regressions: true
+    - Performance acceptable: true
+    - Documentation complete: true
+
+  deployment_risks: []
+
+  post_deployment_monitoring:
+    - Monitor YAML parsing performance in production
+    - Track user feedback on CLI responsiveness
+    - Watch for any unreported edge cases
+
+  rollback_plan:
+    available: true
+    tested: true
+    details: "Upgrade system supports rollback to v1.0.0, tested in regression suite"
+
+recommendations:
+  immediate:
+    - Proceed to qa-impl-consolidate for final report
+    - Handoff to devops for deployment preparation
+    - Update baselines (test, performance) for future releases
+
+  short_term:
+    - Monitor performance metrics in production
+    - Collect user feedback on new features
+    - Schedule follow-up for minor issues in backlog
+
+  long_term:
+    - Evaluate alternative YAML parser for performance
+    - Consider CLI performance optimizations if user complaints
+    - Review and update quality gates based on production data
+
+summary_for_stakeholders:
+  status: APPROVED FOR DEPLOYMENT
+  quality_level: EXCELLENT
+  score: 96 / 100
+
+  highlights:
+    - Zero test failures across 247 tests
+    - Zero critical or high security vulnerabilities
+    - Zero functional regressions
+    - Performance improved by 5.2% overall
+
+  trade_offs_accepted:
+    - YAML parsing 20% slower due to security fix (acceptable)
+    - Minor increase in validation time for improved reliability (acceptable)
+
+  deployment_recommendation: "Deploy to production immediately. All quality criteria met or exceeded."
+
+input_reports:
+  - file: test-results.yaml
+    status: valid
+    outcome: PASS
+    score_contribution: 24 / 25
+
+  - file: sast-report.yaml
+    status: valid
+    outcome: PASS
+    score_contribution: 25 / 25
+
+  - file: regression-report.yaml
+    status: valid
+    outcome: PASS
+    score_contribution: 24 / 25
+
+  - file: performance-report.yaml
+    status: valid
+    outcome: PASS
+    score_contribution: 23 / 25
+
+session_update:
+  agent: qa-implementation
+  task: qa-impl-verdict
+  status: completed
+  verdict: PASS
+  score: 96
+  autonomous_transition: approved
+  next_agent: qa-impl-consolidate
+  timestamp: YYYY-MM-DDTHH:MM:SSZ
+
+next_steps:
+  - qa-impl-consolidate: Compile final QA implementation report
+  - devops: Begin deployment preparation (after consolidate)
+  - Archive QA reports for audit trail
+  - Update project baselines
+
+audit_trail:
+  qa_planning_score: 96
+  qa_implementation_score: 96
+  total_qa_score: 96
+  verdict: PASS
+  autonomous_gates_passed: 2
+  manual_overrides: 0
+  deviation_protocol_invoked: 0
+```