chati-dev 1.4.0 → 2.0.2

package/framework/tasks/qa-planning-consolidate.md

@@ -0,0 +1,309 @@
+---
+id: qa-planning-consolidate
+agent: qa-planning
+trigger: qa-planning-risk-matrix
+phase: clarity
+requires_input: false
+parallelizable: false
+outputs: [qa-plan.yaml]
+handoff_to: dev
+autonomous_gate: true
+criteria:
+  - Overall QA plan score >= 95%
+  - All quality gates defined
+  - Coverage targets set for all modules
+  - Risk matrix complete
+  - Test strategy documented
+---
+
+# QA Planning Consolidation
+
+## Purpose
+Compile all QA planning outputs into a final qa-plan.yaml document, calculate an overall quality score, and determine readiness for autonomous transition to the build phase.
+
+## Prerequisites
+- test-strategy.yaml (testing approach and tools)
+- quality-gates.yaml (gates and thresholds)
+- coverage-plan.yaml (module coverage targets)
+- risk-matrix.yaml (feature risk mapping)
+
+## Steps
+
+1. **Validate Input Documents**
+   - Verify all four prerequisite files exist and are valid YAML
+   - Check schema compliance for each document
+   - Ensure cross-references are consistent (e.g., risk levels in coverage-plan match risk-matrix)
+   - Flag any missing or incomplete sections
+
+2. **Calculate Test Strategy Score (25 points)**
+   - **Testing pyramid defined** (5 points): Unit, integration, e2e, manual scopes clear
+   - **Risk areas identified** (5 points): Critical, high, medium, low risks documented
+   - **Tools selected** (5 points): Test framework, SAST, performance, coverage tools specified
+   - **Test environments planned** (5 points): Local, CI, isolation, matrix defined
+   - **Execution order established** (5 points): Pre-commit, pre-push, CI, release order set
+
+3. **Calculate Quality Gates Score (25 points)**
+   - **Gates for all stages** (7 points): Pre-commit, clarity, build, validate, deploy gates defined
+   - **Thresholds with pass/fail** (6 points): Clear numeric or boolean thresholds for each gate
+   - **Enforcement mechanisms** (6 points): Automated, agent-enforced, manual processes specified
+   - **Bypass procedures** (3 points): Deviation protocol, manual approval, exceptions documented
+   - **Severity levels** (3 points): BLOCKER, CRITICAL, MAJOR, MINOR defined
+
+4. **Calculate Coverage Plan Score (25 points)**
+   - **All modules have targets** (8 points): Every module/component has specific coverage percentage
+   - **Target justifications** (5 points): Rationale provided for each target
+   - **Measurement approach** (5 points): Tool, metrics, enforcement, reporting defined
+   - **Hard-to-test strategies** (4 points): Mocking, isolation, alternative validation approaches
+   - **Exceptions documented** (3 points): Uncovered areas with justifications
+
+5. **Calculate Risk Matrix Score (25 points)**
+   - **All features mapped** (8 points): Every feature has impact, likelihood, risk level
+   - **Risk scoring methodology** (5 points): Clear formula for calculating risk
+   - **Mitigation strategies** (5 points): Technical, process, monitoring mitigations per risk
+   - **Testing priorities** (4 points): Priority 1-4 with corresponding test approaches
+   - **Risk acceptances** (3 points): Known accepted risks with residual risk documentation
+
+6. **Calculate Overall Score**
+   - Sum all four category scores (max 100 points)
+   - Apply penalties for inconsistencies:
+     - Cross-reference mismatch: -5 points per issue (e.g., coverage-plan risk level ≠ risk-matrix)
+     - Missing critical sections: -10 points per section
+     - Unrealistic targets: -5 points (e.g., 100% coverage on hard-to-test code)
+   - Final score = raw score - penalties
+
+7. **Assess Autonomous Transition Readiness**
+   - **Score >= 95%**: Autonomous transition to build phase approved
+   - **Score 90-94%**: Manual approval required, flag for user review
+   - **Score < 90%**: Insufficient quality, identify gaps and remediation
+
+8. **Identify Gaps and Recommendations**
+   - List specific missing or weak sections
+   - Provide concrete recommendations for improvement
+   - Prioritize gaps by impact (critical gaps block transition)
+   - Include estimated effort to close gaps
+
+9. **Cross-Validate with Constitution**
+   - Verify mode governance rules are respected (autonomous transition at 95%)
+   - Ensure quality gates align with mode constraints (clarity = read all, write chati.dev/ only)
+   - Check that agent handoff logic is valid (qa-planning → dev)
+   - Validate autonomous_gate: true is correctly set
+
+10. **Compile Consolidated QA Plan**
+    - Create qa-plan.yaml with summary of all planning documents
+    - Include overall score and subscores
+    - Document autonomous transition decision
+    - Add executive summary for stakeholders
+
+11. **Generate Handoff Artifacts**
+    - If score >= 95%, prepare handoff to dev agent
+    - Include prioritized test implementation roadmap
+    - Provide quick-start guide for dev agent
+    - Document any open questions or assumptions
+
+12. **Log Completion and Next Steps**
+    - Update session.yaml with qa-planning completion status
+    - Log overall score and autonomous transition decision
+    - If autonomous transition approved, trigger dev agent
+    - If manual approval needed, prompt orchestrator for user input
+
+## Decision Points
+
+- **Score in 90-94% Range**: This is the gray zone. Assess whether gaps are minor (documentation completeness) or substantive (missing risk areas). For minor gaps, recommend manual approval. For substantive gaps, require remediation.
+
+- **Conflicting Cross-References**: If coverage-plan.yaml and risk-matrix.yaml disagree on risk levels, trust risk-matrix as the source of truth (it's more comprehensive). Update coverage-plan mentally or flag for correction.
+
+- **Unrealistic Targets**: If coverage targets are excessively high (e.g., 100% on integration tests with external dependencies), apply realism penalty. Suggest adjusted targets in recommendations.
+
+## Error Handling
+
+**Missing Input Document**
+- Cannot proceed without all four planning documents
+- Log error with specific missing file
+- Return to orchestrator with dependency resolution request
+- Do not calculate partial score
+
+**Schema Validation Failure**
+- If a document fails schema validation, attempt to identify specific issue
+- Log detailed error (line number, field name, expected format)
+- Escalate to orchestrator for manual correction
+- Do not proceed with invalid data
+
+**Cross-Reference Inconsistencies**
+- If inconsistencies are minor (e.g., slight wording differences), document and proceed with penalty
+- If inconsistencies are major (e.g., module in coverage-plan not in risk-matrix), flag as critical gap
+- Provide specific remediation steps in recommendations
+
+**Score Below Threshold**
+- If overall score < 90%, compilation succeeds but transition fails
+- Generate detailed gap analysis with prioritized action items
+- Escalate to orchestrator for user decision (remediate or override via deviation protocol)
+- Log decision and rationale in session.yaml
+
+## Output Format
+
+```yaml
+# qa-plan.yaml
+version: 1.0.0
+created: YYYY-MM-DD
+agent: qa-planning
+
+overall_score: 96
+autonomous_transition: approved
+transition_to: dev
+
+subscores:
+  test_strategy: 24 / 25
+  quality_gates: 25 / 25
+  coverage_plan: 24 / 25
+  risk_matrix: 23 / 25
+
+scoring_breakdown:
+  test_strategy:
+    testing_pyramid_defined: 5 / 5
+    risk_areas_identified: 5 / 5
+    tools_selected: 5 / 5
+    environments_planned: 4 / 5 # CI matrix incomplete
+    execution_order: 5 / 5
+
+  quality_gates:
+    gates_all_stages: 7 / 7
+    thresholds_clear: 6 / 6
+    enforcement_mechanisms: 6 / 6
+    bypass_procedures: 3 / 3
+    severity_levels: 3 / 3
+
+  coverage_plan:
+    modules_have_targets: 8 / 8
+    target_justifications: 5 / 5
+    measurement_approach: 5 / 5
+    hard_to_test_strategies: 3 / 4 # IDE integration strategy weak
+    exceptions_documented: 3 / 3
+
+  risk_matrix:
+    features_mapped: 8 / 8
+    risk_scoring: 5 / 5
+    mitigation_strategies: 5 / 5
+    testing_priorities: 2 / 4 # Priority 4 features under-specified
+    risk_acceptances: 3 / 3
+
+penalties:
+  - issue: CI matrix incomplete (only 2 OSes specified, need 3)
+    penalty: -1
+  - issue: IDE integration testing strategy needs more detail
+    penalty: -1
+  - issue: Priority 4 features lack test case examples
+    penalty: -2
+
+gaps:
+  minor:
+    - CI matrix incomplete: Add Windows to existing Ubuntu + macOS
+    - IDE integration strategy: Add headless IDE API approach for automated testing
+    - Priority 4 test cases: Provide 2-3 example test cases for low-risk features
+
+  major: []
+
+recommendations:
+  - Close minor gaps in next iteration (estimated 2 hours)
+  - Current score 96% exceeds autonomous transition threshold
+  - Proceed to build phase with dev agent
+  - Revisit IDE testing strategy in Phase 1 (after installer complete)
+
+constitution_compliance:
+  mode_governance: compliant
+  autonomous_threshold: met (96% >= 95%)
+  agent_handoff: valid (qa-planning -> dev)
+  autonomous_gate: set correctly
+
+executive_summary:
+  status: APPROVED
+  confidence: high
+  readiness: 96%
+  key_strengths:
+    - Comprehensive risk matrix covering all features
+    - Clear quality gates with enforcement mechanisms
+    - Realistic coverage targets with justifications
+    - Thorough test strategy with tools and environments
+
+  areas_for_improvement:
+    - CI matrix should include Windows testing
+    - IDE integration testing needs automation approach
+    - Low-priority features need example test cases
+
+  recommendation: Proceed to build phase with dev agent. Address minor gaps during Phase 1 implementation.
+
+handoff_to_dev:
+  priority_1_tests:
+    - Mode transition validation (state management, orchestrator)
+    - session.yaml write operations (state management)
+    - Version migration execution (upgrade system)
+
+  priority_2_tests:
+    - Agent routing logic (orchestrator)
+    - Task definition loading (agents)
+    - config.yaml parsing (configuration)
+
+  test_implementation_order:
+    1. Set up test framework (Vitest) and coverage tool (c8)
+    2. Implement Priority 1 tests (critical risk)
+    3. Set up CI pipeline with quality gates
+    4. Implement Priority 2 tests (high risk)
+    5. Add integration tests for agent workflows
+    6. Implement Priority 3 tests (medium risk)
+
+  quick_start:
+    - Review test-strategy.yaml for testing approach
+    - Reference coverage-plan.yaml for module-specific targets
+    - Use risk-matrix.yaml to prioritize test implementation
+    - Enforce quality-gates.yaml thresholds in CI
+
+  open_questions: []
+
+  assumptions:
+    - Vitest is compatible with project Node.js version (22.x)
+    - CI environment (GitHub Actions assumed) supports matrix builds
+    - MCP tool responses can be mocked with fixture data
+
+input_documents:
+  - file: test-strategy.yaml
+    status: valid
+    score_contribution: 24 / 25
+
+  - file: quality-gates.yaml
+    status: valid
+    score_contribution: 25 / 25
+
+  - file: coverage-plan.yaml
+    status: valid
+    score_contribution: 24 / 25
+
+  - file: risk-matrix.yaml
+    status: valid
+    score_contribution: 23 / 25
+
+cross_validation:
+  coverage_vs_risk:
+    status: consistent
+    issues: []
+
+  gates_vs_risk:
+    status: consistent
+    issues: []
+
+  strategy_vs_coverage:
+    status: consistent
+    issues: []
+
+session_update:
+  agent: qa-planning
+  status: completed
+  score: 96
+  autonomous_transition: approved
+  next_agent: dev
+  timestamp: YYYY-MM-DDTHH:MM:SSZ
+
+next_steps:
+  - Orchestrator triggers dev agent (autonomous transition)
+  - Dev agent begins Phase 0 (Installer) implementation
+  - QA-Implementation agent on standby for build phase
+  - Revisit minor gaps during Phase 1 review
+```

package/framework/tasks/qa-planning-coverage-plan.md

@@ -0,0 +1,338 @@
+---
+id: qa-planning-coverage-plan
+agent: qa-planning
+trigger: qa-planning-gate-define
+phase: clarity
+requires_input: false
+parallelizable: false
+outputs: [coverage-plan.yaml]
+handoff_to: qa-planning-risk-matrix
+autonomous_gate: false
+criteria:
+  - Coverage targets set for all modules
+  - Target justifications documented
+  - Measurement approach defined
+---
+
+# Test Coverage Planning
+
+## Purpose
+Create a detailed test coverage plan with specific targets for each module, component, and layer of the chati.dev system, aligned with risk levels and quality gates.
+
+## Prerequisites
+- test-strategy.yaml defining overall testing approach
+- quality-gates.yaml with coverage thresholds
+- Architecture document showing system structure
+- Task breakdown identifying all components
+
+## Steps
+
+1. **Map System Architecture to Testable Units**
+   - Identify all modules: orchestrator, agents (13), state management, file operations, CLI, validators, parsers
+   - List core utilities: YAML handlers, i18n, error handling, logging
+   - Note external interfaces: MCP tools, IDE integration, package managers
+   - Document data structures: session.yaml schema, config.yaml, task definitions
+
+2. **Categorize Components by Testability**
+   - **High Testability**: Pure functions (validators, parsers, formatters, calculators)
+   - **Medium Testability**: Stateful components (agent task execution, state management)
+   - **Low Testability**: External dependencies (MCP calls, file system operations, IDE interactions)
+   - **Manual Only**: UI/UX (TUI dashboard, color rendering, i18n display)
+
+3. **Define Coverage Targets by Layer**
+   - **Core Logic Layer**: 80-90% (business rules, validation, state transitions)
+   - **Service Layer**: 70-80% (agent orchestration, task execution, file operations)
+   - **Integration Layer**: 60-70% (MCP interfaces, CLI commands, external tools)
+   - **Presentation Layer**: 40-50% (formatting, rendering, i18n output)
+
+4. **Set Module-Specific Targets**
+   - **Orchestrator**: 85% (critical routing and mode governance logic)
+   - **State Management**: 90% (session.yaml read/write, consistency checks)
+   - **Agent Task Execution**: 75% (task loading, validation, execution)
+   - **File Operations**: 80% (read/write/create, permission handling, error recovery)
+   - **Validators**: 90% (schema validation, input sanitization)
+   - **Parsers**: 85% (YAML parsing, frontmatter extraction, template rendering)
+   - **CLI**: 70% (argument parsing, command routing, help generation)
+   - **i18n**: 60% (translation loading, locale handling; manual testing for display)
+   - **Error Handling**: 75% (error construction, formatting, recovery)
+
+5. **Identify Coverage Measurement Approach**
+   - **Line Coverage**: Primary metric, tracks executed lines
+   - **Branch Coverage**: Secondary metric for conditional logic
+   - **Function Coverage**: Ensures all exported functions tested
+   - **Statement Coverage**: Granular metric for complex expressions
+   - **Tool**: c8 (Istanbul-based coverage for Node.js)
+
+6. **Plan for Hard-to-Test Areas**
+   - **File System Operations**: Use temporary directories and mock fs module
+   - **MCP Tool Calls**: Mock tool responses with fixture data
+   - **IDE Integration**: Test CLI contract, manual verification of IDE behavior
+   - **Process Exit/Signals**: Mock process object, test exit code logic
+   - **Async Operations**: Use async/await test patterns, avoid timing dependencies
+
+7. **Define Uncovered Areas (Explicitly Excluded)**
+   - **Third-Party Dependencies**: Coverage tracked in dependencies' own tests
+   - **Generated Code**: Auto-generated files excluded from coverage
+   - **Prototype/Experimental**: Features flagged as experimental excluded until stable
+   - **Deprecated Code**: Legacy code marked for removal excluded
+
+8. **Establish Coverage Baselines**
+   - **Initial Baseline**: Measure current coverage (if brownfield) or set 0% (if greenfield)
+   - **Sprint Targets**: Define incremental improvement goals (e.g., +5% per sprint)
+   - **Release Targets**: Set minimum coverage for each release milestone
+   - **Maintenance Target**: Steady-state coverage after initial development (e.g., 75%)
+
+9. **Plan Coverage Enforcement**
+   - **Pre-Commit**: Prevent commits that reduce coverage below threshold
+   - **CI Pipeline**: Fail build if coverage drops below gate threshold
+   - **PR Checks**: Require new code to meet higher coverage bar (80%)
+   - **Reporting**: Generate coverage reports in CI, publish to artifact storage
+
+10. **Define Coverage Exceptions Process**
+    - **Justification Required**: Why coverage target cannot be met
+    - **Alternative Validation**: How quality is ensured without coverage (e.g., manual testing)
+    - **Approval**: Who approves exception (tech lead, QA lead)
+    - **Review Cadence**: When exception is revisited (quarterly)
+
+11. **Create Coverage Improvement Roadmap**
+    - **Phase 0 (Installer)**: 70% target (focus on CLI and validation logic)
+    - **Phase 1-3 (Core Agents)**: 75% target (add agent task execution tests)
+    - **Phase 4-5 (Quality/Build)**: 80% target (comprehensive integration tests)
+    - **Phase 6-7 (Deploy/Validate)**: 85% target (full pipeline coverage)
+
+12. **Compile Coverage Plan Document**
+    - List all modules with specific targets
+    - Document measurement approach and tooling
+    - Include exceptions and justifications
+    - Map to quality gates and risk levels
+
+## Decision Points
+
+- **Target Adjustment for Legacy Code**: If brownfield project has existing low coverage, set realistic incremental targets rather than immediate 80% requirement. Example: 40% current → 50% sprint 1 → 60% sprint 2 → 70% sprint 3.
+
+- **Integration Test Coverage**: For external dependencies (MCPs, file system), decide between mocking (higher coverage, less realistic) vs real integration (lower coverage, more realistic). Recommend hybrid: unit tests with mocks, separate integration tests with real dependencies.
+
+- **Coverage vs Quality Trade-off**: High coverage doesn't guarantee quality. If team debates target strictness, emphasize meaningful tests over coverage percentage. Document agreement in coverage-plan.yaml.
+
+## Error Handling
+
+**Missing Architecture Details**
+- If architecture document lacks module boundaries, use best judgment based on file structure
+- Document assumptions and flag for architect review
+- Propose default targets (75% for unclear modules)
+
+**Conflicting Quality Gate Thresholds**
+- If quality-gates.yaml specifies 70% global but risk areas need 90%, document per-module targets
+- Ensure weighted average meets global threshold
+- Escalate if unresolvable conflict
+
+**Tool Limitations**
+- If coverage tool cannot measure certain code patterns (e.g., dynamic imports), document limitation
+- Propose alternative validation (manual code review, runtime monitoring)
+- Exclude unmeasurable code from coverage calculation with justification
+
+## Output Format
+
+```yaml
+# coverage-plan.yaml
+version: 1.0.0
+created: YYYY-MM-DD
+agent: qa-planning
+
+global_targets:
+  overall: 75%
+  new_code: 80%
+  core_logic: 85%
+  baseline: 0% # greenfield project
+
+measurement:
+  tool: c8
+  metrics:
+    primary: line_coverage
+    secondary: [branch_coverage, function_coverage]
+  enforcement:
+    pre_commit: true
+    ci_pipeline: true
+    pr_check: true
+  reporting:
+    format: [html, json, lcov]
+    artifact_storage: .coverage/
+
+modules:
+  orchestrator:
+    target: 85%
+    justification: Critical routing and mode governance logic
+    risk_level: critical
+    files:
+      - chati.dev/orchestrator.js
+      - chati.dev/mode-governance.js
+      - chati.dev/deviation-handler.js
+    hard_to_test:
+      - IDE detection: mock process.env and fs
+      - User prompts: mock readline interface
+
+  state_management:
+    target: 90%
+    justification: State corruption is critical risk
+    risk_level: critical
+    files:
+      - chati.dev/state/session-manager.js
+      - chati.dev/state/validators.js
+      - chati.dev/state/consistency-checker.js
+    hard_to_test:
+      - File system race conditions: use temporary directories
+      - Concurrent access: test with multiple async operations
+
+  agents:
+    target: 75%
+    justification: Task execution logic with external dependencies
+    risk_level: high
+    files:
+      - chati.dev/agents/*.js
+      - chati.dev/tasks/*.md (loaded dynamically)
+    hard_to_test:
+      - MCP tool calls: mock tool responses
+      - File writes: use temporary directories
+      - User input: mock readline
+
+  file_operations:
+    target: 80%
+    justification: File I/O errors are high risk
+    risk_level: high
+    files:
+      - chati.dev/utils/file-ops.js
+      - chati.dev/utils/yaml-handler.js
+    hard_to_test:
+      - Permission errors: mock fs with error injection
+      - Path traversal: test with malicious paths
+
+  validators:
+    target: 90%
+    justification: Pure functions, highly testable
+    risk_level: high
+    files:
+      - chati.dev/validators/*.js
+    hard_to_test: []
+
+  parsers:
+    target: 85%
+    justification: Parsing errors can corrupt workflow
+    risk_level: high
+    files:
+      - chati.dev/parsers/yaml-parser.js
+      - chati.dev/parsers/frontmatter.js
+      - chati.dev/parsers/template-renderer.js
+    hard_to_test:
+      - Malformed YAML: test with edge cases
+
+  cli:
+    target: 70%
+    justification: Argument parsing with many edge cases
+    risk_level: medium
+    files:
+      - chati.dev/cli/index.js
+      - chati.dev/cli/commands/*.js
+    hard_to_test:
+      - Process exit: mock process.exit
+      - Signal handling: mock process signals
+
+  i18n:
+    target: 60%
+    justification: Translation loading testable, display manual
+    risk_level: medium
+    files:
+      - chati.dev/i18n/loader.js
+      - chati.dev/i18n/translations/*.json
+    hard_to_test:
+      - Terminal rendering: manual verification
+      - Locale detection: mock process.env.LANG
+
+  error_handling:
+    target: 75%
+    justification: Error construction and formatting logic
+    risk_level: medium
+    files:
+      - chati.dev/errors/*.js
+    hard_to_test:
+      - Stack trace formatting: test with real errors
+
+  utilities:
+    target: 80%
+    justification: Shared utilities with broad usage
+    risk_level: medium
+    files:
+      - chati.dev/utils/*.js (excluding file-ops.js)
+    hard_to_test: []
+
+excluded:
+  - "node_modules/**"
+  - "**/*.test.js"
+  - "**/*.spec.js"
+  - "**/fixtures/**"
+  - "**/mocks/**"
+  - "build/**"
+  - "dist/**"
+
+exceptions:
+  - module: IDE integration
+    reason: Requires live IDE environment
+    alternative: Manual testing with 7 IDE compatibility matrix
+    approved_by: QA Lead
+    review_date: quarterly
+
+  - module: TUI dashboard
+    reason: Visual rendering difficult to test
+    alternative: Snapshot tests for text output, manual visual verification
+    approved_by: Tech Lead
+    review_date: quarterly
+
+baselines:
+  sprint_1: 70%
+  sprint_2: 75%
+  sprint_3: 80%
+  release_1.0: 75%
+  maintenance: 75%
+
+improvement_roadmap:
+  phase_0_installer:
+    target: 70%
+    focus: [cli, validators, file_operations]
+
+  phase_1_3_agents:
+    target: 75%
+    focus: [orchestrator, agents, state_management]
+
+  phase_4_5_quality:
+    target: 80%
+    focus: [integration tests, error_handling]
+
+  phase_6_7_deploy:
+    target: 85%
+    focus: [full pipeline coverage, edge cases]
+
+hard_to_test_strategies:
+  file_system:
+    approach: Temporary directories + mock fs
+    example: "Use tmp package for isolated test directories"
+
+  mcp_tools:
+    approach: Mock tool responses with fixtures
+    example: "Create fixtures/mcp-responses/*.json for each tool"
+
+  ide_integration:
+    approach: Test CLI contract, manual IDE verification
+    example: "Test npx chati-dev command output, verify in 7 IDEs"
+
+  async_operations:
+    approach: async/await test patterns
+    example: "Use async test functions, avoid setTimeout"
+
+  process_lifecycle:
+    approach: Mock process object
+    example: "Mock process.exit, process.on('SIGINT')"
+
+next_steps:
+  - Build risk matrix mapping features to coverage targets
+  - Validate coverage targets against timeline
+  - Begin test implementation in high-risk modules
+```