aios-core 2.1.0

package/.aios-core/infrastructure/tests/regression-suite-v2.md

@@ -0,0 +1,621 @@
+# Regression Test Suite V2.0
+
+**Created:** 2025-01-17  
+**Purpose:** Reusable test suite for validating V2.0 task format backward compatibility  
+**Story:** STORY-6.1.7.2 - Task Execution Validation
+
+---
+
+## Overview
+
+This regression test suite provides standardized test scenarios for validating that migrated V2.0 tasks maintain backward compatibility with V1.0 behavior while supporting new V2.0 features.
+
+---
+
+## Test Environment Setup
+
+### Prerequisites
+
+1. **Node.js:** v18+ installed and verified
+   ```bash
+   node --version
+   ```
+
+2. **Git CLI:** Available and configured
+   ```bash
+   git --version
+   ```
+
+3. **PM Tool Access:** Configured (ClickUp/GitHub/Jira)
+   - Check `.aios-core/core-config.yaml` for PM tool setting
+   - Ensure API credentials configured if required
+
+4. **Test Database:** Supabase local or test instance
+   - Database connection configured
+   - Migration directory accessible
+
+### Test Data Setup
+
+#### 1. Sample Story
+- **Location:** `docs/stories/test/test-001-sample-story.md`
+- **Story ID:** `STORY-TEST-001`
+- **Status:** Draft
+- **Purpose:** Used for story development and validation workflows
+
+#### 2. Sample Requirements
+- **Location:** `docs/requirements/test-sample-requirements.md`
+- **Format:** Standard PRD template
+- **Purpose:** Used for story creation workflow testing
+
+#### 3. Sample Migration
+- **Location:** `supabase/migrations/20250117000000_test_migration.sql`
+- **Type:** Creates `test_validation_table`
+- **Purpose:** Used for database migration workflow testing
+
+#### 4. PM Tool Test Story
+- **ClickUp:** Task ID format `12345678` (8-digit numeric)
+- **GitHub:** Issue number format `#123`
+- **Jira:** Task key format `TEST-123`
+- **Purpose:** Used for PM tool integration workflow testing
+
+---
+
+## Task Execution Tests
+
+### Priority 1 - Core Development Tasks
+
+#### Test 1.1: dev-develop-story.md
+
+**Test Data:**
+- Story file: `docs/stories/test/test-001-sample-story.md`
+- Mode: yolo (autonomous)
+
+**Execution Steps:**
+1. Activate dev agent
+2. Execute: `*develop-yolo docs/stories/test/test-001-sample-story.md`
+3. Verify story implementation completes
+4. Check Dev Agent Record updated
+5. Validate File List updated
+
+**Expected Results:**
+- ✅ Task executes without errors
+- ✅ Story status updated to "Ready for Review"
+- ✅ Dev Agent Record sections populated
+- ✅ Output format: Duration line 7, Tokens line 8, Metrics last
+
+**Validation Criteria:**
+- Exit code: 0
+- Required outputs generated
+- Post-conditions met
+- No error messages
+
+---
+
+#### Test 1.2: qa-gate.md
+
+**Test Data:**
+- Story file: Completed story from Test 1.1
+
+**Execution Steps:**
+1. Activate qa agent
+2. Execute: `*qa-gate {story-id}`
+3. Verify QA review completes
+4. Check QA Results section updated
+
+**Expected Results:**
+- ✅ QA gate document created
+- ✅ Test results documented
+- ✅ Issues identified (if any)
+- ✅ Final verdict provided
+
+**Validation Criteria:**
+- QA gate report generated
+- All test results documented
+- Clear pass/fail verdict
+
+---
+
+#### Test 1.3: create-next-story.md
+
+**Test Data:**
+- Requirements: `docs/requirements/test-sample-requirements.md`
+
+**Execution Steps:**
+1. Activate sm agent (or po agent)
+2. Execute: `*create-story` with requirements
+3. Verify story file created
+4. Validate story structure
+
+**Expected Results:**
+- ✅ Story file created in correct location
+- ✅ All required sections present
+- ✅ Story follows template format
+- ✅ Story ID assigned correctly
+
+**Validation Criteria:**
+- Story file exists
+- Template compliance verified
+- All sections populated
+
+---
+
+#### Test 1.4: validate-next-story.md
+
+**Test Data:**
+- Story file: Draft story (e.g., Test 1.3 output)
+
+**Execution Steps:**
+1. Activate po agent
+2. Execute: `*validate-story-draft {story-path}`
+3. Verify validation report generated
+4. Check for issues identified
+
+**Expected Results:**
+- ✅ Validation report created
+- ✅ Issues categorized (Critical/Should-Fix/Nice-to-Have)
+- ✅ Implementation readiness score provided
+- ✅ Clear GO/NO-GO decision
+
+**Validation Criteria:**
+- Validation report generated
+- All validation checks executed
+- Actionable feedback provided
+
+---
+
+#### Test 1.5: execute-checklist.md
+
+**Test Data:**
+- Checklist: `po-master-checklist.md` or `story-dod-checklist.md`
+
+**Execution Steps:**
+1. Execute: `*execute-checklist {checklist-name}`
+2. Verify checklist items checked
+3. Validate completion report
+
+**Expected Results:**
+- ✅ Checklist executed successfully
+- ✅ All items validated
+- ✅ Completion report generated
+- ✅ Issues documented (if any)
+
+**Validation Criteria:**
+- Checklist completes without errors
+- All items checked appropriately
+- Report generated
+
+---
+
+### Priority 2 - Agent-Specific Tasks
+
+#### Test 2.1: po-pull-story.md
+
+**Test Data:**
+- PM Tool: Configured PM tool (ClickUp/GitHub/Jira)
+- Task ID: Pre-created task in PM tool
+
+**Execution Steps:**
+1. Activate po agent
+2. Execute: `*pull-story {task-id}`
+3. Verify story file synced locally
+4. Validate story content matches PM tool
+
+**Expected Results:**
+- ✅ Story file created/updated locally
+- ✅ Content matches PM tool task
+- ✅ Metadata synced correctly
+- ✅ No data loss
+
+**Validation Criteria:**
+- Story file synced successfully
+- Content accuracy verified
+- Metadata preserved
+
+---
+
+#### Test 2.2: db-apply-migration.md
+
+**Test Data:**
+- Migration file: `supabase/migrations/20250117000000_test_migration.sql`
+
+**Execution Steps:**
+1. Activate db-sage agent (or dev agent)
+2. Execute: `*db-apply-migration {migration-file}`
+3. Verify migration applied
+4. Validate database state
+
+**Expected Results:**
+- ✅ Migration applied successfully
+- ✅ Database schema updated
+- ✅ No errors during migration
+- ✅ Rollback available if needed
+
+**Validation Criteria:**
+- Migration completes successfully
+- Schema changes verified
+- No data corruption
+
+---
+
+#### Test 2.3: create-agent.md
+
+**Test Data:**
+- Agent spec: Sample agent definition YAML
+
+**Execution Steps:**
+1. Execute: `*create-agent` with agent spec
+2. Verify agent file created
+3. Validate agent structure
+
+**Expected Results:**
+- ✅ Agent file created in correct location
+- ✅ Agent follows template format
+- ✅ All required sections present
+- ✅ Agent ID assigned correctly
+
+**Validation Criteria:**
+- Agent file exists
+- Template compliance verified
+- Valid agent definition
+
+---
+
+#### Test 2.4: create-task.md
+
+**Test Data:**
+- Task spec: Sample task definition
+
+**Execution Steps:**
+1. Execute: `*create-task` with task spec
+2. Verify task file created
+3. Validate task structure
+
+**Expected Results:**
+- ✅ Task file created in correct location
+- ✅ Task follows V2.0 format
+- ✅ All required sections present
+- ✅ Task ID assigned correctly
+
+**Validation Criteria:**
+- Task file exists
+- V2.0 format compliance verified
+- Valid task definition
+
+---
+
+#### Test 2.5: qa-run-tests.md
+
+**Test Data:**
+- Test suite: Project test suite
+
+**Execution Steps:**
+1. Activate qa agent
+2. Execute: `*run-tests`
+3. Verify tests execute
+4. Validate test results
+
+**Expected Results:**
+- ✅ All tests execute successfully
+- ✅ Test results documented
+- ✅ Coverage reported
+- ✅ Failures identified (if any)
+
+**Validation Criteria:**
+- Tests complete execution
+- Results accurately reported
+- Coverage metrics provided
+
+---
+
+### Priority 3 - Utility Tasks
+
+#### Test 3.1: correct-course.md
+
+**Test Data:**
+- Deviation scenario: Sample process deviation
+
+**Execution Steps:**
+1. Execute: `*correct-course` with deviation
+2. Verify correction identified
+3. Validate recommendations provided
+
+**Expected Results:**
+- ✅ Deviation identified
+- ✅ Root cause analyzed
+- ✅ Corrective actions recommended
+- ✅ Process updated if needed
+
+**Validation Criteria:**
+- Deviation correctly identified
+- Actionable recommendations provided
+- Process correction documented
+
+---
+
+#### Test 3.2: create-doc.md
+
+**Test Data:**
+- Template: PRD template or other doc template
+
+**Execution Steps:**
+1. Execute: `*create-doc` with template
+2. Verify document created
+3. Validate document structure
+
+**Expected Results:**
+- ✅ Document created successfully
+- ✅ Template followed correctly
+- ✅ All sections populated
+- ✅ Document formatted properly
+
+**Validation Criteria:**
+- Document exists
+- Template compliance verified
+- Content quality acceptable
+
+---
+
+#### Test 3.3: security-scan.md
+
+**Test Data:**
+- Target: Codebase or specific directory
+
+**Execution Steps:**
+1. Execute: `*security-scan {target}`
+2. Verify scan completes
+3. Validate security report generated
+
+**Expected Results:**
+- ✅ Security scan completes
+- ✅ Vulnerabilities identified
+- ✅ Severity levels assigned
+- ✅ Remediation recommendations provided
+
+**Validation Criteria:**
+- Scan completes successfully
+- All vulnerabilities reported
+- Recommendations actionable
+
+---
+
+#### Test 3.4: sync-documentation.md
+
+**Test Data:**
+- Target: `docs/` directory
+
+**Execution Steps:**
+1. Execute: `*sync-documentation {target}`
+2. Verify documentation synced
+3. Validate consistency
+
+**Expected Results:**
+- ✅ Documentation synced successfully
+- ✅ Consistency verified
+- ✅ Updates documented
+- ✅ No conflicts
+
+**Validation Criteria:**
+- Sync completes successfully
+- Documentation consistent
+- Changes documented
+
+---
+
+#### Test 3.5: improve-self.md
+
+**Test Data:**
+- Feedback: Sample improvement feedback
+
+**Execution Steps:**
+1. Execute: `*improve-self` with feedback
+2. Verify improvements identified
+3. Validate action plan created
+
+**Expected Results:**
+- ✅ Improvements identified
+- ✅ Action plan created
+- ✅ Prioritization provided
+- ✅ Implementation tracked
+
+**Validation Criteria:**
+- Improvements correctly identified
+- Action plan actionable
+- Prioritization logical
+
+---
+
+## Workflow Execution Tests
+
+### Workflow 1: Story Development Flow
+
+**Test Steps:**
+1. PO creates story from requirements
+2. Dev implements story
+3. QA runs quality gate
+
+**Expected Results:**
+- ✅ Story created successfully
+- ✅ Story implemented completely
+- ✅ QA gate passes
+- ✅ End-to-end flow works
+
+**Validation Criteria:**
+- All steps complete successfully
+- Data flows correctly between steps
+- No integration issues
+
+---
+
+### Workflow 2: PM Tool Integration Flow
+
+**Test Steps:**
+1. PO pulls story from PM tool
+2. Dev updates story locally
+3. PO syncs story back to PM tool
+
+**Expected Results:**
+- ✅ Story pulled successfully
+- ✅ Local updates committed
+- ✅ PM tool updated correctly
+- ✅ Bidirectional sync works
+
+**Validation Criteria:**
+- All steps complete successfully
+- Data integrity maintained
+- PM tool reflects changes
+
+---
+
+### Workflow 3: Database Migration Flow
+
+**Test Steps:**
+1. Dev runs dry-run validation
+2. Dev applies migration
+3. QA runs smoke test
+
+**Expected Results:**
+- ✅ Dry-run validates successfully
+- ✅ Migration applies without errors
+- ✅ Smoke test passes
+- ✅ Database state correct
+
+**Validation Criteria:**
+- All steps complete successfully
+- Database integrity maintained
+- Rollback available if needed
+
+---
+
+## Output Format Validation
+
+### Expected Output Format
+
+All task outputs must follow this format:
+
+```
+[Task execution content]
+
+Duration: [time]
+Tokens: [count]
+Metrics: [additional metrics]
+```
+
+**Validation:**
+- Duration on line 7 (or specified line)
+- Tokens on line 8 (or specified line)
+- Metrics at end
+
+---
+
+## Regression Detection
+
+### Critical Regression Criteria
+- Task fails to execute
+- Required output not generated
+- Post-conditions not met
+- Data corruption or loss
+
+### Minor Regression Criteria
+- Output format slightly different (but parseable)
+- Performance degraded >10% (but acceptable)
+- Warning messages (non-blocking)
+
+### Non-Regression (Acceptable)
+- Output format enhanced (additional fields)
+- Performance improved
+- Better error messages
+
+---
+
+## Performance Baseline
+
+### V1.0 Baseline
+- Not available (establishing V2.0 baseline)
+
+### V2.0 Baseline
+- To be established during first execution
+- Document duration, tokens, and metrics
+- Use as comparison for future changes
+
+---
+
+## Test Execution Log
+
+| Test ID | Task/Workflow | Status | Duration | Result | Notes |
+|---------|---------------|--------|----------|--------|-------|
+| 1.1 | dev-develop-story | Pending | - | - | - |
+| 1.2 | qa-gate | Pending | - | - | - |
+| 1.3 | create-next-story | Pending | - | - | - |
+| 1.4 | validate-next-story | Pending | - | - | - |
+| 1.5 | execute-checklist | Pending | - | - | - |
+| 2.1 | po-pull-story | Pending | - | - | - |
+| 2.2 | db-apply-migration | Pending | - | - | - |
+| 2.3 | create-agent | Pending | - | - | - |
+| 2.4 | create-task | Pending | - | - | - |
+| 2.5 | qa-run-tests | Pending | - | - | - |
+| 3.1 | correct-course | Pending | - | - | - |
+| 3.2 | create-doc | Pending | - | - | - |
+| 3.3 | security-scan | Pending | - | - | - |
+| 3.4 | sync-documentation | Pending | - | - | - |
+| 3.5 | improve-self | Pending | - | - | - |
+| W1 | Story Development Flow | Pending | - | - | - |
+| W2 | PM Tool Integration Flow | Pending | - | - | - |
+| W3 | Database Migration Flow | Pending | - | - | - |
+
+---
+
+## Automated Validation Scripts
+
+### Script 1: Output Format Validator
+
+```bash
+#!/bin/bash
+# Validates task output format
+
+validate_output_format() {
+    local output_file=$1
+    # Check for Duration line
+    # Check for Tokens line
+    # Check for Metrics section
+    # Return 0 if valid, 1 if invalid
+}
+```
+
+### Script 2: Regression Detector
+
+```bash
+#!/bin/bash
+# Detects regressions by comparing outputs
+
+detect_regressions() {
+    local baseline=$1
+    local current=$2
+    # Compare outputs
+    # Identify differences
+    # Classify as regression/enhancement/acceptable
+}
+```
+
+---
+
+## Maintenance
+
+### Updating Test Suite
+
+When updating this test suite:
+1. Document changes in Change Log
+2. Update test execution log
+3. Verify all tests still pass
+4. Update baseline metrics if needed
+
+### Version History
+
+| Version | Date | Changes | Author |
+|---------|------|---------|--------|
+| 1.0 | 2025-01-17 | Initial test suite creation | Dex (Dev) |
+
+---
+
+**Last Updated:** 2025-01-17  
+**Maintained By:** Dev Team
+