ace-test 0.6.0

data/handbook/templates/e2e-sandbox-checklist.template.md

@@ -0,0 +1,289 @@
+---
+doc-type: template
+title: E2E Sandbox Setup Checklist
+purpose: E2E sandbox setup checklist
+ace-docs:
+  last-updated: 2026-03-04
+  last-checked: 2026-03-21
+---
+
+# E2E Sandbox Setup Checklist
+
+**Test ID**: TS-{{AREA}}-{{NNN}}
+**Package**: {{package}}
+**Date**: {{date}}
+
+## Pre-Setup Verification
+
+### Tool Requirements
+
+- [ ] Required tools installed:
+  - [ ] {{tool_1}} (version: {{version}})
+  - [ ] {{tool_2}} (version: {{version}})
+- [ ] Tool versions verified:
+  ```bash
+  {{tool_1}} --version
+  {{tool_2}} --version
+  ```
+
+### Environment Requirements
+
+- [ ] Ruby version: {{version}}
+- [ ] Working directory writable
+- [ ] Network access (if needed): {{yes/no}}
+- [ ] External API access (if needed): {{yes/no}}
+
+---
+
+## Sandbox Creation
+
+### 1. Create Isolated Directory
+
+```bash
+TIMESTAMP_ID="$(ace-b36ts encode)"
+SHORT_PKG="{{short-pkg}}"  # Package name without ace- prefix
+SHORT_ID="ts{{nnn}}"       # Lowercase test number
+TEST_DIR=".ace-local/test-e2e/${TIMESTAMP_ID}-${SHORT_PKG}-${SHORT_ID}"
+REPORTS_DIR="${TEST_DIR}-reports"
+
+mkdir -p "$TEST_DIR"
+mkdir -p "$REPORTS_DIR"
+cd "$TEST_DIR"
+
+# Capture for cleanup
+echo "$TEST_DIR" > /tmp/current-e2e-sandbox
+```
+
+- [ ] Directory created
+- [ ] Reports directory created
+- [ ] Path recorded for cleanup
+
+### 2. Set Environment Variables
+
+```bash
+# Isolate from main project
+export PROJECT_ROOT_PATH="$TEST_DIR"
+
+# Disable any caching that might pollute
+export ACE_TEST_MODE=true
+
+# Set test-specific config if needed
+export ACE_CONFIG_PATH="$TEST_DIR/.ace"
+```
+
+- [ ] PROJECT_ROOT_PATH set to sandbox
+- [ ] Test mode enabled
+- [ ] Config path isolated
+
+### 3. Initialize Git Repository (if needed)
+
+```bash
+git init --quiet .
+git config user.email "test@example.com"
+git config user.name "Test User"
+```
+
+- [ ] Git initialized
+- [ ] User config set (prevents prompts)
+
+---
+
+## External API Safety
+
+### API Token Handling
+
+- [ ] Using test/sandbox tokens, NOT production
+- [ ] Token scope limited to minimum required
+- [ ] Token stored in environment variable, not file
+
+```bash
+# Example: GitHub test token with limited scope
+export GITHUB_TOKEN="${TEST_GITHUB_TOKEN:-}"
+if [ -z "$GITHUB_TOKEN" ]; then
+  echo "SKIP: No test token available"
+  exit 0
+fi
+```
+
+### API Call Safety
+
+- [ ] Test endpoints used (not production)
+- [ ] Rate limiting considered
+- [ ] Cleanup API calls planned
+
+| API | Test Endpoint | Production Endpoint | Using |
+|-----|---------------|---------------------|-------|
+| GitHub | api.github.com (test org) | api.github.com | Test |
+| LLM | - | - | Skipped/Mocked |
+
+### Rollback Plan
+
+If test creates resources that need cleanup:
+
+```bash
+# Record resources created
+echo "{{resource_type}}: {{resource_id}}" >> "$REPORTS_DIR/resources-created.txt"
+
+# Cleanup function
+cleanup_resources() {
+  while read -r line; do
+    type=$(echo "$line" | cut -d: -f1)
+    id=$(echo "$line" | cut -d: -f2)
+    # Delete resource
+    {{cleanup_command}} "$id"
+  done < "$REPORTS_DIR/resources-created.txt"
+}
+```
+
+- [ ] Resources tracked
+- [ ] Cleanup function defined
+- [ ] Cleanup runs on success AND failure
+
+---
+
+## Test Data Setup
+
+### Create Test Files
+
+```bash
+cat > "$TEST_DIR/{{filename}}" << 'EOF'
+{{file_content}}
+EOF
+```
+
+- [ ] Test files created
+- [ ] File permissions correct
+- [ ] Content matches test requirements
+
+### Create Test Configuration
+
+```bash
+mkdir -p "$TEST_DIR/.ace/{{tool}}"
+cat > "$TEST_DIR/.ace/{{tool}}/config.yml" << 'EOF'
+{{config_content}}
+EOF
+```
+
+- [ ] Config directory created
+- [ ] Config file created
+- [ ] Config values appropriate for test
+
+---
+
+## Execution Safety
+
+### Timeout Protection
+
+```bash
+# Prevent runaway tests
+timeout 300 {{command}} || {
+  echo "FAIL: Command timed out after 300s"
+  exit 1
+}
+```
+
+- [ ] Timeout set for long-running commands
+- [ ] Timeout value appropriate
+
+### Output Capture
+
+```bash
+# Capture all output for analysis
+{{command}} > "$REPORTS_DIR/stdout.txt" 2> "$REPORTS_DIR/stderr.txt"
+EXIT_CODE=$?
+echo "$EXIT_CODE" > "$REPORTS_DIR/exit_code.txt"
+```
+
+- [ ] stdout captured
+- [ ] stderr captured
+- [ ] Exit code recorded
+
+---
+
+## Cleanup
+
+### On Success
+
+```bash
+# Optional: Remove sandbox if test passed
+if [ "$ALL_TESTS_PASSED" = "true" ]; then
+  rm -rf "$TEST_DIR"
+  echo "Sandbox cleaned up"
+fi
+```
+
+### On Failure
+
+```bash
+# Keep sandbox for debugging
+echo "Sandbox preserved at: $TEST_DIR"
+echo "Reports at: $REPORTS_DIR"
+```
+
+### Always
+
+```bash
+# Restore environment
+unset PROJECT_ROOT_PATH
+unset ACE_TEST_MODE
+unset ACE_CONFIG_PATH
+
+# Clean up any API resources
+cleanup_resources 2>/dev/null || true
+```
+
+- [ ] Environment variables restored
+- [ ] API resources cleaned up
+- [ ] Sandbox disposition documented
+
+---
+
+## Verification Checklist
+
+Before running test:
+
+- [ ] Sandbox is isolated (PROJECT_ROOT_PATH set)
+- [ ] Git is initialized (if needed)
+- [ ] Test tokens are non-production
+- [ ] API scopes are minimal
+- [ ] Cleanup is defined
+- [ ] Timeouts are set
+- [ ] Output capture is configured
+
+After running test:
+
+- [ ] Results recorded in reports directory
+- [ ] Resources cleaned up (or cleanup planned)
+- [ ] Sandbox preserved (failure) or cleaned (success)
+- [ ] No pollution to main project
+
+---
+
+## Quick Setup Script
+
+```bash
+#!/bin/bash
+# e2e-sandbox-setup.sh
+
+set -e
+
+TIMESTAMP_ID="$(ace-b36ts encode)"
+SHORT_PKG="${1:-unknown}"
+SHORT_ID="${2:-ts000}"
+
+TEST_DIR=".ace-local/test-e2e/${TIMESTAMP_ID}-${SHORT_PKG}-${SHORT_ID}"
+REPORTS_DIR="${TEST_DIR}-reports"
+
+mkdir -p "$TEST_DIR" "$REPORTS_DIR"
+cd "$TEST_DIR"
+
+export PROJECT_ROOT_PATH="$TEST_DIR"
+export ACE_TEST_MODE=true
+
+git init --quiet .
+git config user.email "test@example.com"
+git config user.name "Test User"
+
+echo "Sandbox ready: $TEST_DIR"
+echo "Reports: $REPORTS_DIR"
+```

data/handbook/templates/test-case.template.md

@@ -0,0 +1,56 @@
+---
+doc-type: template
+title: "Test Case: {{FEATURE_NAME}}"
+purpose: Template for writing test cases
+ace-docs:
+  last-updated: 2026-01-23
+  last-checked: 2026-03-21
+---
+
+# Test Case: {{FEATURE_NAME}}
+
+## Description
+
+Brief description of what this test case covers.
+
+## Test Scenario
+
+### Given
+- Precondition 1
+- Precondition 2
+
+### When
+- Action or event occurs
+
+### Then
+- Expected outcome 1
+- Expected outcome 2
+
+## Test Implementation
+
+```ruby
+# frozen_string_literal: true
+
+require_relative "test_helper"
+
+module {{MODULE_NAME}}
+  class {{FEATURE_NAME}}Test < TestCase
+    def test_{{scenario_name}}
+      # Arrange
+      # Setup test conditions
+
+      # Act
+      # Execute the behavior being tested
+
+      # Assert
+      # Verify expected outcomes
+    end
+  end
+end
+```
+
+## Edge Cases to Consider
+
+1. Edge case 1
+2. Edge case 2
+3. Edge case 3

data/handbook/templates/test-performance-audit.template.md

@@ -0,0 +1,132 @@
+---
+doc-type: template
+title: Test Suite Performance Audit
+purpose: Test performance audit report
+ace-docs:
+  last-updated: 2026-02-01
+  last-checked: 2026-03-21
+---
+
+# Test Suite Performance Audit
+
+**Date**: {{YYYY-MM-DD}}
+**Scope**: {{package or "full monorepo"}}
+**Auditor**: {{name}}
+
+## Executive Summary
+
+| Metric | Value | Status |
+|--------|-------|--------|
+| Total packages | {{N}} | - |
+| Total tests | {{N}} | - |
+| Suite time | {{N}}s | OK / Warning / Critical |
+| Tests >100ms | {{N}} | OK / Warning / Critical |
+| Tests >200ms | {{N}} | OK / Warning / Critical |
+| Flake rate | {{N}}% | OK / Warning / Critical |
+
+**Overall Health**: Good / Warning / Critical
+
+## Performance Profile
+
+### Slowest Tests (Top 20)
+
+| # | Package | Test | Time | Category | Action |
+|---|---------|------|------|----------|--------|
+| 1 | {{pkg}} | {{test_name}} | {{time}}ms | Unit/Integration/E2E | Fix/Move/OK |
+
+### Package Summary
+
+| Package | Tests | Time | Avg | Slowest | Status |
+|---------|-------|------|-----|---------|--------|
+| {{pkg}} | {{N}} | {{N}}s | {{N}}ms | {{N}}ms | OK/Warning/Critical |
+
+## Threshold Violations
+
+### Critical (>200ms) - Must Fix
+
+| Package | Test | Time | Root Cause | Remediation |
+|---------|------|------|------------|-------------|
+| {{pkg}} | {{test}} | {{N}}ms | Zombie mock / Subprocess leak / Wrong layer | {{action}} |
+
+### Warning (100-200ms) - Should Fix
+
+| Package | Test | Time | Root Cause | Remediation |
+|---------|------|------|------------|-------------|
+| {{pkg}} | {{test}} | {{N}}ms | {{cause}} | {{action}} |
+
+## Zombie Mock Detection
+
+Tests that passed but showed performance anomalies:
+
+| Package | Test | Stub Target | Actual Code Path | Fix |
+|---------|------|-------------|------------------|-----|
+| {{pkg}} | {{test}} | {{stubbed_method}} | {{actual_method}} | Update stub to {{new_target}} |
+
+## Layer Classification Audit
+
+Tests potentially at wrong layer:
+
+| Package | Test | Current Layer | Suggested Layer | Reason |
+|---------|------|---------------|-----------------|--------|
+| {{pkg}} | {{test}} | Integration | E2E | Uses real subprocess |
+| {{pkg}} | {{test}} | Integration | Unit | No real I/O needed |
+
+## Trend Analysis
+
+### Compared to Previous Audit ({{previous_date}})
+
+| Metric | Previous | Current | Change |
+|--------|----------|---------|--------|
+| Suite time | {{N}}s | {{N}}s | +/-{{N}}% |
+| Tests >100ms | {{N}} | {{N}} | +/-{{N}} |
+| Flake rate | {{N}}% | {{N}}% | +/-{{N}}% |
+
+### Regression Sources
+
+- {{package}}: +{{N}}s due to {{reason}}
+- {{package}}: +{{N}} slow tests due to {{reason}}
+
+## Flakiness Report
+
+| Package | Test | Failure Rate | Cause | Fix |
+|---------|------|--------------|-------|-----|
+| {{pkg}} | {{test}} | {{N}}/{{N}} runs | Timing / Ordering / External | {{action}} |
+
+## Action Items
+
+### Immediate (This Week)
+
+- [ ] Fix critical violations in {{package}}
+- [ ] Update zombie mocks in {{package}}
+- [ ] Move {{test}} to E2E layer
+
+### Short-term (This Month)
+
+- [ ] Address warning-level violations
+- [ ] Investigate flaky tests
+- [ ] Update composite helpers in {{package}}
+
+### Long-term (This Quarter)
+
+- [ ] Improve coverage in {{area}}
+- [ ] Review E2E test relevance
+- [ ] Update mock data for API drift
+
+## Recommendations
+
+1. {{recommendation_1}}
+2. {{recommendation_2}}
+3. {{recommendation_3}}
+
+## Appendix: Commands Used
+
+```bash
+# Profile all packages
+ace-test-suite --profile
+
+# Profile specific package
+ace-test {{package}} --profile 20
+
+# Check flakiness (5 runs)
+for i in {1..5}; do ace-test-suite --quiet 2>&1 | grep FAIL; done
+```

data/handbook/templates/test-responsibility-map.template.md

@@ -0,0 +1,92 @@
+---
+doc-type: template
+title: "Test Responsibility Map: {{SCOPE}}"
+purpose: Test responsibility mapping document
+ace-docs:
+  last-updated: 2026-02-19
+  last-checked: 2026-03-21
+---
+
+# Test Responsibility Map: {{SCOPE}}
+
+**Scope**: {{feature_or_package}}
+**Owner**: {{owner}}
+**Date**: {{date}}
+**Status**: Draft | Review | Approved
+
+## Summary
+
+- Total behaviors: {{count}}
+- High risk: {{high_risk_count}}
+- Unit tests planned: {{unit_count}}
+- Integration tests planned: {{integration_count}}
+- E2E tests planned: {{e2e_count}}
+
+## Behaviors and Coverage
+
+| Behavior | Risk | Layer | Test File / ID | Source of Truth | Notes |
+|----------|------|-------|----------------|-----------------|-------|
+| {{behavior}} | High/Med/Low | Unit/Integration/E2E | {{test_file}} | {{source}} | {{notes}} |
+
+### Layer Legend
+
+- **Unit (atoms)**: Pure logic, no I/O, <10ms
+- **Unit (molecules)**: Composed logic, stubbed I/O, <50ms
+- **Integration (organisms)**: Component wiring, stubbed external, <500ms
+- **E2E**: Real I/O, sandboxed, seconds
+
+### Risk Legend
+
+- **High**: Security, data integrity, core business, user-facing errors
+- **Medium**: Important functionality, configuration, integrations
+- **Low**: Logging, cosmetic, internal helpers
+
+## Gaps Identified
+
+Behaviors without test coverage:
+
+- [ ] {{gap_1}}
+- [ ] {{gap_2}}
+
+## Redundancies to Remove
+
+Tests that duplicate coverage:
+
+- [ ] {{redundancy_1}} - covered by {{other_test}}
+- [ ] {{redundancy_2}} - covered by {{other_test}}
+
+## E2E Coverage Summary
+
+Critical workflows requiring E2E validation:
+
+| Workflow | E2E Test ID | Status |
+|----------|-------------|--------|
+| {{workflow_1}} | TS-{{AREA}}-{{NNN}} | Planned/Exists |
+| {{workflow_2}} | TS-{{AREA}}-{{NNN}} | Planned/Exists |
+
+## Fixtures and Contracts Needed
+
+| Fixture/Contract | Purpose | Source |
+|------------------|---------|--------|
+| {{fixture_name}} | {{purpose}} | Real API snapshot / Schema / Manual |
+
+## Action Items
+
+- [ ] Create unit tests for: {{list}}
+- [ ] Create integration tests for: {{list}}
+- [ ] Create E2E test: TS-{{AREA}}-{{NNN}}
+- [ ] Remove redundant tests: {{list}}
+- [ ] Update fixtures: {{list}}
+
+## Review Checklist
+
+- [ ] Each behavior tested at lowest possible layer
+- [ ] Critical workflows have E2E coverage
+- [ ] Edge cases in fast loop, not E2E
+- [ ] No duplicate tests across layers
+- [ ] Source of truth identified for each behavior
+- [ ] Risk levels assigned appropriately
+
+## Notes
+
+{{additional_notes}}