ace-test 0.6.0

data/handbook/guides/test-organization.g.md

@@ -0,0 +1,140 @@
+---
+doc-type: guide
+title: Test Organization
+purpose: Test file organization
+ace-docs:
+  last-updated: 2026-01-23
+  last-checked: 2026-03-21
+---
+
+# Test Organization
+
+## Flat Directory Structure
+
+All ACE gems use a **flat test directory structure** that mirrors the ATOM architecture:
+
+```
+test/
+├── test_helper.rb
+├── search_test.rb              # Main module test
+├── atoms/
+│   ├── pattern_analyzer_test.rb
+│   ├── result_parser_test.rb
+│   └── tool_checker_test.rb
+├── molecules/
+│   ├── preset_manager_test.rb
+│   └── git_scope_filter_test.rb
+├── organisms/
+│   ├── unified_searcher_test.rb
+│   └── result_formatter_test.rb
+├── models/
+│   └── search_result_test.rb
+└── integration/
+    └── cli_integration_test.rb
+```
+
+## Key Conventions
+
+- **Flat structure**: `test/atoms/`, not `test/ace/search/atoms/`
+- **Suffix naming**: `pattern_analyzer_test.rb`, not `test_pattern_analyzer.rb`
+- **Layer directories match ATOM architecture**: atoms, molecules, organisms
+- **Integration tests in separate `integration/` directory**
+
+## Benefits
+
+- Easier to navigate and find tests
+- Matches layer boundaries clearly
+- Consistent across all ACE gems
+- Less nesting = simpler paths
+
+See `ace-taskflow/test/` for reference implementation.
+
+## Naming Conventions
+
+### Test Files
+
+- Use `*_test.rb` suffix (Minitest convention)
+- Name matches the class being tested: `PatternAnalyzer` → `pattern_analyzer_test.rb`
+- One test file per class/module
+
+### Test Methods
+
+- Use `test_` prefix: `def test_finds_patterns_in_code`
+- Be descriptive: `test_returns_empty_when_no_matches` not `test_empty`
+- Include the scenario: `test_raises_error_on_invalid_input`
+
+### Test Classes
+
+- Mirror the class hierarchy: `class PatternAnalyzerTest < Minitest::Test`
+- Group related tests with modules if needed
+
+## Test Data
+
+### Fixtures
+
+Store test data in `test/fixtures/`:
+
+```
+test/fixtures/
+├── sample_config.yml
+├── git_diff_output.txt
+└── api_responses/
+    └── github_pr_123.json
+```
+
+### Creating Fixtures
+
+Use `yaml_fixture` helper for YAML fixtures:
+
+```ruby
+def test_loads_config
+  config = yaml_fixture("sample_config.yml")
+  assert_equal "expected_value", config["key"]
+end
+```
+
+### Inline Data
+
+Prefer inline data for small test cases:
+
+```ruby
+def test_parses_simple_input
+  input = "key: value"
+  result = Parser.parse(input)
+  assert_equal "value", result["key"]
+end
+```
+
+## Test Helpers
+
+### Location
+
+Place shared helpers in `test/test_helper.rb` or a dedicated `test/support/` directory:
+
+```
+test/
+├── test_helper.rb
+└── support/
+    ├── mock_git_repo.rb
+    └── api_stubs.rb
+```
+
+### Including Helpers
+
+```ruby
+# test_helper.rb
+require_relative "support/mock_git_repo"
+
+module TestHelpers
+  include MockGitRepo
+end
+
+class Minitest::Test
+  include TestHelpers
+end
+```
+
+## Related Guides
+
+- [Testing Philosophy](guide://testing-philosophy) - Why this structure
+- [Mocking Patterns](guide://mocking-patterns) - Test isolation patterns

data/handbook/guides/test-performance.g.md

@@ -0,0 +1,353 @@
+---
+doc-type: guide
+title: Test Performance
+purpose: Test performance optimization
+ace-docs:
+  last-updated: 2026-01-23
+  last-checked: 2026-03-21
+---
+
+# Test Performance
+
+## Performance Targets
+
+Define explicit performance expectations for each test layer based on patterns established during optimization of 9 ACE packages (60% average improvement).
+
+### Performance Thresholds by Test Type
+
+| Test Layer | Target Time | Hard Limit | Common Issues |
+|------------|-------------|------------|---------------|
+| Unit (atoms) | <10ms | 50ms | Real git ops, subprocess spawns |
+| Unit (molecules) | <50ms | 100ms | Unstubbed dependencies |
+| Unit (organisms) | <100ms | 200ms | Missing composite helpers |
+| Integration | <500ms | 1s | Too many real operations |
+| E2E | <2s | 5s | Should be rare - ONE per file |
+
+### Performance Cost Reference
+
+Know the cost of common operations to guide optimization:
+
+| Operation | Typical Cost | Notes |
+|-----------|--------------|-------|
+| Real `git init` | ~150-200ms | Use MockGitRepo instead |
+| Real `git commit` | ~50-100ms | Stub in unit tests |
+| Subprocess spawn (`Open3.capture3`) | ~150ms | Stub or use API calls |
+| Sleep in retry tests | 1-2s per sleep | Stub `Kernel.sleep` |
+| Cross-package require (cold) | ~50-100ms | Cache or stub dependencies |
+| `ace-nav` subprocess | ~150-400ms | Use `stub_synthesizer_prompt_path` |
+| Real LLM API call | 1-20s | Use WebMock to stub HTTP |
+| Real GitHub API call | 100-500ms | Use WebMock to stub HTTP |
+
+### When Tests Exceed Targets
+
+1. **Profile first**: Run `ace-test --profile 10` to identify actual bottlenecks
+2. **Check for zombie mocks**: Stubs that don't match actual code paths (see Zombie Mocks section)
+3. **Verify stubbing layer**: Stub at the boundary closest to your test subject
+4. **Consider composite helpers**: Reduce setup overhead with consolidated mock helpers
+5. **Apply E2E rule**: Keep ONE E2E test per file, convert rest to mocked versions
+
+## Sleep Stubbing for Retry Tests
+
+Tests with retry logic often include `sleep` calls that add seconds to test runtime.
+
+### Pattern: Stub Kernel.sleep
+
+```ruby
+def with_stubbed_sleep
+  Kernel.stub :sleep, nil do
+    yield
+  end
+end
+
+def test_retry_logic
+  with_stubbed_sleep do
+    # Retry tests now instant instead of 3+ seconds
+    result = RetryableOperation.call(max_retries: 3, delay: 1.0)
+    assert result.eventually_succeeded?
+  end
+end
+```
+
+### Alternative: Inject Sleep Dependency
+
+```ruby
+# Production code
+class RetryableOperation
+  def initialize(sleeper: Kernel)
+    @sleeper = sleeper
+  end
+
+  def call
+    attempts = 0
+    loop do
+      result = try_operation
+      return result if result.success?
+      attempts += 1
+      break if attempts >= max_retries
+      @sleeper.sleep(delay)
+    end
+  end
+end
+
+# Test with null sleeper
+def test_retry_without_delay
+  null_sleeper = Object.new
+  null_sleeper.define_singleton_method(:sleep) { |_| nil }
+
+  op = RetryableOperation.new(sleeper: null_sleeper)
+  result = op.call
+  assert result.eventually_succeeded?
+end
+```
+
+## E2E Test Strategy: Keep ONE Per Integration File
+
+Keep exactly ONE E2E test per integration test file that exercises real subprocess calls. Convert all other tests to use mocked versions.
+
+### When to Use Real E2E Tests
+
+**Keep as E2E (real subprocess)**:
+- CLI parity validation (CLI vs API produce same result)
+- Critical path smoke tests
+- Tool availability checks (gitleaks, git-filter-repo)
+- One representative test per integration file
+
+**Convert to Mocked**:
+- Flag/option permutation tests
+- Error handling tests
+- Edge case tests
+- Performance-critical paths
+
+### Migration Pattern: E2E to Mocked
+
+```ruby
+# BEFORE: E2E test using subprocess (~500ms each, 5 tests = 2.5s)
+def test_cli_with_verbose_flag
+  output, status = Open3.capture3(BIN, "analyze", "--verbose")
+  assert status.success?
+  assert_includes output, "Verbose output"
+end
+
+def test_cli_with_quiet_flag
+  output, status = Open3.capture3(BIN, "analyze", "--quiet")
+  assert status.success?
+  refute_includes output, "Debug"
+end
+
+# AFTER: Keep ONE E2E, convert rest to API tests (~5ms each)
+def test_cli_parity_with_api  # Keep this ONE E2E test
+  cli_output, status = Open3.capture3(BIN, "analyze", "file.rb")
+  api_result = Ace::MyModule.analyze("file.rb")
+  assert status.success?
+  assert_equal api_result.output, cli_output.strip
+end
+
+def test_verbose_flag  # Converted to API test
+  result = Ace::MyModule.analyze("file.rb", verbose: true)
+  assert result.success?
+  assert_includes result.output, "Verbose output"
+end
+
+def test_quiet_flag  # Converted to API test
+  result = Ace::MyModule.analyze("file.rb", quiet: true)
+  assert result.success?
+  refute_includes result.output, "Debug"
+end
+```
+
+### Real Example: ace-test-runner Optimization
+
+Task 175 reduced ace-test-runner tests from 8.25s to 3.3s (60% reduction) by:
+1. Keeping ONE E2E test for genuine CLI validation
+2. Converting 2 redundant E2E tests to use `run_ace_test_with_mock` helper
+3. Adding TestRunnerMocks infrastructure to ace-support-test-helpers
+
+```ruby
+# Helper for mocked CLI tests
+def run_ace_test_with_mock(args, expected_output: "", expected_status: 0)
+  mock_status = Object.new
+  mock_status.define_singleton_method(:success?) { expected_status == 0 }
+  mock_status.define_singleton_method(:exitstatus) { expected_status }
+
+  Open3.stub :capture3, [expected_output, "", mock_status] do
+    yield
+  end
+end
+```
+
+## Composite Test Helpers
+
+Reduce deeply nested stubs by creating composite helpers that combine related mocks.
+
+### The Problem: Deep Nesting
+
+```ruby
+# BAD: 6-7 levels of nesting (hard to read, slow due to setup overhead)
+def test_complex_operation
+  mock_config_loader do
+    mock_diff_generator do
+      mock_diff_filter do
+        mock_branch_info do
+          mock_pr_fetcher do
+            mock_commits_fetcher do
+              result = SUT.call
+              assert result.success?
+            end
+          end
+        end
+      end
+    end
+  end
+end
+```
+
+### The Solution: Composite Helpers
+
+```ruby
+# GOOD: Single composite helper with keyword options
+def test_complex_operation
+  with_mock_repo_load(branch: "feature", task_pattern: "123") do
+    result = SUT.call
+    assert result.success?
+  end
+end
+
+# In test_helper.rb - consolidates 6 stubs into one helper
+def with_mock_repo_load(branch: "main", task_pattern: nil, usable: true)
+  branch_info = build_mock_branch_info(name: branch, task_pattern: task_pattern)
+  mock_config = build_mock_config
+  mock_diff = Ace::Git::Models::DiffResult.empty
+
+  Ace::Config.stub :create, mock_config do
+    Ace::Git::Molecules::BranchInfo.stub :fetch, branch_info do
+      Ace::Git::Organisms::DiffOrchestrator.stub :generate, mock_diff do
+        yield
+      end
+    end
+  end
+end
+```
+
+### Composite Helper Design Principles
+
+1. **Sensible Defaults**: Most tests need standard values; customize only what matters
+2. **Keyword Arguments**: Allow targeted overrides without changing unrelated values
+3. **Clear Naming**: `with_mock_<context>` pattern indicates scope
+4. **Single Responsibility**: Each helper handles one "thing" completely
+
+### Examples from ACE Packages
+
+| Package | Helper | Purpose |
+|---------|--------|---------|
+| ace-git | `with_mock_repo_load` | Combines 6 stubs for RepoStatusLoader |
+| ace-git | `with_mock_diff_orchestrator` | ConfigLoader + DiffGenerator + DiffFilter |
+| ace-git-secrets | `with_rewrite_test_mocks` | gitleaks + rewriter + working directory |
+| ace-taskflow | `with_real_test_project` | ConfigResolver + project setup |
+| ace-docs | `with_empty_git_diff` | Simple DiffOrchestrator stub |
+| ace-review | `stub_synthesizer_prompt_path` | ace-nav subprocess stub |
+
+### Implementation Pattern
+
+```ruby
+# In test_helper.rb
+module CompositeHelpers
+  def with_empty_git_diff
+    empty_result = Ace::Git::Models::DiffResult.empty
+    Ace::Git::Organisms::DiffOrchestrator.stub(:generate, empty_result) do
+      yield
+    end
+  end
+
+  def with_mock_diff(content:, files: [])
+    mock_result = Ace::Git::Models::DiffResult.new(
+      content: content,
+      stats: { additions: 1, deletions: 0, files: files.size },
+      files: files
+    )
+    Ace::Git::Organisms::DiffOrchestrator.stub(:generate, mock_result) do
+      yield
+    end
+  end
+
+  def build_mock_status(success: true, exitstatus: 0)
+    status = Object.new
+    status.define_singleton_method(:success?) { success }
+    status.define_singleton_method(:exitstatus) { exitstatus }
+    status
+  end
+end
+
+class MyTestCase < Minitest::Test
+  include CompositeHelpers
+end
+```
+
+## Zombie Mocks Pattern
+
+"Zombie Mocks" occur when mocks stub methods that are no longer called by the implementation, but tests continue to pass because the real code path happens to work (slowly or otherwise).
+
+### Symptoms
+
+- Tests pass but are unexpectedly slow
+- Mock setup doesn't match actual code implementation
+- Refactored code still uses old mock patterns
+
+### Case Study: ace-docs ChangeDetector
+
+**Problem**: Tests stubbed `ChangeDetector.stub :execute_git_command` but the implementation had evolved to use `Ace::Git::Organisms::DiffOrchestrator.generate`. Tests passed but each ran real git operations (~1 second each).
+
+```ruby
+# ZOMBIE MOCK - stubs method no longer in code path
+ChangeDetector.stub :execute_git_command, "" do
+  result = ChangeDetector.get_diff_for_documents(docs, since: "HEAD~1")
+end
+
+# CORRECT - stubs actual method being called
+mock_result = Ace::Git::Models::DiffResult.empty
+Ace::Git::Organisms::DiffOrchestrator.stub :generate, mock_result do
+  result = ChangeDetector.get_diff_for_documents(docs, since: "HEAD~1")
+end
+```
+
+**Detection**: Run `ace-test --profile 10` to find slow unit tests. Tests taking >100ms often indicate zombie mocks.
+
+**Result**: Fixing zombie mocks reduced test time from 14s to 1.5s (89% improvement).
+
+### Prevention
+
+1. **Profile regularly**: Add `ace-test --profile 10` to development workflow
+2. **Review mock targets**: When refactoring, update test mocks to match new code paths
+3. **Extract helpers**: Create reusable mock helpers (like `with_empty_git_diff`) that are easy to maintain
+4. **Test the mocks**: Verify mocks are being hit by temporarily breaking them
+
+## When to Investigate Test Performance
+
+1. Run tests with profiling: `ace-test --profile 20`
+2. Look for patterns in slow tests (similar names, same file)
+3. Check for:
+   - Subprocess spawning
+   - Network I/O
+   - Disk I/O
+   - Sleep statements
+   - Large data processing
+
+## Monitoring Test Performance
+
+Add to your CI pipeline:
+
+```yaml
+- name: Check test performance
+  run: |
+    ace-test --profile 20 | tee profile.txt
+    # Fail if any test takes >100ms (except integration tests)
+    if grep -E "^\s+[0-9]+\.\s+test_(?!integration)" profile.txt | awk '{print $NF}' | grep -E "[0-9]+\.[1-9][0-9][0-9]s"; then
+      echo "Tests taking >100ms detected"
+      exit 1
+    fi
+```
+
+## Related Guides
+
+- [Mocking Patterns](guide://mocking-patterns) - How to stub properly
+- [Testing Philosophy](guide://testing-philosophy) - Why performance matters
+- [Testable Code Patterns](guide://testable-code-patterns) - Designing for fast tests

data/handbook/guides/test-responsibility-map.g.md

@@ -0,0 +1,220 @@
+---
+doc-type: guide
+title: Test Responsibility Map Guide
+purpose: Test responsibility mapping and risk-based coverage
+ace-docs:
+  last-updated: 2026-02-22
+  last-checked: 2026-03-21
+---
+
+# Test Responsibility Map Guide
+
+## Goal
+
+A Test Responsibility Map assigns each behavior to the **lowest test layer** that can prove it. This:
+- Prevents duplicate testing across layers
+- Keeps the fast loop fast
+- Ensures critical workflows get E2E coverage
+- Makes coverage gaps visible
+
+## Core Principle
+
+> Each behavior belongs to exactly ONE test layer. Test it at the lowest layer possible, promote only when necessary.
+
+## The Mapping Process
+
+### Step 1: List Behaviors
+
+Identify all behaviors from requirements:
+
+```markdown
+## Behaviors for ConfigParser
+
+1. Parse valid YAML file
+2. Return defaults for missing keys
+3. Raise error for malformed YAML
+4. Handle empty file
+5. Merge cascading configs
+6. CLI reports config errors with exit code 1
+```
+
+### Step 2: Assign Risk Levels
+
+| Risk Level | Criteria | Testing Intensity |
+|------------|----------|-------------------|
+| **High** | Security, data integrity, core business, user-facing errors | Must have unit + E2E |
+| **Medium** | Important functionality, configuration, integrations | Unit required |
+| **Low** | Logging, cosmetic, internal helpers | Unit if time permits |
+
+### Step 3: Map to Layers
+
+For each behavior, ask:
+
+1. **Can a unit test prove this?** (No I/O needed) → Unit
+2. **Does it need component interaction?** (Stubbed I/O) → Integration
+3. **Does it require real I/O to prove?** (CLI, network, filesystem) → E2E
+
+### Step 4: Build the Map
+
+| Behavior | Risk | Layer | Test File | Source of Truth |
+|----------|------|-------|-----------|-----------------|
+| Parse valid YAML | Medium | Unit | config_parser_test.rb | YAML schema |
+| Return defaults | Medium | Unit | config_parser_test.rb | defaults.yml |
+| Malformed YAML error | High | Unit | config_parser_test.rb | Exception spec |
+| Config cascade merge | Medium | Integration | config_resolver_test.rb | Merge rules |
+| CLI exit code 1 | High | E2E | TS-CONFIG-001 | CLI spec |
+
+## Layer Decision Rules
+
+### Unit Test If:
+
+- Pure logic with no side effects
+- Data transformation or validation
+- Error handling for invalid input
+- Edge cases (nil, empty, boundaries)
+
+**Stub everything**: filesystem, network, subprocess, git
+
+### Integration Test If:
+
+- Multiple components interact
+- Data flows between modules
+- Error propagation matters
+- ONE CLI parity check needed
+
+**Stub external dependencies**: APIs, subprocess calls
+
+### E2E Test If:
+
+- Complete user workflow
+- Real tool interaction required
+- Environment-specific behavior
+- Cannot be proven without real I/O
+
+**Use real I/O**: sandboxed, with cleanup
+
+## Avoiding Redundancy
+
+### Anti-Pattern: Testing Same Behavior at Multiple Layers
+
+```markdown
+# BAD: Same behavior tested 3 times
+- Unit: test_config_parser_returns_defaults
+- Integration: test_config_loader_uses_defaults
+- E2E: TC-001 verifies defaults in CLI output
+```
+
+### Pattern: Test at Lowest Layer, Verify at Higher
+
+```markdown
+# GOOD: Behavior at lowest, workflow at highest
+- Unit: test_config_parser_returns_defaults (proves logic)
+- Integration: (skip - unit covers it)
+- E2E: TC-001 verifies full config workflow (one test, not per feature)
+```
+
+## Risk-Based Coverage
+
+### High Risk Behaviors
+
+Must be tested thoroughly:
+
+```markdown
+| Behavior | Risk | Why High | Coverage |
+|----------|------|----------|----------|
+| Auth token validation | High | Security | Unit + E2E |
+| Data persistence | High | Data integrity | Unit + Integration + E2E |
+| Payment processing | High | Business critical | Unit + Integration + E2E |
+```
+
+### Low Risk Behaviors
+
+Basic coverage sufficient:
+
+```markdown
+| Behavior | Risk | Why Low | Coverage |
+|----------|------|---------|----------|
+| Log formatting | Low | Cosmetic | Unit happy path |
+| Debug output | Low | Internal | Skip or minimal |
+```
+
+## Template Usage
+
+Use the template at `templates/test-responsibility-map.template.md` when:
+
+- Starting a new feature
+- Auditing existing coverage
+- Planning test refactoring
+- Reviewing PR test coverage
+
+## Review Questions
+
+When reviewing a responsibility map:
+
+- [ ] Is each behavior at the lowest possible layer?
+- [ ] Are high-risk behaviors covered by E2E?
+- [ ] Are edge cases in unit tests, not E2E?
+- [ ] Any duplicate coverage across layers?
+- [ ] Source of truth identified for each behavior?
+
+## Common Mistakes
+
+### Mistake 1: E2E for Edge Cases
+
+```markdown
+# BAD: Testing every edge case in E2E
+- E2E: TC-001 valid config
+- E2E: TC-002 empty config
+- E2E: TC-003 missing key
+- E2E: TC-004 invalid YAML
+- E2E: TC-005 circular reference
+
+# GOOD: Edge cases in unit, workflow in E2E
+- Unit: test_empty_config, test_missing_key, test_invalid_yaml, test_circular_ref
+- E2E: TC-001 complete config workflow (happy + one error)
+```
+
+### Mistake 2: Missing High-Risk E2E
+
+```markdown
+# BAD: High-risk behavior only unit tested
+- Unit: test_auth_token_validation ✓
+- E2E: (none)
+
+# GOOD: High-risk has E2E verification
+- Unit: test_auth_token_validation ✓
+- E2E: TC-001 authentication workflow ✓
+```
+
+### Mistake 3: No Source of Truth
+
+```markdown
+# BAD: Mock data invented
+mock_response = { status: "ok" }  # Where does this come from?
+
+# GOOD: Mock data from real source
+mock_response = JSON.parse(File.read("fixtures/api_response.json"))
+# fixtures/api_response.json is snapshot from real API
+```
+
+## Integration with Workflows
+
+### With /ace-test-plan
+
+1. Generate responsibility map
+2. Identify gaps
+3. Plan tests by layer
+4. Output test plan
+
+### With /ace-test-verify-suite
+
+1. Check existing tests against map
+2. Identify redundancies
+3. Flag missing coverage
+4. Suggest optimizations
+
+## See Also
+
+- [Test Layer Decision](guide://test-layer-decision) - Layer decision matrix
+- [Test Mocking Patterns](guide://test-mocking-patterns) - How to stub
+- [Test Suite Health](guide://test-suite-health) - Metrics and audits