ace-test 0.6.0

data/handbook/workflow-instructions/test/optimize.wf.md

@@ -0,0 +1,368 @@
+---
+doc-type: workflow
+title: Optimize Tests Workflow
+purpose: Systematically improve test suite performance
+ace-docs:
+  last-updated: 2026-03-12
+  last-checked: 2026-03-21
+---
+
+# Optimize Tests Workflow
+
+## Purpose
+
+Systematically optimize test performance by:
+1. Profiling to find slow tests
+2. Identifying root causes
+3. Applying appropriate fixes
+4. Migrating tests to correct layers
+5. Verifying improvements
+
+## When to Use
+
+- Test suite exceeds time budget
+- After `ace-bundle wfi://test/verify-suite` identifies issues
+- Before major releases
+- When developer feedback loop feels slow
+
+## Prerequisites
+
+- Package has existing tests
+- `ace-test` available
+- Understanding of test layer decision (see guide)
+
+## Workflow Steps
+
+### Step 1: Establish Baseline
+
+Profile current performance:
+
+```bash
+# Run 3 times to account for variance
+for i in 1 2 3; do
+  ace-test <package> --profile 10 2>&1 | tee profile-$i.txt
+done
+
+# Extract consistent slow tests
+cat profile-*.txt | grep -E "^\s+[0-9]+\." | sort | uniq -c | sort -rn
+```
+
+Record baseline:
+- Total suite time: ___s
+- Number of tests: ___
+- Slowest test: ___ (___ms)
+- Tests >100ms: ___
+
+### Step 2: Categorize Slow Tests
+
+For each slow test, identify the cause:
+
+| Cause | Symptoms | Fix |
+|-------|----------|-----|
+| Subprocess spawn | `Open3`, `system()` in stack | Stub availability + execution |
+| Real git operations | `git init`, `git commit` | Use MockGitRepo |
+| Network calls | HTTP requests | WebMock stubs |
+| Filesystem I/O | Large file operations | Use temp files, mock content |
+| Sleep statements | Retry logic with delays | Stub `Kernel.sleep` |
+| Zombie mocks | Stub exists but test still slow | Update stub target |
+| Wrong layer | E2E test in unit folder | Move to e2e/ |
+
+```bash
+# Search for subprocess calls in test
+ace-search "Open3\|system\(" <package>/test/
+
+# Search for real git operations
+ace-search "git init\|`git " <package>/test/
+
+# Search for sleep
+ace-search "sleep" <package>/test/
+```
+
+### Step 3: Apply Quick Wins
+
+#### 3a: Add Missing Availability Stubs
+
+Pattern found in many optimizations:
+
+```ruby
+# BEFORE: Stubs run but not available?
+Runner.stub(:run, mock_result) do
+  subject.lint(file)  # Calls available?() first!
+end
+
+# AFTER: Stub entire chain
+Runner.stub(:available?, true) do
+  Runner.stub(:run, mock_result) do
+    subject.lint(file)
+  end
+end
+```
+
+#### 3b: Stub Sleep in Retry Tests
+
+```ruby
+# Add to test or helper
+def with_stubbed_sleep
+  Kernel.stub :sleep, nil do
+    yield
+  end
+end
+
+# Use in tests
+def test_retry_logic
+  with_stubbed_sleep do
+    result = subject.retry_operation(max: 3, delay: 1.0)
+  end
+end
+```
+
+#### 3c: Replace Real Git with MockGitRepo
+
+```ruby
+# BEFORE: Real git (~150ms per init)
+def setup
+  @repo_path = Dir.mktmpdir
+  system("git", "-C", @repo_path, "init")
+  # ...
+end
+
+# AFTER: MockGitRepo (~0ms)
+def setup
+  @repo = MockGitRepo.new
+  @repo.add_commit("abc123", message: "test", files: ["test.rb"])
+end
+```
+
+### Step 4: Create Composite Helpers
+
+When multiple tests need the same stubs, create a helper:
+
+```ruby
+# In test_helper.rb
+module OptimizationHelpers
+  def with_mock_lint_context(validators: [:standardrb])
+    # Stub all validators as available
+    validators.each do |v|
+      runner = Ace::Lint::Atoms.const_get("#{v.to_s.capitalize}Runner")
+      runner.stub(:available?, true) do
+        runner.stub(:run, mock_lint_result) do
+          yield
+        end
+      end
+    end
+  end
+
+  def mock_lint_result
+    Ace::Lint::Models::LintResult.new(
+      issues: [],
+      exit_code: 0,
+      output: "No issues found"
+    )
+  end
+end
+```
+
+### Step 5: Migrate E2E Tests
+
+For tests that need real subprocess/git/filesystem:
+
+#### 5a: Identify E2E Candidates
+
+Tests that should move to E2E:
+- CLI output format validation
+- Tool availability checking
+- Full workflow with real dependencies
+- Tests that are still slow after stubbing
+
+#### 5b: Create E2E Test Directory
+
+```bash
+# Create E2E test scenario directory
+mkdir -p <package>/test/e2e/TS-<AREA>-00N-<slug>
+```
+
+**scenario.yml:**
+```yaml
+test-id: TS-<AREA>-00N
+title: <Descriptive Title>
+area: <area>
+package: <package>
+priority: medium
+requires:
+  tools: [<required-tools>]
+
+setup:
+  - git-init
+  - copy-fixtures
+  - env:
+      PROJECT_ROOT_PATH: "."
+```
+
+**TC-001-<scenario>.tc.md:**
+```markdown
+---
+tc-id: TC-001
+title: <Scenario>
+---
+
+## Objective
+
+Validate <what this tests> with real dependencies.
+
+## Steps
+
+1. <step>
+   ```bash
+   <command>
+   ```
+
+2. Verify result
+   ```bash
+   [ <assertion> ] && echo "PASS: <description>" || echo "FAIL: <description>"
+   ```
+
+## Expected
+
+- <assertion>
+```
+
+#### 5c: Delete or Convert Original Test
+
+```ruby
+# Option 1: Delete if fully covered by E2E
+# Just remove the test method
+
+# Option 2: Convert to mocked version
+def test_cli_output_format
+  # Mock subprocess, test via API
+  mock_result = { output: "Expected format" }
+  Runner.stub(:execute, mock_result) do
+    result = subject.generate_output
+    assert_equal "Expected format", result
+  end
+end
+```
+
+### Step 6: Pre-warm Caches
+
+If package has caching:
+
+```ruby
+# In test_helper.rb, at load time (not in setup)
+# Pre-warm availability caches
+Ace::Package::ValidatorRegistry.available?(:tool_a)
+Ace::Package::ValidatorRegistry.available?(:tool_b)
+```
+
+### Step 7: Verify Improvements
+
+Re-profile after changes:
+
+```bash
+ace-test <package> --profile 10
+```
+
+Compare to baseline:
+- Total suite time: ___s -> ___s (___% improvement)
+- Slowest test: ___ms -> ___ms
+- Tests >100ms: ___ -> ___
+
+### Step 8: Document Changes
+
+Create retro or update test helper comments:
+
+```ruby
+# test_helper.rb
+#
+# Performance Optimizations Applied:
+# - Pre-warm validator caches at startup (prevents subprocess on first access)
+# - with_mock_lint_context helper stubs all validators
+# - Real CLI tests moved to test/e2e/
+#
+# See: .ace-taskflow/.../retros/<timestamp>-<package>-test-optimization.md
+```
+
+## Common Optimization Patterns
+
+### Pattern: Subprocess Stubbing
+
+```ruby
+def with_stubbed_subprocess(output: "", status: 0)
+  mock_status = Object.new
+  mock_status.define_singleton_method(:success?) { status == 0 }
+  mock_status.define_singleton_method(:exitstatus) { status }
+
+  Open3.stub :capture3, [output, "", mock_status] do
+    yield
+  end
+end
+```
+
+### Pattern: Git Stubbing
+
+```ruby
+def with_mock_git_repo
+  repo = MockGitRepo.new
+  yield repo
+end
+
+def with_stubbed_git_status(clean: true)
+  Ace::Git::Atoms::StatusChecker.stub :clean?, clean do
+    yield
+  end
+end
+```
+
+### Pattern: Config Stubbing
+
+```ruby
+def with_stubbed_config(config_hash)
+  mock_config = Ace::Support::Config::Models::Config.wrap(config_hash)
+  Ace::Support::Config.stub :create, ->(*) { mock_config } do
+    yield
+  end
+end
+```
+
+### Pattern: LLM Stubbing
+
+```ruby
+def with_mock_llm_response(content:)
+  mock_response = {
+    "choices" => [{ "message" => { "content" => content } }]
+  }
+
+  WebMock.stub_request(:post, /api\.anthropic\.com|api\.openai\.com/)
+    .to_return(body: mock_response.to_json)
+
+  yield
+end
+```
+
+## Checklist
+
+- [ ] Baseline established (3 runs)
+- [ ] Slow tests categorized by cause
+- [ ] Availability stubs added
+- [ ] Sleep calls stubbed
+- [ ] Real git replaced with mocks
+- [ ] Composite helpers created
+- [ ] E2E tests migrated
+- [ ] Caches pre-warmed
+- [ ] Improvements verified
+- [ ] Changes documented
+
+## Expected Results
+
+| Before | After | Target |
+|--------|-------|--------|
+| 30s suite | <10s suite | 70%+ reduction |
+| 5 tests >100ms | 0 tests >100ms | Zero violations |
+| Flaky timing | Consistent | <10% variance |
+
+## See Also
+
+- [Test Performance Guide](guide://test-performance)
+- [Test Layer Decision Guide](guide://test-layer-decision)
+- [Test Mocking Patterns Guide](guide://test-mocking-patterns)
+- [Verify Test Suite Workflow](wfi://test/verify-suite)

data/handbook/workflow-instructions/test/performance-audit.wf.md

@@ -0,0 +1,17 @@
+---
+doc-type: workflow
+title: Test Performance Audit Workflow
+purpose: test performance audit workflow
+ace-docs:
+  last-updated: 2026-03-12
+  last-checked: 2026-03-21
+---
+
+# Test Performance Audit Workflow
+
+## Instructions
+
+1. Run `ace-test --profile 20 $ARGUMENTS`.
+2. Read `tmpl://test-performance-audit` and use it to structure findings.
+3. Read `guide://test-performance` for optimization guidance.
+4. Document slow tests, root causes, and proposed fixes.

data/handbook/workflow-instructions/test/plan.wf.md

@@ -0,0 +1,323 @@
+---
+doc-type: workflow
+title: Plan Tests Workflow
+purpose: Ensure comprehensive test coverage with tests at the right layer
+ace-docs:
+  last-updated: 2026-03-12
+  last-checked: 2026-03-21
+---
+
+# Plan Tests Workflow
+
+## Purpose
+
+Before writing code, plan what tests are needed at each layer. This prevents:
+- Missing test coverage
+- Tests at the wrong layer (slow unit tests, missing E2E)
+- Duplicate testing of same behavior at multiple layers
+
+This workflow embodies the **Test Planner** role (deciding WHAT and WHERE to test).
+The **Test Writer** role (implementing tests) follows separately.
+
+## Roles
+
+### Test Planner (This Workflow)
+
+**Focus**: Strategic decisions
+- WHAT behaviors need testing
+- WHERE (which layer) each belongs
+- WHAT risk level applies
+- WHAT fixtures/contracts are needed
+
+**Output**: Test Responsibility Map
+
+### Test Writer (Separate)
+
+**Focus**: Tactical implementation
+- HOW to implement each test
+- HOW to stub dependencies
+- HOW to assert results
+- HOW to maintain performance (<100ms)
+
+**Output**: Test files
+
+## When to Use
+
+- Before implementing a new feature
+- Before fixing a bug (plan regression test)
+- When reviewing existing test coverage
+- As part of `ace-bundle wfi://task/work`
+
+## Input
+
+- Feature description or task specification
+- List of files to be modified
+- Existing test coverage (optional)
+
+## Workflow Steps
+
+### Step 1: Understand the Change
+
+Analyze the feature/change to identify:
+
+1. **Pure logic components** (algorithms, transformations, validations)
+2. **Integration points** (component interactions, data flow)
+3. **External dependencies** (filesystem, network, subprocess, git)
+4. **User-facing behavior** (CLI output, exit codes, error messages)
+
+```
+Questions to answer:
+- What new functions/methods will be added?
+- What existing behavior might change?
+- What external systems are involved?
+- What are the error scenarios?
+```
+
+### Step 2: Identify Behaviors to Test
+
+For each component, list specific behaviors:
+
+```markdown
+## Behaviors to Test
+
+### Component: ConfigParser
+- [ ] Parses valid YAML file
+- [ ] Returns default values for missing keys
+- [ ] Raises error for malformed YAML
+- [ ] Handles empty file gracefully
+
+### Component: WorkflowOrchestrator
+- [ ] Executes steps in order
+- [ ] Stops on first failure
+- [ ] Reports partial progress on failure
+- [ ] Handles empty step list
+```
+
+### Step 3: Assign Risk Levels
+
+For each behavior, assess risk:
+
+| Risk Level | Criteria | Coverage Required |
+|------------|----------|-------------------|
+| **High** | Security, data integrity, core business, user-facing errors | Unit + E2E |
+| **Medium** | Important functionality, configuration | Unit required |
+| **Low** | Logging, cosmetic, internal helpers | Unit if time permits |
+
+Example:
+
+| Behavior | Risk | Why |
+|----------|------|-----|
+| Parse valid YAML | Medium | Core functionality |
+| Malformed YAML error | High | User-facing error handling |
+| CLI exit codes | High | User workflow |
+| Debug logging | Low | Internal only |
+
+### Step 4: Classify by Test Layer
+
+For each behavior, decide the appropriate layer:
+
+| Behavior | Risk | Layer | Rationale |
+|----------|------|-------|-----------|
+| Parse valid YAML | Medium | Unit | Pure function, no I/O |
+| Default values for missing keys | Medium | Unit | Data transformation |
+| Malformed YAML error | High | Unit | Error handling |
+| Steps execute in order | Medium | Integration | Component interaction |
+| CLI shows progress | High | E2E | User-facing behavior |
+
+**Decision criteria**:
+
+```
+Unit Test if:
+- Pure logic, no side effects
+- Can be tested with simple input/output
+- No external dependencies needed
+
+Integration Test if:
+- Multiple components interact
+- Needs controlled I/O (temp files)
+- Tests error propagation
+
+E2E Test if:
+- Tests complete user workflow
+- Requires real external tools
+- Validates CLI behavior
+```
+
+### Step 5: Define Mock Strategy
+
+For each non-E2E test, identify what to mock:
+
+```markdown
+## Mock Strategy
+
+### Unit Tests
+- Stub `FileSystem.read` with test content
+- Stub `Time.now` for timestamp tests
+- Use `MockGitRepo` for commit data
+
+### Integration Tests
+- Stub `Open3.capture3` for subprocess calls
+- Stub `WebMock` for API calls
+- Use temp directory for file operations
+
+### What NOT to Mock
+- The system under test itself
+- Simple value objects
+- Pure functions
+```
+
+### Step 6: Identify Edge Cases
+
+For each behavior, list edge cases:
+
+```markdown
+## Edge Cases
+
+### ConfigParser
+- Empty string input
+- Nil input
+- Very large file (>1MB)
+- Unicode characters in keys
+- Circular references
+
+### WorkflowOrchestrator
+- Zero steps
+- 100+ steps
+- Step throws exception
+- Step returns nil
+- Concurrent execution
+```
+
+### Step 7: Check for Existing Coverage
+
+Search for existing tests:
+
+```bash
+# Find existing tests for the component
+ace-search "class ConfigParser" --type test
+ace-search "def test.*config" --type test
+```
+
+Identify:
+- Tests that already cover some behaviors
+- Tests that need updating
+- Gaps in coverage
+
+### Step 8: Generate Test Responsibility Map
+
+Output the Test Responsibility Map document:
+
+```markdown
+# Test Responsibility Map: [Feature Name]
+
+## Summary
+- Total behaviors: N
+- High risk: N (require E2E coverage)
+- Unit tests planned: N
+- Integration tests planned: N
+- E2E tests planned: N
+
+## Responsibility Matrix
+
+| Behavior | Risk | Layer | Test File | Source of Truth |
+|----------|------|-------|-----------|-----------------|
+| Parse valid YAML | Medium | Unit | config_parser_test.rb | YAML schema |
+| Malformed YAML error | High | Unit | config_parser_test.rb | Error messages |
+| CLI exit codes | High | E2E | TS-CONFIG-001 | CLI spec |
+
+## Unit Tests (atoms/molecules)
+
+### File: test/atoms/config_parser_test.rb
+
+#### test_parses_valid_yaml
+- Input: Valid YAML string
+- Expected: Parsed hash with correct values
+- Mocks: None (pure function)
+
+#### test_returns_defaults_for_missing_keys
+- Input: YAML without optional keys
+- Expected: Hash with default values filled
+- Mocks: None
+
+#### test_raises_on_malformed_yaml
+- Input: Invalid YAML syntax
+- Expected: ParseError with line number
+- Mocks: None
+
+## Integration Tests (organisms)
+
+### File: test/organisms/workflow_orchestrator_test.rb
+
+#### test_executes_steps_in_order
+- Setup: Create 3 mock steps
+- Action: Execute orchestrator
+- Verify: Steps called in order
+- Mocks: Step executors
+
+#### test_stops_on_first_failure
+- Setup: Step 2 returns failure
+- Action: Execute orchestrator
+- Verify: Step 3 not called, error reported
+- Mocks: Step executors
+
+## E2E Tests
+
+### Directory: test/e2e/TS-FEATURE-001-workflow-execution/
+
+#### TC-001: Complete workflow success
+- Steps: Create config, run CLI, verify output
+- Expected: Exit 0, output contains success message
+
+#### TC-002: Workflow failure handling
+- Steps: Create invalid config, run CLI
+- Expected: Exit 1, error message is actionable
+
+## Mock Data Needed
+
+- fixtures/valid_config.yml
+- fixtures/malformed_config.yml
+- fixtures/large_config.yml
+
+## Composite Helpers Needed
+
+- with_mock_steps(count:, failing_at:)
+- with_temp_config(content:)
+```
+
+## Output
+
+The test plan should be saved to:
+- Task folder: `.ace-taskflow/.../task-XXX/test-plan.md`
+- Or reviewed in conversation before implementation
+
+## Checklist Before Implementation
+
+- [ ] All new behaviors have tests planned
+- [ ] Tests are at appropriate layers
+- [ ] Edge cases identified
+- [ ] Mock strategy defined
+- [ ] No duplicate testing across layers
+- [ ] E2E tests cover critical user paths only
+
+## Integration with Other Workflows
+
+### With `ace-bundle wfi://task/work`
+
+1. Load task specification
+2. **Load `ace-bundle wfi://test/plan`**
+3. Implement feature
+4. Write tests according to plan
+5. Verify coverage
+
+### With `ace-bundle wfi://test/create-cases`
+
+Use this workflow first to plan, then `ace-bundle wfi://test/create-cases` to generate test code.
+
+## See Also
+
+- [Test Layer Decision Guide](guide://test-layer-decision)
+- [Test Responsibility Map Guide](guide://test-responsibility-map)
+- [Test Mocking Patterns Guide](guide://test-mocking-patterns)
+- [Test Review Checklist](guide://test-review-checklist)
+- [Create Test Cases Workflow](wfi://test/create-cases)
+- [Verify Test Suite Workflow](wfi://test/verify-suite)

data/handbook/workflow-instructions/test/review.wf.md

@@ -0,0 +1,16 @@
+---
+doc-type: workflow
+title: Test Review Workflow
+purpose: test review workflow
+ace-docs:
+  last-updated: 2026-03-12
+  last-checked: 2026-03-21
+---
+
+# Test Review Workflow
+
+## Instructions
+
+1. Read `guide://test-review-checklist`.
+2. Review the requested tests for layer fit, mock quality, coverage, and performance.
+3. Read `guide://test-responsibility-map` when layer placement is unclear.