ace-test 0.6.0

data/handbook/templates/test-review-checklist.template.md

@@ -0,0 +1,163 @@
+---
+doc-type: template
+title: Test Review Checklist
+purpose: Test PR review checklist
+ace-docs:
+  last-updated: 2026-02-19
+  last-checked: 2026-03-21
+---
+
+# Test Review Checklist
+
+**PR**: #{{number}}
+**Package**: {{package}}
+**Reviewer**: {{name}}
+**Date**: {{date}}
+
+## Quick Summary
+
+- [ ] Tests added/modified: {{count}}
+- [ ] Test type: Unit / Integration / E2E
+- [ ] Performance verified: Yes / No / N/A
+
+---
+
+## 1. Layer Appropriateness
+
+Is each test at the correct layer?
+
+| Test | Current Layer | Correct? | Notes |
+|------|---------------|----------|-------|
+| {{test}} | Unit/Integration/E2E | Yes/No | {{notes}} |
+
+**Checklist**:
+- [ ] Unit tests have NO real I/O (subprocess, network, filesystem)
+- [ ] Integration tests stub external dependencies
+- [ ] E2E tests are in `test/e2e/TS-*/` format (scenario.yml + TC-*.tc.md)
+- [ ] No flag permutation tests in E2E (should be unit)
+- [ ] ONE CLI parity test per integration file max
+
+## 2. Stubbing Quality
+
+Are mocks/stubs correctly implemented?
+
+**Checklist**:
+- [ ] Boundary methods stubbed (not just inner methods)
+- [ ] `available?` checks stubbed if `run` is stubbed
+- [ ] No zombie mocks (stub targets match actual code)
+- [ ] Mock data is realistic (from snapshots or schemas)
+- [ ] Composite helpers used where appropriate
+
+**Red Flags**:
+- [ ] Deep nesting (>3 levels) without composite helper
+- [ ] Stubbing private methods
+- [ ] Mock expectations without behavior assertions
+
+## 3. Behavior vs Implementation
+
+Do tests verify behavior, not implementation details?
+
+**Checklist**:
+- [ ] Tests assert on OUTPUT, not method calls
+- [ ] Tests survive internal refactoring
+- [ ] Mock expectations only for side-effect methods
+- [ ] No testing of private method order
+
+**Example Check**:
+```ruby
+# BAD: Tests implementation
+mock.verify  # "Was X called?"
+
+# GOOD: Tests behavior
+assert_equal expected, result.output
+```
+
+## 4. Performance
+
+Will tests run fast enough?
+
+**Checklist**:
+- [ ] Profiled with `ace-test --profile 5`
+- [ ] Unit tests <100ms each
+- [ ] No `sleep` calls without stubbing
+- [ ] No subprocess calls without stubbing
+- [ ] Cache pre-warming if needed
+
+**Performance Check**:
+```bash
+ace-test {{package}} --profile 10
+# Verify no new tests >100ms
+```
+
+## 5. Coverage Quality
+
+Do tests actually catch bugs?
+
+**Checklist**:
+- [ ] Happy path tested
+- [ ] Error cases tested
+- [ ] Edge cases tested (nil, empty, boundaries)
+- [ ] Test fails when code is broken (try breaking it)
+
+**Negative Test Check**:
+- [ ] At least one error scenario tested
+- [ ] Invalid input handling verified
+- [ ] Exception/error messages checked
+
+## 6. E2E Specific (if applicable)
+
+For E2E tests in TS-format (`TC-*.tc.md`):
+
+**Checklist**:
+- [ ] PASS/FAIL assertions are explicit
+- [ ] File paths discovered at runtime, not hardcoded
+- [ ] Error test cases included (not just happy path)
+- [ ] Exit codes verified for error scenarios
+- [ ] Cleanup documented
+- [ ] Prerequisites listed
+
+## 7. Test Organization
+
+Is the test well-structured?
+
+**Checklist**:
+- [ ] Test file in correct directory (atoms/molecules/organisms/e2e)
+- [ ] Test name describes behavior (`test_returns_error_for_invalid_input`)
+- [ ] Arrange-Act-Assert pattern followed
+- [ ] No test interdependencies
+- [ ] Fixtures in `test/fixtures/` if shared
+
+---
+
+## Verdict
+
+- [ ] **Approve**: Tests are well-designed and performant
+- [ ] **Request Changes**: Issues identified above
+- [ ] **Needs Discussion**: Architectural concerns
+
+**Comments**:
+
+{{reviewer_comments}}
+
+---
+
+## Quick Reference
+
+### Performance Thresholds
+
+| Layer | Target | Warning | Critical |
+|-------|--------|---------|----------|
+| Unit (atoms) | <10ms | >50ms | >100ms |
+| Unit (molecules) | <50ms | >100ms | >200ms |
+| Integration | <500ms | >1s | >2s |
+
+### Stub the Boundary Pattern
+
+```ruby
+# Always stub availability check if stubbing execution
+Runner.stub(:available?, true) do
+  Runner.stub(:run, result) do
+    subject.process
+  end
+end
+```

data/handbook/workflow-instructions/test/analyze-failures.wf.md

@@ -0,0 +1,120 @@
+---
+doc-type: workflow
+title: Analyze Test Failures Workflow
+purpose: analyze-test-failures workflow instruction
+ace-docs:
+  last-updated: 2026-02-24
+  last-checked: 2026-03-21
+---
+
+# Analyze Test Failures Workflow
+
+## Goal
+
+Analyze failing automated tests and classify each failure before any fix is applied.
+
+This workflow produces a decision report that answers:
+- Is this failure caused by implementation code?
+- Is this failure caused by test code/spec?
+- Is this failure caused by test infrastructure/environment?
+
+## Hard Rule
+
+- Do not edit application code or test files in this workflow.
+- Do not run formatting/autofix commands in this workflow.
+- This workflow ends with an analysis report only.
+- Do not ask the user where/how to fix during this workflow; decide from evidence.
+
+## Prerequisites
+
+- Failing tests have already been executed
+- Failure output is available (logs, stack traces, failing test list)
+- Project context can be loaded
+
+## Project Context Loading
+
+- Read and follow: `ace-bundle wfi://bundle`
+- Check recent changes: `git log --oneline -10`
+
+## Classification Categories
+
+Use exactly one category per failing test:
+
+1. `implementation-bug`
+- Product/runtime behavior is wrong
+- Test expectation is valid
+
+2. `test-defect`
+- Test assertion/setup/fixture is stale or incorrect
+- Product behavior appears correct for current requirements
+
+3. `test-infrastructure`
+- Environment/tooling/framework/configuration/isolation issue
+- Failure is not specific to business behavior
+
+## Analysis Procedure
+
+1. Collect failing tests
+- Identify failing file/test IDs from latest run output
+- Capture exact error signatures
+
+2. Gather evidence per failure
+- Primary stacktrace line
+- Related test file and assertion context
+- Related implementation file/entrypoint context
+- Environment/tooling context (timeouts, missing deps, DB state, network/mocks)
+
+3. Classify each failure
+- Assign one category (`implementation-bug`, `test-defect`, `test-infrastructure`)
+- Add confidence: `high`, `medium`, or `low`
+- Record one disconfirming check (what could prove this classification wrong)
+- If confidence is `medium` or `low`, run at least one additional diagnostic read/search before final decision
+
+4. Determine fix target
+- `implementation code`
+- `test code`
+- `test infrastructure`
+
+5. Choose autonomous fix decision
+- Select a single primary fix action per failure
+- Provide concrete file targets in priority order
+- Define explicit no-touch boundaries
+- Do not emit option lists that require user selection
+
+## Required Output Contract
+
+Produce this section before exiting:
+
+```markdown
+## Failure Analysis Report
+
+| Failure | Category | Evidence | Fix Target | Fix Target Layer | Primary Candidate Files | Fallback Candidate Files | Do-Not-Touch Boundaries | Confidence | Disconfirming Check |
+|---|---|---|---|---|---|---|---|---|---|
+| path/to/test_file.rb:TestName | implementation-bug | stacktrace + behavior mismatch summary | implementation code | implementation | app/service.rb, app/model.rb | test/integration/foo_test.rb | test/e2e/** | high | run related tests after patch |
+```
+
+Then include:
+
+```markdown
+## Fix Decisions
+- First item to fix: ...
+- Chosen fix decision: ...
+- Why this target first: ...
+
+### Execution Plan Input
+- Primary failure to fix first: ...
+- Why first: ...
+- Required verification commands: ...
+- Expected pass criteria per command: ...
+```
+
+## Success Criteria
+
+- Every failing test is classified
+- Evidence is concrete and traceable
+- Fix target is explicit per failure
+- Fix target files are explicit per failure (primary + fallback)
+- No-touch boundaries are explicit per failure
+- A single autonomous chosen fix decision is present per failure
+- A prioritized first failure is selected
+- No code/test edits were made in this workflow