ace-test 0.6.0

data/handbook/agents/test.ag.md

@@ -0,0 +1,99 @@
+---
+name: test
+description: Run tests with smart defaults and helpful diagnostics
+expected_params:
+  required: []
+  optional:
+  - target: 'Test target - file path, directory (atoms, molecules), or group name'
+  - profile: 'Profile N slowest tests to identify performance issues'
+  - verbose: 'Show detailed test output'
+last_modified: '2026-01-22'
+type: agent
+source: ace-test
+---
+
+You are a testing specialist using the **ace-test** command-line tool.
+
+## Core Responsibilities
+
+Your primary role is to run tests efficiently and help diagnose issues:
+- Run tests with smart defaults based on context
+- Profile slow tests to identify performance bottlenecks
+- Help interpret test failures and suggest fixes
+- Guide users toward proper test organization
+
+## Primary Tool: ace-test
+
+Use the **ace-test** command for all test execution.
+
+## Commands
+
+### Run All Tests
+```bash
+# Run all tests in current package
+ace-test
+
+# Run all tests across monorepo
+ace-test-suite
+```
+
+### Run Specific Tests
+```bash
+# Run single test file
+ace-test test/atoms/pattern_analyzer_test.rb
+
+# Run test directory/group
+ace-test atoms
+ace-test molecules
+ace-test organisms
+ace-test integration
+```
+
+### Profile Tests
+```bash
+# Profile 10 slowest tests
+ace-test --profile 10
+
+# Profile slowest tests in a group
+ace-test atoms --profile 10
+```
+
+## Test Performance Targets
+
+When profiling, use these thresholds to identify issues:
+
+| Test Layer | Target Time | Hard Limit |
+|------------|-------------|------------|
+| Unit (atoms) | <10ms | 50ms |
+| Unit (molecules) | <50ms | 100ms |
+| Unit (organisms) | <100ms | 200ms |
+| Integration | <500ms | 1s |
+| E2E | <2s | 5s |
+
+## Interpreting Results
+
+When tests fail:
+1. Check the error message and stack trace
+2. Look for patterns (same file, similar names)
+3. Consider recent changes (`git log --oneline -10`)
+4. Check for zombie mocks if tests are slow but passing
+
+When tests are slow:
+1. Profile to identify bottlenecks
+2. Check for real I/O in unit tests
+3. Look for unstubbed subprocess calls
+4. Verify mocks are hitting actual code paths
+
+## Related Guides
+
+- [Quick Reference](guide://quick-reference) - TL;DR testing patterns
+- [Test Performance](guide://test-performance) - Optimization strategies
+- [Mocking Patterns](guide://mocking-patterns) - How to stub properly
+
+## Response Format
+
+When providing results:
+1. Show test command and summary
+2. Highlight failures with file:line references
+3. Suggest fixes based on error patterns
+4. Recommend profiling if tests seem slow

data/handbook/guides/SUMMARY.md

@@ -0,0 +1,95 @@
+# ACE Testing Guide Index
+
+Navigation index for all testing guides in the ace-test package.
+
+## Core Testing Guides
+
+| Guide | Protocol | Description |
+|-------|----------|-------------|
+| Quick Reference | `guide://quick-reference` | TL;DR of testing patterns - flat structure, naming, IO isolation |
+| Testing Philosophy | `guide://testing-philosophy` | Testing pyramid, IO isolation principle, when real IO is allowed |
+| Test Organization | `guide://test-organization` | Flat directory structure, naming conventions, layer boundaries |
+| Mocking Patterns | `guide://mocking-patterns` | MockGitRepo, WebMock, subprocess stubbing, ENV testing patterns |
+| Test Performance | `guide://test-performance` | Performance targets by layer, composite helpers, zombie mocks detection |
+| Testable Code Patterns | `guide://testable-code-patterns` | Avoiding exit calls, returning status codes, exception patterns |
+| Testing Guide | `guide://testing` | General testing guidelines and best practices |
+| TDD Cycle | `guide://testing-tdd-cycle` | Test-driven development implementation cycle |
+| Embedded Testing | `guide://embedded-testing-guide` | Embedded testing in workflows |
+
+## Test Strategy & Planning
+
+Decision frameworks for test design and layer assignment.
+
+| Guide | Protocol | Description |
+|-------|----------|-------------|
+| Testing Strategy | `guide://testing-strategy` | Fast/Slow loop strategy for high-performance test suites |
+| Test Layer Decision | `guide://test-layer-decision` | Decision matrix for unit vs integration vs E2E |
+| Test Responsibility Map | `guide://test-responsibility-map` | Map behaviors to test layers to avoid redundant coverage |
+
+## Test Quality & Health
+
+Patterns for maintaining test suite quality and performance.
+
+| Guide | Protocol | Description |
+|-------|----------|-------------|
+| Test Mocking Patterns | `guide://test-mocking-patterns` | Behavior testing, zombie mock detection, contract testing |
+| Test Suite Health | `guide://test-suite-health` | Metrics, CI integration, periodic audits |
+| Test Review Checklist | `guide://test-review-checklist` | Quick checklist for reviewing test PRs |
+
+## Technology-Specific Guides
+
+### Testing by Technology
+
+| Guide | File | Description |
+|-------|------|-------------|
+| RSpec Patterns | `testing/ruby-rspec.md` | Ruby RSpec-specific testing patterns |
+| RSpec Config Examples | `testing/ruby-rspec-config-examples.md` | RSpec configuration examples |
+| Rust Testing | `testing/rust.md` | Rust testing patterns |
+| TypeScript/Bun | `testing/typescript-bun.md` | Bun test patterns for TypeScript |
+| Vue/Vitest | `testing/vue-vitest.md` | Vue + Vitest testing patterns |
+| Vue/Firebase Auth | `testing/vue-firebase-auth.md` | Vue with Firebase authentication testing |
+| Test Maintenance | `testing/test-maintenance.md` | Test maintenance and refactoring guidelines |
+
+## TDD Cycle Guides
+
+### Test-Driven Development by Platform
+
+| Guide | File | Description |
+|-------|------|-------------|
+| Ruby Gem TDD | `test-driven-development-cycle/ruby-gem.md` | TDD workflow for Ruby gems |
+| Ruby Application TDD | `test-driven-development-cycle/ruby-application.md` | TDD workflow for Ruby applications |
+| Rust CLI TDD | `test-driven-development-cycle/rust-cli.md` | TDD for Rust CLI tools |
+| Rust WASM/Zed TDD | `test-driven-development-cycle/rust-wasm-zed.md` | TDD for Rust WASM/Zed extensions |
+| TypeScript Vue TDD | `test-driven-development-cycle/typescript-vue.md` | TDD for Vue applications |
+| TypeScript Nuxt TDD | `test-driven-development-cycle/typescript-nuxt.md` | TDD for Nuxt applications |
+| Meta Documentation TDD | `test-driven-development-cycle/meta-documentation.md` | TDD for documentation projects |
+
+## Access Guides via ace-nav
+
+```bash
+# Quick reference
+ace-nav guide://quick-reference
+
+# Testing philosophy
+ace-nav guide://testing-philosophy
+
+# Mocking patterns
+ace-nav guide://mocking-patterns
+
+# TDD cycle
+ace-nav guide://testing-tdd-cycle
+
+# New test strategy guides
+ace-nav guide://testing-strategy
+ace-nav guide://test-layer-decision
+ace-nav guide://test-mocking-patterns
+ace-nav guide://test-suite-health
+ace-nav guide://test-responsibility-map
+ace-nav guide://test-review-checklist
+```
+
+## See Also
+
+- [Workflows](../workflow-instructions/) - Testing-related workflows
+- [Agents](../agents/) - Testing automation agents
+- [Templates](../templates/) - Test case templates

data/handbook/guides/embedded-testing-guide.g.md

@@ -0,0 +1,261 @@
+---
+doc-type: guide
+title: Embedding Tests in AI Agent Workflows
+purpose: Embedded testing in workflows
+ace-docs:
+  last-updated: 2026-01-23
+  last-checked: 2026-03-21
+---
+
+# Embedding Tests in AI Agent Workflows
+
+This guide details the standard for incorporating tests directly within AI agent workflow instruction files.
+Integrating tests makes workflows more robust, provides faster feedback, and improves the reliability of
+automated tasks.
+
+## Purpose
+
+Embedding tests directly into workflow instructions allows an AI agent to:
+
+- Verify pre-conditions before starting complex operations
+- Validate the outcome of individual actions or tool uses
+- Confirm that a series of steps achieved the desired overall result
+- Request user verification for subjective or critical outputs
+- Ensure adherence to safety guardrails and compliance requirements
+- Support self-contained workflow execution without external test files
+
+This immediate feedback loop helps catch errors early, reduces the need for extensive manual checking,
+and improves automated processes. In the context of self-contained workflows, embedded tests are
+essential for validation without external dependencies.
+
+## Test Categories
+
+The following categories of tests can be embedded in workflows:
+
+1. **Pre-condition Checks:**
+   - **Description:** Verify that the environment and inputs are ready before an action or task begins.
+   - **Examples:** Ensure input files exist, required tools are available, API keys are set.
+
+2. **Action Validation (Tool-Specific Tests):**
+   - **Description:** Validate the immediate output or effect of a specific tool usage or agent action.
+   - **Examples:** Check if a file was correctly modified, a command ran successfully, an API call returned expected data.
+
+3. **Post-condition Checks (Task-Level Outcome Validation):**
+   - **Description:** Verify that a sequence of actions has achieved its broader goal.
+   - **Examples:** Confirm a generated report contains all necessary sections, a
+   refactoring task didn\'t break existing tests.
+
+4. **Output Validation (Against External Systems or Ground Truth):**
+   - **Description:** Compare the agent\'s final output with an external reference or known correct state.
+   - Examples: Check if a deployed service responds correctly, if data written
+     to a database matches expectations.
+
+5. **Guardrail Tests (Safety and Compliance):**
+   - **Description:** Ensure the agent\'s operations stay within safe boundaries and meet compliance rules.
+   - Examples: Prevent accidental deletion of critical files,
+     check for hardcoded secrets,
+     ensure generated code meets linting standards.
+
+6. **User Feedback/Verification Prompts:**
+   - **Description:** Solicit explicit confirmation from a human user for steps that are subjective, critical, or
+     difficult to automate verification for.
+   - **Examples:** Ask user to review a generated summary, confirm a proposed destructive action.
+
+## Syntax for Embedding Tests
+
+Tests are embedded in workflow markdown files using a specific blockquote structure. There are two main keywords:
+`TEST` for automated checks and `VERIFY` for user feedback prompts.
+
+```markdown
+```markdown
+
+> TEST: <Test Name (brief, human-readable)>
+>   Type: <Pre-condition | Action Validation | Post-condition | Output Validation | Guardrail>
+>   Assert: <Human-readable description of what\\\\\\\'s being checked>
+>   [Command: <executable command or script call, e.g., `bin/test --check-file ...`>]
+>   [File: <path_to_file_to_check_or_use_in_command>]
+>   [Pattern: <regex_or_string_pattern for_grep_like_checks>]\
+>   [Expected: <expected_value_or_outcome_description>]
+
+> VERIFY: <Verification Point Name>
+>   Type: User Feedback
+>   Prompt: <Text to display to the user for verification>
+>   [Options: <e.g., (yes/no), (proceed/abort/edit)>]
+```
+
+**Fields:**
+
+- `TEST:` / `VERIFY:`: Keyword to initiate the test or verification block. Followed by a human-readable name.
+- `Type:`: One of the defined test categories.
+- `Assert:` (for `TEST`): A clear, human-readable statement of the condition being checked.
+- `Command:` (optional, for `TEST`): The shell command the agent should execute to perform the test. A non-zero
+  exit code indicates failure. This is the **preferred method for automated checks**.
+- `File:` (optional, for `TEST`): Path to a file relevant to the test. Can be used by the `Command` or by the agent
+  for simple checks if no `Command` is provided.
+- `Pattern:` (optional, for `TEST`): A regex or string pattern to search for, typically within the `File`.
+- `Expected:` (optional, for `TEST`): A description of the expected outcome or value, useful if the `Command` doesn\'t
+  directly assert this.
+- `Prompt:` (for `VERIFY`): The question or instruction presented to the user.
+- `Options:` (optional, for `VERIFY`): Suggested responses for the user.
+
+## Agent Interpretation and Execution
+
+The AI agent should:
+
+1. **Parse Blocks:** Identify `> TEST:` and `> VERIFY:` blocks in the workflow markdown.
+2. **Execute `TEST` Blocks:**
+   - If a `Command:` is provided, execute it (e.g., using a `terminal` tool). The `bin/test` utility is designed to
+     be the common target for these commands. A non-zero exit status from the command signifies a test failure.
+   - If no `Command:` is provided, the agent may attempt simple, direct checks based on `File:`, `Pattern:`, and
+     `Assert:`, such as file existence or basic content checks. However, complex logic should always be
+     encapsulated in a `Command:`.
+3. **Handle `VERIFY` Blocks:**
+    - Display the `Prompt:` text to the user.
+    - Wait for user input. The workflow may pause or proceed based on the response.
+4. **Handle Test Outcomes:**
+    - **Success:** Proceed with the workflow.
+    - **Failure (Automated Test or Negative User Feedback):** Report the failure (including test name, assertion,
+      and command output if any). Halt the current task or follow specific error-handling instructions in the
+      workflow, then await user guidance.
+5. **Log:** Record all test executions and their outcomes.
+
+## Integration with Self-Contained Workflows
+
+Embedded tests are crucial for maintaining workflow independence:
+
+1. **No External Test Scripts**: Tests should be defined inline rather than referencing external test files
+2. **Self-Validating Steps**: Each major step can include its own validation
+3. **Context-Aware Testing**: Tests can reference files loaded in the Project Context Loading section
+4. **Technology-Agnostic Commands**: Use placeholder commands that can be adapted to different stacks
+
+### Example: Self-Contained Test in Planning Step
+
+```markdown
+### Planning Steps
+* [ ] Review dependency analysis findings
+  > TEST: Analysis Review Complete
+  > Type: Pre-condition Check
+  > Assert: Dependency analysis results are incorporated into refactoring plan
+  > Command: grep -l "dependency" workflow-independence-plan.md
+```
+
+## Examples
+
+### Simple, Fast Feedback Tests
+
+### Example 1: Check if a file was created
+
+## Step 2: Generate Configuration File
+
+The agent will generate `config.json` based on the inputs.
+
+> TEST: Config File Created
+> Type: Post-condition Check
+> Assert: The `config.json` file exists in the output directory.
+> File: output/config.json
+> Command: bin/test --check-file-exists output/config.json
+
+### Example 2: Check if a file contains specific text
+
+## Step 3: Update README
+
+The agent will add a "## Usage" section to `README.md`.
+
+```markdown
+
+> TEST: README Usage Section Added
+>   Type: Action Validation
+>   Assert: The `README.md` file now contains the \"## Usage\" heading.
+>   File: README.md
+>   Pattern: \"## Usage\"
+>   Command: bin/test --check-file-contains-pattern \"## Usage\" README.md
+
+```
+
+### Higher-Level Verification Tests
+
+### Example 3: Confirm a command achieves an outcome (e.g., code formatting)
+
+## Step 4: Format Source Code
+
+The agent will run the project's code formatter on all `.py` files.
+
+> TEST: Code Formatting Applied
+> Type: Post-condition Check
+> Assert: The code formatter reports no changes are needed, indicating formatting was successful.
+Command: black --check .  # (Assumes \\\'black\\\' exits non-zero if changes are needed)
+
+```markdown
+
+*Note: For commands like linters or formatters that exit 0 if successful (or no changes needed) and non-zero
+if issues are found/changes would be made, the agent might need to interpret the exit code accordingly.
+A wrapper script via `bin/test` could invert this if needed (e.g. `bin/test --expect-exit-code 0 \"black --check .\"`).*
+
+### Example 4: User verification of generated content
+
+## Step 5: Generate Project Summary
+
+The agent will write a summary of the project to `docs/summary.md`.
+
+> VERIFY: Summary Accuracy
+> Type: User Feedback
+> Prompt: Please review `docs/summary.md`. Does it accurately reflect the project\'s current state and goals?
+> Options: (Yes, Accurate / No, Needs Revision)
+```markdown
+
+
+**Example 5: Pre-condition check for API key**
+
+## Step 1: Initialize API Client
+
+The agent will prepare to make calls to an external service.
+
+> TEST: API Key Available
+>   Type: Pre-condition Check
+>   Assert: The `EXTERNAL_SERVICE_API_KEY` environment variable is set.
+>   Command: bin/test --check-env-var-set EXTERNAL_SERVICE_API_KEY
+```markdown
+
+## The `bin/test` Utility
+
+A helper script, `bin/test`, is envisioned to simplify common test operations invoked via the `Command:` field.
+This script would provide a consistent interface for checks like:
+
+- File existence (`--check-file-exists <path>`)
+- File non-existence (`--check-file-not-exists <path>`)
+- File is not empty (`--check-file-not-empty <path>`)
+- File contains a string/pattern (`--check-file-contains-pattern "<pattern>" <path>`)
+- Environment variable is set (`--check-env-var-set <VAR_NAME>`)
+- Arbitrary command execution and exit code checking (`--exec "your command here" --expect-exit-code <N>`)
+
+**Note**: In self-contained workflows, test commands should be generic placeholders when `bin/test`
+isn't available, with comments showing language-specific alternatives:
+
+```markdown
+> TEST: Version File Updated
+> Type: Action Validation
+> Assert: Version number updated in project files
+> Command: grep "version.*X.Y.Z" package.json || grep "version.*X.Y.Z" Cargo.toml || grep "VERSION.*X.Y.Z" setup.py
+```
+
+## Best Practices for Embedded Tests
+
+### In Self-Contained Workflows
+
+1. **Embed Test Logic**: Include test commands directly rather than referencing external test suites
+2. **Use Generic Patterns**: Write tests that can work across different technology stacks
+3. **Provide Alternatives**: Show multiple command options for different environments
+4. **Keep Tests Simple**: Focus on existence checks and pattern matching
+5. **Document Expected Output**: Be clear about what constitutes success
+
+### Example: Technology-Agnostic Test
+
+```markdown
+> TEST: Project Tests Pass
+> Type: Post-condition Check
+> Assert: All project tests pass successfully
+> Command: bin/test || npm test || bundle exec rspec || pytest || cargo test
+```
+
+By adopting this standard, AI agent workflows can become significantly more reliable and easier to debug, leading
+to more efficient and trustworthy automation, while maintaining complete independence from external resources.