ace-test-runner-e2e 0.29.0

data/handbook/templates/scenario.yml.template.yml

@@ -0,0 +1,60 @@
+# E2E Test Scenario Configuration
+# Standalone runner/verifier pair format
+# See: handbook/guides/scenario-yml-reference.g.md
+
+# Required: Unique test identifier
+test-id: TS-{AREA}-{NNN}
+title: {Descriptive Title}
+area: {area-name}
+package: {package-name}
+
+# Optional: Priority level (high|medium|low)
+priority: high
+
+# Optional: Estimated duration
+duration: ~{X}min
+
+# Optional: Cost profile (smoke|happy-path|deep)
+cost-tier: smoke
+
+# Optional: Tags for discovery-time filtering (--tags/--exclude-tags, OR semantics)
+tags: [{cost-tier}, "use-case:{area}"]
+
+# Optional: Why this scenario must be E2E (not unit-only)
+e2e-justification: "{Requires real CLI/tools/filesystem behavior}"
+
+# Optional: Unit test files reviewed during Value Gate analysis
+unit-coverage-reviewed:
+  - test/{layer}/{file}_test.rb
+
+# Optional: Primary command under test
+tool-under-test: {ace-tool}
+
+# Optional: Declared sandbox artifact layout
+sandbox-layout:
+  results/tc/01/: "Goal 1 artifacts"
+
+# Optional: Prerequisites
+requires:
+  tools: [{tool1}, {tool2}]
+  ruby: ">= 3.0"
+
+# Setup directives (executed before test cases)
+# Available: git-init, run:, copy-fixtures, write-file:, agent-env:, tmux-session
+# Rules:
+# - setup is fail-fast (do not use `|| true`)
+# - fixtures/setup own sandbox preparation; runner TCs do not re-implement setup
+setup:
+  - git-init
+  # Optional detached tmux session for test isolation (uses unique run ID)
+  # - tmux-session:
+  #     name-source: run-id
+  - run: "cp $PROJECT_ROOT_PATH/mise.toml mise.toml && mise trust mise.toml"
+  # Uncomment as needed:
+  # - copy-fixtures         # Copy fixtures/ directory to sandbox
+  # - agent-env:            # Environment variables passed to runner/verifier agent subprocess
+  #     PROJECT_ROOT_PATH: "."
+
+# Optional: Verification tracking
+# last-verified: YYYY-MM-DD
+# verified-by: {agent-name}

data/handbook/templates/tc-file.template.md

@@ -0,0 +1,45 @@
+---
+doc-type: template
+title: Goal {N} - {Goal Title}
+purpose: Documentation for ace-test-runner-e2e/handbook/templates/tc-file.template.md
+ace-docs:
+  last-updated: 2026-02-25
+  last-checked: 2026-03-21
+---
+
+# Goal {N} - {Goal Title}
+
+## Goal
+
+{Outcome to achieve}
+
+## Workspace
+
+- Working directory: {sandbox-root}
+- Output directory: `results/tc/{NN}/`
+
+## Constraints
+
+- Use only declared scenario tools (`ace-*` and explicit exceptions)
+- Keep artifacts under `results/tc/{NN}/`
+- Do not write outside sandbox
+- Execute actions only; do not assign PASS/FAIL in runner file
+
+<!--
+Companion verifier file (`TC-{NNN}-{slug}.verify.md`) example:
+
+# Goal {N} - {Goal Title}
+
+## Expectations
+
+- Impact Checks:
+  - {Sandbox/project impact expectation}
+- Artifact Checks:
+  - {Artifact expectation}
+- Debug Fallback:
+  - {Optional stdout/stderr/exit evidence when needed}
+
+## Verdict
+
+- Pass when impact and artifact checks are satisfied from sandbox evidence.
+-->

data/handbook/templates/test-report.template.md

@@ -0,0 +1,94 @@
+---
+doc-type: template
+title: "E2E Test Report: {test-id}"
+purpose: Documentation for ace-test-runner-e2e/handbook/templates/test-report.template.md
+ace-docs:
+  last-updated: 2026-02-24
+  last-checked: 2026-03-21
+---
+
+# E2E Test Report: {test-id}
+
+## Test Information
+
+| Field | Value |
+|-------|-------|
+| Test ID | {test-id} |
+| Title | {test-title} |
+| Package | {package} |
+| Agent | {agent-name} |
+| Executed | {timestamp} |
+| Duration | {duration} | <!-- Format: "1m 23s" or "45s" -->
+
+## Results Summary
+
+| Test Case | Description | Status |
+|-----------|-------------|--------|
+| TC-001 | {description} | Pass/Fail |
+| TC-002 | {description} | Pass/Fail |
+| TC-003 | {description} | Pass/Fail |
+
+## Overall Status: {PASS/FAIL/PARTIAL}
+
+**Passed:** {count} | **Failed:** {count} | **Total:** {count}
+
+## Failed Test Details
+
+```yaml
+failed:
+  - tc: TC-{NNN}
+    category: tool-bug|runner-error|test-spec-error|infrastructure-error
+    evidence: "{Brief description of failure and supporting artifacts}"
+```
+
+### TC-{NNN}: {Test Case Name}
+
+**Objective:** {What this test case was verifying}
+
+**Expected:**
+- {Expected result 1}
+- {Expected result 2}
+
+**Actual:**
+- {Actual result 1}
+- {Actual result 2}
+
+**Error Output:**
+```
+{Captured error output if any}
+```
+
+**Analysis:** {Brief analysis of why the test failed}
+
+## Test Environment
+
+{Record environment details relevant to test execution.}
+
+| Component | Version/Value |
+|-----------|---------------|
+| Ruby | {version} |
+| Tool 1 | {version} |
+| Tool 2 | {version} |
+
+## Observations
+
+{Any observations, edge cases, or issues discovered during test execution that aren't failures but worth noting.}
+
+- {Observation 1}
+- {Observation 2}
+
+## Artifacts
+
+{List any artifacts created during test execution.}
+
+| Artifact | Path | Description |
+|----------|------|-------------|
+| Test data | `artifacts/` | Test input files |
+| Logs | `artifacts/output.log` | Command output logs |
+
+## Next Steps
+
+{Recommendations based on test results.}
+
+- [ ] {Action item if tests failed}
+- [ ] {Follow-up investigation if needed}

data/handbook/workflow-instructions/e2e/analyze-failures.wf.md

@@ -0,0 +1,126 @@
+---
+doc-type: workflow
+title: Analyze E2E Failures Workflow
+purpose: analyze-e2e-failures workflow instruction
+ace-docs:
+  last-updated: 2026-03-04
+  last-checked: 2026-03-21
+---
+
+# Analyze E2E Failures Workflow
+
+## Goal
+
+Analyze failing E2E scenarios and classify each failed test case before any fix is applied.
+
+This workflow determines whether each failure is caused by:
+- application/tool code
+- E2E test definition/spec
+- E2E runner/infrastructure
+
+## Hard Rule
+
+- Do not edit package code, scenario files, or runner code in this workflow.
+- Do not run rewrite/fix actions here.
+- This workflow ends with an analysis report only.
+- Do not ask the user where/how to fix during this workflow; decide from evidence.
+
+## Prerequisites
+
+- E2E tests have already run and produced cache artifacts
+- Reports are available under `.ace-local/test-e2e/*-reports/`
+
+## Project Context Loading
+
+- Read and follow: `ace-bundle wfi://bundle`
+- Read E2E guide: `ace-bundle guide://e2e-testing`
+- Check recent changes: `git log --oneline -10`
+
+## Classification Categories
+
+Use exactly one category per failed TC:
+
+1. `code-issue`
+- Tool behavior is incorrect relative to expected product behavior
+
+2. `test-issue`
+- Scenario/TC expectation, fixture, or steps are stale/incorrect
+
+3. `runner-infrastructure-issue`
+- Sandbox/setup/provider/parsing/orchestration issue
+
+## Required Evidence Sources
+
+Use these files as primary evidence:
+- `summary.r.md`
+- `experience.r.md`
+- `metadata.yml`
+- Relevant artifacts in `results/tc/{NN}/`
+
+## Analysis Procedure
+
+1. Locate latest failing report directories
+```bash
+ls -lt .ace-local/test-e2e/*-reports/ 2>/dev/null | head -20
+```
+
+2. For each failing scenario, extract:
+- failed TC IDs
+- reported category/evidence from metadata
+- corroborating artifact evidence
+
+3. Reclassify each failed TC if needed
+- Use `code-issue`, `test-issue`, or `runner-infrastructure-issue`
+- Add confidence: `high|medium|low`
+- Add one disconfirming check per TC
+- If confidence is `medium` or `low`, run at least one additional diagnostic read/search before final decision
+
+4. Recommend rerun scope (cost-aware)
+- `scenario` (default)
+- `package`
+- `suite`
+with explicit rationale
+
+5. Choose autonomous fix decision per failed TC
+- Select a single primary fix action
+- Provide concrete file targets in priority order
+- Define explicit no-touch boundaries
+- Do not emit option lists that require user selection
+
+## Required Output Contract
+
+Produce this section before exiting:
+
+```markdown
+## E2E Failure Analysis Report
+
+| Scenario / TC | Category | Evidence | Fix Target | Fix Target Layer | Primary Candidate Files | Fallback Candidate Files | Do-Not-Touch Boundaries | Confidence | Disconfirming Check | Rerun Scope |
+|---|---|---|---|---|---|---|---|---|---|---|
+| TS-FOO-001 / TC-003 | test-issue | summary + artifact mismatch details | scenario files | test-scenario-runner | TC-003-foo.runner.md | TC-003-foo.verify.md | lib/** | high | re-run scenario after spec adjustment | scenario |
+```
+
+Then include:
+
+```markdown
+## Fix Decisions
+- First item to fix: ...
+- Chosen fix decision: ...
+- Why this target first (unblocks most): ...
+
+### Execution Plan Input
+- First item to fix: ...
+- Why first (unblocks most): ...
+- Required verification commands: ...
+- Expected pass criteria per command: ...
+```
+
+## Success Criteria
+
+- Every failed TC has a category and evidence
+- Category is traceable to report/artifact facts
+- Fix target is explicit per failed TC
+- Fix target files are explicit per failed TC (primary + fallback)
+- No-touch boundaries are explicit per failed TC
+- A single autonomous chosen fix decision is present per failed TC
+- Rerun scope recommendation is cost-aware
+- No code/scenario/runner edits were made in this workflow