ace-test-runner-e2e 0.29.0

data/handbook/workflow-instructions/e2e/fix.wf.md

@@ -0,0 +1,166 @@
+---
+doc-type: workflow
+title: Fix E2E Tests Workflow
+purpose: fix-e2e-tests workflow instruction
+ace-docs:
+  last-updated: 2026-03-13
+  last-checked: 2026-03-21
+---
+
+# Fix E2E Tests Workflow
+
+## Goal
+
+Apply targeted fixes for failing E2E scenarios based on an existing E2E failure analysis report.
+
+This workflow is execution-only. Root cause classification is handled by `wfi://e2e/analyze-failures`.
+
+## Hard Gate (Required Before Any Fix)
+
+Do not apply any fix until an analysis report exists with:
+- scenario / TC identifier
+- category (`code-issue`, `test-issue`, `runner-infrastructure-issue`)
+- evidence from reports/artifacts
+- fix target
+- fix target layer
+- primary candidate files
+- do-not-touch boundaries
+- rerun scope recommendation
+
+If analysis is missing or incomplete, stop and run:
+```bash
+ace-bundle wfi://e2e/analyze-failures
+```
+
+## Required Input
+
+Use the output section from `e2e/analyze-failures`:
+- `## E2E Failure Analysis Report`
+- `## Fix Decisions`
+- `### Execution Plan Input`
+
+## Autonomy Rule
+
+- Do not ask the user to choose fix target, category, or rerun scope.
+- If analysis is incomplete, auto-complete missing decision fields via local evidence (reports, artifacts, scenario files, implementation), then proceed.
+- Only stop for hard blockers (missing files/tools/permissions).
+
+## Execution Environment Guardrail
+
+- Do **not** run E2E commands autonomously in constrained/sandboxed agent environments.
+- Treat `ace-test-e2e` as **user-executed verification** by default.
+- Provide exact rerun commands for the user instead of executing them when environment fidelity is uncertain (missing `mise`, restricted HOME/state dirs, missing provider credentials, restricted tmux/socket access).
+- Run E2E commands directly only when the user explicitly requests execution in the current environment and confirms it is properly configured.
+
+## Priority Order
+
+Apply fixes in this order:
+1. `runner-infrastructure-issue` (can unblock many scenarios)
+2. `code-issue`
+3. `test-issue`
+
+## Fix Procedure
+
+1. Pick the first prioritized item from analysis
+- Use the selected "First item to fix"
+- Confirm category, fix target, and rerun scope
+- Apply the "Chosen fix decision" and primary candidate files directly
+
+2. Apply category-specific fix
+
+### Category: runner-infrastructure-issue
+- Fix runner/sandbox/provider/reporting/orchestration behavior
+- Verify with runner tests when applicable: `ace-test ace-test-runner-e2e`
+
+### Category: code-issue
+- Fix package/tool behavior in implementation code
+- Add/update unit tests if needed
+
+### Category: test-issue
+- Fix scenario definition, runner/verifier criteria, fixtures, or setup steps
+- Preserve role split: runner is execution-only, verifier is impact-first verdict
+- Keep implementation unchanged unless analysis is revised
+
+3. Rerun the selected failing scope after each fix
+
+After every implemented fix, rerun the analysis-selected failing scope before moving to the next item or recommending release.
+
+```text
+# scenario scope (default)
+# user executes locally
+ace-test-e2e {package} {test-id}
+
+# package scope (only if analysis recommended)
+# user executes locally
+ace-test-e2e {package}
+```
+
+Rules:
+- Scenario rerun is the default after each fix iteration.
+- Use package rerun only when analysis explicitly selected package scope.
+- For multiple failing scenarios, rerun each scenario explicitly.
+```text
+ace-test-e2e ace-assign TS-ASSIGN-001
+ace-test-e2e ace-assign TS-ASSIGN-002
+ace-test-e2e ace-bundle TS-BUNDLE-001
+```
+- Record the rerun command and result in the execution summary for every fix item.
+
+4. Re-check classification when evidence conflicts
+- If outcome contradicts analysis, return to `e2e/analyze-failures`
+- Update analysis report and re-select a new autonomous chosen fix decision before continuing
+
+5. Iterate until all targeted failures are resolved
+- Keep one active scenario/TC at a time
+- Preserve cost-conscious rerun discipline
+
+6. Run a final explicit failing-scenario checkpoint before concluding the fix session
+
+After the currently targeted failures are addressed, require one final:
+
+```bash
+# user executes locally
+ace-test-e2e {package} {test-id}
+```
+
+Use one explicit command per previously failing scenario to confirm no targeted failure remains in the active set before ending the fix session or recommending release.
+
+## Cost-Conscious Rules
+
+- Do not run suite reruns by default
+- Prefer scenario reruns while iterating
+- Use package reruns only when analysis explicitly recommends broader scope
+
+## Required Output
+
+```markdown
+## E2E Fix Execution Summary
+
+| Scenario / TC | Category | Change Applied | Verification Command | Result |
+|---|---|---|---|---|
+| ... | ... | ... | ... | pass/fail |
+```
+
+Include one final row for the batch checkpoint:
+- Verification Command: one explicit rerun command per remaining failed scenario (`ace-test-e2e {package} {test-id}`)
+- Result: `pass` or remaining failing scenarios
+- If failures remain, continue the fix loop instead of treating the session as complete
+
+If unresolved:
+
+```markdown
+## Blockers
+- Scenario / TC: ...
+- Why unresolved: ...
+- New evidence: ...
+- Re-analysis required: yes/no
+```
+
+## Success Criteria
+
+- Fixes are traceable to analyzed failures
+- Verification scope matches analysis recommendation, including mandatory reruns after each fix
+- Cost-conscious rerun strategy was followed
+- Final explicit per-scenario rerun checkpoint for all targeted failures was completed before concluding the fix session
+- No user clarification was required for fix targeting/scope in normal flow
+- Targeted failures pass, or blockers are explicitly documented

data/handbook/workflow-instructions/e2e/manage.wf.md

@@ -0,0 +1,179 @@
+---
+doc-type: workflow
+title: Manage E2E Tests Workflow
+purpose: Orchestrate the 3-stage E2E test lifecycle pipeline (review → plan → rewrite)
+ace-docs:
+  last-updated: 2026-03-12
+  last-checked: 2026-03-21
+---
+
+# Manage E2E Tests Workflow
+
+This workflow is a lightweight orchestrator that chains the 3-stage E2E test pipeline. It delegates all logic to the specialized stage workflows.
+
+```text
+ace-bundle wfi://e2e/manage
+         │
+         ├─► Stage 1: ace-bundle wfi://e2e/review   (explore)
+         │        └─► Coverage matrix
+         │
+         ├─► Stage 2: ace-bundle wfi://e2e/plan-changes    (decide)
+         │        └─► Change plan
+         │
+         ├─► User confirmation gate
+         │
+         ├─► Stage 3: ace-bundle wfi://e2e/rewrite   (execute)
+         │        └─► Updated test files
+         │
+         └─► (optional) `ace-test-e2e` verification     (verify)
+```
+
+## Arguments
+
+- `PACKAGE` (required) - The package to manage (e.g., `ace-lint`)
+- `--dry-run` (optional) - Stop after presenting the change plan (skip execution)
+- `--run-tests` (optional) - Run all E2E tests after rewriting
+- `--tags TAG,...` (optional) - Include only scenarios matching any specified tag (OR semantics)
+- `--exclude-tags TAG,...` (optional) - Exclude scenarios matching any specified tag (OR semantics)
+
+## Guardrail
+
+- Do not execute `ace-test-e2e` / `ace-test-e2e-suite` autonomously in constrained or uncertain environments.
+- When verification is requested, provide exact commands for user execution unless the user explicitly requests local execution and confirms environment fidelity.
+
+## Workflow Steps
+
+### 1. Invoke Stage 1: Review
+
+Run the exploration stage to produce a coverage matrix:
+
+```bash
+ace-bundle wfi://e2e/review
+```
+
+Capture the full review report output including coverage matrix, overlap analysis, gap analysis, and health status.
+
+If the review finds no E2E tests and no features worth testing, report this and stop.
+
+### 2. Invoke Stage 2: Plan
+
+Run the decision stage to produce a concrete change plan:
+
+```bash
+ace-bundle wfi://e2e/plan-changes
+```
+
+Capture the change plan with its REMOVE / KEEP / MODIFY / CONSOLIDATE / ADD classifications and the proposed scenario structure.
+
+### 3. Present Plan for Confirmation
+
+Display the change plan from Stage 2 to the user. The plan includes:
+- Impact summary (current vs proposed scenarios/TCs/cost)
+- Classified actions for each TC
+- Proposed scenario structure
+
+**If `--dry-run`:** Stop here after presenting the plan. Do not execute.
+
+**If not dry-run:** Wait for user confirmation before proceeding to Stage 3.
+
+If the user requests modifications, re-run Stage 2 with their feedback incorporated.
+
+### 4. Invoke Stage 3: Rewrite
+
+After user confirms the plan, execute it:
+
+```bash
+ace-bundle wfi://e2e/rewrite
+```
+
+Capture the execution summary with files created, modified, and deleted.
+
+### 5. Run Tests (if --run-tests)
+
+If `--run-tests` flag is provided, verify the rewritten tests:
+
+```bash
+ace-test-e2e {PACKAGE} {TEST_ID_1}
+ace-test-e2e {PACKAGE} {TEST_ID_2}
+```
+
+Provide an explicit `--items` list from the scenarios you want to verify.
+
+Capture test results for the final summary.
+
+### 6. Report Final Summary
+
+Produce a combined summary of the full pipeline run:
+
+```markdown
+## E2E Management Summary: {package}
+
+**Executed:** {timestamp}
+**Pipeline:** review → plan → rewrite {→ test}
+
+### Pipeline Results
+
+| Stage | Status | Key Output |
+|-------|--------|------------|
+| Review | Done | {n} features, {n} unit tests, {n} E2E TCs mapped |
+| Plan | Done | {n} REMOVE, {n} KEEP, {n} MODIFY, {n} CONSOLIDATE, {n} ADD |
+| Rewrite | Done | {n} files created, {n} modified, {n} deleted |
+| Test | {Done/Skipped} | {n}/{n} passed or "not run" |
+
+### Final State
+
+| Metric | Before | After |
+|--------|--------|-------|
+| Scenarios | {n} | {n} |
+| Test Cases | {n} | {n} |
+
+### Next Steps
+
+1. {If tests not run: run `ace-test-e2e {PACKAGE} {TEST_ID}` to verify}
+2. {If tests failed: Investigate failures}
+3. Commit changes with `ace-git-commit`
+```
+
+## Example Invocations
+
+**Full lifecycle (review → plan → confirm → rewrite):**
+```bash
+ace-bundle wfi://e2e/manage
+```
+
+**Dry-run (review → plan → stop):**
+```bash
+ace-bundle wfi://e2e/manage
+```
+
+**Full lifecycle with test verification:**
+```bash
+ace-bundle wfi://e2e/manage
+```
+
+## Error Handling
+
+### No Tests Found
+
+If Stage 1 finds no E2E tests:
+```
+No E2E tests found for {package}.
+
+Use `ace-bundle wfi://e2e/create` to create the first test,
+or load `ace-bundle wfi://e2e/review` to see the unit test coverage.
+```
+
+### User Cancellation
+
+If the user declines the plan at step 3:
+1. Ask what they want to change
+2. Re-invoke Stage 2 with their feedback
+3. Present the updated plan
+4. Or exit if the user wants to stop entirely
+
+### Stage Failure
+
+If any stage fails:
+1. Report which stage failed and why
+2. Show any partial output from the failed stage
+3. Suggest running the failed stage individually for debugging

data/handbook/workflow-instructions/e2e/plan-changes.wf.md

@@ -0,0 +1,255 @@
+---
+doc-type: workflow
+title: Plan E2E Changes Workflow
+purpose: Analyze coverage matrix and produce a concrete E2E test change plan
+ace-docs:
+  last-updated: 2026-03-12
+  last-checked: 2026-03-21
+---
+
+# Plan E2E Changes Workflow
+
+This workflow takes the review output (coverage matrix) from Stage 1 and produces a concrete change plan with classified actions for each existing and proposed TC.
+
+**Pipeline position:** Stage 2 of 3 (Decide)
+
+```text
+ace-bundle wfi://e2e/review  →  ace-bundle wfi://e2e/plan-changes  →  ace-bundle wfi://e2e/rewrite
+     (explore)                           ▶ (decide) ◀                           (execute)
+```
+
+## Arguments
+
+- `PACKAGE` (required) - The package to plan changes for (e.g., `ace-lint`)
+- `--review-report <path>` (optional) - Path to review report from Stage 1. If omitted, load `ace-bundle wfi://e2e/review` first.
+- `--scope <scenario-id>` (optional) - Limit planning to a single scenario (e.g., `TS-LINT-001`)
+
+## Workflow Steps
+
+### 1. Load Review Output
+
+**If `--review-report` provided:**
+Read the file at the given path. Verify it contains a coverage matrix with the expected structure (features × unit tests × E2E columns).
+
+**If no review report:**
+Load `ace-bundle wfi://e2e/review` and capture the full output including coverage matrix, overlap analysis, gap analysis, and health status.
+
+If `--scope` is provided, filter the review data to only the specified scenario and its related features.
+
+### 2. Analyze Recent Changes
+
+Determine what has changed in the package since each TC was last verified:
+
+```bash
+# Get last-verified dates from review output, then check changes since then
+git log --oneline --since="{oldest-last-verified}" -- {PACKAGE}/lib/ {PACKAGE}/bin/
+```
+
+```bash
+# Changed files relative to current state
+git diff --name-only HEAD~20 -- {PACKAGE}/lib/ {PACKAGE}/bin/
+```
+
+Build a change inventory:
+- **New features** — files/modules added since last verification
+- **Modified features** — existing code with changes since last verification
+- **Removed features** — deleted files or deprecated modules
+- **Unchanged features** — stable code with no recent modifications
+
+### 3. Classify Each Existing TC
+
+For each TC listed in the coverage matrix, assign exactly one classification:
+
+**REMOVE** — The TC should be deleted. Criteria (any one is sufficient):
+- Full overlap with unit tests AND the TC does not test real binary/subprocess/filesystem
+- The TC tests behavior that has been removed from the package
+- The TC is a duplicate of another TC (same CLI invocation + same assertions)
+
+For REMOVE due to overlap, replacement evidence is mandatory:
+- Reference existing unit test file(s) and assertions that cover the removed behavior, OR
+- Add a follow-up unit test action to the plan (file + behavior) before removal is considered complete.
+
+**KEEP** — The TC has genuine E2E value and needs no changes. Criteria (all must be true):
+- TC passes the E2E Value Gate (tests real CLI binary + external tools + filesystem I/O)
+- Related source code has no changes since `last-verified`
+- TC structure is valid and assertions are current
+
+**MODIFY** — The TC has E2E value but needs updates. Criteria (any one is sufficient):
+- Related source code changed since `last-verified` (assertions may be outdated)
+- TC scope is too broad (should be narrowed to only E2E-exclusive aspects)
+- TC scope is too narrow (missing assertions for related behavior in same CLI invocation)
+- TC has structure issues flagged in the review
+
+**CONSOLIDATE** — The TC should merge with another TC. Criteria (any one is sufficient):
+- Multiple TCs share the same CLI invocation and could be a single TC with multiple assertions
+- A scenario has more than 5 TCs (merge related TCs to reduce count)
+- Separate TCs each check one assertion after the same setup
+
+For each classification, document:
+- The TC identifier
+- The classification reason (specific, not generic)
+- For REMOVE (overlap): replacement evidence (`existing unit tests` or `planned unit backfill`)
+- For MODIFY: what specifically needs to change
+- For CONSOLIDATE: the target TC and which assertions merge
+
+### 4. Identify New TCs Needed
+
+Review the coverage matrix for gaps that warrant new E2E tests:
+
+**Candidates for new TCs:**
+- Features with no E2E coverage that pass the E2E Value Gate
+- New features from git changes (step 2) without any test coverage
+- Error paths not covered by any existing TC
+
+**For each proposed new TC, document:**
+- Proposed title and objective
+- What CLI command it exercises
+- What it verifies that unit tests cannot
+- Which scenario it belongs to (existing or new)
+
+**Filter through Value Gate:**
+For each candidate, answer: "Does this require the full CLI binary + real external tools + real filesystem I/O?"
+- If NO: skip — unit tests cover this (or add explicit unit test action if coverage is missing)
+- If YES: include in the plan
+
+### 5. Propose Scenario Structure
+
+Group all planned TCs (KEEP + MODIFY + CONSOLIDATE targets + ADD) into scenarios:
+
+**Grouping rules:**
+- 2-5 TCs per scenario (shared setup context)
+- Group by CLI command or feature area
+- Each scenario has a clear theme (e.g., "config validation", "report generation")
+
+**For each scenario:**
+- Name: `TS-{AREA}-{NNN}-{slug}`
+- Tags: `[{cost-tier}, "use-case:{area}"]`
+- List of TCs with ordering (errors first, happy path, structure verification, lifecycle)
+- Shared setup requirements
+- Fixtures needed
+
+**Cost estimation:**
+- Each TC ≈ 1 LLM invocation when run
+- Estimate total scenarios × avg TCs × cost per invocation
+
+### 6. Present Plan to User
+
+Format the complete change plan:
+
+```markdown
+## E2E Change Plan: {package}
+
+**Generated:** {timestamp}
+**Based on:** {review-report path or "inline review"}
+**Scope:** {package-wide or scenario-id}
+
+### Impact Summary
+
+| Metric | Current | Proposed | Change |
+|--------|---------|----------|--------|
+| Scenarios | {n} | {n} | {±n} |
+| Test Cases | {n} | {n} | {±n} |
+| Est. cost/run | ~${x} | ~${x} | {-n%} |
+
+### REMOVE ({n} TCs)
+
+| TC | Reason | Replacement Evidence |
+|----|--------|----------------------|
+| {tc-id} | Unit tests in {file} cover this fully | Existing: {test-file}:{test-name} |
+| {tc-id} | Duplicate of {other-tc-id} | Existing: {test-file}:{test-name} |
+
+### Unit Coverage Backfill ({n} actions)
+
+| Action | File | Behavior |
+|--------|------|----------|
+| Add unit test | {test-file} | {behavior replacing removed E2E assertion} |
+
+### KEEP ({n} TCs)
+
+| TC | Notes |
+|----|-------|
+| {tc-id} | Genuine E2E value, no changes needed |
+
+### MODIFY ({n} TCs)
+
+| TC | Change Needed |
+|----|---------------|
+| {tc-id} | Update assertions — {feature} behavior changed in {commit} |
+| {tc-id} | Narrow scope — remove assertions covered by unit tests |
+
+### CONSOLIDATE ({n} TCs → {n} TCs)
+
+| Source TCs | Target TC | Merged Assertions |
+|------------|-----------|-------------------|
+| {tc-a}, {tc-b} | {tc-a} | Combine {n} assertions into single TC |
+
+### ADD ({n} new TCs)
+
+| Proposed TC | Scenario | Verifies |
+|-------------|----------|----------|
+| {title} | {scenario-id} | {what it tests that units cannot} |
+
+### Proposed Scenario Structure
+
+```
+TS-{AREA}-001-{slug}/  ({n} TCs)
+  TC-001: {title}
+  TC-002: {title}
+
+TS-{AREA}-002-{slug}/  ({n} TCs)
+  TC-001: {title}
+  TC-002: {title}
+  TC-003: {title}
+```
+
+### Next Steps
+
+- Review and approve this plan
+- Run `ace-bundle wfi://e2e/rewrite` to execute
+- Or modify the plan and re-run
+```
+
+**If any classifications are uncertain**, flag them with a `?` and ask the user to confirm before proceeding.
+
+## Example Invocations
+
+**Plan changes with a prior review report:**
+```bash
+ace-bundle wfi://e2e/plan-changes
+```
+
+**Plan changes (runs review automatically):**
+```bash
+ace-bundle wfi://e2e/plan-changes
+```
+
+**Plan changes for a single scenario:**
+```bash
+ace-bundle wfi://e2e/plan-changes
+```
+
+## Error Handling
+
+### No Review Data
+
+If no `--review-report` is provided and `ace-bundle wfi://e2e/review` finds no tests:
+```
+No E2E tests found for {package}. Nothing to plan changes for.
+
+To create the first E2E test: `ace-bundle wfi://e2e/create`
+```
+
+### Empty Coverage Matrix
+
+If the review shows no features or tests:
+```
+Coverage matrix is empty for {package}. Ensure the package has both
+implementation code and at least one E2E test before planning changes.
+```
+
+### User Rejection
+
+If the user rejects the plan:
+1. Ask which classifications they disagree with
+2. Adjust the plan based on feedback
+3. Re-present the updated plan