agentic-orchestrator 0.1.27 → 0.2.0

package/spec-files/outstanding/agentic_orchestrator_interactive_resume_and_reconciliation_disposition_spec.md

@@ -0,0 +1,412 @@
+# Feature Spec: Interactive Resume Recovery and Reconciliation Disposition
+
+> Purpose: Fix two critical orchestrator defects exposed by `agentic_orchestrator_runtime_inspection_spec`: (1) interactive builder runs can be incorrectly blocked as `provider_output_invalid` even when the provider returned a valid NOTE and the real failure is checkpoint/worktree validity or stale gate evidence, and (2) post-QA reconciliation collapses "no plan change needed" into a misleading `no_update` result even when execution is still blocked.
+
+**Version:** 1.0
+**Date:** 2026-03-12
+**Status:** Completed
+**Roadmap Mapping:** M49
+**Depends On:** Interactive execution checkpoints (Entry 213), gate config parity (Entry 214)
+
+---
+
+## 0. Scope and Standards
+
+### 0.1 Primary User Job
+
+**Operator/Recovery Job:** When a feature is resumed in interactive mode, the orchestrator must correctly determine whether the agent:
+
+1. made valid new worktree changes,
+2. made no new changes,
+3. is blocked by checkpoint/worktree violations,
+4. needs gates re-run against a newer worktree, or
+5. needs a plan update.
+
+The orchestrator must not mislabel those cases as malformed provider output, and its reconciliation logs must state what execution should do next.
+
+The same correctness rules must apply to both:
+
+- `resume`-driven recovery loops, and
+- explicit `retry` gate reruns initiated by the operator.
+
+### 0.2 Concrete Failure Being Fixed
+
+On **March 12, 2026**, run `run:4d7dbec7-f70a-404a-ac9b-41ccc4c178fa` repeatedly resumed feature `agentic_orchestrator_runtime_inspection_spec` in interactive mode. The feature ended each cycle as:
+
+- `status: blocked`
+- `gates.fast: fail`
+- `build decision: provider_output_invalid`
+- `post_qa_reconciliation decision: no_update`
+
+even though the existing worktree already contained the intended implementation and manual verification succeeded outside the orchestrator.
+
+Observed failure chain:
+
+1. Builder completed with `output_types:["NOTE"]` in interactive mode.
+2. Worker/build logic marked the turn invalid and blocked the feature as `provider_output_invalid`.
+3. No fresh fast gate ran for the current worktree.
+4. Planner reconciliation ran against stale failed gate evidence and logged `no_update`.
+5. Logs therefore implied the planner was done while the feature remained operationally stuck.
+
+### 0.3 Quality Standards
+
+- TypeScript strict mode passes
+- ESLint zero warnings
+- Coverage >= 90% for all changed files
+- `npm run verify` passes
+- Existing state-machine invariants remain unchanged: no illegal status transitions
+- New logging semantics must be machine-readable and human-debuggable
+
+---
+
+## 1. Defects
+
+### 1.1 Defect A: Interactive Completion Is Misclassified
+
+Today, interactive provider output is parsed permissively:
+
+- non-JSON or NOTE-only output can be converted into a valid `NOTE`
+- direct worktree edits are expected to be validated via checkpoints
+
+However, the later decision pipeline still collapses several distinct failure modes into `provider_output_invalid`:
+
+- malformed provider JSON
+- invalid checkpoint validation
+- plan-scope/worktree violations
+- NOTE-only completion with no net-new work
+
+This is wrong because only the first case is truly malformed output.
+
+### 1.2 Defect B: Reconciliation Overloads `no_update`
+
+Today, post-QA reconciliation answers only:
+
+- "Does the accepted plan need to change?"
+
+It does not answer:
+
+- "What should execution do next?"
+
+As a result, `no_update` currently means both:
+
+- "plan is aligned and feature may be converged"
+- "plan is aligned, but execution is still blocked by stale evidence / invalid checkpoint / provider failure"
+
+This ambiguity is operationally unacceptable.
+
+---
+
+## 2. Goals
+
+### 2.1 Must Have
+
+1. Distinguish malformed provider output from invalid interactive checkpoint/worktree state.
+2. Allow interactive NOTE-only completions to count as valid progress when backed by a valid checkpoint with net-new work.
+3. Detect stale gate evidence relative to the current validated worktree and rerun required gates instead of forcing needless builder loops.
+4. Change reconciliation outputs so they separately express:
+   - plan decision
+   - execution disposition
+5. Ensure logs and dashboard artifacts make the real blocker obvious.
+6. Scope freshness to the required gate mode (`fast` vs `full` vs `merge`) instead of whichever evidence artifact was most recently written.
+
+### 2.2 Non-Goals
+
+1. Do not add new top-level feature statuses beyond the current state machine.
+2. Do not trust user-run external commands such as manual `npm run verify` as authoritative orchestrator evidence unless ingested by orchestrator-owned artifacts.
+3. Do not redesign planner/build/QA wave ordering.
+
+---
+
+## 3. Required Behavior Changes
+
+### 3.1 Interactive Outcome Classification
+
+Interactive worker completion must be classified into one of the following outcome buckets:
+
+1. **`progress_applied`**
+   - provider output is syntactically valid for interactive mode (`NOTE` or `REQUEST`)
+   - latest checkpoint is valid
+   - checkpoint reports `net_new_worktree_change === true`
+
+2. **`no_new_work`**
+   - provider output is syntactically valid for interactive mode
+   - latest checkpoint is valid
+   - checkpoint reports `net_new_worktree_change === false`
+   - no request/plan action was emitted
+
+3. **`checkpoint_invalid`**
+   - provider output is syntactically valid for interactive mode
+   - latest checkpoint is invalid due to diff validation or allowed-area / planned-file violations
+
+4. **`provider_output_invalid`**
+   - provider output cannot be parsed into any allowed output form for the current execution mode
+
+These outcomes must not be collapsed together.
+
+### 3.2 New Error/Reason Semantics
+
+The system must introduce explicit machine-readable reasons for interactive failures. Exact error-code naming may vary, but semantics must be distinct:
+
+- `interactive_checkpoint_invalid`
+- `interactive_no_new_work`
+- `stale_gate_evidence`
+- `provider_output_invalid`
+
+`provider_output_invalid` must only be used for malformed provider output, never for checkpoint validation failures.
+
+### 3.3 Gate Evidence Freshness
+
+Gate evidence must be tied to the validated worktree it was produced from.
+
+Each gate artifact (`latest-fast.json`, `latest-full.json`, `latest-merge.json`, evidence history) must record:
+
+- `input_diff_hash`
+- `input_checkpoint_id` when available
+- `input_worktree_validity` (`valid` | `invalid` | `unknown`)
+- `finished_at`
+
+The orchestrator must compare the current validated worktree/checkpoint against the latest evidence for the required gate mode, not merely the most recently written evidence file overall.
+
+Freshness rules:
+
+1. A gate artifact is **current** only when its `input_diff_hash` matches the current validated worktree diff hash for the gate mode being evaluated.
+2. A gate artifact is **stale** when the current validated worktree diff hash differs from the artifact's `input_diff_hash`, regardless of whether the stored result is `pass` or `fail`.
+3. A stale passing artifact must never be treated as sufficient to advance or converge the feature.
+
+If all of the following are true:
+
+1. current worktree/checkpoint is valid,
+2. current diff hash differs from the latest gate artifact's `input_diff_hash`,
+3. latest gate artifact is stale or missing,
+
+then the feature is in **gate-refresh-needed** state, not **builder-needs-to-edit** state.
+
+In that case, resume/recovery must rerun the required gate (`fast` from build, `full` from QA, `merge` before merge) before asking the builder or QA worker to perform another edit pass or before treating the feature as converged.
+
+### 3.4 Worker-Wave Semantics for Interactive NOTE Completions
+
+When builder returns `NOTE` in interactive mode:
+
+- if outcome is `progress_applied`, proceed to fast gate execution in the same cycle
+- if outcome is `checkpoint_invalid`, block feature with the checkpoint-specific reason and record violations
+- if outcome is `no_new_work` but gate evidence is stale relative to the current valid worktree, rerun the fast gate instead of incrementing no-progress
+- if outcome is `no_new_work` and gate evidence is current, treat as true no-progress
+
+When QA returns `NOTE` in interactive mode:
+
+- if outcome is `progress_applied`, proceed to full gate execution in the same cycle
+- if outcome is `checkpoint_invalid`, block feature with the checkpoint-specific reason and record violations
+- if outcome is `no_new_work` but full-gate evidence is stale relative to the current valid worktree, rerun the full gate instead of incrementing no-progress
+- if outcome is `no_new_work` and full-gate evidence is current, treat as true no-progress
+
+This prevents endless NOTE-only loops when the worktree is already ready for verification.
+
+### 3.5 Resume / Recovery Semantics
+
+Blocked-feature recovery must consider three independent dimensions:
+
+1. latest gate result
+2. latest checkpoint validity
+3. gate-evidence freshness relative to current worktree
+
+Recovery behavior:
+
+- **Stale failed gate + newer valid worktree**: recover into gate rerun path
+- **Stale passed gate + newer valid worktree**: recover into gate rerun path, not advance on stale evidence
+- **Invalid checkpoint/worktree**: remain blocked with checkpoint-specific reason
+- **Current failing gate on current worktree**: remain blocked or gate-repair retry per existing policy
+- **Passing gate evidence on current worktree**: advance normally
+
+The same freshness-aware routing must apply to `retry` command handling:
+
+- `retry` must rerun the requested gate against the current worktree
+- after retry, failed reruns must not blindly reroute the feature back to `building`/`qa` if the real blocker is checkpoint invalidity or stale evidence already resolved into a gate rerun
+- `retry` responses must reflect whether the outcome was a gate rerun, checkpoint block, or true worker-retry condition
+
+### 3.6 Reconciliation Disposition Model
+
+Post-QA reconciliation must return two explicit dimensions:
+
+```ts
+{
+  plan_decision: 'update_required' | 'unchanged';
+  execution_disposition:
+    | 'converged'
+    | 'retry_build'
+    | 'retry_qa'
+    | 'rerun_fast_gate'
+    | 'rerun_full_gate'
+    | 'blocked_checkpoint_invalid'
+    | 'blocked_provider_failure'
+    | 'blocked_stale_evidence'
+    | 'blocked_other';
+  reasons: string[];
+}
+```
+
+Rules:
+
+1. `plan_decision: unchanged` must not be logged as plain `no_update` without execution disposition.
+2. `execution_disposition: converged` is only valid when the feature is not blocked and required gate evidence is current.
+3. If feature status is `blocked`, reconciliation must never imply success by omission.
+4. If status is `blocked` and latest evidence is stale, reconciliation must say so explicitly.
+
+### 3.7 Decision Log Contract
+
+`decisions.md` reconciliation entries must include:
+
+- `plan_decision`
+- `execution_disposition`
+- `status`
+- `latest_gate_overall`
+- `gate_evidence_freshness`
+- `checkpoint_validity`
+- `reasons`
+
+Example:
+
+```json
+{
+  "phase": "post_qa_reconciliation",
+  "iteration": 3,
+  "plan_decision": "unchanged",
+  "execution_disposition": "blocked_checkpoint_invalid",
+  "status": "blocked",
+  "latest_gate_overall": "fail",
+  "gate_evidence_freshness": "stale",
+  "checkpoint_validity": "invalid",
+  "reasons": ["checkpoint_outside_allowed_areas", "fast_gate_evidence_stale"]
+}
+```
+
+The legacy `decision: "no_update"` shape must be retired for reconciliation entries.
+
+---
+
+## 4. Required Implementation Areas
+
+Minimum expected files:
+
+- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
+- `apps/control-plane/src/supervisor/build-wave-executor.ts`
+- `apps/control-plane/src/supervisor/qa-wave-executor.ts`
+- `apps/control-plane/src/supervisor/planning-wave-executor.ts`
+- `apps/control-plane/src/supervisor/run-coordinator.ts`
+- `apps/control-plane/src/application/services/checkpoint-service.ts`
+- `apps/control-plane/src/application/services/worktree-watchdog-service.ts`
+- `apps/control-plane/src/core/error-codes.ts`
+- `apps/control-plane/src/core/tool-caller.ts`
+- `apps/control-plane/src/core/gates.ts`
+- `apps/control-plane/src/application/services/gate-service.ts`
+- `apps/control-plane/src/cli/retry-command-handler.ts`
+- `agentic/orchestrator/schemas/state.schema.json`
+- `config/agentic/orchestrator/gates.yaml` if gate metadata schema needs extension
+- tests covering the above
+
+Optional but recommended:
+
+- runtime inspector/dashboard surfaces that expose checkpoint validity and gate-evidence freshness
+
+---
+
+## 5. Acceptance Criteria
+
+### 5.1 Interactive Builder Classification
+
+1. A valid interactive `NOTE` plus valid net-new checkpoint is not marked `provider_output_invalid`.
+2. A valid interactive `NOTE` plus invalid checkpoint is blocked with checkpoint-specific reason, and violations are preserved.
+3. Malformed provider output still yields `provider_output_invalid`.
+
+### 5.2 Gate Freshness
+
+1. Resume on a feature with stale failed fast-gate evidence and a newer valid worktree reruns `fast` without requiring another builder edit.
+2. Resume on a feature with stale failed full-gate evidence and a newer valid worktree reruns `full` without requiring another QA edit.
+3. Stale passing gate evidence on a newer worktree does not count as current evidence and forces the required gate rerun before advance/convergence.
+4. Gate freshness checks are mode-scoped; a newer `full` artifact does not satisfy `fast`, and vice versa.
+5. Latest gate artifacts record the diff/checkpoint identity they validated.
+
+### 5.3 Reconciliation
+
+1. Reconciliation log entries always include both `plan_decision` and `execution_disposition`.
+2. Reconciliation never reports convergence while feature status is still `blocked`.
+3. Reconciliation never reports convergence when the required gate evidence is stale, even if the stale result was `pass`.
+4. Reconciliation on blocked features can explicitly say `plan unchanged, execution blocked`.
+
+### 5.4 End-to-End Recovery
+
+Given a feature whose worktree already contains complete, valid planned changes and whose old fast-gate artifact is stale:
+
+1. `resume --execution-mode interactive` does not enter an endless NOTE-only build block loop.
+2. The orchestrator reruns the required gate against the current worktree.
+3. If the gate passes, the feature advances to the next phase without planner plan churn.
+4. Running `retry` on the same stale-evidence scenario does not bounce the feature back into a builder/QA retry loop.
+
+---
+
+## 6. Test Plan
+
+### 6.1 Unit Tests
+
+- `worker-decision-loop.spec.ts`
+  - NOTE + valid checkpoint + net-new work => progress true, invalid false
+  - NOTE + invalid checkpoint => checkpoint-specific invalid outcome
+  - malformed output => provider output invalid
+
+- `build-wave-executor.spec.ts` or existing supervisor unit coverage
+  - stale gate evidence + valid current worktree => rerun fast gate
+  - checkpoint-invalid outcome => blocked with checkpoint-specific reason
+  - NOTE-only no-new-work + current evidence => true no-progress
+
+- `qa-wave-executor.spec.ts` or existing supervisor unit coverage
+  - stale full-gate evidence + valid current worktree => rerun full gate
+  - checkpoint-invalid QA outcome => blocked with checkpoint-specific reason
+
+- `planning-wave-executor.spec.ts`
+  - blocked + unchanged plan + stale evidence => `plan_decision=unchanged`, `execution_disposition=blocked_stale_evidence`
+  - blocked + unchanged plan + invalid checkpoint => `blocked_checkpoint_invalid`
+  - ready-to-merge case => `converged`
+
+- `run-coordinator.spec.ts`
+  - recovery selects gate rerun when evidence is stale and worktree is newer
+  - recovery does not recycle blocked features into builder when only gate refresh is needed
+
+- `gates.spec.ts`
+  - gate evidence records input diff/checkpoint metadata
+  - freshness evaluation uses the requested gate mode, not the globally latest evidence artifact
+
+- `retry-command-handler.spec.ts`
+  - retry on stale evidence reruns the required gate without forcing worker-loop reroute
+  - retry on checkpoint-invalid state preserves checkpoint-specific blocked reason
+
+### 6.2 Integration Tests
+
+Add at least one supervisor integration test that reproduces this exact defect class:
+
+1. feature starts in `blocked`
+2. current worktree contains already-complete valid changes
+3. latest fast evidence is stale and failed
+4. builder emits NOTE only
+5. resume reruns fast gate and advances instead of blocking on `provider_output_invalid`
+
+---
+
+## 7. Risks and Constraints
+
+1. The fix must preserve strict status-transition invariants.
+2. The fix must not silently bless out-of-plan worktree changes.
+3. The fix must not trust manual shell verification unless orchestrator-generated evidence is updated.
+4. Backward compatibility may require tolerating old decision-log entries while writing the new format going forward.
+
+---
+
+## 8. Definition of Done
+
+- New interactive outcome taxonomy implemented and tested
+- Gate-evidence freshness metadata implemented and tested
+- Resume/recovery uses freshness-aware gate reruns
+- Reconciliation logs `plan_decision` + `execution_disposition`
+- `npm run typecheck` passes
+- `npm run lint` passes
+- `npm test` passes at coverage thresholds
+- `npm run verify` passes
+- `spec-files/progress.md` updated with implementation entry