@mmerterden/multi-agent-pipeline 20.2.1 → 20.3.0

package/pipeline/multi-agent-refs/features/unattended-gates.md

@@ -0,0 +1,316 @@
+# Feature: Unattended Verification Gates
+
+<!-- toc -->
+- [When the gates run, and what the environment adds](#when-the-gates-run-and-what-the-environment-adds)
+- [The gate ledger](#the-gate-ledger)
+- [Phase 1 -> Phase 2: open questions](#phase-1---phase-2-open-questions)
+- [Spec consistency before development](#spec-consistency-before-development)
+- [Plan critique before development](#plan-critique-before-development)
+- [Phase 2 exit: what "it passed" has to show](#phase-2-exit-what-it-passed-has-to-show)
+- [Phase 3: triage citations and pre-existing claims](#phase-3-triage-citations-and-pre-existing-claims)
+- [Symbol existence in detail](#symbol-existence-in-detail)
+- [Phase 4 (Commit): plan coverage](#phase-4-commit-plan-coverage)
+- [Limits](#limits)
+- [Reference](#reference)
+<!-- /toc -->
+
+**Pattern**: a run nobody is asked about has nobody to notice an invented
+localization key, a question it answered for itself, a test that passes without
+the change, or a finding waved off as "pre-existing". These gates check those
+claims against the repository, the base commit and the run's own record, and
+park the run when a claim does not hold. A normal interactive run is unchanged.
+
+## When the gates run, and what the environment adds
+
+Two questions, answered by `lib/unattended.mjs` and its shell twin
+`lib/unattended.sh`, and kept apart:
+
+| Question | Answer | Decides |
+|---|---|---|
+| Do the quality gates run? `gatesActive(state, env)` / `ma_gates_active <state>` | `MULTI_AGENT_UNATTENDED=1`, or `state.autopilot === true` | every gate below, their ledger entries, the Section 14 fallback |
+| Is anybody watching? `isUnattended(env)` / `ma_unattended` | `MULTI_AGENT_UNATTENDED=1` only, set by the launcher (the autopilot runner passes it to its child) | security enforcement: the commit hook's ledger requirement, and agent-guard's push / PR / install / protected-path blocking (`features/unattended-security.md`) |
+
+A terminal `/multi-agent:autopilot` run has no variable and `autopilot: true`:
+the gates run, the commit hook does not demand the ledger, and
+`phase0-exit-gate.mjs` accepts its autopilot resolutions as it always did. A run
+with the variable set cannot switch the gates off by writing
+`autopilot: false`. The security side never reads the state: the run writes it,
+and the hook's environment is the launcher's.
+
+With neither, no gate writes a ledger entry, touches the state or parks the
+run, and `state.gates[]` stays absent. What each one prints:
+
+| Gate | Attended |
+|---|---|
+| `spec-consistency-gate.mjs`, `plan-critique-gate.mjs`, `review-decision-gate.mjs` | the report, marked advisory; exit 0 (a usage error exits 3, or 2 for `review-decision-gate.mjs`) |
+| `open-questions-gate.mjs`, `symbol-existence-gate.mjs` | a not-applicable verdict; exit 0 |
+| `evidence-gate.mjs`, `verify-citations.mjs`, `plan-coverage-gate.mjs` | the attended verdict and exit code they always had (`test/attended-parity.test.mjs` holds them to it) |
+| `test-summary.mjs`, `test-strength.mjs` | their verdict and exit code; neither is called by an attended phase |
+
+`smoke-gates-interactive-noop.sh` holds that, and the terminal-autopilot case
+beside it.
+
+## The gate ledger
+
+Every gate that runs appends one entry per verdict to
+`state.gates[]` through `scripts/gate-ledger.mjs`, under the `write-state.mjs`
+lock:
+
+```json
+{ "gate": "test-summary", "verdict": "pass|fail|not-applicable",
+  "at": "<ISO time>", "sha": "<HEAD>", "detail": { } }
+```
+
+The state is found from `--state`, else the run whose `sessionId` is
+`MULTI_AGENT_SESSION_ID`, else the run whose `worktreePath` is the repository.
+`sha` is the worktree's HEAD. The pipeline commits only in Phase 4, so it is the
+base the run started from.
+
+Mandatory at commit: `open-questions`, `evidence-gate/build`,
+`evidence-gate/test`, `test-summary`, `test-strength`, `symbol-existence`,
+`verify-citations`, `review-decision`, `plan-coverage`. `pre-commit-check.sh`, as the PreToolUse
+hook on `git commit`, runs `gate-ledger.mjs check` only when the variable is set
+in the hook's own environment: the latest entry of each mandatory gate for HEAD
+must be `pass` or `not-applicable`. A missing entry blocks the commit like a
+failing one. `spec-consistency` is recorded but not mandatory: a run whose item carries no
+requirement id has nothing for it to check (`features/constitution.md`). Nor is
+`plan-critique`: the resume tail commits with no Plan phase (`features/plan-critic.md`). No run state found also blocks. A direct run of the script (Phase 2
+Gate 4) is the secret scan only.
+
+## Phase 1 -> Phase 2: open questions
+
+Autopilot defers every unanswered Section 20 row into
+`state.analysis.openQuestions[]`. As the last step of Phase 1 (Step 12, after
+the plan critique), whatever `state.analysis.docStatus` says, so a bugfix with
+no analysis document runs it too:
+
+```bash
+node "$HOME/.claude/scripts/open-questions-gate.mjs" --state "$STATE_FILE"
+```
+
+| Exit | Verdict in the ledger | The run |
+|---|---|---|
+| 0 | `pass`: the list is empty; `not-applicable`: no list recorded (no analysis in this run) | continues |
+| 1 | `fail`: rows are open, or the answer was `abort` | parked by the gate itself; Phase 2 does not start, nothing is retried |
+| 2 | nothing | usage error; fix the call |
+| 3 | nothing: no run state found | Phase 2 does not start; the run halts with the gate's message |
+
+The gate runs unconditionally because it is in `MANDATORY_AT_COMMIT`: every
+gated path has to leave an entry for HEAD, and a list that was never recorded
+is `not-applicable`, which the commit check accepts. The pipeline tail
+(`/multi-agent:resume` over branch work with no run behind it) has no Phase 1,
+so it runs the same call in its Phase 0 and records `not-applicable`.
+
+Open rows park the run: the gate writes `waitingFor: "question"`,
+`status: "awaiting_input"`, a `haltReason` naming the count and a
+`pendingQuestion` with `stepId: "phase-1/open-questions"`. The runner reports
+it as `awaiting-answer` and never re-runs it on its own.
+
+The question is answered like any other parked question (`resume <id> --answer
+<optionId>`, or a client through answer-question.mjs). Its stepId,
+`phase-1/open-questions`, is this gate: resume re-enters at the end of Phase 1,
+the plan already made, and re-runs `open-questions-gate.mjs` before anything
+else. The gate reads `state.lastAnswer` (that stepId, answered after its latest
+`fail` entry) and applies it, so the ledger's latest entry is the answered one
+and not the park:
+
+| Option | The gate | The run then |
+|---|---|---|
+| `ask-now` | with `MULTI_AGENT_UNATTENDED=1`: parks again on the same question (`fail`, exit 1), since asking needs a person in the session. Without it: evaluates the list as usual | asks each row in the session, as an attended run would, empties `openQuestions[]`, re-runs the gate (`pass`), revises the plan where an answer changes it, and enters Phase 2 |
+| `assume` | moves every row to `state.analysis.assumptions[]`, empties `openQuestions[]`, clears `awaiting_input`: `pass`, exit 0 | writes each assumption into the analysis doc's limitations and enters Phase 2 |
+| `abort` | `fail`, and parks the run as `verification-failed --gate open-questions`: exit 1 | stops; nothing is developed |
+
+The re-run is the step's and not `answer-question.mjs`'s: the answer may come
+from a client through the contract server, whose environment is not the
+run's, and `answer-result.schema.json` fixes what an answer returns.
+
+Before a parked run is left for a person, the autopilot runner gives it a
+research pass (`features/research.md`): `research-gate.mjs` closes the rows a
+verified source answers, re-runs this gate on the rest and records a new
+`open-questions` entry. Only the rows it leaves open are asked. The same holds
+for a maturity blocker at Phase 0.
+
+## Spec consistency before development
+
+After Phase 1 Step 12, beside the open-questions gate:
+
+```bash
+node "$HOME/.claude/scripts/spec-consistency-gate.mjs" --state "$STATE_FILE" --plan "$WORKTREE/.pipeline/plan.json"
+```
+
+| Exit | Verdict in the ledger | The run |
+|---|---|---|
+| 0 | `pass`: every requirement has a task and a test, every task a requirement, no binding constitution rule waived | continues |
+| 1 | `fail` | revises the plan once and re-runs; still 1, parks with `park --outcome verification-failed --gate spec-consistency` |
+| 2 | `not-applicable`: no requirement id in the analysis | continues |
+| 3 | `fail`: a document, plan or constitution unreadable | parks the same way |
+
+`/multi-agent:analysis-jira` runs the same check inside `analysis-story-tree.mjs`,
+where a gap is exit 4 and no issue is created. The checks, the constitution and
+what an attended run sees: `features/constitution.md`.
+
+## Plan critique before development
+
+After spec consistency and before the open-questions gate that closes Phase 1: one critic, one round
+(`features/plan-critic.md`). The `plan-critic` persona writes objections to
+`$WORKTREE/.pipeline/plan-critique.json`, the planner answers each once, then:
+
+```bash
+node "$HOME/.claude/scripts/plan-critique-gate.mjs" --state "$STATE_FILE" \
+  --critique "$WORKTREE/.pipeline/plan-critique.json" --plan "$WORKTREE/.pipeline/plan.json"
+```
+
+| Exit | Verdict in the ledger | The run |
+|---|---|---|
+| 0 | `pass`: no binding constitution objection left unresolved, one reply per objection | continues; advisory objections go to the plan and the PR summary |
+| 1 | `fail`: a binding objection unresolved, a reply missing or doubled, or a second round | parked by the gate as `verification-failed --gate plan-critique`; not retried |
+| 2 | `not-applicable`: no plan | continues |
+| 3 | `fail`: an input unreadable or the critique not in schema shape | parked the same way |
+
+## Phase 2 exit: what "it passed" has to show
+
+In addition to the attended exit gate:
+
+```bash
+S="$HOME/.claude/scripts"
+STACK=$(node "$S/_stack-adapter.mjs" resolve --dir "$WORKTREE" | cut -d' ' -f1)
+node "$S/evidence-gate.mjs" --claim build --status passed --evidence "$WORKTREE/.build.log" --stack "$STACK"
+node "$S/evidence-gate.mjs" --claim test  --status passed --evidence "$WORKTREE/.test.log"  --stack "$STACK"
+node "$S/test-summary.mjs" --log "$WORKTREE/.test.log" --stack "$STACK"
+node "$S/test-strength.mjs" --repo "$WORKTREE" --base HEAD --head worktree --stack "$STACK"
+node "$S/symbol-existence-gate.mjs" --repo "$WORKTREE" --base HEAD --stack "$STACK"
+```
+
+| Gate | Verdict in the ledger |
+|---|---|
+| `evidence-gate --stack` | the adapter's markers and test counts; a status that is not `passed` is recorded as a failure. A test claim needs a positive executed-test count, from the log or from `--junit <dir>` / `--xcresult <bundle>`: markers alone fail. Gradle prints no counts on success and `xcodebuild -quiet` only its verdict, so pass the adapter's `resultFiles` directory of this run (`build/test-results`, `target/surefire-reports`) or the `-resultBundlePath` bundle; wipe the directory before the run so an old result cannot be counted. An adapter's `skipped` markers (Maven `Tests are skipped.`, a runner's "no test files") and a Gradle log whose every test task is `SKIPPED` or `NO-SOURCE` count as zero tests |
+| `test-summary` | `passed` pass; `failed` and `no-tests` fail; `unparsed` not-applicable (the markers still decide) |
+| `test-strength --head worktree` | every changed test red: pass. Any still-green, or a test failing on head: fail. No test file or no production change: not-applicable. A test that goes red only by failing to compile or load without the change is `unjudged`, never red: red plus unjudged is a pass, unjudged alone not-applicable. The uncommitted tree is snapshotted through a private index, so the checkout is not touched |
+| `symbol-existence-gate` | every key, resource and endpoint the added lines reference exists in the adapter's truth sources (section 6); nothing to check against: not-applicable |
+
+A failure here gets the attended rework budget. `test-summary` exit 3 (zero tests
+executed) is not a rework case: the run parks instead of iterating.
+
+```bash
+node "$S/gate-ledger.mjs" park --outcome verification-failed --gate test-summary --reason "no tests executed"
+```
+
+The runner reports `state.verificationFailed` as `verification-failed`: parked,
+not retried, because a retry repeats the same work into the same rejection. The
+same applies to any gate still failing when the rework budget is spent.
+
+## Phase 3: triage citations and pre-existing claims
+
+After triage writes its output (including the Step 3.1 empty result):
+
+```bash
+node "$S/verify-citations.mjs" "$TRIAGE_FILE" --repo "$WORKTREE" --worktree --state "$STATE_FILE"
+```
+
+- Every bucket is resolved: `accepted[]` and the `finding` inside each
+  `deferred[]` / `rejected[]` item.
+- A citation carrying `quote` must match: whitespace collapsed, the quote is in
+  the cited line (a multi-line quote in the lines from there), with at least 8
+  non-space characters unless it is the whole line.
+- Nothing to check (no citation, no pre-existing claim) is `not-applicable`,
+  not `pass`.
+- An attended run resolves `file:line` only, as it always did: flat findings,
+  quotes not read.
+- Gates active: every finding marked `preExisting: true`, or deferred / rejected
+  with a reason that says pre-existing, needs `baselineEvidence.sha` naming the
+  base commit or an ancestor of it (`--base`, else `state.baseline.sha`, else
+  the merge-base with `baseBranch`, else HEAD, which is the base until Phase 4
+  commits). A code finding must cite a line, the line must not be one the
+  diff from that commit adds or changes (the same text elsewhere in the base
+  file does not count), and it must not be blank or punctuation only. A
+  failing test (`baselineEvidence.testId`) must be in
+  `state.baseline.tests.failing`, or `baselineEvidence.logPath` must be a log
+  inside the checkout or the run's state directory that names the test and
+  shows a failure by the stack adapter's markers.
+
+Exit 1: re-run triage once with the rejected list quoted. A claim still
+unbacked moves the finding to `accepted[]`; it is this run's to fix.
+
+Then `review-decision-gate.mjs` on the same file: a blocker stays blocking
+only when two independent reviewers raised it or a failing verify-by-test log
+backs it, otherwise it is lowered to important with a reason, and a rebuttal
+round is refused (`features/review-decision.md`).
+
+## Symbol existence in detail
+
+- Only added lines are read (base..working tree, untracked files included, or
+  base..head with `--head`). An old reference is not this change's claim.
+- References come from the adapter's `referencePatterns` (`(?<ref>...)`); truth
+  sources from its `truthSources` for the same kind (`localization`,
+  `endpoints`), read at the head, so a key added to strings.xml in the same
+  change exists.
+- A file that is itself a truth source holds declarations and is skipped; so is
+  a test file.
+- Reading by format: `.xcstrings` / i18n JSON keys (nested keys by their full
+  dotted path only: `title` under `login` is `login.title`, never `title`),
+  `strings.xml` `name=`, `.strings`, `.stringsdict`, `.properties`, `.po`,
+  `.ftl`, YAML/TOML keys, OpenAPI `paths`; any other file (a generated Swift key
+  enum, an endpoints file) contributes its declared identifiers and string
+  literals. A dotted reference such as `L10n.Profile.header` exists when every
+  segment is an identifier of such a file (or of a YAML/TOML source, read
+  without its nesting); segments that exist only as JSON keys do not combine
+  into an invented path.
+- Endpoints compare with parameters normalised (`{id}`, `:id`, `${id}`,
+  `<id>`), and a path that ends the other at a segment boundary matches (base
+  URL on one side, mount prefix on the other).
+- No truth-source file for any kind in the repository: not-applicable, never a
+  failure.
+
+## Phase 4 (Commit): plan coverage
+
+With the gates active, Phase 1 always writes plan todos (Step 11 runs whatever
+`prefs.global.planTodos.enabled` says), so a gated run has a step list to
+audit. `plan-coverage-gate.mjs` then exits:
+
+| Exit | Verdict | When |
+|---|---|---|
+| 0 | `pass` | every step accounted for, every Section 14 `Add new` file present |
+| 1 | `fail` | a gap, a zero-step plan, or a run that had a Plan phase (`analysis.docStatus` recorded, or `phases["1"]` entered) with no todos and no Section 14 `Add new` row to fall back on |
+| 2 | `not-applicable` | no Plan phase: the pipeline tail |
+| 3 | `fail` | unreadable input |
+
+With no todos it first falls back to the analysis Section 14 `Add new` rows
+and exits 0 or 1 on those. A planned run with neither source is a gap and not
+a not-applicable: it planned work and left nothing to check it against.
+
+Attended, the exit contract is the one it always had: 0 clean, 1 gap, 2 for
+everything else (no todos, verdict `not-applicable`; a zero-step plan, verdict
+`unusable`; usage or unreadable input, no verdict). Phase 4 runs it with
+`--json` and reads only exit 2 with verdict `not-applicable` as
+not-applicable, so a zero-step plan or unreadable input is a gap on either
+path.
+
+An unattended gap is not taken to a person (a terminal autopilot run shows it
+and asks, as attended): it goes back to Phase 2 once, then parks as
+`verification-failed`.
+
+## Limits
+
+- The in-run gates are invoked by the agent and are advisory against an agent
+  that skips them. The commit hook is what holds: its environment is the
+  launcher's, and a skipped gate is a missing ledger entry.
+- `sha` binds an entry to HEAD, not to the working tree. A gate that passed
+  before a later edit still counts until it is re-run; the latest entry wins,
+  so re-running after every rework is what keeps the ledger current.
+
+## Reference
+
+Scripts: `gate-ledger.mjs`, `symbol-existence-gate.mjs`,
+`open-questions-gate.mjs`, `research-gate.mjs`, `spec-consistency-gate.mjs`,
+`plan-critique-gate.mjs`, `review-decision-gate.mjs`,
+`_pre-existing.mjs`, and the unattended paths of
+`evidence-gate.mjs`, `test-summary.mjs`, `test-strength.mjs`,
+`verify-citations.mjs`, `plan-coverage-gate.mjs`, `phase0-exit-gate.mjs`,
+`pre-commit-check.sh`. Tests: `test/gate-ledger.test.mjs`,
+`test/symbol-existence-gate.test.mjs`, `test/open-questions-gate.test.mjs`,
+`test/verify-citations-triage.test.mjs`, `test/plan-coverage-gate.test.mjs`,
+`test/pre-commit-ledger.test.mjs`, `test/unattended-ledger-gates.test.mjs`,
+`test/unattended.test.mjs`, `test/research-gate.test.mjs`,
+`test/spec-consistency-gate.test.mjs`, `test/plan-critique-gate.test.mjs`,
+`test/review-decision-gate.test.mjs`; smoke
+`smoke-gates-interactive-noop.sh`.