@ai-dossier/sched 0.19.0 → 0.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +42 -6
- package/dist/batch-dispatch.d.ts +45 -20
- package/dist/batch-dispatch.d.ts.map +1 -1
- package/dist/batch-dispatch.js +262 -42
- package/dist/batch-dispatch.js.map +1 -1
- package/dist/dispatch.d.ts +24 -0
- package/dist/dispatch.d.ts.map +1 -1
- package/dist/dispatch.js +58 -5
- package/dist/dispatch.js.map +1 -1
- package/dist/engine.d.ts.map +1 -1
- package/dist/engine.js +31 -10
- package/dist/engine.js.map +1 -1
- package/dist/index.d.ts +3 -3
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +6 -4
- package/dist/index.js.map +1 -1
- package/dist/persist.js +12 -1
- package/dist/persist.js.map +1 -1
- package/dist/recovery.d.ts.map +1 -1
- package/dist/recovery.js +4 -1
- package/dist/recovery.js.map +1 -1
- package/dist/state.d.ts.map +1 -1
- package/dist/state.js +18 -2
- package/dist/state.js.map +1 -1
- package/dist/status.d.ts.map +1 -1
- package/dist/status.js +17 -0
- package/dist/status.js.map +1 -1
- package/dist/types.d.ts +70 -2
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js +3 -1
- package/dist/types.js.map +1 -1
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -82,7 +82,15 @@ where every mechanical supervision decision is code, not remembered prose:
|
|
|
82
82
|
"waiting for it to finish" abandons the run with the subprocess still going; the
|
|
83
83
|
instruction tells it to run such commands in the foreground and wait, or poll until
|
|
84
84
|
they finish. `DEFAULT_REPORT_PROMPT_TEMPLATE` is excluded — it never spawns a long
|
|
85
|
-
command.
|
|
85
|
+
command. The prompt instruction alone was not enough (#591): agents kept arming the
|
|
86
|
+
`Monitor` tool to wait on a background command and ending their turn anyway, which the
|
|
87
|
+
engine can only see as an unverified exit. Every `claude`-family command template
|
|
88
|
+
(top-level `command` and each tier's own `commandTemplate`, #527) gets
|
|
89
|
+
`--disallowedTools Monitor` appended automatically — set `dispatch.disallowed_tools: []`
|
|
90
|
+
in `config.json` to opt out, or list your own tools to deny instead of the default
|
|
91
|
+
`["Monitor"]`. Matched on the binary's basename, so an absolute or wrapper path
|
|
92
|
+
(`/usr/local/bin/claude`) still gets it; never applied to a non-`claude` command or one
|
|
93
|
+
that already carries the flag itself, so an `opencode` tier is unaffected.
|
|
86
94
|
2. **Completion verification (AC2)** — an agent exiting is never proof of completion.
|
|
87
95
|
On exit, the unit completes only when ground truth confirms it: the issue's latest
|
|
88
96
|
runstate milestone is `report done`, or GitHub says the issue is closed — except a
|
|
@@ -455,6 +463,8 @@ ready → executing(member i/N) ⟲ → validating → reviewing → shipping
|
|
|
455
463
|
failure rails: executing → dissolving (a member self-reports blocked)
|
|
456
464
|
validating → attributing → (fixing | evicting) → validating → dissolving
|
|
457
465
|
validating → blocked (suite report unreadable, #562) → validating
|
|
466
|
+
executing → blocked (gate-inconclusive:<cap>, #583) → executing
|
|
467
|
+
(`sched resume --batch <id>` re-runs the gate; nothing requeued/reverted)
|
|
458
468
|
```
|
|
459
469
|
|
|
460
470
|
- **One shared worktree/branch per batch**, claimed once by a deterministic (no LLM)
|
|
@@ -473,7 +483,12 @@ failure rails: executing → dissolving (a member self-reports blocked)
|
|
|
473
483
|
kept on `BatchEntry.ranges` for eviction. An incremental gate (`ai-dossier cap run
|
|
474
484
|
typecheck.run` / `test.focused`, when the repo has a manifest) runs after each member
|
|
475
485
|
before advancing — a second, independent check that the member's self-reported "done"
|
|
476
|
-
is real.
|
|
486
|
+
is real. Three-way outcome policy (#583): `task-failed` evicts the member (same rail
|
|
487
|
+
as a self-reported block); `automation-broken`/`capability-unavailable` — the gate
|
|
488
|
+
itself couldn't reach a verdict — block the batch instead of silently proceeding
|
|
489
|
+
(`gate-inconclusive:<cap>`, `member_gates`/`blocked_reason` on `BatchEntry`, surfaced
|
|
490
|
+
in `sched status`); `sched resume --batch <id>` re-runs the gate later to resolve the
|
|
491
|
+
block once the capability is fixed.
|
|
477
492
|
- **The batch's single slot is claimed FRESH for each live step** (a member, the tail
|
|
478
493
|
agent, the report agent, a bounded fix agent) — never held across a wait. The aggregate
|
|
479
494
|
suite itself runs with NO slot claimed at all (deterministic engine work, not an LLM
|
|
@@ -504,10 +519,21 @@ batch-setup had pool integration). Config schema moves to 1.3.0: `dispatch` gain
|
|
|
504
519
|
`member_prompt`, `batch_tail_prompt` and `batch_report_prompt` (the three new agent
|
|
505
520
|
prompt templates).
|
|
506
521
|
|
|
522
|
+
Schema 1.11.0 (#583): `BatchEntry` gains `member_gates` (most recent incremental-gate
|
|
523
|
+
result per member, keyed by issue number as a string — `{capability, outcome,
|
|
524
|
+
output_tail, at}`) and `blocked_reason` (why the batch is `blocked` — persisted so
|
|
525
|
+
`sched status` can show it; previously `blockBatch` only journaled/posted the reason,
|
|
526
|
+
never stored it on the entry, so this also retroactively covers the #562 case). 1.10.0
|
|
527
|
+
states migrate on load: no gate has ever produced a non-`ok` verdict, and no batch has
|
|
528
|
+
ever been blocked, under them, so `{}`/`null` is the exact backfill, not a guess.
|
|
529
|
+
|
|
507
530
|
New journal events: `batch-setup-done`, `batch-setup-failed`, `member-advanced`,
|
|
508
531
|
`batch-warmup-done`, `batch-warmup-failed` (#561 — the cold-path warm step only; a pool
|
|
509
|
-
claim emits neither).
|
|
510
|
-
|
|
532
|
+
claim emits neither). `gate-inconclusive` (#583 — the incremental gate came back
|
|
533
|
+
`automation-broken`/`capability-unavailable` rather than a definite `ok`/`task-failed`;
|
|
534
|
+
sits alongside `batch-blocked` as the per-member analogue of the aggregate suite's
|
|
535
|
+
"block, don't dissolve" precedent). Member/tail/report/fix-agent spawn, progress,
|
|
536
|
+
completion and park events reuse the existing unit-generic names (`assigned`/`spawned`/`unit-failed`/
|
|
511
537
|
`external-advance`/`pr-parked`/`merge-accepted`/`report-dispatched`/`teardown-done`/
|
|
512
538
|
`teardown-failed`) with `unit = batch:<id>`.
|
|
513
539
|
|
|
@@ -624,10 +650,12 @@ import {
|
|
|
624
650
|
// the issue pass — loads/saves state itself, holds no lock
|
|
625
651
|
// across the call
|
|
626
652
|
type BatchDispatchDeps, // inject store/journal/groundTruth/spawnDeps/exec/runSuite/
|
|
627
|
-
// runCapability(optional)/fsExists(optional)
|
|
653
|
+
// runCapability(optional, returns CapabilityGateResult)/fsExists(optional)
|
|
628
654
|
type BatchTickResult, // spawned/completed/parked/mergeAccepted/failed (batch:<id> ids)
|
|
629
655
|
// + blocked (issue numbers, dissolve-requeued)
|
|
630
656
|
type CapOutcome, // ok | task-failed | automation-broken | capability-unavailable
|
|
657
|
+
type CapabilityGateResult, // {outcome: CapOutcome, outputTail?, reason?} — runCapability's return shape (#583)
|
|
658
|
+
resumeBlockedGate, // #583: sched resume --batch <id> — re-run the gate that blocked a batch
|
|
631
659
|
buildMemberPrompt, buildBatchTailPrompt, buildBatchReportPrompt, // #523 prompt builders
|
|
632
660
|
DEFAULT_MEMBER_PROMPT_TEMPLATE, DEFAULT_BATCH_TAIL_PROMPT_TEMPLATE,
|
|
633
661
|
DEFAULT_BATCH_REPORT_PROMPT_TEMPLATE,
|
|
@@ -656,7 +684,7 @@ telemetry" below.
|
|
|
656
684
|
├── config.json # durable intent: max_slots, stall_timeout_ms, reconcile_interval_ms,
|
|
657
685
|
│ # pr_poll_interval_ms, dispatch (incl. report_prompt,
|
|
658
686
|
│ # phase_stall_timeout_ms, fence_takeover_timeout_ms, tiers — #527,
|
|
659
|
-
│ # suite_command — #562), auto_upgrade — #537,
|
|
687
|
+
│ # suite_command — #562, disallowed_tools — #591), auto_upgrade — #537,
|
|
660
688
|
│ # dissolve_policy — #563
|
|
661
689
|
├── events.jsonl # append-only event journal (the operator's flight recorder)
|
|
662
690
|
├── runs/ # per-unit agent output logs (issue-<n>.log)
|
|
@@ -677,6 +705,14 @@ every place a dispatch's exit is first detected: the dead-pid rail, the
|
|
|
677
705
|
external-advance rail, a stall-timeout kill, and a dependents-blocked kill), sourced
|
|
678
706
|
from the agent's `modelUsage` map — never blended with the top-level `usage` block,
|
|
679
707
|
the fix for a ~43% fabricated-saving discrepancy the two blocks were found to produce.
|
|
708
|
+
`recordDispatchRunLog`/`recordMemberRunLog` also return the last tool the dispatch called
|
|
709
|
+
(`parseLastToolUse`, `@ai-dossier/core`, #591), when the log yielded one — the exit itself
|
|
710
|
+
attributes to a concrete cause (e.g. `Monitor`) without opening the transcript. It rides the
|
|
711
|
+
non-terminal `verify-incomplete` event on every unverified exit and, once the escalation
|
|
712
|
+
ladder is exhausted, the terminal `unit-failed` (`agent-exited-unverified` /
|
|
713
|
+
`unverified-exit-at-strongest-tier`) as `last_tool`; a stall-timeout kill carries it too. Only
|
|
714
|
+
the dead-pid detection rail and the stall kill record a fresh log slice in the same tick —
|
|
715
|
+
a slot already `exited`/`verifying` when reconciled again has none to attribute.
|
|
680
716
|
|
|
681
717
|
The dispatch log (`runs/<unit>.log`) is per-UNIT and opened in append mode
|
|
682
718
|
(`createSpawnDeps`), so a redispatched unit's second agent writes its output AFTER the
|
package/dist/batch-dispatch.d.ts
CHANGED
|
@@ -67,15 +67,8 @@ import type { SchedStore } from './persist';
|
|
|
67
67
|
import type { ExecFn } from './project';
|
|
68
68
|
import { type SuiteResult } from './recovery';
|
|
69
69
|
import { type FsExists } from './teardown';
|
|
70
|
-
import type { SchedConfig } from './types';
|
|
71
|
-
|
|
72
|
-
* The four `ai-dossier cap run` outcomes (docs/reference/capabilities.md):
|
|
73
|
-
* `ok` = the capability ran and passed; `task-failed` = it ran and the TASK
|
|
74
|
-
* itself failed (trust the result); `automation-broken` = do not trust the
|
|
75
|
-
* machinery (missing tool, timeout, bad manifest); `capability-unavailable` =
|
|
76
|
-
* no manifest / no such id / `lifecycle: shadow` — no fast path here.
|
|
77
|
-
*/
|
|
78
|
-
export type CapOutcome = 'ok' | 'task-failed' | 'automation-broken' | 'capability-unavailable';
|
|
70
|
+
import type { CapabilityGateResult, SchedConfig } from './types';
|
|
71
|
+
export type { CapOutcome } from './types';
|
|
79
72
|
/** Everything batch dispatch needs from the outside world. */
|
|
80
73
|
export interface BatchDispatchDeps {
|
|
81
74
|
store: SchedStore;
|
|
@@ -100,19 +93,24 @@ export interface BatchDispatchDeps {
|
|
|
100
93
|
/**
|
|
101
94
|
* Runs one `ai-dossier cap run <capabilityId>` in a batch worktree. Two call
|
|
102
95
|
* sites, different degrade contracts:
|
|
103
|
-
* - the per-member incremental gate (#523 AC2
|
|
104
|
-
*
|
|
105
|
-
* this
|
|
106
|
-
*
|
|
107
|
-
*
|
|
108
|
-
*
|
|
96
|
+
* - the per-member incremental gate (#523 AC2, revised #583): without this
|
|
97
|
+
* hook, the gate is skipped entirely (the member's own `slot-cycle` run
|
|
98
|
+
* already attempted this fast path before ever posting `review done`, so
|
|
99
|
+
* a repo with no manifest loses nothing but the engine's independent
|
|
100
|
+
* re-check). With the hook: `ok` advances, `task-failed` evicts,
|
|
101
|
+
* `automation-broken`/`capability-unavailable` BLOCK the batch
|
|
102
|
+
* (`gate-inconclusive`) rather than silently proceeding — #583 found the
|
|
103
|
+
* old "proceed on anything but task-failed" policy let a script's own
|
|
104
|
+
* "I could not run this" signal (a non-zero exit reporting it never
|
|
105
|
+
* really tested anything) masquerade as either a pass or a real failure.
|
|
109
106
|
* - batch-setup's `worktree.prepare` warm step (#561, `warmColdBatchWorktree`):
|
|
110
|
-
*
|
|
111
|
-
*
|
|
112
|
-
*
|
|
113
|
-
*
|
|
107
|
+
* unaffected by #583's gate policy change — `undefined`/
|
|
108
|
+
* `capability-unavailable`/`automation-broken` still fall through to
|
|
109
|
+
* package-manager detection, and a declared-and-`task-failed` capability
|
|
110
|
+
* still hard-fails the whole batch setup (a repo that owns its warm-up
|
|
111
|
+
* should never be silently second-guessed by a fallback underneath it).
|
|
114
112
|
*/
|
|
115
|
-
runCapability?: (worktree: string, capabilityId: string) =>
|
|
113
|
+
runCapability?: (worktree: string, capabilityId: string) => CapabilityGateResult;
|
|
116
114
|
fsExists?: FsExists;
|
|
117
115
|
/**
|
|
118
116
|
* Home directory for `~/.dossier/runs.jsonl` (#564) — mirrors `EngineDeps.homeDir`.
|
|
@@ -130,6 +128,33 @@ export interface BatchTickResult {
|
|
|
130
128
|
/** Issue numbers requeued full-cycle by a dissolve — matches `TickResult.blocked`'s shape. */
|
|
131
129
|
blocked: number[];
|
|
132
130
|
}
|
|
131
|
+
/**
|
|
132
|
+
* `sched resume --batch <id>` (#583 AC4): an operator-triggered, synchronous
|
|
133
|
+
* one-shot recheck of a batch blocked on `gate-inconclusive:<capabilityId>`
|
|
134
|
+
* — re-runs exactly that capability against the current member and resolves
|
|
135
|
+
* the block:
|
|
136
|
+
*
|
|
137
|
+
* - still `automation-broken`/`capability-unavailable` → stays `blocked`,
|
|
138
|
+
* no state change (the capability still isn't fixed).
|
|
139
|
+
* - `task-failed` → the member really is broken; evict via the same rail
|
|
140
|
+
* the live gate uses.
|
|
141
|
+
* - `ok` → the member really was fine; complete it via the same rail the
|
|
142
|
+
* live gate uses.
|
|
143
|
+
*
|
|
144
|
+
* Deliberately NOT a generic engine tick: `runBatchTick`'s own "executing
|
|
145
|
+
* with no live slot" wedge-recovery path (`spawnMemberContinuation`,
|
|
146
|
+
* unconditional) exists for a dispatch that never happened — reusing it here
|
|
147
|
+
* would redispatch a fresh agent for a member whose work is already
|
|
148
|
+
* committed and reviewed. This function transitions `blocked → executing`
|
|
149
|
+
* and immediately, synchronously, calls the same completion/eviction
|
|
150
|
+
* functions the live gate calls — no intervening tick ever sees the batch
|
|
151
|
+
* `executing` with nothing in flight.
|
|
152
|
+
*/
|
|
153
|
+
export declare function resumeBlockedGate(deps: BatchDispatchDeps, config: SchedConfig, dispatch: ResolvedDispatch, batchId: string, now: Date): {
|
|
154
|
+
outcome: 'still-blocked' | 'evicted' | 'completed';
|
|
155
|
+
capability: string;
|
|
156
|
+
detail?: string;
|
|
157
|
+
};
|
|
133
158
|
/**
|
|
134
159
|
* One batch reconcile+refill pass. Called from `engine.ts`'s `tick()` after
|
|
135
160
|
* the issue-level pass — this pass never claims a slot the issue pass already
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"batch-dispatch.d.ts","sourceRoot":"","sources":["../src/batch-dispatch.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6DG;
|
|
1
|
+
{"version":3,"file":"batch-dispatch.d.ts","sourceRoot":"","sources":["../src/batch-dispatch.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6DG;AAkBH,OAAO,EAUL,KAAK,gBAAgB,EAErB,KAAK,SAAS,EACf,MAAM,YAAY,CAAC;AACpB,OAAO,EACL,KAAK,WAAW,EAOjB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,KAAK,OAAO,EAAa,MAAM,WAAW,CAAC;AACpD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AAExC,OAAO,EAUL,KAAK,WAAW,EACjB,MAAM,YAAY,CAAC;AAYpB,OAAO,EAAE,KAAK,QAAQ,EAA2D,MAAM,YAAY,CAAC;AACpG,OAAO,KAAK,EAGV,oBAAoB,EAIpB,WAAW,EAIZ,MAAM,SAAS,CAAC;AAMjB,YAAY,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AAE1C,8DAA8D;AAC9D,MAAM,WAAW,iBAAiB;IAChC,KAAK,EAAE,UAAU,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,WAAW,EAAE,WAAW,CAAC;IACzB,SAAS,EAAE,SAAS,CAAC;IACrB,GAAG,EAAE,MAAM,IAAI,CAAC;IAChB,uFAAuF;IACvF,OAAO,EAAE,MAAM,CAAC;IAChB,0FAA0F;IAC1F,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,sGAAsG;IACtG,QAAQ,EAAE,CAAC,QAAQ,EAAE,MAAM,KAAK,WAAW,CAAC;IAC5C;;;;;;;;;;;;;;;;;;;OAmBG;IACH,aAAa,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,YAAY,EAAE,MAAM,KAAK,oBAAoB,CAAC;IACjF,QAAQ,CAAC,EAAE,QAAQ,CAAC;IACpB;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,8FAA8F;AAC9F,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,8FAA8F;IAC9F,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAywCD;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,wBAAgB,iBAAiB,CAC/B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,GAAG,EAAE,IAAI,GACR;IAAE,OAAO,EAAE,eAAe,GAAG,SAAS,GAAG,WAAW,CAAC;IAAC,UAAU,EAAE,MAAM,CAAC;IAAC,MAAM,CAAC,EAAE,MAAM,CAAA;CAAE,CA+E7F;AAofD;;;;;;;;;;;;;GAaG;AACH,wBAAgB,YAAY,CAC1B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,GACzB,eAAe,CAsFjB"}
|
package/dist/batch-dispatch.js
CHANGED
|
@@ -95,9 +95,11 @@ var __importStar = (this && this.__importStar) || (function () {
|
|
|
95
95
|
};
|
|
96
96
|
})();
|
|
97
97
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
98
|
+
exports.resumeBlockedGate = resumeBlockedGate;
|
|
98
99
|
exports.runBatchTick = runBatchTick;
|
|
99
100
|
const fs = __importStar(require("node:fs"));
|
|
100
101
|
const path = __importStar(require("node:path"));
|
|
102
|
+
const core_1 = require("@ai-dossier/core");
|
|
101
103
|
const worktree_pool_1 = require("@ai-dossier/worktree-pool");
|
|
102
104
|
const attribution_1 = require("./attribution");
|
|
103
105
|
const dispatch_1 = require("./dispatch");
|
|
@@ -140,6 +142,65 @@ function sanitizeUntrustedText(value) {
|
|
|
140
142
|
function journalEvent(deps, event, unitId, extra = {}) {
|
|
141
143
|
deps.journal.append((0, journal_1.unitEvent)(event, unitId, extra), deps.now());
|
|
142
144
|
}
|
|
145
|
+
/**
|
|
146
|
+
* Last ~500 bytes of a gate's output tail, falling back to the envelope's
|
|
147
|
+
* `reason` when no subprocess ran (#583 AC1/AC3 review: `capability-unavailable`
|
|
148
|
+
* and a failed assumption probe carry no output_tail — `reason` is the only
|
|
149
|
+
* explanation available for those) — journal details stay compact; the full
|
|
150
|
+
* tail lives in the per-gate log file. UTF-8-safe (never splits a multi-byte
|
|
151
|
+
* character) — chars would risk exactly that, hence `Buffer`, matching
|
|
152
|
+
* `cli/src/capability.ts`'s `truncateTailBytes`.
|
|
153
|
+
*/
|
|
154
|
+
function gateDetailExcerpt(outputTail, reason) {
|
|
155
|
+
const text = outputTail || reason;
|
|
156
|
+
if (!text)
|
|
157
|
+
return undefined;
|
|
158
|
+
const buf = Buffer.from(text, 'utf-8');
|
|
159
|
+
return buf.length > 500 ? buf.subarray(buf.length - 500).toString('utf-8') : text;
|
|
160
|
+
}
|
|
161
|
+
/**
|
|
162
|
+
* Best-effort per-gate diagnostic log (#583 AC1) — mirrors `appendCapLog`'s
|
|
163
|
+
* never-crash contract: a log-write failure must not interrupt the gate
|
|
164
|
+
* decision itself.
|
|
165
|
+
*/
|
|
166
|
+
function writeGateLog(deps, batchId, capabilityId, issue, outputTail) {
|
|
167
|
+
if (!outputTail)
|
|
168
|
+
return;
|
|
169
|
+
try {
|
|
170
|
+
const logPath = (0, dispatch_1.batchGateLogPath)(deps.store.runsDir, batchId, capabilityId, issue);
|
|
171
|
+
fs.mkdirSync(path.dirname(logPath), { recursive: true, mode: 0o700 });
|
|
172
|
+
fs.writeFileSync(logPath, outputTail, { mode: 0o600 });
|
|
173
|
+
}
|
|
174
|
+
catch {
|
|
175
|
+
// diagnostic only — never fail the gate decision over a log write
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
/**
|
|
179
|
+
* Persist the incremental gate's most recent verdict for a member on
|
|
180
|
+
* `BatchEntry.member_gates` (#583 AC4) — shared by the live gate
|
|
181
|
+
* (`reconcileMemberSlot`) and `sched resume --batch`'s recheck
|
|
182
|
+
* (`resumeBlockedGate`), so a resumed batch's status never shows a stale
|
|
183
|
+
* outcome from before the recheck ran.
|
|
184
|
+
*/
|
|
185
|
+
function recordMemberGate(deps, batchId, memberIssue, gate, now) {
|
|
186
|
+
deps.store.withLock((s) => {
|
|
187
|
+
const b = (0, state_1.findBatch)(s, batchId);
|
|
188
|
+
return {
|
|
189
|
+
state: (0, state_1.patchBatch)(s, batchId, {
|
|
190
|
+
member_gates: {
|
|
191
|
+
...(b?.member_gates ?? {}),
|
|
192
|
+
[String(memberIssue)]: {
|
|
193
|
+
capability: gate.id,
|
|
194
|
+
outcome: gate.outcome,
|
|
195
|
+
output_tail: gate.outputTail ?? null,
|
|
196
|
+
at: now.toISOString(),
|
|
197
|
+
},
|
|
198
|
+
},
|
|
199
|
+
}, now),
|
|
200
|
+
result: undefined,
|
|
201
|
+
};
|
|
202
|
+
});
|
|
203
|
+
}
|
|
143
204
|
function slotFor(state, batchId) {
|
|
144
205
|
return state.slots.find((s) => s.unit === unit(batchId));
|
|
145
206
|
}
|
|
@@ -288,7 +349,7 @@ poolNote = '') {
|
|
|
288
349
|
}), now);
|
|
289
350
|
return { ok: true };
|
|
290
351
|
};
|
|
291
|
-
const capOutcome = deps.runCapability?.(worktree, 'worktree.prepare');
|
|
352
|
+
const capOutcome = deps.runCapability?.(worktree, 'worktree.prepare')?.outcome;
|
|
292
353
|
if (capOutcome === 'ok')
|
|
293
354
|
return done('cap:worktree.prepare');
|
|
294
355
|
if (capOutcome === 'task-failed')
|
|
@@ -932,6 +993,187 @@ function evictMemberAndContinue(deps, config, dispatch, batchId, batch, memberIs
|
|
|
932
993
|
}
|
|
933
994
|
advanceMemberOrValidate(deps, config, dispatch, batchId, batch.members.length, batch.executing_member, memberIssue, now, result);
|
|
934
995
|
}
|
|
996
|
+
/**
|
|
997
|
+
* The incremental gate (#523 AC2, revised #583): typecheck + focused tests
|
|
998
|
+
* via `cap run`, when the repo has a manifest for them — a second,
|
|
999
|
+
* independent check that the member's own self-reported "done" is real,
|
|
1000
|
+
* matching this codebase's "never trust a claimed completion" ethos
|
|
1001
|
+
* (AC2/#464's `isVerifiedComplete`). Three-way policy: `task-failed` evicts
|
|
1002
|
+
* the member directly, same rail as a self-reported block (RFC F.1) — no
|
|
1003
|
+
* aggregate suite has run yet, so there is nothing to attribute.
|
|
1004
|
+
* `automation-broken`/`capability-unavailable` — the gate itself could not
|
|
1005
|
+
* reach a verdict — BLOCK the batch instead of silently proceeding (#583: a
|
|
1006
|
+
* script that legitimately could not run its suite must not be read as
|
|
1007
|
+
* either a pass or a real failure). Only both `ok` falls through.
|
|
1008
|
+
*
|
|
1009
|
+
* Returns `true` when the gate already decided the member's fate (evicted or
|
|
1010
|
+
* blocked, both of which `return` from the caller); `false` when no hook is
|
|
1011
|
+
* configured or both checks came back `ok`, meaning the caller should treat
|
|
1012
|
+
* the member as genuinely complete.
|
|
1013
|
+
*/
|
|
1014
|
+
function runIncrementalGate(deps, config, dispatch, batchId, batch, memberIssue, now, result) {
|
|
1015
|
+
if (batch.worktree === null || !deps.runCapability)
|
|
1016
|
+
return false;
|
|
1017
|
+
const worktree = batch.worktree;
|
|
1018
|
+
const runCapability = deps.runCapability;
|
|
1019
|
+
const gateResults = ['typecheck.run', 'test.focused'].map((id) => ({
|
|
1020
|
+
id,
|
|
1021
|
+
...runCapability(worktree, id),
|
|
1022
|
+
}));
|
|
1023
|
+
const gateFailure = gateResults.find((r) => r.outcome === 'task-failed');
|
|
1024
|
+
const gateInconclusive = gateResults.find((r) => r.outcome === 'automation-broken' || r.outcome === 'capability-unavailable');
|
|
1025
|
+
const worstGate = gateFailure ?? gateInconclusive;
|
|
1026
|
+
if (worstGate) {
|
|
1027
|
+
recordMemberGate(deps, batchId, memberIssue, worstGate, now);
|
|
1028
|
+
}
|
|
1029
|
+
if (gateFailure) {
|
|
1030
|
+
const reason = `incremental-gate-failed:${gateFailure.id}`;
|
|
1031
|
+
writeGateLog(deps, batchId, gateFailure.id, memberIssue, gateFailure.outputTail);
|
|
1032
|
+
const excerpt = gateDetailExcerpt(gateFailure.outputTail, gateFailure.reason);
|
|
1033
|
+
journalEvent(deps, 'unit-failed', unit(batchId), {
|
|
1034
|
+
issue: memberIssue,
|
|
1035
|
+
reason,
|
|
1036
|
+
detail: excerpt
|
|
1037
|
+
? `cap run ${gateFailure.id} reported task-failed after member review done: ${excerpt}`
|
|
1038
|
+
: `cap run ${gateFailure.id} reported task-failed after member review done`,
|
|
1039
|
+
});
|
|
1040
|
+
deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
|
|
1041
|
+
evictMemberAndContinue(deps, config, dispatch, batchId, batch, memberIssue, reason, now, result);
|
|
1042
|
+
return true;
|
|
1043
|
+
}
|
|
1044
|
+
if (gateInconclusive) {
|
|
1045
|
+
const reason = `gate-inconclusive:${gateInconclusive.id}`;
|
|
1046
|
+
writeGateLog(deps, batchId, gateInconclusive.id, memberIssue, gateInconclusive.outputTail);
|
|
1047
|
+
const excerpt = gateDetailExcerpt(gateInconclusive.outputTail, gateInconclusive.reason);
|
|
1048
|
+
journalEvent(deps, 'gate-inconclusive', unit(batchId), {
|
|
1049
|
+
issue: memberIssue,
|
|
1050
|
+
reason,
|
|
1051
|
+
detail: excerpt
|
|
1052
|
+
? `cap run ${gateInconclusive.id} reported ${gateInconclusive.outcome} after member review done: ${excerpt}`
|
|
1053
|
+
: `cap run ${gateInconclusive.id} reported ${gateInconclusive.outcome} after member review done`,
|
|
1054
|
+
});
|
|
1055
|
+
deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
|
|
1056
|
+
const stateNow = deps.store.load();
|
|
1057
|
+
const rDeps = recoveryDeps(deps, config, batch, now);
|
|
1058
|
+
const blocked = (0, recovery_1.blockBatch)(stateNow, batchId, { reason, milestonePhase: 'batch-review' }, rDeps);
|
|
1059
|
+
deps.store.withLock((s) => ({
|
|
1060
|
+
state: applyBatchAndIssues(s, blocked.state, batchId, []),
|
|
1061
|
+
result: undefined,
|
|
1062
|
+
}));
|
|
1063
|
+
result.failed.push(unit(batchId));
|
|
1064
|
+
return true;
|
|
1065
|
+
}
|
|
1066
|
+
return false;
|
|
1067
|
+
}
|
|
1068
|
+
/**
|
|
1069
|
+
* The member is verifiably done — self-reported complete AND (when checked)
|
|
1070
|
+
* the incremental gate agrees. Recompute the member's commit range, mark it
|
|
1071
|
+
* validated, and advance the batch. Shared by `reconcileMemberSlot`'s
|
|
1072
|
+
* immediate "both gate checks ok" fallthrough and `resumeBlockedGate` (#583)
|
|
1073
|
+
* — a member confirmed complete via a delayed `sched resume --batch` recheck
|
|
1074
|
+
* gets exactly the same treatment as one confirmed complete on the first try.
|
|
1075
|
+
* `releaseSlot` is idempotent (a no-op once the slot is already idle), so
|
|
1076
|
+
* this is safe to call whether or not the caller already released it.
|
|
1077
|
+
*/
|
|
1078
|
+
function completeMemberGate(deps, config, dispatch, batchId, batch, memberIssue, now, result) {
|
|
1079
|
+
// The commit-range recompute (`git log`) is a blocking subprocess call — it
|
|
1080
|
+
// must run OUTSIDE the lock, like every other exec in this module; the
|
|
1081
|
+
// result then lands as a pure data patch under the lock (Convention review:
|
|
1082
|
+
// `recordRanges` used to run `git log` INSIDE the withLock mutator, which
|
|
1083
|
+
// is exactly what `engine.ts`'s own "a slow git call never holds the lock"
|
|
1084
|
+
// invariant exists to prevent).
|
|
1085
|
+
const ranges = (0, attribution_1.memberRanges)(boundaryCommits(deps, batch));
|
|
1086
|
+
deps.store.withLock((s) => {
|
|
1087
|
+
let n = releaseSlot(s, batchId, now);
|
|
1088
|
+
n = (0, state_1.patchBatch)(n, batchId, { ranges }, now);
|
|
1089
|
+
n = advanceMemberToValidated(n, memberIssue, now);
|
|
1090
|
+
return { state: n, result: undefined };
|
|
1091
|
+
});
|
|
1092
|
+
result.completed.push(unit(batchId));
|
|
1093
|
+
advanceMemberOrValidate(deps, config, dispatch, batchId, batch.members.length, batch.executing_member, memberIssue, now, result);
|
|
1094
|
+
}
|
|
1095
|
+
/**
|
|
1096
|
+
* `sched resume --batch <id>` (#583 AC4): an operator-triggered, synchronous
|
|
1097
|
+
* one-shot recheck of a batch blocked on `gate-inconclusive:<capabilityId>`
|
|
1098
|
+
* — re-runs exactly that capability against the current member and resolves
|
|
1099
|
+
* the block:
|
|
1100
|
+
*
|
|
1101
|
+
* - still `automation-broken`/`capability-unavailable` → stays `blocked`,
|
|
1102
|
+
* no state change (the capability still isn't fixed).
|
|
1103
|
+
* - `task-failed` → the member really is broken; evict via the same rail
|
|
1104
|
+
* the live gate uses.
|
|
1105
|
+
* - `ok` → the member really was fine; complete it via the same rail the
|
|
1106
|
+
* live gate uses.
|
|
1107
|
+
*
|
|
1108
|
+
* Deliberately NOT a generic engine tick: `runBatchTick`'s own "executing
|
|
1109
|
+
* with no live slot" wedge-recovery path (`spawnMemberContinuation`,
|
|
1110
|
+
* unconditional) exists for a dispatch that never happened — reusing it here
|
|
1111
|
+
* would redispatch a fresh agent for a member whose work is already
|
|
1112
|
+
* committed and reviewed. This function transitions `blocked → executing`
|
|
1113
|
+
* and immediately, synchronously, calls the same completion/eviction
|
|
1114
|
+
* functions the live gate calls — no intervening tick ever sees the batch
|
|
1115
|
+
* `executing` with nothing in flight.
|
|
1116
|
+
*/
|
|
1117
|
+
function resumeBlockedGate(deps, config, dispatch, batchId, now) {
|
|
1118
|
+
const state = deps.store.load();
|
|
1119
|
+
const batch = (0, state_1.findBatch)(state, batchId);
|
|
1120
|
+
if (!batch)
|
|
1121
|
+
throw new types_1.SchedNotFoundError(`Batch not found: ${batchId}`);
|
|
1122
|
+
if (batch.status !== 'blocked' || !batch.blocked_reason?.startsWith('gate-inconclusive:')) {
|
|
1123
|
+
throw new types_1.IllegalTransitionError('batch', batch.status, 'executing');
|
|
1124
|
+
}
|
|
1125
|
+
const capabilityId = batch.blocked_reason.slice('gate-inconclusive:'.length);
|
|
1126
|
+
const memberIssue = batch.members[batch.executing_member - 1];
|
|
1127
|
+
if (batch.worktree === null || !deps.runCapability || memberIssue === undefined) {
|
|
1128
|
+
throw new types_1.SchedNotFoundError(`Batch ${batchId} has no worktree/current member/gate hook to recheck`);
|
|
1129
|
+
}
|
|
1130
|
+
const recheck = deps.runCapability(batch.worktree, capabilityId);
|
|
1131
|
+
const result = emptyResult();
|
|
1132
|
+
const excerpt = gateDetailExcerpt(recheck.outputTail, recheck.reason);
|
|
1133
|
+
recordMemberGate(deps, batchId, memberIssue, { id: capabilityId, ...recheck }, now);
|
|
1134
|
+
if (recheck.outcome === 'automation-broken' || recheck.outcome === 'capability-unavailable') {
|
|
1135
|
+
// Still inconclusive: leave an audit trail (log + journal) exactly like
|
|
1136
|
+
// the live gate does, even though the batch stays `blocked` — otherwise
|
|
1137
|
+
// a repeated `sched resume --batch` leaves no record of when it was last
|
|
1138
|
+
// checked or what it said (#583 review).
|
|
1139
|
+
writeGateLog(deps, batchId, capabilityId, memberIssue, recheck.outputTail);
|
|
1140
|
+
journalEvent(deps, 'gate-inconclusive', unit(batchId), {
|
|
1141
|
+
issue: memberIssue,
|
|
1142
|
+
reason: `gate-inconclusive:${capabilityId}`,
|
|
1143
|
+
detail: excerpt
|
|
1144
|
+
? `sched resume --batch: cap run ${capabilityId} still reports ${recheck.outcome} on recheck: ${excerpt}`
|
|
1145
|
+
: `sched resume --batch: cap run ${capabilityId} still reports ${recheck.outcome} on recheck`,
|
|
1146
|
+
});
|
|
1147
|
+
return {
|
|
1148
|
+
outcome: 'still-blocked',
|
|
1149
|
+
capability: capabilityId,
|
|
1150
|
+
detail: excerpt,
|
|
1151
|
+
};
|
|
1152
|
+
}
|
|
1153
|
+
deps.store.withLock((s) => ({
|
|
1154
|
+
state: (0, state_1.patchBatch)((0, state_1.transitionBatch)(s, batchId, 'executing', {}, now), batchId, { blocked_reason: null }, now),
|
|
1155
|
+
result: undefined,
|
|
1156
|
+
}));
|
|
1157
|
+
if (recheck.outcome === 'task-failed') {
|
|
1158
|
+
const reason = `incremental-gate-failed:${capabilityId}`;
|
|
1159
|
+
writeGateLog(deps, batchId, capabilityId, memberIssue, recheck.outputTail);
|
|
1160
|
+
journalEvent(deps, 'unit-failed', unit(batchId), {
|
|
1161
|
+
issue: memberIssue,
|
|
1162
|
+
reason,
|
|
1163
|
+
detail: excerpt
|
|
1164
|
+
? `sched resume --batch: cap run ${capabilityId} reported task-failed on recheck: ${excerpt}`
|
|
1165
|
+
: `sched resume --batch: cap run ${capabilityId} reported task-failed on recheck`,
|
|
1166
|
+
});
|
|
1167
|
+
evictMemberAndContinue(deps, config, dispatch, batchId, batch, memberIssue, reason, now, result);
|
|
1168
|
+
return { outcome: 'evicted', capability: capabilityId, detail: excerpt };
|
|
1169
|
+
}
|
|
1170
|
+
journalEvent(deps, 'external-advance', unit(batchId), {
|
|
1171
|
+
issue: memberIssue,
|
|
1172
|
+
detail: `sched resume --batch: cap run ${capabilityId} reported ok on recheck`,
|
|
1173
|
+
});
|
|
1174
|
+
completeMemberGate(deps, config, dispatch, batchId, batch, memberIssue, now, result);
|
|
1175
|
+
return { outcome: 'completed', capability: capabilityId };
|
|
1176
|
+
}
|
|
935
1177
|
/**
|
|
936
1178
|
* Record one member dispatch's tokens/cost to `runs.jsonl` (#564) — the
|
|
937
1179
|
* `batch-dispatch.ts` analogue of `engine.ts`'s `recordDispatchRunLog`.
|
|
@@ -957,7 +1199,7 @@ function recordMemberRunLog(deps, dispatch, state, batchId, batch, memberIssue,
|
|
|
957
1199
|
reason: slot.spawned_at === null ? 'never-spawned' : `already-recorded-${slot.status}`,
|
|
958
1200
|
slot: slot.id,
|
|
959
1201
|
});
|
|
960
|
-
return;
|
|
1202
|
+
return null;
|
|
961
1203
|
}
|
|
962
1204
|
const tier = (0, state_1.findEntry)(state, memberIssue)?.tier ?? 'mid';
|
|
963
1205
|
const { cmd, model } = (0, dispatch_1.resolveTierSpawn)(dispatch, tier, memberIssue);
|
|
@@ -976,6 +1218,9 @@ function recordMemberRunLog(deps, dispatch, state, batchId, batch, memberIssue,
|
|
|
976
1218
|
tier,
|
|
977
1219
|
});
|
|
978
1220
|
(0, run_log_1.finalizeRunLogEntry)(runEntry, logContent, deps.homeDir, (event, extra) => journalEvent(deps, event, unit(batchId), extra), { issue: memberIssue, log: logFile });
|
|
1221
|
+
// #591: the last tool this member dispatch called — attributes an
|
|
1222
|
+
// `agent-exited-unverified` failure to a concrete cause without opening the transcript.
|
|
1223
|
+
return (0, core_1.parseLastToolUse)(logContent);
|
|
979
1224
|
}
|
|
980
1225
|
function reconcileMemberSlot(deps, config, dispatch, batchId, slot, now, result) {
|
|
981
1226
|
const state0 = deps.store.load();
|
|
@@ -1016,46 +1261,15 @@ function reconcileMemberSlot(deps, config, dispatch, batchId, slot, now, result)
|
|
|
1016
1261
|
// incremental gate below decides — record its telemetry once here (#564)
|
|
1017
1262
|
// rather than at each of this branch's two later exit points.
|
|
1018
1263
|
recordMemberRunLog(deps, dispatch, state0, batchId, batch, memberIssue, slot, now);
|
|
1019
|
-
// Incremental gate (#523 AC2
|
|
1020
|
-
//
|
|
1021
|
-
//
|
|
1022
|
-
//
|
|
1023
|
-
//
|
|
1024
|
-
|
|
1025
|
-
|
|
1026
|
-
if (batch.worktree !== null && deps.runCapability) {
|
|
1027
|
-
const gateFailure = ['typecheck.run', 'test.focused']
|
|
1028
|
-
.map((id) => ({ id, outcome: deps.runCapability?.(batch.worktree, id) }))
|
|
1029
|
-
.find((r) => r.outcome === 'task-failed');
|
|
1030
|
-
if (gateFailure) {
|
|
1031
|
-
const reason = `incremental-gate-failed:${gateFailure.id}`;
|
|
1032
|
-
journalEvent(deps, 'unit-failed', unit(batchId), {
|
|
1033
|
-
issue: memberIssue,
|
|
1034
|
-
reason,
|
|
1035
|
-
detail: `cap run ${gateFailure.id} reported task-failed after member review done`,
|
|
1036
|
-
});
|
|
1037
|
-
deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
|
|
1038
|
-
evictMemberAndContinue(deps, config, dispatch, batchId, batch, memberIssue, reason, now, result);
|
|
1039
|
-
return;
|
|
1040
|
-
}
|
|
1041
|
-
// `ok` / `automation-broken` / `capability-unavailable` all proceed —
|
|
1042
|
-
// only a definite task failure blocks a member here.
|
|
1264
|
+
// Incremental gate (#523 AC2, revised #583) — see `runIncrementalGate`'s
|
|
1265
|
+
// own doc comment for the three-way policy. A `true` return means the
|
|
1266
|
+
// gate already decided the member's fate (evicted or blocked) and
|
|
1267
|
+
// returned early; `false` means both checks were `ok` and the member is
|
|
1268
|
+
// genuinely done.
|
|
1269
|
+
if (runIncrementalGate(deps, config, dispatch, batchId, batch, memberIssue, now, result)) {
|
|
1270
|
+
return;
|
|
1043
1271
|
}
|
|
1044
|
-
|
|
1045
|
-
// it must run OUTSIDE the lock, like every other exec in this module;
|
|
1046
|
-
// the result then lands as a pure data patch under the lock (Convention
|
|
1047
|
-
// review: `recordRanges` used to run `git log` INSIDE the withLock
|
|
1048
|
-
// mutator, which is exactly what `engine.ts`'s own "a slow git call never
|
|
1049
|
-
// holds the lock" invariant exists to prevent).
|
|
1050
|
-
const ranges = (0, attribution_1.memberRanges)(boundaryCommits(deps, batch));
|
|
1051
|
-
deps.store.withLock((s) => {
|
|
1052
|
-
let n = releaseSlot(s, batchId, now);
|
|
1053
|
-
n = (0, state_1.patchBatch)(n, batchId, { ranges }, now);
|
|
1054
|
-
n = advanceMemberToValidated(n, memberIssue, now);
|
|
1055
|
-
return { state: n, result: undefined };
|
|
1056
|
-
});
|
|
1057
|
-
result.completed.push(unit(batchId));
|
|
1058
|
-
advanceMemberOrValidate(deps, config, dispatch, batchId, batch.members.length, batch.executing_member, memberIssue, now, result);
|
|
1272
|
+
completeMemberGate(deps, config, dispatch, batchId, batch, memberIssue, now, result);
|
|
1059
1273
|
return;
|
|
1060
1274
|
}
|
|
1061
1275
|
if ((0, groundtruth_1.isMemberBlocked)(milestone) || dead) {
|
|
@@ -1065,11 +1279,17 @@ function reconcileMemberSlot(deps, config, dispatch, batchId, slot, now, result)
|
|
|
1065
1279
|
: dead
|
|
1066
1280
|
? 'agent-exited-unverified'
|
|
1067
1281
|
: 'member-blocked';
|
|
1068
|
-
recordMemberRunLog(deps, dispatch, state0, batchId, batch, memberIssue, slot, now);
|
|
1282
|
+
const lastTool = recordMemberRunLog(deps, dispatch, state0, batchId, batch, memberIssue, slot, now);
|
|
1069
1283
|
journalEvent(deps, 'unit-failed', unit(batchId), {
|
|
1070
1284
|
issue: memberIssue,
|
|
1071
1285
|
reason,
|
|
1072
1286
|
detail: 'member blocked',
|
|
1287
|
+
// #591: attributes an unverified member exit to a concrete cause (e.g.
|
|
1288
|
+
// `Monitor`) without opening the transcript. Gated on the RESOLVED reason, not
|
|
1289
|
+
// `dead` — a member can be simultaneously `dead` AND carry a milestone-posted
|
|
1290
|
+
// `reason` (it posted `blocked` and then exited), and that real block has no
|
|
1291
|
+
// log-derived cause to attribute; only `agent-exited-unverified` does.
|
|
1292
|
+
...(reason === 'agent-exited-unverified' && lastTool !== null ? { last_tool: lastTool } : {}),
|
|
1073
1293
|
});
|
|
1074
1294
|
deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
|
|
1075
1295
|
// A member that never went green (RFC F.1) evicts DIRECTLY — no aggregate
|