@cat-factory/executor-harness 1.114.0 → 1.118.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -115,6 +115,38 @@ Bootstrap differs at the ends: it may start from an empty dir, and **resets
115
115
  history to one commit and force-pushes** the default branch instead of opening a
116
116
  PR. Blueprint **commits onto a branch** (no history reset) and returns the tree.
117
117
 
118
+ ### The work-branch push is CHECKPOINTED, so it is lease-guarded
119
+
120
+ Step 8's push is not the run's first: every `JOB_CHECKPOINT_INTERVAL_MS` (60s) the harness pushes
121
+ whatever the agent has committed and NOT yet published, so an evicted container's work survives on
122
+ the branch and a retry resumes on top of it. The interval is a **loss window**, not a push rate:
123
+ `unpublishedWorkBranchTip` skips a tick whose branch tip is already published, so a long run pushes
124
+ once per commit the agent makes rather than once a minute, and nothing here needs tuning per model.
125
+
126
+ That makes the harness its own competing writer. A commit is published within a minute of being
127
+ made, the agent cannot observe that from inside the container, and amending or resetting it
128
+ afterwards is ordinary git hygiene, so the final push used to be refused as a non-fast-forward and
129
+ failed the whole run with its work already on the branch.
130
+
131
+ Every push after the first therefore carries `--force-with-lease` against **the sha this pass
132
+ itself published**, never a tip it merely cloned. Two rules make that bound real, and both are
133
+ easy to get wrong:
134
+
135
+ - **The published sha comes from the push itself** (`pushBranch` names an explicit
136
+ `<sha>:refs/heads/<branch>` source and returns it), not from `refs/remotes/origin/<branch>`. A
137
+ fresh coding run clones a single branch, so `git push` creates no tracking ref for the work
138
+ branch and a lease read back from one never arms at all.
139
+ - **The lease is withheld unless the branch still contains the tip this pass started from**
140
+ (`workBranchLease`). Once a checkpoint has landed, a rewrite reaching below that tip would lease
141
+ successfully against our own commit and carry an earlier run's work away with it.
142
+
143
+ The run's own rewrite lands; a SECOND writer's commits, and a rewrite this pass cannot claim, still
144
+ refuse the push. A refused push is not reported as a generic `git` fault but as the
145
+ `branch-contended` failure cause, which the engine recovers from by re-dispatching the step onto
146
+ the branch as it now stands (bounded by `MAX_BRANCH_CONTENTION_RECOVERIES`, counted as
147
+ `container.branch_contended` and recorded on the step for the debug API). The agents are told the
148
+ matching half of the rule: add commits, never rewrite them (`PLATFORM_DELIVERY_CONTRACT`).
149
+
118
150
  ### Reference designs
119
151
 
120
152
  A job body for a kind that CAPTURES views (the UI tester, or a deployment's own browser-driven kind)
@@ -296,13 +328,14 @@ Kimi / DeepSeek) and meters spend. The provider key never enters the container.
296
328
  | `src/pi.ts` | Pi provider config, non-interactive run, JSON-line event + todo-progress parsing, global `AGENTS.md` guidance. |
297
329
  | `src/pi-reduction.ts` | Reducing a Pi event stream to what the run PRODUCED (summary, stats, diagnostics, terminal failure), FOLDED as records stream rather than over a retained array — memory is O(largest record), not O(records). The array-taking entry points offline tooling uses are defined in terms of the same reducer. |
298
330
  | `src/tool-silence.ts` | The tool-silence watchdog (F13) and the `ToolProgressWindow` an agent stream opens, beats and closes. Separate from the phase marker on purpose: a window is only meaningful while something able to reset it is running. |
299
- | `src/git.ts` | clone / branch / commit / push + GitHub PR creation; bootstrap history reset + force-push. |
331
+ | `src/git.ts` | clone / branch / commit / push (lease-guarded: [The work-branch push is CHECKPOINTED, so it is lease-guarded](#the-work-branch-push-is-checkpointed-so-it-is-lease-guarded)) + GitHub PR creation; bootstrap history reset + force-push. |
300
332
  | `src/bootstrap.ts` | The `/bootstrap` handler (clone-or-empty → adapt → reinit + force-push). |
301
333
  | `src/blueprint.ts` | The `/blueprint` handler (decompose → render `blueprints/` → commit on branch). |
302
334
  | `src/embed.ts` | Bundled assets/templates written into the workspace. |
303
335
  | `src/package-registries.ts` | Private-registry (npm) auth: renders the job's allowlisted entries into an npmrc; the user `~/.npmrc` in a container, a per-job file pointed at by `npm_config_userconfig` for a native job. |
304
336
  | `src/agent-runner.ts` | The subscription-harness runners (`runClaudeCode` / `runCodex`): talk direct to the vendor with a leased OAuth token, lift per-turn usage/telemetry off the CLI event stream. |
305
337
  | `src/claude-call-aggregator.ts` | Folds Claude Code's per-CONTENT-BLOCK `stream-json` envelopes back into the model calls they belong to (by `message.id`), reconstructs each call's request transcript, and routes subagent turns off the parent's chain. **Exported as the `./claude-call-aggregator` subpath and driven by the BACKEND too** (`runtimes/local`, for an inline step running on the developer's host `claude`), so it stays the ONE implementation: the per-envelope over-count it fixes inflated a measured 1.47M tokens to 5.53M, and both drivers have to learn that only once. That second driver is why the transcript is retained only to `MAX_TRANSCRIPT_CHARS` (stating what it stopped retaining) and why assembling bodies at all is a `bodies` switch: in a container the reconstruction is one job's memory in a box sized for it, in the backend it is per concurrent inline step in the orchestrator process. Unlike the compile-only `./embed`, this subpath is a `dist` import, which is why the package emits declarations, and why a consumer's typecheck depends on Turbo's `^build` edge having built this package first (see `tsconfig.json`'s `comment:buildOrder`). |
338
+ | `src/usage-attribution.ts` | Reconciles a subscription CLI's TWO token channels: the per-turn usage its stream narrates and the cumulative total its terminal event reports. They disagree routinely and in one direction (Claude Code's per-turn `output_tokens` is the message-START snapshot, single digits), so whatever the turns did not account for becomes ONE extra metric standing for the job (`standsForJob`, filed with a null turn index) rather than tokens grafted onto a real turn, which would make a derived number read as a measured one. Reconciled against the PARENT loop's calls alone, since the terminal cumulative covers only that conversation. |
306
339
  | `src/transcript-retention.ts` | Lifts the CLI session transcripts (`projects/` / `sessions/`) out of the isolated, credential-bearing config home before it is deleted, and prunes them on a TTL (debugging artifact retention). |
307
340
  | `src/captured-command.ts` | The one way the harness runs a declared shell command on its own behalf: `sh -c` with a per-command watchdog, abort handling, conventional exit codes (124/127/130) and a scrub-then-bound output capture. Shared by both pre-PR verification phases so a fix to one cannot miss the other. |
308
341
  | `src/dependency-install.ts` | Dependency prepopulation: `prepopulateDependencies` is the ONE seam every checkout-having mode calls; it runs the service's install command before the agent's first turn, excludes what the install materialised from git so no `git add -A` can sweep a dependency tree into the PR, and builds the prompt note describing the outcome. Best-effort: every failure shape becomes a note, never a failed job. Generic: keyed off the job body, never the agent kind. |
@@ -3,11 +3,12 @@ import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
3
3
  import { tmpdir } from 'node:os';
4
4
  import { dirname, join } from 'node:path';
5
5
  import { claudeAssistantContent, isObject, numberOf, redactBody } from './claude-stream.js';
6
- import { createClaudeRunTelemetry, subagentDispatchId } from './claude-call-aggregator.js';
6
+ import { claudeUsage, unaccountedUsageCall } from './usage-attribution.js';
7
+ import { createClaudeRunTelemetry, subagentDispatchId, } from './claude-call-aggregator.js';
7
8
  import { ToolCallTracker, recordClaudeToolResults, } from './tool-trajectory.js';
8
9
  import { log } from './logger.js';
9
10
  import { NO_TOOL_WINDOW } from './tool-silence.js';
10
- import { createCallMetricPublisher, publishCallMetric, } from './pi.js';
11
+ import { publishCallMetric, } from './pi.js';
11
12
  import { claudeAllowedToolPatterns, mcpServerSecretValues, observeClaudeMcpInit, writeClaudeMcpConfig, } from './agent-capabilities.js';
12
13
  import { codexImageGapNote, createCodexHome, disposeCodexHome } from './codex-home.js';
13
14
  import { ProgressGuard } from './progress-guard.js';
@@ -19,22 +20,6 @@ import { createSliceTracker, startSubagentWatcher } from './subagents.js';
19
20
  import { createTaskPlanTracker, mergeProgress, normalizeStatus, pickProgress, toProgress, todosToProgress, } from './progress.js';
20
21
  import { assertOnboardingKeysCurrent, writeOnboardingPreseed } from './onboarding-preseed.js';
21
22
  import { retainSessionTranscripts } from './transcript-retention.js';
22
- /**
23
- * Fallback token attribution: if a CLI reported a cumulative total but no per-turn
24
- * usage (so every captured call has zero tokens), pin the whole total onto the LAST
25
- * call rather than dropping it — the run's tokens are still accounted, just not split
26
- * per turn. A no-op when the calls already carry per-turn tokens.
27
- */
28
- function attributeCumulativeUsage(calls, usage) {
29
- if (!usage || calls.length === 0)
30
- return;
31
- const anyTokens = calls.some((c) => c.inputTokens > 0 || c.outputTokens > 0);
32
- if (anyTokens)
33
- return;
34
- const last = calls[calls.length - 1];
35
- last.inputTokens = usage.inputTokens;
36
- last.outputTokens = usage.outputTokens;
37
- }
38
23
  /**
39
24
  * Drive one CLI subprocess to completion, streaming LF-framed JSONL from stdout
40
25
  * through `onEvent`. Mirrors `runPi`'s lifecycle: prompt over stdin (out-of-band,
@@ -472,6 +457,59 @@ function openToolWindow(opts) {
472
457
  function carriesToolResult(content) {
473
458
  return content.some((block) => isObject(block) && block.type === 'tool_result');
474
459
  }
460
+ /**
461
+ * Open the per-call telemetry capture for one claude-code run.
462
+ *
463
+ * It reconstructs the full per-call request/response bodies from the stream.
464
+ * `--output-format stream-json --verbose` emits a near-verbatim Anthropic Messages envelope per
465
+ * response CONTENT BLOCK (not per call), so the aggregator folds the envelopes sharing a
466
+ * `message.id` back into one call and buffers that call's `user` tool_result turns — together the
467
+ * growing prompt transcript, in the shape the model was actually sent. It is SEEDED with the inputs
468
+ * the harness supplies (they never appear in the stream): the system + first user message when the
469
+ * prompt rides argv, or a single folded user turn when it doesn't, so the reconstruction never shows
470
+ * a system turn that was never sent. Bodies are credential-scrubbed (they can echo the leased token).
471
+ *
472
+ * The parent loop's calls are tracked SEPARATELY, by reference into the same list, because the
473
+ * terminal `result` event's cumulative usage covers only the parent conversation. In `ambientAuth`
474
+ * mode there is no transcript watcher, so the CLI's tagged subagent turns are captured here too and
475
+ * `calls` holds both; reconciling against that mixed list is what once billed a subagent for the
476
+ * parent's whole output shortfall.
477
+ */
478
+ function openClaudeCallCapture(opts, stream) {
479
+ const calls = [];
480
+ const parentCalls = [];
481
+ const publish = (metric) => publishCallMetric(calls, metric, opts.onCallMetric);
482
+ // `watcherOwnsSubagents` tracks the `startSubagentWatcher` wiring in the caller: it is started
483
+ // only when the CLI has an isolated config home to watch, which an `ambientAuth` run does not
484
+ // have. The telemetry routes the CLI's tagged subagent turns accordingly — see
485
+ // `createClaudeRunTelemetry`.
486
+ const telemetry = createClaudeRunTelemetry({
487
+ seed: stream.folded
488
+ ? [{ role: 'user', content: stream.prompt }]
489
+ : [
490
+ { role: 'system', content: opts.systemPrompt },
491
+ { role: 'user', content: opts.userPrompt },
492
+ ],
493
+ secrets: stream.secrets,
494
+ watcherOwnsSubagents: !opts.ambientAuth,
495
+ publish: (metric) => {
496
+ parentCalls.push(metric);
497
+ publish(metric);
498
+ },
499
+ publishSubagent: publish,
500
+ });
501
+ return {
502
+ calls,
503
+ telemetry,
504
+ settleUsage: (usage) => {
505
+ // Published like any other call so the live drain records it too, which is also what stamps
506
+ // its `seq` and therefore its stable row id.
507
+ const remainder = unaccountedUsageCall(parentCalls, usage);
508
+ if (remainder)
509
+ publish(remainder);
510
+ },
511
+ };
512
+ }
475
513
  export async function runClaudeCode(opts) {
476
514
  const stats = { toolCalls: 0, assistantChars: 0 };
477
515
  let summary = '';
@@ -488,34 +526,9 @@ export async function runClaudeCode(opts) {
488
526
  bytes: Buffer.byteLength(opts.systemPrompt, 'utf8'),
489
527
  });
490
528
  }
491
- // Reconstruct the full per-call request/response bodies for telemetry from the
492
- // stream. `--output-format stream-json --verbose` emits a near-verbatim Anthropic
493
- // Messages envelope per response CONTENT BLOCK (not per call), so the aggregator below
494
- // folds the envelopes sharing a `message.id` back into one call and buffers that call's
495
- // `user` tool_result turns — together the growing prompt transcript, in the shape the
496
- // model was actually sent. We seed it with the inputs the harness supplies (they never
497
- // appear in the stream): the system + first user message when the prompt rides argv, or
498
- // a single folded user turn when it doesn't — so the reconstruction never shows a system
499
- // turn that was never sent. Bodies are credential-scrubbed (they can echo the leased token).
500
529
  const secrets = opts.subscriptionToken ? secretsToRedact(opts.subscriptionToken) : [];
501
- const calls = [];
502
- // Streams each call as the CLI yields it, EXCEPT one whose tokens `attributeCumulativeUsage`
503
- // may still rewrite below (a published call must be final — see the publisher).
504
- const publisher = createCallMetricPublisher(calls, opts.onCallMetric);
505
- // `watcherOwnsSubagents` tracks the `startSubagentWatcher` wiring below: it is started only when
506
- // the CLI has an isolated config home to watch, which an `ambientAuth` run does not have. The
507
- // telemetry routes the CLI's tagged subagent turns accordingly — see `createClaudeRunTelemetry`.
508
- const telemetry = createClaudeRunTelemetry({
509
- seed: folded
510
- ? [{ role: 'user', content: prompt }]
511
- : [
512
- { role: 'system', content: opts.systemPrompt },
513
- { role: 'user', content: opts.userPrompt },
514
- ],
515
- secrets,
516
- watcherOwnsSubagents: !opts.ambientAuth,
517
- publish: (metric) => publisher.publish(metric),
518
- });
530
+ const capture = openClaudeCallCapture(opts, { prompt, folded, secrets });
531
+ const telemetry = capture.telemetry;
519
532
  // ADR 0026 D2.1 + ADR 0027 Defect B: surface live slice progress from the two views the run
520
533
  // produces of the SAME slicing. The parent's subagent dispatches + their terminal tool_results
521
534
  // appear on this stream (as do the subagents' own intermediate turns, tagged with the dispatch
@@ -668,8 +681,7 @@ export async function runClaudeCode(opts) {
668
681
  summary,
669
682
  stats,
670
683
  stderrTail,
671
- calls,
672
- publisher,
684
+ capture,
673
685
  usage,
674
686
  subagents,
675
687
  expectSubagentCalls: telemetry.expectsWatcherCalls(),
@@ -678,11 +690,11 @@ export async function runClaudeCode(opts) {
678
690
  }
679
691
  catch (err) {
680
692
  // The stream ended abnormally (guard trip, watchdog kill, CLI crash). Complete the call in
681
- // flight anyway, and release whatever the publisher was withholding: a killed run never
682
- // returns an outcome, so the live channel is the ONLY record of what it spent, and dropping
683
- // its last turn is what the streaming exists to avoid.
693
+ // flight anyway: a killed run never returns an outcome, so the live channel is the ONLY record
694
+ // of what it spent, and dropping its last turn is what the streaming exists to avoid. No
695
+ // terminal `result` event arrived, so there is no cumulative total to reconcile against and no
696
+ // remainder row to file — every captured turn already streamed as it was completed.
684
697
  telemetry.flush();
685
- publisher.flush();
686
698
  // A tripped no-progress guard aborted the CLI; streamCli rejects with its generic abort
687
699
  // message, so replace it with the guard's actionable diagnostic — carrying the stderr tail it
688
700
  // attached, since that is usually the only evidence of what the CLI was doing when it was
@@ -808,14 +820,11 @@ function buildClaudeEnv(opts, configHome) {
808
820
  * loop's telemetry must filter them (`subagentDispatchId`) for this concatenation to hold.
809
821
  */
810
822
  async function assembleClaudeOutcome(args) {
811
- const { summary, stats, stderrTail, calls, publisher, usage, subagents } = args;
812
- // The parent's cumulative-usage fallback applies to the PARENT calls only (before the
813
- // subagent calls, which carry their own per-turn tokens, are concatenated).
814
- attributeCumulativeUsage(calls, usage);
815
- // The withheld calls are final only NOW, so stream them: the completion poll drains them
816
- // alongside the result, and the backend records the attributed numbers rather than the zeros
817
- // they carried while the run was in flight.
818
- publisher.flush();
823
+ const { summary, stats, stderrTail, capture, usage, subagents } = args;
824
+ const calls = capture.calls;
825
+ // What the parent's narrated turns did not account for, as its OWN row (never tokens grafted onto
826
+ // a real turn).
827
+ capture.settleUsage(usage);
819
828
  // Final drain of any subagent transcript writes that landed after the last poll, then
820
829
  // fold the subagents' usage + per-call telemetry into the run's outcome.
821
830
  await subagents?.stop();
@@ -840,21 +849,6 @@ async function assembleClaudeOutcome(args) {
840
849
  ...(mergedCalls.length ? { callMetrics: mergedCalls } : {}),
841
850
  };
842
851
  }
843
- function claudeUsage(raw) {
844
- if (!isObject(raw))
845
- return undefined;
846
- // Count every input bucket Anthropic bills: fresh input plus BOTH cache reads and
847
- // cache writes (cache_creation_input_tokens), which are real consumed tokens — and
848
- // are the dominant share on a long agent run. Omitting them under-weights a token's
849
- // true load in the usage-aware rotation window.
850
- const input = numberOf(raw.input_tokens) +
851
- numberOf(raw.cache_read_input_tokens) +
852
- numberOf(raw.cache_creation_input_tokens);
853
- const output = numberOf(raw.output_tokens);
854
- if (input === 0 && output === 0)
855
- return undefined;
856
- return { inputTokens: input, outputTokens: output };
857
- }
858
852
  // ---------------------------------------------------------------------------
859
853
  // Codex
860
854
  // ---------------------------------------------------------------------------
@@ -157,8 +157,17 @@ export interface ClaudeRunTelemetry {
157
157
  * `ambientAuth` run has no isolated config home to watch — the tagged turns are recorded here
158
158
  * instead, on per-dispatch transcripts of their own. Dropping them in that case would leave the run
159
159
  * billed by neither channel, and an under-count reads as a cheap run rather than as an error.
160
+ *
161
+ * Which is also why the two are published through SEPARATE callbacks. A caller reconciling the
162
+ * parent's terminal cumulative usage needs the parent's calls alone, and with one shared callback
163
+ * the fallback channel silently mixed subagent turns into that list — where they both understated
164
+ * the shortfall and, being last, attracted it (`unaccountedUsageCall`). `publishSubagent` is
165
+ * optional so a caller that draws no distinction (a test, the settled-transcript path where nothing
166
+ * arrives here anyway) keeps one sink.
160
167
  */
161
168
  export declare function createClaudeRunTelemetry(opts: ClaudeStreamTelemetryOptions & {
162
169
  watcherOwnsSubagents: boolean;
170
+ /** Where a SUBAGENT conversation's call goes. Absent ⇒ `publish`, the parent's sink. */
171
+ publishSubagent?: (metric: HarnessCallMetric) => void;
163
172
  }): ClaudeRunTelemetry;
164
173
  export {};
@@ -311,10 +311,19 @@ function createSubagentStreamTelemetry(opts) {
311
311
  * `ambientAuth` run has no isolated config home to watch — the tagged turns are recorded here
312
312
  * instead, on per-dispatch transcripts of their own. Dropping them in that case would leave the run
313
313
  * billed by neither channel, and an under-count reads as a cheap run rather than as an error.
314
+ *
315
+ * Which is also why the two are published through SEPARATE callbacks. A caller reconciling the
316
+ * parent's terminal cumulative usage needs the parent's calls alone, and with one shared callback
317
+ * the fallback channel silently mixed subagent turns into that list — where they both understated
318
+ * the shortfall and, being last, attracted it (`unaccountedUsageCall`). `publishSubagent` is
319
+ * optional so a caller that draws no distinction (a test, the settled-transcript path where nothing
320
+ * arrives here anyway) keeps one sink.
314
321
  */
315
322
  export function createClaudeRunTelemetry(opts) {
316
323
  const parent = createClaudeStreamTelemetry(opts);
317
- const subagents = opts.watcherOwnsSubagents ? undefined : createSubagentStreamTelemetry(opts);
324
+ const subagents = opts.watcherOwnsSubagents
325
+ ? undefined
326
+ : createSubagentStreamTelemetry({ ...opts, publish: opts.publishSubagent ?? opts.publish });
318
327
  let sawSubagentTurn = false;
319
328
  return {
320
329
  onAssistant(dispatchId, message) {
@@ -1,7 +1,7 @@
1
1
  import { mkdir } from 'node:fs/promises';
2
2
  import { join } from 'node:path';
3
3
  import { runCapturedCommand } from './captured-command.js';
4
- import { branchAheadOfBase, changedFilesSinceBase, branchHasCommitsSince, cloneExistingBranch, cloneRepo, commitTrackedEdits, createBranch, excludeFromGit, fetchReferenceBranches, headCommit, listUntrackedFiles, prepareExistingCheckout, pushBranch, refreshFromBaseIfClean, remoteBranchExists, } from './git.js';
4
+ import { branchAheadOfBase, changedFilesSinceBase, branchHasCommitsSince, cloneExistingBranch, cloneRepo, commitTrackedEdits, createBranch, excludeFromGit, fetchReferenceBranches, headCommit, listUntrackedFiles, prepareExistingCheckout, pushBranch, refreshFromBaseIfClean, remoteBranchExists, unpublishedWorkBranchTip, workBranchLease, } from './git.js';
5
5
  import { FOLLOW_UPS_FILENAME, FollowUpTailer } from './follow-ups.js';
6
6
  import { EFFORT_REPORT_FILE } from './effort.js';
7
7
  import { PR_DESCRIPTION_FILE, readPrDescription, } from './pr-description.js';
@@ -51,24 +51,65 @@ function followUpPollIntervalMs() {
51
51
  * concurrently: overlapping pushes race on the remote ref and can make a push fail with a
52
52
  * ref-lock / non-fast-forward error — which, on the FINAL push, would fail the whole run even
53
53
  * though the work is committed. `pushWorkOnce` coalesces concurrent callers onto one push and only
54
- * pushes once the branch has advanced past `baseSha`.
54
+ * pushes what is UNPUBLISHED ({@link unpublishedWorkBranchTip}: past `baseSha`, and not already the
55
+ * tip the last push published).
55
56
  *
56
- * Only push once the branch has advanced past its pre-run tip: pushing while it still sits at
57
- * `baseSha` would create the work branch at the base commit (a zero-diff branch), which a later
58
- * retry would see via `remoteBranchExists` and treat as resumable work then fail to open a PR
59
- * ("no commits between base and head"). So a run that never commits leaves NO branch behind,
60
- * preserving the clean no-op outcome.
57
+ * Every push after the first LEASES against the sha this pass published (see
58
+ * {@link pushBranch}), because the checkpoint makes the harness its own competing writer: it
59
+ * publishes a commit within a minute of the agent making it, and the agent is then free to amend,
60
+ * reset or rebase that commit, which is perfectly ordinary git hygiene, and the delivery contract
61
+ * asks it to validate AFTER committing, exactly the sequence that produces an amend. Without the
62
+ * lease the final push is refused as a non-fast-forward and the whole run fails with its work
63
+ * already on the branch. The lease is what keeps that recovery from becoming a blanket `--force`:
64
+ * a SECOND writer (a concurrent dispatch for the same block) still refuses the push, which is the
65
+ * "never clobber another run's commits" property the resume design leans on.
66
+ *
67
+ * The lease alone does not bound the force to THIS pass's own commits, and that is the property
68
+ * the design promises, so it is checked rather than assumed: once one checkpoint has landed, a
69
+ * rewrite that drops `baseSha` (the tip the pass started from, which on a RESUMED branch is an
70
+ * earlier run's published work) would lease successfully against our own checkpoint and take the
71
+ * earlier commits with it. So the lease is armed only while the branch still CONTAINS `baseSha`;
72
+ * withheld, the push goes out plain, git refuses it, and the engine re-dispatches onto the branch
73
+ * as it stands. A rewrite this pass cannot prove is its own is never forced away.
74
+ *
75
+ * What is pushable is {@link unpublishedWorkBranchTip}'s question, and both of its answers matter
76
+ * here. A branch still at `baseSha` must not be pushed at all, or a later retry resumes a zero-diff
77
+ * branch and cannot open a PR for it. A branch already at the published tip has nothing to add, and
78
+ * skipping it is what keeps the interval a LOSS WINDOW rather than a push rate: an hour-long run
79
+ * that commits eight times pushes eight times, not sixty. That skip is invisible to the outcome by
80
+ * construction: `finalizeCodingRun` decides `pushed` from the BRANCH (advanced this pass, or
81
+ * resumed), never from whether the final call issued a `git push`, because a tip the checkpoint
82
+ * already published is published.
61
83
  */
62
84
  function createWorkBranchPusher(args) {
63
85
  const { dir, spec, baseSha, logger, signal } = args;
64
86
  let pushInFlight = null;
87
+ // The sha THIS pass last published to the work branch, and the only value it will ever lease a
88
+ // force push against. Starts unset even on a RESUMED branch: the tip we merely cloned is an
89
+ // earlier run's work, so a rewrite of it is refused (and re-driven) rather than forced away.
90
+ let publishedSha;
65
91
  const pushWorkOnce = () => {
66
92
  if (pushInFlight)
67
93
  return pushInFlight;
68
94
  pushInFlight = (async () => {
69
- if (!(await branchHasCommitsSince(dir, baseSha, signal)))
95
+ if (!(await unpublishedWorkBranchTip({ dir, baseSha, publishedSha, signal })))
70
96
  return;
71
- await pushBranch(dir, spec.pushBranch, spec.ghToken, signal);
97
+ // The rule the lease is entitled to lives beside the push ({@link workBranchLease}); the
98
+ // warn is here, because a withheld lease is how a rewrite this pass cannot claim fails the
99
+ // push it is about to make, and the run's log is where that is read.
100
+ const lease = await workBranchLease({
101
+ dir,
102
+ branch: spec.pushBranch,
103
+ baseSha,
104
+ publishedSha,
105
+ signal,
106
+ onWithheld: (probe) => logger.warn('coding-agent: push lease withheld, the branch dropped its pre-run tip', {
107
+ baseSha,
108
+ publishedSha,
109
+ probe,
110
+ }),
111
+ });
112
+ publishedSha = await pushBranch(dir, spec.pushBranch, spec.ghToken, signal, lease);
72
113
  })().finally(() => {
73
114
  pushInFlight = null;
74
115
  });
package/dist/failure.d.ts CHANGED
@@ -11,13 +11,19 @@
11
11
  * went quiet, this one says the model rabbit-holed while streaming.
12
12
  * - `agent` — the agent ran but produced an unusable/failed result, or threw.
13
13
  * - `git` — a git operation failed (clone/push/merge/PR).
14
+ * - `branch-contended`: a push to the work branch was REFUSED because the branch carries
15
+ * commits this push would drop (a second writer, or a rewrite of an
16
+ * earlier run's history). Split out of `git` because it is the one git
17
+ * fault the ENGINE can recover from on its own: re-dispatching the step
18
+ * resumes the branch as it now stands, where every other `git` failure
19
+ * would only fail again.
14
20
  * - `api` — an upstream API call failed (e.g. the GitHub/GitLab PR/MR REST call).
15
21
  * - `llm-upstream` — the model provider rejected every call (auth/quota/rate-limit) and Pi
16
22
  * exhausted its retries, so the run never produced a result.
17
23
  * - `no-usable-output` — the agent finished but returned no usable report / structured output.
18
24
  * - `no-changes` — a coding agent finished without producing any change to push.
19
25
  */
20
- export declare const FAILURE_CAUSES: readonly ['inactivity-timeout', 'max-duration', 'no-tool-progress', 'agent', 'git', 'api', 'llm-upstream', 'no-usable-output', 'no-changes'];
26
+ export declare const FAILURE_CAUSES: readonly ['inactivity-timeout', 'max-duration', 'no-tool-progress', 'agent', 'git', 'branch-contended', 'api', 'llm-upstream', 'no-usable-output', 'no-changes'];
21
27
  /**
22
28
  * See {@link FAILURE_CAUSES}. Derived from the array rather than declared beside it so the two
23
29
  * cannot disagree, and so the list is ENUMERABLE at runtime — which is what lets
package/dist/failure.js CHANGED
@@ -25,6 +25,12 @@
25
25
  * went quiet, this one says the model rabbit-holed while streaming.
26
26
  * - `agent` — the agent ran but produced an unusable/failed result, or threw.
27
27
  * - `git` — a git operation failed (clone/push/merge/PR).
28
+ * - `branch-contended`: a push to the work branch was REFUSED because the branch carries
29
+ * commits this push would drop (a second writer, or a rewrite of an
30
+ * earlier run's history). Split out of `git` because it is the one git
31
+ * fault the ENGINE can recover from on its own: re-dispatching the step
32
+ * resumes the branch as it now stands, where every other `git` failure
33
+ * would only fail again.
28
34
  * - `api` — an upstream API call failed (e.g. the GitHub/GitLab PR/MR REST call).
29
35
  * - `llm-upstream` — the model provider rejected every call (auth/quota/rate-limit) and Pi
30
36
  * exhausted its retries, so the run never produced a result.
@@ -37,6 +43,7 @@ export const FAILURE_CAUSES = [
37
43
  'no-tool-progress',
38
44
  'agent',
39
45
  'git',
46
+ 'branch-contended',
40
47
  'api',
41
48
  'llm-upstream',
42
49
  'no-usable-output',
package/dist/git.d.ts CHANGED
@@ -9,6 +9,27 @@ export declare const NON_INTERACTIVE_CREDENTIAL_ARGS: string[];
9
9
  * so it must NOT be reported here as a git timeout. Pure, so the classification is unit-tested.
10
10
  */
11
11
  export declare function isGitTimeoutKill(err: unknown, aborted: boolean): boolean;
12
+ /**
13
+ * Why a push to the work branch was REFUSED. Both mean the branch carries commits this push
14
+ * would drop, and git tells them apart by whether our object database HOLDS the tip the remote
15
+ * reports: it does for a tip our own checkout created, so the two are distinguishable and need
16
+ * different remedies (see {@link PUSH_REJECTION_REMEDIES}).
17
+ *
18
+ * - `local-rewrite`: we HAVE the remote's tip and are no longer descended from it, i.e. this
19
+ * checkout amended / reset / rebased a commit that had already been pushed. Git labels it
20
+ * `(non-fast-forward)`.
21
+ * - `remote-writer`: the remote's tip is a commit this checkout has never seen (`(fetch first)`),
22
+ * or our lease found the branch moved past what we published (`(stale info)`), so a SECOND
23
+ * writer owns the branch.
24
+ */
25
+ export type PushRejection = 'local-rewrite' | 'remote-writer';
26
+ /**
27
+ * Whether `stderr` is a REFUSED push, and which shape. Ordered: the lease/fetch-first shapes are
28
+ * checked first, because a `(stale info)` refusal also prints the generic "failed to push some
29
+ * refs" line the non-fast-forward shape shares. Pure, so both branches are unit-tested against
30
+ * real git output rather than inferred.
31
+ */
32
+ export declare function classifyPushRejection(stderr: string): PushRejection | undefined;
12
33
  /**
13
34
  * Classify the common shapes of git's own stderr into an actionable remedy, else undefined
14
35
  * (an unrecognized failure keeps just its raw stderr). This is the FIRST-WRAP-POINT for
@@ -370,10 +391,104 @@ export declare function fetchPullRequestHead(opts: {
370
391
  onSkip?: (reason: string) => void;
371
392
  }): Promise<boolean>;
372
393
  /**
373
- * Push the work branch to origin. The remote URL carries only the username, so
374
- * the token is supplied here via the askpass env (never in argv).
394
+ * Push the work branch to origin and return the sha it PUBLISHED. The remote URL carries only the
395
+ * username, so the token is supplied here via the askpass env (never in argv).
396
+ *
397
+ * The push names an explicit SOURCE COMMIT (`<sha>:refs/heads/<branch>`) rather than the branch,
398
+ * which is what makes the return value exact rather than a guess. The agent commits while this
399
+ * runs, so `git push origin <branch>` publishes whatever the branch ref holds at the moment git
400
+ * reads it, and a caller that leases against a sha it read either side of that has leased against
401
+ * the wrong commit. Reading it back from `refs/remotes/origin/<branch>` afterwards is worse than
402
+ * inexact, it is EMPTY on the production checkout: a fresh coding run clones one branch
403
+ * (`cloneRepo`), so the remote's fetch refspec covers the base alone and `git push` creates no
404
+ * tracking ref for the work branch at all. Naming the sha needs no ref and no round trip.
405
+ *
406
+ * `-u` goes with it: with a non-branch source git sets no upstream config (verified), nothing in
407
+ * the harness reads that config, and the agent is told never to push or pull.
408
+ *
409
+ * `expectRemoteSha` turns the push into a LEASED force (`--force-with-lease=<branch>:<sha>`), which
410
+ * is how a run whose own checkpoint push it has since rewritten still lands. It is deliberately NOT
411
+ * a plain `--force`: the lease succeeds only while the remote still holds the sha THIS run
412
+ * published, so a second writer's commits refuse the push (`(stale info)`) instead of being
413
+ * clobbered. Callers therefore pass only a sha this same pass published; leasing against a tip we
414
+ * merely CLONED would force over an earlier run's work.
415
+ */
416
+ export declare function pushBranch(dir: string, branch: string, ghToken: string, signal?: AbortSignal, opts?: {
417
+ expectRemoteSha?: string;
418
+ }): Promise<string>;
419
+ /**
420
+ * Whether `sha` is still reachable from `branch`'s tip, i.e. the branch CONTAINS it:
421
+ * `git rev-list --count --max-count=1 <sha> --not refs/heads/<branch>` is 0 when everything
422
+ * reachable from `sha` is reachable from the branch too (the tip itself counts as contained).
423
+ *
424
+ * Phrased as a rev-list rather than `merge-base --is-ancestor` on purpose: the latter answers "no"
425
+ * by EXITING 1, which is indistinguishable here from a broken checkout, and this probe's whole job
426
+ * is to be trusted only when it is a definite answer. Tri-state for the same reason (as
427
+ * {@link branchAheadOfBase} is):
428
+ *
429
+ * - `true`: confirmed contained.
430
+ * - `false`: confirmed dropped, so the branch was rewritten below `sha`.
431
+ * - `undefined`: could not determine (an unknown object, a rev-list error). A caller must not read
432
+ * a failed probe as either answer.
433
+ *
434
+ * The work-branch lease is gated on this: see {@link workBranchLease}.
435
+ */
436
+ export declare function branchContainsCommit(dir: string, branch: string, sha: string, signal?: AbortSignal): Promise<boolean | undefined>;
437
+ /**
438
+ * The work branch's tip when it holds something UNPUBLISHED, else undefined: the answer to whether a
439
+ * checkpoint tick has anything to do. Two ways of having nothing:
440
+ *
441
+ * - the tip is still `baseSha`, so this pass has committed nothing. Pushing here would create the
442
+ * work branch at the base commit, and a later retry would see that zero-diff branch via
443
+ * `remoteBranchExists`, resume it as work, and fail to open a PR ("no commits between base and
444
+ * head"). A pass that never commits must leave NO branch behind.
445
+ * - the tip is `publishedSha`, so the last push already published it. Without this the checkpoint
446
+ * re-pushed an unchanged branch on every tick: an hour-long run committing eight times issued
447
+ * ~60 pushes, ~52 of them a full authenticated round trip answering "Everything up-to-date",
448
+ * each one counting against the host's push rate limits.
449
+ *
450
+ * That second condition is also what keeps the INTERVAL the right knob. It expresses the acceptable
451
+ * loss window when a container dies (a property of the deployment's infra churn), not a rate: gated
452
+ * this way, the tick publishes at most one push per commit the agent makes, whatever the model or
453
+ * the run's length, so nothing here needs to be tuned per model.
454
+ */
455
+ export declare function unpublishedWorkBranchTip(args: {
456
+ dir: string;
457
+ /** The branch tip this pass started from. */
458
+ baseSha: string;
459
+ /** The sha this pass published, if any ({@link pushBranch}'s return). */
460
+ publishedSha: string | undefined;
461
+ signal?: AbortSignal;
462
+ }): Promise<string | undefined>;
463
+ /**
464
+ * The lease a work-branch push is entitled to (the `opts` {@link pushBranch} takes): the sha this
465
+ * pass last published, and nothing at all before it has published one.
466
+ *
467
+ * The extra condition is what bounds the force to THIS pass's own commits, which the lease alone
468
+ * does not do and the design promises. Once one checkpoint has landed, a rewrite that drops
469
+ * `baseSha` (the tip the pass started from, which on a RESUMED branch is an earlier run's published
470
+ * work) would still lease successfully against our own checkpoint and carry those earlier commits
471
+ * away with it. So the lease is withheld unless the branch still CONTAINS `baseSha`: the push then
472
+ * goes out plain, git refuses it as a non-fast-forward, and the engine re-dispatches onto the
473
+ * branch as it stands.
474
+ *
475
+ * A probe that could not answer withholds it too (`onWithheld('unreadable')`), because the two
476
+ * mistakes are not symmetric: withholding costs a refused rewrite and one re-dispatch, trusting an
477
+ * unreadable probe costs commits.
375
478
  */
376
- export declare function pushBranch(dir: string, branch: string, ghToken: string, signal?: AbortSignal): Promise<void>;
479
+ export declare function workBranchLease(args: {
480
+ dir: string;
481
+ branch: string;
482
+ /** The branch tip this pass started from. */
483
+ baseSha: string;
484
+ /** The sha this pass published, if any (`pushBranch`'s return). */
485
+ publishedSha: string | undefined;
486
+ signal?: AbortSignal;
487
+ /** Told why the lease was withheld, so the harness can log it with its own logger. */
488
+ onWithheld?: (probe: 'unreadable' | 'dropped') => void;
489
+ }): Promise<{
490
+ expectRemoteSha?: string;
491
+ }>;
377
492
  /**
378
493
  * Reset the working tree's git history to a single bootstrap commit and push it
379
494
  * to the target repository's default branch. Wiping `.git` before re-initialising