@cat-factory/executor-harness 1.112.0 → 1.116.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -1
- package/dist/coding-agent.js +50 -9
- package/dist/failure.d.ts +7 -1
- package/dist/failure.js +7 -0
- package/dist/git.d.ts +118 -3
- package/dist/git.js +162 -9
- package/package.json +4 -4
- package/src/coding-agent.ts +52 -8
- package/src/failure.ts +7 -0
- package/src/git.ts +212 -9
package/README.md
CHANGED
|
@@ -115,6 +115,38 @@ Bootstrap differs at the ends: it may start from an empty dir, and **resets
|
|
|
115
115
|
history to one commit and force-pushes** the default branch instead of opening a
|
|
116
116
|
PR. Blueprint **commits onto a branch** (no history reset) and returns the tree.
|
|
117
117
|
|
|
118
|
+
### The work-branch push is CHECKPOINTED, so it is lease-guarded
|
|
119
|
+
|
|
120
|
+
Step 8's push is not the run's first: every `JOB_CHECKPOINT_INTERVAL_MS` (60s) the harness pushes
|
|
121
|
+
whatever the agent has committed and NOT yet published, so an evicted container's work survives on
|
|
122
|
+
the branch and a retry resumes on top of it. The interval is a **loss window**, not a push rate:
|
|
123
|
+
`unpublishedWorkBranchTip` skips a tick whose branch tip is already published, so a long run pushes
|
|
124
|
+
once per commit the agent makes rather than once a minute, and nothing here needs tuning per model.
|
|
125
|
+
|
|
126
|
+
That makes the harness its own competing writer. A commit is published within a minute of being
|
|
127
|
+
made, the agent cannot observe that from inside the container, and amending or resetting it
|
|
128
|
+
afterwards is ordinary git hygiene, so the final push used to be refused as a non-fast-forward and
|
|
129
|
+
failed the whole run with its work already on the branch.
|
|
130
|
+
|
|
131
|
+
Every push after the first therefore carries `--force-with-lease` against **the sha this pass
|
|
132
|
+
itself published**, never a tip it merely cloned. Two rules make that bound real, and both are
|
|
133
|
+
easy to get wrong:
|
|
134
|
+
|
|
135
|
+
- **The published sha comes from the push itself** (`pushBranch` names an explicit
|
|
136
|
+
`<sha>:refs/heads/<branch>` source and returns it), not from `refs/remotes/origin/<branch>`. A
|
|
137
|
+
fresh coding run clones a single branch, so `git push` creates no tracking ref for the work
|
|
138
|
+
branch and a lease read back from one never arms at all.
|
|
139
|
+
- **The lease is withheld unless the branch still contains the tip this pass started from**
|
|
140
|
+
(`workBranchLease`). Once a checkpoint has landed, a rewrite reaching below that tip would lease
|
|
141
|
+
successfully against our own commit and carry an earlier run's work away with it.
|
|
142
|
+
|
|
143
|
+
The run's own rewrite lands; a SECOND writer's commits, and a rewrite this pass cannot claim, still
|
|
144
|
+
refuse the push. A refused push is not reported as a generic `git` fault but as the
|
|
145
|
+
`branch-contended` failure cause, which the engine recovers from by re-dispatching the step onto
|
|
146
|
+
the branch as it now stands (bounded by `MAX_BRANCH_CONTENTION_RECOVERIES`, counted as
|
|
147
|
+
`container.branch_contended` and recorded on the step for the debug API). The agents are told the
|
|
148
|
+
matching half of the rule: add commits, never rewrite them (`PLATFORM_DELIVERY_CONTRACT`).
|
|
149
|
+
|
|
118
150
|
### Reference designs
|
|
119
151
|
|
|
120
152
|
A job body for a kind that CAPTURES views (the UI tester, or a deployment's own browser-driven kind)
|
|
@@ -296,7 +328,7 @@ Kimi / DeepSeek) and meters spend. The provider key never enters the container.
|
|
|
296
328
|
| `src/pi.ts` | Pi provider config, non-interactive run, JSON-line event + todo-progress parsing, global `AGENTS.md` guidance. |
|
|
297
329
|
| `src/pi-reduction.ts` | Reducing a Pi event stream to what the run PRODUCED (summary, stats, diagnostics, terminal failure), FOLDED as records stream rather than over a retained array — memory is O(largest record), not O(records). The array-taking entry points offline tooling uses are defined in terms of the same reducer. |
|
|
298
330
|
| `src/tool-silence.ts` | The tool-silence watchdog (F13) and the `ToolProgressWindow` an agent stream opens, beats and closes. Separate from the phase marker on purpose: a window is only meaningful while something able to reset it is running. |
|
|
299
|
-
| `src/git.ts` | clone / branch / commit / push + GitHub PR creation; bootstrap history reset + force-push.
|
|
331
|
+
| `src/git.ts` | clone / branch / commit / push (lease-guarded: [The work-branch push is CHECKPOINTED, so it is lease-guarded](#the-work-branch-push-is-checkpointed-so-it-is-lease-guarded)) + GitHub PR creation; bootstrap history reset + force-push. |
|
|
300
332
|
| `src/bootstrap.ts` | The `/bootstrap` handler (clone-or-empty → adapt → reinit + force-push). |
|
|
301
333
|
| `src/blueprint.ts` | The `/blueprint` handler (decompose → render `blueprints/` → commit on branch). |
|
|
302
334
|
| `src/embed.ts` | Bundled assets/templates written into the workspace. |
|
package/dist/coding-agent.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { mkdir } from 'node:fs/promises';
|
|
2
2
|
import { join } from 'node:path';
|
|
3
3
|
import { runCapturedCommand } from './captured-command.js';
|
|
4
|
-
import { branchAheadOfBase, changedFilesSinceBase, branchHasCommitsSince, cloneExistingBranch, cloneRepo, commitTrackedEdits, createBranch, excludeFromGit, fetchReferenceBranches, headCommit, listUntrackedFiles, prepareExistingCheckout, pushBranch, refreshFromBaseIfClean, remoteBranchExists, } from './git.js';
|
|
4
|
+
import { branchAheadOfBase, changedFilesSinceBase, branchHasCommitsSince, cloneExistingBranch, cloneRepo, commitTrackedEdits, createBranch, excludeFromGit, fetchReferenceBranches, headCommit, listUntrackedFiles, prepareExistingCheckout, pushBranch, refreshFromBaseIfClean, remoteBranchExists, unpublishedWorkBranchTip, workBranchLease, } from './git.js';
|
|
5
5
|
import { FOLLOW_UPS_FILENAME, FollowUpTailer } from './follow-ups.js';
|
|
6
6
|
import { EFFORT_REPORT_FILE } from './effort.js';
|
|
7
7
|
import { PR_DESCRIPTION_FILE, readPrDescription, } from './pr-description.js';
|
|
@@ -51,24 +51,65 @@ function followUpPollIntervalMs() {
|
|
|
51
51
|
* concurrently: overlapping pushes race on the remote ref and can make a push fail with a
|
|
52
52
|
* ref-lock / non-fast-forward error — which, on the FINAL push, would fail the whole run even
|
|
53
53
|
* though the work is committed. `pushWorkOnce` coalesces concurrent callers onto one push and only
|
|
54
|
-
* pushes
|
|
54
|
+
* pushes what is UNPUBLISHED ({@link unpublishedWorkBranchTip}: past `baseSha`, and not already the
|
|
55
|
+
* tip the last push published).
|
|
55
56
|
*
|
|
56
|
-
*
|
|
57
|
-
*
|
|
58
|
-
*
|
|
59
|
-
*
|
|
60
|
-
*
|
|
57
|
+
* Every push after the first LEASES against the sha this pass published (see
|
|
58
|
+
* {@link pushBranch}), because the checkpoint makes the harness its own competing writer: it
|
|
59
|
+
* publishes a commit within a minute of the agent making it, and the agent is then free to amend,
|
|
60
|
+
* reset or rebase that commit, which is perfectly ordinary git hygiene, and the delivery contract
|
|
61
|
+
* asks it to validate AFTER committing, exactly the sequence that produces an amend. Without the
|
|
62
|
+
* lease the final push is refused as a non-fast-forward and the whole run fails with its work
|
|
63
|
+
* already on the branch. The lease is what keeps that recovery from becoming a blanket `--force`:
|
|
64
|
+
* a SECOND writer (a concurrent dispatch for the same block) still refuses the push, which is the
|
|
65
|
+
* "never clobber another run's commits" property the resume design leans on.
|
|
66
|
+
*
|
|
67
|
+
* The lease alone does not bound the force to THIS pass's own commits, and that is the property
|
|
68
|
+
* the design promises, so it is checked rather than assumed: once one checkpoint has landed, a
|
|
69
|
+
* rewrite that drops `baseSha` (the tip the pass started from, which on a RESUMED branch is an
|
|
70
|
+
* earlier run's published work) would lease successfully against our own checkpoint and take the
|
|
71
|
+
* earlier commits with it. So the lease is armed only while the branch still CONTAINS `baseSha`;
|
|
72
|
+
* withheld, the push goes out plain, git refuses it, and the engine re-dispatches onto the branch
|
|
73
|
+
* as it stands. A rewrite this pass cannot prove is its own is never forced away.
|
|
74
|
+
*
|
|
75
|
+
* What is pushable is {@link unpublishedWorkBranchTip}'s question, and both of its answers matter
|
|
76
|
+
* here. A branch still at `baseSha` must not be pushed at all, or a later retry resumes a zero-diff
|
|
77
|
+
* branch and cannot open a PR for it. A branch already at the published tip has nothing to add, and
|
|
78
|
+
* skipping it is what keeps the interval a LOSS WINDOW rather than a push rate: an hour-long run
|
|
79
|
+
* that commits eight times pushes eight times, not sixty. That skip is invisible to the outcome by
|
|
80
|
+
* construction: `finalizeCodingRun` decides `pushed` from the BRANCH (advanced this pass, or
|
|
81
|
+
* resumed), never from whether the final call issued a `git push`, because a tip the checkpoint
|
|
82
|
+
* already published is published.
|
|
61
83
|
*/
|
|
62
84
|
function createWorkBranchPusher(args) {
|
|
63
85
|
const { dir, spec, baseSha, logger, signal } = args;
|
|
64
86
|
let pushInFlight = null;
|
|
87
|
+
// The sha THIS pass last published to the work branch, and the only value it will ever lease a
|
|
88
|
+
// force push against. Starts unset even on a RESUMED branch: the tip we merely cloned is an
|
|
89
|
+
// earlier run's work, so a rewrite of it is refused (and re-driven) rather than forced away.
|
|
90
|
+
let publishedSha;
|
|
65
91
|
const pushWorkOnce = () => {
|
|
66
92
|
if (pushInFlight)
|
|
67
93
|
return pushInFlight;
|
|
68
94
|
pushInFlight = (async () => {
|
|
69
|
-
if (!(await
|
|
95
|
+
if (!(await unpublishedWorkBranchTip({ dir, baseSha, publishedSha, signal })))
|
|
70
96
|
return;
|
|
71
|
-
|
|
97
|
+
// The rule the lease is entitled to lives beside the push ({@link workBranchLease}); the
|
|
98
|
+
// warn is here, because a withheld lease is how a rewrite this pass cannot claim fails the
|
|
99
|
+
// push it is about to make, and the run's log is where that is read.
|
|
100
|
+
const lease = await workBranchLease({
|
|
101
|
+
dir,
|
|
102
|
+
branch: spec.pushBranch,
|
|
103
|
+
baseSha,
|
|
104
|
+
publishedSha,
|
|
105
|
+
signal,
|
|
106
|
+
onWithheld: (probe) => logger.warn('coding-agent: push lease withheld, the branch dropped its pre-run tip', {
|
|
107
|
+
baseSha,
|
|
108
|
+
publishedSha,
|
|
109
|
+
probe,
|
|
110
|
+
}),
|
|
111
|
+
});
|
|
112
|
+
publishedSha = await pushBranch(dir, spec.pushBranch, spec.ghToken, signal, lease);
|
|
72
113
|
})().finally(() => {
|
|
73
114
|
pushInFlight = null;
|
|
74
115
|
});
|
package/dist/failure.d.ts
CHANGED
|
@@ -11,13 +11,19 @@
|
|
|
11
11
|
* went quiet, this one says the model rabbit-holed while streaming.
|
|
12
12
|
* - `agent` — the agent ran but produced an unusable/failed result, or threw.
|
|
13
13
|
* - `git` — a git operation failed (clone/push/merge/PR).
|
|
14
|
+
* - `branch-contended`: a push to the work branch was REFUSED because the branch carries
|
|
15
|
+
* commits this push would drop (a second writer, or a rewrite of an
|
|
16
|
+
* earlier run's history). Split out of `git` because it is the one git
|
|
17
|
+
* fault the ENGINE can recover from on its own: re-dispatching the step
|
|
18
|
+
* resumes the branch as it now stands, where every other `git` failure
|
|
19
|
+
* would only fail again.
|
|
14
20
|
* - `api` — an upstream API call failed (e.g. the GitHub/GitLab PR/MR REST call).
|
|
15
21
|
* - `llm-upstream` — the model provider rejected every call (auth/quota/rate-limit) and Pi
|
|
16
22
|
* exhausted its retries, so the run never produced a result.
|
|
17
23
|
* - `no-usable-output` — the agent finished but returned no usable report / structured output.
|
|
18
24
|
* - `no-changes` — a coding agent finished without producing any change to push.
|
|
19
25
|
*/
|
|
20
|
-
export declare const FAILURE_CAUSES: readonly ['inactivity-timeout', 'max-duration', 'no-tool-progress', 'agent', 'git', 'api', 'llm-upstream', 'no-usable-output', 'no-changes'];
|
|
26
|
+
export declare const FAILURE_CAUSES: readonly ['inactivity-timeout', 'max-duration', 'no-tool-progress', 'agent', 'git', 'branch-contended', 'api', 'llm-upstream', 'no-usable-output', 'no-changes'];
|
|
21
27
|
/**
|
|
22
28
|
* See {@link FAILURE_CAUSES}. Derived from the array rather than declared beside it so the two
|
|
23
29
|
* cannot disagree, and so the list is ENUMERABLE at runtime — which is what lets
|
package/dist/failure.js
CHANGED
|
@@ -25,6 +25,12 @@
|
|
|
25
25
|
* went quiet, this one says the model rabbit-holed while streaming.
|
|
26
26
|
* - `agent` — the agent ran but produced an unusable/failed result, or threw.
|
|
27
27
|
* - `git` — a git operation failed (clone/push/merge/PR).
|
|
28
|
+
* - `branch-contended`: a push to the work branch was REFUSED because the branch carries
|
|
29
|
+
* commits this push would drop (a second writer, or a rewrite of an
|
|
30
|
+
* earlier run's history). Split out of `git` because it is the one git
|
|
31
|
+
* fault the ENGINE can recover from on its own: re-dispatching the step
|
|
32
|
+
* resumes the branch as it now stands, where every other `git` failure
|
|
33
|
+
* would only fail again.
|
|
28
34
|
* - `api` — an upstream API call failed (e.g. the GitHub/GitLab PR/MR REST call).
|
|
29
35
|
* - `llm-upstream` — the model provider rejected every call (auth/quota/rate-limit) and Pi
|
|
30
36
|
* exhausted its retries, so the run never produced a result.
|
|
@@ -37,6 +43,7 @@ export const FAILURE_CAUSES = [
|
|
|
37
43
|
'no-tool-progress',
|
|
38
44
|
'agent',
|
|
39
45
|
'git',
|
|
46
|
+
'branch-contended',
|
|
40
47
|
'api',
|
|
41
48
|
'llm-upstream',
|
|
42
49
|
'no-usable-output',
|
package/dist/git.d.ts
CHANGED
|
@@ -9,6 +9,27 @@ export declare const NON_INTERACTIVE_CREDENTIAL_ARGS: string[];
|
|
|
9
9
|
* so it must NOT be reported here as a git timeout. Pure, so the classification is unit-tested.
|
|
10
10
|
*/
|
|
11
11
|
export declare function isGitTimeoutKill(err: unknown, aborted: boolean): boolean;
|
|
12
|
+
/**
|
|
13
|
+
* Why a push to the work branch was REFUSED. Both mean the branch carries commits this push
|
|
14
|
+
* would drop, and git tells them apart by whether our object database HOLDS the tip the remote
|
|
15
|
+
* reports: it does for a tip our own checkout created, so the two are distinguishable and need
|
|
16
|
+
* different remedies (see {@link PUSH_REJECTION_REMEDIES}).
|
|
17
|
+
*
|
|
18
|
+
* - `local-rewrite`: we HAVE the remote's tip and are no longer descended from it, i.e. this
|
|
19
|
+
* checkout amended / reset / rebased a commit that had already been pushed. Git labels it
|
|
20
|
+
* `(non-fast-forward)`.
|
|
21
|
+
* - `remote-writer`: the remote's tip is a commit this checkout has never seen (`(fetch first)`),
|
|
22
|
+
* or our lease found the branch moved past what we published (`(stale info)`), so a SECOND
|
|
23
|
+
* writer owns the branch.
|
|
24
|
+
*/
|
|
25
|
+
export type PushRejection = 'local-rewrite' | 'remote-writer';
|
|
26
|
+
/**
|
|
27
|
+
* Whether `stderr` is a REFUSED push, and which shape. Ordered: the lease/fetch-first shapes are
|
|
28
|
+
* checked first, because a `(stale info)` refusal also prints the generic "failed to push some
|
|
29
|
+
* refs" line the non-fast-forward shape shares. Pure, so both branches are unit-tested against
|
|
30
|
+
* real git output rather than inferred.
|
|
31
|
+
*/
|
|
32
|
+
export declare function classifyPushRejection(stderr: string): PushRejection | undefined;
|
|
12
33
|
/**
|
|
13
34
|
* Classify the common shapes of git's own stderr into an actionable remedy, else undefined
|
|
14
35
|
* (an unrecognized failure keeps just its raw stderr). This is the FIRST-WRAP-POINT for
|
|
@@ -370,10 +391,104 @@ export declare function fetchPullRequestHead(opts: {
|
|
|
370
391
|
onSkip?: (reason: string) => void;
|
|
371
392
|
}): Promise<boolean>;
|
|
372
393
|
/**
|
|
373
|
-
* Push the work branch to origin. The remote URL carries only the
|
|
374
|
-
* the token is supplied here via the askpass env (never in argv).
|
|
394
|
+
* Push the work branch to origin and return the sha it PUBLISHED. The remote URL carries only the
|
|
395
|
+
* username, so the token is supplied here via the askpass env (never in argv).
|
|
396
|
+
*
|
|
397
|
+
* The push names an explicit SOURCE COMMIT (`<sha>:refs/heads/<branch>`) rather than the branch,
|
|
398
|
+
* which is what makes the return value exact rather than a guess. The agent commits while this
|
|
399
|
+
* runs, so `git push origin <branch>` publishes whatever the branch ref holds at the moment git
|
|
400
|
+
* reads it, and a caller that leases against a sha it read either side of that has leased against
|
|
401
|
+
* the wrong commit. Reading it back from `refs/remotes/origin/<branch>` afterwards is worse than
|
|
402
|
+
* inexact, it is EMPTY on the production checkout: a fresh coding run clones one branch
|
|
403
|
+
* (`cloneRepo`), so the remote's fetch refspec covers the base alone and `git push` creates no
|
|
404
|
+
* tracking ref for the work branch at all. Naming the sha needs no ref and no round trip.
|
|
405
|
+
*
|
|
406
|
+
* `-u` goes with it: with a non-branch source git sets no upstream config (verified), nothing in
|
|
407
|
+
* the harness reads that config, and the agent is told never to push or pull.
|
|
408
|
+
*
|
|
409
|
+
* `expectRemoteSha` turns the push into a LEASED force (`--force-with-lease=<branch>:<sha>`), which
|
|
410
|
+
* is how a run whose own checkpoint push it has since rewritten still lands. It is deliberately NOT
|
|
411
|
+
* a plain `--force`: the lease succeeds only while the remote still holds the sha THIS run
|
|
412
|
+
* published, so a second writer's commits refuse the push (`(stale info)`) instead of being
|
|
413
|
+
* clobbered. Callers therefore pass only a sha this same pass published; leasing against a tip we
|
|
414
|
+
* merely CLONED would force over an earlier run's work.
|
|
415
|
+
*/
|
|
416
|
+
export declare function pushBranch(dir: string, branch: string, ghToken: string, signal?: AbortSignal, opts?: {
|
|
417
|
+
expectRemoteSha?: string;
|
|
418
|
+
}): Promise<string>;
|
|
419
|
+
/**
|
|
420
|
+
* Whether `sha` is still reachable from `branch`'s tip, i.e. the branch CONTAINS it:
|
|
421
|
+
* `git rev-list --count --max-count=1 <sha> --not refs/heads/<branch>` is 0 when everything
|
|
422
|
+
* reachable from `sha` is reachable from the branch too (the tip itself counts as contained).
|
|
423
|
+
*
|
|
424
|
+
* Phrased as a rev-list rather than `merge-base --is-ancestor` on purpose: the latter answers "no"
|
|
425
|
+
* by EXITING 1, which is indistinguishable here from a broken checkout, and this probe's whole job
|
|
426
|
+
* is to be trusted only when it is a definite answer. Tri-state for the same reason (as
|
|
427
|
+
* {@link branchAheadOfBase} is):
|
|
428
|
+
*
|
|
429
|
+
* - `true`: confirmed contained.
|
|
430
|
+
* - `false`: confirmed dropped, so the branch was rewritten below `sha`.
|
|
431
|
+
* - `undefined`: could not determine (an unknown object, a rev-list error). A caller must not read
|
|
432
|
+
* a failed probe as either answer.
|
|
433
|
+
*
|
|
434
|
+
* The work-branch lease is gated on this: see {@link workBranchLease}.
|
|
435
|
+
*/
|
|
436
|
+
export declare function branchContainsCommit(dir: string, branch: string, sha: string, signal?: AbortSignal): Promise<boolean | undefined>;
|
|
437
|
+
/**
|
|
438
|
+
* The work branch's tip when it holds something UNPUBLISHED, else undefined: the answer to whether a
|
|
439
|
+
* checkpoint tick has anything to do. Two ways of having nothing:
|
|
440
|
+
*
|
|
441
|
+
* - the tip is still `baseSha`, so this pass has committed nothing. Pushing here would create the
|
|
442
|
+
* work branch at the base commit, and a later retry would see that zero-diff branch via
|
|
443
|
+
* `remoteBranchExists`, resume it as work, and fail to open a PR ("no commits between base and
|
|
444
|
+
* head"). A pass that never commits must leave NO branch behind.
|
|
445
|
+
* - the tip is `publishedSha`, so the last push already published it. Without this the checkpoint
|
|
446
|
+
* re-pushed an unchanged branch on every tick: an hour-long run committing eight times issued
|
|
447
|
+
* ~60 pushes, ~52 of them a full authenticated round trip answering "Everything up-to-date",
|
|
448
|
+
* each one counting against the host's push rate limits.
|
|
449
|
+
*
|
|
450
|
+
* That second condition is also what keeps the INTERVAL the right knob. It expresses the acceptable
|
|
451
|
+
* loss window when a container dies (a property of the deployment's infra churn), not a rate: gated
|
|
452
|
+
* this way, the tick publishes at most one push per commit the agent makes, whatever the model or
|
|
453
|
+
* the run's length, so nothing here needs to be tuned per model.
|
|
454
|
+
*/
|
|
455
|
+
export declare function unpublishedWorkBranchTip(args: {
|
|
456
|
+
dir: string;
|
|
457
|
+
/** The branch tip this pass started from. */
|
|
458
|
+
baseSha: string;
|
|
459
|
+
/** The sha this pass published, if any ({@link pushBranch}'s return). */
|
|
460
|
+
publishedSha: string | undefined;
|
|
461
|
+
signal?: AbortSignal;
|
|
462
|
+
}): Promise<string | undefined>;
|
|
463
|
+
/**
|
|
464
|
+
* The lease a work-branch push is entitled to (the `opts` {@link pushBranch} takes): the sha this
|
|
465
|
+
* pass last published, and nothing at all before it has published one.
|
|
466
|
+
*
|
|
467
|
+
* The extra condition is what bounds the force to THIS pass's own commits, which the lease alone
|
|
468
|
+
* does not do and the design promises. Once one checkpoint has landed, a rewrite that drops
|
|
469
|
+
* `baseSha` (the tip the pass started from, which on a RESUMED branch is an earlier run's published
|
|
470
|
+
* work) would still lease successfully against our own checkpoint and carry those earlier commits
|
|
471
|
+
* away with it. So the lease is withheld unless the branch still CONTAINS `baseSha`: the push then
|
|
472
|
+
* goes out plain, git refuses it as a non-fast-forward, and the engine re-dispatches onto the
|
|
473
|
+
* branch as it stands.
|
|
474
|
+
*
|
|
475
|
+
* A probe that could not answer withholds it too (`onWithheld('unreadable')`), because the two
|
|
476
|
+
* mistakes are not symmetric: withholding costs a refused rewrite and one re-dispatch, trusting an
|
|
477
|
+
* unreadable probe costs commits.
|
|
375
478
|
*/
|
|
376
|
-
export declare function
|
|
479
|
+
export declare function workBranchLease(args: {
|
|
480
|
+
dir: string;
|
|
481
|
+
branch: string;
|
|
482
|
+
/** The branch tip this pass started from. */
|
|
483
|
+
baseSha: string;
|
|
484
|
+
/** The sha this pass published, if any (`pushBranch`'s return). */
|
|
485
|
+
publishedSha: string | undefined;
|
|
486
|
+
signal?: AbortSignal;
|
|
487
|
+
/** Told why the lease was withheld, so the harness can log it with its own logger. */
|
|
488
|
+
onWithheld?: (probe: 'unreadable' | 'dropped') => void;
|
|
489
|
+
}): Promise<{
|
|
490
|
+
expectRemoteSha?: string;
|
|
491
|
+
}>;
|
|
377
492
|
/**
|
|
378
493
|
* Reset the working tree's git history to a single bootstrap commit and push it
|
|
379
494
|
* to the target repository's default branch. Wiping `.git` before re-initialising
|
package/dist/git.js
CHANGED
|
@@ -91,6 +91,48 @@ export function isGitTimeoutKill(err, aborted) {
|
|
|
91
91
|
function gitSubcommand(args) {
|
|
92
92
|
return args.find((a) => a !== '' && !a.startsWith('-')) ?? 'command';
|
|
93
93
|
}
|
|
94
|
+
/**
|
|
95
|
+
* Whether `stderr` is a REFUSED push, and which shape. Ordered: the lease/fetch-first shapes are
|
|
96
|
+
* checked first, because a `(stale info)` refusal also prints the generic "failed to push some
|
|
97
|
+
* refs" line the non-fast-forward shape shares. Pure, so both branches are unit-tested against
|
|
98
|
+
* real git output rather than inferred.
|
|
99
|
+
*/
|
|
100
|
+
export function classifyPushRejection(stderr) {
|
|
101
|
+
// A HOST-side refusal is not contention, and re-dispatching cannot help: branch protection, a
|
|
102
|
+
// pre-receive hook or a token policy is declining the write itself, and GitHub's protected-branch
|
|
103
|
+
// message says "refusing to allow a non-fast-forward push", which would otherwise read as a
|
|
104
|
+
// rewrite. Git's own labels separate the two cleanly (`! [remote rejected]` is the server
|
|
105
|
+
// declining, `! [rejected]` is git's own fast-forward/lease check), so such a failure stays a
|
|
106
|
+
// plain `git` fault with the write-access remedy below.
|
|
107
|
+
if (/remote rejected|protected branch|hook declined|refusing to allow/i.test(stderr)) {
|
|
108
|
+
return undefined;
|
|
109
|
+
}
|
|
110
|
+
if (/\(stale info\)|\(fetch first\)|remote contains work that you do not/i.test(stderr)) {
|
|
111
|
+
return 'remote-writer';
|
|
112
|
+
}
|
|
113
|
+
if (/\(non-fast-forward\)|tip of your current branch is behind|branch tip is behind/i.test(stderr)) {
|
|
114
|
+
return 'local-rewrite';
|
|
115
|
+
}
|
|
116
|
+
return undefined;
|
|
117
|
+
}
|
|
118
|
+
/**
|
|
119
|
+
* The remedy each {@link PushRejection} earns. A `Record`, so a new rejection shape cannot be
|
|
120
|
+
* classified without saying what a human should do about it. Neither is "run `git pull`", which is
|
|
121
|
+
* what git's own hint advises and is advice for a person at a terminal, not for an autonomous run.
|
|
122
|
+
*/
|
|
123
|
+
const PUSH_REJECTION_REMEDIES = {
|
|
124
|
+
'local-rewrite': 'The push was refused because the commit it publishes is not descended from the one the work ' +
|
|
125
|
+
'branch already holds: this checkout rewrote history that had already been pushed (an amend, ' +
|
|
126
|
+
"reset or rebase of an existing commit). The platform checkpoint-pushes the agent's commits " +
|
|
127
|
+
'while it works and lets a run force over its OWN published checkpoint, so what stays refused ' +
|
|
128
|
+
'is a rewrite it cannot attribute to this pass: commits an earlier run published, or a rewrite ' +
|
|
129
|
+
'that dropped the branch tip this pass started from. The engine re-dispatches the step to ' +
|
|
130
|
+
'resume from the branch as it stands; work already on the branch is never dropped.',
|
|
131
|
+
'remote-writer': 'The push was refused because another writer advanced this work branch while the run was ' +
|
|
132
|
+
'working (a second dispatch for the same block, or a person pushing to it). Nothing is lost: ' +
|
|
133
|
+
"the other writer's commits stay on the branch and the engine re-dispatches the step so the " +
|
|
134
|
+
'agent resumes on top of them. If it recurs, check whether two runs are active for the same block.',
|
|
135
|
+
};
|
|
94
136
|
/**
|
|
95
137
|
* Classify the common shapes of git's own stderr into an actionable remedy, else undefined
|
|
96
138
|
* (an unrecognized failure keeps just its raw stderr). This is the FIRST-WRAP-POINT for
|
|
@@ -102,6 +144,11 @@ function gitSubcommand(args) {
|
|
|
102
144
|
*/
|
|
103
145
|
export function describeGitFailure(stderr) {
|
|
104
146
|
const s = stderr.toLowerCase();
|
|
147
|
+
// A refused push first: its stderr carries neither an auth nor an access shape, so a miss here
|
|
148
|
+
// would leave the operator git's own "use 'git pull' before pushing again" hint and nothing else.
|
|
149
|
+
const rejection = classifyPushRejection(stderr);
|
|
150
|
+
if (rejection)
|
|
151
|
+
return PUSH_REJECTION_REMEDIES[rejection];
|
|
105
152
|
// Rate-limit / abuse-detection first: the host returns these as a 403, which would
|
|
106
153
|
// otherwise fall into the write-access shape below and be mislabeled as a permission
|
|
107
154
|
// problem — but the fix is to wait, not to grant access.
|
|
@@ -154,13 +201,21 @@ function gitFailure(err, args, aborted) {
|
|
|
154
201
|
}
|
|
155
202
|
const stderr = typeof e?.stderr === 'string' ? e.stderr : (e?.stderr?.toString() ?? '');
|
|
156
203
|
const base = e instanceof Error ? e.message : String(err);
|
|
157
|
-
|
|
158
|
-
//
|
|
159
|
-
//
|
|
160
|
-
//
|
|
204
|
+
// `execFile` builds its rejection message as `Command failed: <cmd>\n<stderr>`, so for the
|
|
205
|
+
// ordinary non-zero exit the stderr is ALREADY in `base`, and appending it again printed every
|
|
206
|
+
// git failure's output twice, which reads as two attempts. Append only what `base` lacks
|
|
207
|
+
// (a killed/other rejection whose message carries no output).
|
|
208
|
+
const tail = stderr.trim();
|
|
209
|
+
const combined = tail && !base.includes(tail) ? `${base}\n${tail}` : base;
|
|
210
|
+
// Append a cause + fix for the recognized auth/access/push-rejection shapes, keeping the raw
|
|
211
|
+
// (scrubbed) stderr above it as the detail. The remedy is static text with no secrets, so it is
|
|
212
|
+
// added after redaction.
|
|
161
213
|
const remedy = describeGitFailure(combined);
|
|
162
214
|
const message = remedy ? `${redactSecrets(combined)}\n${remedy}` : redactSecrets(combined);
|
|
163
|
-
|
|
215
|
+
// A REFUSED push is not a generic `git` fault: the branch moved under this run, which the engine
|
|
216
|
+
// recovers from by re-dispatching the step onto the branch as it now stands. It gets its own
|
|
217
|
+
// structured cause so that recovery keys off a classification rather than this message.
|
|
218
|
+
const failure = new HarnessFailure(classifyPushRejection(combined) ? 'branch-contended' : 'git', message);
|
|
164
219
|
if (e?.stack)
|
|
165
220
|
failure.stack = redactSecrets(e.stack);
|
|
166
221
|
return failure;
|
|
@@ -851,15 +906,113 @@ export async function fetchPullRequestHead(opts) {
|
|
|
851
906
|
}
|
|
852
907
|
}
|
|
853
908
|
/**
|
|
854
|
-
* Push the work branch to origin. The remote URL carries only the
|
|
855
|
-
* the token is supplied here via the askpass env (never in argv).
|
|
909
|
+
* Push the work branch to origin and return the sha it PUBLISHED. The remote URL carries only the
|
|
910
|
+
* username, so the token is supplied here via the askpass env (never in argv).
|
|
911
|
+
*
|
|
912
|
+
* The push names an explicit SOURCE COMMIT (`<sha>:refs/heads/<branch>`) rather than the branch,
|
|
913
|
+
* which is what makes the return value exact rather than a guess. The agent commits while this
|
|
914
|
+
* runs, so `git push origin <branch>` publishes whatever the branch ref holds at the moment git
|
|
915
|
+
* reads it, and a caller that leases against a sha it read either side of that has leased against
|
|
916
|
+
* the wrong commit. Reading it back from `refs/remotes/origin/<branch>` afterwards is worse than
|
|
917
|
+
* inexact, it is EMPTY on the production checkout: a fresh coding run clones one branch
|
|
918
|
+
* (`cloneRepo`), so the remote's fetch refspec covers the base alone and `git push` creates no
|
|
919
|
+
* tracking ref for the work branch at all. Naming the sha needs no ref and no round trip.
|
|
920
|
+
*
|
|
921
|
+
* `-u` goes with it: with a non-branch source git sets no upstream config (verified), nothing in
|
|
922
|
+
* the harness reads that config, and the agent is told never to push or pull.
|
|
923
|
+
*
|
|
924
|
+
* `expectRemoteSha` turns the push into a LEASED force (`--force-with-lease=<branch>:<sha>`), which
|
|
925
|
+
* is how a run whose own checkpoint push it has since rewritten still lands. It is deliberately NOT
|
|
926
|
+
* a plain `--force`: the lease succeeds only while the remote still holds the sha THIS run
|
|
927
|
+
* published, so a second writer's commits refuse the push (`(stale info)`) instead of being
|
|
928
|
+
* clobbered. Callers therefore pass only a sha this same pass published; leasing against a tip we
|
|
929
|
+
* merely CLONED would force over an earlier run's work.
|
|
856
930
|
*/
|
|
857
|
-
export async function pushBranch(dir, branch, ghToken, signal) {
|
|
858
|
-
await git(['
|
|
931
|
+
export async function pushBranch(dir, branch, ghToken, signal, opts = {}) {
|
|
932
|
+
const sha = (await git(['rev-parse', '--verify', `refs/heads/${branch}`], { cwd: dir, signal })).trim();
|
|
933
|
+
const lease = opts.expectRemoteSha ? [`--force-with-lease=${branch}:${opts.expectRemoteSha}`] : [];
|
|
934
|
+
await git(['push', ...lease, 'origin', `${sha}:refs/heads/${branch}`], {
|
|
859
935
|
cwd: dir,
|
|
860
936
|
signal,
|
|
861
937
|
env: await authEnv(ghToken),
|
|
862
938
|
});
|
|
939
|
+
return sha;
|
|
940
|
+
}
|
|
941
|
+
/**
|
|
942
|
+
* Whether `sha` is still reachable from `branch`'s tip, i.e. the branch CONTAINS it:
|
|
943
|
+
* `git rev-list --count --max-count=1 <sha> --not refs/heads/<branch>` is 0 when everything
|
|
944
|
+
* reachable from `sha` is reachable from the branch too (the tip itself counts as contained).
|
|
945
|
+
*
|
|
946
|
+
* Phrased as a rev-list rather than `merge-base --is-ancestor` on purpose: the latter answers "no"
|
|
947
|
+
* by EXITING 1, which is indistinguishable here from a broken checkout, and this probe's whole job
|
|
948
|
+
* is to be trusted only when it is a definite answer. Tri-state for the same reason (as
|
|
949
|
+
* {@link branchAheadOfBase} is):
|
|
950
|
+
*
|
|
951
|
+
* - `true`: confirmed contained.
|
|
952
|
+
* - `false`: confirmed dropped, so the branch was rewritten below `sha`.
|
|
953
|
+
* - `undefined`: could not determine (an unknown object, a rev-list error). A caller must not read
|
|
954
|
+
* a failed probe as either answer.
|
|
955
|
+
*
|
|
956
|
+
* The work-branch lease is gated on this: see {@link workBranchLease}.
|
|
957
|
+
*/
|
|
958
|
+
export async function branchContainsCommit(dir, branch, sha, signal) {
|
|
959
|
+
try {
|
|
960
|
+
const out = await git(['rev-list', '--count', '--max-count=1', sha, '--not', `refs/heads/${branch}`], { cwd: dir, signal });
|
|
961
|
+
const count = Number(out.trim());
|
|
962
|
+
return Number.isNaN(count) ? undefined : count === 0;
|
|
963
|
+
}
|
|
964
|
+
catch {
|
|
965
|
+
return undefined;
|
|
966
|
+
}
|
|
967
|
+
}
|
|
968
|
+
/**
|
|
969
|
+
* The work branch's tip when it holds something UNPUBLISHED, else undefined: the answer to whether a
|
|
970
|
+
* checkpoint tick has anything to do. Two ways of having nothing:
|
|
971
|
+
*
|
|
972
|
+
* - the tip is still `baseSha`, so this pass has committed nothing. Pushing here would create the
|
|
973
|
+
* work branch at the base commit, and a later retry would see that zero-diff branch via
|
|
974
|
+
* `remoteBranchExists`, resume it as work, and fail to open a PR ("no commits between base and
|
|
975
|
+
* head"). A pass that never commits must leave NO branch behind.
|
|
976
|
+
* - the tip is `publishedSha`, so the last push already published it. Without this the checkpoint
|
|
977
|
+
* re-pushed an unchanged branch on every tick: an hour-long run committing eight times issued
|
|
978
|
+
* ~60 pushes, ~52 of them a full authenticated round trip answering "Everything up-to-date",
|
|
979
|
+
* each one counting against the host's push rate limits.
|
|
980
|
+
*
|
|
981
|
+
* That second condition is also what keeps the INTERVAL the right knob. It expresses the acceptable
|
|
982
|
+
* loss window when a container dies (a property of the deployment's infra churn), not a rate: gated
|
|
983
|
+
* this way, the tick publishes at most one push per commit the agent makes, whatever the model or
|
|
984
|
+
* the run's length, so nothing here needs to be tuned per model.
|
|
985
|
+
*/
|
|
986
|
+
export async function unpublishedWorkBranchTip(args) {
|
|
987
|
+
const head = await headCommit(args.dir, args.signal);
|
|
988
|
+
if (head === args.baseSha || head === args.publishedSha)
|
|
989
|
+
return undefined;
|
|
990
|
+
return head;
|
|
991
|
+
}
|
|
992
|
+
/**
|
|
993
|
+
* The lease a work-branch push is entitled to (the `opts` {@link pushBranch} takes): the sha this
|
|
994
|
+
* pass last published, and nothing at all before it has published one.
|
|
995
|
+
*
|
|
996
|
+
* The extra condition is what bounds the force to THIS pass's own commits, which the lease alone
|
|
997
|
+
* does not do and the design promises. Once one checkpoint has landed, a rewrite that drops
|
|
998
|
+
* `baseSha` (the tip the pass started from, which on a RESUMED branch is an earlier run's published
|
|
999
|
+
* work) would still lease successfully against our own checkpoint and carry those earlier commits
|
|
1000
|
+
* away with it. So the lease is withheld unless the branch still CONTAINS `baseSha`: the push then
|
|
1001
|
+
* goes out plain, git refuses it as a non-fast-forward, and the engine re-dispatches onto the
|
|
1002
|
+
* branch as it stands.
|
|
1003
|
+
*
|
|
1004
|
+
* A probe that could not answer withholds it too (`onWithheld('unreadable')`), because the two
|
|
1005
|
+
* mistakes are not symmetric: withholding costs a refused rewrite and one re-dispatch, trusting an
|
|
1006
|
+
* unreadable probe costs commits.
|
|
1007
|
+
*/
|
|
1008
|
+
export async function workBranchLease(args) {
|
|
1009
|
+
if (!args.publishedSha)
|
|
1010
|
+
return {};
|
|
1011
|
+
const contains = await branchContainsCommit(args.dir, args.branch, args.baseSha, args.signal);
|
|
1012
|
+
if (contains === true)
|
|
1013
|
+
return { expectRemoteSha: args.publishedSha };
|
|
1014
|
+
args.onWithheld?.(contains === undefined ? 'unreadable' : 'dropped');
|
|
1015
|
+
return {};
|
|
863
1016
|
}
|
|
864
1017
|
/**
|
|
865
1018
|
* Reset the working tree's git history to a single bootstrap commit and push it
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@cat-factory/executor-harness",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.116.0",
|
|
4
4
|
"description": "Container payload: a thin TypeScript wrapper that runs the Pi coding agent against a cloned repo and opens a PR. Runs in the Cloudflare Container (and, in local native mode, as a host process); carries no secrets.",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -30,9 +30,9 @@
|
|
|
30
30
|
"hono": "^4.13.1",
|
|
31
31
|
"typescript": "7.0.2",
|
|
32
32
|
"vitest": "^4.1.10",
|
|
33
|
-
"@cat-factory/kernel": "0.
|
|
34
|
-
"@cat-factory/server": "0.
|
|
35
|
-
"@cat-factory/spend": "0.15.
|
|
33
|
+
"@cat-factory/kernel": "0.298.0",
|
|
34
|
+
"@cat-factory/server": "0.284.0",
|
|
35
|
+
"@cat-factory/spend": "0.15.89"
|
|
36
36
|
},
|
|
37
37
|
"scripts": {
|
|
38
38
|
"build": "tsc -p tsconfig.json",
|
package/src/coding-agent.ts
CHANGED
|
@@ -24,6 +24,8 @@ import {
|
|
|
24
24
|
pushBranch,
|
|
25
25
|
refreshFromBaseIfClean,
|
|
26
26
|
remoteBranchExists,
|
|
27
|
+
unpublishedWorkBranchTip,
|
|
28
|
+
workBranchLease,
|
|
27
29
|
} from './git.js'
|
|
28
30
|
import { FOLLOW_UPS_FILENAME, FollowUpTailer } from './follow-ups.js'
|
|
29
31
|
import type { HarnessCallMetric } from './pi.js'
|
|
@@ -280,13 +282,35 @@ function followUpPollIntervalMs(): number {
|
|
|
280
282
|
* concurrently: overlapping pushes race on the remote ref and can make a push fail with a
|
|
281
283
|
* ref-lock / non-fast-forward error — which, on the FINAL push, would fail the whole run even
|
|
282
284
|
* though the work is committed. `pushWorkOnce` coalesces concurrent callers onto one push and only
|
|
283
|
-
* pushes
|
|
285
|
+
* pushes what is UNPUBLISHED ({@link unpublishedWorkBranchTip}: past `baseSha`, and not already the
|
|
286
|
+
* tip the last push published).
|
|
284
287
|
*
|
|
285
|
-
*
|
|
286
|
-
*
|
|
287
|
-
*
|
|
288
|
-
*
|
|
289
|
-
*
|
|
288
|
+
* Every push after the first LEASES against the sha this pass published (see
|
|
289
|
+
* {@link pushBranch}), because the checkpoint makes the harness its own competing writer: it
|
|
290
|
+
* publishes a commit within a minute of the agent making it, and the agent is then free to amend,
|
|
291
|
+
* reset or rebase that commit, which is perfectly ordinary git hygiene, and the delivery contract
|
|
292
|
+
* asks it to validate AFTER committing, exactly the sequence that produces an amend. Without the
|
|
293
|
+
* lease the final push is refused as a non-fast-forward and the whole run fails with its work
|
|
294
|
+
* already on the branch. The lease is what keeps that recovery from becoming a blanket `--force`:
|
|
295
|
+
* a SECOND writer (a concurrent dispatch for the same block) still refuses the push, which is the
|
|
296
|
+
* "never clobber another run's commits" property the resume design leans on.
|
|
297
|
+
*
|
|
298
|
+
* The lease alone does not bound the force to THIS pass's own commits, and that is the property
|
|
299
|
+
* the design promises, so it is checked rather than assumed: once one checkpoint has landed, a
|
|
300
|
+
* rewrite that drops `baseSha` (the tip the pass started from, which on a RESUMED branch is an
|
|
301
|
+
* earlier run's published work) would lease successfully against our own checkpoint and take the
|
|
302
|
+
* earlier commits with it. So the lease is armed only while the branch still CONTAINS `baseSha`;
|
|
303
|
+
* withheld, the push goes out plain, git refuses it, and the engine re-dispatches onto the branch
|
|
304
|
+
* as it stands. A rewrite this pass cannot prove is its own is never forced away.
|
|
305
|
+
*
|
|
306
|
+
* What is pushable is {@link unpublishedWorkBranchTip}'s question, and both of its answers matter
|
|
307
|
+
* here. A branch still at `baseSha` must not be pushed at all, or a later retry resumes a zero-diff
|
|
308
|
+
* branch and cannot open a PR for it. A branch already at the published tip has nothing to add, and
|
|
309
|
+
* skipping it is what keeps the interval a LOSS WINDOW rather than a push rate: an hour-long run
|
|
310
|
+
* that commits eight times pushes eight times, not sixty. That skip is invisible to the outcome by
|
|
311
|
+
* construction: `finalizeCodingRun` decides `pushed` from the BRANCH (advanced this pass, or
|
|
312
|
+
* resumed), never from whether the final call issued a `git push`, because a tip the checkpoint
|
|
313
|
+
* already published is published.
|
|
290
314
|
*/
|
|
291
315
|
function createWorkBranchPusher(args: {
|
|
292
316
|
dir: string
|
|
@@ -301,11 +325,31 @@ function createWorkBranchPusher(args: {
|
|
|
301
325
|
} {
|
|
302
326
|
const { dir, spec, baseSha, logger, signal } = args
|
|
303
327
|
let pushInFlight: Promise<void> | null = null
|
|
328
|
+
// The sha THIS pass last published to the work branch, and the only value it will ever lease a
|
|
329
|
+
// force push against. Starts unset even on a RESUMED branch: the tip we merely cloned is an
|
|
330
|
+
// earlier run's work, so a rewrite of it is refused (and re-driven) rather than forced away.
|
|
331
|
+
let publishedSha: string | undefined
|
|
304
332
|
const pushWorkOnce = (): Promise<void> => {
|
|
305
333
|
if (pushInFlight) return pushInFlight
|
|
306
334
|
pushInFlight = (async () => {
|
|
307
|
-
if (!(await
|
|
308
|
-
|
|
335
|
+
if (!(await unpublishedWorkBranchTip({ dir, baseSha, publishedSha, signal }))) return
|
|
336
|
+
// The rule the lease is entitled to lives beside the push ({@link workBranchLease}); the
|
|
337
|
+
// warn is here, because a withheld lease is how a rewrite this pass cannot claim fails the
|
|
338
|
+
// push it is about to make, and the run's log is where that is read.
|
|
339
|
+
const lease = await workBranchLease({
|
|
340
|
+
dir,
|
|
341
|
+
branch: spec.pushBranch,
|
|
342
|
+
baseSha,
|
|
343
|
+
publishedSha,
|
|
344
|
+
signal,
|
|
345
|
+
onWithheld: (probe) =>
|
|
346
|
+
logger.warn('coding-agent: push lease withheld, the branch dropped its pre-run tip', {
|
|
347
|
+
baseSha,
|
|
348
|
+
publishedSha,
|
|
349
|
+
probe,
|
|
350
|
+
}),
|
|
351
|
+
})
|
|
352
|
+
publishedSha = await pushBranch(dir, spec.pushBranch, spec.ghToken, signal, lease)
|
|
309
353
|
})().finally(() => {
|
|
310
354
|
pushInFlight = null
|
|
311
355
|
})
|
package/src/failure.ts
CHANGED
|
@@ -26,6 +26,12 @@
|
|
|
26
26
|
* went quiet, this one says the model rabbit-holed while streaming.
|
|
27
27
|
* - `agent` — the agent ran but produced an unusable/failed result, or threw.
|
|
28
28
|
* - `git` — a git operation failed (clone/push/merge/PR).
|
|
29
|
+
* - `branch-contended`: a push to the work branch was REFUSED because the branch carries
|
|
30
|
+
* commits this push would drop (a second writer, or a rewrite of an
|
|
31
|
+
* earlier run's history). Split out of `git` because it is the one git
|
|
32
|
+
* fault the ENGINE can recover from on its own: re-dispatching the step
|
|
33
|
+
* resumes the branch as it now stands, where every other `git` failure
|
|
34
|
+
* would only fail again.
|
|
29
35
|
* - `api` — an upstream API call failed (e.g. the GitHub/GitLab PR/MR REST call).
|
|
30
36
|
* - `llm-upstream` — the model provider rejected every call (auth/quota/rate-limit) and Pi
|
|
31
37
|
* exhausted its retries, so the run never produced a result.
|
|
@@ -38,6 +44,7 @@ export const FAILURE_CAUSES = [
|
|
|
38
44
|
'no-tool-progress',
|
|
39
45
|
'agent',
|
|
40
46
|
'git',
|
|
47
|
+
'branch-contended',
|
|
41
48
|
'api',
|
|
42
49
|
'llm-upstream',
|
|
43
50
|
'no-usable-output',
|
package/src/git.ts
CHANGED
|
@@ -112,6 +112,69 @@ function gitSubcommand(args: string[]): string {
|
|
|
112
112
|
return args.find((a) => a !== '' && !a.startsWith('-')) ?? 'command'
|
|
113
113
|
}
|
|
114
114
|
|
|
115
|
+
/**
|
|
116
|
+
* Why a push to the work branch was REFUSED. Both mean the branch carries commits this push
|
|
117
|
+
* would drop, and git tells them apart by whether our object database HOLDS the tip the remote
|
|
118
|
+
* reports: it does for a tip our own checkout created, so the two are distinguishable and need
|
|
119
|
+
* different remedies (see {@link PUSH_REJECTION_REMEDIES}).
|
|
120
|
+
*
|
|
121
|
+
* - `local-rewrite`: we HAVE the remote's tip and are no longer descended from it, i.e. this
|
|
122
|
+
* checkout amended / reset / rebased a commit that had already been pushed. Git labels it
|
|
123
|
+
* `(non-fast-forward)`.
|
|
124
|
+
* - `remote-writer`: the remote's tip is a commit this checkout has never seen (`(fetch first)`),
|
|
125
|
+
* or our lease found the branch moved past what we published (`(stale info)`), so a SECOND
|
|
126
|
+
* writer owns the branch.
|
|
127
|
+
*/
|
|
128
|
+
export type PushRejection = 'local-rewrite' | 'remote-writer'
|
|
129
|
+
|
|
130
|
+
/**
|
|
131
|
+
* Whether `stderr` is a REFUSED push, and which shape. Ordered: the lease/fetch-first shapes are
|
|
132
|
+
* checked first, because a `(stale info)` refusal also prints the generic "failed to push some
|
|
133
|
+
* refs" line the non-fast-forward shape shares. Pure, so both branches are unit-tested against
|
|
134
|
+
* real git output rather than inferred.
|
|
135
|
+
*/
|
|
136
|
+
export function classifyPushRejection(stderr: string): PushRejection | undefined {
|
|
137
|
+
// A HOST-side refusal is not contention, and re-dispatching cannot help: branch protection, a
|
|
138
|
+
// pre-receive hook or a token policy is declining the write itself, and GitHub's protected-branch
|
|
139
|
+
// message says "refusing to allow a non-fast-forward push", which would otherwise read as a
|
|
140
|
+
// rewrite. Git's own labels separate the two cleanly (`! [remote rejected]` is the server
|
|
141
|
+
// declining, `! [rejected]` is git's own fast-forward/lease check), so such a failure stays a
|
|
142
|
+
// plain `git` fault with the write-access remedy below.
|
|
143
|
+
if (/remote rejected|protected branch|hook declined|refusing to allow/i.test(stderr)) {
|
|
144
|
+
return undefined
|
|
145
|
+
}
|
|
146
|
+
if (/\(stale info\)|\(fetch first\)|remote contains work that you do not/i.test(stderr)) {
|
|
147
|
+
return 'remote-writer'
|
|
148
|
+
}
|
|
149
|
+
if (
|
|
150
|
+
/\(non-fast-forward\)|tip of your current branch is behind|branch tip is behind/i.test(stderr)
|
|
151
|
+
) {
|
|
152
|
+
return 'local-rewrite'
|
|
153
|
+
}
|
|
154
|
+
return undefined
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
/**
|
|
158
|
+
* The remedy each {@link PushRejection} earns. A `Record`, so a new rejection shape cannot be
|
|
159
|
+
* classified without saying what a human should do about it. Neither is "run `git pull`", which is
|
|
160
|
+
* what git's own hint advises and is advice for a person at a terminal, not for an autonomous run.
|
|
161
|
+
*/
|
|
162
|
+
const PUSH_REJECTION_REMEDIES: Record<PushRejection, string> = {
|
|
163
|
+
'local-rewrite':
|
|
164
|
+
'The push was refused because the commit it publishes is not descended from the one the work ' +
|
|
165
|
+
'branch already holds: this checkout rewrote history that had already been pushed (an amend, ' +
|
|
166
|
+
"reset or rebase of an existing commit). The platform checkpoint-pushes the agent's commits " +
|
|
167
|
+
'while it works and lets a run force over its OWN published checkpoint, so what stays refused ' +
|
|
168
|
+
'is a rewrite it cannot attribute to this pass: commits an earlier run published, or a rewrite ' +
|
|
169
|
+
'that dropped the branch tip this pass started from. The engine re-dispatches the step to ' +
|
|
170
|
+
'resume from the branch as it stands; work already on the branch is never dropped.',
|
|
171
|
+
'remote-writer':
|
|
172
|
+
'The push was refused because another writer advanced this work branch while the run was ' +
|
|
173
|
+
'working (a second dispatch for the same block, or a person pushing to it). Nothing is lost: ' +
|
|
174
|
+
"the other writer's commits stay on the branch and the engine re-dispatches the step so the " +
|
|
175
|
+
'agent resumes on top of them. If it recurs, check whether two runs are active for the same block.',
|
|
176
|
+
}
|
|
177
|
+
|
|
115
178
|
/**
|
|
116
179
|
* Classify the common shapes of git's own stderr into an actionable remedy, else undefined
|
|
117
180
|
* (an unrecognized failure keeps just its raw stderr). This is the FIRST-WRAP-POINT for
|
|
@@ -123,6 +186,10 @@ function gitSubcommand(args: string[]): string {
|
|
|
123
186
|
*/
|
|
124
187
|
export function describeGitFailure(stderr: string): string | undefined {
|
|
125
188
|
const s = stderr.toLowerCase()
|
|
189
|
+
// A refused push first: its stderr carries neither an auth nor an access shape, so a miss here
|
|
190
|
+
// would leave the operator git's own "use 'git pull' before pushing again" hint and nothing else.
|
|
191
|
+
const rejection = classifyPushRejection(stderr)
|
|
192
|
+
if (rejection) return PUSH_REJECTION_REMEDIES[rejection]
|
|
126
193
|
// Rate-limit / abuse-detection first: the host returns these as a 403, which would
|
|
127
194
|
// otherwise fall into the write-access shape below and be mislabeled as a permission
|
|
128
195
|
// problem — but the fix is to wait, not to grant access.
|
|
@@ -200,13 +267,24 @@ function gitFailure(err: unknown, args: string[], aborted: boolean): HarnessFail
|
|
|
200
267
|
}
|
|
201
268
|
const stderr = typeof e?.stderr === 'string' ? e.stderr : (e?.stderr?.toString() ?? '')
|
|
202
269
|
const base = e instanceof Error ? e.message : String(err)
|
|
203
|
-
|
|
204
|
-
//
|
|
205
|
-
//
|
|
206
|
-
//
|
|
270
|
+
// `execFile` builds its rejection message as `Command failed: <cmd>\n<stderr>`, so for the
|
|
271
|
+
// ordinary non-zero exit the stderr is ALREADY in `base`, and appending it again printed every
|
|
272
|
+
// git failure's output twice, which reads as two attempts. Append only what `base` lacks
|
|
273
|
+
// (a killed/other rejection whose message carries no output).
|
|
274
|
+
const tail = stderr.trim()
|
|
275
|
+
const combined = tail && !base.includes(tail) ? `${base}\n${tail}` : base
|
|
276
|
+
// Append a cause + fix for the recognized auth/access/push-rejection shapes, keeping the raw
|
|
277
|
+
// (scrubbed) stderr above it as the detail. The remedy is static text with no secrets, so it is
|
|
278
|
+
// added after redaction.
|
|
207
279
|
const remedy = describeGitFailure(combined)
|
|
208
280
|
const message = remedy ? `${redactSecrets(combined)}\n${remedy}` : redactSecrets(combined)
|
|
209
|
-
|
|
281
|
+
// A REFUSED push is not a generic `git` fault: the branch moved under this run, which the engine
|
|
282
|
+
// recovers from by re-dispatching the step onto the branch as it now stands. It gets its own
|
|
283
|
+
// structured cause so that recovery keys off a classification rather than this message.
|
|
284
|
+
const failure = new HarnessFailure(
|
|
285
|
+
classifyPushRejection(combined) ? 'branch-contended' : 'git',
|
|
286
|
+
message,
|
|
287
|
+
)
|
|
210
288
|
if (e?.stack) failure.stack = redactSecrets(e.stack)
|
|
211
289
|
return failure
|
|
212
290
|
}
|
|
@@ -1054,20 +1132,145 @@ export async function fetchPullRequestHead(opts: {
|
|
|
1054
1132
|
}
|
|
1055
1133
|
|
|
1056
1134
|
/**
|
|
1057
|
-
* Push the work branch to origin. The remote URL carries only the
|
|
1058
|
-
* the token is supplied here via the askpass env (never in argv).
|
|
1135
|
+
* Push the work branch to origin and return the sha it PUBLISHED. The remote URL carries only the
|
|
1136
|
+
* username, so the token is supplied here via the askpass env (never in argv).
|
|
1137
|
+
*
|
|
1138
|
+
* The push names an explicit SOURCE COMMIT (`<sha>:refs/heads/<branch>`) rather than the branch,
|
|
1139
|
+
* which is what makes the return value exact rather than a guess. The agent commits while this
|
|
1140
|
+
* runs, so `git push origin <branch>` publishes whatever the branch ref holds at the moment git
|
|
1141
|
+
* reads it, and a caller that leases against a sha it read either side of that has leased against
|
|
1142
|
+
* the wrong commit. Reading it back from `refs/remotes/origin/<branch>` afterwards is worse than
|
|
1143
|
+
* inexact, it is EMPTY on the production checkout: a fresh coding run clones one branch
|
|
1144
|
+
* (`cloneRepo`), so the remote's fetch refspec covers the base alone and `git push` creates no
|
|
1145
|
+
* tracking ref for the work branch at all. Naming the sha needs no ref and no round trip.
|
|
1146
|
+
*
|
|
1147
|
+
* `-u` goes with it: with a non-branch source git sets no upstream config (verified), nothing in
|
|
1148
|
+
* the harness reads that config, and the agent is told never to push or pull.
|
|
1149
|
+
*
|
|
1150
|
+
* `expectRemoteSha` turns the push into a LEASED force (`--force-with-lease=<branch>:<sha>`), which
|
|
1151
|
+
* is how a run whose own checkpoint push it has since rewritten still lands. It is deliberately NOT
|
|
1152
|
+
* a plain `--force`: the lease succeeds only while the remote still holds the sha THIS run
|
|
1153
|
+
* published, so a second writer's commits refuse the push (`(stale info)`) instead of being
|
|
1154
|
+
* clobbered. Callers therefore pass only a sha this same pass published; leasing against a tip we
|
|
1155
|
+
* merely CLONED would force over an earlier run's work.
|
|
1059
1156
|
*/
|
|
1060
1157
|
export async function pushBranch(
|
|
1061
1158
|
dir: string,
|
|
1062
1159
|
branch: string,
|
|
1063
1160
|
ghToken: string,
|
|
1064
1161
|
signal?: AbortSignal,
|
|
1065
|
-
|
|
1066
|
-
|
|
1162
|
+
opts: { expectRemoteSha?: string } = {},
|
|
1163
|
+
): Promise<string> {
|
|
1164
|
+
const sha = (
|
|
1165
|
+
await git(['rev-parse', '--verify', `refs/heads/${branch}`], { cwd: dir, signal })
|
|
1166
|
+
).trim()
|
|
1167
|
+
const lease = opts.expectRemoteSha ? [`--force-with-lease=${branch}:${opts.expectRemoteSha}`] : []
|
|
1168
|
+
await git(['push', ...lease, 'origin', `${sha}:refs/heads/${branch}`], {
|
|
1067
1169
|
cwd: dir,
|
|
1068
1170
|
signal,
|
|
1069
1171
|
env: await authEnv(ghToken),
|
|
1070
1172
|
})
|
|
1173
|
+
return sha
|
|
1174
|
+
}
|
|
1175
|
+
|
|
1176
|
+
/**
|
|
1177
|
+
* Whether `sha` is still reachable from `branch`'s tip, i.e. the branch CONTAINS it:
|
|
1178
|
+
* `git rev-list --count --max-count=1 <sha> --not refs/heads/<branch>` is 0 when everything
|
|
1179
|
+
* reachable from `sha` is reachable from the branch too (the tip itself counts as contained).
|
|
1180
|
+
*
|
|
1181
|
+
* Phrased as a rev-list rather than `merge-base --is-ancestor` on purpose: the latter answers "no"
|
|
1182
|
+
* by EXITING 1, which is indistinguishable here from a broken checkout, and this probe's whole job
|
|
1183
|
+
* is to be trusted only when it is a definite answer. Tri-state for the same reason (as
|
|
1184
|
+
* {@link branchAheadOfBase} is):
|
|
1185
|
+
*
|
|
1186
|
+
* - `true`: confirmed contained.
|
|
1187
|
+
* - `false`: confirmed dropped, so the branch was rewritten below `sha`.
|
|
1188
|
+
* - `undefined`: could not determine (an unknown object, a rev-list error). A caller must not read
|
|
1189
|
+
* a failed probe as either answer.
|
|
1190
|
+
*
|
|
1191
|
+
* The work-branch lease is gated on this: see {@link workBranchLease}.
|
|
1192
|
+
*/
|
|
1193
|
+
export async function branchContainsCommit(
|
|
1194
|
+
dir: string,
|
|
1195
|
+
branch: string,
|
|
1196
|
+
sha: string,
|
|
1197
|
+
signal?: AbortSignal,
|
|
1198
|
+
): Promise<boolean | undefined> {
|
|
1199
|
+
try {
|
|
1200
|
+
const out = await git(
|
|
1201
|
+
['rev-list', '--count', '--max-count=1', sha, '--not', `refs/heads/${branch}`],
|
|
1202
|
+
{ cwd: dir, signal },
|
|
1203
|
+
)
|
|
1204
|
+
const count = Number(out.trim())
|
|
1205
|
+
return Number.isNaN(count) ? undefined : count === 0
|
|
1206
|
+
} catch {
|
|
1207
|
+
return undefined
|
|
1208
|
+
}
|
|
1209
|
+
}
|
|
1210
|
+
|
|
1211
|
+
/**
|
|
1212
|
+
* The work branch's tip when it holds something UNPUBLISHED, else undefined: the answer to whether a
|
|
1213
|
+
* checkpoint tick has anything to do. Two ways of having nothing:
|
|
1214
|
+
*
|
|
1215
|
+
* - the tip is still `baseSha`, so this pass has committed nothing. Pushing here would create the
|
|
1216
|
+
* work branch at the base commit, and a later retry would see that zero-diff branch via
|
|
1217
|
+
* `remoteBranchExists`, resume it as work, and fail to open a PR ("no commits between base and
|
|
1218
|
+
* head"). A pass that never commits must leave NO branch behind.
|
|
1219
|
+
* - the tip is `publishedSha`, so the last push already published it. Without this the checkpoint
|
|
1220
|
+
* re-pushed an unchanged branch on every tick: an hour-long run committing eight times issued
|
|
1221
|
+
* ~60 pushes, ~52 of them a full authenticated round trip answering "Everything up-to-date",
|
|
1222
|
+
* each one counting against the host's push rate limits.
|
|
1223
|
+
*
|
|
1224
|
+
* That second condition is also what keeps the INTERVAL the right knob. It expresses the acceptable
|
|
1225
|
+
* loss window when a container dies (a property of the deployment's infra churn), not a rate: gated
|
|
1226
|
+
* this way, the tick publishes at most one push per commit the agent makes, whatever the model or
|
|
1227
|
+
* the run's length, so nothing here needs to be tuned per model.
|
|
1228
|
+
*/
|
|
1229
|
+
export async function unpublishedWorkBranchTip(args: {
|
|
1230
|
+
dir: string
|
|
1231
|
+
/** The branch tip this pass started from. */
|
|
1232
|
+
baseSha: string
|
|
1233
|
+
/** The sha this pass published, if any ({@link pushBranch}'s return). */
|
|
1234
|
+
publishedSha: string | undefined
|
|
1235
|
+
signal?: AbortSignal
|
|
1236
|
+
}): Promise<string | undefined> {
|
|
1237
|
+
const head = await headCommit(args.dir, args.signal)
|
|
1238
|
+
if (head === args.baseSha || head === args.publishedSha) return undefined
|
|
1239
|
+
return head
|
|
1240
|
+
}
|
|
1241
|
+
|
|
1242
|
+
/**
|
|
1243
|
+
* The lease a work-branch push is entitled to (the `opts` {@link pushBranch} takes): the sha this
|
|
1244
|
+
* pass last published, and nothing at all before it has published one.
|
|
1245
|
+
*
|
|
1246
|
+
* The extra condition is what bounds the force to THIS pass's own commits, which the lease alone
|
|
1247
|
+
* does not do and the design promises. Once one checkpoint has landed, a rewrite that drops
|
|
1248
|
+
* `baseSha` (the tip the pass started from, which on a RESUMED branch is an earlier run's published
|
|
1249
|
+
* work) would still lease successfully against our own checkpoint and carry those earlier commits
|
|
1250
|
+
* away with it. So the lease is withheld unless the branch still CONTAINS `baseSha`: the push then
|
|
1251
|
+
* goes out plain, git refuses it as a non-fast-forward, and the engine re-dispatches onto the
|
|
1252
|
+
* branch as it stands.
|
|
1253
|
+
*
|
|
1254
|
+
* A probe that could not answer withholds it too (`onWithheld('unreadable')`), because the two
|
|
1255
|
+
* mistakes are not symmetric: withholding costs a refused rewrite and one re-dispatch, trusting an
|
|
1256
|
+
* unreadable probe costs commits.
|
|
1257
|
+
*/
|
|
1258
|
+
export async function workBranchLease(args: {
|
|
1259
|
+
dir: string
|
|
1260
|
+
branch: string
|
|
1261
|
+
/** The branch tip this pass started from. */
|
|
1262
|
+
baseSha: string
|
|
1263
|
+
/** The sha this pass published, if any (`pushBranch`'s return). */
|
|
1264
|
+
publishedSha: string | undefined
|
|
1265
|
+
signal?: AbortSignal
|
|
1266
|
+
/** Told why the lease was withheld, so the harness can log it with its own logger. */
|
|
1267
|
+
onWithheld?: (probe: 'unreadable' | 'dropped') => void
|
|
1268
|
+
}): Promise<{ expectRemoteSha?: string }> {
|
|
1269
|
+
if (!args.publishedSha) return {}
|
|
1270
|
+
const contains = await branchContainsCommit(args.dir, args.branch, args.baseSha, args.signal)
|
|
1271
|
+
if (contains === true) return { expectRemoteSha: args.publishedSha }
|
|
1272
|
+
args.onWithheld?.(contains === undefined ? 'unreadable' : 'dropped')
|
|
1273
|
+
return {}
|
|
1071
1274
|
}
|
|
1072
1275
|
|
|
1073
1276
|
/**
|