omp-conductor 0.3.6 → 0.3.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +71 -24
- package/package.json +1 -1
- package/src/briefs/orchestrator.md +5 -2
- package/src/briefs/worker.md +1 -1
- package/src/cli.ts +28 -14
- package/src/config.ts +16 -5
- package/src/daemon.ts +130 -40
- package/src/lifecycle.ts +203 -12
- package/src/omp.ts +5 -0
- package/src/plugin.ts +32 -2
- package/src/setup.ts +3 -1
- package/src/types.ts +5 -2
- package/src/worker.ts +37 -5
- package/src/worktree.ts +61 -9
package/README.md
CHANGED
|
@@ -355,8 +355,11 @@ a yes over Telegram, which is the same protocol it uses for any other amendment.
|
|
|
355
355
|
omp-conductor daemon --once
|
|
356
356
|
```
|
|
357
357
|
|
|
358
|
-
The loop ticks every 5 minutes. `omp-conductor stop`
|
|
359
|
-
|
|
358
|
+
The loop ticks every 5 minutes. `omp-conductor stop` shuts the loop down after
|
|
359
|
+
the current tick rather than mid-run. When the live process is the MainPID of
|
|
360
|
+
`omp-conductor.service`, stop goes through `systemctl stop` so a unit with
|
|
361
|
+
`Restart=on-failure` cannot bring it straight back; otherwise it is a raw
|
|
362
|
+
`SIGTERM` (then `SIGKILL` after 10 seconds).
|
|
360
363
|
|
|
361
364
|
## How one tick works
|
|
362
365
|
|
|
@@ -454,19 +457,48 @@ A `claimed` or `running` row is a promise that a worker process exists, and a
|
|
|
454
457
|
daemon that just started knows that promise is broken: its workers died with the
|
|
455
458
|
previous process. At startup — unless another daemon is alive, so a foreground
|
|
456
459
|
`daemon --once` cannot orphan a running daemon's real workers — every such row is
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
clean orphan has its label released so the next
|
|
468
|
-
attempts still count toward `maxAttemptsPerIssue`,
|
|
469
|
-
instead of redispatching forever.
|
|
460
|
+
**salvaged first** (dirty tree → `wip(#N): attempt N killed by a daemon restart —
|
|
461
|
+
auto-salvaged` on the run's branch, same path as a turns-cap kill), then moved to
|
|
462
|
+
`orphaned`, with a log line naming the issue, the attempt and the worktree. That
|
|
463
|
+
frees the slots immediately; a fleet must never resume as deadlocked as it
|
|
464
|
+
crashed, and uncommitted edits must not wait for a human with `bun -e`.
|
|
465
|
+
|
|
466
|
+
Only the rows change after salvage. The issue keeps `agent:in-progress` — the
|
|
467
|
+
label is the crash guard against double-dispatch — and deciding what the dead
|
|
468
|
+
worker's remains are worth is the orchestrator's drain-duty judgement, spelled
|
|
469
|
+
out in its brief: an open green PR goes to the merge path, a salvaged sha is a
|
|
470
|
+
continuation hand-off, and a clean orphan has its label released so the next
|
|
471
|
+
tick re-claims it. Orphaned attempts still count toward `maxAttemptsPerIssue`,
|
|
472
|
+
so a crash loop escalates instead of redispatching forever.
|
|
473
|
+
|
|
474
|
+
### Deploying a new package onto a busy fleet
|
|
475
|
+
|
|
476
|
+
`systemctl restart` / `omp-conductor restart` is safe for **work product** once
|
|
477
|
+
this version is installed: startup salvage commits dirty trees before orphaning
|
|
478
|
+
rows, and salvage rewrites the mirror's managed `info/exclude` to the package's
|
|
479
|
+
current list before `git add` so a narrowed ignore cannot hide deliverables.
|
|
480
|
+
|
|
481
|
+
It is still disruptive for **in-flight sessions** (the worker process dies; the
|
|
482
|
+
attempt is spent). Prefer draining when you can wait:
|
|
483
|
+
|
|
484
|
+
1. `omp-conductor pause` — stop new claims; live workers finish.
|
|
485
|
+
2. Wait until `omp-conductor status` shows `workers 0 / N` (no `deploy` hint line).
|
|
486
|
+
3. Install the new package (`bun add -g omp-conductor@…`, `omp plugin install …`).
|
|
487
|
+
4. `systemctl restart omp-conductor` (or `omp-conductor restart`).
|
|
488
|
+
5. `omp-conductor resume` if you left it paused.
|
|
489
|
+
|
|
490
|
+
If you cannot wait:
|
|
491
|
+
|
|
492
|
+
1. `omp-conductor pause` (optional but keeps new claims off during the swap).
|
|
493
|
+
2. Install.
|
|
494
|
+
3. Restart — salvage runs on boot for every live worktree, then rows go `orphaned`.
|
|
495
|
+
4. Resume; the pane orchestrator triages `agent:in-progress` orphans (continuation
|
|
496
|
+
/ merge / release label). Status prints a `deploy` line while live workers > 0
|
|
497
|
+
so you can see the risk before you restart.
|
|
498
|
+
|
|
499
|
+
Do **not** edit files under the running install and expect the daemon to keep
|
|
500
|
+
dispatching — the integrity tripwire pauses and pages. Install, then restart, so
|
|
501
|
+
the new process records a fresh baseline.
|
|
470
502
|
|
|
471
503
|
### What settles a green PR
|
|
472
504
|
|
|
@@ -496,6 +528,17 @@ merged and whose issues were all closed, through two daemon restarts — and bec
|
|
|
496
528
|
the active set *is* the busy set, those three issues were permanently unclaimable.
|
|
497
529
|
A status page that has stopped being evidence is worse than no status page.
|
|
498
530
|
|
|
531
|
+
### Continuation runs
|
|
532
|
+
|
|
533
|
+
When a worktree is provisioned onto a branch that already exists in the mirror
|
|
534
|
+
(reattach after a prior attempt, orphan, or turns-cap auto-requeue), the worker
|
|
535
|
+
brief includes a **Continuation** section: read `git log` / `git diff` against
|
|
536
|
+
the default branch first, and do not recreate work already on the branch.
|
|
537
|
+
|
|
538
|
+
A **turns-cap kill with attempts remaining** salvages the tree, puts the queue
|
|
539
|
+
label back on, and skips the failed label so the next tick reclaims as a
|
|
540
|
+
continuation automatically.
|
|
541
|
+
|
|
499
542
|
### Branch names
|
|
500
543
|
|
|
501
544
|
`<type>/<slug>`, where the type is `fix` when any label's last segment (after `:`
|
|
@@ -539,14 +582,14 @@ rest. `0` is a real value (a hard stop), not "unset".
|
|
|
539
582
|
| Cap | Default | What it protects |
|
|
540
583
|
| --- | --- | --- |
|
|
541
584
|
| `maxConcurrentWorkers` | `2` | Parallel omp sessions. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. Raise it only if you actually have the runners. |
|
|
542
|
-
| `dailySpendUsd` | `25` | Rolling-day spend ceiling
|
|
585
|
+
| `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total`. |
|
|
543
586
|
| `workerMaxTurns` | `120` | Turn ceiling for one worker. Catches a session looping without converging. |
|
|
544
587
|
| `workerWallClockMs` | `5400000` (90 minutes) | Wall-clock ceiling for one worker. A session that is merely stuck spends no turns, so turns alone cannot detect it. |
|
|
545
588
|
| `maxAttemptsPerIssue` | `2` | Retries per issue before it escalates. One clean retry recovers from flaky CI; a third attempt almost always means the issue itself is underspecified. |
|
|
546
589
|
|
|
547
590
|
Days are counted from **local midnight**, matching how a human reads "today".
|
|
548
591
|
|
|
549
|
-
Hitting `dailySpendUsd` is not the same as hitting the other caps. A concurrency
|
|
592
|
+
Set `dailySpendUsd` to `null` (wizard: blank) for no money gate — turns and wall-clock still apply. Hitting a numeric `dailySpendUsd` is not the same as hitting the other caps. A concurrency
|
|
550
593
|
limit simply defers work to a later tick. The spend cap **pauses the daemon and
|
|
551
594
|
pages at Tier 2**: a loop that is burning money has to halt itself, because
|
|
552
595
|
waiting for someone to notice tomorrow is how a runaway becomes expensive.
|
|
@@ -1065,9 +1108,9 @@ omp-conductor help
|
|
|
1065
1108
|
| Command | Behaviour |
|
|
1066
1109
|
| --- | --- |
|
|
1067
1110
|
| `start` | Spawn the loop in the background, detached, and wait until it answers `GET /healthz` on `:8787`. Refuses if one is already live, naming its pid. If the process dies or never serves, `start` cleans up after it and quotes the tail of `daemon.log`. |
|
|
1068
|
-
| `stop` | `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop. |
|
|
1069
|
-
| `restart` | `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them
|
|
1070
|
-
| `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. A `.conductor-stalled` marker in the state directory adds an `orchestrator STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
|
|
1111
|
+
| `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
|
|
1112
|
+
| `restart` | Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
|
|
1113
|
+
| `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. While live workers > 0, prints a `deploy` line naming the count so a busy restart is visible before you take it. A `.conductor-stalled` marker in the state directory adds an `orchestrator STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
|
|
1071
1114
|
| `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
|
|
1072
1115
|
| `unblock <issue>` | Remove that issue's `blocked` and `failed` state labels through the tracker, so the next tick can claim it again. This is the supported way back for an escalation you answered: eligibility disqualifies any issue carrying a state label, so an answered issue that keeps one is never re-claimed and the answer is inert. Removing a label the issue does not carry is a no-op, so both are always cleared and neither has to be looked up first. `agent:in-progress` is deliberately not touched — it means a worker process exists, which is not something an answer changes. The run history is left exactly as it is: an answered block still spent a worker, so it still counts toward `maxAttemptsPerIssue`, and the output says how many attempts remain — or warns that the next tick will escalate instead of dispatching, when none do. Exits `2` with `unblock needs an issue number` on a missing or malformed positional. |
|
|
1073
1116
|
| `daemon` | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. This is what `start` launches, and what a systemd unit should call. Writes the pidfile itself, and refuses with `another daemon is alive (pid N); stop it first` rather than becoming a second dispatcher. |
|
|
@@ -1187,10 +1230,14 @@ Known and deliberate in this version:
|
|
|
1187
1230
|
retried rather than corrupted.
|
|
1188
1231
|
- **Mirrors grow one branch ref per run.** Unpushed work is never discarded, so
|
|
1189
1232
|
refs accumulate until you reap them.
|
|
1190
|
-
- **`stop` is a deadline, not a clean drain.**
|
|
1191
|
-
|
|
1192
|
-
|
|
1193
|
-
|
|
1233
|
+
- **`stop` is a deadline, not a clean drain.** Whether it goes through
|
|
1234
|
+
`systemctl` or a raw signal, the loop is asked to finish the tick it is on, and
|
|
1235
|
+
a tick with a worker in flight can run for that worker's whole wall clock; the
|
|
1236
|
+
signal path escalates to `SIGKILL` after 10 seconds. There is no "stop once the
|
|
1237
|
+
current worker lands". A unit that supervises the daemon should set
|
|
1238
|
+
`SuccessExitStatus=0 143` (belt-and-braces for a handled `SIGTERM` exit) and
|
|
1239
|
+
operators should prefer `omp-conductor stop` / `systemctl stop` over a raw
|
|
1240
|
+
`kill`, so `Restart=on-failure` cannot misread a deliberate stop as a crash.
|
|
1194
1241
|
- **A failed orchestrator degrades quietly.** The daemon logs a warning and keeps
|
|
1195
1242
|
running, but tier-1 escalations then land in issue comments — which is exactly the
|
|
1196
1243
|
"nobody reads it until morning" path the orchestrator exists to avoid. The warning
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "omp-conductor",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.9",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
|
|
@@ -100,8 +100,11 @@ checked in this order:
|
|
|
100
100
|
- **Commits — pushed or not — or a PR that is not green.** Safe either way:
|
|
101
101
|
pushed work lives on the remote, and unpushed commits live on the run's branch
|
|
102
102
|
in the mirror, which a re-claim deliberately reattaches so the next worker
|
|
103
|
-
starts from them
|
|
104
|
-
|
|
103
|
+
starts from them with a **continuation brief** (read the log/diff first; do
|
|
104
|
+
not recreate existing work). Note what exists and release the label; the
|
|
105
|
+
attempt counter still bounds a loop of deaths. A turns-cap kill with attempts
|
|
106
|
+
left is re-queued automatically by the daemon — you should still notice it on
|
|
107
|
+
drain, but you do not have to invent the continuation prompt.
|
|
105
108
|
- **Genuinely nothing** (clean tree, no commits, no PR). Release the label and let
|
|
106
109
|
the next tick re-claim it clean.
|
|
107
110
|
|
package/src/briefs/worker.md
CHANGED
|
@@ -28,7 +28,7 @@ Then read the repo's own guidance before writing anything: `AGENTS.md`,
|
|
|
28
28
|
`CLAUDE.md`, `CONTEXT.md`, and any `docs/adr/` entry the issue touches. Those
|
|
29
29
|
files are canonical; your priors are not.
|
|
30
30
|
|
|
31
|
-
## Acceptance criteria
|
|
31
|
+
{{CONTINUATION}}## Acceptance criteria
|
|
32
32
|
|
|
33
33
|
{{ACCEPTANCE_CRITERIA}}
|
|
34
34
|
|
package/src/cli.ts
CHANGED
|
@@ -16,6 +16,7 @@ import {
|
|
|
16
16
|
DEFAULT_PORT,
|
|
17
17
|
healthCheck,
|
|
18
18
|
livingDaemon,
|
|
19
|
+
restartDaemon,
|
|
19
20
|
startDaemon,
|
|
20
21
|
stopDaemon,
|
|
21
22
|
writeRecord,
|
|
@@ -46,9 +47,14 @@ usage:
|
|
|
46
47
|
start run the dispatch loop in the background and wait until it answers
|
|
47
48
|
GET /healthz on :8787 (override with --port). Refuses if one is
|
|
48
49
|
already running.
|
|
49
|
-
stop
|
|
50
|
+
stop stop the running daemon. Uses systemctl when the omp-conductor
|
|
51
|
+
unit owns the process (so Restart=on-failure cannot bring it back);
|
|
52
|
+
otherwise SIGTERM then SIGKILL.
|
|
50
53
|
restart stop then start, keeping the running daemon's port and project
|
|
51
|
-
unless a flag overrides them.
|
|
54
|
+
unless a flag overrides them. On boot the new process salvages any
|
|
55
|
+
dirty live worktrees before orphaning those rows — see README
|
|
56
|
+
"Deploying a new package onto a busy fleet". Goes through systemctl
|
|
57
|
+
when the unit owns the live pid.
|
|
52
58
|
status show pause state, caps, active runs, today's usage, and whether a
|
|
53
59
|
daemon is alive.
|
|
54
60
|
tail follow the newest run for <issue>: the worker's assistant text and
|
|
@@ -398,27 +404,35 @@ try {
|
|
|
398
404
|
}
|
|
399
405
|
|
|
400
406
|
case "stop": {
|
|
401
|
-
// Read the pid before it stops existing, so the confirmation can name it.
|
|
402
|
-
const pid = livingDaemon()?.pid;
|
|
403
407
|
const result = await stopDaemon();
|
|
404
|
-
|
|
408
|
+
if (result.kind === "not-running") {
|
|
409
|
+
process.stdout.write("not running\n");
|
|
410
|
+
} else {
|
|
411
|
+
process.stdout.write(
|
|
412
|
+
`stopped — pid ${result.pid}${result.via === "systemctl" ? " (via systemctl)" : ""}\n`,
|
|
413
|
+
);
|
|
414
|
+
}
|
|
405
415
|
break;
|
|
406
416
|
}
|
|
407
417
|
|
|
408
418
|
case "restart": {
|
|
409
419
|
// Inherit the running daemon's port and project: a restart that quietly
|
|
410
420
|
// moved to the default port would leave every existing health check
|
|
411
|
-
// pointing at nothing.
|
|
412
|
-
|
|
413
|
-
const
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
port: portFlag(argv) ?? previous?.port,
|
|
417
|
-
project: flag(argv, "project") ?? previous?.project,
|
|
421
|
+
// pointing at nothing. When the unit owns the live pid, restartDaemon
|
|
422
|
+
// goes through systemctl so the replacement stays supervised.
|
|
423
|
+
const { previous, record, via } = await restartDaemon({
|
|
424
|
+
port: portFlag(argv),
|
|
425
|
+
project: flag(argv, "project"),
|
|
418
426
|
});
|
|
427
|
+
if (previous !== undefined) {
|
|
428
|
+
process.stdout.write(
|
|
429
|
+
`stopped — pid ${previous.pid}${via === "systemctl" ? " (via systemctl)" : ""}\n`,
|
|
430
|
+
);
|
|
431
|
+
}
|
|
419
432
|
process.stdout.write(
|
|
420
|
-
`started — pid ${
|
|
421
|
-
`${
|
|
433
|
+
`started — pid ${record.pid}, /healthz on :${record.port}` +
|
|
434
|
+
`${record.project === undefined ? "" : `, project ${record.project}`}` +
|
|
435
|
+
`${via === "systemctl" ? " (via systemctl)" : ""}\nlog ${record.logFile}\n`,
|
|
422
436
|
);
|
|
423
437
|
break;
|
|
424
438
|
}
|
package/src/config.ts
CHANGED
|
@@ -139,15 +139,16 @@ export function saveConfig(c: ConductorConfig): void {
|
|
|
139
139
|
/**
|
|
140
140
|
* Layers a project's overrides on the global defaults, field by field, so a
|
|
141
141
|
* project that pins one cap still inherits the other five. `??` not `||`: a
|
|
142
|
-
* deliberate `dailySpendUsd: 0` is a hard stop, not "unset"
|
|
143
|
-
*
|
|
144
|
-
*
|
|
142
|
+
* deliberate `dailySpendUsd: 0` is a hard stop, not "unset", and `null` is a
|
|
143
|
+
* deliberate "no spend gate" that must not fall through to the default.
|
|
144
|
+
* Spelled out per field so adding a `Caps` member fails to compile here.
|
|
145
145
|
*/
|
|
146
146
|
export function resolveCaps(p: ProjectConfig, defaults: Caps): Caps {
|
|
147
147
|
const o: Partial<Caps> = p.caps ?? {};
|
|
148
148
|
return {
|
|
149
149
|
maxConcurrentWorkers: o.maxConcurrentWorkers ?? defaults.maxConcurrentWorkers,
|
|
150
|
-
|
|
150
|
+
// nullish only — `null` means off; do not coalesce it to the default.
|
|
151
|
+
dailySpendUsd: o.dailySpendUsd !== undefined ? o.dailySpendUsd : defaults.dailySpendUsd,
|
|
151
152
|
workerMaxTurns: o.workerMaxTurns ?? defaults.workerMaxTurns,
|
|
152
153
|
workerWallClockMs: o.workerWallClockMs ?? defaults.workerWallClockMs,
|
|
153
154
|
maxAttemptsPerIssue: o.maxAttemptsPerIssue ?? defaults.maxAttemptsPerIssue,
|
|
@@ -520,8 +521,18 @@ function coerceCaps(parsed: unknown, label: string, problems: string[], legacy:
|
|
|
520
521
|
for (const key of CAP_KEYS) {
|
|
521
522
|
const v = raw[key];
|
|
522
523
|
if (v === undefined) continue;
|
|
524
|
+
// Spend is the only cap that may be null (= no gate). Every other ceiling
|
|
525
|
+
// is a non-negative number; 0 remains a hard stop where it already was.
|
|
526
|
+
if (key === "dailySpendUsd" && v === null) {
|
|
527
|
+
out.dailySpendUsd = null;
|
|
528
|
+
continue;
|
|
529
|
+
}
|
|
523
530
|
if (typeof v !== "number" || !Number.isFinite(v) || v < 0) {
|
|
524
|
-
problems.push(
|
|
531
|
+
problems.push(
|
|
532
|
+
key === "dailySpendUsd"
|
|
533
|
+
? `${label}.${key} must be a non-negative finite number or null (no cap), found ${JSON.stringify(v)}`
|
|
534
|
+
: `${label}.${key} must be a non-negative finite number, found ${JSON.stringify(v)}`,
|
|
535
|
+
);
|
|
525
536
|
continue;
|
|
526
537
|
}
|
|
527
538
|
out[key] = v;
|
package/src/daemon.ts
CHANGED
|
@@ -444,10 +444,35 @@ export async function buildBrief(
|
|
|
444
444
|
r: Routed,
|
|
445
445
|
branch: string,
|
|
446
446
|
worktree: string,
|
|
447
|
+
opts: { continuation?: boolean; defaultBranch?: string } = {},
|
|
447
448
|
): Promise<string> {
|
|
448
449
|
// Read per dispatch rather than caching: editing the brief then takes effect
|
|
449
450
|
// on the next issue instead of needing a daemon restart.
|
|
450
451
|
const template = await Bun.file(BRIEF_TEMPLATE_PATH).text();
|
|
452
|
+
const defaultBranch = opts.defaultBranch ?? r.repo.defaultBranch;
|
|
453
|
+
const continuation =
|
|
454
|
+
opts.continuation === true
|
|
455
|
+
? [
|
|
456
|
+
"",
|
|
457
|
+
"## Continuation — do not start from zero",
|
|
458
|
+
"",
|
|
459
|
+
`You are **resuming** issue #${r.issue.number}. Branch \`${branch}\` already exists`,
|
|
460
|
+
"and was reattached with prior commits (and possibly a salvaged WIP tip).",
|
|
461
|
+
"Before writing anything:",
|
|
462
|
+
"",
|
|
463
|
+
"```bash",
|
|
464
|
+
"git log --oneline origin/" + defaultBranch + "..HEAD",
|
|
465
|
+
"git diff --stat origin/" + defaultBranch + "...HEAD",
|
|
466
|
+
"git status --porcelain",
|
|
467
|
+
"```",
|
|
468
|
+
"",
|
|
469
|
+
"Read that history. **Do not recreate work that already exists.** Finish",
|
|
470
|
+
"what remains against the same acceptance criteria. If a prior attempt",
|
|
471
|
+
"left a `wip(#…): … auto-salvaged` commit, treat it as your starting",
|
|
472
|
+
"point, not as trash to rewrite from scratch.",
|
|
473
|
+
"",
|
|
474
|
+
].join("\n")
|
|
475
|
+
: "";
|
|
451
476
|
return renderBrief(template, {
|
|
452
477
|
ISSUE_NUMBER: String(r.issue.number),
|
|
453
478
|
ISSUE_TITLE: r.issue.title,
|
|
@@ -461,6 +486,7 @@ export async function buildBrief(
|
|
|
461
486
|
// placeholder sits flush against the next list item in the template, so an
|
|
462
487
|
// unconfigured render leaves no blank line where a hint would have gone.
|
|
463
488
|
GRAPH_HINT: graphHint(r.repo),
|
|
489
|
+
CONTINUATION: continuation,
|
|
464
490
|
});
|
|
465
491
|
}
|
|
466
492
|
|
|
@@ -513,14 +539,14 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
513
539
|
// would cost a second network fetch per attempt.
|
|
514
540
|
const mirrorPath = mirrorPathFor(r.repo, project.mirrorRoot);
|
|
515
541
|
await removeWorktree(mirrorPath, worktreePathFor(project.workspaceRoot, issue));
|
|
516
|
-
|
|
517
|
-
worktreePath = await addWorktree(
|
|
542
|
+
const provisioned = await addWorktree(
|
|
518
543
|
r.repo,
|
|
519
544
|
project.mirrorRoot,
|
|
520
545
|
project.workspaceRoot,
|
|
521
546
|
issue,
|
|
522
547
|
branch,
|
|
523
548
|
);
|
|
549
|
+
worktreePath = provisioned.path;
|
|
524
550
|
|
|
525
551
|
// The SDK names the transcript itself, so the daemon supplies the parent
|
|
526
552
|
// directory and learns the real path back from the result. Inventing one
|
|
@@ -529,10 +555,16 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
529
555
|
mkdirSync(sessionDir, { recursive: true });
|
|
530
556
|
store.updateRun(runId, { worktree: worktreePath, state: "running" });
|
|
531
557
|
|
|
532
|
-
log(
|
|
558
|
+
log(
|
|
559
|
+
`#${issue} attempt ${attempt} → ${r.repo.name} ${branch}` +
|
|
560
|
+
(provisioned.reattached ? " (continuation: reattached existing branch)" : ""),
|
|
561
|
+
);
|
|
533
562
|
|
|
534
563
|
const result = await runWorker({
|
|
535
|
-
brief: await buildBrief(project, r, branch, worktreePath
|
|
564
|
+
brief: await buildBrief(project, r, branch, worktreePath, {
|
|
565
|
+
continuation: provisioned.reattached,
|
|
566
|
+
defaultBranch: r.repo.defaultBranch,
|
|
567
|
+
}),
|
|
536
568
|
cwd: worktreePath,
|
|
537
569
|
caps,
|
|
538
570
|
sessionDir,
|
|
@@ -572,30 +604,61 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
572
604
|
detail: [`${r.issue.title}`, r.issue.url, "", result.report].join("\n"),
|
|
573
605
|
});
|
|
574
606
|
} else if (result.state === "failed" || result.state === "killed") {
|
|
575
|
-
|
|
576
|
-
//
|
|
577
|
-
//
|
|
607
|
+
// Turns-cap with attempts left: salvage, put the issue back on the queue,
|
|
608
|
+
// and skip the failed label so the next tick reclaims as a continuation
|
|
609
|
+
// instead of burning a human triage cycle (#50 / #21).
|
|
610
|
+
const continueTurns =
|
|
611
|
+
result.killedBy === "turns" && attempt < caps.maxAttemptsPerIssue;
|
|
578
612
|
const salvaged = await salvage(issue, attempt, endedBy(result.killedBy), worktreePath);
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
issue,
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
:
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
613
|
+
|
|
614
|
+
if (continueTurns) {
|
|
615
|
+
await tracker.removeLabel(issue, inProgress);
|
|
616
|
+
await tracker.addLabel(issue, project.queueLabel);
|
|
617
|
+
log(
|
|
618
|
+
`#${issue} turns-cap on attempt ${attempt}/${caps.maxAttemptsPerIssue} — ` +
|
|
619
|
+
`salvaged and re-queued for continuation`,
|
|
620
|
+
);
|
|
621
|
+
await safeEscalate(d, {
|
|
622
|
+
tier: 1,
|
|
623
|
+
project: project.name,
|
|
624
|
+
issue,
|
|
625
|
+
runId,
|
|
626
|
+
summary: `#${issue} hit the turns cap on attempt ${attempt} — auto-requeued for continuation`,
|
|
627
|
+
detail: [
|
|
628
|
+
`${r.issue.title}`,
|
|
629
|
+
r.issue.url,
|
|
630
|
+
...salvaged,
|
|
631
|
+
`Session: ${result.sessionFile ?? "(no transcript)"}`,
|
|
632
|
+
"",
|
|
633
|
+
"The queue label is back on; the next tick should reattach the branch",
|
|
634
|
+
"and open a continuation brief. No failed label was applied.",
|
|
635
|
+
"",
|
|
636
|
+
result.report,
|
|
637
|
+
].join("\n"),
|
|
638
|
+
});
|
|
639
|
+
} else {
|
|
640
|
+
await swapLabel(tracker, issue, inProgress, project.stateLabels.failed);
|
|
641
|
+
await safeEscalate(d, {
|
|
642
|
+
tier: 1,
|
|
643
|
+
project: project.name,
|
|
644
|
+
issue,
|
|
645
|
+
runId,
|
|
646
|
+
// The dedup key includes the summary, so the attempt number is what
|
|
647
|
+
// lets a genuine second failure page again while a tick that keeps
|
|
648
|
+
// seeing the same dead issue stays quiet.
|
|
649
|
+
summary: result.killedBy
|
|
650
|
+
? `#${issue} was killed on attempt ${attempt} by the ${result.killedBy} cap`
|
|
651
|
+
: `#${issue} failed on attempt ${attempt}`,
|
|
652
|
+
detail: [
|
|
653
|
+
`${r.issue.title}`,
|
|
654
|
+
r.issue.url,
|
|
655
|
+
...salvaged,
|
|
656
|
+
`Session: ${result.sessionFile ?? "(no transcript)"}`,
|
|
657
|
+
"",
|
|
658
|
+
result.report,
|
|
659
|
+
].join("\n"),
|
|
660
|
+
});
|
|
661
|
+
}
|
|
599
662
|
} else {
|
|
600
663
|
// pushed-green: the PR belongs to a human now. The in-progress label
|
|
601
664
|
// stays on until the merge closes the issue, which is also what keeps
|
|
@@ -923,9 +986,10 @@ async function tick(d: Deps): Promise<void> {
|
|
|
923
986
|
|
|
924
987
|
// Spend is the one cap that stops the fleet instead of merely deferring work.
|
|
925
988
|
// A loop that is burning money has to halt itself; waiting for a human to
|
|
926
|
-
// notice tomorrow is how a runaway becomes expensive.
|
|
989
|
+
// notice tomorrow is how a runaway becomes expensive. `null` means the
|
|
990
|
+
// operator opted out — turns and wall-clock still brake every run (#46).
|
|
927
991
|
const spent = store.spendSince(project.name, since);
|
|
928
|
-
if (spent >= caps.dailySpendUsd) {
|
|
992
|
+
if (caps.dailySpendUsd !== null && spent >= caps.dailySpendUsd) {
|
|
929
993
|
setPaused(true);
|
|
930
994
|
await safeEscalate(d, {
|
|
931
995
|
tier: 2,
|
|
@@ -1011,7 +1075,9 @@ export function formatStatus(s: StatusSnapshot): string {
|
|
|
1011
1075
|
"caps",
|
|
1012
1076
|
` workers ${s.liveWorkers} / ${s.caps.maxConcurrentWorkers}`,
|
|
1013
1077
|
` issues today ${s.runsToday}`,
|
|
1014
|
-
|
|
1078
|
+
s.caps.dailySpendUsd === null
|
|
1079
|
+
? ` spend today $${s.spendTodayUsd.toFixed(2)} (no daily cap)`
|
|
1080
|
+
: ` spend today $${s.spendTodayUsd.toFixed(2)} / $${s.caps.dailySpendUsd.toFixed(2)}`,
|
|
1015
1081
|
` worker max turns ${s.caps.workerMaxTurns}`,
|
|
1016
1082
|
` worker wall clock ${Math.round(s.caps.workerWallClockMs / 60_000)}m`,
|
|
1017
1083
|
` attempts per issue ${s.caps.maxAttemptsPerIssue}`,
|
|
@@ -1029,6 +1095,15 @@ export function formatStatus(s: StatusSnapshot): string {
|
|
|
1029
1095
|
);
|
|
1030
1096
|
}
|
|
1031
1097
|
}
|
|
1098
|
+
// Deploy hint: a restart while workers are live orphans them (salvage runs
|
|
1099
|
+
// first — #35). Prefer pause + drain to zero live workers when you can wait.
|
|
1100
|
+
if (s.liveWorkers > 0) {
|
|
1101
|
+
lines.push(
|
|
1102
|
+
"",
|
|
1103
|
+
`deploy ${s.liveWorkers} live worker(s) — restart salvages dirty trees then orphans the rows; ` +
|
|
1104
|
+
`pause and wait for workers 0/${s.caps.maxConcurrentWorkers} when you can drain instead`,
|
|
1105
|
+
);
|
|
1106
|
+
}
|
|
1032
1107
|
return lines.join("\n");
|
|
1033
1108
|
}
|
|
1034
1109
|
|
|
@@ -1085,18 +1160,18 @@ export function armConductor(): void {
|
|
|
1085
1160
|
}
|
|
1086
1161
|
|
|
1087
1162
|
/**
|
|
1088
|
-
* Settles
|
|
1163
|
+
* Settles `claimed`/`running` rows left by a dead daemon process and, before
|
|
1164
|
+
* marking each one `orphaned`, salvages any dirty worktree.
|
|
1089
1165
|
*
|
|
1090
|
-
*
|
|
1091
|
-
*
|
|
1092
|
-
*
|
|
1093
|
-
*
|
|
1094
|
-
*
|
|
1095
|
-
* (found live, after a host restart killed two workers mid-run).
|
|
1166
|
+
* Found live after a host restart killed two workers mid-run, and again on
|
|
1167
|
+
* every package deploy that restarted while workers were live (#35): without
|
|
1168
|
+
* the salvage call the next attempt's `worktree remove --force` destroyed
|
|
1169
|
+
* uncommitted edits that had no other copy. Cap-kills already salvaged (#27);
|
|
1170
|
+
* this is the same call site for the restart path.
|
|
1096
1171
|
*
|
|
1097
1172
|
* Only the rows change. The issue keeps its in-progress label — that label is
|
|
1098
1173
|
* the crash guard against double-dispatch, and deciding what a dead worker's
|
|
1099
|
-
* remains are worth (an open PR?
|
|
1174
|
+
* remains are worth (an open PR? a salvaged sha? a clean tree?) is the
|
|
1100
1175
|
* orchestrator's drain-duty judgement, not something to automate here. The
|
|
1101
1176
|
* rows also keep counting toward `maxAttemptsPerIssue`, so a loop of deaths
|
|
1102
1177
|
* still escalates instead of retrying forever.
|
|
@@ -1107,12 +1182,21 @@ export function armConductor(): void {
|
|
|
1107
1182
|
* them is {@link settlePushedGreen}, on the tick, by asking the tracker what
|
|
1108
1183
|
* became of the PR — the one question a restart cannot answer by inference.
|
|
1109
1184
|
*/
|
|
1110
|
-
export function reconcileOrphanedRuns(
|
|
1185
|
+
export async function reconcileOrphanedRuns(
|
|
1186
|
+
store: Store,
|
|
1187
|
+
project: string,
|
|
1188
|
+
): Promise<RunRecord[]> {
|
|
1111
1189
|
// Live runs only: `pushed-green` holds no process, so it cannot be orphaned by
|
|
1112
1190
|
// a process dying — it is finished work waiting on a human merge.
|
|
1113
1191
|
const stale = store.liveRuns(project);
|
|
1114
1192
|
const endedAt = Date.now();
|
|
1115
1193
|
for (const r of stale) {
|
|
1194
|
+
// Salvage before the row flips: the worktree path is on the record, and
|
|
1195
|
+
// salvageWip is a no-op for a missing/clean tree. Reason string matches
|
|
1196
|
+
// the cap-kill wording so triage reads the same either way.
|
|
1197
|
+
if (r.worktree !== "") {
|
|
1198
|
+
await salvage(r.issue, r.attempt, "a daemon restart", r.worktree);
|
|
1199
|
+
}
|
|
1116
1200
|
store.updateRun(r.id, { state: "orphaned", endedAt });
|
|
1117
1201
|
}
|
|
1118
1202
|
return stale;
|
|
@@ -1141,7 +1225,7 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
|
|
|
1141
1225
|
// daemon must not orphan that daemon's real, live workers).
|
|
1142
1226
|
const alive = livingDaemon();
|
|
1143
1227
|
if (alive === undefined || alive.pid === process.pid) {
|
|
1144
|
-
for (const r of reconcileOrphanedRuns(store, project.name)) {
|
|
1228
|
+
for (const r of await reconcileOrphanedRuns(store, project.name)) {
|
|
1145
1229
|
log(
|
|
1146
1230
|
`#${r.issue} orphaned by a previous daemon (attempt ${r.attempt}, was ${r.state}, worktree ${r.worktree}) — ` +
|
|
1147
1231
|
`slot freed; the ${project.stateLabels.inProgress} label stays until the orchestrator triages what the worker left`,
|
|
@@ -1283,5 +1367,11 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
|
|
|
1283
1367
|
await orchestrator?.dispose();
|
|
1284
1368
|
store.close();
|
|
1285
1369
|
log("stopped");
|
|
1370
|
+
// A handled SIGTERM still leaves some runtimes with a non-zero default
|
|
1371
|
+
// (historically 128+signal). Under systemd `Restart=on-failure` that looks
|
|
1372
|
+
// like a crash and the unit comes straight back — the exact failure mode
|
|
1373
|
+
// `omp-conductor stop` hit on the reference fleet. Force success so a
|
|
1374
|
+
// graceful drain is not a restart.
|
|
1375
|
+
process.exitCode = 0;
|
|
1286
1376
|
}
|
|
1287
1377
|
}
|
package/src/lifecycle.ts
CHANGED
|
@@ -10,16 +10,30 @@
|
|
|
10
10
|
* operator real time are `start` refusing against a ghost and `status`
|
|
11
11
|
* reporting a daemon that died hours ago.
|
|
12
12
|
*
|
|
13
|
+
* When the live pid is the MainPID of the `omp-conductor.service` unit,
|
|
14
|
+
* `stop`/`restart` go through `systemctl` rather than a raw `SIGTERM`. A
|
|
15
|
+
* raw signal against a unit with `Restart=on-failure` is read as a crash and
|
|
16
|
+
* the unit comes straight back — the bug that made `omp-conductor stop` look
|
|
17
|
+
* like a no-op on a systemd-managed host.
|
|
18
|
+
*
|
|
13
19
|
* Deliberately free of every other module in this package: nothing here opens
|
|
14
20
|
* the store, loads the config or talks to `gh`, so `stop` and `status` keep
|
|
15
21
|
* working when the config is the very thing that is broken.
|
|
16
22
|
*/
|
|
17
23
|
|
|
18
|
-
import { spawn } from "node:child_process";
|
|
24
|
+
import { spawn, spawnSync } from "node:child_process";
|
|
19
25
|
import { chmodSync, closeSync, mkdirSync, openSync, readFileSync, readSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs";
|
|
20
26
|
import { homedir } from "node:os";
|
|
21
27
|
import { join } from "node:path";
|
|
22
28
|
|
|
29
|
+
/**
|
|
30
|
+
* The systemd unit name operators are expected to install for a supervised
|
|
31
|
+
* daemon. Hardcoded rather than configured: one name on the host is the
|
|
32
|
+
* whole point of a unit, and a wrong name would silently fall back to the
|
|
33
|
+
* SIGTERM path and reintroduce the restart loop this module exists to avoid.
|
|
34
|
+
*/
|
|
35
|
+
export const SYSTEMD_UNIT = "omp-conductor.service";
|
|
36
|
+
|
|
23
37
|
/**
|
|
24
38
|
* Mirrors `DEFAULT_PORT` in ./daemon.ts. Duplicated rather than imported so
|
|
25
39
|
* this module stays free of the dispatcher's dependency tree; exported so the
|
|
@@ -271,23 +285,67 @@ export async function startDaemon(o: { port?: number; project?: string } = {}):
|
|
|
271
285
|
}
|
|
272
286
|
|
|
273
287
|
/**
|
|
274
|
-
*
|
|
288
|
+
* How the last stop actually landed. Callers print this so an operator can
|
|
289
|
+
* tell a supervised stop from a bare SIGTERM without reading the journal.
|
|
290
|
+
*/
|
|
291
|
+
export type StopResult =
|
|
292
|
+
| { kind: "stopped"; pid: number; via: "systemctl" | "signal" }
|
|
293
|
+
| { kind: "not-running" };
|
|
294
|
+
|
|
295
|
+
/**
|
|
296
|
+
* Stops the daemon.
|
|
275
297
|
*
|
|
276
|
-
* `
|
|
277
|
-
*
|
|
278
|
-
*
|
|
279
|
-
*
|
|
280
|
-
*
|
|
281
|
-
*
|
|
298
|
+
* Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is
|
|
299
|
+
* the live daemon: systemd then owns the stop and will not schedule a restart
|
|
300
|
+
* for the exit it just requested. A raw `SIGTERM` against a unit with
|
|
301
|
+
* `Restart=on-failure` is what made `omp-conductor stop` look like a no-op on
|
|
302
|
+
* the reference fleet — exit 143 was a "failure", and the unit came back five
|
|
303
|
+
* seconds later.
|
|
304
|
+
*
|
|
305
|
+
* Falls back to `SIGTERM` then `SIGKILL` when there is no unit, the unit is
|
|
306
|
+
* not running, or its MainPID is somebody else (a hand-started daemon next to
|
|
307
|
+
* a stopped unit). The grace period is a deadline, not a clean drain: a tick
|
|
308
|
+
* with a worker in flight can run for that worker's whole wall clock.
|
|
282
309
|
*/
|
|
283
|
-
export async function stopDaemon(o: { timeoutMs?: number } = {}): Promise<
|
|
310
|
+
export async function stopDaemon(o: { timeoutMs?: number } = {}): Promise<StopResult> {
|
|
284
311
|
const rec = livingDaemon();
|
|
285
312
|
if (rec === undefined) {
|
|
286
313
|
// `livingDaemon` already cleared a stale file; this covers the unparseable
|
|
287
|
-
// one it refused to read.
|
|
314
|
+
// one it refused to read. Still ask systemd: a unit can be running with
|
|
315
|
+
// no pidfile (boot race, wiped runtime dir) and stop must still land.
|
|
316
|
+
const unitPid = systemdMainPid();
|
|
317
|
+
if (unitPid !== undefined && (await stopViaSystemd(undefined))) {
|
|
318
|
+
const deadline = Date.now() + (o.timeoutMs ?? STOP_TIMEOUT_MS);
|
|
319
|
+
while (isAlive(unitPid) && Date.now() < deadline) await sleep(100);
|
|
320
|
+
if (isAlive(unitPid)) {
|
|
321
|
+
throw new Error(
|
|
322
|
+
`daemon pid ${unitPid} still alive after systemctl stop ${SYSTEMD_UNIT} — ` +
|
|
323
|
+
`check \`systemctl status ${SYSTEMD_UNIT}\``,
|
|
324
|
+
);
|
|
325
|
+
}
|
|
326
|
+
clearRecord();
|
|
327
|
+
return { kind: "stopped", pid: unitPid, via: "systemctl" };
|
|
328
|
+
}
|
|
288
329
|
clearRecord();
|
|
289
|
-
return "not-running";
|
|
330
|
+
return { kind: "not-running" };
|
|
290
331
|
}
|
|
332
|
+
|
|
333
|
+
if (await stopViaSystemd(rec.pid)) {
|
|
334
|
+
// Wait out the unit: systemctl stop is synchronous for Type=simple, but
|
|
335
|
+
// a slow drain still holds the old pid briefly and the next start would
|
|
336
|
+
// refuse against it.
|
|
337
|
+
const deadline = Date.now() + (o.timeoutMs ?? STOP_TIMEOUT_MS);
|
|
338
|
+
while (isAlive(rec.pid) && Date.now() < deadline) await sleep(100);
|
|
339
|
+
if (isAlive(rec.pid)) {
|
|
340
|
+
throw new Error(
|
|
341
|
+
`daemon pid ${rec.pid} still alive after systemctl stop ${SYSTEMD_UNIT} — ` +
|
|
342
|
+
`check \`systemctl status ${SYSTEMD_UNIT}\``,
|
|
343
|
+
);
|
|
344
|
+
}
|
|
345
|
+
clearRecord();
|
|
346
|
+
return { kind: "stopped", pid: rec.pid, via: "systemctl" };
|
|
347
|
+
}
|
|
348
|
+
|
|
291
349
|
const gone = await terminate(rec.pid, o.timeoutMs ?? STOP_TIMEOUT_MS);
|
|
292
350
|
if (!gone) {
|
|
293
351
|
// The record stays: something is still holding that pid, and forgetting
|
|
@@ -295,13 +353,146 @@ export async function stopDaemon(o: { timeoutMs?: number } = {}): Promise<"stopp
|
|
|
295
353
|
throw new Error(`daemon pid ${rec.pid} survived SIGTERM and SIGKILL — it may belong to another user`);
|
|
296
354
|
}
|
|
297
355
|
clearRecord();
|
|
298
|
-
return "stopped";
|
|
356
|
+
return { kind: "stopped", pid: rec.pid, via: "signal" };
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
/**
|
|
360
|
+
* Restarts the daemon.
|
|
361
|
+
*
|
|
362
|
+
* Same ownership rule as {@link stopDaemon}: when the unit owns the live pid,
|
|
363
|
+
* `systemctl restart` keeps systemd in charge of the replacement process so
|
|
364
|
+
* the new MainPID is still the unit's. Falling back to stop+start for a
|
|
365
|
+
* hand-started daemon would leave the unit dead and the new process
|
|
366
|
+
* unsupervised — fine for a laptop, wrong for the host that installed a unit
|
|
367
|
+
* specifically so a crash comes back.
|
|
368
|
+
*
|
|
369
|
+
* Returns the record of the process that is now answering `/healthz`.
|
|
370
|
+
*/
|
|
371
|
+
export async function restartDaemon(
|
|
372
|
+
o: { port?: number; project?: string; timeoutMs?: number } = {},
|
|
373
|
+
): Promise<{ previous: DaemonRecord | undefined; record: DaemonRecord; via: "systemctl" | "cli" }> {
|
|
374
|
+
const previous = livingDaemon();
|
|
375
|
+
const unitPid = systemdMainPid();
|
|
376
|
+
const unitOwns =
|
|
377
|
+
unitPid !== undefined && (previous === undefined || previous.pid === unitPid);
|
|
378
|
+
|
|
379
|
+
if (unitOwns) {
|
|
380
|
+
const ran = systemctl(["restart", SYSTEMD_UNIT]);
|
|
381
|
+
if (ran.ok) {
|
|
382
|
+
// systemctl restart returns once the new MainPID is up; the pidfile is
|
|
383
|
+
// written by the daemon itself on boot, so wait for that rather than
|
|
384
|
+
// inventing a record from the unit alone.
|
|
385
|
+
const deadline = Date.now() + READY_TIMEOUT_MS;
|
|
386
|
+
for (;;) {
|
|
387
|
+
const rec = livingDaemon();
|
|
388
|
+
if (rec !== undefined) {
|
|
389
|
+
const health = await healthCheck(rec.port);
|
|
390
|
+
if (health.ok) return { previous, record: rec, via: "systemctl" };
|
|
391
|
+
}
|
|
392
|
+
if (Date.now() >= deadline) break;
|
|
393
|
+
await sleep(READY_POLL_MS);
|
|
394
|
+
}
|
|
395
|
+
throw new Error(
|
|
396
|
+
`systemctl restart ${SYSTEMD_UNIT} returned, but the daemon never answered /healthz`,
|
|
397
|
+
);
|
|
398
|
+
}
|
|
399
|
+
// Unit exists and owns the pid but systemctl refused (permissions, dbus
|
|
400
|
+
// down). Fall through to the signal path rather than stranding the
|
|
401
|
+
// operator with "restart failed" and a still-running daemon they cannot
|
|
402
|
+
// reach through the unit.
|
|
403
|
+
}
|
|
404
|
+
|
|
405
|
+
await stopDaemon({ timeoutMs: o.timeoutMs });
|
|
406
|
+
const record = await startDaemon({ port: o.port ?? previous?.port, project: o.project ?? previous?.project });
|
|
407
|
+
return { previous, record, via: "cli" };
|
|
408
|
+
}
|
|
409
|
+
|
|
410
|
+
/**
|
|
411
|
+
* The MainPID of {@link SYSTEMD_UNIT}, or `undefined` when systemd is absent,
|
|
412
|
+
* the unit is unknown, or it is not running. Never throws: a missing binary
|
|
413
|
+
* or a dbus blip is "no unit", and stop falls back to SIGTERM.
|
|
414
|
+
*/
|
|
415
|
+
export function systemdMainPid(unit = SYSTEMD_UNIT): number | undefined {
|
|
416
|
+
const ran = systemctl(["show", unit, "--property=MainPID", "--property=ActiveState", "--value"]);
|
|
417
|
+
if (!ran.ok) return undefined;
|
|
418
|
+
// `systemctl show --value` prints one property per line, MainPID then
|
|
419
|
+
// ActiveState, in the order requested. Tolerate either order and blank
|
|
420
|
+
// lines so a future systemctl rearrange does not silently disable the path.
|
|
421
|
+
const lines = ran.stdout
|
|
422
|
+
.split("\n")
|
|
423
|
+
.map((l) => l.trim())
|
|
424
|
+
.filter((l) => l.length > 0);
|
|
425
|
+
let pid: number | undefined;
|
|
426
|
+
let active: string | undefined;
|
|
427
|
+
for (const line of lines) {
|
|
428
|
+
if (/^\d+$/.test(line)) {
|
|
429
|
+
const n = Number(line);
|
|
430
|
+
if (Number.isInteger(n) && n > 1) pid = n;
|
|
431
|
+
} else {
|
|
432
|
+
active = line;
|
|
433
|
+
}
|
|
434
|
+
}
|
|
435
|
+
if (active !== undefined && active !== "active" && active !== "reactivating") return undefined;
|
|
436
|
+
return pid;
|
|
299
437
|
}
|
|
300
438
|
|
|
301
439
|
// ---------------------------------------------------------------------------
|
|
302
440
|
// internals
|
|
303
441
|
// ---------------------------------------------------------------------------
|
|
304
442
|
|
|
443
|
+
/**
|
|
444
|
+
* Ask systemd to stop the unit, but only when it actually owns `pid`.
|
|
445
|
+
*
|
|
446
|
+
* `pid === undefined` means "no pidfile" — still stop the unit if it is
|
|
447
|
+
* active, because that is the only process that could be the daemon. A unit
|
|
448
|
+
* whose MainPID is some other process is left alone: stopping it would take
|
|
449
|
+
* down a neighbour, and the signal path handles *our* pid.
|
|
450
|
+
*/
|
|
451
|
+
async function stopViaSystemd(pid: number | undefined): Promise<boolean> {
|
|
452
|
+
const main = systemdMainPid();
|
|
453
|
+
if (main === undefined) return false;
|
|
454
|
+
if (pid !== undefined && main !== pid) return false;
|
|
455
|
+
const ran = systemctl(["stop", SYSTEMD_UNIT]);
|
|
456
|
+
return ran.ok;
|
|
457
|
+
}
|
|
458
|
+
|
|
459
|
+
/**
|
|
460
|
+
* Run one `systemctl` invocation. Captures output and never throws: absence
|
|
461
|
+
* of the binary, a missing unit, or a permission error are all "not ok", and
|
|
462
|
+
* the caller decides whether to fall back.
|
|
463
|
+
*
|
|
464
|
+
* The default shells out. Tests replace it with {@link setSystemctlForTest}
|
|
465
|
+
* so the ownership decision is exercised without a real systemd.
|
|
466
|
+
*/
|
|
467
|
+
export type SystemctlFn = (args: string[]) => { ok: boolean; stdout: string; stderr: string };
|
|
468
|
+
|
|
469
|
+
function defaultSystemctl(args: string[]): { ok: boolean; stdout: string; stderr: string } {
|
|
470
|
+
try {
|
|
471
|
+
const res = spawnSync("systemctl", args, {
|
|
472
|
+
encoding: "utf8",
|
|
473
|
+
// A hung dbus is not worth blocking stop on; the signal path is right there.
|
|
474
|
+
timeout: 15_000,
|
|
475
|
+
env: process.env,
|
|
476
|
+
});
|
|
477
|
+
if (res.error) return { ok: false, stdout: "", stderr: res.error.message };
|
|
478
|
+
return {
|
|
479
|
+
ok: res.status === 0,
|
|
480
|
+
stdout: res.stdout ?? "",
|
|
481
|
+
stderr: res.stderr ?? "",
|
|
482
|
+
};
|
|
483
|
+
} catch (err) {
|
|
484
|
+
return { ok: false, stdout: "", stderr: err instanceof Error ? err.message : String(err) };
|
|
485
|
+
}
|
|
486
|
+
}
|
|
487
|
+
|
|
488
|
+
let systemctl: SystemctlFn = defaultSystemctl;
|
|
489
|
+
|
|
490
|
+
/** Test-only: replace the `systemctl` runner. Pass `undefined` to restore. */
|
|
491
|
+
export function setSystemctlForTest(fn: SystemctlFn | undefined): void {
|
|
492
|
+
systemctl = fn ?? defaultSystemctl;
|
|
493
|
+
}
|
|
494
|
+
|
|
495
|
+
|
|
305
496
|
function sleep(ms: number): Promise<void> {
|
|
306
497
|
return new Promise<void>((resolve) => {
|
|
307
498
|
const t = setTimeout(resolve, ms);
|
package/src/omp.ts
CHANGED
|
@@ -162,6 +162,11 @@ export async function createSession(opts: {
|
|
|
162
162
|
// sharing it fails to start. The daemon runs `maxConcurrentWorkers`
|
|
163
163
|
// (2 by default) workers at once, which makes this the normal path.
|
|
164
164
|
agentRegistry: new mod.AgentRegistry(),
|
|
165
|
+
// Default true in the SDK, but pass it explicitly so a harness change
|
|
166
|
+
// cannot silently strip MCP from workers. Discovery walks cwd + user
|
|
167
|
+
// agentDir (~/.omp/agent/mcp.json) — without this, workers grep-only and
|
|
168
|
+
// burn the turns cap on discovery (#29).
|
|
169
|
+
enableMCP: true,
|
|
165
170
|
});
|
|
166
171
|
const raw = asRawSession(created);
|
|
167
172
|
// Surfaced rather than swallowed: this is how a quiet downgrade to a weaker
|
package/src/plugin.ts
CHANGED
|
@@ -193,6 +193,30 @@ async function askNumber(ctx: CommandContext, title: string, fallback: number):
|
|
|
193
193
|
return value;
|
|
194
194
|
}
|
|
195
195
|
|
|
196
|
+
/**
|
|
197
|
+
* Daily spend ceiling. Blank / "none" / "off" → null (no gate). Unparseable
|
|
198
|
+
* non-empty input keeps the current value. Distinct from askNumber so operators
|
|
199
|
+
* can turn the money brake off without writing a magic 0 (which is a hard stop).
|
|
200
|
+
*/
|
|
201
|
+
async function askSpendCap(
|
|
202
|
+
ctx: CommandContext,
|
|
203
|
+
title: string,
|
|
204
|
+
fallback: number | null,
|
|
205
|
+
): Promise<number | null> {
|
|
206
|
+
const seed = fallback === null ? "" : String(fallback);
|
|
207
|
+
const raw = (await ask(ctx, title, seed)).trim().toLowerCase();
|
|
208
|
+
if (raw === "" || raw === "none" || raw === "off" || raw === "null") return null;
|
|
209
|
+
const value = Number(raw);
|
|
210
|
+
if (!Number.isFinite(value) || value < 0) {
|
|
211
|
+
ctx.ui.notify(
|
|
212
|
+
`"${raw}" is not a non-negative number or blank — keeping ${fallback === null ? "no cap" : fallback}.`,
|
|
213
|
+
"warning",
|
|
214
|
+
);
|
|
215
|
+
return fallback;
|
|
216
|
+
}
|
|
217
|
+
return value;
|
|
218
|
+
}
|
|
219
|
+
|
|
196
220
|
/**
|
|
197
221
|
* Pre-push gates as one comma-separated line, `cmd @ cwd` for a subdirectory:
|
|
198
222
|
* `bun run check, bun test @ server`. Shown through `formatGates`, the same
|
|
@@ -500,10 +524,12 @@ const askGraph: AreaAsker = async (ctx, a) => {
|
|
|
500
524
|
* answer for anyone who has not measured their own runners. */
|
|
501
525
|
const askCaps: AreaAsker = async (ctx, a) => {
|
|
502
526
|
const caps: Partial<Caps> = { ...a.caps };
|
|
527
|
+
const spendLabel =
|
|
528
|
+
DEFAULT_CAPS.dailySpendUsd === null ? "no spend cap" : `$${DEFAULT_CAPS.dailySpendUsd}/day`;
|
|
503
529
|
const tuneCaps = await ctx.ui.confirm(
|
|
504
530
|
"Caps",
|
|
505
531
|
`Defaults: ${DEFAULT_CAPS.maxConcurrentWorkers} workers, ` +
|
|
506
|
-
|
|
532
|
+
`${spendLabel}, ${DEFAULT_CAPS.workerMaxTurns} turns and ` +
|
|
507
533
|
`${Math.round(DEFAULT_CAPS.workerWallClockMs / 60000)} min per worker, ` +
|
|
508
534
|
`${DEFAULT_CAPS.maxAttemptsPerIssue} attempts per issue. Change them?`,
|
|
509
535
|
);
|
|
@@ -516,7 +542,11 @@ const askCaps: AreaAsker = async (ctx, a) => {
|
|
|
516
542
|
"Max concurrent workers",
|
|
517
543
|
caps.maxConcurrentWorkers ?? DEFAULT_CAPS.maxConcurrentWorkers,
|
|
518
544
|
);
|
|
519
|
-
caps.dailySpendUsd = await
|
|
545
|
+
caps.dailySpendUsd = await askSpendCap(
|
|
546
|
+
ctx,
|
|
547
|
+
"Spend ceiling per rolling day (USD) — blank = no spend cap",
|
|
548
|
+
caps.dailySpendUsd !== undefined ? caps.dailySpendUsd : DEFAULT_CAPS.dailySpendUsd,
|
|
549
|
+
);
|
|
520
550
|
caps.workerMaxTurns = await askNumber(ctx, "Turn ceiling per worker", caps.workerMaxTurns ?? DEFAULT_CAPS.workerMaxTurns);
|
|
521
551
|
caps.workerWallClockMs = await askNumber(
|
|
522
552
|
ctx,
|
package/src/setup.ts
CHANGED
|
@@ -954,9 +954,11 @@ export const AMEND_AREAS: {
|
|
|
954
954
|
describe: (p) => {
|
|
955
955
|
const c = resolveCaps(p, DEFAULT_CAPS);
|
|
956
956
|
const answered = Object.keys(p.caps).length > 0;
|
|
957
|
+
const spend =
|
|
958
|
+
c.dailySpendUsd === null ? "no spend cap" : `$${c.dailySpendUsd}/day`;
|
|
957
959
|
return (
|
|
958
960
|
`${c.maxConcurrentWorkers} workers, ${c.workerMaxTurns} turns, ` +
|
|
959
|
-
`${Math.round(c.workerWallClockMs / 60000)}m,
|
|
961
|
+
`${Math.round(c.workerWallClockMs / 60000)}m, ${spend}, ` +
|
|
960
962
|
`${c.maxAttemptsPerIssue} attempts${answered ? "" : " (all defaults)"} — ` +
|
|
961
963
|
`${p.workerModel === undefined ? "harness default model" : `model ${p.workerModel}`}`
|
|
962
964
|
);
|
package/src/types.ts
CHANGED
|
@@ -16,8 +16,11 @@ export interface Caps {
|
|
|
16
16
|
/** Parallel omp sessions. Two by default: on a small self-hosted runner pool
|
|
17
17
|
* a third worker would starve its own PR checks. */
|
|
18
18
|
maxConcurrentWorkers: number;
|
|
19
|
-
/**
|
|
20
|
-
|
|
19
|
+
/**
|
|
20
|
+
* Rolling-day spend ceiling. `null` means no spend gate (turns + wall-clock
|
|
21
|
+
* still apply). `0` is a hard stop — deliberate, not "unset".
|
|
22
|
+
*/
|
|
23
|
+
dailySpendUsd: number | null;
|
|
21
24
|
/** Turn ceiling for one worker — catches loops that are burning tokens
|
|
22
25
|
* without converging. */
|
|
23
26
|
workerMaxTurns: number;
|
package/src/worker.ts
CHANGED
|
@@ -216,15 +216,22 @@ export async function runWorker(
|
|
|
216
216
|
// is its report, whether it finished cleanly or was cut off.
|
|
217
217
|
const text = reportText(field(message, "content"));
|
|
218
218
|
if (text !== "") report = text;
|
|
219
|
+
|
|
220
|
+
// Real cost lives on assistant messages as `usage.cost.total` (live hermes
|
|
221
|
+
// transcripts, 2026-08-07). The earlier agent_end.telemetry path never
|
|
222
|
+
// fired, so every run recorded $0 and the daily cap was theater (#46).
|
|
223
|
+
const cost = costUsdFromMessage(message);
|
|
224
|
+
if (cost !== undefined) spendUsd += cost;
|
|
219
225
|
});
|
|
220
226
|
|
|
221
227
|
session.on("agent_end", (event) => {
|
|
222
|
-
//
|
|
223
|
-
// spend can legitimately read 0 and the daily-spend cap then leans on the
|
|
224
|
-
// turn and wall-clock ceilings. Upgrade path: pass a telemetry config
|
|
225
|
-
// through `createSession` once the harness exposes it on the SDK options.
|
|
228
|
+
// Fallback for harnesses that only attach cost on the terminal event.
|
|
226
229
|
const estimated = field(field(field(event, "telemetry"), "cost"), "estimatedUsd");
|
|
227
|
-
if (typeof estimated === "number" && Number.isFinite(estimated)
|
|
230
|
+
if (typeof estimated === "number" && Number.isFinite(estimated) && estimated > 0) {
|
|
231
|
+
// Prefer message totals when both exist — do not double-count a run that
|
|
232
|
+
// already accumulated per-message costs.
|
|
233
|
+
if (spendUsd === 0) spendUsd += estimated;
|
|
234
|
+
}
|
|
228
235
|
|
|
229
236
|
// Anything that is not literally `false` — including garbage or nothing at
|
|
230
237
|
// all — is a finished run.
|
|
@@ -282,6 +289,31 @@ export async function runWorker(
|
|
|
282
289
|
);
|
|
283
290
|
}
|
|
284
291
|
|
|
292
|
+
/**
|
|
293
|
+
* USD cost from one assistant message's `usage.cost` block.
|
|
294
|
+
*
|
|
295
|
+
* Prefer `total` when present; otherwise sum the component fields the live
|
|
296
|
+
* harness emits (input/output/cacheRead/cacheWrite). Exported so a unit test
|
|
297
|
+
* can pin the shape without standing up a session.
|
|
298
|
+
*/
|
|
299
|
+
export function costUsdFromMessage(message: unknown): number | undefined {
|
|
300
|
+
const usage = field(message, "usage");
|
|
301
|
+
const cost = field(usage, "cost");
|
|
302
|
+
if (cost === null || typeof cost !== "object") return undefined;
|
|
303
|
+
const total = field(cost, "total");
|
|
304
|
+
if (typeof total === "number" && Number.isFinite(total) && total >= 0) return total;
|
|
305
|
+
let sum = 0;
|
|
306
|
+
let any = false;
|
|
307
|
+
for (const key of ["input", "output", "cacheRead", "cacheWrite"] as const) {
|
|
308
|
+
const v = field(cost, key);
|
|
309
|
+
if (typeof v === "number" && Number.isFinite(v) && v >= 0) {
|
|
310
|
+
sum += v;
|
|
311
|
+
any = true;
|
|
312
|
+
}
|
|
313
|
+
}
|
|
314
|
+
return any ? sum : undefined;
|
|
315
|
+
}
|
|
316
|
+
|
|
285
317
|
/**
|
|
286
318
|
* Read one property off an unvalidated harness event. The event union lives in
|
|
287
319
|
* the peer dependency, so the worker narrows the handful of fields it reads
|
package/src/worktree.ts
CHANGED
|
@@ -102,15 +102,27 @@ const EXCLUDE_END = "# <<< omp-conductor";
|
|
|
102
102
|
|
|
103
103
|
/**
|
|
104
104
|
* Appended to every mirror's `info/exclude`, and so in force in every worktree
|
|
105
|
-
* cut from it
|
|
106
|
-
*
|
|
105
|
+
* cut from it: a worker's own scaffolding is not the repo's business, and
|
|
106
|
+
* keeping it out of `git status` keeps it out of the worker's own `git add -A`
|
|
107
|
+
* as well as out of salvage.
|
|
107
108
|
*
|
|
108
|
-
*
|
|
109
|
-
*
|
|
110
|
-
*
|
|
111
|
-
*
|
|
109
|
+
* **Only the directory form of `.scratch*`; the slash is the design.** An ignore
|
|
110
|
+
* here applies to every repo this fleet touches, and an ignored *new* file is
|
|
111
|
+
* invisible to salvage — `git add -A` skips it, and a tree holding only such
|
|
112
|
+
* files reports `nothing`. So a name that could plausibly be a deliverable must
|
|
113
|
+
* never appear in this list. The first version also carried `.env.local` and
|
|
114
|
+
* `*.local.sh`; its first correction still carried an unqualified `.scratch*`.
|
|
115
|
+
* All three make the same mistake: repos can legitimately ship an `.env.local`
|
|
116
|
+
* template, `bootstrap.local.sh`, `.scratchrc`, or `.scratchpad`, and a worker
|
|
117
|
+
* asked to add one would watch it vanish.
|
|
118
|
+
*
|
|
119
|
+
* The directory-only pattern survives because its trailing slash limits it to a
|
|
120
|
+
* directory whose name announces that its contents are disposable. That was
|
|
121
|
+
* the 2026-08-07 incident's exact shape (`.scratch82/env.sh`). Broader
|
|
122
|
+
* conventions belong in a repo's own `.gitignore`, where its operator chooses
|
|
123
|
+
* them, rather than being imposed by whatever dispatcher happens to be driving.
|
|
112
124
|
*/
|
|
113
|
-
const LOCAL_EXCLUDE = [".scratch*/"
|
|
125
|
+
const LOCAL_EXCLUDE = [".scratch*/"];
|
|
114
126
|
|
|
115
127
|
/**
|
|
116
128
|
* Adds the managed block to an `info/exclude`, preserving everything else.
|
|
@@ -182,6 +194,40 @@ async function configureMirror(mirrorPath: string): Promise<void> {
|
|
|
182
194
|
writeFileSync(exclude, mergeExclude(existsSync(exclude) ? readFileSync(exclude, "utf8") : ""));
|
|
183
195
|
}
|
|
184
196
|
|
|
197
|
+
/**
|
|
198
|
+
* Rewrites this worktree's common `info/exclude` managed block to the package's
|
|
199
|
+
* current list. Salvage must do this itself: it is the path that runs after a
|
|
200
|
+
* package swap, when the mirror may still carry a previous release's patterns,
|
|
201
|
+
* and `git add -A` would otherwise silently skip legitimate new files the old
|
|
202
|
+
* list happened to name (dogfood 2026-08-07 / #44).
|
|
203
|
+
*
|
|
204
|
+
* Failures are swallowed — a missing common dir is "no exclude to heal", and
|
|
205
|
+
* the salvage dirty check still runs. Never throws into the salvage outcome.
|
|
206
|
+
*/
|
|
207
|
+
function refreshManagedExclude(worktree: string): void {
|
|
208
|
+
try {
|
|
209
|
+
// `--git-common-dir` is relative for linked worktrees; resolve against the
|
|
210
|
+
// tree so bare-mirror layouts and plain clones both land on info/exclude.
|
|
211
|
+
const common = Bun.spawnSync(["git", "rev-parse", "--git-common-dir"], {
|
|
212
|
+
cwd: worktree,
|
|
213
|
+
stdin: "ignore",
|
|
214
|
+
stdout: "pipe",
|
|
215
|
+
stderr: "pipe",
|
|
216
|
+
env: { ...process.env, GIT_TERMINAL_PROMPT: "0" },
|
|
217
|
+
});
|
|
218
|
+
if (common.exitCode !== 0) return;
|
|
219
|
+
const raw = common.stdout.toString().trim();
|
|
220
|
+
if (raw === "") return;
|
|
221
|
+
const commonDir = raw.startsWith("/") ? raw : join(worktree, raw);
|
|
222
|
+
const exclude = join(commonDir, "info", "exclude");
|
|
223
|
+
mkdirSync(dirname(exclude), { recursive: true });
|
|
224
|
+
writeFileSync(exclude, mergeExclude(existsSync(exclude) ? readFileSync(exclude, "utf8") : ""));
|
|
225
|
+
} catch {
|
|
226
|
+
// ponytail: exclude heal is best-effort; salvage still prefers a commit of
|
|
227
|
+
// whatever git can see over failing the whole orphan path.
|
|
228
|
+
}
|
|
229
|
+
}
|
|
230
|
+
|
|
185
231
|
/**
|
|
186
232
|
* Returns the path of the bare mirror for `repo`, cloning it on first use and
|
|
187
233
|
* refreshing it otherwise.
|
|
@@ -239,7 +285,7 @@ export async function addWorktree(
|
|
|
239
285
|
workspaceRoot: string,
|
|
240
286
|
issue: number,
|
|
241
287
|
branch: string,
|
|
242
|
-
): Promise<string> {
|
|
288
|
+
): Promise<{ path: string; reattached: boolean }> {
|
|
243
289
|
const mirrorPath = await ensureMirror(repo, mirrorRoot);
|
|
244
290
|
mkdirSync(workspaceRoot, { recursive: true });
|
|
245
291
|
|
|
@@ -323,7 +369,7 @@ export async function addWorktree(
|
|
|
323
369
|
}
|
|
324
370
|
}
|
|
325
371
|
|
|
326
|
-
return worktreePath;
|
|
372
|
+
return { path: worktreePath, reattached: branchExists };
|
|
327
373
|
}
|
|
328
374
|
|
|
329
375
|
/**
|
|
@@ -394,6 +440,12 @@ export async function salvageWip(
|
|
|
394
440
|
// code, and "no tree" is not a salvage failure worth alarming anyone with.
|
|
395
441
|
if (!existsSync(worktree)) return { kind: "nothing" };
|
|
396
442
|
|
|
443
|
+
// Heal the managed ignore *before* status/add. A mirror left by an older
|
|
444
|
+
// build may still list patterns this release dropped; without this, an
|
|
445
|
+
// untracked deliverable matching the stale list is invisible to salvage
|
|
446
|
+
// and dies with the next `worktree remove --force` (#44).
|
|
447
|
+
refreshManagedExclude(worktree);
|
|
448
|
+
|
|
397
449
|
if ((await git(["status", "--porcelain"], worktree)) === "") return { kind: "nothing" };
|
|
398
450
|
|
|
399
451
|
// The tree's own branch, not one the caller believes it should be on: this
|