omp-conductor 0.18.2 → 0.19.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +105 -40
- package/REFERENCE.md +865 -30
- package/package.json +1 -1
- package/schema/config.schema.json +26 -0
- package/src/admission.ts +212 -26
- package/src/ask.ts +288 -1
- package/src/briefs/orchestrator.md +6 -5
- package/src/cli.ts +5 -1
- package/src/command-help.ts +9 -1
- package/src/command-manifest.ts +36 -3
- package/src/commands/arm.ts +5 -1
- package/src/commands/context.ts +2 -0
- package/src/commands/message.ts +26 -2
- package/src/commands/reconcile-units.ts +104 -0
- package/src/commands/release-composition.ts +232 -0
- package/src/commands/resume.ts +2 -27
- package/src/commands/setup.ts +101 -16
- package/src/commands/stats.ts +11 -30
- package/src/commands/tail.ts +31 -1
- package/src/commands/upgrade.ts +20 -3
- package/src/commands/verb.ts +2 -1
- package/src/config-schema.ts +19 -0
- package/src/config.ts +80 -0
- package/src/credential-class.ts +366 -0
- package/src/daemon.ts +1218 -288
- package/src/dashboard/app.js +504 -2
- package/src/dashboard/controls.ts +336 -0
- package/src/dashboard/index.html +30 -0
- package/src/dashboard/server.ts +271 -30
- package/src/dashboard/style.css +116 -0
- package/src/dashboard/transcript.ts +173 -0
- package/src/doctor.ts +379 -22
- package/src/failure-class.ts +59 -0
- package/src/fleet.ts +511 -101
- package/src/host.ts +6 -130
- package/src/omp.ts +29 -0
- package/src/orchestrator-tick.ts +343 -88
- package/src/pause.ts +233 -0
- package/src/settlement.ts +159 -2
- package/src/setup-answers.ts +97 -0
- package/src/setup-host.ts +325 -1159
- package/src/setup-install.ts +204 -27
- package/src/setup-wizard.ts +111 -50
- package/src/setup.ts +33 -0
- package/src/spend-telemetry.ts +117 -0
- package/src/stats.ts +35 -0
- package/src/status-render.ts +348 -19
- package/src/store.ts +1229 -55
- package/src/telegram-freshness.ts +269 -0
- package/src/to-spec.ts +27 -0
- package/src/types.ts +697 -4
- package/src/unblock.ts +22 -0
- package/src/unit-reconcile.ts +303 -0
- package/src/upgrade-verify.ts +8 -1
- package/src/upgrade.ts +326 -47
- package/src/verbs/actions.ts +124 -10
- package/src/verbs/protocol.ts +70 -2
- package/src/verbs/server.ts +447 -8
- package/src/wake.ts +48 -0
- package/src/worker.ts +403 -3
package/REFERENCE.md
CHANGED
|
@@ -180,6 +180,19 @@ than as drift (#511). It still flags genuine drift in `User=`, `ExecStart=`,
|
|
|
180
180
|
other `Environment=` lines, `Restart=` and `SuccessExitStatus=`, and the unit
|
|
181
181
|
itself keeps pinning `SHELL=` so panes do not fall back to dash (#463).
|
|
182
182
|
|
|
183
|
+
One finding is a conjunction rather than a fault. `bootstrap-deadlock` (#910)
|
|
184
|
+
fires when no worker on this host can start — because the *installed* conductor
|
|
185
|
+
cannot load its harness, exercised the way a worker does by running
|
|
186
|
+
`harness-loader.ts` under `bun --no-install` — and says the other half out loud:
|
|
187
|
+
a release cannot publish the fix either, because publishing needs a merged
|
|
188
|
+
worker pull request, so the broken install blocks its own replacement. It is
|
|
189
|
+
deliberately its own finding rather than a sentence appended to the stale-setup
|
|
190
|
+
ones, because the two states call for opposite actions: "run `setup host`" is
|
|
191
|
+
correct advice for a stale install and useless here, since no amount of
|
|
192
|
+
re-running setup replaces the installed package. Its fix names the one command
|
|
193
|
+
that breaks the cycle — `upgrade --bootstrap <sha> --source <checkout>` — and an
|
|
194
|
+
ordinary stale host still reads exactly as it did.
|
|
195
|
+
|
|
183
196
|
## Onboarding
|
|
184
197
|
|
|
185
198
|
`omp-conductor setup` is the whole of it. One command, in a plain terminal, doing
|
|
@@ -271,6 +284,27 @@ names the first missing key instead of prompting. Add `--save-answers FILE`
|
|
|
271
284
|
to an interactive run to capture only accepted answers for deterministic
|
|
272
285
|
replay. Unknown keys are ignored so one file can cover a larger interview.
|
|
273
286
|
|
|
287
|
+
**A failed apply keeps its answers (#864).** The interview is the expensive half
|
|
288
|
+
and it usually is not what failed: on 2026-08-21 `setup policy` failed at its
|
|
289
|
+
post-interview gates several times, and every retry re-asked the whole merge,
|
|
290
|
+
release, arming and review interview — the same `runs-settled` value was typed
|
|
291
|
+
five times while the real fault was a runtime gate. So when the apply phase
|
|
292
|
+
throws, the confirmed answers are written to
|
|
293
|
+
`<state dir>/setup-resume-<project>.json` (mode `0600`, beside the config they
|
|
294
|
+
describe) and the failure names the exact command that replays them:
|
|
295
|
+
|
|
296
|
+
```bash
|
|
297
|
+
omp-conductor setup policy --project conductor --resume ~/.omp/conductor/setup-resume-conductor.json
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
`--resume` is `--answers` with a binding, and the binding is the point: the file
|
|
301
|
+
records a hash of the config it was answered against and the version that wrote
|
|
302
|
+
it. A changed config refuses, naming both generations — replaying an old plan
|
|
303
|
+
over newer policy would silently overwrite whatever changed it — and so does a
|
|
304
|
+
version mismatch, because the answer keys are a schema an upgrade may have
|
|
305
|
+
renamed. A completed setup deletes its own resume file, so a stale one cannot
|
|
306
|
+
invite a replay of answers that already landed.
|
|
307
|
+
|
|
274
308
|
To fill in or revise just the brief later — the two `POLICY.md` sections above — run
|
|
275
309
|
the `brief` area, which re-asks the judgment questions and re-runs the probes:
|
|
276
310
|
|
|
@@ -446,6 +480,23 @@ Per tick, for the daemon's project:
|
|
|
446
480
|
4. **Filter and route.** An issue is eligible only if it carries the queue label
|
|
447
481
|
and none of the three state labels (`inProgress`, `blocked`, `failed`). Eligible
|
|
448
482
|
issues are partitioned into routable and unroutable.
|
|
483
|
+
|
|
484
|
+
A lifecycle label read from that **list** is a suspicion, not a verdict
|
|
485
|
+
(#891). `listReady` is a label-filtered search, and GitHub's search index is
|
|
486
|
+
eventually consistent, so an `unblock` that has already landed can still come
|
|
487
|
+
back carrying the label it cleared. Before a candidate is dropped for a
|
|
488
|
+
lifecycle label, the daemon re-reads that one issue's live labels
|
|
489
|
+
(`issueSnapshot`, a direct REST read) and admits normally when the exact read
|
|
490
|
+
is clean. Measured 2026-08-22T09:18Z: `unblock 889` cleared `agent:failed`,
|
|
491
|
+
`gh issue view` returned the clean set immediately, and the pass woken right
|
|
492
|
+
after still held it `stale-lifecycle` with the fleet at 0/5.
|
|
493
|
+
|
|
494
|
+
The re-read is bounded to candidates a lifecycle label would otherwise drop,
|
|
495
|
+
and skips two of those: an issue whose newest run is active (its own row is
|
|
496
|
+
the authority) and an operator-parked one (a park is a decision, not a stale
|
|
497
|
+
label). An unreadable re-read is **fail-closed** — the issue stays out of the
|
|
498
|
+
pass and is held as `issue-state-lookup-error`, never as `stale-lifecycle`,
|
|
499
|
+
because "the tracker could not say" is not evidence of a residual label.
|
|
449
500
|
5. **Escalate the unroutable** at Tier 1, quoting the repo labels actually seen and
|
|
450
501
|
the configured repo names. These are never dispatched.
|
|
451
502
|
6. **Check spend.** If spend since local midnight has reached `dailySpendUsd`, the
|
|
@@ -797,6 +848,75 @@ the queue label is refused with `file-lane-unparseable` and an actionable
|
|
|
797
848
|
message, because promoting beside overlapping work on a section that plainly
|
|
798
849
|
tried to declare is the exact gap the interlock exists to close.
|
|
799
850
|
|
|
851
|
+
#### What status shows
|
|
852
|
+
|
|
853
|
+
`status` renders the two lifecycle kinds as two blocks, because "a branch is
|
|
854
|
+
holding a file" and "a worker is editing that file" need different answers:
|
|
855
|
+
|
|
856
|
+
- **`mutation leases`** — a live worker, or a dispatched unsettled review
|
|
857
|
+
revision. Each carries the run holding the lease, how long it has held it, the
|
|
858
|
+
read the interlock proves occupancy with (`worktree` for a live checkout,
|
|
859
|
+
`branch` for a worker-free row's mirror branch — the same condition
|
|
860
|
+
`probeRunLane` uses), and the files its declaration occupies.
|
|
861
|
+
|
|
862
|
+
Two further lines hang off a lease when they have something to say. **`pane`**
|
|
863
|
+
names the Herdr pane representing that worker, or `degraded` with the reason
|
|
864
|
+
there is none (#841) — see [the worker-pane
|
|
865
|
+
contract](#a-worker-gets-a-herdr-pane-but-the-pane-is-never-the-worker-840).
|
|
866
|
+
**`tokens`** appears only when a run's reasoning share of its output tokens is
|
|
867
|
+
disproportionate (≥80%, #518): the fact that distinguishes a run deliberating
|
|
868
|
+
expensively in wall clock but cheaply in dollars from one that is idle or
|
|
869
|
+
stuck. Run 42f73f52 read `24/180 turns $0.05` after 43 minutes and looked
|
|
870
|
+
barely started; 96% of its 106.3k output tokens were reasoning, its turns were
|
|
871
|
+
arriving at 200 seconds each, and the 180-turn cap was therefore unreachable —
|
|
872
|
+
the wall clock was the only ceiling that could ever fire. Both numbers come off
|
|
873
|
+
the same `usage` block the spend already comes from, so neither costs a
|
|
874
|
+
provider call, and both are recorded live rather than at settlement: the moment
|
|
875
|
+
this explains is forty minutes before `wall-clock-cap-spinning` (#494) fires.
|
|
876
|
+
An ordinary reasoning share prints nothing, and a run with no token counts
|
|
877
|
+
makes no claim about them — absent is not 0%.
|
|
878
|
+
- **`preserved artifacts`** — worker-free `pushed-green` / `pushed-pending` rows.
|
|
879
|
+
Durable work with nothing editing it, so it is shown with how long it has been
|
|
880
|
+
waiting and on what: a PR to review and merge, or (with no PR) a branch
|
|
881
|
+
`conductor_pr_recover` can still publish. No turn rate and no cap projection —
|
|
882
|
+
those describe a session in progress, and printing them beside work nobody is
|
|
883
|
+
running is how a settled artifact came to read as a busy one.
|
|
884
|
+
|
|
885
|
+
Membership is the union of the run's own live state and the snapshot's recorded
|
|
886
|
+
`leasedRunIds`, never the ids alone: a live worker is a lease by definition, and
|
|
887
|
+
the recorded set exists to add the case a state read cannot see — a dispatched
|
|
888
|
+
revision on a row that still says `pushed-green`. Both blocks are rendered from
|
|
889
|
+
the snapshot alone, so a status read never probes git and never mutates anything.
|
|
890
|
+
|
|
891
|
+
#### What holds a lane, and what merely owns an artifact
|
|
892
|
+
|
|
893
|
+
Occupancy follows the **mutation lease** (#899): the runs something is writing
|
|
894
|
+
through right now — a live worker (`claimed`/`running`), or a review revision the
|
|
895
|
+
daemon has dispatched and that has not settled. A worker-free `pushed-green` /
|
|
896
|
+
`pushed-pending` row still owns its issue and its PR, so its work is never
|
|
897
|
+
re-implemented, but it no longer holds its files: holding them meant one settled
|
|
898
|
+
artifact idled every overlapping candidate until a human merged it.
|
|
899
|
+
|
|
900
|
+
The trade is deliberate. A merge conflict is a later, recoverable event
|
|
901
|
+
(`conductor_pr_update_branch`, or the `merge-conflict` failure class); an idle
|
|
902
|
+
fleet is an immediate, unrecoverable one.
|
|
903
|
+
|
|
904
|
+
Because a released branch's files may legitimately be taken by someone else,
|
|
905
|
+
every path that *resumes* writing one re-proves the lane first (#925):
|
|
906
|
+
|
|
907
|
+
- a **continuation** is held (`file-lane`) when the branch it resumes actually
|
|
908
|
+
carries a file a lease holds — read from the branch, not from the candidate's
|
|
909
|
+
declaration, which may be narrower or absent;
|
|
910
|
+
- **`conductor_pr_recover`** refuses (`recovery-lane-occupied`) rather than
|
|
911
|
+
publishing a frozen branch into review beside a worker still rewriting one of
|
|
912
|
+
its files. Adopting an already-open PR is not gated: it publishes nothing.
|
|
913
|
+
|
|
914
|
+
The two sides fail in opposite directions, on purpose. Admission is unattended
|
|
915
|
+
and continuous, so an unreadable probe admits (fail open) rather than stalling
|
|
916
|
+
the queue on a transient git read. Recovery is one deliberate call whose refusal
|
|
917
|
+
is legible and retryable, so an unreadable lane refuses
|
|
918
|
+
(`recovery-lane-unprovable`) rather than publishing on an unproven one.
|
|
919
|
+
|
|
800
920
|
## Routing
|
|
801
921
|
|
|
802
922
|
An issue must carry **exactly one** `repo:<name>` label naming a repo in
|
|
@@ -875,7 +995,8 @@ rest. `0` is a real value (a hard stop), not "unset".
|
|
|
875
995
|
| --- | --- | --- |
|
|
876
996
|
| `maxConcurrentWorkers` | `2` (setup may write `1` on <16 GiB hosts) | Parallel omp sessions, each a child process of the daemon and all inside its cgroup. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. On hosts under ~16 GiB RAM, prefer `1` so the unit stays out of swap ([host sizing](#host-sizing-and-memory)). Raise it only if you actually have the runners *and* the RAM. |
|
|
877
997
|
| `maxConcurrentWorkersPerRepo` | `1` | Max live workers in the **same repo**. The mirror, branch-protection staleness and shared CI egress are all per-repo collision domains, so extra slots should land on other repos. Raise it only when a repo genuinely needs two workers at once. |
|
|
878
|
-
| `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total
|
|
998
|
+
| `dailySpendUsd` | `25` | Rolling-day spend ceiling in **estimated** USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total` — OMP's own local price for the tokens a session recorded, not the provider's bill. |
|
|
999
|
+
| `maxRunSpendUsd` | `null` (derived: `dailySpendUsd / maxConcurrentWorkers`) | Ceiling on what **one run** may spend, and the amount admission reserves from the day's budget before launching it. Positive, or `null` to derive; never larger than `dailySpendUsd`. See [Per-run spend reservations](#per-run-spend-reservations-851). |
|
|
879
1000
|
| `planUsage` | `null` (unmetered) | Subscription/plan allowance guard: `{ "windowId": "anthropic:7d", "maxUsedFraction": 0.85 }`, or `null` for no plan gate. Independent of `dailySpendUsd` — see [Plan allowance](#plan-allowance-planusage) below. |
|
|
880
1001
|
| `workerMaxTurns` | `120` | Base ceiling for each new worker. Catches a session looping without converging; use `omp-conductor extend` to raise one live run or one issue's next attempt without changing this default. |
|
|
881
1002
|
| `workerMaxTurnsCeiling` | `240` (twice the effective `workerMaxTurns` when omitted) | Upper bound for per-issue turn extensions. Prevents the loopback control from granting an unbounded worker budget. |
|
|
@@ -891,6 +1012,149 @@ pages at Tier 2**: a loop that is burning money has to halt itself, because
|
|
|
891
1012
|
waiting for someone to notice tomorrow is how a runaway becomes expensive.
|
|
892
1013
|
Work resumes only after `omp-conductor resume`.
|
|
893
1014
|
|
|
1015
|
+
### Install-surface parity, recorded (#919)
|
|
1016
|
+
|
|
1017
|
+
`omp-conductor` installs onto three surfaces — the Bun-global CLI/daemon tree,
|
|
1018
|
+
the omp plugin, and the herdr recovery plugin — and on a host whose installs are
|
|
1019
|
+
manual they diverge silently. Measured on this host, 2026-08-22: the omp plugin
|
|
1020
|
+
sat on the withdrawn 0.18.1 release beside a 0.18.0 daemon for about a day, with
|
|
1021
|
+
no finding, no status line and no escalation.
|
|
1022
|
+
|
|
1023
|
+
The dispatch pass records the three identities once per pass, and the cheap
|
|
1024
|
+
surfaces read that row:
|
|
1025
|
+
|
|
1026
|
+
- `status` is silent while they agree, prints the mismatch with
|
|
1027
|
+
`omp-conductor upgrade --to <version>` when they disagree, and says
|
|
1028
|
+
**`not observed yet`** before the first pass has looked — never "they agree";
|
|
1029
|
+
- the tick digest names a **proven** mismatch only.
|
|
1030
|
+
|
|
1031
|
+
Neither spawns a process for it, and that is the design rather than an
|
|
1032
|
+
optimisation: probing costs three subprocesses, and doing it at render time took
|
|
1033
|
+
the tick's own suite from 8.4s to 83.4s while spawning three children every
|
|
1034
|
+
fifteen minutes to answer a question that changes only when someone installs
|
|
1035
|
+
something.
|
|
1036
|
+
|
|
1037
|
+
An absent surface, a `local:` herdr link and a pin whose release could not be
|
|
1038
|
+
verified are steady states someone chose, so they produce no status row and no
|
|
1039
|
+
tick line — a warning repeated every fifteen minutes trains an operator to
|
|
1040
|
+
ignore the row that matters. `omp-conductor doctor` keeps the full nuance,
|
|
1041
|
+
including those cases, and is still the on-demand answer.
|
|
1042
|
+
|
|
1043
|
+
### The one peer plugin that is also a contract (#961)
|
|
1044
|
+
|
|
1045
|
+
The same pass records three more versions, about `omp-telegram`: what is
|
|
1046
|
+
installed, what its running daemon reports, and what is published. That plugin
|
|
1047
|
+
gets this treatment and no other because a *conductor* instruction depends on its
|
|
1048
|
+
behaviour — the floor tells every session answering an inbound message to call
|
|
1049
|
+
`telegram_send` with neither `chat_id` nor `thread_id`, so the reply stays in the
|
|
1050
|
+
topic it arrived in.
|
|
1051
|
+
|
|
1052
|
+
Measured on this host: the installed 0.12.1 predated that ladder. Its send path
|
|
1053
|
+
had two rungs (explicit `chat_id`, then in-process `lastTarget`), so the mandated
|
|
1054
|
+
call refused with `no active telegram chat` three times on 2026-08-21, and each
|
|
1055
|
+
refusal cost an improvised recovery — once by passing route ids by hand, which is
|
|
1056
|
+
precisely the anti-pattern #882 names as its own silent fake. The published
|
|
1057
|
+
0.12.2 has the four-rung ladder plus per-rung diagnostics. **The fix had been on
|
|
1058
|
+
npm for days, and no surface said so**, so three sessions did transcript
|
|
1059
|
+
archaeology to reach a conclusion `npm view` answers in one call.
|
|
1060
|
+
|
|
1061
|
+
- `status` grows a `tg-plugin` row **only** on a proven divergence: installed
|
|
1062
|
+
behind published, or a daemon serving something other than what is installed.
|
|
1063
|
+
Silent otherwise, and silent when nothing was recorded — a fleet that does not
|
|
1064
|
+
use Telegram must not grow a permanent row about a plugin it does not have.
|
|
1065
|
+
- `doctor` reports the finding at full nuance, always `warn` and never `fail`:
|
|
1066
|
+
the fleet dispatches, merges and reports fine on a stale plugin. One
|
|
1067
|
+
instruction does not, and the remedy is an install the operator owns.
|
|
1068
|
+
|
|
1069
|
+
Behind-published outranks daemon-stale deliberately, because the remedies
|
|
1070
|
+
compose: an install without a restart leaves the daemon on the old code. And an
|
|
1071
|
+
unreachable registry is **`unknown`**, never `current` — a check that could not
|
|
1072
|
+
run must not certify, and a host with no outbound npm access must not grow a
|
|
1073
|
+
permanent warning about it. No minimum version is compiled in: a baked-in
|
|
1074
|
+
baseline goes stale in exactly the way this check exists to catch, and would
|
|
1075
|
+
re-create #904's divergence with one more copy of the number.
|
|
1076
|
+
|
|
1077
|
+
### Per-run spend reservations (#851)
|
|
1078
|
+
|
|
1079
|
+
A daily cap alone is a *post-spend* stop: it compares spend-to-date at
|
|
1080
|
+
admission and pauses the fleet on the **next** pass, so one expensive run can
|
|
1081
|
+
cross the ceiling by its whole cost before anything objects. Measured on this
|
|
1082
|
+
package's own fleet, 2026-08-21: one `@slow` run spent an estimated $25.16
|
|
1083
|
+
against a $25.00 daily cap, and the fleet paused only after it had settled.
|
|
1084
|
+
|
|
1085
|
+
So the cap is enforced in two places, and both are mechanical:
|
|
1086
|
+
|
|
1087
|
+
- **At admission, as a reservation.** Before a run launches, its whole per-run
|
|
1088
|
+
allowance is reserved out of the day's budget and recorded on the run row. A
|
|
1089
|
+
candidate is held (`spend-reservation`) when spend-to-date plus every live
|
|
1090
|
+
run's reservation plus its own would exceed `dailySpendUsd`. The reservation
|
|
1091
|
+
lives on the row, so a daemon restart re-reads it instead of forgetting it,
|
|
1092
|
+
and a run that settles for less than it reserved releases the remainder
|
|
1093
|
+
simply by ceasing to be live — there is no release step to forget.
|
|
1094
|
+
- **Inside the run, as a ceiling.** A session that reaches its allowance is
|
|
1095
|
+
killed like any other cap kill, through the ordinary salvage path, so its
|
|
1096
|
+
work is preserved and its PR (if any) survives. Response cost is known only
|
|
1097
|
+
when a response completes, so the **declared overshoot is exactly one model
|
|
1098
|
+
response** — never zero, and never a whole run.
|
|
1099
|
+
|
|
1100
|
+
The allowance is `maxRunSpendUsd` when set, and otherwise
|
|
1101
|
+
`dailySpendUsd / maxConcurrentWorkers`: the largest per-run reservation that
|
|
1102
|
+
still lets the fleet run at its configured concurrency. Reserving the whole
|
|
1103
|
+
daily cap per run would bound overshoot just as well and would silently
|
|
1104
|
+
serialise every fleet on defaults, which is a concurrency change disguised as a
|
|
1105
|
+
spend guard; N reservations of cap/N are still the cap, so nothing is given
|
|
1106
|
+
away. Set `maxRunSpendUsd` explicitly to bound one run harder.
|
|
1107
|
+
|
|
1108
|
+
`status` labels the figure **estimated** for the same reason the table above
|
|
1109
|
+
does: on the 2026-08-21 incident the local estimate read $25.16 while roughly
|
|
1110
|
+
$16 of provider credit actually moved, so presenting it as billed spend would
|
|
1111
|
+
assert something conductor cannot see. Reservations are shown beside it, because
|
|
1112
|
+
that — not spend-to-date — is what the next admission subtracts.
|
|
1113
|
+
|
|
1114
|
+
#### And whether that figure means anything (#970)
|
|
1115
|
+
|
|
1116
|
+
Both halves of the cap read one column: `spendUsd`, summed by
|
|
1117
|
+
`store.spendSince`. When the harness stops reporting cost the column reads
|
|
1118
|
+
`0.00`, the sum shrinks, and the cap compares a fraction of reality against its
|
|
1119
|
+
ceiling — #46's *"the daily spend cap is theater"*, recurring. The check's own
|
|
1120
|
+
wording has always said the quiet part: **$0.00 spend is not proof of no spend.**
|
|
1121
|
+
|
|
1122
|
+
Measured on this fleet, counting only runs that did any work:
|
|
1123
|
+
|
|
1124
|
+
| day | worked | reported $0.00 |
|
|
1125
|
+
|---|---|---|
|
|
1126
|
+
| 2026-08-15 → 20 | 15–118 | 0.0–3.8% |
|
|
1127
|
+
| **2026-08-21** | 22 | **31.8%** |
|
|
1128
|
+
| **2026-08-22** | 12 | **75.0%** |
|
|
1129
|
+
|
|
1130
|
+
Six days of baseline, then a regression — with one run taking 213 turns for
|
|
1131
|
+
$0.00. Three things were wrong with the detection, and all three are fixed:
|
|
1132
|
+
|
|
1133
|
+
- **It was all-or-nothing.** The predicate was `window.every(spendUsd === 0)`,
|
|
1134
|
+
so it fired only on total loss. Replayed over all 410 windows in this fleet's
|
|
1135
|
+
recorded history it would have fired 14 times while staying silent through
|
|
1136
|
+
**37** windows that had lost a majority — partial loss, the common case, was
|
|
1137
|
+
the invisible one. On 2026-08-21 every window contained one metered run, so it
|
|
1138
|
+
was silent all day. It is now a share of the window, reported at a majority.
|
|
1139
|
+
- **It counted runs that never worked.** A kill before the first turn records
|
|
1140
|
+
`$0.00` honestly, and 26 of this project's 48 zero-spend rows are exactly
|
|
1141
|
+
those. They are excluded from both sides now, so a burst of administrative
|
|
1142
|
+
kills neither fires a telemetry finding nor dilutes a real one. The sampler
|
|
1143
|
+
over-reads to compensate.
|
|
1144
|
+
- **It was on-demand only.** `spend-telemetry` lived in `doctor` and nowhere
|
|
1145
|
+
else, which is precisely the gap #919 closed for install surfaces — and the
|
|
1146
|
+
stake here is a spend *control*. `status` now carries a `spend telemetry` row
|
|
1147
|
+
directly beneath the figure it qualifies, computed from the same store read,
|
|
1148
|
+
silent while telemetry is healthy.
|
|
1149
|
+
|
|
1150
|
+
The threshold is a majority rather than any single zero, because the healthy
|
|
1151
|
+
baseline is 0–3.8% and not zero: a legitimately cheap run happens, and a row
|
|
1152
|
+
that fires on one of them is a row an operator learns to skip.
|
|
1153
|
+
|
|
1154
|
+
Deliberately absent: any attempt to *estimate* the missing cost from turns or
|
|
1155
|
+
tokens. That would convert a known unknown into a confidently wrong number, on
|
|
1156
|
+
the one control whose job is to stop the fleet spending money.
|
|
1157
|
+
|
|
894
1158
|
`workerMaxTurns` and `workerWallClockMs` are enforced inside the session driver.
|
|
895
1159
|
The daemon reads a live run's effective turn ceiling at every turn boundary. Use
|
|
896
1160
|
`omp-conductor extend <issue> --turns N [--project NAME]` to raise it without
|
|
@@ -1019,6 +1283,123 @@ honour the pattern it says so, and the daemon logs that per run:
|
|
|
1019
1283
|
Worth reading the log for. A run that quietly used a weaker model than you chose
|
|
1020
1284
|
otherwise looks like a run that was merely unlucky.
|
|
1021
1285
|
|
|
1286
|
+
### Cap escalation (`workerEscalationModel`)
|
|
1287
|
+
|
|
1288
|
+
A run that reaches a ceiling with no PR, no head and no salvage commit is
|
|
1289
|
+
classified `turn-cap-spinning` or `wall-clock-cap-spinning`, and normally that
|
|
1290
|
+
is a decomposition verdict: the slice was too big, and switching models would
|
|
1291
|
+
hide it. The cap classes are excluded from `modelFallbacks` for exactly that
|
|
1292
|
+
reason.
|
|
1293
|
+
|
|
1294
|
+
`workerEscalationModel` buys one exception per issue, for the case where the
|
|
1295
|
+
slice was fine and the tier was not:
|
|
1296
|
+
|
|
1297
|
+
```json
|
|
1298
|
+
"workerEscalationModel": "@slow"
|
|
1299
|
+
```
|
|
1300
|
+
|
|
1301
|
+
It is an opaque selector like `workerModel` — a role, a pattern, whatever omp
|
|
1302
|
+
resolves — and conductor never interprets it. On the **first** artifact-free
|
|
1303
|
+
cap kill in an issue's run chain the settlement sweep records a one-shot marker
|
|
1304
|
+
and hands the queue label back — one store transaction, so a daemon that dies
|
|
1305
|
+
mid-recovery restarts into either "nothing happened" or "the continuation is
|
|
1306
|
+
owed", never a chain that reads as escalated with nothing queued — and the
|
|
1307
|
+
next dispatch launches on that selector; the page it sends says so instead of
|
|
1308
|
+
asking a human to requeue by hand. The run
|
|
1309
|
+
row records the selector it dispatched on (`model`) and, once the session
|
|
1310
|
+
settles, the model the harness actually resolved (`resolvedModel`).
|
|
1311
|
+
|
|
1312
|
+
What it deliberately does not do:
|
|
1313
|
+
|
|
1314
|
+
- **It never fires twice for one issue.** The marker is never removed, so a
|
|
1315
|
+
restart cannot repeat it and a second cap follows the existing escalation
|
|
1316
|
+
path — no climb to a third tier.
|
|
1317
|
+
- **It never fires for a `-progress` cap.** A cap kill with work to continue from
|
|
1318
|
+
keeps decomposing.
|
|
1319
|
+
- **It is not a provider fallback.** `modelFallbacks` answers provider aborts,
|
|
1320
|
+
credit refusals and capacity throttling, and still does: an escalated chain
|
|
1321
|
+
fails over normally on top of its new primary, and a cap never advances or
|
|
1322
|
+
resets the provider streak.
|
|
1323
|
+
|
|
1324
|
+
Omit it and nothing changes: a spinning cap escalates to a human exactly as it
|
|
1325
|
+
did before, and the page says no escalation target is configured.
|
|
1326
|
+
|
|
1327
|
+
### Required credential class (`requireOauthProviders`)
|
|
1328
|
+
|
|
1329
|
+
On 2026-08-21 an Anthropic OAuth grant's refresh failed with `invalid_grant`. The
|
|
1330
|
+
exact selector `anthropic/claude-opus-5` still resolved — right provider, right
|
|
1331
|
+
model — through the *same provider's* API key, and one run spent an estimated
|
|
1332
|
+
$25.16 on billing nobody chose.
|
|
1333
|
+
|
|
1334
|
+
That is documented harness behaviour, not a bug. `AuthStorage.getApiKey`
|
|
1335
|
+
resolves first-match-wins across runtime override → config override → stored
|
|
1336
|
+
OAuth → login-sourced API key → provider env var → other stored API key →
|
|
1337
|
+
fallback resolver, and **a model selector does not stop that cascade**. Nothing
|
|
1338
|
+
omp exposes can constrain it: `models.yml` `auth: "oauth"` only shapes the
|
|
1339
|
+
request, a provider-id variant like `xai-oauth` is a real provider that also
|
|
1340
|
+
accepts `XAI_API_KEY`, and there is no setting, env var, selector suffix or
|
|
1341
|
+
request option for it. So this is conductor's fence, not a config passthrough:
|
|
1342
|
+
|
|
1343
|
+
```json
|
|
1344
|
+
"requireOauthProviders": ["anthropic", "openai-codex"]
|
|
1345
|
+
```
|
|
1346
|
+
|
|
1347
|
+
**Declared per provider, never per model or per run.** A credential belongs to a
|
|
1348
|
+
provider — a model has none of its own — and a run's provider is not knowable in
|
|
1349
|
+
advance: `workerModel` is an opaque selector that may be a role alias, and the
|
|
1350
|
+
provider-fault fallback chain can move a live run onto another provider
|
|
1351
|
+
mid-flight. A per-run inference would be a guess dressed as a fact. So the
|
|
1352
|
+
declaration reads "this provider must bill to its subscription", and while that
|
|
1353
|
+
is untrue every candidate waits.
|
|
1354
|
+
|
|
1355
|
+
It is checked at **three** points, and each closes a window the others cannot:
|
|
1356
|
+
|
|
1357
|
+
- **Admission** holds every candidate with the `credential-class` reason, and the
|
|
1358
|
+
hold's detail names the provider, what it resolves to instead, the disabled
|
|
1359
|
+
cause (`invalid_grant` — the actionable half) and the remediation. Fleet-wide
|
|
1360
|
+
like `plan-usage-cap`, and self-clearing: re-authenticate and the next pass
|
|
1361
|
+
admits. Nothing pages, because a page for something an operator fixes in a
|
|
1362
|
+
minute costs more attention than the guard saves.
|
|
1363
|
+
- **Before provisioning**, so a run that cannot launch never clones a mirror or
|
|
1364
|
+
cuts a worktree.
|
|
1365
|
+
- **Immediately before the spawn**, which is the only one that closes the
|
|
1366
|
+
admission-to-session window — a grant can be disabled in it. A run closed here
|
|
1367
|
+
settles `stopped`, with the reason in its report, and **does not spend a failed
|
|
1368
|
+
attempt**: the issue is not what is wrong, the host's credentials are, and
|
|
1369
|
+
charging an attempt would exhaust an issue while the operator re-authenticates.
|
|
1370
|
+
|
|
1371
|
+
The reader is a runnable probe (`src/credential-class.ts`) spawned per check with
|
|
1372
|
+
`--no-install`, exactly as `harness-loader.ts` asks the installed harness its own
|
|
1373
|
+
question. Out of process for two reasons: it keeps the harness's module graph and
|
|
1374
|
+
credential database out of the long-lived daemon, and every read is taken *now*
|
|
1375
|
+
rather than against a pool this process cached an hour ago. It calls the harness's
|
|
1376
|
+
own exported API — `getCredentialOrigin`, `listDisabledCredentials` — never a
|
|
1377
|
+
query against `agent.db`, and prints only an origin kind, an optional
|
|
1378
|
+
environment-variable *name* and disabled classes with their causes. There is
|
|
1379
|
+
nothing in that surface that could carry credential material.
|
|
1380
|
+
|
|
1381
|
+
**This gate fails closed, and that is deliberate.** A probe that cannot answer —
|
|
1382
|
+
no harness, no `bun`, malformed output, an answer about a different provider —
|
|
1383
|
+
refuses, because "we could not check" costs exactly what "it is wrong" costs.
|
|
1384
|
+
That is the opposite posture from the lane interlock, which is inert without its
|
|
1385
|
+
probe.
|
|
1386
|
+
|
|
1387
|
+
**The residue, which this does not fix and must not pretend to.** It is a state
|
|
1388
|
+
check, not an atomic guarantee. If OAuth becomes invalid *during* a request, the
|
|
1389
|
+
installed resolver still disables it and falls through to a same-provider API
|
|
1390
|
+
key, and nothing conductor can pass prevents that: there is no class constraint,
|
|
1391
|
+
and no after-the-fact answer either — `getApiKey` returns a bearer string with no
|
|
1392
|
+
class, assistant messages carry no credential field, and session `credential_pin`
|
|
1393
|
+
entries are OAuth-only *and* change-only, so neither a stale pin nor a missing one
|
|
1394
|
+
proves anything. A transcript audit of billing class would be a fabrication. The
|
|
1395
|
+
only hard lever is omp's own `disabledProviders`, which removes the provider
|
|
1396
|
+
before credential checks — and also removes the subscription route, which is why
|
|
1397
|
+
it stays an operator action. The real fix is upstream: an `AuthApiKeyOptions` that
|
|
1398
|
+
can require a credential class and refuse rather than descend.
|
|
1399
|
+
|
|
1400
|
+
Omit it and nothing changes — no probe runs, and dispatch is byte-for-byte what
|
|
1401
|
+
it was.
|
|
1402
|
+
|
|
1022
1403
|
## Omp settings overlay
|
|
1023
1404
|
|
|
1024
1405
|
`ompSettings` on a project is the fleet-owned channel for saying "this project's
|
|
@@ -1333,6 +1714,18 @@ the command prints that held-notice id instead of claiming delivery — a
|
|
|
1333
1714
|
not default to the marker. It is also not a report — it
|
|
1334
1715
|
leaves no `reports` row, and nothing retries it.
|
|
1335
1716
|
|
|
1717
|
+
Every held notice also names the live path (#882). A hold reaches nobody now, so
|
|
1718
|
+
it is the wrong answer for somebody who is waiting: on 2026-08-21T20:48Z an
|
|
1719
|
+
interrupted tick fell back to this command and the operator's answer sat in the
|
|
1720
|
+
outbox until the 23:30 digest while they waited in the topic they had written in.
|
|
1721
|
+
The hold was correct policy and "nothing was sent" was already printed; what was
|
|
1722
|
+
missing is that `telegram_send` (or `telegram_ask` for a choice) reaches that
|
|
1723
|
+
person in their own topic right now. The line is conditional, because this
|
|
1724
|
+
command runs in a different process from the session and cannot know whether an
|
|
1725
|
+
inbound message is waiting — guessing would either nag every legitimate
|
|
1726
|
+
digest-only note or quietly bypass the availability policy on a hunch. The caller
|
|
1727
|
+
knows; one sentence makes holding a choice rather than a default.
|
|
1728
|
+
|
|
1336
1729
|
### Delivery is at-least-once, and the docs will not pretend otherwise
|
|
1337
1730
|
|
|
1338
1731
|
The Telegram Bot API accepts no client-supplied idempotency key and offers the
|
|
@@ -1515,13 +1908,33 @@ omp-conductor watch list
|
|
|
1515
1908
|
A watch is a row `kind: watch` that the orchestrator opened for itself — either
|
|
1516
1909
|
a `--resolves-when` condition (the daemon checks it and wakes the next tick when
|
|
1517
1910
|
met, exactly as for a question), or a plain carry note the next tick should
|
|
1518
|
-
read. It renders under its own "Watches" heading
|
|
1519
|
-
resolve
|
|
1520
|
-
nobody answered; silently expiring a watch could drop a release the day its
|
|
1521
|
-
condition finally fires. A superseded watch is closed with the ordinary
|
|
1911
|
+
read. It renders under its own "Watches" heading and is never offered to you to
|
|
1912
|
+
resolve. A superseded watch is closed with the ordinary
|
|
1522
1913
|
`decision withdraw <id>`. `decision open --resolves-when` still records an
|
|
1523
1914
|
operator question: a condition governs *when* to ask, not *who* answers.
|
|
1524
1915
|
|
|
1916
|
+
**The seven-day deadline starts when the condition fires, not when the watch was
|
|
1917
|
+
opened** (#966). While a watch is waiting — an unfired condition, or no condition
|
|
1918
|
+
at all — it never expires, because that deadline is for a question nobody
|
|
1919
|
+
answered and silently dropping a watch could lose a release the day its condition
|
|
1920
|
+
finally fires. The moment `conditionMetAt` is stamped the row stops waiting: it
|
|
1921
|
+
is an actionable item, which is exactly what the question deadline bounds, so it
|
|
1922
|
+
gets the ordinary window measured from that instant and closes itself if nobody
|
|
1923
|
+
acts.
|
|
1924
|
+
|
|
1925
|
+
Measured on this fleet 2026-08-23, which is why: three watches had carried
|
|
1926
|
+
`[CONDITION MET — act on this now]` for 10–12 hours over work that was entirely
|
|
1927
|
+
closed (#809 and #888 closed, `omp-conductor@0.18.2` published *and* installed),
|
|
1928
|
+
and would have carried it forever. The codebase already names that cost for
|
|
1929
|
+
install surfaces — *a warning repeated every fifteen minutes trains an operator
|
|
1930
|
+
to ignore the row that matters* — and a permanent "act on this now" is that,
|
|
1931
|
+
aimed at the orchestrator's own attention.
|
|
1932
|
+
|
|
1933
|
+
Two details keep it honest. Re-observing an already-met condition is refused, so
|
|
1934
|
+
the deadline cannot be pushed out on every pass. And clearing a met condition —
|
|
1935
|
+
which `pr-checks-green` and `pr-review-ready` do when the head moves (#808) —
|
|
1936
|
+
restores `never`: a PR that went red again is waiting, not ignored.
|
|
1937
|
+
|
|
1525
1938
|
## Failure classes and recovery by class (#132)
|
|
1526
1939
|
|
|
1527
1940
|
Every run that did not reach a merged PR used to end at a human. The
|
|
@@ -1593,11 +2006,57 @@ Positive evidence only: a tracker that cannot list answers empty, and an empty
|
|
|
1593
2006
|
answer removes nothing — the label is the interlock that keeps two workers off
|
|
1594
2007
|
one issue.
|
|
1595
2008
|
|
|
2009
|
+
### Grooming verdicts are reconciled too (#964)
|
|
2010
|
+
|
|
2011
|
+
Same reasoning, one surface over. A verdict describes an **open** issue —
|
|
2012
|
+
`promotable` means "promote this" — and nothing retired one when its issue
|
|
2013
|
+
closed. `reconcileGrooming` clears only the two per-pass admission holds
|
|
2014
|
+
(`file-lane`, `depends-on`), which are correctly self-clearing; every other row
|
|
2015
|
+
was written once and never removed.
|
|
2016
|
+
|
|
2017
|
+
Measured on this fleet 2026-08-23: **64 of 65 rows named a closed issue, and all
|
|
2018
|
+
26 `promotable` ones did**, so `status` advertised 26 candidates ready to promote
|
|
2019
|
+
on a repo whose only open issue was blocked on an operator. The tick prompt said
|
|
2020
|
+
"Promote the promotable or groom new issues" over the same rows.
|
|
2021
|
+
|
|
2022
|
+
Each pass now retires every row whose issue is absent from a complete
|
|
2023
|
+
`listOpenIssues` snapshot. Three properties make that safe rather than
|
|
2024
|
+
destructive:
|
|
2025
|
+
|
|
2026
|
+
- **A complete snapshot, never a filtered one.** `listOpenIssues` paginates to
|
|
2027
|
+
the end and is conditional (an unchanged repo costs a 304). Reconciling against
|
|
2028
|
+
`listReady` or `listLabeled` would condemn every verdict the filter did not
|
|
2029
|
+
mention — and a promotable issue carries no label at all, which is also why
|
|
2030
|
+
this cannot live inside the label reconcile above.
|
|
2031
|
+
- **Fail closed.** An unreadable tracker retires nothing and logs why. An
|
|
2032
|
+
unreadable tracker and "no issues are open" look identical, and acting on the
|
|
2033
|
+
second the first time GitHub rate-limits a pass would empty the table.
|
|
2034
|
+
- **Silent when nothing changed.** A line every fifteen minutes saying "0
|
|
2035
|
+
retired" is a line nobody reads.
|
|
2036
|
+
|
|
2037
|
+
Note what was *not* broken: candidate selection already built its pool from a
|
|
2038
|
+
live open-issue snapshot (#848) and used grooming rows only as exclusions, so a
|
|
2039
|
+
closed issue was never offered to a to-spec batch. The rot was in what the
|
|
2040
|
+
surfaces reported, and — latently — in `claimable = routed − knownBlocked`, which
|
|
2041
|
+
is arithmetic over the one subset that *was* being reconciled.
|
|
2042
|
+
|
|
1596
2043
|
### Where you see it
|
|
1597
2044
|
|
|
1598
|
-
- `omp-conductor status` grows
|
|
1599
|
-
|
|
1600
|
-
|
|
2045
|
+
- `omp-conductor status` grows up to two blocks, and they answer different
|
|
2046
|
+
questions. **`failure classes (awaiting recovery)`** counts rows whose
|
|
2047
|
+
recovery has not run *and* whose action the sweep will actually take; it is
|
|
2048
|
+
omitted when empty, because empty is the healthy answer. **`failure classes
|
|
2049
|
+
(classified, no action by design)`** counts the two recorded-only actions,
|
|
2050
|
+
`none` and `hold` — a `hold` waits on a human by design, and `none` is what
|
|
2051
|
+
the settle sweep writes beside `returned-for-revision` when a reviewer closes
|
|
2052
|
+
pushed work without merging, where the remedy is the queue label the sweep
|
|
2053
|
+
deliberately leaves on rather than anything a sweep does.
|
|
2054
|
+
|
|
2055
|
+
They were one list until it grew a monotonically increasing tally: on this
|
|
2056
|
+
fleet `returned-for-revision` reached **44** while every actionable class sat
|
|
2057
|
+
at zero and invisible beneath it. That is the #109 defect one level up — a row
|
|
2058
|
+
state is not an issue state, and a count nobody can act on is not a backlog.
|
|
2059
|
+
Classes rather than row states, for the same original reason.
|
|
1601
2060
|
- The board appends `[<class>]` to a card whose newest run carries one.
|
|
1602
2061
|
- The tick prompt carries one line — `Auto-recovered since last tick: 3
|
|
1603
2062
|
(merge-conflict #365, admin-kill #82, …) — already handled, do not re-triage
|
|
@@ -1927,16 +2386,41 @@ and a decision made out of a model's own wording is one no two runs spell the
|
|
|
1927
2386
|
same way. A reason outside its set is refused, and the refusal names every
|
|
1928
2387
|
accepted value.
|
|
1929
2388
|
|
|
2389
|
+
Read from `VERB_SPECS` on 2026-08-23. This table had drifted twice — `label
|
|
2390
|
+
change` was missing `completed`, and `pr review` was absent altogether — which is
|
|
2391
|
+
the ordinary hazard of writing a closed set down in two places. The refusal names
|
|
2392
|
+
the live set, and is the authority when they disagree.
|
|
2393
|
+
|
|
1930
2394
|
| Verb | Accepted reasons |
|
|
1931
2395
|
| --- | --- |
|
|
1932
2396
|
| merge | `preconditions-met`, `behind-base-refreshed`, `operator-instructed`, `release-blocking` |
|
|
1933
2397
|
| release | `batch-complete`, `epic-closed`, `hotfix`, `operator-instructed` |
|
|
1934
|
-
| label change | `promoted-to-queue`, `re-briefed`, `needs-human`, `duplicate`, `superseded`, `out-of-scope` |
|
|
2398
|
+
| label change | `promoted-to-queue`, `re-briefed`, `completed`, `needs-human`, `duplicate`, `superseded`, `out-of-scope` |
|
|
2399
|
+
| pr review | `blocking-findings` |
|
|
1935
2400
|
|
|
1936
2401
|
Free-form rationale still has a home: it rides alongside as a separate
|
|
1937
2402
|
`rationale` field, is written into the audit trail verbatim, and is never
|
|
1938
2403
|
parsed or matched by anything.
|
|
1939
2404
|
|
|
2405
|
+
**And the refusal says so** (#968), because for a long time it did not. An
|
|
2406
|
+
out-of-enum `reason` was refused with the field, every accepted value and the
|
|
2407
|
+
rejected text quoted back — everything except where to put the sentence. Measured
|
|
2408
|
+
on this fleet 2026-08-23, `reason-not-in-enum` was the **largest single refusal
|
|
2409
|
+
cause in the ledger**: 99 of 241, across all four closed-set verbs, over 7.6
|
|
2410
|
+
days. Every one of the 99 carried real audit content — *"landed directly in PR
|
|
2411
|
+
#940; queue label would re-dispatch finished work"* — and **none supplied
|
|
2412
|
+
`rationale`**, because the moment a caller has a sentence to record is the moment
|
|
2413
|
+
they are told only that `reason` is closed. 7578 characters of accounting were
|
|
2414
|
+
refused and discarded.
|
|
2415
|
+
|
|
2416
|
+
So the refusal now ends: *Pick the enum value that fits and put that sentence in
|
|
2417
|
+
`"rationale"`, which is logged verbatim.* It is keyed on the verb's own spec, not
|
|
2418
|
+
a hardcoded list — advising a field the verb would then refuse as
|
|
2419
|
+
`unknown-argument` turns one wasted round trip into two — and it is offered only
|
|
2420
|
+
for `reason`, never for another enum argument like `action`, where prose was
|
|
2421
|
+
never the intent. A test pins the invariant that makes the advice always
|
|
2422
|
+
available: every verb with a closed `reason` set also declares `rationale`.
|
|
2423
|
+
|
|
1940
2424
|
## Orchestrator tick
|
|
1941
2425
|
|
|
1942
2426
|
The escalation path above assumes an orchestrator session that is actually
|
|
@@ -2007,6 +2491,41 @@ The refusal of the raw tool is mechanical (the tool-call gate), not a prompt
|
|
|
2007
2491
|
reminder: a model cannot wait unbounded on a local tick even by omitting the
|
|
2008
2492
|
timeout argument.
|
|
2009
2493
|
|
|
2494
|
+
#### The spec-out questionnaire (#947)
|
|
2495
|
+
|
|
2496
|
+
Decomposing a big issue needs several judgement calls about **one** issue, and
|
|
2497
|
+
asking them one at a time is what left #291's two — *is a Gitea fleet actually
|
|
2498
|
+
wanted*, *refuse-and-document versus emulate epic links* — as prose in an issue
|
|
2499
|
+
body that nothing tracked. `conductor_questionnaire` asks them together:
|
|
2500
|
+
|
|
2501
|
+
- `spec-issue` names the issue being specced, and every item's decision row
|
|
2502
|
+
carries it. That binding is the point: the answers become that issue's
|
|
2503
|
+
provenance, so the next reader sees why a slice is shaped the way it is
|
|
2504
|
+
instead of re-litigating it. A group without one is refused.
|
|
2505
|
+
- `items` are ordinary bounded asks — the same schema, composed rather than
|
|
2506
|
+
restated, so an item can never accept something `conductor_ask` refuses. At
|
|
2507
|
+
most six, and an item that fails validation (including the question-shape
|
|
2508
|
+
ceiling) refuses the **whole group** rather than delivering a question set the
|
|
2509
|
+
rows behind it do not match.
|
|
2510
|
+
- Every row is written in **one transaction, before the single delivery**, so an
|
|
2511
|
+
operator is never shown a question nothing is waiting on.
|
|
2512
|
+
- **One delivery, N decisions.** The items go out as a single numbered message,
|
|
2513
|
+
answerable in any order, and each resolves independently: answering item 2
|
|
2514
|
+
leaves item 1 open. The group is bounded by the same ask ceiling — a
|
|
2515
|
+
questionnaire never waits longer than one `conductor_ask` may.
|
|
2516
|
+
- At the ceiling each unanswered item takes its **own** declared `on-timeout`:
|
|
2517
|
+
`auto-proceed` resolves with `"<option> (auto-applied on ask timeout)"`,
|
|
2518
|
+
`park` stays open. An item the operator answered keeps that answer — the group
|
|
2519
|
+
never resolves atomically, because that would discard the answers they gave.
|
|
2520
|
+
- This path is deliberately plain text rather than six button posts, since six
|
|
2521
|
+
posts is exactly what it exists not to do. So a prose reply is not itself a row
|
|
2522
|
+
resolution — the same contract a degraded single ask has — and the result names
|
|
2523
|
+
each row id plus `omp-conductor decision resolve <id> --answer "…"`.
|
|
2524
|
+
|
|
2525
|
+
Both surfaces are mounted and unmounted together: a beat that cannot route one
|
|
2526
|
+
cannot route the other, and a half-present ask surface would leave a model
|
|
2527
|
+
choosing between a working tool and a broken one.
|
|
2528
|
+
|
|
2010
2529
|
#### Upgrading from one shared arm marker
|
|
2011
2530
|
|
|
2012
2531
|
Before per-project markers every project was given the same `<state dir>/armed`,
|
|
@@ -2256,14 +2775,14 @@ least of all on a fleet whose session lives somewhere else.
|
|
|
2256
2775
|
## CLI reference
|
|
2257
2776
|
|
|
2258
2777
|
```bash
|
|
2259
|
-
omp-conductor setup [area] [--no-ai] [--answers FILE] [--save-answers FILE] [--project NAME]
|
|
2778
|
+
omp-conductor setup [area] [--no-ai] [--answers FILE] [--save-answers FILE] [--resume FILE] [--project NAME]
|
|
2260
2779
|
omp-conductor setup host [NAME] [--project NAME]
|
|
2261
2780
|
omp-conductor setup graph [--no-seed] [--print] [--project NAME]
|
|
2262
2781
|
omp-conductor start [--port N] [--project NAME]
|
|
2263
2782
|
omp-conductor --version
|
|
2264
2783
|
omp-conductor stop
|
|
2265
2784
|
omp-conductor restart [--now] [--timeout SECONDS] [--port N] [--project NAME]
|
|
2266
|
-
omp-conductor upgrade [--to VERSION] [--project NAME]
|
|
2785
|
+
omp-conductor upgrade [--to VERSION] [--bootstrap SHA --source PATH] [--project NAME]
|
|
2267
2786
|
omp-conductor upgrade-install --to VERSION [--project NAME]
|
|
2268
2787
|
omp-conductor upgrade-rollback
|
|
2269
2788
|
omp-conductor status [--project NAME] [--json]
|
|
@@ -2305,13 +2824,13 @@ omp-conductor help
|
|
|
2305
2824
|
| Command | Scope | Behaviour |
|
|
2306
2825
|
| --- | --- | --- |
|
|
2307
2826
|
| `setup [area] [--no-ai] [--answers FILE] [--save-answers FILE] [--project NAME]` | project | The deterministic interview, with styled Clack prompts on an interactive TTY and byte-stable plain output for pipes or `OMP_CONDUCTOR_PLAIN_UI=1`. `--answers` validates a JSON object of stable prompt keys before setup and replaces every prompt; a missing required key exits `1` naming the key and file instead of hanging. `--save-answers` records accepted interactive answers as replayable JSON after a successful run. Bare setup is a full first run, or — when the project already exists — a chooser of which area to amend. Naming an area positionally skips that chooser and amends only that area: `tracker`, `gates`, `caps`, `code-graph`, `authority`, `policy`, `escalation`, `reporting`, `brief`. `host` and `graph` are install subcommands rather than areas and are matched first; anything else exits `2` listing both vocabularies. Every prompt shows its current value as the default, and Enter accepts what you see; `Ctrl-C` at any prompt abandons the run and writes nothing. Setup also **reads your repos to propose answers**: the gates prompt is pre-filled from what CI actually runs, and the brief's `## Project context` and release procedure are drafted from every routing repo and shown for confirmation before anything is written. Each probe is a short session with **no shell, no editor and no verbs** in a throwaway shallow clone, and every answer is a proposal you edit or decline — a probe that cannot clone, cannot reach a model, or answers unusably costs you one warning and the shipped stub. `--no-ai` asks every question with the reading half removed. |
|
|
2308
|
-
| `setup host [--project NAME]` | host | Re-render and stage the systemd unit, then **run** the install: `install -m 0644` into `/etc/systemd/system`, `daemon-reload`, `enable`, `restart`. Stages the fleet recovery oneshot (`omp-conductor-recover.service`) and its playbook (`/usr/local/sbin/omp-conductor-recover`) alongside, and installs them **before** the fleet units: both fleet units carry `OnFailure=` to the recovery unit, so a crash-looped daemon or herdr session now collects evidence durably, attempts one bounded recovery, and pages tier-2 instead of dying silently (#485). Every command is shown with its exact argv, one confirm covers the batch, and `sudo` asks for your password once before the first step — or is skipped entirely on a fleet that genuinely runs as root. The first failure stops the rest and prints the un-run remainder verbatim so you can finish by hand. Refuses an *escalated* invocation (`sudo`, or `sudo -i`/`su -` detected by the invoking account disagreeing with the fleet's) before writing anything, naming both accounts, because staging derives the unit's `User=`/`HOME=` from whoever ran it. On a non-Linux host the files are still staged and only the `systemctl` steps are refused. |
|
|
2827
|
+
| `setup host [--project NAME]` | host | Re-render and stage the systemd unit, then **run** the install: `install -m 0644` into `/etc/systemd/system`, `daemon-reload`, `enable`, `restart`. Stages the fleet recovery oneshot (`omp-conductor-recover.service`) and its playbook (`/usr/local/sbin/omp-conductor-recover`) alongside, and installs them **before** the fleet units: both fleet units carry `OnFailure=` to the recovery unit, so a crash-looped daemon or herdr session now collects evidence durably, attempts one bounded recovery, and pages tier-2 instead of dying silently (#485). Every command is shown with its exact argv, one confirm covers the batch, and `sudo` asks for your password once before the first step — or is skipped entirely on a fleet that genuinely runs as root. The first failure stops the rest and prints the un-run remainder verbatim so you can finish by hand. Refuses an *escalated* invocation (`sudo`, or `sudo -i`/`su -` detected by the invoking account disagreeing with the fleet's) before writing anything, naming both accounts, because staging derives the unit's `User=`/`HOME=` from whoever ran it. On a non-Linux host the files are still staged and only the `systemctl` steps are refused. On a herdr host the last step is `systemctl restart herdr-fleet.service`, which kills every pane in that session — so an invocation from *inside* it is refused before the plan is staged or a single step runs, naming the unit and a context the restart cannot reach (a login shell outside herdr, or a detached `systemd-run` unit). The hazard is the reason, not the herdr host: a plan with no such restart step — a non-herdr host, or a no-op re-run — installs normally from a pane, and `upgrade` has refused the same context since #486 (#834). It also decides who owns the fleet Herdr session before it mutates anything (#893): systemd cannot adopt an already-running process, so an unmanaged `herdr --session <name> server` holding the session makes every start of the unit exit 1 — and because the unit is `Type=simple`, `systemctl restart` returns 0 before the child discovers that. An unmanaged session is therefore stopped inside the same confirmed batch (unprivileged, before the unit install) so the unit can take it over, and a fact that cannot be read is refused rather than guessed — the remedy stops somebody's live session. Success is then proven rather than assumed: a bounded readiness gate re-reads until the unit is `active (running)` with a main pid AND the session answers, so an auto-restart loop is a failed setup. That failure leaves dispatch **paused** (unlike an ordinary failed step, which restores it) and names what it saw: a fleet nobody supervises must not admit workers. |
|
|
2309
2828
|
| `setup graph [--no-seed] [--print] [--project NAME]` | project | The code-graph install end to end, in one preview and one confirm: check the prerequisites read-only and stop before installing anything when `codebase-memory-mcp` is absent or no MCP entry mounts it (printing the entry to add); stage this project's `cbm-reindex-<project>.{sh,service,timer}` **after** the confirm — never before, so a declined run leaves the staged tree byte-identical — refusing a stem that already belongs to another project or to a file it did not generate (#720); `git clone` each missing index-only checkout **as you, never through sudo**; install and enable `cbm-reindex-<project>.timer` as root; then seed one indexing run so the first fetch happens while you watch, and verify with the same probe `status` uses. A repo that does not verify is a failure with the remediation, not a success — staged-but-not-trusted is how you discover months later that no worker read an index. `--no-seed` enables the timer without the seeding run and says plainly the graph is unusable until it first fires; it never skips the prerequisite or clone steps. `--print` changes nothing. Exits `1` when no repo has [`graphProject`](#configuration). |
|
|
2310
2829
|
| `start` | host | Start `herdr-fleet.service` when that optional unit is installed, clearing a previous pane-recovery pin, then start the dispatch daemon and wait until it answers `GET /healthz`. When `omp-conductor.service` is installed, systemd is the only start path: even `start --project NAME` restores the shared unit and uses the name only to verify that `/healthz` serves the requested project. A detached daemon is allowed only when the unit is proven absent. It never clears pause or arms ticks. Refuses if a daemon is already live, naming its pid; manager refusal or unprovable ownership is an error rather than a detached fallback. |
|
|
2311
2830
|
| `stop` | fleet | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
|
|
2312
2831
|
| `restart [--now] [--timeout SECONDS] [--port N] [--project NAME]` | host | Drains the fleet by default: pause new claims, wait until live workers reach `0 / N` (bounded by `--timeout SECONDS`, default 1800 = 30 min), restart, then restore the prior dispatch state. A daemon serving multiple configured projects makes restart host-wide: `--project` is rejected because draining one queue and restarting the shared process would kill another project's workers. Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; only a host proven not to have the installed unit may fall back to the standalone stop/start path. `--now` skips the drain and restarts immediately, orphaning any live runs (old behaviour). A drain that hits `--timeout` restarts nothing and leaves dispatch paused — `omp-conductor resume` lifts it, or re-run `restart` to keep waiting. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
|
|
2313
|
-
| `upgrade [--to VERSION] [--project NAME]` | host | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. Host-wide by default: one daemon serves every configured project, so a bare run drains all of them and refreshes every brief. `--project` is rejected when the live daemon serves several projects — draining one queue and restarting the shared daemon would kill another's workers. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. The same transaction runs detached, without a human, as the fleet-installs-itself path: the orchestrator calls the `conductor_install` verb under the granted `install` shape, the daemon validates the version against npm and starts a transient systemd unit (`upgrade-install`) outside the pane and the daemon, and the first tick after the restart verifies version, `/healthz`, ticks, pane and `doctor` against the durable upgrade journal before restoring dispatch — rolling back and paging tier-2 on any gap. |
|
|
2314
|
-
| `status [--project NAME] [--json]` | fleet | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. `--json` emits the stable structured snapshot directly, without ANSI or prose, for automation. The text project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. Active-run lines overlay cooperative worker `paused`/`pausing` from `/healthz` without changing SQLite `running` state or the live worker count. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
|
|
2832
|
+
| `upgrade [--to VERSION] [--bootstrap SHA --source PATH] [--project NAME]` | host | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. Host-wide by default: one daemon serves every configured project, so a bare run drains all of them and refreshes every brief. `--project` is rejected when the live daemon serves several projects — draining one queue and restarting the shared daemon would kill another's workers. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. The same transaction runs detached, without a human, as the fleet-installs-itself path: the orchestrator calls the `conductor_install` verb under the granted `install` shape, the daemon validates the version against npm and starts a transient systemd unit (`upgrade-install`) outside the pane and the daemon, and the first tick after the restart verifies version, `/healthz`, ticks, pane and `doctor` against the durable upgrade journal before restoring dispatch — rolling back and paging tier-2 on any gap. **`--bootstrap SHA --source PATH`** is the second accepted identity kind (#908), for the case a published version cannot express: the installed conductor is what is broken, and publishing the fix needs the workers that are broken. The sha is the whole identity (40 hex, never a range); `--source` is a checkout that must be AT that sha — verified with `git rev-parse HEAD` — and its only job is to be the tree `bun run check` runs against before anything is installed. A wrong sha, an unreadable tree, or a failing check **refuses**, naming the identity and what failed, with nothing installed. Otherwise it is the same transaction: the same journal (with the checks recorded as evidence), the same three surfaces from that one identity, the same attestation, the same rollback. The `--to` and `--bootstrap` identities are mutually exclusive. |
|
|
2833
|
+
| `status [--project NAME] [--json]` | fleet | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. `--json` emits the stable structured snapshot directly, without ANSI or prose, for automation. The text project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. Active-run lines overlay cooperative worker `paused`/`pausing` from `/healthz` without changing SQLite `running` state or the live worker count. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. The `dispatch` line judges a **fence's owning process** when it has one (#938): a `setup` transaction records `owner=<pid>` in its sentinel, because its fence lives exactly as long as that process — so a killed setup renders `ABANDONED — its process (pid N) is gone` with the one command that clears it, while a live one renders `held by a live process`. An operator `hold`, and the durable hold a failed apply leaves behind, record no owner at all and are never described that way, whatever their age. Nothing here clears anything: a pid is reusable, a paused fleet is the safe state, and `doctor` carries the same verdict as a `dispatch-fence:<project>` warning. |
|
|
2315
2834
|
| `ledger [--issue N] [--limit N] [--json]` | project | The action audit: every [mediated-verb](#the-mediated-verbs-126) mutation and every next-attempt turn budget. Verb entries include the arguments, decision, named refusal, and resulting SHA. Turn-budget entries remain after an override is replaced or consumed. Reads (`conductor_pr_status`) are absent so polling cannot bury the signal. `--issue` narrows both histories; `--limit` defaults to 50. `--json` emits one stable object with `project`, optional `issue`, `entries`, `refused`, and `turnOverrides`. Recent verb refusals and pending turn overrides also appear in `status`. |
|
|
2316
2835
|
| `board [--project NAME]` | fleet | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, the last 24 hours of Merged and Settled, and Parked (an issue the tracker has not confirmed closed — still open, or a label read that failed — so nothing dispatches it until a human labels it). Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
|
|
2317
2836
|
| `hold [--keep-ticks] [--project NAME]` | fleet | Soft stop: pause claiming **and** disarm ticks. Daemon and pane stay up. Prefer this when the intent is "stop the conductor" without killing processes. `--keep-ticks` pauses claiming but leaves the arm marker, so the heartbeat keeps reporting and `resume` alone restores the fleet — no fresh arm challenge. See [Stop the conductor](README.md#stop-the-conductor-hold--stop). |
|
|
@@ -2323,15 +2842,16 @@ omp-conductor help
|
|
|
2323
2842
|
| `extend <issue> --turns N [--project NAME]` | project | Raise a live worker's effective turn ceiling through its owning daemon without restarting its session. If the latest run is failed, killed, orphaned, or blocked and has no live controller, store a one-shot ceiling for that issue's next claimed attempt instead. A next-attempt value must exceed the project base, every extension must stay at or below `workerMaxTurnsCeiling`, and live extensions remain monotonic. The pending value appears in `status`, is recorded in `ledger`, and is consumed atomically by one claim. |
|
|
2324
2843
|
| `worker pause <issue>` / `worker resume <issue>` | project | Cooperatively park one live worker without changing its run state or lane. Pause aborts the active turn to harness idle and freezes the remaining wall-clock budget; resume continues the same session with a prompt to re-check its last action before repeating it. This is separate from fleet-level `hold`, which refuses new claims and work-starting mutations while allowing pre-pause completion work and releases. |
|
|
2325
2844
|
| `worker stop <issue> --reason TEXT [--project NAME]` | project | Terminally end a running or cooperatively paused worker. The reason is required (1–500 characters) and persisted on the run. The command waits for settlement, records the distinct `stopped` state, salvages and publishes dirty work, removes `agent:in-progress` through the durable label outbox, and consumes neither failed-attempt nor continuation budget. If salvage fails, the tree holding the only copy stays in place and the command names it. Repeating stop is idempotent and reports the run's already-terminal state. |
|
|
2326
|
-
| `unblock <issue> [--force] [--no-requeue]` | project | Remove that issue's `blocked` and `failed` labels so an answered escalation can be claimed again, and restore the project queue label by default so the dispatcher actually sees it. `agent:in-progress` comes off too, but only when the newest recorded run is terminal — that row is the proof no worker still owns the issue, so a live run keeps the label (and the queue label stays off until that run settles), and so does an issue with no run row at all. Run history remains intact: blocks consume the independent continuation budget, not failed implementation attempts. The output reports both budgets and warns when either will make the next tick escalate instead of dispatch.
|
|
2327
|
-
| `
|
|
2845
|
+
| `unblock <issue> [--force] [--no-requeue]` | project | Remove that issue's `blocked` and `failed` labels so an answered escalation can be claimed again, and restore the project queue label by default so the dispatcher actually sees it. `agent:in-progress` comes off too, but only when the newest recorded run is terminal — that row is the proof no worker still owns the issue, so a live run keeps the label (and the queue label stays off until that run settles), and so does an issue with no run row at all. Run history remains intact: blocks consume the independent continuation budget, not failed implementation attempts. The output reports both budgets and warns when either will make the next tick escalate instead of dispatch. |
|
|
2846
|
+
| `release-composition declare --campaign ID [--pr URL ...] [--reason TEXT]` · `override --pr URL [--reason TEXT]` · `complete|cancel [--reason TEXT]` · `status` | project | Declare the project's one active release (campaign + exact PR URLs), admit exactly one extra PR by recorded override, retire the guard, or inspect it. While a release is active, `conductor_pr_merge` refuses every other PR with `outside-active-release` — the [active release composition](#the-active-release-composition-850) guard — and the refusal clears only through this lifecycle, never through a merge reason. Every transition survives restarts and is recorded as an immutable material event. |
|
|
2847
|
+
| `verb <conductor_*> [--arg k=v ...]` | project | Run one [mediated verb](#the-mediated-verbs-126) as the orchestrator, from the CLI — the external-orchestrator half of the verb surface. Every argument goes in as a `--arg k=v` string; an orchestrator can merge (`conductor_pr_merge`), label (`conductor_label`), release (`conductor_release`), update a branch (`conductor_pr_update_branch`) or title/body (`conductor_pr_update`), recover a settled run's missing PR (`conductor_pr_recover`), clear the review evidence standing at one exact head (`conductor_pr_review_clear`), or read PR state (`conductor_pr_status`). The daemon applies the same checks and writes the same ledger rows a session's call would; a missing `--arg` is refused exactly as a missing tool argument is, worker-only verbs (`conductor_push`, `conductor_pr_create`) are refused with `role-not-allowed`, and a refusal exits `3`. An unknown verb…
|
|
2328
2848
|
| `friction <kind> --detail TEXT [--issue N]` | project | Record one bounded judgment the daemon cannot infer: an escalation belonged in a digest, or a tick report was noise/surprising. The detail is limited to 160 characters. One event never changes policy; three observations inside seven days make the aggregate eligible for one Learning-loop prompt, followed by a seven-day cooldown. |
|
|
2329
2849
|
| `report --text TEXT [--kind material|digest|tier2|decision-needed|fleet-stopped|confirmed-failure]` | project | Hand a rendered report to the daemon's durable outbox. The command persists the text **before** anything can send and prints a durable handoff id. A material report submitted during quiet hours becomes a held notice until the window opens; otherwise it becomes a report whose delivery the daemon owns, retries with bounded backoff, and records. Delivery is [at-least-once](#report-delivery-the-outbox), so a crash mid-send is retried as a possible repeat and `delivered` never proves exactly one message. `--kind digest` is accepted at most once per local day, decided from the ledger; an unknown `--kind` exits `2`. The remaining kinds declare the report's interrupt category — the escalation handoff: the reporting policy decides between immediate delivery and a durable hold exactly as for a daemon escalation of that category, A repeated identical call exits `2` only while the earlier handoff is still queued undelivered; once it lands, the same text is admitted again (the handoff state decides, not a permanent ledger). Anything still owed appears in `status` with its age. |
|
|
2330
2850
|
| `decision open --question TEXT [--blocks TEXT] [--resolves-when COND]` | project | Record a question the orchestrator has put to you, and print its id. A question that lives only in a session's context is lost at the next compaction — after which it is either asked twice or dropped silently. `--resolves-when` attaches a machine-checkable condition: `pr-merged:<https url>`, `pr-checks-green:<https url>`, `pr-mergeable:<https url>`, `issue-closed:<n>`, `npm-version:<pkg>@<version>`, or `rate-limit-reset:github`; anything else exits `2` listing the six forms. See [The decision ledger](#the-decision-ledger-136). |
|
|
2331
2851
|
| `decision resolve <id> --answer TEXT` | project | Record what you decided. Exits `1` naming the id when it is unknown or no longer open, so a second answer cannot overwrite the first. |
|
|
2332
2852
|
| `decision withdraw <id> [--reason TEXT]` | project | Close a question the session stopped needing, with why. Same guard as `resolve`. |
|
|
2333
2853
|
| `decision list [--json]` | project | Open questions, oldest first: id, age, what each blocks, whether its condition is met, and the question. `--json` emits `{ project, decisions }`; the empty state is an empty array. Prints `no open decisions` in text mode when there are none. Watches are not listed here — `watch list` shows those. |
|
|
2334
|
-
| `watch add --note TEXT [--blocks TEXT] [--resolves-when COND]` | project | Record a condition or carry note the orchestrator set for itself, with no human in the loop (#459). `--resolves-when` attaches a machine-checkable condition the daemon checks for you; a met watch wakes the next tick exactly as a met question does. Renders under its own "Watches" heading, is never counted in `decisions N open`, and
|
|
2854
|
+
| `watch add --note TEXT [--blocks TEXT] [--resolves-when COND]` | project | Record a condition or carry note the orchestrator set for itself, with no human in the loop (#459). `--resolves-when` attaches a machine-checkable condition the daemon checks for you; a met watch wakes the next tick exactly as a met question does. Renders under its own "Watches" heading, is never counted in `decisions N open`, and never expires while it is *waiting* — once a condition fires, the ordinary seven-day window runs from that instant (#966). |
|
|
2335
2855
|
| `watch list [--json]` | project | Open watches, oldest first: id, age, what each blocks, whether its condition is met, and the note. `--json` emits `{ project, watches }`; the empty state is an empty array. Prints `no watches` in text mode when there are none. |
|
|
2336
2856
|
| `daemon` | host | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. Admitted workers run in a tracked background pool, so settlement and capacity checks remain periodic while they work; shutdown drains the pool before closing the store. This is what `start` launches and what a systemd unit should call. |
|
|
2337
2857
|
| `daemon --once` | host | Run a single tick, wait for workers admitted by that tick, and exit. No HTTP server or pidfile — a drill must not register itself as the daemon, or the next reader believes it and the real daemon's in-flight runs get reconciled as orphans. |
|
|
@@ -2519,6 +3039,20 @@ Until that uid exists, a root-or-operator daemon still has a mechanical
|
|
|
2519
3039
|
worktree gate on structured tools and an integrity tripwire on its own package —
|
|
2520
3040
|
but `bash` plus host credentials remain a prompt-and-deploy problem.
|
|
2521
3041
|
|
|
3042
|
+
**One attempt at this shipped and was withdrawn — read it before rebuilding it.**
|
|
3043
|
+
0.18.1 created an `omp-worker` account, launched sessions under it with
|
|
3044
|
+
`setpriv`, projected the operator's agent config to it by ACL, and bind-mounted
|
|
3045
|
+
the operator's `node_modules` read-only so the worker could resolve the harness.
|
|
3046
|
+
Every admitted worker then died before its first turn, one resolution layer at a
|
|
3047
|
+
time: config ACLs, a missing `zod`/`yaml`, a native addon looked up through a
|
|
3048
|
+
different per-UID Bun cache, package-export failures out of the bound tree. The
|
|
3049
|
+
lesson is not "fix the mount": exposing package *files* to another uid is not the
|
|
3050
|
+
same as giving it a runnable runtime. The rollback (#892) restored the
|
|
3051
|
+
fleet-account launch, and `setup host`/`upgrade` now retire that host state
|
|
3052
|
+
wherever it still exists. A next attempt needs an immutable self-contained worker
|
|
3053
|
+
runtime (a compiled binary or a package image), a real end-to-end session
|
|
3054
|
+
preflight before the switch, and atomic rollback — not a narrower ACL.
|
|
3055
|
+
|
|
2522
3056
|
### The orchestrator is unconfined, deliberately
|
|
2523
3057
|
|
|
2524
3058
|
There is **no mechanical file gate on the orchestrator session**, and that is an
|
|
@@ -2559,11 +3093,12 @@ daemon, across a process boundary, not in a prompt the model can rewrite.
|
|
|
2559
3093
|
| --- | --- | --- |
|
|
2560
3094
|
| `conductor_push` | the worker owning the run | The ref is exactly `refs/heads/<that run's branch>`. Fast-forward only; there is no force argument to reject because none is declared. |
|
|
2561
3095
|
| `conductor_pr_create` | the worker owning the run | The run has no open PR (the same guard admission uses); head is the run branch; base is the repo's configured `defaultBranch`. |
|
|
2562
|
-
| `conductor_pr_review` | **orchestrator only** | The PR is one a run of this project opened; the run is in a revisable settled state (`pushed-green`, or `failed`/`killed` after pushing green); the live head still equals the reviewed head and the checks at it are green; no review revision is already in flight
|
|
2563
|
-
| `
|
|
3096
|
+
| `conductor_pr_review` | **orchestrator only** | The PR is one a run of this project opened; the run is in a revisable settled state (`pushed-green`, or `failed`/`killed` after pushing green); the live head still equals the reviewed head and the checks at it are green; no review revision is already in flight. At the review-round ceiling it no longer refuses: it admits one [final adjudication](#the-review-ceiling-adjudication-lifecycle-874) for that exact head under the project's `review.adjudicator` role, idempotently, without consuming a worker round (#932). |
|
|
3097
|
+
| `conductor_pr_review_clear` | **orchestrator only** | The PR is one a run of this project recorded (a `pushed-green` run is the normal case, so revisability is deliberately *not* required); `headSha` equals the live head; something actually blocks the merge at that head; `reason` is non-empty. Green is not required — the merge gate re-checks it. Records an append-only clearance plus a material event, and the [exact-head review gate](#the-exact-head-review-gate-888) then stops refusing that head. |
|
|
3098
|
+
| `conductor_pr_recover` | **orchestrator only** | The target issue has a recorded terminal run whose branch still exists at its exact recorded 40-hex head; the run is not live and not merged; the issue is open; no open PR already closes the issue at that head, and a recorded PR is open (returned), closed/merged (refused), definitively missing (replaced) or unreadable (retryable). No live [mutation lease](#what-holds-a-lane-and-what-merely-owns-an-artifact) holds a file the preserved branch carries — an overlap refuses `recovery-lane-occupied`, and a lane that cannot be read refuses `recovery-lane-unprovable` rather than publishing on an unproven one (#925). Creates the one missing PR from that branch to the configured `defaultBranch` and records it on the run. |
|
|
2564
3099
|
| `conductor_pr_status` | worker or orchestrator | Read-only — nothing to gate. A worker reads only its own run's PR; an orchestrator may name any syntactically valid PR URL, open, merged, or closed, and gets its live state and head (checks are reported when available; a merged or closed PR reports its state instead of an `expected OPEN` refusal). |
|
|
2565
3100
|
| `conductor_pr_update_branch` | orchestrator, or the worker owning the run | The PR belongs to this project and is open. A worker may only name its own run's PR. |
|
|
2566
|
-
| `conductor_pr_merge` | **orchestrator only** | Ordinarily, `authority.merge` equals the caller. A hand-edited `recoveryMerges` entry may instead authorize one exact unrecorded PR/head/reason while held. In both paths, `headSha` equals the live head *at execution time*; checks are green at that same SHA; the project route and migration chain are valid; the project's single merge slot is free. |
|
|
3101
|
+
| `conductor_pr_merge` | **orchestrator only** | Ordinarily, `authority.merge` equals the caller. A hand-edited `recoveryMerges` entry may instead authorize one exact unrecorded PR/head/reason while held. In both paths, while the project has an [active release composition](#the-active-release-composition-850), the PR must be named in it or carry an explicit operator override; then `headSha` equals the live head *at execution time*; checks are green at that same SHA; no [review evidence](#the-exact-head-review-gate-888) stands at that head without a recorded `conductor_pr_review_clear`; the project route and migration chain are valid; the project's single merge slot is free. |
|
|
2567
3102
|
| `conductor_label` | **orchestrator only** | The label is in the project's own vocabulary. Lifecycle labels stay the daemon's. Adding the queue label echoes the parsed [file lane](#the-file-lane-declaration), or refuses with `file-lane-unparseable` when a clearly delimited write-lane section parsed nothing — it never claims fail-open beside a declaration that was actually attempted. |
|
|
2568
3103
|
| `conductor_release` | **orchestrator only** | `authority.release` equals the caller; the per-shape grant permits it; the artefact or environment was declared; the release preconditions hold; the `reason` is in the closed enum. `version-bump-pr` creates or re-validates one deterministic version-only PR and, on a later call, merges only its exact green head through the project's single merge slot. |
|
|
2569
3104
|
| `conductor_install` | **orchestrator only** | Gated like a release act: the `install` shape defaults to `human` and a grant is what moves it. The daemon refuses a version npm does not expose with a full `gitHead`, refuses while another install is still in flight, and otherwise starts a detached transient unit that pauses, drains, installs the CLI/omp plugin/Herdr plugin and reloads — outside this session and the daemon. The unit never declares its own success; the first tick after the restart verifies and reports through the durable outbox. |
|
|
@@ -2584,6 +3119,216 @@ live-head or check validation, or make a general class of PRs mergeable.
|
|
|
2584
3119
|
Authorizations are re-read from config on every call and every attempted merge
|
|
2585
3120
|
is written to the ordinary verb ledger, including refusals.
|
|
2586
3121
|
|
|
3122
|
+
#### The refusals reach `status`, not only `ledger` (#972)
|
|
3123
|
+
|
|
3124
|
+
`status` closes with a bounded verb-ledger block — newest first, five entries,
|
|
3125
|
+
silent when nothing has been called:
|
|
3126
|
+
|
|
3127
|
+
```
|
|
3128
|
+
verb ledger 20 recent call(s), 3 refused (omp-conductor ledger for the rest)
|
|
3129
|
+
2026-08-23 01:13:23 REFUSE conductor_label orchestrator [reason-not-in-enum]
|
|
3130
|
+
```
|
|
3131
|
+
|
|
3132
|
+
The header counts every row the snapshot carries (`STATUS_LEDGER_SCAN`, 20), not
|
|
3133
|
+
the five printed: a count computed from the visible rows would report "1
|
|
3134
|
+
refused" on a fleet that refused eleven things that morning.
|
|
3135
|
+
|
|
3136
|
+
This is the surface an operator reads, and the refusal *class* is what they come
|
|
3137
|
+
for. It had never appeared there. The block was written for `status` in #133 —
|
|
3138
|
+
its own doc comment and both constants say so — but was wired into a second,
|
|
3139
|
+
unshipped copy of the status body in `daemon.ts`, which was already not the live
|
|
3140
|
+
renderer on the day it landed. So for a fortnight the snapshot performed a
|
|
3141
|
+
20-row read on every single `status` call and discarded the result, while the
|
|
3142
|
+
dead copy was kept alive by exactly one test.
|
|
3143
|
+
|
|
3144
|
+
What that cost is measurable: #968 — an out-of-enum `reason` accounting for 41%
|
|
3145
|
+
of every refusal on this fleet — was found by querying the sqlite file by hand.
|
|
3146
|
+
A header reading `99 refused` would have surfaced it in the first week.
|
|
3147
|
+
|
|
3148
|
+
### A review round is charged only when its worker worked (#903)
|
|
3149
|
+
|
|
3150
|
+
The review-round ceiling bounds *polishing*, so it counts rounds a worker
|
|
3151
|
+
actually ran. A revision dispatch that dies before its resumed session takes a
|
|
3152
|
+
turn — a host permission fault, a spawn failure, a daemon shutdown between the
|
|
3153
|
+
claim and the launch — worked no round: the same row is returned to the pending
|
|
3154
|
+
set with its round number, findings and target session unchanged, the run goes
|
|
3155
|
+
back to the exact `pushed-green` state the review verb recorded, and the next
|
|
3156
|
+
dispatch pass resumes it. No label changes and nothing escalates, because
|
|
3157
|
+
nothing has failed that a human can act on.
|
|
3158
|
+
|
|
3159
|
+
The discriminator is "did the resumed session take a turn", never the error
|
|
3160
|
+
text: during the 2026-08-21/22 worker-mount outage every dispatch died at turn
|
|
3161
|
+
zero on an `EACCES` no signature list contained, each death consumed one of the
|
|
3162
|
+
PR's three rounds, and the ceiling then refused any further round — an outage
|
|
3163
|
+
turned into a permanently unmergeable PR. Matching on errno would only wait for
|
|
3164
|
+
the next outage to spell itself differently.
|
|
3165
|
+
|
|
3166
|
+
Retries are counted on the row itself, so they survive a restart, and they are
|
|
3167
|
+
bounded: after three the round settles `failed` exactly as it always did, the
|
|
3168
|
+
issue takes the failed label, and one escalation names it. A round whose worker
|
|
3169
|
+
took even one turn is charged normally, whatever it then did — that is what
|
|
3170
|
+
keeps the ceiling meaningful.
|
|
3171
|
+
|
|
3172
|
+
### The exact-head review gate (#888)
|
|
3173
|
+
|
|
3174
|
+
Green checks say nothing about the review lifecycle, so `conductor_pr_merge`
|
|
3175
|
+
also refuses — with `merge-blocked-by-review` — when durable review evidence
|
|
3176
|
+
stands at exactly the head being merged: a review round that is queued, one
|
|
3177
|
+
that was dispatched and crashed mid-review, or one that settled `failed`
|
|
3178
|
+
there. The evidence is keyed by project + PR + head, so a finding tied to an
|
|
3179
|
+
older head never poisons a corrected push.
|
|
3180
|
+
|
|
3181
|
+
That leaves the question of how a head that will *not* be corrected ever
|
|
3182
|
+
merges — a PR at the review-round ceiling, or one whose findings a human read
|
|
3183
|
+
and dismissed. Re-reviewing the same head is not the answer and never was:
|
|
3184
|
+
`conductor_pr_review` at an unchanged head appends to the pending round or
|
|
3185
|
+
opens a new one, and both block. The answer is an explicit, recorded
|
|
3186
|
+
disposition:
|
|
3187
|
+
|
|
3188
|
+
```bash
|
|
3189
|
+
omp-conductor verb conductor_pr_review_clear --project acme \
|
|
3190
|
+
--arg prUrl=https://github.com/acme/api/pull/12 \
|
|
3191
|
+
--arg headSha=<the live head> \
|
|
3192
|
+
--arg reason="findings answered in review thread; ceiling reached"
|
|
3193
|
+
```
|
|
3194
|
+
|
|
3195
|
+
Three properties make it a disposition rather than a blanket override, and all
|
|
3196
|
+
three are mechanical:
|
|
3197
|
+
|
|
3198
|
+
- **It clears backwards only.** A clearance suppresses evidence whose own last
|
|
3199
|
+
activity is at or before the moment it was taken. A round recorded *after*
|
|
3200
|
+
it blocks the merge again — so a clearance cannot be banked as a standing
|
|
3201
|
+
pass.
|
|
3202
|
+
- **It is head-scoped and non-vacuous.** It names the exact live head, and a
|
|
3203
|
+
clearance at a head where nothing blocks is refused (`review-clearance-vacuous`).
|
|
3204
|
+
- **It bypasses nothing else.** Authority, pause, base-red-freeze, exact-head,
|
|
3205
|
+
green checks, the migration chain, the active release composition and
|
|
3206
|
+
single-flight all still apply to the merge that follows.
|
|
3207
|
+
|
|
3208
|
+
Like the composition override, it is deliberately a separate verb: a merge
|
|
3209
|
+
call's own `reason` is not a clearance. Each clearance is an append-only row
|
|
3210
|
+
with its reason, a verb-ledger entry, and a material event in the digest.
|
|
3211
|
+
|
|
3212
|
+
### The review-ceiling adjudication lifecycle (#874)
|
|
3213
|
+
|
|
3214
|
+
A PR that reaches `review.maxRounds` has no worker rounds left, and returning it
|
|
3215
|
+
to the operator is what conductor exists to avoid. So the ceiling escalates the
|
|
3216
|
+
**automation**: `review.adjudicator` names an OMP model role (a role name, never
|
|
3217
|
+
a provider/model — OMP owns that selection), and a terminal adjudication decides
|
|
3218
|
+
the head with a stronger model.
|
|
3219
|
+
|
|
3220
|
+
What exists today is the durable lifecycle underneath it. One adjudication is
|
|
3221
|
+
recorded per project + PR + **exact head**, and that uniqueness is a database
|
|
3222
|
+
index rather than a flag a caller remembers to check, so a repeated tick, two
|
|
3223
|
+
dispatchers and a daemon restart all re-read the same row instead of launching a
|
|
3224
|
+
second adjudicator. A live adjudication on the same PR at a *different* head is
|
|
3225
|
+
refused: the PR moved, and two live adjudications on one PR is exactly the
|
|
3226
|
+
duplicate this prevents.
|
|
3227
|
+
|
|
3228
|
+
It is deliberately **not** a round in `review_revisions`. Reusing round N+1
|
|
3229
|
+
would spend a ceiling that is already exhausted, and would resume the same
|
|
3230
|
+
session and model for a nominal fourth opinion — the failure the escalation
|
|
3231
|
+
exists to end. The two tables also answer different questions: a revision says
|
|
3232
|
+
"go and change this", an adjudication says "decide about this".
|
|
3233
|
+
|
|
3234
|
+
The terminal states **are** the verdict — `cleared`, `rejected`,
|
|
3235
|
+
`unavailable-model`, `stale-head`, `failed` — with no second verdict column to
|
|
3236
|
+
disagree with them. Stored beside the state is what it cannot say: the
|
|
3237
|
+
`evidence` that produced it, and the `disposition` applied because of it.
|
|
3238
|
+
`status` renders each one under its run as `adjudication <state> role <asked>
|
|
3239
|
+
→ <model actually launched>`, matched on PR + head so a corrected push never
|
|
3240
|
+
appears to carry the older head's decision.
|
|
3241
|
+
|
|
3242
|
+
**Reaching the ceiling now admits one instead of refusing.** `conductor_pr_review`
|
|
3243
|
+
at `maxRounds` used to refuse and tell the orchestrator to leave the PR open and
|
|
3244
|
+
escalate; it now records the adjudication for the head it just verified live and
|
|
3245
|
+
green, carrying the findings that provoked it — the most relevant single input the
|
|
3246
|
+
adjudicator gets. The ceiling on *worker* rounds is unchanged: no fourth revision
|
|
3247
|
+
is dispatched and no round is consumed. A repeated call at the same head is
|
|
3248
|
+
idempotent; a call whose head has moved has already refused (`head-stale`) before
|
|
3249
|
+
the admission, and one made while an adjudication is live at a different head
|
|
3250
|
+
refuses `review-in-flight`.
|
|
3251
|
+
|
|
3252
|
+
**The daemon's own pass launches it**, next to review-revision dispatch and never
|
|
3253
|
+
inside it: a revision resumes an implementation worker to change code, an
|
|
3254
|
+
adjudication opens a fresh read-only session to decide about code, and folding
|
|
3255
|
+
them together is how "escalate the automation" quietly becomes "resume the same
|
|
3256
|
+
worker with a different label". The pass re-reads the live head first (a moved
|
|
3257
|
+
head settles `stale-head` without launching, an unreadable one stays pending for
|
|
3258
|
+
the next tick), claims `pending → running` — that transition is the single-flight
|
|
3259
|
+
guard — assembles the evidence from durable rows plus live tracker reads, and
|
|
3260
|
+
settles the row from the verdict:
|
|
3261
|
+
|
|
3262
|
+
| Result | State |
|
|
3263
|
+
|---|---|
|
|
3264
|
+
| verdict `cleared` | `cleared` — the ordinary orchestrator-owned merge path |
|
|
3265
|
+
| verdict `rejected` | `rejected` — #876's autonomous disposition |
|
|
3266
|
+
| a verdict naming a different head | `stale-head`, fail-closed |
|
|
3267
|
+
| no verdict, zero turns | `unavailable-model` — the role never launched; a configuration fault |
|
|
3268
|
+
| no verdict, turns spent | `failed` |
|
|
3269
|
+
|
|
3270
|
+
An adjudication occupies no worker slot and no issue, charges no failed attempt
|
|
3271
|
+
and no continuation, and its turn ceiling (`ADJUDICATION_MAX_TURNS`) is
|
|
3272
|
+
deliberately small: a verdict is a read and an answer, not an implementation. A
|
|
3273
|
+
daemon restart settles a `running` adjudication `failed` — its session died with
|
|
3274
|
+
the process, an adjudicator has no branch or transcript worth resuming, and
|
|
3275
|
+
re-queueing it would be a second launch for one head.
|
|
3276
|
+
|
|
3277
|
+
**The verdict is then acted on** (#876), in its own pass, because a verdict and
|
|
3278
|
+
what was done about it are two facts:
|
|
3279
|
+
|
|
3280
|
+
- **`cleared`** records a [`conductor_pr_review_clear`](#the-exact-head-review-gate-888)
|
|
3281
|
+
at that exact head, by `adjudication:<role>`. That is the whole mechanism — the
|
|
3282
|
+
merge gate already reads clearances, so there is no adjudication-shaped
|
|
3283
|
+
exception inside `conductor_pr_merge` and one audit trail covers both. The head
|
|
3284
|
+
is re-read first: if it moved, no clearance is recorded and the row says so.
|
|
3285
|
+
- **Every other terminal state** — `rejected`, `failed`, `unavailable-model`,
|
|
3286
|
+
`stale-head` — closes the pull request, having first posted the adjudicator's
|
|
3287
|
+
reasons and every preserved review finding on it. They all mean "this head is
|
|
3288
|
+
not merging and no further round is coming", and the blocking artefact is the
|
|
3289
|
+
same in each case: an open PR occupying the issue. The branch is untouched.
|
|
3290
|
+
|
|
3291
|
+
Nothing here opens a review round, launches a second adjudicator, or asks an
|
|
3292
|
+
operator for anything. No budget is reset either: the closed PR is charged by the
|
|
3293
|
+
ordinary settle sweep exactly as any PR closed without merging, which is what
|
|
3294
|
+
lets this compose with a separate corrective-worker path instead of resetting its
|
|
3295
|
+
counters.
|
|
3296
|
+
|
|
3297
|
+
The recorded `disposition` is the idempotency key — a row that already has one is
|
|
3298
|
+
skipped, so a restart or a retried tick cannot close one PR twice — and it is
|
|
3299
|
+
written only *after* the act succeeds, so a failed close leaves the row
|
|
3300
|
+
undisposed and retryable rather than claiming work it did not do. As a backstop,
|
|
3301
|
+
`conductor_pr_merge` refuses any head whose adjudication settled anything other
|
|
3302
|
+
than `cleared`, in case a close failed or raced.
|
|
3303
|
+
|
|
3304
|
+
### The active release composition (#850)
|
|
3305
|
+
|
|
3306
|
+
An early patch is only trustworthy while nothing unrelated slips into it, so a
|
|
3307
|
+
project can declare **one active release composition**: a campaign identifier
|
|
3308
|
+
(say `v0.18.1-reliability`) plus the full PR URLs allowed into that release.
|
|
3309
|
+
While one is active, `conductor_pr_merge` refuses every other PR with the
|
|
3310
|
+
`outside-active-release` refusal naming the campaign and the conflicting PR —
|
|
3311
|
+
checked before the merge lock and any tracker call, from SQLite rows that
|
|
3312
|
+
survive daemon restarts, so the guard is mechanical rather than a policy
|
|
3313
|
+
sentence or a status warning.
|
|
3314
|
+
|
|
3315
|
+
The guard moves only through the explicit operator lifecycle:
|
|
3316
|
+
|
|
3317
|
+
```bash
|
|
3318
|
+
omp-conductor release-composition declare --campaign v0.18.1-reliability \
|
|
3319
|
+
--pr https://github.com/acme/api/pull/12 # open the release
|
|
3320
|
+
omp-conductor release-composition override --pr https://github.com/acme/api/pull/13 \
|
|
3321
|
+
--reason "hotfix belongs in this patch" # admit exactly one extra PR
|
|
3322
|
+
omp-conductor release-composition complete # or cancel — retires the guard
|
|
3323
|
+
```
|
|
3324
|
+
|
|
3325
|
+
A merge call's free-form `reason` is never an override, and an override bypasses
|
|
3326
|
+
only the composition check: authority, pause, base-red-freeze, review, exact-head,
|
|
3327
|
+
green-checks and single-flight all still apply to the admitted merge. Every
|
|
3328
|
+
transition writes an immutable material event, overrides stay in the audit trail
|
|
3329
|
+
scoped to their campaign, and merging a member PR never retires the composition —
|
|
3330
|
+
only `complete` or `cancel` does.
|
|
3331
|
+
|
|
2587
3332
|
For a repo with `release.versionFile`, call `conductor_release` with
|
|
2588
3333
|
`shape=version-bump-pr` and the intended `v<semver>` tag. The requested version
|
|
2589
3334
|
must be newer than the live semantic version. The first call creates
|
|
@@ -2607,6 +3352,31 @@ A `github-release` for the same repo likewise requires that reviewed tag to be
|
|
|
2607
3352
|
present on origin and verifies the tag's version file before creating the
|
|
2608
3353
|
release; it never lets GitHub synthesize the missing tag.
|
|
2609
3354
|
|
|
3355
|
+
**Pinning a tag to one commit (#695).** Both tag shapes accept
|
|
3356
|
+
`--arg sha=<commit>`, and it removes the dependence on *when* the tag is cut.
|
|
3357
|
+
Without it the target is whatever the default branch is at cut time, which
|
|
3358
|
+
deadlocks any fleet whose queue is not empty: cutting needs a window with no
|
|
3359
|
+
unsettled run, reaching that window needs the open PR merged, and merging moves
|
|
3360
|
+
the branch past the tag that was just cut — leaving it unpushable (behind main),
|
|
3361
|
+
un-re-cuttable (a tag the tool cut is never force-moved) and with no shape able
|
|
3362
|
+
to clean it up. The 2026-08-18 occurrence burned a version number.
|
|
3363
|
+
|
|
3364
|
+
With a pinned commit:
|
|
3365
|
+
|
|
3366
|
+
- `git-tag` cuts at exactly that commit, and the "behind live main" refusal does
|
|
3367
|
+
not apply — being behind is the point;
|
|
3368
|
+
- `git-push-tags` publishes that tag **unchanged**, instead of re-pointing it to
|
|
3369
|
+
the live branch head as an unpinned push does. A local tag standing somewhere
|
|
3370
|
+
other than the pinned commit is a refusal, never a silent move;
|
|
3371
|
+
- a commit the released repository does not have, or one **not reachable from
|
|
3372
|
+
the released branch**, is refused naming the sha: a tag pointing off-branch is
|
|
3373
|
+
a release of code nobody merged;
|
|
3374
|
+
- omitting `sha` leaves every existing behaviour exactly as it was, and the
|
|
3375
|
+
argument is refused on any shape that cuts no tag rather than being ignored.
|
|
3376
|
+
|
|
3377
|
+
Deliberately not fixed here: the `runs-settled` precondition's scope (see #603,
|
|
3378
|
+
already repo-scoped) and any tag-delete or re-cut shape.
|
|
3379
|
+
|
|
2610
3380
|
### The transport
|
|
2611
3381
|
|
|
2612
3382
|
Identity is never an argument. `project`, `run`, `issue` and the caller's role
|
|
@@ -2775,16 +3545,81 @@ Known and deliberate in this version:
|
|
|
2775
3545
|
re-derived, not forgotten), and one closing notice lands when the orchestrator
|
|
2776
3546
|
recovers. What remains a separate watchdog is the `.conductor-stalled` wedge
|
|
2777
3547
|
path, for a session that stays alive but stops draining its queue.
|
|
2778
|
-
- **
|
|
2779
|
-
worker is an omp session the daemon starts as a child process
|
|
2780
|
-
|
|
2781
|
-
|
|
2782
|
-
|
|
2783
|
-
|
|
2784
|
-
|
|
2785
|
-
|
|
2786
|
-
|
|
2787
|
-
|
|
3548
|
+
- **A worker gets a Herdr pane, but the pane is never the worker (#840).** Each
|
|
3549
|
+
worker is an omp session the daemon starts as a child process, and that child
|
|
3550
|
+
stays the authoritative one: the typed control socket, the mediated verb
|
|
3551
|
+
socket, the worktree, the transcript, the accounting and the output-schema
|
|
3552
|
+
settlement are all its. What the workspace shows is a *representation* of that
|
|
3553
|
+
exact child — a pane named `worker-<project>-<issue>-a<attempt>-<run>`, created
|
|
3554
|
+
at spawn from the pid the spawn path itself reports, and reported to Herdr
|
|
3555
|
+
through its external-supervisor verbs (`pane report-agent-session` for the run
|
|
3556
|
+
and transcript identity, `pane report-agent` for `working`/`idle`/`blocked`,
|
|
3557
|
+
`pane release-agent` when the child is gone).
|
|
3558
|
+
|
|
3559
|
+
Nothing starts a second OMP process, and nothing reads the pane back: no run
|
|
3560
|
+
state, settlement, turn, spend or model fact is ever derived from terminal
|
|
3561
|
+
output. The pane displays `omp-conductor tail`, the existing read-only
|
|
3562
|
+
follower. Input cannot reach the session either, and that is structural rather
|
|
3563
|
+
than a check — the authoritative child is spawned with `stdin: "ignore"`, so
|
|
3564
|
+
there is no descriptor for a keystroke to travel down, and the pane's own
|
|
3565
|
+
process is a follower whose stdin reaches only itself.
|
|
3566
|
+
|
|
3567
|
+
**The identity is durable, so the workspace survives the process (#842).** The
|
|
3568
|
+
pid, pane id and label are written onto the run row the moment the pane is
|
|
3569
|
+
established — before it is announced, because a pane the store does not know
|
|
3570
|
+
about is a pane a restart cannot reconcile. Three consequences follow, and each
|
|
3571
|
+
is the answer to a way the pre-#842 representation went wrong:
|
|
3572
|
+
|
|
3573
|
+
- **A re-entered launch adopts, never duplicates.** A run that already carries a
|
|
3574
|
+
pane identity is re-reported against that pane; only a run with none creates
|
|
3575
|
+
one. Two panes claiming one worker is worse than none, because both look real.
|
|
3576
|
+
- **A restart releases what it inherits.** The startup orphan sweep hands the
|
|
3577
|
+
pane of every reaped run back to Herdr (`releaseOrphanedWorkerPane`), so a
|
|
3578
|
+
workspace never shows a live worker for a child that died with the previous
|
|
3579
|
+
daemon. The recorded pid is deliberately *not* consulted for liveness: a
|
|
3580
|
+
`session-host` child dies with the daemon that owned its verb socket, so the
|
|
3581
|
+
worker is gone whatever the pid says — and pids are reused, so probing one is
|
|
3582
|
+
exactly how a stranger's process comes to read as a live worker.
|
|
3583
|
+
- **State is projected from transitions, never scraped.** Pause and resume reach
|
|
3584
|
+
the pane through the control registry's own `onPhase` notifier — the phase the
|
|
3585
|
+
control actually reached, fired only on success, so a refused pause never
|
|
3586
|
+
shows as `idle`. A cap kill reports `unknown`. Each report carries the run's
|
|
3587
|
+
own monotonic `seq`, so a late report cannot overwrite a newer state.
|
|
3588
|
+
|
|
3589
|
+
**Herdr unavailability is a named state, and retention is finite (#841).** A
|
|
3590
|
+
worker with no pane keeps working — visibility is never a precondition — but it
|
|
3591
|
+
never does so quietly: the reason is written to its run row and `omp-conductor
|
|
3592
|
+
status` prints `pane degraded — <reason>` on that run's own line, so a fleet
|
|
3593
|
+
running blind cannot be mistaken for a fleet with nothing to show. Every
|
|
3594
|
+
dispatch pass then reconciles the workspace against the live run set
|
|
3595
|
+
(`reconcileWorkerPanes`), which is what makes the surface survive a *Herdr*
|
|
3596
|
+
restart — an event conductor is never told about:
|
|
3597
|
+
|
|
3598
|
+
- A live run whose pane Herdr still has is left alone (`intact`), so repeated
|
|
3599
|
+
passes converge instead of churning panes.
|
|
3600
|
+
- A live run whose pane Herdr lost is re-associated: one new pane, for the pid
|
|
3601
|
+
already on the row. No second OMP process is started, ever. A run with no
|
|
3602
|
+
recorded pid stays `untracked` rather than getting a pane with nothing behind
|
|
3603
|
+
it.
|
|
3604
|
+
- A conductor-owned pane whose run id is not live is released — the retention
|
|
3605
|
+
bound: a settled worker's pane is conductor's only until the next pass. It is
|
|
3606
|
+
matched by the run id Herdr itself reports, never by label or age, so a live
|
|
3607
|
+
worker's pane can never be cleaned up and a stale lookalike can never be
|
|
3608
|
+
mistaken for one. Release, never close: nothing here can terminate a live
|
|
3609
|
+
authoritative worker, because the only mutation is handing lifecycle
|
|
3610
|
+
authority back.
|
|
3611
|
+
- An unreadable or absent Herdr reconciles nothing and says so on every pass.
|
|
3612
|
+
Failing to read the workspace is not evidence that a worker's pane is stale.
|
|
3613
|
+
|
|
3614
|
+
Fleet recovery knows the difference too: `herdr/bin/recover.sh` excludes any
|
|
3615
|
+
pane carrying the `omp-conductor` source from its resume candidates, so a
|
|
3616
|
+
released worker pane — which has no agent and sits in the fleet cwd, and so
|
|
3617
|
+
looks free to every other check — is never where the orchestrator gets started.
|
|
3618
|
+
|
|
3619
|
+
The cap still does the work. The admission loop (`admitCandidates` in
|
|
3620
|
+
`src/daemon.ts`) computes `slots = maxConcurrentWorkers - live workers`, admits
|
|
3621
|
+
at most that many issues per tick, and dispatches them together. `omp-conductor
|
|
3622
|
+
status` remains the authoritative list of occupied issues.
|
|
2788
3623
|
- **Report delivery is at-least-once, never exactly-once.** The Telegram Bot API
|
|
2789
3624
|
takes no client-supplied idempotency key, so the window between "Telegram
|
|
2790
3625
|
accepted it" and "SQLite recorded that" is irreducible. The daemon resolves it
|