omp-conductor 0.15.10 → 0.15.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +273 -2544
- package/REFERENCE.md +2680 -0
- package/package.json +3 -2
- package/schema/config.schema.json +11 -23
- package/src/arm-challenge.ts +112 -0
- package/src/ask.ts +434 -0
- package/src/board.ts +81 -15
- package/src/brief-upgrade.ts +114 -8
- package/src/briefs/orchestrator.md +113 -34
- package/src/briefs/policy.md +33 -8
- package/src/briefs/worker.md +18 -9
- package/src/chain-check.ts +1 -1
- package/src/check-trailing-newlines.ts +82 -0
- package/src/cli.ts +225 -1406
- package/src/commands/arm.ts +21 -0
- package/src/commands/board.ts +23 -0
- package/src/commands/brief-upgrade.ts +186 -0
- package/src/commands/context.ts +150 -0
- package/src/commands/daemon.ts +71 -0
- package/src/commands/dashboard.ts +74 -0
- package/src/commands/decision.ts +103 -0
- package/src/commands/disarm.ts +21 -0
- package/src/commands/doctor.ts +100 -0
- package/src/commands/event.ts +62 -0
- package/src/commands/extend.ts +64 -0
- package/src/commands/friction.ts +56 -0
- package/src/commands/help.ts +9 -0
- package/src/commands/hold.ts +26 -0
- package/src/commands/intake.ts +155 -0
- package/src/commands/ledger.ts +69 -0
- package/src/commands/message.ts +96 -0
- package/src/commands/report.ts +206 -0
- package/src/commands/restart.ts +76 -0
- package/src/commands/resume.ts +58 -0
- package/src/commands/setup.ts +143 -0
- package/src/commands/start.ts +23 -0
- package/src/commands/stats.ts +131 -0
- package/src/commands/status.ts +48 -0
- package/src/commands/stop.ts +51 -0
- package/src/commands/tail.ts +109 -0
- package/src/commands/unblock.ts +39 -0
- package/src/commands/upgrade-install.ts +31 -0
- package/src/commands/upgrade-rollback.ts +32 -0
- package/src/commands/upgrade.ts +25 -0
- package/src/commands/verb.ts +83 -0
- package/src/commands/version.ts +30 -0
- package/src/commands/worker.ts +100 -0
- package/src/config-schema.ts +47 -1
- package/src/config.ts +45 -4
- package/src/daemon.ts +972 -101
- package/src/dashboard/app.js +459 -0
- package/src/dashboard/index.html +61 -0
- package/src/dashboard/server.ts +481 -0
- package/src/dashboard/style.css +348 -0
- package/src/decisions.ts +39 -14
- package/src/diff-flags.ts +131 -241
- package/src/doctor.ts +932 -0
- package/src/escalate.ts +2 -2
- package/src/failure-class.ts +66 -3
- package/src/fleet.ts +58 -1
- package/src/gitops.ts +157 -0
- package/src/graph-health.ts +1 -1
- package/src/label-projection.ts +1 -1
- package/src/lifecycle.ts +198 -2
- package/src/model-fallback.ts +177 -0
- package/src/notices.ts +9 -0
- package/src/omp.ts +93 -13
- package/src/orchestrator-tick.ts +414 -20
- package/src/orchestrator.ts +4 -4
- package/src/privileged.ts +10 -0
- package/src/release-policy.ts +342 -30
- package/src/reports.ts +19 -5
- package/src/session-host.ts +11 -5
- package/src/setup-host.ts +663 -25
- package/src/setup-install.ts +292 -28
- package/src/setup-wizard.ts +255 -74
- package/src/setup.ts +156 -35
- package/src/stats.ts +331 -0
- package/src/store.ts +219 -22
- package/src/tracker/github.ts +74 -4
- package/src/types.ts +215 -31
- package/src/unblock.ts +55 -11
- package/src/upgrade-journal.ts +220 -0
- package/src/upgrade-verify.ts +506 -0
- package/src/upgrade.ts +385 -58
- package/src/verbs/actions.ts +73 -1
- package/src/verbs/protocol.ts +29 -4
- package/src/verbs/server.ts +183 -20
- package/src/worker.ts +3 -3
- package/systemd/omp-conductor-recover.sh +433 -0
- package/systemd/omp-conductor.service.example +14 -3
- package/systemd/recover-unit-test.sh +428 -0
package/README.md
CHANGED
|
@@ -36,12 +36,12 @@ The package ships two deployables:
|
|
|
36
36
|
| Deployable | Entry | What it is for |
|
|
37
37
|
| --- | --- | --- |
|
|
38
38
|
| Everything an operator does | `omp-conductor` binary | The only operator surface: setup, inspect, control, and the dispatch loop itself as a background process (`start` / `stop` / `restart`) with a `/healthz` endpoint for a supervisor. |
|
|
39
|
-
| Orchestrator heartbeat | omp extension, activated by `.conductor-tick.json` | Prompts a 24/7 orchestrator session on a fixed interval so its standing loop actually runs, and marks the session stalled when its prompts stop being consumed. Inert in every other session — including a second session opened in the fleet's own directory. See [Orchestrator tick](#orchestrator-tick). |
|
|
39
|
+
| Orchestrator heartbeat | omp extension, activated by `.conductor-tick.json` | Prompts a 24/7 orchestrator session on a fixed interval so its standing loop actually runs, and marks the session stalled when its prompts stop being consumed. Inert in every other session — including a second session opened in the fleet's own directory. See [Orchestrator tick](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#orchestrator-tick). |
|
|
40
40
|
|
|
41
41
|
**There is no slash command and no skill.** Earlier releases shipped a
|
|
42
42
|
`/conductor` command and a `skill://conductor-onboarding`; both are gone. The
|
|
43
43
|
interview they wrapped now lives in the binary as `omp-conductor setup`, which is
|
|
44
|
-
the documented path for every operator and agent — see [Onboarding](#onboarding).
|
|
44
|
+
the documented path for every operator and agent — see [Onboarding](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#onboarding).
|
|
45
45
|
An omp chat session that wants conductor state or wants to change it **shells out
|
|
46
46
|
to `omp-conductor`**, which is why the verb ledger records who asked: a session
|
|
47
47
|
that ran a verb is indistinguishable from an operator who ran the same verb, and
|
|
@@ -86,50 +86,6 @@ runtime acts on:
|
|
|
86
86
|
| `decisions` (recommended) | Tier-2 decisions and fleet-stopping conditions, when operator availability permits. | Every other material event is recorded durably and ships with the configured digest as one message. |
|
|
87
87
|
| `quiet` | Tier-2 escalations, fleet stops, and confirmed failures, when operator availability permits. | Everything else waits for one daily rollup. |
|
|
88
88
|
|
|
89
|
-
**What the scope does:** the [orchestrator heartbeat](#orchestrator-tick) appends
|
|
90
|
-
the current constraint to every tick it sends, so the reporting contract arrives
|
|
91
|
-
with the prompt instead of only in a brief the session read hours ago. Explicit
|
|
92
|
-
policies name their actual interrupt categories and digest cadence. The policy is
|
|
93
|
-
re-read from `~/.omp/conductor/config.json` on **every** tick, and escalation,
|
|
94
|
-
direct Telegram, and durable report paths apply it mechanically. Turning the
|
|
95
|
-
volume up or down — `omp-conductor setup` again, or an edit to the file — therefore
|
|
96
|
-
binds the next tick without restarting the session.
|
|
97
|
-
|
|
98
|
-
No config, an unreadable or invalid config, or several unnamed projects fall
|
|
99
|
-
back to the legacy `material` scope for the heartbeat and log the reason once.
|
|
100
|
-
An invalid live availability policy blocks autonomous Telegram fail-closed; it
|
|
101
|
-
does not guess that the operator is awake.
|
|
102
|
-
|
|
103
|
-
Changing the key later does not rewrite an `ORCHESTRATOR.md` you already have.
|
|
104
|
-
The generated `POLICY.md` describes every scope without pinning the current
|
|
105
|
-
choice; the tick constraint remains derived from live config. Keep any
|
|
106
|
-
operator-owned reporting additions in `ORCHESTRATOR.md` consistent with it.
|
|
107
|
-
|
|
108
|
-
## Where issues come from
|
|
109
|
-
|
|
110
|
-
**GitHub Issues is the only supported tracker in v1.** `tracker.kind` accepts
|
|
111
|
-
exactly one value, `"github"`, and every tracker operation shells out to your
|
|
112
|
-
already-authenticated `gh` CLI — the conductor never stores a token of its own.
|
|
113
|
-
Gitea, Jira, and file-based trackers are not supported yet; the seam for them is
|
|
114
|
-
`src/tracker/github.ts`, which implements the whole nine-method `Tracker`
|
|
115
|
-
interface in `src/types.ts` (`listReady`, `addLabel`, `removeLabel`, `comment`,
|
|
116
|
-
`close`, `linkParent`, `parentOf`, `openCloserFor`, `prState`) that a future
|
|
117
|
-
backend would swap in.
|
|
118
|
-
|
|
119
|
-
You tell the conductor where to look with three keys, all in
|
|
120
|
-
`~/.omp/conductor/config.json` (the [Configuration](#configuration) section has
|
|
121
|
-
the full annotated example, and `omp-conductor setup` will interview you for these
|
|
122
|
-
and create any missing labels):
|
|
123
|
-
|
|
124
|
-
| Key | Meaning |
|
|
125
|
-
| --- | --- |
|
|
126
|
-
| `tracker.repo` | The **one** `owner/repo` whose issue list is the queue. This is your planning repo — it does not have to contain any code. |
|
|
127
|
-
| `queueLabel` | Open issues in `tracker.repo` carrying this label are the work queue. Nothing else is ever read. Required: the wizard pre-fills `ready-for-agent`, but a config that omits the key is rejected, not defaulted. |
|
|
128
|
-
| `routing.repos` + `repo:<name>` labels | Each queued issue must also carry exactly one routing label naming which code repo the work lands in. The conductor cuts the worktree and PR there, from `routing.repos[name].cloneUrl`. An issue with zero or two routing labels is reported as unroutable and skipped — never guessed. |
|
|
129
|
-
|
|
130
|
-
So: one tracker repo supplies the queue, routing labels fan issues out to any
|
|
131
|
-
number of code repos, and both label names are yours to configure.
|
|
132
|
-
|
|
133
89
|
## Install
|
|
134
90
|
|
|
135
91
|
```bash
|
|
@@ -140,379 +96,23 @@ That installs the `omp-conductor` binary and the orchestrator heartbeat extensio
|
|
|
140
96
|
It registers **no slash command and no skills** — everything an operator does is a
|
|
141
97
|
verb on the binary, starting with `omp-conductor setup`.
|
|
142
98
|
|
|
99
|
+
After install — and after every `omp-conductor upgrade` — run
|
|
100
|
+
`omp-conductor doctor` once: it checks the deployment faults that have
|
|
101
|
+
previously cost debugging sessions (gh auth/scopes, exact-case labels, systemd
|
|
102
|
+
unit drift and dir ownership, config + backup freshness, sqlite integrity,
|
|
103
|
+
spend telemetry, reporting timezones, Telegram health) and exits 0 only when
|
|
104
|
+
nothing failed.
|
|
105
|
+
|
|
143
106
|
From a checkout of the monorepo, `./setup.sh` checks both plugins. It preserves
|
|
144
107
|
an existing npm-managed `omp-conductor` and links only the Herdr half, so running
|
|
145
108
|
setup on a release-based fleet cannot silently switch omp to mutable source.
|
|
146
109
|
`./setup.sh install --force-link` is the explicit opt-in to link both checkout
|
|
147
110
|
directories.
|
|
148
111
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
Also required on the host:
|
|
155
|
-
|
|
156
|
-
- `bun`: the CLI and the daemon run on it (`Bun.serve` backs `/healthz`).
|
|
157
|
-
- **A model credential the *daemon's own account* can reach.** Sessions are child
|
|
158
|
-
processes of the daemon and inherit its environment and `$HOME` unmodified, so a
|
|
159
|
-
session authenticates with exactly what the daemon authenticates with — nothing
|
|
160
|
-
is injected and nothing is scrubbed. Any shape the harness itself understands
|
|
161
|
-
works, including the ordinary one:
|
|
162
|
-
- an **OAuth login** already recorded for that account (`omp` login state under
|
|
163
|
-
its `~/.omp/agent`). This is the common case and needs no configuration.
|
|
164
|
-
- a model **API key** in the daemon's environment (`ANTHROPIC_API_KEY`,
|
|
165
|
-
`OPENAI_API_KEY`, …). Note systemd starts the service with a clean
|
|
166
|
-
environment, so it has to be an `Environment=` line on the unit, not something
|
|
167
|
-
exported in your shell.
|
|
168
|
-
- the harness's **auth broker**, configured in that account's
|
|
169
|
-
`~/.omp/agent/config.yml`.
|
|
170
|
-
|
|
171
|
-
The account matters more than the shape: a login recorded under a *different*
|
|
172
|
-
account is invisible to the service. A unit running `User=fleet` cannot see
|
|
173
|
-
`root`'s login, and workers then die at turn 0 with `No model selected`. Such a
|
|
174
|
-
run is classified `env-start-failure` and charges **neither** the failure budget
|
|
175
|
-
nor a continuation — an environment fault is not a failed implementation, and a
|
|
176
|
-
run that recorded no turn, no commit and no error did not attempt anything — but
|
|
177
|
-
nothing dispatches successfully until the credential is reachable.
|
|
178
|
-
- `gh`, already authenticated: every tracker operation shells out to it, so the
|
|
179
|
-
daemon never handles a GitHub token itself.
|
|
180
|
-
- `git`: mirrors and worktrees.
|
|
181
|
-
- **[omp-telegram](https://www.npmjs.com/package/omp-telegram)**, for the
|
|
182
|
-
escalation channel. It is a separate package and is not vendored here.
|
|
183
|
-
|
|
184
|
-
Two different things depend on it, and they need different amounts of it:
|
|
185
|
-
|
|
186
|
-
- **Tier-2 paging** needs only its bot token. This package reads
|
|
187
|
-
`TELEGRAM_BOT_TOKEN` out of `$OMP_TELEGRAM_STATE_DIR/.env` (default
|
|
188
|
-
`~/.omp/agent/telegram/.env`) and posts to the chat id you configure. No
|
|
189
|
-
pairing required, and no token ever passes through this package's own config.
|
|
190
|
-
- **The interactive channel** — replying to an escalation, approving a brief
|
|
191
|
-
amendment from your phone — needs omp-telegram actually paired, which is what
|
|
192
|
-
writes `access.json`. The fleet heartbeat also reads that file and refuses to
|
|
193
|
-
tick unless exactly one owner is paired, on the grounds that unattended
|
|
194
|
-
dispatch is only defensible while a tier-2 page can reach a person.
|
|
195
|
-
- **Approving a Learning-loop amendment from a heartbeat tick** needs one more
|
|
196
|
-
setting than pairing: a notify destination. `/telegram notify` writes
|
|
197
|
-
`notifyMode` and `notifyChat` into the same `access.json`. omp-telegram
|
|
198
|
-
mounts its `telegram_ask` tool only for a turn that resolves a notify target,
|
|
199
|
-
and a locally injected tick resolves one only through that setting — so
|
|
200
|
-
without it the orchestrator can page you but cannot put a yes/no question in
|
|
201
|
-
front of you, which is the one thing the Learning loop's approval step
|
|
202
|
-
requires. `omp-conductor status` reports this on the `telegram` row, and a
|
|
203
|
-
tick that cannot ask says so in its own prompt and falls back to
|
|
204
|
-
`telegram_send`.
|
|
205
|
-
|
|
206
|
-
Set `notifyChat` even on a forum fleet. `/telegram topics` routes to a topic
|
|
207
|
-
this session claims at runtime, and that claim is not visible in
|
|
208
|
-
`access.json` — so a file carrying only `topicsChat` is reported as
|
|
209
|
-
unconfigured rather than guessed at, on the grounds that a health row which
|
|
210
|
-
reads green over a broken contract is worse than one that overstates a fault.
|
|
211
|
-
|
|
212
|
-
- **A fleet that answers instead of narrating** needs one key, set once:
|
|
213
|
-
`/telegram set profile daemon` (omp-telegram 0.11.0 or newer). Without it the
|
|
214
|
-
bridge behaves as it does on a laptop: it finalizes a real Telegram message
|
|
215
|
-
per assistant turn for as long as a conversation is active — so one answer
|
|
216
|
-
arrives as several messages, and a message that lands mid-tick keeps relaying
|
|
217
|
-
that tick's internal turns — and it posts every local run's closing text to
|
|
218
|
-
`notifyChat`, which on a host whose runs are heartbeat ticks means each tick's
|
|
219
|
-
working prose. The profile switches all of it off at the transport: text
|
|
220
|
-
reaches Telegram only through `telegram_send` / `telegram_ask`, the idle post
|
|
221
|
-
is suppressed, and `telegram_ask` stays mounted and aimed at the paired owner
|
|
222
|
-
on every turn — including a locally injected tick, so it also removes the need
|
|
223
|
-
for `notifyMode` above. Approval and blocked-input pings still fire; those
|
|
224
|
-
mean a human is needed, which is the point of the channel.
|
|
225
|
-
|
|
226
|
-
`omp-conductor status` reports an interactive profile on the `telegram` row,
|
|
227
|
-
and every tick composed on one carries a prompt line saying so. Note the two
|
|
228
|
-
settings pull against each other before the profile exists: setting
|
|
229
|
-
`notifyMode` to make `telegram_ask` mountable is exactly what arms the idle
|
|
230
|
-
post, so the correctly askable fleet was also the loud one.
|
|
231
|
-
|
|
232
|
-
With neither, tier 2 degrades to a comment on the issue. Nothing is broken in
|
|
233
|
-
that configuration: it is supported, just slower to reach you.
|
|
234
|
-
|
|
235
|
-
## Updating
|
|
236
|
-
|
|
237
|
-
Run one command from a shell outside the target `herdr-fleet.service`:
|
|
238
|
-
|
|
239
|
-
```bash
|
|
240
|
-
omp-conductor upgrade
|
|
241
|
-
```
|
|
242
|
-
|
|
243
|
-
It resolves the latest published npm release, pauses new claims, drains active
|
|
244
|
-
workers, and pins that exact release across the Bun-global CLI, omp plugin, and
|
|
245
|
-
Herdr plugin. It also recomposes the conductor-owned brief floor, restarts Herdr
|
|
246
|
-
and the daemon, waits for pane recovery, verifies the installed identities and
|
|
247
|
-
layered fleet status twice, then restores the original dispatch state.
|
|
248
|
-
|
|
249
|
-
Use `omp-conductor upgrade --to X.Y.Z` for an explicit published version. The
|
|
250
|
-
command exits without changing anything when all three surfaces already use that
|
|
251
|
-
release, the Herdr plugin is pinned to its exact `gitHead`, and the brief is
|
|
252
|
-
current.
|
|
253
|
-
|
|
254
|
-
The upgrade is host-wide, because everything it replaces is: one daemon serves
|
|
255
|
-
every configured project, so its restart lands on all of them at once. A bare
|
|
256
|
-
`omp-conductor upgrade` therefore drains **every** project's workers and
|
|
257
|
-
refreshes **every** project's brief, and pauses them with the fleet-wide
|
|
258
|
-
sentinel — which leaves any per-project `hold` you set standing when it
|
|
259
|
-
restores dispatch.
|
|
260
|
-
|
|
261
|
-
`--project` narrows that only when it is truthful to do so. When a live daemon
|
|
262
|
-
recorded a single project, the command drains and restarts that project, and an
|
|
263
|
-
explicit `--project` naming a different one is rejected before pause or
|
|
264
|
-
installation. When the daemon serves every configured project and there is more
|
|
265
|
-
than one, `--project` is rejected too: draining one queue and then restarting
|
|
266
|
-
the shared daemon would kill another queue's workers without ever counting
|
|
267
|
-
them. Re-run without the flag.
|
|
268
|
-
|
|
269
|
-
Ticks remain in their existing armed or disarmed state, so an ordinary update
|
|
270
|
-
does not halt the exact pane or require another Telegram arm challenge. Progress
|
|
271
|
-
names the Bun-global CLI, omp plugin, Herdr plugin, brief, reloads, and both
|
|
272
|
-
verification passes separately.
|
|
273
|
-
|
|
274
|
-
An installation, brief, reload, or verification failure pauses dispatch and
|
|
275
|
-
attempts to restore the exact CLI/plugin identities that were present before the
|
|
276
|
-
command. If rollback also fails, the error names every failed restoration and
|
|
277
|
-
keeps dispatch paused; it never brings a known mixed fleet back into service.
|
|
278
|
-
The command never publishes npm, edits an install root, or delegates lifecycle
|
|
279
|
-
steps to an AI session. It refuses to run inside a Herdr-managed session because
|
|
280
|
-
an updater that restarts itself cannot verify the result.
|
|
281
|
-
|
|
282
|
-
## Onboarding
|
|
283
|
-
|
|
284
|
-
`omp-conductor setup` is the whole of it. One command, in a plain terminal, doing
|
|
285
|
-
the two jobs onboarding has always had:
|
|
286
|
-
|
|
287
|
-
| Half | What it does |
|
|
288
|
-
| --- | --- |
|
|
289
|
-
| **The interview** | Asks the judgment no amount of repo reading produces, then writes it into `POLICY.md` as prose you own and can edit. |
|
|
290
|
-
| **The probes** | Reads your repos and *proposes* the rest — real CI gates, the project context, the release procedure — each a default you edit or a draft you confirm. |
|
|
291
|
-
|
|
292
|
-
The split matters because the two halves fail differently. A wrong config value is
|
|
293
|
-
a run that errors on the next tick; a wrong release boundary is a fleet that
|
|
294
|
-
publishes something at 03:00. The first is worth a validated prompt. The second is
|
|
295
|
-
worth being asked properly, which is why it is asked and never guessed.
|
|
296
|
-
|
|
297
|
-
### What only you can answer
|
|
298
|
-
|
|
299
|
-
Always asked: **where the roadmap lives and what the current priority is.** A
|
|
300
|
-
tracker shows what is *open*, never what *matters*, and an orchestrator that cannot
|
|
301
|
-
rank work grooms by recency — which is how a stale issue outranks the thing you are
|
|
302
|
-
shipping this month.
|
|
303
|
-
|
|
304
|
-
Asked only when you grant the orchestrator a release shape, because a
|
|
305
|
-
humans-release fleet has no boundary to draw:
|
|
306
|
-
|
|
307
|
-
- **Where the orchestrator's leg ENDS**, in one sentence. If it cannot be said in
|
|
308
|
-
one sentence it is not a boundary, and a vague release mandate is what eventually
|
|
309
|
-
publishes something at 03:00.
|
|
310
|
-
- **What** may be released and from which branch; **when** — batched how, after
|
|
311
|
-
which *named* checks; **what proof** must be held first, results actually read
|
|
312
|
-
rather than an impression; **what must still be asked** every time; and **what
|
|
313
|
-
stays permanently forbidden**.
|
|
314
|
-
- **What makes a release worth cutting** — a sprint, an epic's children all closed,
|
|
315
|
-
N merged issues. Without it the orchestrator either releases per merge, a stream
|
|
316
|
-
of meaningless versions burning shared runners, or never releases at all.
|
|
317
|
-
- **Who owns the rollback.** Name a person and setup says so plainly: that person
|
|
318
|
-
already owns the release, so the honest configuration ends the agent's leg
|
|
319
|
-
*before* the irreversible step. It offers to move the boundary there; declining
|
|
320
|
-
is a choice, not a mistake.
|
|
321
|
-
|
|
322
|
-
Grant every release shape and setup pushes back once — credentials sitting in the
|
|
323
|
-
environment of a session that runs unattended for weeks, and a 03:00 rollback being
|
|
324
|
-
a judgement call under time pressure with partial information — then records what
|
|
325
|
-
you decide. It is your fleet.
|
|
326
|
-
|
|
327
|
-
### What setup reads for you
|
|
328
|
-
|
|
329
|
-
Setup discovers factual defaults before it asks for them. `git remote get-url
|
|
330
|
-
origin` supplies the tracker repo and single-repo routing key; bounded `gh` calls
|
|
331
|
-
supply the default branch, existing queue/state labels, branch-protection checks,
|
|
332
|
-
environments, an unambiguous npm package name, and GitHub Projects/open milestones.
|
|
333
|
-
Each discovered value is shown with its evidence and remains editable at the same
|
|
334
|
-
prompt. Discovery only seeds a fresh interview: a re-run starts from the saved
|
|
335
|
-
project, so an operator-edited value is never guessed again.
|
|
336
|
-
|
|
337
|
-
Judgment and prose still belong to the existing confined model probes, and **every
|
|
338
|
-
answer is a proposal**:
|
|
339
|
-
|
|
340
|
-
- **Gates.** Reads each routing repo's CI workflows, `package.json` scripts and
|
|
341
|
-
`Makefile`/`justfile`, then pre-fills the [gates](#configuration) prompt with the
|
|
342
|
-
exact commands and the `cwd` each runs from, so your gates match what CI runs. It
|
|
343
|
-
reports the evidence it used, and an honest "this repo has no cheap local check"
|
|
344
|
-
is a real answer rather than an invented `npm test`.
|
|
345
|
-
- **Project context** and **the release procedure.** Drafted across *every* routing
|
|
346
|
-
repo — which repo owns which concern, which ship together, where the release
|
|
347
|
-
machinery actually lives — then shown to you in full and kept **only if you
|
|
348
|
-
confirm**. `POLICY.md` is re-read on every tick, so a paragraph you never read
|
|
349
|
-
would become an instruction the orchestrator follows all week.
|
|
350
|
-
|
|
351
|
-
A model probe has **no shell, no editor and no verbs**: it reads files and answers,
|
|
352
|
-
and a tool it was not given is refused rather than allowed. It is not a sandbox —
|
|
353
|
-
it runs as your own user and reads what you can read — which is why it is pointed
|
|
354
|
-
only at repos you configured yourself.
|
|
355
|
-
|
|
356
|
-
**Setup never fails because discovery or a probe did.** No `gh`, no auth or
|
|
357
|
-
network, a private repo, no omp peer, a clone failure, a timeout, or a malformed
|
|
358
|
-
reply each produces a warning and leaves the typed default in place. Every question
|
|
359
|
-
is still asked.
|
|
360
|
-
|
|
361
|
-
Skip the reading half entirely with `--no-ai`:
|
|
362
|
-
|
|
363
|
-
```bash
|
|
364
|
-
omp-conductor setup --no-ai
|
|
365
|
-
```
|
|
366
|
-
|
|
367
|
-
To fill in or revise just the brief later — the two `POLICY.md` sections above — run
|
|
368
|
-
the `brief` area, which re-asks the judgment questions and re-runs the probes:
|
|
369
|
-
|
|
370
|
-
```bash
|
|
371
|
-
omp-conductor setup brief
|
|
372
|
-
```
|
|
373
|
-
|
|
374
|
-
### Changing one setting
|
|
375
|
-
|
|
376
|
-
`config.json` is wizard-written, so changing a value means running the wizard —
|
|
377
|
-
and a wizard that re-asks twenty questions to add one key is a wizard people edit
|
|
378
|
-
the file behind instead. So a re-run against a project that is already configured
|
|
379
|
-
opens with one question:
|
|
380
|
-
|
|
381
|
-
```text
|
|
382
|
-
"platform" is already configured — what would you like to do?
|
|
383
|
-
> Change one area
|
|
384
|
-
asks one area's questions; every other answer is carried through from the saved config
|
|
385
|
-
Walk every question again
|
|
386
|
-
the full interview, every prompt pre-filled with what is configured now
|
|
387
|
-
Add another project
|
|
388
|
-
full interview for a new project; existing projects stay as they are
|
|
389
|
-
```
|
|
390
|
-
|
|
391
|
-
Amending is the default. Pick it and the eight areas are listed with what each one
|
|
392
|
-
says right now, so the row you want is the row you can see:
|
|
393
|
-
|
|
394
|
-
```text
|
|
395
|
-
Which area? Each row shows what it says now
|
|
396
|
-
tracker & repos — acme/platform, queue "ready-for-agent", "repo:" → platform, api, web, worker
|
|
397
|
-
gates — platform: bun run check; api: ruff check . @ backend; web: pnpm lint…
|
|
398
|
-
caps & worker model — 2 workers, 120 turns, 90m, $25/day, 2 attempts (all defaults) — harness default model
|
|
399
|
-
code graph — not configured — workers grep
|
|
400
|
-
authority — merge=orchestrator, release=orchestrator
|
|
401
|
-
escalation & triage — tier 2 pages Telegram 123456789, comments too, triage external
|
|
402
|
-
reporting scope — material — escalations, plus green PRs, second failures, and anything that stops the fleet
|
|
403
|
-
orchestrator brief — none at ~/.omp/conductor/worktrees/ORCHESTRATOR.md
|
|
404
|
-
```
|
|
405
|
-
|
|
406
|
-
Only that area's questions are asked. Every other answer is read back out of
|
|
407
|
-
`config.json` and written again unchanged — the same answers, the same builder,
|
|
408
|
-
the same single confirm, so there is still exactly one thing in this package that
|
|
409
|
-
writes a config, and it still writes nothing before you agree. The consent screen
|
|
410
|
-
leads with the delta and then shows the whole project as it would be written:
|
|
411
|
-
|
|
412
|
-
```text
|
|
413
|
-
amending code graph — project platform
|
|
414
|
-
was not configured — workers grep
|
|
415
|
-
now ~/.cache/conductor-graph/acme — 4 clone(s): platform, api, web, worker
|
|
416
|
-
carried over tracker & repos, gates, caps & worker model, authority, escalation & triage, reporting scope, orchestrator brief
|
|
417
|
-
read back from ~/.omp/conductor/config.json and rewritten unchanged
|
|
418
|
-
```
|
|
419
|
-
|
|
420
|
-
A first run, or a project name this config has never seen, never sees either
|
|
421
|
-
question: there is nothing to amend, so it is the full interview exactly as
|
|
422
|
-
before. Choosing *Walk every question again* asks once to confirm the replace
|
|
423
|
-
(so a silent overwrite cannot happen from muscle-memory Enter) — every prompt
|
|
424
|
-
pre-filled with what is configured, Enter to keep it — with one wrinkle worth
|
|
425
|
-
knowing: the two authority confirms and the orchestrator-session confirm cannot
|
|
426
|
-
start on "yes", so Entering through the full interview **revokes** a delegation
|
|
427
|
-
rather than renewing it. Amending the `authority` area names the current grant in
|
|
428
|
-
the question, which is the safer way to leave one alone.
|
|
429
|
-
|
|
430
|
-
### Adding another project
|
|
431
|
-
|
|
432
|
-
One daemon serves every configured project. To put a second (or third) fleet on
|
|
433
|
-
the same host without touching the first:
|
|
434
|
-
|
|
435
|
-
```bash
|
|
436
|
-
omp-conductor setup --project second
|
|
437
|
-
# or pick "Add another project" from the re-run chooser
|
|
438
|
-
```
|
|
439
|
-
|
|
440
|
-
That is a full interview for the new name only. Defaults land under
|
|
441
|
-
`~/.omp/conductor/projects/<name>/{worktrees,mirrors}` so two fleets never share
|
|
442
|
-
a cwd; the first project's existing flat `worktrees`/`mirrors` paths are never
|
|
443
|
-
migrated. A `workspaceRoot` that collides with another project is refused with
|
|
444
|
-
both names in the error. Re-using an existing name asks amend-or-replace before
|
|
445
|
-
anything is written.
|
|
446
|
-
|
|
447
|
-
After apply, setup provisions labels, brief, tick config (with `project` +
|
|
448
|
-
`agentName`), topic binding, smoke, and arm for the new project only, then prints:
|
|
449
|
-
|
|
450
|
-
- `omp-conductor restart --now` — the running daemon picks up the new project
|
|
451
|
-
only after reload (printed, not auto-run when workers are live)
|
|
452
|
-
- a copy-pasteable **herdr handoff**: `herdr --session conductor workspace create
|
|
453
|
-
--cwd <workspaceRoot> --label <project> --no-focus`, then `herdr --session
|
|
454
|
-
conductor agent start <project> --kind omp --pane <pane-id>` into that empty
|
|
455
|
-
pane (never into a live orchestrator), plus the `FLEET_CWDS` list for
|
|
456
|
-
multi-fleet recovery
|
|
457
|
-
|
|
458
|
-
`omp-conductor setup gates --project second` (and every other area) still amends
|
|
459
|
-
only that project.
|
|
460
|
-
|
|
461
|
-
### Keeping a brief current
|
|
462
|
-
|
|
463
|
-
The standing prompt is two layers:
|
|
464
|
-
|
|
465
|
-
| Layer | File | Updates how? |
|
|
466
|
-
| --- | --- | --- |
|
|
467
|
-
| Package floor | `src/briefs/orchestrator.md` | Every tick recomposes it into `ORCHESTRATOR.md` from the installed package. Upgrade the package in this host's existing install root + restart is enough. |
|
|
468
|
-
| Fleet policy | `POLICY.md` | Yours. Setup writes the scaffold once; the Learning loop edits only this file. |
|
|
469
|
-
| Composed view | `ORCHESTRATOR.md` | Regenerated from floor + `POLICY.md` on each tick (and at setup). Do not hand-amend it for durable policy. |
|
|
470
|
-
| Worker brief | `src/briefs/worker.md` | Read per run from the package. |
|
|
471
|
-
|
|
472
|
-
```bash
|
|
473
|
-
omp-conductor brief-upgrade # report overlay / legacy state
|
|
474
|
-
omp-conductor brief-upgrade --migrate # dry-run: bannered ORCHESTRATOR.md → POLICY.md
|
|
475
|
-
omp-conductor brief-upgrade --migrate --apply
|
|
476
|
-
omp-conductor brief-upgrade --retrofit # #20: propose YOURS TO EDIT cut on a hand-written brief
|
|
477
|
-
omp-conductor brief-upgrade --retrofit --apply
|
|
478
|
-
```
|
|
479
|
-
|
|
480
|
-
- **Overlay already active** (`POLICY.md` present): protocol updates need no brief-upgrade.
|
|
481
|
-
- **Legacy bannered brief**: `--migrate` lifts the owned half into `POLICY.md`
|
|
482
|
-
and recomposes. Previous brief and policy versions go to
|
|
483
|
-
`$OMP_CONDUCTOR_HOME/backups/briefs/` (default
|
|
484
|
-
`~/.omp/conductor/backups/briefs/`), named with their source filename and
|
|
485
|
-
timestamp.
|
|
486
|
-
- **Hand-written brief** (no banner): `--retrofit` inserts the banner before the first Releases / Project context / Reporting / Amendments heading; then `--migrate`.
|
|
487
|
-
- **The legacy single-file merge is gone** (0.4.3). A bare `--apply` exits `2`
|
|
488
|
-
naming the two paths that remain, rather than rewriting a brief nobody asked
|
|
489
|
-
it to. `--migrate` is the cross-version ABI: an upgrade keeps calling it, and
|
|
490
|
-
an overlay fleet tolerates its absence because the floor recomposes each tick.
|
|
491
|
-
- **`status` names the layout**, so a fleet still on a legacy brief is visible
|
|
492
|
-
where an operator already looks: `brief overlay (package floor + POLICY.md)`,
|
|
493
|
-
or `brief legacy-bannered — run omp-conductor brief-upgrade`.
|
|
494
|
-
- **Existing sidecars**: a composed refresh relocates conductor-generated
|
|
495
|
-
`ORCHESTRATOR.md.bak-<timestamp>` and `POLICY.md.bak-<timestamp>` files into
|
|
496
|
-
that backup directory. Other `.bak` files stay untouched.
|
|
497
|
-
|
|
498
|
-
`--file PATH` checks a brief that is not where the wizard would have put it.
|
|
499
|
-
|
|
500
|
-
The **Learning loop** proposes diffs against `POLICY.md` for you to approve over
|
|
501
|
-
Telegram. It also learns from repeated operational friction. The daemon
|
|
502
|
-
automatically rolls up repairable admission holds; the orchestrator records
|
|
503
|
-
judgments code cannot make with:
|
|
504
|
-
|
|
505
|
-
```bash
|
|
506
|
-
omp-conductor friction escalation-digest --detail "routine retry belonged in the digest" [--issue N]
|
|
507
|
-
omp-conductor friction report-noise --detail "green status repeated with no operator action"
|
|
508
|
-
omp-conductor friction report-surprise --detail "a material failure was missing from the report"
|
|
509
|
-
```
|
|
510
|
-
|
|
511
|
-
Three observations within seven days make a bounded signal eligible for one
|
|
512
|
-
tick. After it is surfaced, that signal cools down for seven days. A signal is
|
|
513
|
-
evidence to investigate, never an automatic policy edit: the existing one-at-a-
|
|
514
|
-
time Telegram approval, `POLICY.md`-only edit, Hard-boundary prohibition, and
|
|
515
|
-
**Amendments** log still apply.
|
|
112
|
+
For what the host needs before anything runs — `bun`, an authenticated `gh`,
|
|
113
|
+
`git`, a model credential the daemon's own account can reach, and
|
|
114
|
+
`omp-telegram` for the escalation channel — see the
|
|
115
|
+
[Install prerequisites](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#install-prerequisites) in the reference.
|
|
516
116
|
|
|
517
117
|
## Quick start
|
|
518
118
|
|
|
@@ -561,6 +161,19 @@ time Telegram approval, `POLICY.md`-only edit, Hard-boundary prohibition, and
|
|
|
561
161
|
|
|
562
162
|
The result must show a running daemon, a healthy `/healthz`, armed ticks for external orchestration, and the configured project.
|
|
563
163
|
|
|
164
|
+
6. Verify the deployment mechanically before trusting it:
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
omp-conductor doctor
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
`doctor` runs read-only and exits 0 only when nothing failed: gh auth and
|
|
171
|
+
scopes, exact-case labels, installed-systemd-unit drift, runtime-dir
|
|
172
|
+
ownership, config + backup freshness, sqlite integrity, spend telemetry,
|
|
173
|
+
reporting timezones, and Telegram health. Run it again after every
|
|
174
|
+
`omp-conductor upgrade` — every one of these failures has cost a debugging
|
|
175
|
+
session silently, and each has a one-line fix in its finding.
|
|
176
|
+
|
|
564
177
|
### First worker drill
|
|
565
178
|
|
|
566
179
|
Use a disposable target repository for this drill. Replace the values below with labels that the wizard showed.
|
|
@@ -594,6 +207,25 @@ The package also ships a generic unit at
|
|
|
594
207
|
[`systemd/omp-conductor.service.example`](systemd/omp-conductor.service.example).
|
|
595
208
|
|
|
596
209
|
|
|
210
|
+
## Operating the fleet
|
|
211
|
+
|
|
212
|
+
The fleet is a long-lived orchestrator session plus the dispatch daemon behind
|
|
213
|
+
it. These are the verbs you reach for day to day.
|
|
214
|
+
|
|
215
|
+
### The fleet host
|
|
216
|
+
|
|
217
|
+
Both halves point at the same 24/7 omp session on your always-on host. The
|
|
218
|
+
session's working directory is configured as `FLEET_CWD`; it contains
|
|
219
|
+
`.conductor-tick.json`, whose relative `armedFile` also resolves from that
|
|
220
|
+
directory. Conductor state defaults to `~/.omp/conductor`, while `workspaceRoot`
|
|
221
|
+
defaults to its `worktrees/` directory. Setup composes the package floor and the
|
|
222
|
+
fleet-owned `POLICY.md` into `ORCHESTRATOR.md` there. No `/root` layout is built
|
|
223
|
+
into a new deployment.
|
|
224
|
+
|
|
225
|
+
The Herdr half owns recovery, not dispatch or policy: it restores the exact
|
|
226
|
+
session identity, requests an immediate heartbeat, or reports through Telegram
|
|
227
|
+
and a Herdr notification that the fleet is down.
|
|
228
|
+
|
|
597
229
|
### Stop the conductor (hold / stop)
|
|
598
230
|
|
|
599
231
|
Two words, and one of them takes a flag:
|
|
@@ -607,13 +239,13 @@ Two words, and one of them takes a flag:
|
|
|
607
239
|
|
|
608
240
|
`resume` clears pause **and** any `stop --pane` recovery pin, and **never re-arms**. `arm` is proof-gated: it sends a Telegram challenge and writes the arm marker only after your reply appears as a *user* turn in the orchestrator transcript. `stop --pane` targets the configured conductor agent only — it does **not** run `systemctl stop herdr-fleet`. To bounce the daemon without stopping the fleet, use `restart`.
|
|
609
241
|
|
|
610
|
-
**Removed in 0.15.0**, each exiting `2` with a pointer: `halt` (now `stop`), `pause` (use `hold`), `release-pane` (now part of `resume`), and `graph-setup` (now [`setup graph`](#code-graph-discovery)). Dropping `pause` cost one real capability, "stop claiming but keep ticking", which is now `hold --keep-ticks` rather than a fifth verb. It matters because disarming is the expensive half of a hold: re-arming sends a Telegram challenge and blocks until you answer it in the chat, so stopping claims for ten minutes otherwise costs a manual round trip to get the heartbeat back — while the workers a hold deliberately leaves running have nothing shepherding them.
|
|
242
|
+
**Removed in 0.15.0**, each exiting `2` with a pointer: `halt` (now `stop`), `pause` (use `hold`), `release-pane` (now part of `resume`), and `graph-setup` (now [`setup graph`](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#code-graph-discovery)). Dropping `pause` cost one real capability, "stop claiming but keep ticking", which is now `hold --keep-ticks` rather than a fifth verb. It matters because disarming is the expensive half of a hold: re-arming sends a Telegram challenge and blocks until you answer it in the chat, so stopping claims for ten minutes otherwise costs a manual round trip to get the heartbeat back — while the workers a hold deliberately leaves running have nothing shepherding them.
|
|
611
243
|
|
|
612
244
|
`status` prints a layered header (`dispatch` / `ticks` / next tick time / `pane` / `recovery` / `herdr` / `telegram` / `daemon`) so a paused fleet cannot hide an armed orchestrator still spending turns. The Telegram line calls the official `getMe` endpoint to prove the token and API are usable without sending a message, then separately reports whether the inbound bridge is configured.
|
|
613
245
|
|
|
614
246
|
When the tracker is behind what the store decided, `status` adds a
|
|
615
247
|
`labels projection N pending (oldest …)` row: those are label
|
|
616
|
-
changes the dispatcher committed to and the [projector](#
|
|
248
|
+
changes the dispatcher committed to and the [projector](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#how-one-tick-works) has not been
|
|
617
249
|
able to apply yet (a 403, a rate limit). The intended label state is durable, so
|
|
618
250
|
nothing is lost and safety is preserved: for an issue the queue read already
|
|
619
251
|
returns, a pending *state-label removal* is overlaid as applied, so a stale
|
|
@@ -693,2176 +325,273 @@ including when that tick config is the thing that failed to parse.
|
|
|
693
325
|
Clear the pin with `omp-conductor resume` when you want recovery again.
|
|
694
326
|
|
|
695
327
|
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
|
|
710
|
-
|
|
711
|
-
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
|
|
728
|
-
|
|
729
|
-
|
|
730
|
-
|
|
731
|
-
|
|
732
|
-
|
|
733
|
-
|
|
734
|
-
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
749
|
-
|
|
750
|
-
|
|
751
|
-
11. **Record the pass.** Persist ready/routed/admitted counts and group every hold
|
|
752
|
-
under a stable reason code with at most five sample issue numbers. Tracker
|
|
753
|
-
failures mark the summary `DEGRADED`; capacity, sibling, open-PR and budget
|
|
754
|
-
holds remain normal policy state.
|
|
755
|
-
12. **Dispatch** the admitted issues concurrently.
|
|
756
|
-
|
|
757
|
-
Then, per admitted issue:
|
|
758
|
-
|
|
759
|
-
1. **Create the run row (`claimed`) — before any worktree or session exists.**
|
|
760
|
-
This ordering is the whole crash-safety story: the *row*, not a label, is the
|
|
761
|
-
guard against dispatching the same issue twice. It is local and written before
|
|
762
|
-
anything that can fail; if the process dies at any later point, the startup
|
|
763
|
-
orphan sweep marks the left-behind row `orphaned` and the orchestrator's drain
|
|
764
|
-
duty triages it (see below). The `agent:in-progress` label is a write-behind
|
|
765
|
-
projection of that row — enqueued in the same breath and applied to the tracker
|
|
766
|
-
by the projector with retry — so even a tracker that refuses the write cannot
|
|
767
|
-
recreate a duplicate PR while the guard is unavailable.
|
|
768
|
-
2. Run the tick's post-admission projection pass, which applies freshly enqueued
|
|
769
|
-
label ops on the healthy path.
|
|
770
|
-
3. Clear any stale tree for this issue, then add a fresh worktree at
|
|
771
|
-
`<workspaceRoot>/<issue>` cut from the bare mirror at `<mirrorRoot>/<repo>.git`,
|
|
772
|
-
on the run's branch off the repo's default branch.
|
|
773
|
-
4. Allocate a session transcript under `<state dir>/sessions/`, one per attempt,
|
|
774
|
-
and move the run to `running`. The run record keeps the exact path and a
|
|
775
|
-
failure escalation quotes it, so you can read what the worker actually did.
|
|
776
|
-
5. Run one omp session with the rendered brief, under the turn and wall-clock caps.
|
|
777
|
-
6. Record the outcome:
|
|
778
|
-
|
|
779
|
-
| Outcome | Labels | Worktree | Escalation |
|
|
780
|
-
| --- | --- | --- | --- |
|
|
781
|
-
| `pushed-pending` | `agent:in-progress` stays while the daemon rechecks GitHub | removed | none |
|
|
782
|
-
| `pushed-green` | `agent:in-progress` stays while the PR is open | removed | none |
|
|
783
|
-
| `blocked` | swapped to `agent:blocked` | dirty tree committed to the branch, then removed | Tier 1 |
|
|
784
|
-
| `failed` / `killed` | swapped to `agent:failed` | dirty tree committed to the branch, then retained until the PR or issue is terminal | Tier 1 |
|
|
785
|
-
| unexpected error | swapped to `agent:failed` | same | Tier 1 |
|
|
786
|
-
|
|
787
|
-
`pushed-pending` and `pushed-green` are not the end of the row: later ticks
|
|
788
|
-
verify outstanding checks and settle the PR once it resolves, and a row that
|
|
789
|
-
settles gives up its `agent:in-progress` label. See
|
|
790
|
-
[what settles a green PR](#what-settles-a-green-pr).
|
|
791
|
-
|
|
792
|
-
Every label write the dispatcher makes goes through the label projection
|
|
793
|
-
outbox and is applied to the tracker by the projector with retry, strictly in
|
|
794
|
-
the order it was enqueued per issue — a later op for one issue never lands
|
|
795
|
-
before an earlier one that is still owed. A state-label swap on a dispatch
|
|
796
|
-
outcome enqueues the new label ahead of the old one's removal, so the issue
|
|
797
|
-
is never briefly bare (the shape eligibility reads as fresh work), while a
|
|
798
|
-
requeue (`swapToQueue`, `unblock`) enqueues its removals ahead of the queue
|
|
799
|
-
add for the same reason in reverse: the issue must not look claimable before
|
|
800
|
-
its stale state label is gone. A refused write is deferred with backoff
|
|
801
|
-
instead of dropped, and `status` shows any un-applied lag on a
|
|
802
|
-
`labels projection` row.
|
|
803
|
-
|
|
804
|
-
**Every continuable end salvages the tree first.** A turns-cap kill, a
|
|
805
|
-
wall-clock kill, a crash and a graceful block all leave a tree the next
|
|
806
|
-
attempt removes `--force` — only the run's *branch* is preserved across
|
|
807
|
-
attempts. So before the escalation is written, a dirty tree is committed to
|
|
808
|
-
the run's own branch as `wip(#<issue>): attempt <n> <ending> — auto-salvaged`
|
|
809
|
-
(everything, including files git has never seen) and pushed, and the
|
|
810
|
-
escalation says where it went: `WIP committed to <branch> @ <sha>`. A push
|
|
811
|
-
that is refused leaves the commit in this host's mirror and says so.
|
|
812
|
-
|
|
813
|
-
Blocking was excluded from this until #118, on the argument that a worker
|
|
814
|
-
which stops on purpose has turns left to commit for itself. It cost a
|
|
815
|
-
34-file refactor: the worker blocked to ask whether a failing test was
|
|
816
|
-
obsolete — which is precisely a worker declining to commit a half-migrated
|
|
817
|
-
tree — and the daemon removed the tree seconds later, leaving the run branch
|
|
818
|
-
and `origin/main` on the same commit. A `pushed-green` or `pushed-pending`
|
|
819
|
-
run is now the only end that does not salvage: its deliverable is already on
|
|
820
|
-
a remote branch, and appending a WIP commit would turn the PR the daemon
|
|
821
|
-
just verified red.
|
|
822
|
-
|
|
823
|
-
**A salvage that fails keeps the tree and stops the issue.** If git refuses
|
|
824
|
-
the commit, the worktree is the only copy in existence, so it is retained
|
|
825
|
-
whatever the run's outcome was, the row records the failure, and the issue
|
|
826
|
-
is held out of dispatch with the `unsalvaged-wip` reason — because claiming
|
|
827
|
-
it is what would finally destroy the tree. `status` shows it under `wip` as
|
|
828
|
-
`UNSALVAGED`, and `omp-conductor unblock <n>` refuses. Recover the tree by
|
|
829
|
-
hand, then `unblock <n> --force` records that you accepted it and releases
|
|
830
|
-
the hold.
|
|
831
|
-
|
|
832
|
-
**A preserved tip is named to the next worker.** The sha is written to the
|
|
833
|
-
run row, shown by `status` and the board, and the continuation brief tells
|
|
834
|
-
the resuming worker the exact commit it is building on and that it is the
|
|
835
|
-
only copy.
|
|
836
|
-
|
|
837
|
-
Later ticks reap retained failure trees in bounded batches after the tracker
|
|
838
|
-
proves their PR merged/closed or their issue closed, provided no live run or
|
|
839
|
-
queued continuation owns the issue. Cleanup fetches remote refs first and
|
|
840
|
-
keeps any dirty tree or branch with uniquely local commits. Only then does it
|
|
841
|
-
remove the physical tree, prune registrations, and delete the obsolete local
|
|
842
|
-
mirror branch. Unknown tracker, network, repo, or git state is a no-op.
|
|
843
|
-
|
|
844
|
-
### What a restart does to runs that were in flight
|
|
845
|
-
|
|
846
|
-
A `claimed` or `running` row is a promise that a worker process exists, and a
|
|
847
|
-
daemon that just started knows that promise is broken: its workers died with the
|
|
848
|
-
previous process. At startup — unless another daemon is alive, so a foreground
|
|
849
|
-
`daemon --once` cannot orphan a running daemon's real workers — every such row is
|
|
850
|
-
**salvaged first** (dirty tree → `wip(#N): attempt N killed by a daemon restart —
|
|
851
|
-
auto-salvaged` on the run's branch, same path as a turns-cap kill), then moved to
|
|
852
|
-
`orphaned`, with a log line naming the issue, the attempt and the worktree. That
|
|
853
|
-
frees the slots immediately; a fleet must never resume as deadlocked as it
|
|
854
|
-
crashed, and uncommitted edits must not wait for a human with `bun -e`.
|
|
855
|
-
|
|
856
|
-
Only the rows change after salvage. The issue keeps `agent:in-progress` — the
|
|
857
|
-
label is the crash guard against double-dispatch — and deciding what the dead
|
|
858
|
-
worker's remains are worth is the orchestrator's drain-duty judgement, spelled
|
|
859
|
-
out in its brief: an open green PR goes to the merge path, a salvaged sha is a
|
|
860
|
-
continuation hand-off, and a clean orphan has its label released so the next
|
|
861
|
-
tick re-claims it. Orphans consume `maxContinuationsPerIssue`, not failed
|
|
862
|
-
implementation attempts, so crashes cannot starve the retry needed for a real
|
|
863
|
-
code or CI failure — and a crash loop still escalates.
|
|
864
|
-
|
|
865
|
-
### Deploying a new package onto a busy fleet
|
|
866
|
-
|
|
867
|
-
`systemctl restart` / `omp-conductor restart` is safe for **work product** once
|
|
868
|
-
this version is installed: startup salvage commits dirty trees before orphaning
|
|
869
|
-
rows, and salvage rewrites the mirror's managed `info/exclude` to the package's
|
|
870
|
-
current list before `git add` so a narrowed ignore cannot hide deliverables.
|
|
871
|
-
|
|
872
|
-
It is still disruptive for **in-flight sessions** because the worker process dies
|
|
873
|
-
and the attempt is spent. Update through the lifecycle command rather than
|
|
874
|
-
hand-installing or restarting individual surfaces:
|
|
875
|
-
|
|
876
|
-
```bash
|
|
877
|
-
omp-conductor upgrade
|
|
878
|
-
```
|
|
879
|
-
|
|
880
|
-
It pauses new claims, drains workers, installs one pinned release across all
|
|
881
|
-
surfaces, restarts, verifies twice, and resumes only if dispatch was initially
|
|
882
|
-
running. If an install, restart, or verification step fails, dispatch remains
|
|
883
|
-
paused and the command exits nonzero.
|
|
884
|
-
|
|
885
|
-
Do **not** edit files under the running install and expect the daemon to keep
|
|
886
|
-
dispatching — the integrity tripwire pauses and pages. Upgrade by whole release
|
|
887
|
-
so the new process records a fresh baseline.
|
|
888
|
-
|
|
889
|
-
### What settles a green PR
|
|
890
|
-
|
|
891
|
-
The worker watches CI, then reports the PR URL and the exact remote head SHA it
|
|
892
|
-
observed. The daemon independently reads the PR again and requires it to be open,
|
|
893
|
-
non-draft, still at that head, and backed by a non-empty check rollup in which
|
|
894
|
-
every check succeeded or was skipped. Missing or nonterminal checks become
|
|
895
|
-
`pushed-pending` and are rechecked on later ticks; red or cancelled checks become
|
|
896
|
-
`failed` with a bounded job/log digest. Only verified evidence becomes
|
|
897
|
-
`pushed-green`.
|
|
898
|
-
If a failed report mentions PRs only in prose, the daemon retains the last URL
|
|
899
|
-
whose `owner/repo` matches the run's repository; links to other repositories are
|
|
900
|
-
ignored. This preserves the continuation target without trusting an unrelated
|
|
901
|
-
PR mentioned in the same report.
|
|
902
|
-
|
|
903
|
-
What happens after verification is a human's decision, taken minutes to days
|
|
904
|
-
later and never announced to the daemon — so every tick asks the tracker about
|
|
905
|
-
every pushed PR it is still holding:
|
|
906
|
-
|
|
907
|
-
| PR | Row becomes | Why |
|
|
908
|
-
| --- | --- | --- |
|
|
909
|
-
| merged | `merged` | The work landed. This is the state `merged` was reserved for. |
|
|
910
|
-
| closed without merging | `failed`, class `returned-for-revision`, with the PR in `lastError` | A human read the work and asked for another pass. Leaving it `pushed-green` strands the issue forever behind a PR nobody will merge, and calling it `merged` is a lie about code that is not on the base branch. `failed` is the honest row state; the class preserves the review decision and releases the issue so a re-queue can be attempted again. |
|
|
911
|
-
| still open | unchanged | The normal steady state. Its issue must stay occupied, or a second attempt lands on the live PR. |
|
|
912
|
-
| could not be determined | unchanged | A flaky network, a revoked token, a deleted PR. An unknown answer never settles a row; the next tick asks again for free. |
|
|
913
|
-
|
|
914
|
-
Run history is untouched. A PR closed without merging consumes one continuation,
|
|
915
|
-
not a failed implementation attempt; a merge spends neither budget. A settled
|
|
916
|
-
row also loses `agent:in-progress` from its issue: the row transition and label
|
|
917
|
-
removal are one fact, and a terminal answer about the PR proves no worker
|
|
918
|
-
process owns the issue,
|
|
919
|
-
so the duplicate-dispatch guard it exists for is spent. The removal is *enqueued
|
|
920
|
-
on the label projection outbox in the same breath as the terminal write* — a
|
|
921
|
-
durable local write that cannot fail on the tracker — so the row settles at once
|
|
922
|
-
and the projector applies the label with retry. A tracker that refuses the
|
|
923
|
-
removal can no longer strand it: the op stays owed, eligibility overlays the
|
|
924
|
-
pending removal so the issue is not held back by a label that is already decided
|
|
925
|
-
gone, and `status` shows the lag. Anything beyond that one release — a re-queue,
|
|
926
|
-
a `blocked` marker — is still the orchestrator's drain-duty judgement. One
|
|
927
|
-
unreachable PR costs its own row and nothing else; the rest of the sweep still
|
|
928
|
-
settles.
|
|
929
|
-
|
|
930
|
-
Until this existed, nothing ever revisited a `pushed-green` row: the startup
|
|
931
|
-
reconciler only settles rows that held a process, and `merged` went unwritten. On
|
|
932
|
-
2026-08-07 the reference fleet reported three active runs whose PRs were all
|
|
933
|
-
merged and whose issues were all closed, through two daemon restarts — and because
|
|
934
|
-
the active set *is* the busy set, those three issues were permanently unclaimable.
|
|
935
|
-
A status page that has stopped being evidence is worse than no status page.
|
|
936
|
-
|
|
937
|
-
The label half of that outlived the row half by two days. On 2026-08-09 a merged
|
|
938
|
-
PR and a closed-unmerged one both settled their rows correctly and both left their
|
|
939
|
-
issues carrying `agent:in-progress`, which eligibility reads as "a worker owns
|
|
940
|
-
this" — with the brief forbidding the orchestrator from editing a state label and
|
|
941
|
-
`unblock` declining to clear that one, neither issue could ever be claimed again.
|
|
942
|
-
|
|
943
|
-
### Base-branch health after merge
|
|
944
|
-
|
|
945
|
-
The daemon records two different facts after a merge:
|
|
946
|
-
|
|
947
|
-
- The **post-merge audit** attributes a regression to one merge. For up to 24
|
|
948
|
-
hours, it checks only push-triggered workflow runs for the exact merge SHA and
|
|
949
|
-
base branch. A new red result adds `base-branch-red` evidence and escalates.
|
|
950
|
-
- **Current health** drives `status` and release policy. On every sweep, the
|
|
951
|
-
daemon resolves the live head of each branch it merged into during the last
|
|
952
|
-
seven days, then reads only push-triggered runs for that head and branch. The
|
|
953
|
-
status row includes the head SHA and run count.
|
|
954
|
-
|
|
955
|
-
Current health is `green` only when every observed run completed successfully,
|
|
956
|
-
neutrally, or skipped. A failing conclusion is `red`; an in-progress or
|
|
957
|
-
unrecognised conclusion is `pending`; and a head with no push-triggered run is
|
|
958
|
-
`unknown`, never green. Pending and unknown heads are rechecked. Green and red
|
|
959
|
-
heads are read again when the branch moves, so an old verdict cannot describe a
|
|
960
|
-
new commit. If GitHub cannot return the head or its runs, the daemon keeps the
|
|
961
|
-
last honest row instead of replacing evidence with a network failure.
|
|
962
|
-
|
|
963
|
-
The `base-branch-green` release requirement reads this current live-head row for
|
|
964
|
-
the repository being released. Red, pending, unknown, and absent evidence all
|
|
965
|
-
refuse the release.
|
|
966
|
-
|
|
967
|
-
### The settlement audit
|
|
968
|
-
|
|
969
|
-
Verifying `state: pushed-green` left two lines of the same report still taken on
|
|
970
|
-
faith. The worker brief asks for them and, until this existed, nothing read them:
|
|
971
|
-
|
|
972
|
-
```
|
|
973
|
-
gates: <exact commands run and their results>
|
|
974
|
-
changed: <files touched, one line>
|
|
975
|
-
```
|
|
976
|
-
|
|
977
|
-
So at settlement the daemon fetches the pull request's diff and checks the report
|
|
978
|
-
against it. What it finds is a **settlement audit flag** — advisory, never a
|
|
979
|
-
gate. A flagged run settles exactly as an unflagged one does; nothing here can
|
|
980
|
-
change a run's state, hold a merge, or spend an attempt.
|
|
981
|
-
|
|
982
|
-
| Flag | Raised when |
|
|
983
|
-
| --- | --- |
|
|
984
|
-
| `undisclosed-file` | The PR touched a file the `changed:` line never named. Lockfiles are exempt — they are derived from a manifest the report did disclose. |
|
|
985
|
-
| `changed-line-missing` | The report had no usable `changed:` line at all. One flag, not one per file. |
|
|
986
|
-
| `unmatched-claim` | `changed:` named a path the PR never touched. The weaker direction, and reported as such. |
|
|
987
|
-
| `report-format-unparsed` | The same file appeared as both claimed-but-untouched and touched-but-unclaimed, so the `changed:` line's format defeated the parser. Read the diff directly; this is not a trust signal against the worker. |
|
|
988
|
-
| `test-file-deleted` | A test file left the tree with no rename to account for it. |
|
|
989
|
-
| `test-disabled` | A `.skip` / `.only` / `xit` / `@pytest.mark.skip` / `t.Skip` marker appears on a line the PR added. |
|
|
990
|
-
| `assertions-removed` | An assertion was commented out, or a test file lost more assertions than it gained. |
|
|
991
|
-
| `test-timeout-raised` | A named timeout in a test file went up — compared against its own previous value, so a brand-new timeout is not a finding. |
|
|
992
|
-
|
|
993
|
-
A flag on a test file the dispatching issue never names is marked
|
|
994
|
-
`[unattributed]`: that is the "don't weaken tests you didn't write" case, and it
|
|
995
|
-
is the one worth reading first.
|
|
996
|
-
|
|
997
|
-
Flags reach you three ways: appended to the settlement report, stored on the run
|
|
998
|
-
row and shown under the run in `omp-conductor status` for as long as its PR is
|
|
999
|
-
open, and — once per flagged settlement — as a tier-1 escalation to the
|
|
1000
|
-
orchestrator, whose brief says what judgement each flag invites.
|
|
1001
|
-
|
|
1002
|
-
**A clean, accurately reported PR produces nothing.** That is a design
|
|
1003
|
-
constraint, not an aspiration: an audit that fires on honest work gets muted, and
|
|
1004
|
-
a muted audit is worse than none because the fleet still believes it is being
|
|
1005
|
-
checked. Every rule resolves ambiguity towards silence, and each accepts a named
|
|
1006
|
-
blind spot to stay quiet — a renamed test file is not a deleted one (even when
|
|
1007
|
-
git did not detect the rename, matched by basename), a `.skip` inside a string
|
|
1008
|
-
literal or a recorded fixture is not a skip, and a rewritten test that keeps its
|
|
1009
|
-
coverage is not a weakening.
|
|
1010
|
-
|
|
1011
|
-
The analyser is pure: it takes a parsed diff, the report and the issue text, and
|
|
1012
|
-
returns flags. Only `Tracker.prDiff` touches the network, and a diff it cannot
|
|
1013
|
-
read produces no flags *and says so* — silence about a diff nobody read is not a
|
|
1014
|
-
clean bill.
|
|
1015
|
-
|
|
1016
|
-
### Continuation runs
|
|
1017
|
-
|
|
1018
|
-
When a worktree is provisioned onto a branch that already exists in the mirror
|
|
1019
|
-
(reattach after a prior attempt, orphan, or turns-cap auto-requeue), the worker
|
|
1020
|
-
brief includes a **Continuation** section: read `git log` / `git diff` against
|
|
1021
|
-
the default branch first, and do not recreate work already on the branch.
|
|
1022
|
-
|
|
1023
|
-
A **turns-cap kill with attempts remaining** salvages the tree, puts the queue
|
|
1024
|
-
label back on, and skips the failed label so the next tick reclaims as a
|
|
1025
|
-
continuation automatically.
|
|
1026
|
-
|
|
1027
|
-
### Branch names
|
|
1028
|
-
|
|
1029
|
-
`<type>/<slug>`, where the type is `fix` when any label's last segment (after `:`
|
|
1030
|
-
or `/`) is `bug`, and `feat` otherwise. The slug is the issue title folded to
|
|
1031
|
-
`[a-z0-9-]`, and the whole ref is capped at 60 characters. It is computed from the
|
|
1032
|
-
issue alone, so a retried run recomputes the same branch and finds its own work
|
|
1033
|
-
instead of forking a second one.
|
|
1034
|
-
|
|
1035
|
-
## Routing
|
|
1036
|
-
|
|
1037
|
-
An issue must carry **exactly one** `repo:<name>` label naming a repo in
|
|
1038
|
-
`routing.repos`. The prefix is `routing.labelPrefix` and defaults to `repo:`.
|
|
1039
|
-
|
|
1040
|
-
Routing never guesses. An issue it cannot resolve to a single configured checkout
|
|
1041
|
-
is handed back as unroutable:
|
|
1042
|
-
|
|
1043
|
-
| Reason | Condition |
|
|
1044
|
-
| --- | --- |
|
|
1045
|
-
| `no-repo-label` | The issue carries no label starting with the prefix. |
|
|
1046
|
-
| `multiple-repo-labels` | It carries two or more distinct prefixed labels. A repeated identical label is deduplicated, not treated as an ambiguity. |
|
|
1047
|
-
| `unknown-repo` | Its single prefixed label names a repo that is not in `routing.repos`. |
|
|
1048
|
-
|
|
1049
|
-
In all three cases the issue is **escalated at Tier 1 and never dispatched**. The
|
|
1050
|
-
fix is always the same, and the escalation says so: put exactly one
|
|
1051
|
-
`repo:<name>` label on the issue.
|
|
1052
|
-
|
|
1053
|
-
This is deliberate. A request that spans two repos, taken whole by one worker, is
|
|
1054
|
-
the precise failure this guard exists to prevent: the worker cannot open a PR
|
|
1055
|
-
against two checkouts, so it improvises — it vendors a copy, edits the wrong repo,
|
|
1056
|
-
or produces a PR that cannot be merged without the other half. Splitting a
|
|
1057
|
-
multi-repo request is a human decision about contracts; it is not something to
|
|
1058
|
-
infer from a label. Sending the issue back costs a label edit; guessing costs a
|
|
1059
|
-
bad merge.
|
|
1060
|
-
|
|
1061
|
-
## Host sizing and memory
|
|
1062
|
-
|
|
1063
|
-
Workers are **child processes** of the daemon (plus one long-lived orchestrator
|
|
1064
|
-
session), each its own pid, talking back over a unix socket. They are still
|
|
1065
|
-
inside the service's cgroup, so systemd's Memory peak for
|
|
1066
|
-
`omp-conductor.service` is daemon + every live worker + the orchestrator + any
|
|
1067
|
-
MCP stdio children those sessions mount. `MemoryMax=` governs that whole total,
|
|
1068
|
-
not one process.
|
|
1069
|
-
|
|
1070
|
-
The generated unit starts `omp-conductor daemon --port 8787` without a
|
|
1071
|
-
`--project` filter, so one daemon serves every configured project. Its automatic
|
|
1072
|
-
`MemoryMax=` tier uses the sum of resolved `maxConcurrentWorkers` values across
|
|
1073
|
-
all projects: `3G` for a total of one worker, otherwise `5G`.
|
|
1074
|
-
|
|
1075
|
-
On the reference deploy that produced [issue #51](https://github.com/TerrifiedBug/conductor/issues/51):
|
|
1076
|
-
|
|
1077
|
-
| Shape | Observed |
|
|
1078
|
-
| --- | --- |
|
|
1079
|
-
| Idle / workers restarting | ~430 MB RSS for the daemon alone |
|
|
1080
|
-
| Two workers + orchestrator, busy | **3.2–4.2 GB** Memory peak for the unit; up to ~800 MB swap |
|
|
1081
|
-
|
|
1082
|
-
That peak is **expected for concurrent SDK sessions**, not evidence of a
|
|
1083
|
-
conductor-side leak: the SQLite store is disk-backed, admission state is
|
|
1084
|
-
per-tick, and worker sessions are disposed when a run ends. What grows is the
|
|
1085
|
-
session heap (conversation + tool output); a single graph-assisted run has been
|
|
1086
|
-
measured in the hundreds of thousands of characters of tool output.
|
|
1087
|
-
|
|
1088
|
-
**Practical guidance**
|
|
1089
|
-
|
|
1090
|
-
- Prefer **≥16 GiB RAM** for the default `maxConcurrentWorkers: 2`, and do **not**
|
|
1091
|
-
co-locate ClickHouse / other multi-GB services beside that fleet on an ≤8 GiB
|
|
1092
|
-
box.
|
|
1093
|
-
- On hosts under ~16 GiB, keep the **sum** of every project's
|
|
1094
|
-
`maxConcurrentWorkers` at **1**. `omp-conductor setup` chooses that default
|
|
1095
|
-
for a new project when it can read host RAM and warns before applying a
|
|
1096
|
-
configuration whose combined capacity exceeds the host recommendation.
|
|
1097
|
-
- Supervise the daemon with a unit that sets `SuccessExitStatus=0 143`; setup
|
|
1098
|
-
renders `MemoryMax=3G` for one configured worker and `MemoryMax=5G` for two
|
|
1099
|
-
or more.
|
|
1100
|
-
- `omp-conductor status` prints daemon `rss` from `/healthz` when the process is
|
|
1101
|
-
up, so you can see pressure without scraping journald.
|
|
1102
|
-
|
|
1103
|
-
## Caps
|
|
1104
|
-
|
|
1105
|
-
Caps resolve per project: the global `defaults` block, then the project's own
|
|
1106
|
-
`caps` layered on field by field, so a project that pins one cap still inherits the
|
|
1107
|
-
rest. `0` is a real value (a hard stop), not "unset".
|
|
1108
|
-
|
|
1109
|
-
| Cap | Default | What it protects |
|
|
1110
|
-
| --- | --- | --- |
|
|
1111
|
-
| `maxConcurrentWorkers` | `2` (setup may write `1` on <16 GiB hosts) | Parallel omp sessions, each a child process of the daemon and all inside its cgroup. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. On hosts under ~16 GiB RAM, prefer `1` so the unit stays out of swap ([host sizing](#host-sizing-and-memory)). Raise it only if you actually have the runners *and* the RAM. |
|
|
1112
|
-
| `maxConcurrentWorkersPerRepo` | `1` | Max live workers in the **same repo**. The mirror, branch-protection staleness and shared CI egress are all per-repo collision domains, so extra slots should land on other repos. Raise it only when a repo genuinely needs two workers at once. |
|
|
1113
|
-
| `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total`. |
|
|
1114
|
-
| `planUsage` | `null` (unmetered) | Subscription/plan allowance guard: `{ "windowId": "anthropic:7d", "maxUsedFraction": 0.85 }`, or `null` for no plan gate. Independent of `dailySpendUsd` — see [Plan allowance](#plan-allowance-planusage) below. |
|
|
1115
|
-
| `workerMaxTurns` | `120` | Base ceiling for each new worker. Catches a session looping without converging; use `omp-conductor extend` to raise one live run or one issue's next attempt without changing this default. |
|
|
1116
|
-
| `workerMaxTurnsCeiling` | `240` (twice the effective `workerMaxTurns` when omitted) | Upper bound for per-issue turn extensions. Prevents the loopback control from granting an unbounded worker budget. |
|
|
1117
|
-
| `workerWallClockMs` | `5400000` (90 minutes) | Wall-clock ceiling for one worker. A session that is merely stuck spends no turns, so turns alone cannot detect it. |
|
|
1118
|
-
| `maxAttemptsPerIssue` | `2` | Failed implementation or CI attempts before escalation. Operational stops do not consume this budget, so salvage can continue without stealing the retry needed for a real failure. |
|
|
1119
|
-
| `maxContinuationsPerIssue` | `2` | Cap-kill, daemon-orphan and answered-block resumes before escalation. This independently bounds crash/resume loops. |
|
|
1120
|
-
|
|
1121
|
-
Days are counted from **local midnight**, matching how a human reads "today".
|
|
1122
|
-
|
|
1123
|
-
Set `dailySpendUsd` to `null` (wizard: blank) for no money gate — turns and wall-clock still apply. Hitting a numeric `dailySpendUsd` is not the same as hitting the other caps. A concurrency
|
|
1124
|
-
limit simply defers work to a later tick. The spend cap **pauses the daemon and
|
|
1125
|
-
pages at Tier 2**: a loop that is burning money has to halt itself, because
|
|
1126
|
-
waiting for someone to notice tomorrow is how a runaway becomes expensive.
|
|
1127
|
-
Work resumes only after `omp-conductor resume`.
|
|
1128
|
-
|
|
1129
|
-
`workerMaxTurns` and `workerWallClockMs` are enforced inside the session driver.
|
|
1130
|
-
The daemon reads a live run's effective turn ceiling at every turn boundary. Use
|
|
1131
|
-
`omp-conductor extend <issue> --turns N [--project NAME]` to raise it without
|
|
1132
|
-
restarting or reconstructing the session. For a live worker, extension is
|
|
1133
|
-
monotonic: equal or lower values are refused. If the latest run is failed,
|
|
1134
|
-
killed, orphaned, or blocked, the command instead stores a one-shot ceiling for
|
|
1135
|
-
that issue's next attempt. A next-attempt ceiling must exceed the effective
|
|
1136
|
-
project base, and every extension must stay at or below
|
|
1137
|
-
`workerMaxTurnsCeiling`. `status` shows both active ceilings and pending
|
|
1138
|
-
next-attempt overrides. The store consumes an override atomically when it claims
|
|
1139
|
-
the next run, so later attempts return to the project base. Config edits change
|
|
1140
|
-
that base on the next tick but do not change workers already in flight. A cap
|
|
1141
|
-
that fires aborts the run, records it as `killed`, and names the ceiling in the
|
|
1142
|
-
escalation.
|
|
1143
|
-
|
|
1144
|
-
Pause one live worker cooperatively with
|
|
1145
|
-
`omp-conductor worker pause <issue> [--project NAME]`. The daemon aborts the
|
|
1146
|
-
active turn to an idle harness state, freezes the remaining wall-clock budget,
|
|
1147
|
-
and keeps the run in the Running lane. `status` overlays `paused`/`pausing` from `/healthz` on that active-run line while the SQLite row stays `running`. `omp-conductor worker resume <issue>`
|
|
1148
|
-
continues the same session with a prompt to re-check its last action before
|
|
1149
|
-
proceeding. To end that run instead, use
|
|
1150
|
-
`omp-conductor worker stop <issue> --reason TEXT [--project NAME]`. Stop works
|
|
1151
|
-
from running or paused, salvages dirty work before removing the worktree, records
|
|
1152
|
-
the distinct terminal `stopped` state, and removes `agent:in-progress` through
|
|
1153
|
-
the label outbox. A salvage failure keeps the only copy in place and reports its
|
|
1154
|
-
path. Stopped runs consume neither implementation-failure nor continuation
|
|
1155
|
-
budget. Repeating stop reports the already-terminal state. These worker controls
|
|
1156
|
-
are separate from fleet-level `pause`, which stops new claims.
|
|
1157
|
-
|
|
1158
|
-
### Plan allowance (`planUsage`)
|
|
1159
|
-
|
|
1160
|
-
`dailySpendUsd` meters money, which is the only thing an API-billed account can
|
|
1161
|
-
run out of. A fixed-price subscription cannot be expressed that way: the real
|
|
1162
|
-
ceiling is a **provider allowance** — a weekly token window whose marginal
|
|
1163
|
-
dollar cost is zero and whose exhaustion stops every session on the host.
|
|
1164
|
-
Pricing that into the dollar meter would mean inventing a number.
|
|
1165
|
-
|
|
1166
|
-
`planUsage` is the second, independent guard. It reads `omp usage --json` — the
|
|
1167
|
-
structured form of the harness `/usage` view — and holds new claims while the
|
|
1168
|
-
named window is at or over its threshold. Running workers finish normally, and
|
|
1169
|
-
**the daemon is not paused**: the window resets on the provider's clock, so
|
|
1170
|
-
dispatch resumes by itself once a fresh reading is below the threshold. Nothing
|
|
1171
|
-
estimates a plan quota from conductor's own transcript token counts.
|
|
328
|
+
### See the fleet in a browser: `dashboard`
|
|
329
|
+
|
|
330
|
+
`omp-conductor dashboard` serves the same fleet facts `status` and `board`
|
|
331
|
+
render, in a browser: a static UI plus one read endpoint, `GET /api/projects`,
|
|
332
|
+
which answers with every configured project annotated with its daemon state
|
|
333
|
+
(the same `livingDaemon` + `/healthz` classification `status` uses), the
|
|
334
|
+
daemon's port, and the raw `/healthz` body when it answers. It is a separate
|
|
335
|
+
process from the dispatch daemon and only ever reads fleet state — nothing on
|
|
336
|
+
the daemon port changes.
|
|
337
|
+
|
|
338
|
+
Binds `127.0.0.1:8788` by default. First start mints a bearer token at
|
|
339
|
+
`<stateDir()>/dashboard-token` (mode `0600`, next to `config.json` under
|
|
340
|
+
`$OMP_CONDUCTOR_HOME`) and later starts reuse it; the page asks for the token
|
|
341
|
+
once and keeps it in its own `localStorage`. Every `/api/*` request must carry
|
|
342
|
+
`Authorization: Bearer <token>`; static assets are unauthenticated by design.
|
|
343
|
+
|
|
344
|
+
**Tailnet posture.** Loopback is the default because it is the safe one: any
|
|
345
|
+
local user could otherwise read your project list. To look at the dashboard
|
|
346
|
+
from another machine on your tailnet, run
|
|
347
|
+
`omp-conductor dashboard --host <tailnet-ip>`; the server still binds, but
|
|
348
|
+
prints a one-line warning naming the token file — the token is then the only
|
|
349
|
+
thing between anyone on that network and the fleet.
|
|
350
|
+
|
|
351
|
+
### See the fleet: `status` and `board`
|
|
352
|
+
|
|
353
|
+
`omp-conductor status` and `omp-conductor board` are the two windows onto the
|
|
354
|
+
fleet, and both are covered above under
|
|
355
|
+
[Stop the conductor](#stop-the-conductor-hold--stop) because that is where the
|
|
356
|
+
paused-fleet failure hides. `tail`, `extend`, `unblock` and the `worker`
|
|
357
|
+
controls are the day-to-day levers:
|
|
358
|
+
|
|
359
|
+
### Outcomes and cost: `stats`
|
|
360
|
+
|
|
361
|
+
`omp-conductor stats [--since 7d | 30d | YYYY-MM-DD] [--project NAME] [--json]`
|
|
362
|
+
answers the operator's basic product question — is this fleet earning its keep
|
|
363
|
+
— from the local store only, with zero GitHub calls. Over a bounded window
|
|
364
|
+
(default: the last 7 days) it reports, per repo and in total: issues merged,
|
|
365
|
+
issues settled (merged or terminally failed/blocked), merge rate, queue→merge
|
|
366
|
+
lead time (median and p90), attempts per merged issue, metered spend per merged
|
|
367
|
+
issue, the failure-class breakdown of everything that did not merge, and the
|
|
368
|
+
tracked GitHub API calls consumed over the window's days.
|
|
369
|
+
|
|
370
|
+
Continuation chains collapse into one journey: an issue that took six attempts
|
|
371
|
+
is one merged outcome with six runs and one lead time, measured from its first
|
|
372
|
+
queue-label claim to the tick's merge settlement. A chain that began before the
|
|
373
|
+
window still counts its whole lead time and all its attempts.
|
|
374
|
+
|
|
375
|
+
Runs whose spend reads `$0.00` — harness telemetry absent, see
|
|
376
|
+
[Limitations](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#limitations)
|
|
377
|
+
— are counted separately as **unmetered** and never averaged into cost as if
|
|
378
|
+
they were free; a merged issue the harness never metered reports its cost as
|
|
379
|
+
unknown rather than zero.
|
|
380
|
+
|
|
381
|
+
`--json` prints the stable report shape. It is a contract: keys never move, and
|
|
382
|
+
the human rendering shows the same numbers:
|
|
1172
383
|
|
|
1173
384
|
```json
|
|
1174
|
-
|
|
1175
|
-
"
|
|
385
|
+
{
|
|
386
|
+
"project": "demo",
|
|
387
|
+
"window": { "sinceDay": "2026-08-09", "untilDay": "2026-08-16",
|
|
388
|
+
"sinceEpochMs": 1754697600000, "untilEpochMs": 1755302400000 },
|
|
389
|
+
"ghCalls": 41,
|
|
390
|
+
"empty": false,
|
|
391
|
+
"total": {
|
|
392
|
+
"repo": "(all)",
|
|
393
|
+
"merged": 3, "settled": 4, "mergeRate": 0.75,
|
|
394
|
+
"runsPerMerged": 4, "leadTimeMedianMs": 7560000, "leadTimeP90Ms": 34200000,
|
|
395
|
+
"spendUsd": 1.23, "spendPerMerged": 0.41,
|
|
396
|
+
"unmeteredRuns": 2, "unmeteredMerged": 1,
|
|
397
|
+
"failureClasses": { "unknown": 3 }
|
|
398
|
+
},
|
|
399
|
+
"repos": [ { "repo": "acme/api", "merged": 3, "settled": 3, "mergeRate": 1,
|
|
400
|
+
"runsPerMerged": 2, "leadTimeMedianMs": 7560000,
|
|
401
|
+
"leadTimeP90Ms": 34200000, "spendUsd": 1.23,
|
|
402
|
+
"spendPerMerged": 0.41, "unmeteredRuns": 2, "unmeteredMerged": 1,
|
|
403
|
+
"failureClasses": {} } ]
|
|
1176
404
|
}
|
|
1177
405
|
```
|
|
1178
406
|
|
|
1179
|
-
|
|
1180
|
-
|
|
1181
|
-
|
|
1182
|
-
|
|
1183
|
-
|
|
1184
|
-
|
|
1185
|
-
```bash
|
|
1186
|
-
omp usage --json | jq -r '.reports[].limits[] | "\(.id) \(.amount.usedFraction) \(.amount.unit)"'
|
|
1187
|
-
```
|
|
1188
|
-
|
|
1189
|
-
A bare window key (`"7d"`) also works, but **only** when exactly one reported
|
|
1190
|
-
allowance carries it. On an Anthropic account `7d` matches two, and the guard
|
|
1191
|
-
refuses to guess.
|
|
1192
|
-
|
|
1193
|
-
**`maxUsedFraction` is a fraction, not a percentage.** `0.85` holds at 85%.
|
|
1194
|
-
A value outside `0`–`1` is rejected at config load, because `85` would mean
|
|
1195
|
-
"hold at 8500% consumed" — a guard that reads as configured and can never fire.
|
|
1196
|
-
Comparison always goes through the provider's `usedFraction`, never a raw
|
|
1197
|
-
count: `unit` is `percent` for Anthropic and `unknown` with raw counts for
|
|
1198
|
-
`xai-oauth`, so a threshold compared against `used` misreads any non-percent
|
|
1199
|
-
provider by orders of magnitude.
|
|
1200
|
-
|
|
1201
|
-
**Availability policy.** The guard never displays a number it did not read, and
|
|
1202
|
-
never shows a fabricated `0% used`. What each situation does:
|
|
1203
|
-
|
|
1204
|
-
| Situation | `status` / `board` | New claims |
|
|
1205
|
-
| --- | --- | --- |
|
|
1206
|
-
| `planUsage: null` | `unmetered` | admitted |
|
|
1207
|
-
| Window below threshold | `5% / 85% of anthropic:7d used · resets in 6d 2h` | admitted |
|
|
1208
|
-
| Window at or over threshold | same, plus `holding new claims` | **held** (`plan-usage-cap`, Tier 1) |
|
|
1209
|
-
| No provider reports a readable allowance, or `omp usage --json` fails | `unavailable — <reason>` | admitted for up to 30 minutes, then **held** and paged at Tier 2 |
|
|
1210
|
-
| `windowId` names a window the reading does not contain | `window "<id>" is not in this reading — Reported: …` | **held**, Tier 2 |
|
|
1211
|
-
| `windowId` matches more than one allowance | `window "<id>" matches …` | **held**, Tier 2 |
|
|
1212
|
-
| The window reports nothing a fraction can be derived from | `window "<id>" reports no comparable fraction …` | **held**, Tier 2 |
|
|
1213
|
-
|
|
1214
|
-
The split is deliberate. A *read error* is transient — a token refresh, a
|
|
1215
|
-
provider 502, `omp` briefly absent mid-upgrade — and stalling a fleet on one
|
|
1216
|
-
would cost more than admitting through it, since spend, turns, wall clock and
|
|
1217
|
-
concurrency are all still enforced. Half an hour of continuous failure is not
|
|
1218
|
-
an outage, it is a broken meter, and a plan-capped fleet running on a broken
|
|
1219
|
-
meter is how the allowance gets spent to zero unnoticed. A *successful* read
|
|
1220
|
-
that does not contain the configured window is not a read error at all: the
|
|
1221
|
-
source answered, and it says the config names something that is not there. That
|
|
1222
|
-
fails closed immediately, like every other config fault in this package, and
|
|
1223
|
-
recovers by itself as soon as a reading contains the window again.
|
|
1224
|
-
|
|
1225
|
-
Readings are cached for 60 seconds (15 for a failure) so one tick costs one
|
|
1226
|
-
provider call rather than one per candidate, and a cached reading is dropped
|
|
1227
|
-
the moment its own `resetsAt` passes — that is what makes admission resume at
|
|
1228
|
-
the rollover instead of a TTL later. `omp usage invalidate` clears omp's own
|
|
1229
|
-
cache; conductor picks the change up at its next read.
|
|
1230
|
-
|
|
1231
|
-
Both controls are shown separately, never folded together — `status` prints a
|
|
1232
|
-
`spend today` row and a `plan usage` row, and the board's admission line ends
|
|
1233
|
-
with `spend $2.40/$25.00 | plan 5%/85%`.
|
|
1234
|
-
|
|
1235
|
-
## Worker model
|
|
1236
|
-
|
|
1237
|
-
`workerModel` on a project pins the model its workers run on, as a pattern in
|
|
1238
|
-
omp's own model/role syntax (whatever `/model` accepts). It sits beside `caps`
|
|
1239
|
-
rather than inside them, because it is not a ceiling:
|
|
1240
|
-
|
|
1241
|
-
```json
|
|
1242
|
-
"workerModel": "smol"
|
|
1243
|
-
```
|
|
1244
|
-
|
|
1245
|
-
Omit it and the harness picks, which is the right answer until you have a reason.
|
|
1246
|
-
The pattern is passed through unresolved: omp resolves it after its extensions
|
|
1247
|
-
load, so a name this package has never heard of still works. If the harness cannot
|
|
1248
|
-
honour the pattern it says so, and the daemon logs that per run:
|
|
407
|
+
`empty: true` means nothing settled in the window — a fresh store or an idle
|
|
408
|
+
fleet, not zero measurements. `mergeRate`, the lead-time fields,
|
|
409
|
+
`runsPerMerged` and `spendPerMerged` are `null` when there is nothing to
|
|
410
|
+
measure them over, never a fabricated 0. Lead times are settlement
|
|
411
|
+
timestamps (the tick that confirmed the merge), so they carry up to one tick
|
|
412
|
+
of sweep latency.
|
|
1249
413
|
|
|
1250
|
-
|
|
1251
|
-
#412 model fallback: <what the harness substituted>
|
|
1252
|
-
```
|
|
414
|
+
### Watch a run: `tail`
|
|
1253
415
|
|
|
1254
|
-
|
|
1255
|
-
|
|
416
|
+
`omp-conductor tail <issue>` follows the newest run for an issue — its
|
|
417
|
+
`assistant:` text and every `tool:` call as they land — from the top of the
|
|
418
|
+
transcript, and prints `run ended: <state>` when it finishes. Workers are omp
|
|
419
|
+
sessions inside the daemon, not terminals, so this is the only way to watch one
|
|
420
|
+
live. See [`tail`](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#cli-reference).
|
|
1256
421
|
|
|
1257
|
-
|
|
422
|
+
### Intervene: `extend`, `unblock`, `worker`
|
|
1258
423
|
|
|
1259
|
-
|
|
1260
|
-
|
|
1261
|
-
|
|
1262
|
-
`
|
|
1263
|
-
|
|
1264
|
-
|
|
1265
|
-
|
|
424
|
+
- `omp-conductor extend <issue> --turns N` raises a live worker's effective turn
|
|
425
|
+
ceiling, or stores a one-shot ceiling for the issue's next attempt when its
|
|
426
|
+
latest run is terminal.
|
|
427
|
+
- `omp-conductor unblock <issue>` clears an issue's `blocked` / `failed` labels —
|
|
428
|
+
and `agent:in-progress` when the newest run is terminal — so an answered
|
|
429
|
+
escalation can be claimed again. It is the way back after you answer a tier-1.
|
|
430
|
+
- `omp-conductor worker pause/resume/stop <issue>` parks or ends one live worker
|
|
431
|
+
without touching the fleet; `worker stop --reason TEXT` terminates, salvages
|
|
432
|
+
and gives up its in-progress label.
|
|
1266
433
|
|
|
1267
|
-
|
|
434
|
+
Each is a verb on the binary and takes `--project NAME`. Full semantics are in
|
|
435
|
+
the [CLI reference](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#cli-reference).
|
|
1268
436
|
|
|
1269
|
-
|
|
1270
|
-
dispatch. With `graphProject` unset, nothing about dispatch, caps, escalation, or
|
|
1271
|
-
status changes. A fresh host needs both of these before an index is worth
|
|
1272
|
-
anything, and `setup graph` reports them as step 0:
|
|
437
|
+
### Answering a decision
|
|
1273
438
|
|
|
1274
|
-
|
|
1275
|
-
|
|
1276
|
-
|
|
1277
|
-
|
|
1278
|
-
|
|
1279
|
-
|
|
439
|
+
A question the orchestrator put to you — an amendment, a tier-2 decision, "do I
|
|
440
|
+
ship this tonight?" — arrives on your phone and is written to the **decision
|
|
441
|
+
ledger**, where it survives compaction, restarts and long ticks instead of living
|
|
442
|
+
only in a session's memory. You answer with the `omp-conductor decision` verbs
|
|
443
|
+
(`open`, `list`, `resolve`, `withdraw`), and a `--resolves-when` condition lets a
|
|
444
|
+
parked question wake up by itself when a PR merges, an issue closes or a check
|
|
445
|
+
goes green. See [the decision ledger](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#the-decision-ledger-136).
|
|
1280
446
|
|
|
1281
|
-
|
|
1282
|
-
underneath it (default `~/.cache/conductor-graph/<org>/<repo>`) and writes it to
|
|
1283
|
-
each repo's [`graphProject`](#configuration). The only automatic interaction is
|
|
1284
|
-
a bounded, read-only health query; this package never clones, fetches, builds an
|
|
1285
|
-
index, or changes systemd. Dispatch, caps and escalation do not depend on graph
|
|
1286
|
-
health. On a fleet configured before this key existed, `omp-conductor setup` and
|
|
1287
|
-
the `code graph` area add it in two prompts — see
|
|
1288
|
-
[Changing one setting](#changing-one-setting).
|
|
447
|
+
### Capture an idea: `intake`
|
|
1289
448
|
|
|
1290
|
-
|
|
1291
|
-
|
|
1292
|
-
|
|
1293
|
-
being blunt about all three candidates.
|
|
1294
|
-
|
|
1295
|
-
| Directory | Why not |
|
|
1296
|
-
| --- | --- |
|
|
1297
|
-
| **The worker's worktree** | An index is keyed by the realpath of the directory it was built from, and has no git-worktree awareness. A run's `worktrees/<issue>` path is therefore *always* an empty project — a worker that queried its own cwd would get silence, conclude there is no graph, and spend the run grepping. This is why `graphProject` is an absolute path in the config and not something derived at run time. |
|
|
1298
|
-
| **Your own checkout** | Refreshing an index means resetting the clone to its default branch. In a directory you work in, that either destroys uncommitted work or — if it is made safe instead — indexes whatever feature branch you left checked out, so the fleet orients against your WIP. |
|
|
1299
|
-
| **A conductor mirror** | The daemon's mirrors are bare. There is no working tree to index. |
|
|
1300
|
-
|
|
1301
|
-
So `graphProject` names a fourth thing: a clone that exists only to be indexed,
|
|
1302
|
-
that nothing human ever edits, and that is therefore safe to `git reset --hard`
|
|
1303
|
-
every night. The worker brief names that path, tells the session to match it
|
|
1304
|
-
against `list_projects`' `root_path` and query by the `name` beside it, and says
|
|
1305
|
-
plainly that the graph is a snapshot which does **not** contain the worker's own
|
|
1306
|
-
edits — orient with it, then read the real file before changing it.
|
|
1307
|
-
|
|
1308
|
-
### Creating and refreshing them
|
|
449
|
+
An idea that is not an issue yet has nowhere to live. `omp-conductor intake`
|
|
450
|
+
gives it one, durably — the sqlite store, not a session, so a thought captured
|
|
451
|
+
at 02:00 is still there after a restart:
|
|
1309
452
|
|
|
1310
453
|
```bash
|
|
1311
|
-
omp-conductor
|
|
1312
|
-
omp-conductor
|
|
454
|
+
omp-conductor intake "ship the intake command" # prints an id
|
|
455
|
+
omp-conductor intake list # id, age, text — oldest first
|
|
456
|
+
omp-conductor intake dismiss 3f9c2a1b7e04 # drops one by id
|
|
1313
457
|
```
|
|
1314
458
|
|
|
1315
|
-
`
|
|
1316
|
-
|
|
1317
|
-
|
|
1318
|
-
in the state directory and prints the two `sudo` lines that install and enable
|
|
1319
|
-
them; it never runs `systemctl`.
|
|
1320
|
-
|
|
1321
|
-
**Run it as the account the fleet runs as, never under `sudo`** — it refuses if
|
|
1322
|
-
you try. Everything it derives resolves per-account: the config it loads, the
|
|
1323
|
-
state directory it stages into, and the `HOME`/`User=` it bakes into the unit.
|
|
1324
|
-
Under root you get a timer that goes green while writing indexes into
|
|
1325
|
-
`/root/.cache`, where no worker session looks — silent, and indistinguishable
|
|
1326
|
-
from the feature simply not helping. Only installing the units needs root, which
|
|
1327
|
-
is why that is two separate printed commands.
|
|
1328
|
-
|
|
1329
|
-
Two properties of the generated unit are deliberate:
|
|
1330
|
-
|
|
1331
|
-
- **It is a timer, not the server's own watcher.** That watcher lives inside a
|
|
1332
|
-
connected MCP session and dies with it, so an ephemeral worker session keeps
|
|
1333
|
-
nothing fresh. The refresh has to come from outside the fleet.
|
|
1334
|
-
- **It fails loudly.** The refresh is `set -euo pipefail`, then per repo
|
|
1335
|
-
`git fetch --prune origin` and `git reset --hard origin/<its own defaultBranch>`
|
|
1336
|
-
before indexing. Nothing is `|| true`-ed, so a fetch that has been broken for a
|
|
1337
|
-
week turns the unit red instead of quietly re-indexing a stale tree and exiting
|
|
1338
|
-
`0` — a green timer serving a month-old graph is worse than no graph at all.
|
|
1339
|
-
|
|
1340
|
-
The unit spells out `HOME` and an explicit `PATH`, because systemd supplies
|
|
1341
|
-
neither usefully: the indexer resolves its store from `HOME`, systemd's default
|
|
1342
|
-
`PATH` has no `~/.local/bin`, and the indexer shells out to `git`. Both are the
|
|
1343
|
-
user that ran `setup graph`; the unit sets no `User=`, so check them if that is
|
|
1344
|
-
not the account the timer runs as.
|
|
1345
|
-
|
|
1346
|
-
### Seeing whether the graph is usable
|
|
1347
|
-
|
|
1348
|
-
When at least one routed repo has `graphProject`, `omp-conductor status` adds a
|
|
1349
|
-
`code graph` block. It proves the indexer is on `PATH`, the worker MCP config
|
|
1350
|
-
mounts it, every configured clone exists and exactly matches an indexed
|
|
1351
|
-
`root_path`, the refresh timer is enabled and active, and the last service run
|
|
1352
|
-
succeeded within 45 minutes. A running daemon refreshes this evidence every
|
|
1353
|
-
minute and publishes the cached result through `/healthz`; status probes the host
|
|
1354
|
-
directly when that cache is unavailable. Every command is read-only, runs with a
|
|
1355
|
-
one-second timeout, and graph degradation never changes `/healthz.ok` or blocks
|
|
1356
|
-
dispatch. Unconfigured projects omit the block entirely.
|
|
1357
|
-
|
|
1358
|
-
## Escalation tiers
|
|
1359
|
-
|
|
1360
|
-
| Tier | Meaning | Raised by | Delivered to |
|
|
1361
|
-
| --- | --- | --- | --- |
|
|
1362
|
-
| 1 | "Not a human's problem yet" — the run is parked and safe. | Unroutable issue, blocked run, failed or killed run, dispatch error, attempts exhausted. | The orchestrator session, as an injected prompt. Falls back to an issue comment when no orchestrator is running, or when it will not accept the injection. |
|
|
1363
|
-
| 2 | "The fleet is stopped until you look." | Daily spend cap reached; the installed package changed under the running daemon. Either way the project is already paused. | Telegram, when `escalation.telegramChatId` is set and a bot token is readable; otherwise it falls back to the issue comment. |
|
|
1364
|
-
|
|
1365
|
-
**The orchestrator** is one persistent, file-backed session per daemon run, resumed
|
|
1366
|
-
across restarts so it remembers what it has already handled. Its `cwd` is the state
|
|
1367
|
-
directory, deliberately not a checkout. Delivery resolves when the harness *accepts*
|
|
1368
|
-
the prompt, not when the model answers it, so a tick never parks behind a model; an
|
|
1369
|
-
injection arriving mid-thought queues as a follow-up instead of interrupting the
|
|
1370
|
-
turn in flight. Its standing orders are explicit: re-brief the worker, file or
|
|
1371
|
-
comment on issues, or promote to tier 2, and never edit product code or push a
|
|
1372
|
-
branch. Merging is the one line worded from config — see [`authority`](#configuration).
|
|
1373
|
-
If it fails to start, the daemon logs a warning and runs on, with tier-1
|
|
1374
|
-
escalations degraded to issue comments.
|
|
1375
|
-
|
|
1376
|
-
**Or no orchestrator at all.** Set `escalation.orchestrator` to `"external"` when
|
|
1377
|
-
you already run your own supervising session — a visible TUI session in a pane,
|
|
1378
|
-
typically. The daemon then starts none of its own and every tier-1 escalation
|
|
1379
|
-
posts as an issue comment, which is what that session drains. One brain, and it
|
|
1380
|
-
is the one you can watch.
|
|
1381
|
-
|
|
1382
|
-
**Answering a tier 1 is only half of it.** A blocked or failed run leaves its state
|
|
1383
|
-
label on the issue, and eligibility reads any state label as disqualifying, so an
|
|
1384
|
-
answered issue that keeps one is never re-claimed and the answer is inert — nothing
|
|
1385
|
-
fails, the issue just stops existing as far as dispatch is concerned.
|
|
1386
|
-
[`omp-conductor unblock <issue>`](#cli-reference) is the way back: it clears the
|
|
1387
|
-
label through the same tracker the dispatcher writes with, including
|
|
1388
|
-
`agent:in-progress` when the newest recorded run is terminal, since a terminal row
|
|
1389
|
-
is proof the worker process is gone. The brief tells the
|
|
1390
|
-
orchestrator to run that verb rather than edit the label itself, and that is not a
|
|
1391
|
-
formality — orphan detection works by comparing `agent:in-progress` labels against
|
|
1392
|
-
live runs, and it is only trustworthy while every state label on the tracker was
|
|
1393
|
-
written by this package.
|
|
1394
|
-
|
|
1395
|
-
Tier 2 borrows the bot token that `omp-telegram` already owns, at
|
|
1396
|
-
`~/.omp/agent/telegram/.env` (or `$OMP_TELEGRAM_STATE_DIR/.env`). If you run that
|
|
1397
|
-
bot, Tier 2 needs no extra configuration beyond the chat id. If the token is
|
|
1398
|
-
absent, Tier 2 degrades to the issue comment instead of failing. The token is
|
|
1399
|
-
never logged, and it is redacted out of any error text that could reach a public
|
|
1400
|
-
issue comment.
|
|
1401
|
-
|
|
1402
|
-
**Escalations are deduplicated.** The dispatcher re-notices the same unroutable
|
|
1403
|
-
issue on every poll, so a ledger in the store — keyed by project, issue, tier and
|
|
1404
|
-
summary — makes a recurring condition page **once** and suppresses the five-minute
|
|
1405
|
-
repeats. The marker is recorded only on successful delivery, so a page that could
|
|
1406
|
-
not be delivered is retried on the next tick instead of being written off as sent.
|
|
1407
|
-
The spend-cap and integrity-tripwire summaries carry the date, so the same
|
|
1408
|
-
condition pages again tomorrow but only once per day.
|
|
1409
|
-
|
|
1410
|
-
If `fallbackToIssueComment` is off and no Telegram transport is configured,
|
|
1411
|
-
delivery throws instead of dropping silently. The failure is logged and retried,
|
|
1412
|
-
because a swallowed escalation looks exactly like a healthy fleet.
|
|
1413
|
-
|
|
1414
|
-
## Report delivery (the outbox)
|
|
1415
|
-
|
|
1416
|
-
Escalations are the daemon's. **Reports** — the material events and the daily
|
|
1417
|
-
digest your [`reporting.scope`](#your-workflow-vs-the-package) asks for — are
|
|
1418
|
-
written by the orchestrator, and until v0.3.26 they were also *delivered* by it:
|
|
1419
|
-
a report reached you only if the model remembered to call `telegram_send`. On
|
|
1420
|
-
2026-08-06 a suite release and two tier-2 escalations were written that way and
|
|
1421
|
-
none of the three arrived, and nothing anywhere recorded that fact — an undelivered
|
|
1422
|
-
report and a quiet tick look identical.
|
|
1423
|
-
|
|
1424
|
-
### Material events survive the session
|
|
1425
|
-
|
|
1426
|
-
A deferred digest does not use the session transcript as its source of truth.
|
|
1427
|
-
Record each ordinary outcome when it happens:
|
|
1428
|
-
|
|
1429
|
-
```bash
|
|
1430
|
-
omp-conductor event record \
|
|
1431
|
-
--category merge \
|
|
1432
|
-
--summary "#42 merged" \
|
|
1433
|
-
--evidence "https://github.com/acme/api/pull/42"
|
|
1434
|
-
```
|
|
1435
|
-
|
|
1436
|
-
`--category` is a short lowercase slug. `--summary` states the outcome, and
|
|
1437
|
-
`--evidence` names the issue, PR, release, run, commit, or URL that proves it.
|
|
1438
|
-
Use `--occurred-at <ISO timestamp>` when the event happened earlier; otherwise,
|
|
1439
|
-
the command uses the current time. The command writes one row to SQLite and
|
|
1440
|
-
sends nothing. The row survives later ticks, session compaction, session
|
|
1441
|
-
replacement, and daemon restarts.
|
|
1442
|
-
|
|
1443
|
-
When a digest is due, its tick prompt lists a bounded, oldest-first set of
|
|
1444
|
-
owed material events and deferred escalations. Each line includes its ledger id.
|
|
1445
|
-
The prompt gives the exact handoff shape:
|
|
459
|
+
`status` shows an `intake N pending idea(s)` row only while the backlog is
|
|
460
|
+
nonzero, so an empty intake stays invisible instead of becoming noise; grooming
|
|
461
|
+
a pending idea into an actual issue happens later, not on this surface.
|
|
1446
462
|
|
|
1447
|
-
|
|
1448
|
-
omp-conductor report \
|
|
1449
|
-
--kind digest \
|
|
1450
|
-
--events EVENT_ID_1,EVENT_ID_2 \
|
|
1451
|
-
--notices NOTICE_ID_1,NOTICE_ID_2 \
|
|
1452
|
-
--text "<the whole digest>"
|
|
1453
|
-
```
|
|
463
|
+
## Multi-project operation
|
|
1454
464
|
|
|
1455
|
-
|
|
1456
|
-
|
|
1457
|
-
|
|
1458
|
-
|
|
1459
|
-
delivery exhausts its retry budget and the report becomes `failed`, its rows
|
|
1460
|
-
return to the owed backlog, where a replacement digest can claim them.
|
|
1461
|
-
`omp-conductor status` always shows the
|
|
1462
|
-
material-event and held-escalation backlog counts, including the age of the
|
|
1463
|
-
oldest row when one exists.
|
|
465
|
+
One daemon serves every configured project on a host. Each project keeps its own
|
|
466
|
+
queue, caps, reporting scope, tick config and arm marker under
|
|
467
|
+
`~/.omp/conductor/projects/<name>/`. The lifecycle verbs below take
|
|
468
|
+
`--project NAME`, and a few take `--all` for when you mean the whole host at once.
|
|
1464
469
|
|
|
1465
|
-
|
|
1466
|
-
state. The orchestrator still decides what is material and records the evidence.
|
|
1467
|
-
The mechanism only makes that decision durable until a non-failed digest owns it.
|
|
470
|
+
### Adding another project
|
|
1468
471
|
|
|
1469
|
-
|
|
1470
|
-
|
|
472
|
+
To put a second fleet on the same host without touching the first, run the wizard
|
|
473
|
+
for the new name only:
|
|
1471
474
|
|
|
1472
475
|
```bash
|
|
1473
|
-
omp-conductor
|
|
1474
|
-
|
|
1475
|
-
--text "<the whole digest>" --kind digest \
|
|
1476
|
-
--events EVENT_IDS --notices NOTICE_IDS # use row ids from its tick
|
|
476
|
+
omp-conductor setup --project second
|
|
477
|
+
# or pick "Add another project" from the re-run chooser
|
|
1477
478
|
```
|
|
1478
479
|
|
|
1479
|
-
|
|
1480
|
-
|
|
1481
|
-
|
|
1482
|
-
|
|
1483
|
-
`reports` row and prints its report id. Both survive the session being
|
|
1484
|
-
compacted, interrupted or restarted,
|
|
1485
|
-
and the daemon being restarted under it. The daemon delivers over the same bot
|
|
1486
|
-
token tier 2 uses, with bounded retries, and `omp-conductor status` lists
|
|
1487
|
-
anything it still owes. If availability closes after a material report was
|
|
1488
|
-
queued but before its first attempt, the outbox atomically converts that row to
|
|
1489
|
-
the same held-notice path instead of leaking the update through quiet hours.
|
|
1490
|
-
|
|
1491
|
-
### Answering a person, in the thread they wrote in
|
|
1492
|
-
|
|
1493
|
-
A report is an update; an answer is a conversation, and it goes back where the
|
|
1494
|
-
question came from. `telegram_send` keeps the active forum topic **only while it
|
|
1495
|
-
names no chat** — `thread_id` defaults to the active topic when `chat_id` is
|
|
1496
|
-
omitted — so an orchestrator that helpfully supplied `chat_id` (and nothing
|
|
1497
|
-
else) answered three topic messages in the main chat instead (#366). The
|
|
1498
|
-
orchestrator floor now says to name **neither** target or **both**, and the
|
|
1499
|
-
dispatcher refuses `chat_id` without `thread_id` for a project that configured
|
|
1500
|
-
`escalation.telegramTopicId`. A flat-chat project is unaffected. When a pinned
|
|
1501
|
-
topic id has gone stale — the bridge re-claims pane topics across restarts — the
|
|
1502
|
-
live claim for the project's herdr space is used instead, falling back to a
|
|
1503
|
-
claim titled for the project, so a restart does not quietly move every page into
|
|
1504
|
-
the main chat (#407, #412).
|
|
1505
|
-
|
|
1506
|
-
A locally injected tick has no inbound message to inherit a topic from, so a
|
|
1507
|
-
bare `telegram_send` there has nothing to preserve. That turn addresses the
|
|
1508
|
-
operator from config instead:
|
|
1509
|
-
|
|
1510
|
-
```bash
|
|
1511
|
-
omp-conductor message --text "<the message>" # this project's chat and topic
|
|
1512
|
-
omp-conductor message --text "QUESTION: cut 0.16.0 tonight?" # carries the decision category
|
|
1513
|
-
```
|
|
480
|
+
That is a full interview for the new name only. Defaults land under
|
|
481
|
+
`~/.omp/conductor/projects/<name>/{worktrees,mirrors}`, so two fleets never share
|
|
482
|
+
a cwd. After apply, setup provisions labels, brief, tick config, topic binding,
|
|
483
|
+
smoke and arm for that project only, then prints the two follow-ups:
|
|
1514
484
|
|
|
1515
|
-
|
|
1516
|
-
|
|
1517
|
-
|
|
1518
|
-
|
|
1519
|
-
leaves no `reports` row, and nothing retries it.
|
|
485
|
+
- `omp-conductor restart --now` — the running daemon picks the new project up
|
|
486
|
+
only after a reload (printed, not auto-run while workers are live);
|
|
487
|
+
- a copy-pasteable **herdr handoff** that starts the new fleet's orchestrator in
|
|
488
|
+
its own empty pane, never into a live one.
|
|
1520
489
|
|
|
1521
|
-
|
|
490
|
+
Every other area still amends only that project (`setup gates --project second`),
|
|
491
|
+
and a `workspaceRoot` that collides with another project is refused. The full
|
|
492
|
+
walkthrough — what `--project` means for each verb, amendment rules and the
|
|
493
|
+
`--all` caveats — is in [Onboarding](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#adding-another-project).
|
|
1522
494
|
|
|
1523
|
-
|
|
1524
|
-
bot no readable record of what it has already sent. There is nothing to replay a
|
|
1525
|
-
request against and nothing to reconcile with, so **exactly-once delivery cannot
|
|
1526
|
-
be built on this transport** and this package does not claim it. `delivered` is
|
|
1527
|
-
proof that Telegram accepted *an* attempt, never proof that exactly one message
|
|
1528
|
-
exists.
|
|
495
|
+
### Hold, resume, arm — per project, and `--all`
|
|
1529
496
|
|
|
1530
|
-
|
|
1531
|
-
|
|
497
|
+
The lifecycle verbs are per project, so one fleet can be quiet while another
|
|
498
|
+
keeps dispatching:
|
|
1532
499
|
|
|
1533
|
-
|
|
|
500
|
+
| Verb | `--project NAME` | `--all` |
|
|
1534
501
|
| --- | --- | --- |
|
|
1535
|
-
| `
|
|
1536
|
-
| `
|
|
1537
|
-
| `
|
|
1538
|
-
| `
|
|
1539
|
-
|
|
1540
|
-
|
|
1541
|
-
|
|
1542
|
-
|
|
1543
|
-
|
|
1544
|
-
|
|
1545
|
-
|
|
1546
|
-
|
|
1547
|
-
|
|
1548
|
-
|
|
1549
|
-
|
|
1550
|
-
|
|
1551
|
-
|
|
1552
|
-
|
|
1553
|
-
|
|
1554
|
-
|
|
|
1555
|
-
| **Definitive** — nobody has it | Telegram answered and refused it (`{"ok":false}` under any status, or a non-2xx status), or the connection never opened at all (refused, DNS failure) so the request provably never left. | back to `pending`, backoff, attempt counted | nothing special — it *is* a first attempt |
|
|
1556
|
-
| **Outcome unknown** — Telegram might have it | The request was cut off after it left: timeout, abort, socket reset, `EPIPE`. Or the POST came back `200` and the **response body could not be read** — Telegram had already decided and the answer was lost coming back. | stays `sending`, flagged as a possible repeat | `POSSIBLE REPEAT`, with the report id to compare against |
|
|
1557
|
-
|
|
1558
|
-
Anything that cannot be classified confidently is treated as **outcome unknown**.
|
|
1559
|
-
That default is deliberate and is the safe direction: the worst case is a
|
|
1560
|
-
duplicate you were warned about, against a delivered report re-posted as though
|
|
1561
|
-
it were new, with nothing anywhere saying it might be a second copy.
|
|
1562
|
-
|
|
1563
|
-
The half of "never double-post" that *is* achievable is enforced: a report cannot
|
|
1564
|
-
be **concurrently** in flight twice. Claiming a report is a conditional update,
|
|
1565
|
-
so only one attempt can move a `pending` row, and every terminal transition names
|
|
1566
|
-
the attempt it is settling — a request that answers after its row was reclaimed
|
|
1567
|
-
is discarded rather than allowed to overwrite a newer attempt's outcome. That is
|
|
1568
|
-
what stops a retry storm.
|
|
1569
|
-
|
|
1570
|
-
### A report nobody can deliver is itself news
|
|
1571
|
-
|
|
1572
|
-
A report that exhausts its retries is marked `failed` **and escalates as tier 2**.
|
|
1573
|
-
This rides the transport that just failed, which is deliberate and accepted: the
|
|
1574
|
-
common failure is a wrong chat id or a bot kicked from the chat, not a global
|
|
1575
|
-
Telegram outage, and in both of those the page reaches an operator who is
|
|
1576
|
-
otherwise being told nothing at all. If the whole channel is down the page
|
|
1577
|
-
degrades to a line in `daemon.log` and the `reports` block in `status`, which is
|
|
1578
|
-
then the only surface — a report has no tracker issue, so there is no issue
|
|
1579
|
-
comment to fall back to. The page goes through the ordinary escalation ledger and
|
|
1580
|
-
carries the report id, so one undeliverable report pages exactly once.
|
|
1581
|
-
|
|
1582
|
-
### Daily digests are deduplicated from the ledger
|
|
1583
|
-
|
|
1584
|
-
With `digest.cadence: "daily"`, `--kind digest` is accepted at most once per
|
|
1585
|
-
**local** day, per project. The second hand-over on the same day is refused and
|
|
1586
|
-
told which report already holds the slot, including when that report has already
|
|
1587
|
-
been delivered. This is decided from the `reports` table, not from the model's
|
|
1588
|
-
memory of the last tick — a restarted or compacted session cannot send a second
|
|
1589
|
-
daily digest by forgetting the first. A `per-tick` digest carries no daily key,
|
|
1590
|
-
so later ticks can hand off newly accumulated rows. Material reports carry no
|
|
1591
|
-
dedupe key either: two events in a day are two events.
|
|
1592
|
-
|
|
1593
|
-
### What `status` shows
|
|
1594
|
-
|
|
1595
|
-
```text
|
|
1596
|
-
reports 1 pending · 1 sending · 0 failed (delivery is at-least-once — a retry may duplicate)
|
|
1597
|
-
9f2c1ab0d3e4 pending material 12m old attempt 2/6, retry in 1m (telegram sendMessage rejected: {"ok":false,…)
|
|
1598
|
-
4b7c1ad9e001 SENDING digest 3m old attempt 1, outcome unknown — the process that sent it never said; a daemon start retries it and the message will say it may be a repeat
|
|
1599
|
-
```
|
|
1600
|
-
|
|
1601
|
-
`pending` and `sending` are printed differently because they ask different things
|
|
1602
|
-
of you, and every row carries its age — "1 report pending since 08:15Z" is the
|
|
1603
|
-
signal that was missing when the reports went nowhere. Delivered reports leave
|
|
1604
|
-
the block: it is a list of what you are still owed, not a log.
|
|
1605
|
-
|
|
1606
|
-
Delivery keeps running while the fleet is **paused**. Pause stops claiming, not
|
|
1607
|
-
your right to hear about work that already happened. It runs on its own
|
|
1608
|
-
thirty-second timer rather than the five-minute dispatch tick, so a report does
|
|
1609
|
-
not sit in the outbox for the length of a poll interval.
|
|
1610
|
-
|
|
1611
|
-
The tier-2 escalation ledger (`notifications`) is untouched by all of this. It is
|
|
1612
|
-
a bare dedupe key by design — its primary key *is* the key — which is exactly why
|
|
1613
|
-
reports needed a separate table rather than an extension of that one.
|
|
1614
|
-
|
|
1615
|
-
## The decision ledger (#136)
|
|
1616
|
-
|
|
1617
|
-
The outbox above fixed reports the orchestrator sends. This fixes the ones it is
|
|
1618
|
-
**waiting on**. A question put to you — an amendment, a tier-2 decision, "do I
|
|
1619
|
-
ship this tonight?" — lived in exactly one place: the model's context. A
|
|
1620
|
-
compaction, a restart, or a tick that ran long lost the question *and* the fact
|
|
1621
|
-
that one was owed, after which the session either asked again (you answer twice)
|
|
1622
|
-
or dropped it silently (the decision never lands, and nothing anywhere says one
|
|
1623
|
-
is outstanding).
|
|
1624
|
-
|
|
1625
|
-
So questions are written down, and every tick's prompt carries what is still
|
|
1626
|
-
open — read from the store, never from what the session remembers asking:
|
|
1627
|
-
|
|
1628
|
-
```bash
|
|
1629
|
-
omp-conductor decision open --question "ship 0.4.3 tonight?" \
|
|
1630
|
-
--blocks "the release" --resolves-when issue-closed:132
|
|
1631
|
-
omp-conductor decision list
|
|
1632
|
-
omp-conductor decision resolve <id> --answer "yes, after #132 lands"
|
|
1633
|
-
omp-conductor decision withdraw <id> --reason "the release slipped a week"
|
|
1634
|
-
```
|
|
1635
|
-
|
|
1636
|
-
**`--resolves-when` is the part that makes a parked question wake up.** Six
|
|
1637
|
-
conditions, each one something this package can check without asking you:
|
|
1638
|
-
|
|
1639
|
-
| Condition | Met when |
|
|
502
|
+
| `hold [--keep-ticks]` | pause claiming + disarm that project's ticks | every project |
|
|
503
|
+
| `stop [--pane]` | stop that project (daemon, optionally the pane) | every project |
|
|
504
|
+
| `arm` / `disarm` | that project's arm marker | every project's marker |
|
|
505
|
+
| `resume` | clear pause + any `stop --pane` recovery pin | every project |
|
|
506
|
+
|
|
507
|
+
`setup host` writes one arm marker per project (`armed-<project>`), so arming one
|
|
508
|
+
fleet never arms another; `hold --project A` writes `paused-<name>` for that
|
|
509
|
+
project only, while a bare `paused` sentinel pauses every project. Per-project
|
|
510
|
+
tick identity and the shared-marker upgrade are in
|
|
511
|
+
[Orchestrator tick](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#orchestrator-tick).
|
|
512
|
+
|
|
513
|
+
## CLI at a glance
|
|
514
|
+
|
|
515
|
+
Every verb hangs off the single `omp-conductor` binary; there is no in-session
|
|
516
|
+
command. Each takes an optional `--project NAME`, and
|
|
517
|
+
`hold` / `stop` / `arm` / `disarm` / `resume` also take `--all`. The full
|
|
518
|
+
reference — usage strings, flags and per-command behaviour — is in the
|
|
519
|
+
[CLI reference](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#cli-reference).
|
|
520
|
+
|
|
521
|
+
| Verb | What it does |
|
|
1640
522
|
| --- | --- |
|
|
1641
|
-
| `
|
|
1642
|
-
| `
|
|
1643
|
-
| `
|
|
1644
|
-
| `
|
|
1645
|
-
| `
|
|
1646
|
-
| `
|
|
1647
|
-
|
|
1648
|
-
|
|
1649
|
-
|
|
1650
|
-
|
|
1651
|
-
|
|
1652
|
-
|
|
1653
|
-
|
|
1654
|
-
|
|
1655
|
-
|
|
1656
|
-
|
|
1657
|
-
`
|
|
1658
|
-
|
|
1659
|
-
|
|
1660
|
-
|
|
1661
|
-
|
|
1662
|
-
|
|
1663
|
-
|
|
1664
|
-
a condition nobody can check.
|
|
1665
|
-
|
|
1666
|
-
**Expiry is enforced, not remembered.** An unanswered question closes itself
|
|
1667
|
-
after seven days — the deadline the floor's parked-amendment protocol already
|
|
1668
|
-
promised — so the digest stays a list of live questions instead of a graveyard.
|
|
1669
|
-
Answering or withdrawing is explicit, and a second resolution of the same id is
|
|
1670
|
-
refused rather than overwriting the first answer.
|
|
1671
|
-
|
|
1672
|
-
`omp-conductor status` carries one row, `decisions`, reported whether or not
|
|
1673
|
-
anything is open: `decisions 2 open (oldest 26h)`, or `decisions none open`. A
|
|
1674
|
-
row that appeared only when something was outstanding would leave "it forgot to
|
|
1675
|
-
record the question" and "there genuinely is none" looking identical, which is
|
|
1676
|
-
the ambiguity this table exists to remove.
|
|
1677
|
-
|
|
1678
|
-
## Failure classes and recovery by class (#132)
|
|
1679
|
-
|
|
1680
|
-
Every run that did not reach a merged PR used to end at a human. The
|
|
1681
|
-
orchestrator re-derived the same triage on each tick — read the row, read the
|
|
1682
|
-
PR's checks, decide whether to requeue, re-run, settle or escalate — and then
|
|
1683
|
-
threw the conclusion away. Measured on this fleet's own history: **half the
|
|
1684
|
-
spend produced no merged PR**, and a large share of it was not implementation
|
|
1685
|
-
failure at all but daemon restarts, cancelled runners and a base branch moving
|
|
1686
|
-
under a green PR.
|
|
1687
|
-
|
|
1688
|
-
So the daemon classifies each terminal non-success before the next dispatch,
|
|
1689
|
-
persists the class on the row, and performs the one recovery that class names.
|
|
1690
|
-
|
|
1691
|
-
| Class | Signals | Recovery | Budget |
|
|
1692
|
-
| --- | --- | --- | --- |
|
|
1693
|
-
| `env-start-failure` | turn 0 plus an explicit harness start error (`No model selected`, a rejected key) | escalate — the session never read the issue | none |
|
|
1694
|
-
| `settlement-stuck` | a row carrying a PR that has since merged | settle: release the label, mark the row merged | none |
|
|
1695
|
-
| `returned-for-revision` | a `pushed-green` or `pushed-pending` PR was closed without merging | none — preserve the review decision for a human re-queue | continuation |
|
|
1696
|
-
| `merge-conflict` | `pushed-green`, PR open, GitHub reports conflicting | requeue for a rebase continuation | continuation |
|
|
1697
|
-
| `question` | the worker stopped to ask something (`blocked`) | escalate, carrying the worker's own report as evidence | none |
|
|
1698
|
-
| `orphan-dirty` | orphaned with a failed salvage and no operator ack | hold — recorded only; the tree is the only copy | none |
|
|
1699
|
-
| `orphan-clean` | orphaned with nothing uncommitted | requeue | continuation |
|
|
1700
|
-
| `turn-cap-progress` | at the turn ceiling **with** a PR, head or salvage commit | continue from the branch | continuation |
|
|
1701
|
-
| `turn-cap-spinning` | at the ceiling with no PR and no commits | escalate with the last tool calls the transcript recorded — and the completion path deliberately does **not** requeue it | none |
|
|
1702
|
-
| `admin-kill` | killed *below* its own ceiling — a restart or a drain | requeue | none |
|
|
1703
|
-
| `ci-infra` | PR open, every unresolved check cancelled / timed out / stale | re-run the failed jobs | none |
|
|
1704
|
-
| `ci-deterministic` | PR open, a check genuinely reports `FAILURE` | escalate with the failing check names and links | failed attempt |
|
|
1705
|
-
| `dispatch-infra` | the conductor's own Git path failed before the worker's first turn | requeue, bounded by per-class strikes | none |
|
|
1706
|
-
| `provider-credit` | the provider refused the run for credit (HTTP 402, or its own out-of-credit text read off the transcript) | pause the fleet and require `omp-conductor resume` once the provider has credit | none |
|
|
1707
|
-
| `provider-transient` | the provider aborted a request stream before the run produced a verdict | requeue, bounded by per-class strikes | none |
|
|
1708
|
-
| `unknown` | anything unrecognised | escalate | as recorded |
|
|
1709
|
-
|
|
1710
|
-
**Unknown escalates; it never silently retries.** A shape this table does not
|
|
1711
|
-
recognise is a gap in the table, and a quiet requeue would spend a budget on a
|
|
1712
|
-
cause nobody has named — the behaviour this exists to end.
|
|
1713
|
-
|
|
1714
|
-
### The budgets follow the cause
|
|
1715
|
-
|
|
1716
|
-
`failuresFor` (implementation attempts) excludes `ci-infra`, `settlement-stuck`,
|
|
1717
|
-
`env-start-failure`, `dispatch-infra`, `provider-credit`, `provider-transient`
|
|
1718
|
-
and `returned-for-revision`. `continuationsFor` excludes `admin-kill`,
|
|
1719
|
-
`settlement-stuck`, `env-start-failure`, `dispatch-infra`, `provider-credit`
|
|
1720
|
-
and `provider-transient`, but explicitly counts a failed
|
|
1721
|
-
`returned-for-revision` row. Environment, dispatch and provider faults charge
|
|
1722
|
-
neither budget because the issue did not receive a valid implementation
|
|
1723
|
-
attempt. A merge conflict and a returned review both charge a continuation:
|
|
1724
|
-
each asks for more work, but neither is a failed implementation attempt.
|
|
1725
|
-
|
|
1726
|
-
An **unclassified** row (every row written before 0.4.3) counts exactly as it
|
|
1727
|
-
did before classification existed. Upgrading therefore changes no existing
|
|
1728
|
-
budget: the columns are additive and nullable, and a pre-0.4.3 `conductor.db`
|
|
1729
|
-
opens unchanged.
|
|
1730
|
-
|
|
1731
|
-
### Stale labels are reconciled
|
|
1732
|
-
|
|
1733
|
-
On 2026-08-09 four issues carried `agent:failed` while every one of them was
|
|
1734
|
-
already complete — residue of a turns-cap kill two days earlier that nothing in
|
|
1735
|
-
the loop ever revisited. The board counted four phantom failures while the
|
|
1736
|
-
genuinely stuck issues were invisible.
|
|
1737
|
-
|
|
1738
|
-
Each tick now reconciles the three state labels against the tracker:
|
|
1739
|
-
|
|
1740
|
-
- A **closed** issue never keeps an `agent:*` label.
|
|
1741
|
-
- An **open** issue carrying `failed` whose sub-issues have *all* closed loses
|
|
1742
|
-
the label and gets one comment naming them, deduplicated through the same
|
|
1743
|
-
notifications ledger escalations use.
|
|
1744
|
-
|
|
1745
|
-
Positive evidence only: a tracker that cannot list answers empty, and an empty
|
|
1746
|
-
answer removes nothing — the label is the interlock that keeps two workers off
|
|
1747
|
-
one issue.
|
|
1748
|
-
|
|
1749
|
-
### Where you see it
|
|
1750
|
-
|
|
1751
|
-
- `omp-conductor status` grows a `failure classes (unrecovered)` block, counting
|
|
1752
|
-
only rows whose recovery has *not* run. Classes rather than row states,
|
|
1753
|
-
because a row state is not an issue state.
|
|
1754
|
-
- The board appends `[<class>]` to a card whose newest run carries one.
|
|
1755
|
-
- The tick prompt carries one line — `Auto-recovered since last tick: 3
|
|
1756
|
-
(merge-conflict #365, admin-kill #82, …) — already handled, do not re-triage
|
|
1757
|
-
these.` — so the orchestrator stops writing that paragraph by re-deriving it.
|
|
1758
|
-
|
|
1759
|
-
## Configuration
|
|
1760
|
-
|
|
1761
|
-
The config lives at `$OMP_CONDUCTOR_HOME/config.json`, or
|
|
1762
|
-
`~/.omp/conductor/config.json` when that variable is unset. It is written with mode
|
|
1763
|
-
`0600` in a directory created `0700`, because it carries chat ids and clone URLs.
|
|
1764
|
-
That same directory holds the SQLite store (`conductor.db`), the `paused` sentinel,
|
|
1765
|
-
the `sessions/` worker transcripts, the `orchestrator/` session directory,
|
|
1766
|
-
`backups/briefs/` for timestamped brief and policy safety copies, and
|
|
1767
|
-
`release-policy-blocks.jsonl`, the append-only audit of mechanically rejected
|
|
1768
|
-
release/deploy calls.
|
|
1769
|
-
|
|
1770
|
-
Runtime state lives elsewhere, under `$OMP_CONDUCTOR_RUNTIME_DIR` (default
|
|
1771
|
-
`~/.omp/run/daemons/omp-conductor`): `daemon.json`, a mode-`0600` pidfile written
|
|
1772
|
-
atomically, and `daemon.log`, appended across every boot so the previous failure is
|
|
1773
|
-
still there when you go looking. It is kept apart from the config directory because
|
|
1774
|
-
it is meaningless after a reboot, and the pidfile's liveness is probed on every
|
|
1775
|
-
read — a stale one never blocks a `start`. Both `start` and a bare `daemon` write
|
|
1776
|
-
the pidfile, so a daemon run in the foreground under systemd is as visible to
|
|
1777
|
-
`status` as a backgrounded one; `daemon --once` writes nothing, because that drill
|
|
1778
|
-
is exactly what the orphan-reconciliation guard reads the pidfile to protect.
|
|
1779
|
-
|
|
1780
|
-
The file is validated on every read. A malformed config produces one readable error
|
|
1781
|
-
listing every fault, and the daemon refuses to start rather than running with half
|
|
1782
|
-
a project.
|
|
1783
|
-
|
|
1784
|
-
The same vocabulary the loader enforces ships as a JSON Schema at
|
|
1785
|
-
`schema/config.schema.json` in the installed package (draft 2020-12). Anything
|
|
1786
|
-
`saveConfig` writes carries a top-level `"$schema"` reference to that installed
|
|
1787
|
-
copy (resolved from the package's own location, so it points at a real file),
|
|
1788
|
-
which lets an editor that understands JSON Schema validate a hand-edited config
|
|
1789
|
-
as you type; a config without the key is just as valid. Regenerate the shipped
|
|
1790
|
-
schema from `ConfigSchema` (`src/config-schema.ts`) with:
|
|
1791
|
-
|
|
1792
|
-
```sh
|
|
1793
|
-
bun run schema
|
|
1794
|
-
```
|
|
1795
|
-
|
|
1796
|
-
and commit the resulting `schema/config.schema.json`. CI's `bun test` fails if the
|
|
1797
|
-
checked-in schema drifts from what the code renders, so you cannot forget the step.
|
|
1798
|
-
|
|
1799
|
-
`omp-conductor setup` is the only thing here that writes this file, and on a project
|
|
1800
|
-
it already knows it can rewrite one area of it without re-asking the rest — see
|
|
1801
|
-
[Changing one setting](#changing-one-setting).
|
|
1802
|
-
|
|
1803
|
-
`version` is `2`. A `version: 1` file still loads: caps it names that this build no
|
|
1804
|
-
longer enforces are dropped rather than treated as typos, and the next save writes
|
|
1805
|
-
it back as `2`. In a `version: 2` file an unrecognised cap key **is** an error,
|
|
1806
|
-
because there is nothing left to retire — a mistyped `dailySpendUSD` would
|
|
1807
|
-
otherwise read as configured while the real ceiling stayed the default.
|
|
1808
|
-
|
|
1809
|
-
A complete, valid config for one project with two target repos:
|
|
1810
|
-
|
|
1811
|
-
```json
|
|
1812
|
-
{
|
|
1813
|
-
"version": 2,
|
|
1814
|
-
"defaults": {
|
|
1815
|
-
"maxConcurrentWorkers": 2,
|
|
1816
|
-
"dailySpendUsd": 25,
|
|
1817
|
-
"planUsage": { "windowId": "anthropic:7d", "maxUsedFraction": 0.85 },
|
|
1818
|
-
"workerMaxTurns": 120,
|
|
1819
|
-
"workerWallClockMs": 5400000,
|
|
1820
|
-
"maxAttemptsPerIssue": 2,
|
|
1821
|
-
"maxContinuationsPerIssue": 2
|
|
1822
|
-
},
|
|
1823
|
-
"projects": [
|
|
1824
|
-
{
|
|
1825
|
-
"name": "demo",
|
|
1826
|
-
"tracker": { "kind": "github", "repo": "acme/planning" },
|
|
1827
|
-
"queueLabel": "ready-for-agent",
|
|
1828
|
-
"stateLabels": {
|
|
1829
|
-
"inProgress": "agent:in-progress",
|
|
1830
|
-
"blocked": "agent:blocked",
|
|
1831
|
-
"failed": "agent:failed"
|
|
1832
|
-
},
|
|
1833
|
-
"routing": {
|
|
1834
|
-
"labelPrefix": "repo:",
|
|
1835
|
-
"repos": {
|
|
1836
|
-
"api": {
|
|
1837
|
-
"name": "api",
|
|
1838
|
-
"cloneUrl": "git@github.com:acme/api.git",
|
|
1839
|
-
"defaultBranch": "main",
|
|
1840
|
-
"gates": [
|
|
1841
|
-
{ "cmd": "bun run lint", "cwd": "." },
|
|
1842
|
-
{ "cmd": "bun test", "cwd": "." }
|
|
1843
|
-
],
|
|
1844
|
-
"graphProject": "~/.cache/conductor-graph/acme/api",
|
|
1845
|
-
"migrations": { "dir": "backend/alembic/versions" },
|
|
1846
|
-
"release": { "versionFile": "omp/package.json" }
|
|
1847
|
-
},
|
|
1848
|
-
"worker": {
|
|
1849
|
-
"name": "worker",
|
|
1850
|
-
"cloneUrl": "git@github.com:acme/worker.git",
|
|
1851
|
-
"defaultBranch": "main",
|
|
1852
|
-
"gates": [
|
|
1853
|
-
{ "cmd": "ruff check .", "cwd": "." },
|
|
1854
|
-
{ "cmd": "pytest -q", "cwd": "backend" }
|
|
1855
|
-
]
|
|
1856
|
-
}
|
|
1857
|
-
}
|
|
1858
|
-
},
|
|
1859
|
-
"caps": {
|
|
1860
|
-
"maxConcurrentWorkers": 1,
|
|
1861
|
-
"dailySpendUsd": 15
|
|
1862
|
-
},
|
|
1863
|
-
"workerModel": "smol",
|
|
1864
|
-
"escalation": {
|
|
1865
|
-
"telegramChatId": "123456789",
|
|
1866
|
-
"telegramTopicId": 8713,
|
|
1867
|
-
"fallbackToIssueComment": true,
|
|
1868
|
-
"orchestrator": "embedded"
|
|
1869
|
-
},
|
|
1870
|
-
"authority": {
|
|
1871
|
-
"merge": "human",
|
|
1872
|
-
"release": "human"
|
|
1873
|
-
},
|
|
1874
|
-
"releasePolicy": {
|
|
1875
|
-
"version-bump-pr": "human",
|
|
1876
|
-
"git-tag": "human",
|
|
1877
|
-
"git-push-tags": "human",
|
|
1878
|
-
"package-publish": "human",
|
|
1879
|
-
"github-release": "human",
|
|
1880
|
-
"deploy": "human"
|
|
1881
|
-
},
|
|
1882
|
-
"policy": {
|
|
1883
|
-
"merge": {
|
|
1884
|
-
"requiredChecks": ["build", "lint"],
|
|
1885
|
-
"baseFreshness": "up-to-date",
|
|
1886
|
-
"drafts": "block",
|
|
1887
|
-
"whenBehindBase": "update-branch"
|
|
1888
|
-
},
|
|
1889
|
-
"release": {
|
|
1890
|
-
"requires": ["runs-settled", "no-open-prs"],
|
|
1891
|
-
"requiredChecks": ["release"],
|
|
1892
|
-
"artefacts": ["@acme/sdk"],
|
|
1893
|
-
"environments": ["staging"]
|
|
1894
|
-
}
|
|
1895
|
-
},
|
|
1896
|
-
"recoveryMerges": [
|
|
1897
|
-
{
|
|
1898
|
-
"prUrl": "https://github.com/acme/api/pull/381",
|
|
1899
|
-
"headSha": "9a783d8f17071d63f2d5d764d43a29837c365920",
|
|
1900
|
-
"reason": "operator-instructed"
|
|
1901
|
-
}
|
|
1902
|
-
],
|
|
1903
|
-
"reporting": {
|
|
1904
|
-
"scope": "material"
|
|
1905
|
-
},
|
|
1906
|
-
"workspaceRoot": "~/.omp/conductor/worktrees",
|
|
1907
|
-
"mirrorRoot": "~/.omp/conductor/mirrors"
|
|
1908
|
-
}
|
|
1909
|
-
]
|
|
1910
|
-
}
|
|
1911
|
-
```
|
|
1912
|
-
|
|
1913
|
-
Field notes:
|
|
1914
|
-
|
|
1915
|
-
| Field | Notes |
|
|
1916
|
-
| --- | --- |
|
|
1917
|
-
| `version` | Must be `2`. A `version: 1` file still loads, drops the caps this build no longer enforces, and is rewritten as `2` on the next save. Present from day one so a format change can be migrated instead of silently misread. |
|
|
1918
|
-
| `defaults` | Every `Caps` field. Anything omitted falls back to the built-in default. |
|
|
1919
|
-
| `tracker.repo` | `owner/repo`. `tracker.kind` may be omitted; `"github"` is the only accepted value. |
|
|
1920
|
-
| `queueLabel` | The one label meaning "a human has signed this off as agent-ready". Matched exactly, case-sensitively. |
|
|
1921
|
-
| `release.versionFile` | Optional, per repo: a repo-relative JSON file with a top-level string `version`, such as `omp/package.json`. Declares that tags must match the version already landed on the live default branch. A delegated `git-tag` for such a repo requires delegated `version-bump-pr` too; otherwise config loading fails with the missing preparation path instead of granting an impossible release. Absolute paths and `..` are refused. |
|
|
1922
|
-
| `groomBelow` | Optional; default `4`. Routable candidates below this count make the orchestrator's tick prompt say the queue is running low and to groom it (Duty 2). An integer ≥ 1; anything else degrades to the default. |
|
|
1923
|
-
| `stateLabels` | Optional; defaults to `agent:in-progress`, `agent:blocked`, `agent:failed`. |
|
|
1924
|
-
| `routing.labelPrefix` | Optional; defaults to `repo:`. |
|
|
1925
|
-
| `routing.repos` | At least one entry, or nothing can be routed. `name` defaults to the map key, `defaultBranch` to `main`. |
|
|
1926
|
-
| `gates` | The exact cheap commands CI also runs, each with the `cwd` it runs from (`cwd` defaults to `.`). Running the real gate locally is what makes an unattended push safe — a subset lets an error outside the source dir reach the runners. |
|
|
1927
|
-
| `graphProject` | Optional, per repo. Absolute path of the **index-only clone** whose code graph this repo's workers query — conductor's own disposable clone, pinned to the repo's default branch, never a checkout you work in and never a worker's worktree. Written by the wizard; `~` is expanded, and a relative path is an error rather than something resolved against whichever cwd happened to read the file. Absent means this repo has no graph and its briefs say nothing about one. See [Code-graph discovery](#code-graph-discovery). |
|
|
1928
|
-
| `migrations` | Optional, per repo: `{ "dir": "backend/alembic/versions" }`. Names the repo-relative directory of an Alembic-style ordered migration chain (`revision` / `down_revision` in `*.py`). When set, `conductor_pr_merge` **refuses** a merge that would corrupt the chain at the base tip: reusing a revision id another file already declares, deleting a published migration, or a merge that would leave the combined graph with more than one head (so a stale parent is refused, and a fork-repair merge migration that unifies the heads passes). Absent means the repo opts out of the chain check entirely. Repo-relative only: a leading `/` or `..` is an error. |
|
|
1929
|
-
| `caps` | Per-project overrides; omit it or pin only the fields you want to change. |
|
|
1930
|
-
| `escalation.fallbackToIssueComment` | Defaults to `true`. Absent means "yes, still tell me". |
|
|
1931
|
-
| `escalation.telegramTopicId` | Optional forum topic for everything conductor sends: tier-2 pages, reports, digests, arm challenges, `omp-conductor message`. Setup offers the topics omp-telegram has claimed, naming each one's herdr space. The bridge re-claims a pane's topic across restarts — including the restarts `upgrade` and `restart` perform — so a pinned id that is no longer claimed is replaced at send time by the live claim whose **herdr space** is this project, falling back to one titled for the project, logged without ids (#407, #412). The space is read first because the bridge titles a topic `ownAgentName ?? basename(cwd)`, and a multi-project host whose panes sit under one state directory gives every claim the same title. An identity two claims share is treated as no match at all rather than a guess. A pin that is still claimed always wins, so a deliberately separate topic is never hijacked. Absent keeps flat-chat behaviour. |
|
|
1932
|
-
| `escalation.orchestrator` | Optional; `"embedded"` (default) or `"external"`. `external` means an orchestrator session already runs elsewhere: the daemon starts none, and tier-1 escalations post as issue comments for that session to drain. Any other value is an error. |
|
|
1933
|
-
| `authority` | Optional; `{ "merge": …, "release": … }`, each `"human"` (default) or `"orchestrator"`. It grants nothing to the daemon — it words the orchestrator's standing orders and the Releases paragraph of the rendered brief, so the config and the prompt cannot disagree about who holds the merge button. Unknown keys and any other value are errors, never folded to the default. |
|
|
1934
|
-
| `releasePolicy` | Optional; a per-shape map whose values are `"human"` (default) or `"orchestrator"`. Shapes are `version-bump-pr`, `git-tag`, `git-push-tags`, `package-publish`, `github-release`, and `deploy`. The legacy `"none"` denies every shape; legacy `"operator-brief"` grants the artifact-producing shapes, including reviewed version preparation, but keeps deploy human-owned. The in-session tripwire blocks recognised raw release/deploy calls before execution. Every rejection is written to `release-policy-blocks.jsonl`; the heartbeat carries that day's count into the daily digest. This is the mechanical gate; `authority.release` still says who owns the decision. |
|
|
1935
|
-
| `recoveryMerges` | Optional, hand-edited recovery authority for a PR that has no conductor run record. Each entry is an exact `{ prUrl, headSha, reason: "operator-instructed" }` tuple. When all three values match, `conductor_pr_merge` may merge that one PR even while the fleet is held and even when standing merge authority is `"human"`. It still requires a routed project repo, an open PR at that exact live head, green checks, the migration-chain guard, and the single-flight lock—the same safety path as an ordinary merge. Duplicate PR URLs and malformed values make config loading fail closed. Setup preserves entries but never creates them. Remove an entry after the recovery is complete. |
|
|
1936
|
-
| `reporting` | Optional; a **legacy scope preset** (`reporting.scope` — `"material"` default, `"decisions"`, `"escalations"`) or the **explicit form** `{ "interruptOn": [...], "digest": { ... }, "availability": { ... } }`. The preset writes which categories may page the operator (`interruptOn`) and when the rollup happens (`digest.cadence`); the explicit form sets both directly and may add a weekly operator-availability window. The two forms are mutually exclusive in one config. See [Reporting policy](#reporting-policy-reporting). |
|
|
1937
|
-
| `orchestratorReadPaths` | **Retired in 0.4.3.** Still accepted in a config and ignored, so a fleet carrying it upgrades without an edit. It widened the orchestrator's file-tool allowlist; there is no allowlist any more — the orchestrator is [unconfined by design](#the-orchestrator-is-unconfined-deliberately). |
|
|
1938
|
-
| `policy` | Optional; the gating conditions a merge or a release must satisfy, in two sections — `policy.merge` and `policy.release`. Any member may be omitted and the loader fills it from the strict default; an unknown key in either section, or a value outside its vocabulary, is an error naming the field, never a silent downgrade. See [Merge and release preconditions](#merge-and-release-preconditions-policy). |
|
|
1939
|
-
| `workspaceRoot` / `mirrorRoot` | Optional; default to `worktrees/` and `mirrors/` under the state directory. `~` is expanded. |
|
|
1940
|
-
|
|
1941
|
-
Prefer an SSH `cloneUrl`, or an https URL backed by a credential helper. A clone URL
|
|
1942
|
-
with credentials embedded is persisted into the mirror's git config, exactly as it
|
|
1943
|
-
would be for a hand-run clone.
|
|
1944
|
-
|
|
1945
|
-
### Reporting policy (`reporting`)
|
|
1946
|
-
|
|
1947
|
-
What may interrupt the operator's phone, and when the daily rollup happens. Two
|
|
1948
|
-
spellings, mutually exclusive in one config (the loader rejects a `scope` next to
|
|
1949
|
-
`interruptOn`/`digest`):
|
|
1950
|
-
|
|
1951
|
-
- **Preset** — `reporting.scope`, the three legacy values, mapped verbatim:
|
|
1952
|
-
- `material` (default) → `interruptOn: [tier2, decision-needed, fleet-stopped, confirmed-failure, material]`, digest `per-tick`.
|
|
1953
|
-
- `decisions` → `interruptOn: [tier2, decision-needed, fleet-stopped]`, digest `per-tick`.
|
|
1954
|
-
- `escalations` → `interruptOn: [tier2, fleet-stopped]`, digest `daily` (model-timed).
|
|
1955
|
-
- **Explicit** — `reporting: { "interruptOn": ["tier2", "fleet-stopped", ...], "digest": { "cadence": "none" | "per-tick" | "daily" } }`.
|
|
1956
|
-
`interruptOn` must be a non-empty array of known categories (`tier2`, `decision-needed`, `fleet-stopped`, `confirmed-failure`, `material`), each an escalation's tier-2 category. `daily` may add `at` (`HH:MM`, 24h) and `timezone` (a known IANA zone, defaulting to the host zone) — both only valid with `daily`.
|
|
1957
|
-
|
|
1958
|
-
The explicit form may add a weekly local-time window:
|
|
523
|
+
| `setup [area]` | The wizard: interview + probes. `--no-ai`, per-area amend, `host`, `graph`. |
|
|
524
|
+
| `start` / `stop` / `restart` | Run the dispatch daemon. `restart` drains first; `stop --pane` also halts the pane. |
|
|
525
|
+
| `upgrade [--to VERSION]` | Pin one published npm release across CLI, omp plugin, Herdr plugin and brief. |
|
|
526
|
+
| `status` | Layered fleet report: dispatch, ticks, pane, recovery, telegram, daemon, then the project body. |
|
|
527
|
+
| `stats [--since 7d|30d|YYYY-MM-DD] [--json]` | What the fleet accomplished and at what cost, from the local store only: merges, merge rate, lead time, attempts and metered cost per merged issue, failure classes, gh calls. |
|
|
528
|
+
| `doctor [--json] [--probe-telegram]` | Read-only deployment health — gh auth, exact-case labels, systemd drift, config backup, sqlite integrity, spend telemetry, timezones, Telegram. Run after install and after every upgrade; exit 0 only when nothing failed. |
|
|
529
|
+
| `ledger` | The action audit — every mediated verb, refusal and turn budget. |
|
|
530
|
+
| `board` | Live terminal kanban from Queue to Settled. |
|
|
531
|
+
| `dashboard [--port N] [--host ADDR]` | Browser UI plus bearer-authenticated `/api/projects`: every project's daemon state, port and healthz. Loopback by default; the token lives at `<state>/dashboard-token`. |
|
|
532
|
+
| `hold [--keep-ticks]` | Pause claims and disarm ticks — the soft stop. |
|
|
533
|
+
| `arm` / `disarm` | Gate / clear the orchestrator tick's arm marker. |
|
|
534
|
+
| `tail <issue>` | Follow a live worker's transcript. |
|
|
535
|
+
| `extend <issue> --turns N` | Raise one run's, or one next attempt's, turn ceiling. |
|
|
536
|
+
| `worker pause/resume/stop <issue>` | Park or end one live worker. |
|
|
537
|
+
| `unblock <issue>` | Clear blocked/failed/`in-progress` so a settled issue can be re-claimed. |
|
|
538
|
+
| `verb <conductor_*>` | Run a mediated verb from the CLI (external orchestration). |
|
|
539
|
+
| `friction <kind> --detail TEXT` | Record a bounded operator judgment — an escalation that belonged in a digest, or a report that was noise/surprising — feeding the learning loop. |
|
|
540
|
+
| `event record` / `report` / `message` | Record and deliver reports and messages through the durable outbox. |
|
|
541
|
+
| `decision open/resolve/withdraw/list` | Read and answer questions in the decision ledger. |
|
|
542
|
+
| `intake "<text>"` / `intake list` / `intake dismiss <id>` | Capture a raw idea durably (it lives in the store and survives restarts), list what is still pending, dismiss what turned out to be nothing. |
|
|
543
|
+
| `daemon [--once]` | Run the loop in the foreground — systemd's entry point. |
|
|
544
|
+
| `resume` | Clear pause and any `stop --pane` recovery pin; never re-arms. |
|
|
545
|
+
| `brief-upgrade` | Report / migrate / retrofit the `ORCHESTRATOR.md` ↔ `POLICY.md` overlay. |
|
|
1959
546
|
|
|
1960
|
-
|
|
1961
|
-
{
|
|
1962
|
-
"reporting": {
|
|
1963
|
-
"interruptOn": ["tier2", "fleet-stopped"],
|
|
1964
|
-
"digest": { "cadence": "daily", "at": "17:00", "timezone": "Europe/London" },
|
|
1965
|
-
"availability": {
|
|
1966
|
-
"timezone": "Europe/London",
|
|
1967
|
-
"days": ["mon", "tue", "wed", "thu", "fri"],
|
|
1968
|
-
"start": "09:00",
|
|
1969
|
-
"end": "17:00",
|
|
1970
|
-
"bypass": ["fleet-stopped"]
|
|
1971
|
-
}
|
|
1972
|
-
}
|
|
1973
|
-
}
|
|
1974
|
-
```
|
|
1975
|
-
|
|
1976
|
-
`timezone` must be a known IANA zone. For a daily digest, its timezone defaults
|
|
1977
|
-
to this value and must match it when both are set.
|
|
1978
|
-
|
|
1979
|
-
`days` is a non-empty set of `mon` through `sun`; `start` is inclusive and
|
|
1980
|
-
`end` is exclusive. A start later than the end defines an overnight window on
|
|
1981
|
-
the day it opens. `bypass` is an explicit list of known interrupt categories
|
|
1982
|
-
that may still page outside the window; it may be empty. For ordinary notices,
|
|
1983
|
-
a bypass has no effect on a category omitted from `interruptOn`. Urgent recovery
|
|
1984
|
-
notices may bypass category batching when the digest loop itself is unavailable,
|
|
1985
|
-
but they still require the configured availability bypass outside the window.
|
|
1986
|
-
|
|
1987
|
-
The setup wizard offers this as **Weekly availability window** and asks for the
|
|
1988
|
-
zone, days, start/end, bypass categories, and digest schedule: every tick,
|
|
1989
|
-
model-timed daily, disabled, or a fixed daily `HH:MM`. Re-running setup or
|
|
1990
|
-
amending reporting preselects and preserves the configured `none`, `per-tick`,
|
|
1991
|
-
or `daily` cadence. Choosing **Continuous (24-hour interrupts)** is the explicit
|
|
1992
|
-
opt-out and preserves the behavior of every existing config; an absent
|
|
1993
|
-
`availability` key also means continuous operation.
|
|
1994
|
-
|
|
1995
|
-
Outside the window, an otherwise interruptible escalation is stored durably
|
|
1996
|
-
instead of sent. A daily digest may consume it first. Otherwise the daemon
|
|
1997
|
-
atomically queues one working-hours catch-up report when the window opens,
|
|
1998
|
-
including after downtime; associating the held rows before delivery prevents a
|
|
1999
|
-
later tick from authoring a duplicate. Each heartbeat prompt names the
|
|
2000
|
-
mechanically computed current mode and next transition. `status` shows the same
|
|
2001
|
-
state plus the next digest opportunity (`due now`, every tick, disabled, or its
|
|
2002
|
-
next operator-local timestamp). Config, escalation routing, and report transport
|
|
2003
|
-
are re-read at tick or send time, so changing the window or Telegram target
|
|
2004
|
-
does not require a daemon restart.
|
|
2005
|
-
|
|
2006
|
-
Attachment-bearing autonomous Telegram sends cannot be replayed by the text
|
|
2007
|
-
digest, so they are blocked with an explicit “nothing sent or held” error rather
|
|
2008
|
-
than silently dropping their files.
|
|
2009
|
-
|
|
2010
|
-
A tier-2 escalation whose category is **not** in `interruptOn` is not dropped: it
|
|
2011
|
-
is held (`held_notices`) and the next accepted digest is its delivery authority.
|
|
2012
|
-
A `daily` digest is at-most-once per local day (`digest:<YYYY-MM-DD>` in the
|
|
2013
|
-
configured zone), which remains the delivery authority across restarts.
|
|
2014
|
-
`per-tick` digests are not daily-deduplicated, so a later tick can claim newly
|
|
2015
|
-
accumulated rows. A scheduled `daily` digest is only sent on a day it has not
|
|
2016
|
-
already run, once the local clock has passed `at`; a restart after `at` still
|
|
2017
|
-
sends today's (one catch-up), and a fully missed day is skipped, never sent late.
|
|
2018
|
-
|
|
2019
|
-
### Merge and release preconditions (`policy`)
|
|
2020
|
-
|
|
2021
|
-
These used to be sentences in your `POLICY.md`: when a PR may be merged, what
|
|
2022
|
-
must be green, what a release requires. Prose cannot be checked, so every tick
|
|
2023
|
-
re-decided them by reading and interpreting them again. They are configuration
|
|
2024
|
-
now, `POLICY.md` keeps only judgement, and the rendered brief *describes* the
|
|
2025
|
-
policy instead of restating it — no threshold lives in two places.
|
|
2026
|
-
|
|
2027
|
-
`policy.merge`:
|
|
2028
|
-
|
|
2029
|
-
| Field | Values | Default | Means |
|
|
2030
|
-
| --- | --- | --- | --- |
|
|
2031
|
-
| `requiredChecks` | any check names | `[]` | Checks that must have concluded successfully. **Empty is the strict answer** — it means every check the PR reports, not "no checks". |
|
|
2032
|
-
| `baseFreshness` | `up-to-date`, `any` | `up-to-date` | Whether the head must be level with the base branch. `any` accepts a verdict produced against an older base. |
|
|
2033
|
-
| `drafts` | `block`, `allow` | `block` | Whether a draft PR can be merged at all. |
|
|
2034
|
-
| `whenBehindBase` | `update-branch`, `hold`, `escalate` | `update-branch` | What to do with a green PR that fell behind. `update-branch` runs `gh pr update-branch` and waits for the fresh run. Closing it and an admin bypass are not spellable. |
|
|
2035
|
-
|
|
2036
|
-
`policy.release`:
|
|
2037
|
-
|
|
2038
|
-
| Field | Values | Default | Means |
|
|
2039
|
-
| --- | --- | --- | --- |
|
|
2040
|
-
| `requires` | `runs-settled`, `no-open-prs`, `queue-drained`, `base-branch-green`, `epic-children-closed` | `["runs-settled"]` | What must already have landed. `base-branch-green` requires the current live head's push-triggered workflow verdict for that routed repository to be green; pending, unknown, red, or no observation refuses release. Order and duplicates do not matter; the loader canonicalises. |
|
|
2041
|
-
| `requiredChecks` | any check names | `[]` | Checks that must be green on the branch being released. Empty means every check it reports. |
|
|
2042
|
-
| `artefacts` | any names | `[]` | The packages or images this project releases. **Empty denies**: nothing has been authorised to ship. |
|
|
2043
|
-
| `environments` | any names | `[]` | Deploy targets. **Empty denies** every environment. |
|
|
2044
|
-
|
|
2045
|
-
A project with no `policy` block loads as the whole default above, which is the
|
|
2046
|
-
strictest reading of the prose it replaced. `omp-conductor setup` asks for all of
|
|
2047
|
-
it under the **merge & release preconditions** area, so changing one condition
|
|
2048
|
-
costs eight prompts rather than a hand-edit — see
|
|
2049
|
-
[Changing one setting](#changing-one-setting).
|
|
2050
|
-
|
|
2051
|
-
This key grants nothing. Who *may* merge or release is
|
|
2052
|
-
[`authority`](#configuration), and which release tool calls are mechanically
|
|
2053
|
-
permitted is [`releasePolicy`](#configuration). `policy` says what must be true
|
|
2054
|
-
before the act, whoever is doing it.
|
|
2055
|
-
|
|
2056
|
-
#### Reasons are a closed vocabulary
|
|
2057
|
-
|
|
2058
|
-
Where an automated verb takes a `reason`, the argument is one value out of a
|
|
2059
|
-
fixed set, not free text — a reason a rule matches on is a reason that decides,
|
|
2060
|
-
and a decision made out of a model's own wording is one no two runs spell the
|
|
2061
|
-
same way. A reason outside its set is refused, and the refusal names every
|
|
2062
|
-
accepted value.
|
|
2063
|
-
|
|
2064
|
-
| Verb | Accepted reasons |
|
|
2065
|
-
| --- | --- |
|
|
2066
|
-
| merge | `preconditions-met`, `behind-base-refreshed`, `operator-instructed`, `release-blocking` |
|
|
2067
|
-
| release | `batch-complete`, `epic-closed`, `hotfix`, `operator-instructed` |
|
|
2068
|
-
| label change | `promoted-to-queue`, `re-briefed`, `needs-human`, `duplicate`, `superseded`, `out-of-scope` |
|
|
2069
|
-
|
|
2070
|
-
Free-form rationale still has a home: it rides alongside as a separate
|
|
2071
|
-
`rationale` field, is written into the audit trail verbatim, and is never
|
|
2072
|
-
parsed or matched by anything.
|
|
2073
|
-
|
|
2074
|
-
## Orchestrator tick
|
|
2075
|
-
|
|
2076
|
-
The escalation path above assumes an orchestrator session that is actually
|
|
2077
|
-
running its loop. A 24/7 omp session with a standing brief and nobody typing into
|
|
2078
|
-
it never gets prompted, so it never runs anything. Installing
|
|
2079
|
-
`omp plugin install omp-conductor` also installs a heartbeat that prompts it.
|
|
2080
|
-
|
|
2081
|
-
The heartbeat is **inert unless the session cwd contains
|
|
2082
|
-
`.conductor-tick.json`**, so an ordinary session has no timer. `omp-conductor setup`
|
|
2083
|
-
writes this file for external orchestration. A manual configuration has this form:
|
|
2084
|
-
|
|
2085
|
-
```json
|
|
2086
|
-
{
|
|
2087
|
-
"intervalSeconds": 900,
|
|
2088
|
-
"project": "fleet",
|
|
2089
|
-
"armedFile": "/home/fleet/.omp/conductor/armed-fleet",
|
|
2090
|
-
"accessFile": "/home/fleet/.omp/agent/telegram/access.json",
|
|
2091
|
-
"message": "Run your standing loop from ORCHESTRATOR.md now."
|
|
2092
|
-
}
|
|
2093
|
-
```
|
|
2094
|
-
|
|
2095
|
-
| Key | Required | Default | Notes |
|
|
2096
|
-
| --- | --- | --- | --- |
|
|
2097
|
-
| `intervalSeconds` | yes | — | Whole seconds between ticks, minimum `60`. A tick costs a full turn of a frontier model, so a sub-minute period is refused rather than obeyed. |
|
|
2098
|
-
| `project` | no | the only configured project | Which conductor project this fleet session ticks for. `setup host` stamps it, one tick config per fleet cwd, and it is what lets a host with several configured projects resolve *this* fleet's brief, reporting policy, digest ledger and release grants. Omitting it is the pre-multi-project spelling: correct on a single-project host, and on a host with two or more it degrades every tick to the default reporting scope with no release grants — `status` and the tick log then name the one fix (`re-run omp-conductor setup host`). A name no configured project has degrades the same way. |
|
|
2099
|
-
| `budgetSeconds` | no | `600` | Seconds a turn may run before the tick guard refuses its remaining tool calls (#189), and before a queued operator message preempts them. An integer ≥ 60; anything else degrades to the default. |
|
|
2100
|
-
| `armedFile` | no | none — the gate passes | Path to the arm marker. A tick does nothing while the file is missing. **Re-read from disk on every tick**, so a `setup host` restamp onto `armed-<project>` binds on the next heartbeat instead of leaving a live pane watching the path it captured at session start. Relative paths resolve against the session cwd, so `state/armed` means `<cwd>/state/armed`. `setup host` writes `<state dir>/armed-<project>`, one marker per project, so `arm --project A` cannot arm B. A value it did not generate is left alone as your own choice. |
|
|
2101
|
-
| `accessFile` | no | none — the gate passes | Path to the Telegram bridge's `access.json`. Every tick re-reads it and requires `enabled: true` with exactly one entry in `allowFrom`. Relative paths resolve against the session cwd. **Configure this on any fleet deploy** — see below. |
|
|
2102
|
-
| `message` | no | `Tick <ISO timestamp>: re-read <workspaceRoot>/ORCHESTRATOR.md from disk, then run your standing loop from it.`, then the reporting-policy line, delivery rule, and mechanical availability state | When set, this text replaces the ordinary reporting-policy line and delivery rule, but the runtime-owned availability state is still appended: a custom prompt cannot infer whether the operator may be interrupted. Re-read from disk on **every** tick, so rewording it binds the next heartbeat instead of waiting for a session restart; a re-read that fails — caught mid-edit, removed, or invalid — keeps the value read at session start rather than stopping the heartbeat. `intervalSeconds` is *not* re-read: rescheduling a live timer still needs a restart. The default *orders* the session to re-read its brief, naming the path resolved from the project's `workspaceRoot`, because a standing prompt drifts out of a long-lived session's context while the file on disk does not. |
|
|
2103
|
-
| `agentName` | no | the project name, else `fleet` | The herdr agent name the orchestrator's pane is registered under. Under herdr this is the whole of the identity check below. `setup host` writes the project name, so two fleets in one herdr session are distinguishable; when no tick config names one, the fallback matches `AGENT_NAME=${AGENT_NAME:-fleet}` in the recovery plugin's `recover.sh`, so both halves key on one name. Rename the agent and set this to match. |
|
|
2104
|
-
|
|
2105
|
-
#### Upgrading from one shared arm marker
|
|
2106
|
-
|
|
2107
|
-
Before per-project markers every project was given the same `<state dir>/armed`,
|
|
2108
|
-
so arming one fleet armed all of them. `setup host` rewrites that value — and
|
|
2109
|
-
only that value — to `armed-<project>`. The old bare marker is honoured for one
|
|
2110
|
-
more cycle on a **single-project** host, so the upgrade never silently disarms a
|
|
2111
|
-
live fleet, and the next `arm` or `disarm` retires it. On a host with **two or
|
|
2112
|
-
more** projects it arms nothing: `status` reports `legacy global arm marker —
|
|
2113
|
-
re-run setup host, then arm per project`, and every tick stays disarmed until each
|
|
2114
|
-
project is armed on its own marker.
|
|
2115
|
-
|
|
2116
|
-
The same restamp renames the identity this pane ticks under: an `agentName` of
|
|
2117
|
-
`fleet` — the value every project used to be given — becomes the project name.
|
|
2118
|
-
**Under herdr that is an operator step, not a no-op.** Ownership is proved against
|
|
2119
|
-
the pane's registered herdr agent, so after re-running `setup host` the live fleet
|
|
2120
|
-
pane needs the new name.
|
|
2121
|
-
|
|
2122
|
-
**Rename the agent herdr already detects — do not `agent start`.** `herdr agent
|
|
2123
|
-
start` submits omp *into* the pane's existing shell and requires a pane sitting at
|
|
2124
|
-
a shell prompt with no agent on it; the live orchestrator pane is neither, so it
|
|
2125
|
-
is refused at best and starts a second omp in that pane at worst. `rename` touches
|
|
2126
|
-
no process and keeps the session as it is:
|
|
2127
|
-
|
|
2128
|
-
```sh
|
|
2129
|
-
herdr --session <session> agent list # find the fleet's pane_id
|
|
2130
|
-
herdr --session <session> agent rename <pane-id> <project>
|
|
2131
|
-
```
|
|
2132
|
-
|
|
2133
|
-
If the name cannot be reassigned in place, stop and resume rather than starting a
|
|
2134
|
-
second orchestrator — the same shape `recover.sh` uses, so the omp session is
|
|
2135
|
-
preserved rather than replaced:
|
|
2136
|
-
|
|
2137
|
-
```sh
|
|
2138
|
-
herdr --session <session> agent get <pane-id> # note agent_session.value — the session ref
|
|
2139
|
-
# exit omp in that pane (/exit) so the pane is back at a shell prompt, then:
|
|
2140
|
-
herdr --session <session> agent start <project> --kind omp --pane <pane-id> -- --resume=<ref>
|
|
2141
|
-
```
|
|
2142
|
-
|
|
2143
|
-
Until the pane carries the new name it declines to tick and logs which agent it
|
|
2144
|
-
actually is versus the one the tick config names, with the `rename` command in the
|
|
2145
|
-
line — the heartbeat fails closed and says so rather than letting two fleets both
|
|
2146
|
-
answer to `fleet`. Set `agentName` explicitly if you would rather keep the old
|
|
2147
|
-
name; a value that is not the shared default is never rewritten.
|
|
2148
|
-
|
|
2149
|
-
The marker itself needs no restart: `armedFile` is re-read from disk every tick, so
|
|
2150
|
-
the restamped path binds on the next heartbeat. Before 0.15.2 it was read once at
|
|
2151
|
-
session start, and a restamp under a live pane left that pane watching a path the
|
|
2152
|
-
restamp had just replaced — `status` reported `armed` from the file while the
|
|
2153
|
-
heartbeat skipped silently as "not armed", which writes no stall marker.
|
|
2154
|
-
|
|
2155
|
-
Recovery is fail-closed across that window. A restamped `agentName` moves the
|
|
2156
|
-
recovery plugin's own state files to per-agent paths that do not exist yet, and
|
|
2157
|
-
the live pane is still saved under `fleet`, so the snapshot offers no candidate
|
|
2158
|
-
for the new name. `herdr-conductor` treats the pre-rename identity and bootstrap
|
|
2159
|
-
marker as proof a fleet has already lived on this host whatever it is called now:
|
|
2160
|
-
it pages `no fleet identity to recover for agent <project>` instead of
|
|
2161
|
-
provisioning a second workspace beside the live orchestrator. Finish the rename
|
|
2162
|
-
and the next pass recovers normally.
|
|
2163
|
-
|
|
2164
|
-
A default tick sends one message (`customType` `omp-conductor.tick`, attributed
|
|
2165
|
-
to the user): the standing-loop prompt, the reporting-policy constraint re-read
|
|
2166
|
-
from conductor config on every tick, the delivery rule, and the mechanically
|
|
2167
|
-
computed operator-availability state. A configured `message` replaces the first
|
|
2168
|
-
three parts but not that clock state. The delivery rule is there
|
|
2169
|
-
because end-of-turn text reaches the operator's Telegram only on a turn that
|
|
2170
|
-
*began* as an inbound Telegram message: a tick is injected locally, so anything
|
|
2171
|
-
the session merely writes at the end of one is read by nobody, and a reportable
|
|
2172
|
-
event has to be delivered by an explicit `telegram_send` call the session
|
|
2173
|
-
watched succeed. The tick starts a turn if the session is idle; while a turn is
|
|
2174
|
-
streaming it is queued as a follow-up and consumed when that turn ends.
|
|
2175
|
-
It sends **nothing** when:
|
|
2176
|
-
|
|
2177
|
-
- `armedFile` is configured and missing;
|
|
2178
|
-
- `accessFile` is configured and the escalation channel is not verifiably up;
|
|
2179
|
-
- an earlier tick is still queued. Ticks coalesce rather than stack, so a slow
|
|
2180
|
-
turn cannot leave a backlog of heartbeats behind it — and two coalesced ticks
|
|
2181
|
-
in a row are the signal that the session is not slow but wedged, which is
|
|
2182
|
-
what the [stall marker](#a-wedged-session-and-the-marker-that-notices) is for.
|
|
2183
|
-
|
|
2184
|
-
### One session per directory ticks, and it says which
|
|
2185
|
-
|
|
2186
|
-
Activation is a property of the *directory*, so before it arms anything the
|
|
2187
|
-
heartbeat asks whether this session is the orchestrator or merely a session
|
|
2188
|
-
standing in its directory. It has to: opening a second omp session in the fleet's
|
|
2189
|
-
cwd — a shell to read state, say — used to arm a second heartbeat that prompted
|
|
2190
|
-
*that* session with the standing loop, and with
|
|
2191
|
-
[`authority`](#configuration) delegated it would consider itself entitled to
|
|
2192
|
-
merge PRs and cut releases. Two brains, one queue, and nothing in the log to tell
|
|
2193
|
-
them apart.
|
|
2194
|
-
|
|
2195
|
-
**Under herdr** (`HERDR_ENV=1` with a `HERDR_PANE_ID`), the answer is the pane's
|
|
2196
|
-
registered agent name: the heartbeat asks `herdr agent list` for the entry whose
|
|
2197
|
-
`pane_id` is this pane's and ticks only when its `name` equals `agentName`. Fleetness
|
|
2198
|
-
is the *session* — every pane in it shares `HERDR_SESSION` and the cwd — and
|
|
2199
|
-
herdr's `agent` field is the *runtime*, `omp` for the orchestrator and for the
|
|
2200
|
-
shell beside it, so neither can tell them apart. The registered name can, it is
|
|
2201
|
-
what `herdr agent start fleet --kind omp --pane <id>` sets when the recovery plugin
|
|
2202
|
-
starts a fleet into an empty pane — and what `herdr agent rename <pane-id> fleet`
|
|
2203
|
-
sets on a pane whose agent herdr already detects, which is the only safe spelling
|
|
2204
|
-
while omp is running in it. It is the same identity the recovery plugin keys on. A
|
|
2205
|
-
pane with a different name, or no name at all, stays inert.
|
|
2206
|
-
|
|
2207
|
-
**Without herdr**, the session claims the directory in a sibling
|
|
2208
|
-
`.conductor-tick-owner.json` (pid, session file, claim time) and ticks only while
|
|
2209
|
-
it is the live claimant. Liveness is a **pid check, never a timestamp**: a crashed
|
|
2210
|
-
orchestrator's claim is reclaimed by the next session rather than wedging the
|
|
2211
|
-
fleet until somebody deletes a file, and a slow-but-running orchestrator never
|
|
2212
|
-
loses its claim to a lease that expired.
|
|
2213
|
-
|
|
2214
|
-
Declining is logged once, at session start, naming the holder — which is the whole
|
|
2215
|
-
point, because the original failure was that the second ticker was
|
|
2216
|
-
indistinguishable from the first:
|
|
2217
|
-
|
|
2218
|
-
```text
|
|
2219
|
-
[omp-conductor] orchestrator tick inactive: pane w1:p1 (agent "fleet") owns the fleet tick here — this session will not tick
|
|
2220
|
-
[omp-conductor] orchestrator tick inactive: this pane is agent "scratch", not the fleet agent "fleet" — this session will not tick
|
|
2221
|
-
[omp-conductor] orchestrator tick inactive: pid 12345 (claimed 2026-01-02T03:04:05.000Z, session …/fleet.jsonl) owns the fleet tick in /home/conductor/.omp/conductor — this session will not tick
|
|
2222
|
-
```
|
|
2223
|
-
|
|
2224
|
-
A `herdr agent list` that does not answer also declines, for the same reason the
|
|
2225
|
-
escalation channel fails closed: under herdr this session is one pane of several
|
|
2226
|
-
in that directory, and an unproven identity is exactly the case the check exists
|
|
2227
|
-
for. That includes `herdr` not being on the session's `PATH` — worth checking on a
|
|
2228
|
-
fleet host, where the orchestrator's environment comes from a unit file rather
|
|
2229
|
-
than a login shell — and `HERDR_BIN_PATH` names the binary when it is not, the
|
|
2230
|
-
same escape hatch the recovery plugin's `recover.sh` has. On a host with no herdr
|
|
2231
|
-
and no prior claimant — the ordinary single-session case — nothing changes.
|
|
2232
|
-
|
|
2233
|
-
### The escalation channel is a gate, and it fails closed
|
|
2234
|
-
|
|
2235
|
-
Unattended dispatch is only defensible while a tier-2 escalation can reach a
|
|
2236
|
-
person. So `accessFile` is checked on **every** tick and never cached at session
|
|
2237
|
-
start: the bridge is reconfigured out-of-band, and a heartbeat that trusted a
|
|
2238
|
-
startup snapshot would keep dispatching for days after the channel went away. A
|
|
2239
|
-
stale arm marker must not outlive the channel that makes running unattended safe.
|
|
2240
|
-
|
|
2241
|
-
The check passes only when a bot token is resolvable — `TELEGRAM_BOT_TOKEN` in
|
|
2242
|
-
the environment, or in the `.env` beside `accessFile` — and the file parses to an
|
|
2243
|
-
object with `enabled: true` and exactly one `allowFrom` entry. Everything else
|
|
2244
|
-
stops the heartbeat: no token, so nothing outbound works at all; file missing,
|
|
2245
|
-
unreadable or truncated; not JSON, or JSON that is not an object; `enabled`
|
|
2246
|
-
absent or false; zero owners paired (nobody to page) or more than one (ambiguous:
|
|
2247
|
-
the conductor refuses to guess which human is on the hook). Failure modes are
|
|
2248
|
-
deliberately not distinguished in the decision: each one means a page lands
|
|
2249
|
-
nowhere. `omp-conductor status` is where they are told apart — its `telegram` row
|
|
2250
|
-
names the specific fault.
|
|
2251
|
-
|
|
2252
|
-
One caveat the file cannot express: omp-telegram binds its own copy of the token
|
|
2253
|
-
in `startBot()` at session start, and only when the bridge is switched on. It
|
|
2254
|
-
rebinds only on `/telegram token` or `/telegram on`. So writing a token into
|
|
2255
|
-
`.env` out-of-band — or flipping `enabled` to true by hand — restores tier-2
|
|
2256
|
-
paging immediately, because conductor sends those itself, while the bridge's own
|
|
2257
|
-
tools, `telegram_send` and `telegram_ask`, stay dead until you reload it. After
|
|
2258
|
-
either edit, run `/telegram on` in the orchestrator session. Until you do, ticks
|
|
2259
|
-
carry an explicit note that an amendment cannot be approved on this surface, and
|
|
2260
|
-
the conductor never assumes an answer it did not receive.
|
|
2261
|
-
|
|
2262
|
-
Leaving `accessFile` unset passes the gate, because an ordinary developer session
|
|
2263
|
-
that happens to have a `.conductor-tick.json` has no bridge to check. It is not an
|
|
2264
|
-
off switch for the check: **a fleet deploy always sets it.**
|
|
2265
|
-
|
|
2266
|
-
Every tick — sent or skipped — is logged with its reason (`not armed`,
|
|
2267
|
-
`escalation channel down`, `tick already pending`) to the omp log. `omp-conductor
|
|
2268
|
-
hold` is deliberately **not** one of the gates: hold stops the *dispatcher*
|
|
2269
|
-
claiming work, and the tick drives a different session — one whose duties
|
|
2270
|
-
(grooming the queue, draining escalations, reporting) are exactly what stays
|
|
2271
|
-
useful while dispatch is stopped. Its own off switch is the arm marker. Skips
|
|
2272
|
-
are deliberately silent in the UI: a disarmed fleet would otherwise raise a
|
|
2273
|
-
notification every interval, forever. The one exception is a malformed
|
|
2274
|
-
`.conductor-tick.json`,
|
|
2275
|
-
which notifies once at session start and leaves the heartbeat off; silent failure
|
|
2276
|
-
there is the failure mode the heartbeat exists to prevent. A conductor config that
|
|
2277
|
-
cannot supply a reporting scope logs `tick reporting scope: using material` once
|
|
2278
|
-
per session. The interval does not re-log it, because the file is unlikely to fix
|
|
2279
|
-
itself between two ticks.
|
|
2280
|
-
|
|
2281
|
-
### A wedged session, and the marker that notices
|
|
2282
|
-
|
|
2283
|
-
Coalescing is also the only wedge detector this package has. On 2026-08-07 the
|
|
2284
|
-
dogfood fleet's orchestrator finished a turn, logged `ui.loop-blocked` right
|
|
2285
|
-
after an auto-compaction threshold decision, and never started another. The
|
|
2286
|
-
process stayed alive, so herdr's recovery — agent listed AND a non-shell
|
|
2287
|
-
foreground process — read healthy. The dispatch daemon is a separate process and
|
|
2288
|
-
kept working, so `/healthz` was green all night, while the one brain holding
|
|
2289
|
-
merge authority sat on a green PR it never merged. A tick injected two minutes
|
|
2290
|
-
into the wedge and an operator's Telegram message five minutes later both went
|
|
2291
|
-
unconsumed for 23 minutes, until a manual `SIGTERM`. The heartbeat logged `tick
|
|
2292
|
-
skipped: tick already pending` throughout, which is exactly what a merely slow
|
|
2293
|
-
turn looks like.
|
|
2294
|
-
|
|
2295
|
-
So the heartbeat counts them. Two consecutive coalesced ticks — a full hour at
|
|
2296
|
-
the reference 1800-second interval, generous by construction — mean the last
|
|
2297
|
-
prompt was never consumed, and the extension:
|
|
2298
|
-
|
|
2299
|
-
- writes `<session cwd>/.conductor-stalled`, one line of `<ISO timestamp>
|
|
2300
|
-
<diagnosis>`.
|
|
2301
|
-
- logs at **error** level: `orchestrator stalled: 2 ticks queued unconsumed —
|
|
2302
|
-
the agent loop is not draining; see .conductor-stalled`.
|
|
2303
|
-
|
|
2304
|
-
Both escapes deliberately leave the session, because a loop that cannot drain
|
|
2305
|
-
its queue cannot report on itself — that is the whole failure.
|
|
2306
|
-
|
|
2307
|
-
**The daemon reads it.** A marker nobody consumes is an artifact, not an alert,
|
|
2308
|
-
so the dispatch daemon checks it on its own five-minute tick — and *before* its
|
|
2309
|
-
pause check. The orchestrator is a different process and can be wedged while
|
|
2310
|
-
the fleet is deliberately paused, which is precisely the state the dogfood
|
|
2311
|
-
fleet was in when this happened. One tier-2 page per stall, keyed on the
|
|
2312
|
-
marker's own timestamp so a second wedge the same day is not swallowed as a
|
|
2313
|
-
repeat, re-armed when the marker clears, and latched only once the page is
|
|
2314
|
-
confirmed delivered — an escalation channel that fails on the one tick that
|
|
2315
|
-
noticed must not buy permanent silence.
|
|
2316
|
-
|
|
2317
|
-
It restarts nothing. A wedge lands mid-turn, and no other process can tell a
|
|
2318
|
-
half-applied edit from an idle loop; the operator attaches, looks, and decides.
|
|
2319
|
-
|
|
2320
|
-
**herdr-conductor deliberately does not read it**, though its liveness test
|
|
2321
|
-
(agent listed AND a non-shell foreground process) passes straight through a
|
|
2322
|
-
wedge. That plugin only runs on `startup`, `pane.exited` and
|
|
2323
|
-
`pane.agent_detected`, and a session that stays alive and stops working emits
|
|
2324
|
-
none of them — so the check could never fire during the wedge itself. What it
|
|
2325
|
-
*would* catch is the recovery afterwards: the marker survives a restart until
|
|
2326
|
-
the new session consumes a tick, so every operator SIGTERM-and-resume would
|
|
2327
|
-
page about the healthy session they just fixed. Telling those apart needs the
|
|
2328
|
-
process start time against the marker's, and herdr's `pane process-info`
|
|
2329
|
-
reports pids, not start times. The daemon gives up at most one tick of
|
|
2330
|
-
coverage and never cries wolf.
|
|
2331
|
-
|
|
2332
|
-
The first tick that actually sends clears the counter and deletes the marker,
|
|
2333
|
-
and it deletes one it did not write: recovery normally arrives as a fresh
|
|
2334
|
-
process resuming the same transcript, so the session doing the clearing is not
|
|
2335
|
-
the session that stalled. Nothing else removes the file. Neither the write nor
|
|
2336
|
-
the delete can take the heartbeat down — a filesystem error is logged and the
|
|
2337
|
-
tick carries on.
|
|
2338
|
-
|
|
2339
|
-
`omp-conductor status` reads the same marker from the **state directory** and
|
|
2340
|
-
prints one more line under the daemon block:
|
|
2341
|
-
|
|
2342
|
-
```text
|
|
2343
|
-
orchestrator STALLED since 2026-08-07T06:27:55.123Z — 2 ticks queued unconsumed — the agent loop is not draining
|
|
2344
|
-
```
|
|
2345
|
-
|
|
2346
|
-
That reading is the reference deploy's convention — the orchestrator session
|
|
2347
|
-
runs from `~/.omp/conductor`, which is the state directory — and it is
|
|
2348
|
-
one-directional: a line there proves a wedge, and its absence proves nothing,
|
|
2349
|
-
least of all on a fleet whose session lives somewhere else.
|
|
547
|
+
## Where issues come from
|
|
2350
548
|
|
|
2351
|
-
|
|
549
|
+
**GitHub Issues is the only supported tracker in v1.** `tracker.kind` accepts
|
|
550
|
+
exactly one value, `"github"`, and every tracker operation shells out to your
|
|
551
|
+
already-authenticated `gh` CLI — the conductor never stores a token of its own.
|
|
552
|
+
Gitea, Jira, and file-based trackers are not supported yet; the seam for them is
|
|
553
|
+
`src/tracker/github.ts`, which implements the whole nine-method `Tracker`
|
|
554
|
+
interface in `src/types.ts` (`listReady`, `addLabel`, `removeLabel`, `comment`,
|
|
555
|
+
`close`, `linkParent`, `parentOf`, `openCloserFor`, `prState`) that a future
|
|
556
|
+
backend would swap in.
|
|
2352
557
|
|
|
2353
|
-
|
|
2354
|
-
omp
|
|
2355
|
-
omp-conductor setup
|
|
2356
|
-
|
|
2357
|
-
omp-conductor start [--port N] [--project NAME]
|
|
2358
|
-
omp-conductor --version
|
|
2359
|
-
omp-conductor stop
|
|
2360
|
-
omp-conductor restart [--now] [--timeout SECONDS] [--port N] [--project NAME]
|
|
2361
|
-
omp-conductor upgrade [--to VERSION] [--project NAME]
|
|
2362
|
-
omp-conductor status [--project NAME]
|
|
2363
|
-
omp-conductor ledger [--issue N] [--limit N] [--project NAME]
|
|
2364
|
-
omp-conductor board [--project NAME]
|
|
2365
|
-
omp-conductor hold [--keep-ticks] [--project NAME]
|
|
2366
|
-
omp-conductor stop [--pane] [--project NAME]
|
|
2367
|
-
omp-conductor arm [--project NAME]
|
|
2368
|
-
omp-conductor disarm [--project NAME]
|
|
2369
|
-
omp-conductor tail <issue> [--project NAME]
|
|
2370
|
-
omp-conductor extend <issue> --turns N [--project NAME]
|
|
2371
|
-
omp-conductor worker pause <issue> [--project NAME]
|
|
2372
|
-
omp-conductor worker resume <issue> [--project NAME]
|
|
2373
|
-
omp-conductor worker stop <issue> --reason TEXT [--project NAME]
|
|
2374
|
-
omp-conductor unblock <issue> [--force] [--no-requeue] [--project NAME]
|
|
2375
|
-
omp-conductor verb <conductor_*> [--project NAME] [--arg k=v ...]
|
|
2376
|
-
omp-conductor friction <escalation-digest|report-noise|report-surprise> --detail TEXT [--issue N] [--project NAME]
|
|
2377
|
-
omp-conductor event record --category NAME --summary TEXT --evidence REF [--occurred-at ISO] [--project NAME]
|
|
2378
|
-
omp-conductor report --text TEXT [--kind material|digest] [--events IDS] [--notices IDS] [--project NAME]
|
|
2379
|
-
omp-conductor message --text TEXT [--project NAME]
|
|
2380
|
-
omp-conductor decision open --question TEXT [--blocks TEXT] [--resolves-when COND] [--project NAME]
|
|
2381
|
-
omp-conductor decision resolve <id> --answer TEXT [--project NAME]
|
|
2382
|
-
omp-conductor decision withdraw <id> [--reason TEXT] [--project NAME]
|
|
2383
|
-
omp-conductor decision list [--project NAME]
|
|
2384
|
-
omp-conductor daemon [--once] [--port N] [--project NAME]
|
|
2385
|
-
omp-conductor resume [--project NAME]
|
|
2386
|
-
omp-conductor brief-upgrade [--migrate|--retrofit] [--apply] [--file PATH] [--project NAME]
|
|
2387
|
-
omp-conductor help
|
|
2388
|
-
```
|
|
558
|
+
You tell the conductor where to look with three keys, all in
|
|
559
|
+
`~/.omp/conductor/config.json` (the [Configuration](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#configuration) section has
|
|
560
|
+
the full annotated example, and `omp-conductor setup` will interview you for these
|
|
561
|
+
and create any missing labels):
|
|
2389
562
|
|
|
2390
|
-
|
|
|
563
|
+
| Key | Meaning |
|
|
2391
564
|
| --- | --- |
|
|
2392
|
-
| `
|
|
2393
|
-
| `
|
|
2394
|
-
| `
|
|
2395
|
-
| `start` | Start `herdr-fleet.service` when that optional unit is installed, clearing a previous pane-recovery pin, then start the dispatch daemon and wait until it answers `GET /healthz`. When `omp-conductor.service` is installed, systemd is the only start path: even `start --project NAME` restores the shared unit and uses the name only to verify that `/healthz` serves the requested project. A detached daemon is allowed only when the unit is proven absent. It never clears pause or arms ticks. Refuses if a daemon is already live, naming its pid; manager refusal or unprovable ownership is an error rather than a detached fallback. |
|
|
2396
|
-
| `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
|
|
2397
|
-
| `restart [--now] [--timeout SECONDS] [--port N] [--project NAME]` | Drains the fleet by default: pause new claims, wait until live workers reach `0 / N` (bounded by `--timeout SECONDS`, default 1800 = 30 min), restart, then restore the prior dispatch state. A daemon serving multiple configured projects makes restart host-wide: `--project` is rejected because draining one queue and restarting the shared process would kill another project's workers. Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; only a host proven not to have the installed unit may fall back to the standalone stop/start path. `--now` skips the drain and restarts immediately, orphaning any live runs (old behaviour). A drain that hits `--timeout` restarts nothing and leaves dispatch paused — `omp-conductor resume` lifts it, or re-run `restart` to keep waiting. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
|
|
2398
|
-
| `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. Host-wide by default: one daemon serves every configured project, so a bare run drains all of them and refreshes every brief. `--project` is rejected when the live daemon serves several projects — draining one queue and restarting the shared daemon would kill another's workers. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
|
|
2399
|
-
| `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. Active-run lines overlay cooperative worker `paused`/`pausing` from `/healthz` without changing SQLite `running` state or the live worker count. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
|
|
2400
|
-
| `ledger [--issue N] [--limit N]` | The action audit: every [mediated-verb](#the-mediated-verbs-126) mutation and every next-attempt turn budget. Verb entries include the arguments, decision, named refusal, and resulting SHA. Turn-budget entries remain after an override is replaced or consumed. Reads (`conductor_pr_status`) are absent so polling cannot bury the signal. `--issue` narrows both histories; `--limit` defaults to 50. Recent verb refusals and pending turn overrides also appear in `status`. |
|
|
2401
|
-
| `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, the last 24 hours of Merged and Settled, and Parked (an issue the tracker has not confirmed closed — still open, or a label read that failed — so nothing dispatches it until a human labels it). Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
|
|
2402
|
-
| `hold [--keep-ticks] [--project NAME]` | Soft stop: pause claiming **and** disarm ticks. Daemon and pane stay up. Prefer this when the intent is "stop the conductor" without killing processes. `--keep-ticks` pauses claiming but leaves the arm marker, so the heartbeat keeps reporting and `resume` alone restores the fleet — no fresh arm challenge. See [Stop the conductor](#stop-the-conductor-hold--stop). |
|
|
2403
|
-
| `stop [--pane] [--project NAME]` | Stop the conductor: pause claiming, disarm ticks, then stop the dispatch daemon (systemctl-aware). Pane stays up unless `--pane` is passed. `stop --pane` also pins herdr-conductor recovery off for the conductor agent only — it does **not** stop `herdr-fleet.service` or any other herdr session. Fail-closed: exits nonzero unless the agent is proven gone. To bounce the daemon without stopping the fleet, use `restart`. |
|
|
2404
|
-
| `arm [--project NAME]` | Proof-gated: send a Telegram challenge and write this project's arm marker only after your reply appears as a user turn in the orchestrator transcript. The challenge names the project, so a host running two fleets is not ambiguous. Never auto-armed by `resume` / `hold`. |
|
|
2405
|
-
| `disarm [--project NAME]` | Remove this project's arm marker so its ticks skip; another project's ticks keep running. Also clears a pre-per-project shared `armed` marker while that marker is still what holds this fleet's gate open — otherwise the disarm would not disarm. Processes untouched. |
|
|
2406
|
-
| `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
|
|
2407
|
-
| `extend <issue> --turns N [--project NAME]` | Raise a live worker's effective turn ceiling through its owning daemon without restarting its session. If the latest run is failed, killed, orphaned, or blocked and has no live controller, store a one-shot ceiling for that issue's next claimed attempt instead. A next-attempt value must exceed the project base, every extension must stay at or below `workerMaxTurnsCeiling`, and live extensions remain monotonic. The pending value appears in `status`, is recorded in `ledger`, and is consumed atomically by one claim. |
|
|
2408
|
-
| `worker pause <issue>` / `worker resume <issue>` | Cooperatively park one live worker without changing its run state or lane. Pause aborts the active turn to harness idle and freezes the remaining wall-clock budget; resume continues the same session with a prompt to re-check its last action before repeating it. This is separate from fleet-level `hold`, which refuses new claims and work-starting mutations while allowing pre-pause completion work and releases. |
|
|
2409
|
-
| `worker stop <issue> --reason TEXT [--project NAME]` | Terminally end a running or cooperatively paused worker. The reason is required (1–500 characters) and persisted on the run. The command waits for settlement, records the distinct `stopped` state, salvages and publishes dirty work, removes `agent:in-progress` through the durable label outbox, and consumes neither failed-attempt nor continuation budget. If salvage fails, the tree holding the only copy stays in place and the command names it. Repeating stop is idempotent and reports the run's already-terminal state. |
|
|
2410
|
-
| `unblock <issue> [--force] [--no-requeue]` | Remove that issue's `blocked` and `failed` labels so an answered escalation can be claimed again, and restore the project queue label by default so the dispatcher actually sees it. `agent:in-progress` comes off too, but only when the newest recorded run is terminal — that row is the proof no worker still owns the issue, so a live run keeps the label (and the queue label stays off until that run settles), and so does an issue with no run row at all. Run history remains intact: blocks consume the independent continuation budget, not failed implementation attempts. The output reports both budgets and warns when either will make the next tick escalate instead of dispatch. The label changes go through the [label projection outbox](#the-tick): they are applied inline before the command returns, but **a tracker that refuses them (403, rate limit) no longer fails the verb** — it exits `0`, the intended label state is durable and the daemon retries it, and the output says `label sync queued (N pending) — the daemon retries` instead of claiming the labels were restored. Safety is preserved, but the issue is only claimable once the queue label itself lands: the queue read asks GitHub for issues carrying that label, so a refused queue-label add keeps the issue out of dispatch until projection succeeds. `--no-requeue` clears the state labels only, leaving the queue label untouched — the case where you are about to close the issue. **Refuses, clearing nothing and exiting `3`, when the newest attempt's work could not be committed and its worktree is the only copy** — re-claiming removes that tree. `--force` records the operator's acceptance on the run row and then clears; the salvage failure stays in history. Exits `2` when the issue number is missing or malformed. |
|
|
2411
|
-
| `verb <conductor_*> [--arg k=v ...]` | Run one [mediated verb](#the-mediated-verbs-126) as the orchestrator, from the CLI — the external-orchestrator half of the verb surface. Every argument goes in as a `--arg k=v` string; an orchestrator can merge (`conductor_pr_merge`), label (`conductor_label`), release (`conductor_release`), update a branch (`conductor_pr_update_branch`) or title/body (`conductor_pr_update`), or read PR state (`conductor_pr_status`). The daemon applies the same checks and writes the same ledger rows a session's call would; a missing `--arg` is refused exactly as a missing tool argument is, worker-only verbs (`conductor_push`, `conductor_pr_create`) are refused with `role-not-allowed`, and a refusal exits `3`. An unknown verb exits `2`. |
|
|
2412
|
-
| `friction <kind> --detail TEXT [--issue N]` | Record one bounded judgment the daemon cannot infer: an escalation belonged in a digest, or a tick report was noise/surprising. The detail is limited to 160 characters. One event never changes policy; three observations inside seven days make the aggregate eligible for one Learning-loop prompt, followed by a seven-day cooldown. |
|
|
2413
|
-
| `report --text TEXT [--kind material|digest]` | Hand a rendered report to the daemon's durable outbox. The command persists the text **before** anything can send and prints a durable handoff id. A material report submitted during quiet hours becomes a held notice until the window opens; otherwise it becomes a report whose delivery the daemon owns, retries with bounded backoff, and records. Delivery is [at-least-once](#report-delivery-the-outbox), so a crash mid-send is retried as a possible repeat and `delivered` never proves exactly one message. `--kind digest` is accepted at most once per local day, decided from the ledger; an unknown `--kind` exits `2`. Anything still owed appears in `status` with its age. |
|
|
2414
|
-
| `decision open --question TEXT [--blocks TEXT] [--resolves-when COND]` | Record a question the orchestrator has put to you, and print its id. A question that lives only in a session's context is lost at the next compaction — after which it is either asked twice or dropped silently. `--resolves-when` attaches a machine-checkable condition: `pr-merged:<https url>`, `pr-checks-green:<https url>`, `pr-mergeable:<https url>`, `issue-closed:<n>`, `npm-version:<pkg>@<version>`, or `rate-limit-reset:github`; anything else exits `2` listing the six forms. See [The decision ledger](#the-decision-ledger-resolves-when). |
|
|
2415
|
-
| `decision resolve <id> --answer TEXT` | Record what you decided. Exits `1` naming the id when it is unknown or no longer open, so a second answer cannot overwrite the first. |
|
|
2416
|
-
| `decision withdraw <id> [--reason TEXT]` | Close a question the session stopped needing, with why. Same guard as `resolve`. |
|
|
2417
|
-
| `decision list` | Open questions, oldest first: id, age, what each blocks, whether its condition is met, and the question. Prints `no open decisions` when there are none. |
|
|
2418
|
-
| `daemon` | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. Admitted workers run in a tracked background pool, so settlement and capacity checks remain periodic while they work; shutdown drains the pool before closing the store. This is what `start` launches and what a systemd unit should call. |
|
|
2419
|
-
| `daemon --once` | Run a single tick, wait for workers admitted by that tick, and exit. No HTTP server or pidfile — a drill must not register itself as the daemon, or the next reader believes it and the real daemon's in-flight runs get reconciled as orphans. |
|
|
2420
|
-
| `--port N` | Accepted by `start`, `restart` and `daemon`. Both `--port 9000` and `--port=9000` work; missing or out of range exits `2` rather than falling back to the default, because probing the wrong endpoint is worse than a hard failure. |
|
|
2421
|
-
| `--project NAME` | Selects a project for project-scoped commands and foreground `daemon`. On an installed shared service, `start --project NAME` still starts the host-wide unit and uses the name only to verify `/healthz`; a draining `restart --project NAME` is rejected when that daemon serves multiple projects. A project-only daemon is available only through an explicit foreground `daemon --project NAME` or standalone start on a host proven not to have the unit. |
|
|
2422
|
-
| `pause [--reason TEXT]` | Stop new claims and work-starting mutations only. The running daemon notices on its next tick; runs already in flight finish. The orchestrator may still merge, update, or label runs admitted before the pause, and may release when the release policy's own preconditions hold. The orchestrator heartbeat keeps ticking if armed — its gate is the arm marker, not this flag. Per-worker pause is separate. Prefer `hold` to silence both. `--reason TEXT` is recorded in the pause sentinel, which `status` shows as the pause provenance. |
|
|
2423
|
-
| `resume [--project NAME]` | Clear pause and any `stop --pane` recovery pin — does **not** re-arm. Run `arm` after an inbound Telegram proof to resume ticks. |
|
|
2424
|
-
| `--version`, `-V`, `version` | Print the installed `omp-conductor` package version and exit `0`. Works from the global binary and npm/plugin install because it reads the package metadata beside the shipped CLI. |
|
|
2425
|
-
| `brief-upgrade` | Inspect the package-floor + `POLICY.md` overlay. Reports by default; see [Keeping a brief current](#keeping-a-brief-current). |
|
|
2426
|
-
| `--migrate` | Only for `brief-upgrade`. Lift a bannered `ORCHESTRATOR.md` owned half into `POLICY.md` and recompose. Dry-run unless `--apply`. |
|
|
2427
|
-
| `--retrofit` | Only for `brief-upgrade`. Propose (or with `--apply`, write) a `YOURS TO EDIT` banner before the first owned-topic heading on a hand-written brief. |
|
|
2428
|
-
| `--apply` | Only for `brief-upgrade`. Confirms `--migrate` / `--retrofit`. On its own it exits `2`: the legacy single-file merge was removed in 0.4.3. |
|
|
2429
|
-
| `--file PATH` | Only for `brief-upgrade`. Check a brief that is not where the wizard would have put it, on a host that may have no config at all. |
|
|
2430
|
-
| `help`, `--help`, `-h` | Print usage. An unknown or missing verb prints it too, and exits `2`. |
|
|
2431
|
-
|
|
2432
|
-
Pause is a sentinel under the state directory and survives a daemon restart.
|
|
2433
|
-
`hold --project NAME` writes `paused-<name>` for that project only; a bare
|
|
2434
|
-
`paused` file (legacy / all-projects) pauses every project. It refuses new claims
|
|
2435
|
-
and work-starting mutations, allows completion verbs only for runs admitted
|
|
2436
|
-
before the pause, and leaves `conductor_release` to its normal authority, grant,
|
|
2437
|
-
and precondition checks. Per-worker pause is independent. Hold also removes the
|
|
2438
|
-
arm marker the heartbeat reads, so both brains go quiet without killing processes.
|
|
2439
|
-
|
|
2440
|
-
Every one of these is a verb on the `omp-conductor` binary, each taking an optional
|
|
2441
|
-
`--project NAME`. There is no in-session command: an omp session that wants any of
|
|
2442
|
-
them shells out to the binary, which is what keeps one implementation and one ledger
|
|
2443
|
-
entry per action.
|
|
2444
|
-
|
|
2445
|
-
### Health endpoint
|
|
2446
|
-
|
|
2447
|
-
```bash
|
|
2448
|
-
curl -s localhost:8787/healthz
|
|
2449
|
-
```
|
|
2450
|
-
|
|
2451
|
-
```json
|
|
2452
|
-
{
|
|
2453
|
-
"ok": true,
|
|
2454
|
-
"rssBytes": 123456789,
|
|
2455
|
-
"projects": [
|
|
2456
|
-
{
|
|
2457
|
-
"ok": true,
|
|
2458
|
-
"paused": false,
|
|
2459
|
-
"activeRuns": 1,
|
|
2460
|
-
"project": "demo",
|
|
2461
|
-
"dispatch": {
|
|
2462
|
-
"completedAt": 1786185678000,
|
|
2463
|
-
"ready": 8,
|
|
2464
|
-
"routed": 8,
|
|
2465
|
-
"admitted": 0,
|
|
2466
|
-
"degraded": true,
|
|
2467
|
-
"holds": [
|
|
2468
|
-
{ "reason": "parent-lookup-error", "count": 8, "issues": [321, 320, 318] }
|
|
2469
|
-
]
|
|
2470
|
-
},
|
|
2471
|
-
"codeGraph": {
|
|
2472
|
-
"configured": true,
|
|
2473
|
-
"status": "degraded",
|
|
2474
|
-
"checkedAt": "2026-08-08T13:00:00.000Z",
|
|
2475
|
-
"prerequisites": { "indexer": "present", "mcpMount": "missing" },
|
|
2476
|
-
"repos": [
|
|
2477
|
-
{
|
|
2478
|
-
"name": "api",
|
|
2479
|
-
"path": "/home/fleet/.cache/conductor-graph/acme/api",
|
|
2480
|
-
"clone": "present",
|
|
2481
|
-
"index": "present"
|
|
2482
|
-
}
|
|
2483
|
-
],
|
|
2484
|
-
"timer": { "enabled": "enabled", "active": "active" },
|
|
2485
|
-
"refresh": {
|
|
2486
|
-
"result": "success",
|
|
2487
|
-
"fresh": true,
|
|
2488
|
-
"lastSuccessAt": "2026-08-08T12:50:00.000Z",
|
|
2489
|
-
"ageMs": 600000
|
|
2490
|
-
},
|
|
2491
|
-
"reasons": ["worker MCP configuration does not mount the indexer"]
|
|
2492
|
-
}
|
|
2493
|
-
}
|
|
2494
|
-
]
|
|
2495
|
-
}
|
|
2496
|
-
```
|
|
565
|
+
| `tracker.repo` | The **one** `owner/repo` whose issue list is the queue. This is your planning repo — it does not have to contain any code. |
|
|
566
|
+
| `queueLabel` | Open issues in `tracker.repo` carrying this label are the work queue. Nothing else is ever read. Required: the wizard pre-fills `ready-for-agent`, but a config that omits the key is rejected, not defaulted. |
|
|
567
|
+
| `routing.repos` + `repo:<name>` labels | Each queued issue must also carry exactly one routing label naming which code repo the work lands in. The conductor cuts the worktree and PR there, from `routing.repos[name].cloneUrl`. An issue with zero or two routing labels is reported as unroutable and skipped — never guessed. |
|
|
2497
568
|
|
|
2498
|
-
|
|
2499
|
-
|
|
2500
|
-
Per-project blocks keep `paused`, `activeRuns`, `dispatch`, `codeGraph`, and
|
|
2501
|
-
`workers`. Nonfatal admission errors and graph degradation keep `ok` `true` so a
|
|
2502
|
-
supervisor does not restart-loop. Inspect `dispatch.degraded` and its bounded
|
|
2503
|
-
reason groups for queue starvation; inspect `codeGraph` for configured graph
|
|
2504
|
-
health. `activeRuns` counts occupied issues — live workers plus green PRs
|
|
2505
|
-
awaiting merge.
|
|
569
|
+
So: one tracker repo supplies the queue, routing labels fan issues out to any
|
|
570
|
+
number of code repos, and both label names are yours to configure.
|
|
2506
571
|
|
|
2507
|
-
##
|
|
572
|
+
## The reference
|
|
2508
573
|
|
|
2509
|
-
|
|
2510
|
-
|
|
574
|
+
The guide stops at "operate the fleet". Everything deeper moves unchanged to
|
|
575
|
+
[`REFERENCE.md`](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md):
|
|
2511
576
|
|
|
2512
|
-
|
|
|
577
|
+
| Subject | In REFERENCE.md |
|
|
2513
578
|
| --- | --- |
|
|
2514
|
-
|
|
|
2515
|
-
|
|
|
2516
|
-
|
|
|
2517
|
-
|
|
|
2518
|
-
|
|
|
2519
|
-
|
|
|
2520
|
-
|
|
|
2521
|
-
|
|
|
2522
|
-
|
|
2523
|
-
|
|
2524
|
-
|
|
2525
|
-
|
|
2526
|
-
|
|
2527
|
-
|
|
2528
|
-
|
|
2529
|
-
|
|
2530
|
-
|
|
2531
|
-
|
|
2532
|
-
`read` / `grep` / `glob` when the tool's path resolves outside that worktree
|
|
2533
|
-
(symlink-aware). Target selection was already mechanical — only a repo in
|
|
2534
|
-
`routing.repos` is ever checked out — and the caps still bound *how much* work
|
|
2535
|
-
happens.
|
|
2536
|
-
|
|
2537
|
-
General shell access is not confined to the worktree. Its argument is an opaque
|
|
2538
|
-
program, so the brief still forbids path escape and the deploy-level answer is a
|
|
2539
|
-
least-privilege worker uid (below). The narrower release-policy tripwire does
|
|
2540
|
-
inspect explicit command shapes such as `git tag`, `npm publish`, and deploy
|
|
2541
|
-
verbs; it blocks those before execution when `releasePolicy` is `none`.
|
|
2542
|
-
|
|
2543
|
-
#### Integrity tripwire (package self-hash)
|
|
2544
|
-
|
|
2545
|
-
Separately, the conductor watches *itself*. At startup the daemon sha256s every
|
|
2546
|
-
`.ts` and `.md` file of its own installed `src/` — the dispatcher and the briefs
|
|
2547
|
-
both, since rewriting a brief buys more than rewriting the loop — and re-walks
|
|
2548
|
-
that tree on every tick (about 0.6 ms). Any difference at all, changed or added
|
|
2549
|
-
or removed, is read as the package having been modified underneath a running
|
|
2550
|
-
daemon: the tick claims nothing, the fleet is paused, and a tier-2 escalation
|
|
2551
|
-
naming the first few differing paths pages you **once**, not every five minutes.
|
|
2552
|
-
|
|
2553
|
-
**A normal deploy never trips it.** The baseline is recorded per daemon process,
|
|
2554
|
-
so installing a new build and restarting the unit re-records it from the new
|
|
2555
|
-
files; only a change that lands *while* a daemon is holding the package open can
|
|
2556
|
-
diverge from it. That also means `omp-conductor resume` on its own will not hold
|
|
2557
|
-
— the next tick re-walks, still differs, and pauses again. Put the files back, or
|
|
2558
|
-
restart onto the build you meant to be running.
|
|
2559
|
-
|
|
2560
|
-
This catches a worker (or human) that still managed to edit the live install —
|
|
2561
|
-
including via `bash` — after the fact. It is detection for the package boundary,
|
|
2562
|
-
not a substitute for the worktree gate or a dedicated uid.
|
|
2563
|
-
|
|
2564
|
-
#### Least-privilege worker uid (deploy)
|
|
2565
|
-
|
|
2566
|
-
The largest remaining win is OS-level: run the daemon (or at least worker
|
|
2567
|
-
sessions, when the harness supports a uid switch) as a user that can write only
|
|
2568
|
-
its worktrees and mirrors. A sketch that matches the reference single-host
|
|
2569
|
-
deploy:
|
|
2570
|
-
|
|
2571
|
-
1. Create a system user, e.g. `conductor-worker`, with home under
|
|
2572
|
-
`/var/lib/conductor-worker` (or similar).
|
|
2573
|
-
2. `chown` the project's `workspaceRoot` and `mirrorRoot` to that user; leave
|
|
2574
|
-
`~/.omp/conductor/config.json` readable only by the operator/daemon account
|
|
2575
|
-
(`0600` as shipped).
|
|
2576
|
-
3. Do **not** put the worker uid in `docker` / `sudoers`, and do not give it the
|
|
2577
|
-
operator's `gh` auth if a narrower deploy token can open PRs in the routed
|
|
2578
|
-
repos alone.
|
|
2579
|
-
4. Point the [example systemd unit](systemd/omp-conductor.service.example)
|
|
2580
|
-
`User=` / `Group=` at that account once the daemon itself should run
|
|
2581
|
-
unprivileged end-to-end.
|
|
2582
|
-
|
|
2583
|
-
Until that uid exists, a root-or-operator daemon still has a mechanical
|
|
2584
|
-
worktree gate on structured tools and an integrity tripwire on its own package —
|
|
2585
|
-
but `bash` plus host credentials remain a prompt-and-deploy problem.
|
|
2586
|
-
|
|
2587
|
-
### The orchestrator is unconfined, deliberately
|
|
2588
|
-
|
|
2589
|
-
There is **no mechanical file gate on the orchestrator session**, and that is an
|
|
2590
|
-
operator decision rather than an omission (#143).
|
|
2591
|
-
|
|
2592
|
-
A previous release jailed it to an allowlist. That gate could only ever be
|
|
2593
|
-
installed by `createLocalSession`, so it existed exactly in the sessions this
|
|
2594
|
-
daemon spawns — and the supported shape for a heartbeat orchestrator is an
|
|
2595
|
-
`omp` session the operator starts themselves, which never had it. A boundary
|
|
2596
|
-
present in one deployment out of two is not a boundary, and the brief asserting
|
|
2597
|
-
it was absolute was the worse half of the bug: a session that believes it is
|
|
2598
|
-
gated stops checking itself.
|
|
2599
|
-
|
|
2600
|
-
What holds the orchestrator instead:
|
|
2601
|
-
|
|
2602
|
-
| | |
|
|
2603
|
-
| --- | --- |
|
|
2604
|
-
| **The brief** | `ORCHESTRATOR.md`'s hard boundaries — never read or edit a worker's checkout or the mirror cache; when you need a run's code, read its PR. |
|
|
2605
|
-
| **The action ledger** | Every `conductor_*` mutation and operator-selected next-attempt turn budget remains auditable. `omp-conductor ledger` shows both, including refused calls and consumed or replaced budget overrides. |
|
|
2606
|
-
| **The dispatcher** | Merge, label and release authority are checked in the daemon against the operator's grant, across a process boundary, never in the prompt. |
|
|
2607
|
-
|
|
2608
|
-
Unconfined means auditable, not licensed. `orchestratorReadPaths` is retired: it
|
|
2609
|
-
is still accepted in a config and ignored, so a fleet carrying it upgrades
|
|
2610
|
-
without editing anything.
|
|
2611
|
-
|
|
2612
|
-
## The mediated verbs (#126)
|
|
2613
|
-
|
|
2614
|
-
A session can reach `gh`: it inherits the daemon's environment, credentials and
|
|
2615
|
-
all. It is told not to publish with it. These verbs are the sanctioned route
|
|
2616
|
-
instead, because the dispatcher owns the settlement record — a push or a PR the
|
|
2617
|
-
daemon did not perform is a run it cannot account for, and the checks that would
|
|
2618
|
-
have refused it never ran. The point of them is *where those checks run*: in the
|
|
2619
|
-
daemon, across a process boundary, not in a prompt the model can rewrite.
|
|
2620
|
-
|
|
2621
|
-
### The verbs
|
|
2622
|
-
|
|
2623
|
-
| Verb | Allowed caller | What the daemon checks before acting |
|
|
2624
|
-
| --- | --- | --- |
|
|
2625
|
-
| `conductor_push` | the worker owning the run | The ref is exactly `refs/heads/<that run's branch>`. Fast-forward only; there is no force argument to reject because none is declared. |
|
|
2626
|
-
| `conductor_pr_create` | the worker owning the run | The run has no open PR (the same guard admission uses); head is the run branch; base is the repo's configured `defaultBranch`. |
|
|
2627
|
-
| `conductor_pr_status` | worker or orchestrator | Read-only. A worker reads only its own run's PR; an orchestrator may name any open PR in a routed project repo. |
|
|
2628
|
-
| `conductor_pr_update_branch` | orchestrator, or the worker owning the run | The PR belongs to this project and is open. A worker may only name its own run's PR. |
|
|
2629
|
-
| `conductor_pr_merge` | **orchestrator only** | Ordinarily, `authority.merge` equals the caller. A hand-edited `recoveryMerges` entry may instead authorize one exact unrecorded PR/head/reason while held. In both paths, `headSha` equals the live head *at execution time*; checks are green at that same SHA; the project route and migration chain are valid; the project's single merge slot is free. |
|
|
2630
|
-
| `conductor_label` | **orchestrator only** | The label is in the project's own vocabulary. Lifecycle labels stay the daemon's. |
|
|
2631
|
-
| `conductor_release` | **orchestrator only** | `authority.release` equals the caller; the per-shape grant permits it; the artefact or environment was declared; the release preconditions hold; the `reason` is in the closed enum. `version-bump-pr` creates or re-validates one deterministic version-only PR and, on a later call, merges only its exact green head through the project's single merge slot. |
|
|
2632
|
-
|
|
2633
|
-
Standing merge and release authority use the same exact rule: **the caller's
|
|
2634
|
-
role must equal the configured holder.** `authority` has exactly two values, so
|
|
2635
|
-
a `!== "human"` test would have let a *worker* release. A worker is refused
|
|
2636
|
-
every release shape under the most permissive config there is. The exact
|
|
2637
|
-
operator-authored recovery tuple below is the sole authority exception inside
|
|
2638
|
-
`conductor_pr_merge`; a reviewed version bump is instead a `conductor_release`
|
|
2639
|
-
operation governed throughout by release authority.
|
|
2640
|
-
|
|
2641
|
-
`recoveryMerges` is deliberately narrower than standing merge authority. It is
|
|
2642
|
-
an operator-authored, one-PR escape hatch for a recovery branch that cannot have
|
|
2643
|
-
a run row—for example, a conflict repair created after the fleet was held. It
|
|
2644
|
-
does not admit new work, unpause the fleet, widen repository routing, bypass
|
|
2645
|
-
live-head or check validation, or make a general class of PRs mergeable.
|
|
2646
|
-
Authorizations are re-read from config on every call and every attempted merge
|
|
2647
|
-
is written to the ordinary verb ledger, including refusals.
|
|
2648
|
-
|
|
2649
|
-
For a repo with `release.versionFile`, call `conductor_release` with
|
|
2650
|
-
`shape=version-bump-pr` and the intended `v<semver>` tag. The requested version
|
|
2651
|
-
must be newer than the live semantic version. The first call creates
|
|
2652
|
-
`conductor/release-<version>` from the live default branch, changes only the
|
|
2653
|
-
declared JSON `version`, and opens a normal PR. Call it again after CI: the daemon
|
|
2654
|
-
re-reads that exact PR head, verifies the PR contains only the semantic version
|
|
2655
|
-
change, requires green checks, and merges with GitHub's exact-head guard. The
|
|
2656
|
-
ordinary action ledger records both calls. A raw source push is never delegated,
|
|
2657
|
-
and a worker makes no release decision.
|
|
2658
|
-
|
|
2659
|
-
For Git-backed releases, a repo that declares `release.versionFile` refuses both
|
|
2660
|
-
tag creation and a new tag push until the live default branch's version matches
|
|
2661
|
-
the requested tag. `git-tag` is idempotent when the named tag exists locally but
|
|
2662
|
-
has not been pushed: it re-points the tag to the verified live default-branch
|
|
2663
|
-
head. `git-push-tags` performs the same re-point immediately before pushing if
|
|
2664
|
-
the default branch moved between the two calls. A tag already published on
|
|
2665
|
-
origin is immutable: an identical tag is accepted as already complete, while a
|
|
2666
|
-
different published target is refused and must use a new tag name.
|
|
2667
|
-
|
|
2668
|
-
A `github-release` for the same repo likewise requires that reviewed tag to be
|
|
2669
|
-
present on origin and verifies the tag's version file before creating the
|
|
2670
|
-
release; it never lets GitHub synthesize the missing tag.
|
|
2671
|
-
|
|
2672
|
-
### The transport
|
|
2673
|
-
|
|
2674
|
-
Identity is never an argument. `project`, `run`, `issue` and the caller's role
|
|
2675
|
-
come from **which socket the call arrived on**, and a request carrying any of
|
|
2676
|
-
those field names is refused outright, named. So a worker on run X cannot *ask*
|
|
2677
|
-
to merge run Y's PR — on its own channel that request is unexpressible.
|
|
2678
|
-
|
|
2679
|
-
**Each channel is bound to one process, because the modes cannot tell sessions
|
|
2680
|
-
apart.** Every session runs as the daemon's own uid, so it matches the *owner*
|
|
2681
|
-
class here: it can list this directory and connect to any socket in it, including
|
|
2682
|
-
the orchestrator's. Authorisation and the ledger both read the role from the
|
|
2683
|
-
channel, so a worker doing that would have been authorised as the orchestrator
|
|
2684
|
-
(under `authority.merge: "orchestrator"`) *and recorded as* the orchestrator. No
|
|
2685
|
-
file mode closes that — the owner bits belong to the uid the session already has.
|
|
2686
|
-
|
|
2687
|
-
So the daemon binds each channel to the **pid it spawned for that session**, and
|
|
2688
|
-
refuses a connection from anything else without answering it, logged the way an
|
|
2689
|
-
impersonation is. Until a channel is bound it refuses everything, because the
|
|
2690
|
-
socket necessarily exists before the child that connects to it. The kernel
|
|
2691
|
-
supplies the pid: `SO_PEERCRED` on Linux, `LOCAL_PEERPID` on macOS. A host where
|
|
2692
|
-
neither can be asked — no loadable libc, or the call refused — refuses every
|
|
2693
|
-
connection on a bound channel and says so at startup, rather than falling back to
|
|
2694
|
-
the uid, which under one shared uid is no check at all.
|
|
2695
|
-
|
|
2696
|
-
The residual is narrow, real, and worth stating: one uid can `ptrace` and signal
|
|
2697
|
-
its siblings, so a determined session can still interfere with the process that
|
|
2698
|
-
*is* bound. That is a far higher bar than connecting to a socket, and closing it
|
|
2699
|
-
needs separate OS principals.
|
|
2700
|
-
|
|
2701
|
-
```
|
|
2702
|
-
<state dir>/verbs/ daemon-owned, mode 0711
|
|
2703
|
-
run-7-9a783d877d422b9e.sock 0600, bound for run 7
|
|
2704
|
-
run-9-1c40e2a5b6d3f018.sock 0600, bound for run 9
|
|
2705
|
-
orchestrator-4b1f...c2.sock 0600, the orchestrator's
|
|
2706
|
-
```
|
|
2707
|
-
|
|
2708
|
-
**What these modes buy, and what they do not.** They keep every *other local
|
|
2709
|
-
account* out: `0711` on the parent is traversable but not listable, so no other
|
|
2710
|
-
user can enumerate the fleet's sockets, the suffixes are unguessable, and only
|
|
2711
|
-
the daemon's uid can connect to a `0600` socket at all.
|
|
2712
|
-
|
|
2713
|
-
They are **not** a boundary between runs. Sessions are child processes of the
|
|
2714
|
-
daemon running as its own uid, so a session matches the owner class on all of
|
|
2715
|
-
these: it could list the directory and connect to a sibling's socket. Each run is
|
|
2716
|
-
*handed* its own path and nothing else, which is a convention the run has no
|
|
2717
|
-
reason to break — not an enforcement. What makes breaking it visible is the
|
|
2718
|
-
[ledger](#the-ledger): every call is recorded with the channel it
|
|
2719
|
-
arrived on, so a worker calling on another run's socket is in the record.
|
|
2720
|
-
|
|
2721
|
-
Closing that properly needs the sessions to be different OS principals. A
|
|
2722
|
-
per-run credential boundary that did exactly this shipped and was removed in
|
|
2723
|
-
0.5.0 — it worked, and the cost was that it also hid the operator's own model
|
|
2724
|
-
credential from every session, so nothing could start. It is not worth
|
|
2725
|
-
re-litigating without solving that first.
|
|
2726
|
-
|
|
2727
|
-
Before binding, the daemon verifies every component of the path is owned by
|
|
2728
|
-
itself (or root), free of symlinks, and unwritable by anyone else; a failed
|
|
2729
|
-
check **refuses dispatch** rather than degrading. Paths are unguessably
|
|
2730
|
-
suffixed, and only the daemon ever unlinks one.
|
|
2731
|
-
|
|
2732
|
-
Peer credentials are asserted server-side — `getpeereid` on macOS, `SO_PEERCRED`
|
|
2733
|
-
on Linux — and the daemon states at startup exactly what that buys rather than
|
|
2734
|
-
implying more. Sessions are child processes running under the daemon's own uid,
|
|
2735
|
-
so the peer check proves the caller is a local process on this host; it is the
|
|
2736
|
-
socket, not the uid, that says which run is calling. A connection whose peer
|
|
2737
|
-
cannot be read at all is closed with no reply and logged.
|
|
2738
|
-
|
|
2739
|
-
```
|
|
2740
|
-
verb transport: verb sockets in ~/.omp/conductor/verbs (mode 711); each socket
|
|
2741
|
-
0600 under the daemon's own uid; peer uid asserted with getpeereid
|
|
2742
|
-
```
|
|
2743
|
-
|
|
2744
|
-
**No mutation route exists on the HTTP port**, and none may be added. That
|
|
2745
|
-
surface is unauthenticated loopback TCP reachable by any local user; a `PUT` or
|
|
2746
|
-
`POST` at any verb path answers 404, pinned by a test.
|
|
2747
|
-
|
|
2748
|
-
The child-side tool handler is a thin client only. It forwards arguments and
|
|
2749
|
-
renders the answer — no policy branch, no local fallback, no second route. With
|
|
2750
|
-
no socket it fails closed and says so, rather than reaching for `git push`.
|
|
2751
|
-
|
|
2752
|
-
### The ledger
|
|
2753
|
-
|
|
2754
|
-
Every mutating verb call is recorded with its arguments, the decision, the
|
|
2755
|
-
named refusal reason and any resulting SHA. Reads are not: a status poll every
|
|
2756
|
-
thirty seconds would bury the refusals the record exists to surface.
|
|
2757
|
-
|
|
2758
|
-
Every `extend` that sets a next-attempt budget also appends an audit entry.
|
|
2759
|
-
Replacing or consuming the pending override does not erase that history.
|
|
2760
|
-
|
|
2761
|
-
```console
|
|
2762
|
-
$ omp-conductor ledger --issue 7
|
|
2763
|
-
acme — 3 verb call(s), 1 refused (newest first)
|
|
2764
|
-
2026-08-09 11:04:12 REFUSE conductor_pr_merge worker #7 [role-not-allowed]
|
|
2765
|
-
prUrl=https://github.com/acme/api/pull/7 headSha=9a783d8… reason=preconditions-met
|
|
2766
|
-
refused: merge authority is the orchestrator's, never a worker session's.
|
|
2767
|
-
2026-08-09 10:58:03 ALLOW conductor_pr_create worker #7
|
|
2768
|
-
title=fix: settle the head check body=Closes acme/tracker#7
|
|
2769
|
-
opened https://github.com/acme/api/pull/7 (conductor/issue-7 → main).
|
|
2770
|
-
2026-08-09 10:57:41 ALLOW conductor_push worker #7 9a783d877d42
|
|
2771
|
-
(no arguments)
|
|
2772
|
-
pushed refs/heads/conductor/issue-7 at 9a783d877d42….
|
|
2773
|
-
```
|
|
579
|
+
| The upgrade lifecycle (`omp-conductor upgrade`) | [Updating](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#updating) |
|
|
580
|
+
| The wizard internals, editing one setting, the brief overlay | [Onboarding](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#onboarding) |
|
|
581
|
+
| What one dispatch tick actually does | [How one tick works](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#how-one-tick-works) |
|
|
582
|
+
| Routing rules, and why unroutable never guesses | [Routing](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#routing) |
|
|
583
|
+
| Host sizing and memory | [Host sizing and memory](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#host-sizing-and-memory) |
|
|
584
|
+
| Caps and the plan allowance | [Caps](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#caps) |
|
|
585
|
+
| The worker model | [Worker model](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#worker-model) |
|
|
586
|
+
| Code-graph discovery | [Code-graph discovery](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#code-graph-discovery) |
|
|
587
|
+
| Escalation tiers | [Escalation tiers](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#escalation-tiers) |
|
|
588
|
+
| Report delivery (the outbox) | [Report delivery](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#report-delivery-the-outbox) |
|
|
589
|
+
| The decision ledger | [The decision ledger](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#the-decision-ledger-136) |
|
|
590
|
+
| Failure classes and recovery | [Failure classes](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#failure-classes-and-recovery-by-class-132) |
|
|
591
|
+
| Configuration (`config.json`) key by key | [Configuration](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#configuration) |
|
|
592
|
+
| The orchestrator tick internals | [Orchestrator tick](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#orchestrator-tick) |
|
|
593
|
+
| The full CLI reference and health endpoint | [CLI reference](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#cli-reference) |
|
|
594
|
+
| The mediated verbs | [The mediated verbs](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#the-mediated-verbs-126) |
|
|
595
|
+
| Known limitations | [Limitations](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#limitations) |
|
|
596
|
+
| The licence | [License](https://github.com/TerrifiedBug/conductor/blob/main/omp/REFERENCE.md#license) |
|
|
2774
597
|
|
|
2775
|
-
The newest few also appear in `omp-conductor status`, because a refused merge is
|
|
2776
|
-
news: it means a session tried to do something the config does not permit.
|
|
2777
|
-
|
|
2778
|
-
`release-policy.ts` stays installed as defence in depth — it refuses early, in
|
|
2779
|
-
the session, with an explanation the model can act on in the same turn, and it
|
|
2780
|
-
leaves a durable record that something tried. It is no longer what *stops* a
|
|
2781
|
-
release. Treat a block there as evidence about a session's intentions; the
|
|
2782
|
-
daemon is what prevented it.
|
|
2783
|
-
|
|
2784
|
-
|
|
2785
|
-
## Limitations
|
|
2786
|
-
|
|
2787
|
-
Known and deliberate in this version:
|
|
2788
|
-
|
|
2789
|
-
- **`gh` is shelled out to.** Every tracker operation spawns a process and does its
|
|
2790
|
-
own TLS handshake (roughly 200-400 ms each), and failures are classified by
|
|
2791
|
-
matching human-readable stderr rather than a status code. The upside is that no
|
|
2792
|
-
token is ever handled, stored or logged by the daemon.
|
|
2793
|
-
- **`listReady` fetches a single page of 100 issues.** A queue deeper than 100
|
|
2794
|
-
ready issues truncates silently. A backlog that size is a staffing problem before
|
|
2795
|
-
it is a paging one.
|
|
2796
|
-
- **Spend accounting depends on harness telemetry.** Cost arrives only when the
|
|
2797
|
-
harness run carries it; without it `spendUsd` reads `0`, `status` shows `$0.00`,
|
|
2798
|
-
and the daily-spend cap never fires. The turn and wall-clock ceilings are what
|
|
2799
|
-
actually bound a runaway in that case. Do not treat `$0.00` as proof that nothing
|
|
2800
|
-
was spent.
|
|
2801
|
-
- **GitHub is the only tracker.** The internal `Tracker` port is deliberately
|
|
2802
|
-
provider-neutral, but `tracker.kind` accepts only `"github"` today.
|
|
2803
|
-
- **One daemon serves every configured project by default.** `setup host` writes a
|
|
2804
|
-
unit without `--project`. Pass `--project NAME` only to filter a foreground or
|
|
2805
|
-
temporary daemon down to one project.
|
|
2806
|
-
- **Labels are matched exactly and case-sensitively.** `Ready-For-Agent` is not
|
|
2807
|
-
`ready-for-agent`, and the mismatch is silent: the issue is simply never picked
|
|
2808
|
-
up.
|
|
2809
|
-
- **No cross-process lock on the mirrors.** Two dispatch loops fetching the same
|
|
2810
|
-
repo at the same instant can collide on git's ref locks; the run fails and is
|
|
2811
|
-
retried rather than corrupted.
|
|
2812
|
-
- **Uniquely local mirror branches are retained.** Terminal runs are reaped
|
|
2813
|
-
automatically only after every commit exists on a remote ref. A failed salvage
|
|
2814
|
-
push deliberately leaves its branch and tree for an operator rather than
|
|
2815
|
-
trading disk hygiene for data loss.
|
|
2816
|
-
- **`stop` is a bounded best-effort drain.** A signal stops new ticks and the
|
|
2817
|
-
daemon waits for its active worker pool before closing the store. The CLI
|
|
2818
|
-
escalates to `SIGKILL` after 10 seconds, so a worker that needs longer is
|
|
2819
|
-
orphaned and salvaged on restart. Use `pause`, wait for `workers 0 / N`, then
|
|
2820
|
-
stop when a clean drain matters. A supervising unit should set
|
|
2821
|
-
`SuccessExitStatus=0 143`, and operators should prefer `omp-conductor stop` /
|
|
2822
|
-
`systemctl stop` over raw `kill`, so `Restart=on-failure` cannot misread a
|
|
2823
|
-
deliberate stop as a crash.
|
|
2824
|
-
- **A failed orchestrator degrades quietly.** The daemon logs a warning and keeps
|
|
2825
|
-
running, but tier-1 escalations then land in issue comments — which is exactly the
|
|
2826
|
-
"nobody reads it until morning" path the orchestrator exists to avoid. The warning
|
|
2827
|
-
is in `daemon.log`; nothing pages you about it.
|
|
2828
|
-
- **Workers are not terminal panes, so you cannot watch them there.** Each
|
|
2829
|
-
worker is an omp session the daemon starts as a child process. The resident
|
|
2830
|
-
daemon tracks workers in a background pool so the five-minute loop keeps
|
|
2831
|
-
settling PRs and checking capacity; shutdown waits for that pool. Herdr still
|
|
2832
|
-
shows exactly one pane (the orchestrator's) regardless of concurrency.
|
|
2833
|
-
|
|
2834
|
-
The cap does work. The admission loop (`admitCandidates` in `src/daemon.ts`) computes
|
|
2835
|
-
`slots = maxConcurrentWorkers - live workers`, admits at most that many issues
|
|
2836
|
-
per tick, and dispatches them together. To see them, read `omp-conductor
|
|
2837
|
-
status`, which lists every occupied issue, or follow `daemon.log`.
|
|
2838
|
-
- **Report delivery is at-least-once, never exactly-once.** The Telegram Bot API
|
|
2839
|
-
takes no client-supplied idempotency key, so the window between "Telegram
|
|
2840
|
-
accepted it" and "SQLite recorded that" is irreducible. The daemon resolves it
|
|
2841
|
-
toward a duplicate — the report is retried and the retry says it may be a
|
|
2842
|
-
repeat — because a duplicate you can recognise by its report id is cheaper
|
|
2843
|
-
than a silently dropped page. `delivered` means Telegram accepted an attempt,
|
|
2844
|
-
not that exactly one message exists. See
|
|
2845
|
-
[Report delivery](#report-delivery-the-outbox).
|
|
2846
|
-
- **Workers stop at green PRs.** They are never authorised to merge, release or
|
|
2847
|
-
deploy: those actions default to a human, and while setup may grant either to
|
|
2848
|
-
the orchestrator, `authority` never grants them to a worker or the dispatch
|
|
2849
|
-
daemon. The verbs refuse a worker mechanically, and a worker reaching for
|
|
2850
|
-
another session's channel is refused too — each channel is bound to the pid the
|
|
2851
|
-
daemon spawned for it. What remains a prohibition rather than a gate is shelling
|
|
2852
|
-
out to `gh` directly: sessions inherit the daemon's credentials. That shows up as
|
|
2853
|
-
a mutation with no matching ledger entry, which is a mismatch an operator can
|
|
2854
|
-
find.
|
|
2855
|
-
- **The worker gate is partial, and the orchestrator has none.** A worker's
|
|
2856
|
-
structured `write` / `edit` / `read` / `grep` / `glob` calls are gated to its
|
|
2857
|
-
worktree by an inline harness extension; `bash` is not, so a shell one-liner
|
|
2858
|
-
can still leave the tree, and no claim in this README says otherwise. The
|
|
2859
|
-
orchestrator is [unconfined on purpose](#the-orchestrator-is-unconfined-deliberately)
|
|
2860
|
-
— its boundaries are its brief and the verb ledger. Prefer a
|
|
2861
|
-
[least-privilege worker uid](#least-privilege-worker-uid-deploy); the
|
|
2862
|
-
[integrity tripwire](#integrity-tripwire-package-self-hash) still pages if the
|
|
2863
|
-
installed package itself changes under a live daemon.
|
|
2864
|
-
|
|
2865
|
-
|
|
2866
|
-
## License
|
|
2867
|
-
|
|
2868
|
-
MIT
|