omp-conductor 0.3.18 → 0.3.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +344 -179
- package/package.json +1 -1
- package/skills/conductor-onboarding/SKILL.md +43 -50
- package/skills/conductor-update/SKILL.md +15 -165
- package/src/board.ts +734 -0
- package/src/brief-upgrade.ts +102 -19
- package/src/briefs/orchestrator.md +32 -16
- package/src/briefs/worker.md +7 -2
- package/src/cli.ts +136 -53
- package/src/config.ts +22 -6
- package/src/daemon.ts +673 -126
- package/src/fleet.ts +64 -6
- package/src/graph-health.ts +296 -0
- package/src/graph.ts +6 -6
- package/src/omp.ts +15 -4
- package/src/orchestrator-tick.ts +157 -10
- package/src/orchestrator.ts +8 -1
- package/src/plugin.ts +173 -45
- package/src/release-policy.ts +202 -0
- package/src/setup-host.ts +285 -0
- package/src/setup.ts +24 -20
- package/src/store.ts +373 -13
- package/src/tracker/github.ts +171 -3
- package/src/transcript.ts +45 -0
- package/src/types.ts +145 -2
- package/src/unblock.ts +26 -14
- package/src/upgrade.ts +327 -0
- package/src/worker.ts +67 -30
- package/src/worktree.ts +66 -0
- package/systemd/omp-conductor.service.example +5 -3
package/README.md
CHANGED
|
@@ -18,11 +18,11 @@ authority are chosen during setup and default to `human`. Granting either action
|
|
|
18
18
|
to the orchestrator never grants it to a worker or the dispatch daemon. Releases
|
|
19
19
|
remain batched from coherent groups of merged work, never cut one per worker PR.
|
|
20
20
|
|
|
21
|
-
Code counts every limit that decides whether work starts: concurrency, dollars
|
|
22
|
-
day, turns and wall clock per worker,
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
21
|
+
Code counts every limit that decides whether work starts: concurrency, dollars
|
|
22
|
+
per day, turns and wall clock per worker, failed implementation attempts, and
|
|
23
|
+
operational continuations per issue. None is left to the model. The dispatcher
|
|
24
|
+
enforces these budgets before anything is claimed and kills workers that exceed
|
|
25
|
+
their per-run limits.
|
|
26
26
|
|
|
27
27
|
When a run does get stuck, the first responder is not you. A tier-1 escalation is
|
|
28
28
|
injected into a long-lived **orchestrator session** that can read the issue and the
|
|
@@ -39,7 +39,7 @@ The package ships three deployables, plus two skills:
|
|
|
39
39
|
| Standalone daemon | `omp-conductor` binary | The dispatch loop, managed as a background process (`start` / `stop` / `restart`) with a `/healthz` endpoint for a supervisor. |
|
|
40
40
|
| Orchestrator heartbeat | omp extension, activated by `.conductor-tick.json` | Prompts a 24/7 orchestrator session on a fixed interval so its standing loop actually runs, and marks the session stalled when its prompts stop being consumed. Inert in every other session — including a second session opened in the fleet's own directory. See [Orchestrator tick](#orchestrator-tick). |
|
|
41
41
|
| Onboarding skill | `skill://conductor-onboarding` | Directs an omp session to interview you, read your repos for real CI gates, and tailor `ORCHESTRATOR.md` — then finish through the wizard. Discovered automatically once the plugin is installed. See [Onboarding](#onboarding). |
|
|
42
|
-
| Update skill | `skill://conductor-update` |
|
|
42
|
+
| Update skill | `skill://conductor-update` | Thin natural-language wrapper over the deterministic `omp-conductor upgrade` command. See [Updating](#updating). |
|
|
43
43
|
|
|
44
44
|
The first two are thin wrappers over the same `daemon.ts`, so the plugin and the
|
|
45
45
|
CLI cannot disagree about what a cap means or where the state lives. Claiming is
|
|
@@ -134,8 +134,11 @@ number of code repos, and both label names are yours to configure.
|
|
|
134
134
|
omp plugin install omp-conductor
|
|
135
135
|
```
|
|
136
136
|
|
|
137
|
-
From a checkout of the monorepo, `./setup.sh`
|
|
138
|
-
|
|
137
|
+
From a checkout of the monorepo, `./setup.sh` checks both plugins. It preserves
|
|
138
|
+
an existing npm-managed `omp-conductor` and links only the Herdr half, so running
|
|
139
|
+
setup on a release-based fleet cannot silently switch omp to mutable source.
|
|
140
|
+
`./setup.sh install --force-link` is the explicit opt-in to link both checkout
|
|
141
|
+
directories.
|
|
139
142
|
|
|
140
143
|
### Prerequisites
|
|
141
144
|
|
|
@@ -168,25 +171,39 @@ Also required on the host:
|
|
|
168
171
|
|
|
169
172
|
## Updating
|
|
170
173
|
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
the
|
|
174
|
+
Run one command from a shell outside the target `herdr-fleet.service`:
|
|
175
|
+
|
|
176
|
+
```bash
|
|
177
|
+
omp-conductor upgrade
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
It resolves the latest published npm release, pauses new claims, drains active
|
|
181
|
+
workers, and pins that exact release across the Bun-global CLI, omp plugin, and
|
|
182
|
+
Herdr plugin. It also recomposes the conductor-owned brief floor, restarts Herdr
|
|
183
|
+
and the daemon, waits for pane recovery, verifies the installed identities and
|
|
184
|
+
layered fleet status twice, then restores the original dispatch state.
|
|
185
|
+
|
|
186
|
+
Use `omp-conductor upgrade --to X.Y.Z` for an explicit published version. The
|
|
187
|
+
command exits without changing anything when all three surfaces already use that
|
|
188
|
+
release, the Herdr plugin is pinned to its exact `gitHead`, and the brief is
|
|
189
|
+
current.
|
|
190
|
+
|
|
191
|
+
When a daemon is live, the command drains and restarts its recorded project.
|
|
192
|
+
An explicit `--project` that names a different project is rejected before pause
|
|
193
|
+
or installation, so an operator cannot drain one queue and kill another queue's
|
|
194
|
+
workers.
|
|
178
195
|
|
|
179
196
|
Ticks remain in their existing armed or disarmed state, so an ordinary update
|
|
180
197
|
does not halt the exact pane or require another Telegram arm challenge. Any
|
|
181
198
|
installation, reload, or verification failure leaves dispatch paused instead of
|
|
182
|
-
bringing up a mixed fleet. The
|
|
183
|
-
|
|
184
|
-
that
|
|
185
|
-
|
|
199
|
+
bringing up a mixed fleet. The command never publishes npm, edits an install
|
|
200
|
+
root, or delegates lifecycle steps to an AI session. It refuses to run inside a
|
|
201
|
+
Herdr-managed session because an updater that restarts itself cannot verify the
|
|
202
|
+
result.
|
|
186
203
|
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
204
|
+
The bundled `skill://conductor-update` remains available as a natural-language
|
|
205
|
+
entry point, but it only runs this command; the lifecycle is implemented and
|
|
206
|
+
verified by the CLI.
|
|
190
207
|
|
|
191
208
|
## Onboarding
|
|
192
209
|
|
|
@@ -194,8 +211,8 @@ Onboarding this package has two layers, and installing it gives you both.
|
|
|
194
211
|
|
|
195
212
|
| Layer | What it is | What it owns |
|
|
196
213
|
| --- | --- | --- |
|
|
197
|
-
| **`/conductor setup`** | The deterministic wizard.
|
|
198
|
-
| **`skill://conductor-onboarding`** | A skill bundled in this package (`skills/conductor-onboarding/SKILL.md`)
|
|
214
|
+
| **`/conductor setup`** | The deterministic wizard. It shows the label, runtime, and dry-run plans before one consent step. A configured project can amend one area. | **Mechanical setup.** It writes `config.json`, labels, the brief, an external heartbeat config, and a staged systemd unit. It also completes the paused smoke and arm gates. |
|
|
215
|
+
| **`skill://conductor-onboarding`** | A skill bundled in this package (`skills/conductor-onboarding/SKILL.md`). Any omp session can discover it after plugin installation. | **Brief authoring.** It supplies the judgment that fixed prompts cannot. |
|
|
199
216
|
|
|
200
217
|
The split exists because the two halves fail differently. A wrong config value is
|
|
201
218
|
a run that errors on the next tick; a wrong release boundary is a fleet that
|
|
@@ -313,75 +330,113 @@ omp-conductor brief-upgrade --retrofit --apply
|
|
|
313
330
|
```
|
|
314
331
|
|
|
315
332
|
- **Overlay already active** (`POLICY.md` present): protocol updates need no brief-upgrade.
|
|
316
|
-
- **Legacy bannered brief**: `--migrate` lifts the owned half into `POLICY.md`
|
|
333
|
+
- **Legacy bannered brief**: `--migrate` lifts the owned half into `POLICY.md`
|
|
334
|
+
and recomposes. Previous brief and policy versions go to
|
|
335
|
+
`$OMP_CONDUCTOR_HOME/backups/briefs/` (default
|
|
336
|
+
`~/.omp/conductor/backups/briefs/`), named with their source filename and
|
|
337
|
+
timestamp.
|
|
317
338
|
- **Hand-written brief** (no banner): `--retrofit` inserts the banner before the first Releases / Project context / Reporting / Amendments heading; then `--migrate`.
|
|
318
339
|
- **Legacy `--apply`**: still merges a bannered single-file brief when you need the old path.
|
|
340
|
+
- **Existing sidecars**: a composed refresh relocates conductor-generated
|
|
341
|
+
`ORCHESTRATOR.md.bak-<timestamp>` and `POLICY.md.bak-<timestamp>` files into
|
|
342
|
+
that backup directory. Other `.bak` files stay untouched.
|
|
319
343
|
|
|
320
344
|
`--file PATH` checks a brief that is not where the wizard would have put it.
|
|
321
345
|
|
|
322
|
-
The **Learning loop** proposes diffs against `POLICY.md` for you to approve over
|
|
346
|
+
The **Learning loop** proposes diffs against `POLICY.md` for you to approve over
|
|
347
|
+
Telegram. It also learns from repeated operational friction. The daemon
|
|
348
|
+
automatically rolls up repairable admission holds; the orchestrator records
|
|
349
|
+
judgments code cannot make with:
|
|
350
|
+
|
|
351
|
+
```bash
|
|
352
|
+
omp-conductor friction escalation-digest --detail "routine retry belonged in the digest" [--issue N]
|
|
353
|
+
omp-conductor friction report-noise --detail "green status repeated with no operator action"
|
|
354
|
+
omp-conductor friction report-surprise --detail "a material failure was missing from the report"
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
Three observations within seven days make a bounded signal eligible for one
|
|
358
|
+
tick. After it is surfaced, that signal cools down for seven days. A signal is
|
|
359
|
+
evidence to investigate, never an automatic policy edit: the existing one-at-a-
|
|
360
|
+
time Telegram approval, `POLICY.md`-only edit, Hard-boundary prohibition, and
|
|
361
|
+
**Amendments** log still apply.
|
|
323
362
|
|
|
324
363
|
## Quick start
|
|
325
364
|
|
|
326
|
-
1.
|
|
327
|
-
|
|
328
|
-
|
|
365
|
+
1. Install `omp-conductor` and `omp-telegram`. Pair the Telegram bot and enable its bridge.
|
|
366
|
+
2. Open the long-lived omp session in the fleet workspace.
|
|
367
|
+
|
|
368
|
+
The default workspace is `~/.omp/conductor/worktrees`. The wizard shows the actual path before it writes files.
|
|
369
|
+
|
|
370
|
+
3. Run the wizard:
|
|
329
371
|
|
|
330
372
|
```text
|
|
331
373
|
/conductor setup
|
|
332
374
|
```
|
|
333
375
|
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
376
|
+
The wizard reads the tracker with the same routing code as the daemon. It shows every issue that the next tick can route.
|
|
377
|
+
|
|
378
|
+
Nothing changes before the consent step. After consent, setup does these actions:
|
|
379
|
+
|
|
380
|
+
- creates the required labels;
|
|
381
|
+
- writes `config.json` and the selected brief;
|
|
382
|
+
- writes `.conductor-tick.json` for external orchestration;
|
|
383
|
+
- stages `omp-conductor.service` under the conductor state directory;
|
|
384
|
+
- runs `daemon --once` while dispatch is paused;
|
|
385
|
+
- starts a temporary daemon and proves `/healthz` and stored status;
|
|
386
|
+
- stops the temporary daemon;
|
|
387
|
+
- proves the current inbound Telegram path before it arms or resumes an external heartbeat;
|
|
388
|
+
- clears the dispatch pause only after all required gates pass.
|
|
389
|
+
|
|
390
|
+
If the arm proof cannot complete, setup keeps dispatch paused. The result shows the exact recovery commands.
|
|
341
391
|
|
|
342
|
-
|
|
343
|
-
orchestrator should be (`reporting.scope`), and whether to write an
|
|
344
|
-
`ORCHESTRATOR.md` you then own. See
|
|
345
|
-
[Your workflow vs. the package](#your-workflow-vs-the-package). If you would
|
|
346
|
-
rather be interviewed through those two, and have the brief tailored and your
|
|
347
|
-
gates read out of your CI config, start from
|
|
348
|
-
[Onboarding](#onboarding) instead.
|
|
392
|
+
4. On a systemd host, run the install commands that setup prints.
|
|
349
393
|
|
|
350
|
-
|
|
394
|
+
The commands install the staged unit, reload systemd, enable the unit, and restart it. The generated unit contains the current user, paths, project, port, and memory ceiling.
|
|
395
|
+
|
|
396
|
+
On a host without systemd, start the daemon directly:
|
|
351
397
|
|
|
352
398
|
```bash
|
|
353
399
|
omp-conductor start
|
|
354
400
|
```
|
|
355
401
|
|
|
356
|
-
|
|
357
|
-
clearing a previous `halt --pane` recovery pin so Herdr can resume the exact
|
|
358
|
-
conductor pane. Hosts without systemd or without that unit keep the standalone
|
|
359
|
-
daemon behaviour. It then waits until the daemon actually answers
|
|
360
|
-
`GET /healthz`; spawning is not starting. A daemon whose config is broken,
|
|
361
|
-
whose port is taken or whose database is locked exits within a second, and the
|
|
362
|
-
command fails with the tail of `daemon.log` instead of printing a false
|
|
363
|
-
success. It refuses to start a second daemon, naming the live pid. Starting
|
|
364
|
-
processes does not clear `pause` or arm ticks; those remain explicit operator
|
|
365
|
-
decisions.
|
|
366
|
-
|
|
367
|
-
Pane recovery spans two separately installed plugins: npm ships the omp
|
|
368
|
-
heartbeat/status half, while `herdr-conductor` supplies `recover.sh`. After an
|
|
369
|
-
npm upgrade, refresh the Herdr plugin from `TerrifiedBug/conductor/herdr` as
|
|
370
|
-
well; publishing or installing npm alone cannot add the recovery-side tick
|
|
371
|
-
request.
|
|
372
|
-
|
|
373
|
-
For a first run, take a single tick in the foreground and watch it:
|
|
402
|
+
5. Read the layered status:
|
|
374
403
|
|
|
375
404
|
```bash
|
|
376
|
-
omp-conductor
|
|
405
|
+
omp-conductor status
|
|
377
406
|
```
|
|
378
407
|
|
|
379
|
-
The
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
408
|
+
The result must show a running daemon, a healthy `/healthz`, armed ticks for external orchestration, and the configured project.
|
|
409
|
+
|
|
410
|
+
### First worker drill
|
|
411
|
+
|
|
412
|
+
Use a disposable target repository for this drill. Replace the values below with labels that the wizard showed.
|
|
413
|
+
|
|
414
|
+
```bash
|
|
415
|
+
TRACKER=acme/planning
|
|
416
|
+
QUEUE=ready-for-agent
|
|
417
|
+
ROUTE=repo:api
|
|
418
|
+
|
|
419
|
+
gh issue create --repo \"$TRACKER\" \
|
|
420
|
+
--title \"Conductor setup drill: add a marker file\" \
|
|
421
|
+
--body $'Add `conductor-smoke.txt` to the target repository.\\n\\nAcceptance: the file contains `setup path verified` and the pull request checks pass.' \
|
|
422
|
+
--label \"$ROUTE\" \
|
|
423
|
+
--label \"$QUEUE\"
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
The daemon claims the issue, creates a worktree, runs the configured gates, and opens a pull request. Follow it with:
|
|
427
|
+
|
|
428
|
+
```bash
|
|
429
|
+
omp-conductor status
|
|
430
|
+
omp-conductor tail <issue-number>
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
Merge the green pull request. Then make sure that status moves the run to `merged` and frees its worker slot.
|
|
434
|
+
|
|
435
|
+
The loop ticks every 5 minutes while workers remain active. Merge settlement and capacity checks do not wait for the longest worker.
|
|
436
|
+
|
|
437
|
+
`omp-conductor stop` drains active workers before it stops. On systemd, it uses `systemctl stop` to prevent an automatic restart.
|
|
438
|
+
|
|
439
|
+
The package also ships a generic unit at
|
|
385
440
|
[`systemd/omp-conductor.service.example`](systemd/omp-conductor.service.example).
|
|
386
441
|
|
|
387
442
|
|
|
@@ -400,6 +455,25 @@ Four control planes used to answer "stop" differently. The package verbs:
|
|
|
400
455
|
|
|
401
456
|
`status` prints a layered header (`dispatch` / `ticks` / next tick time / `pane` / `recovery` / `herdr` / `telegram` / `daemon`) so a paused fleet cannot hide an armed orchestrator still spending turns. The Telegram line calls the official `getMe` endpoint to prove the token and API are usable without sending a message, then separately reports whether the inbound bridge is configured.
|
|
402
457
|
|
|
458
|
+
`omp-conductor board [--project NAME]` opens the same facts as a live terminal
|
|
459
|
+
kanban instead of a scrolling wall of status text. Its columns are Queue,
|
|
460
|
+
Claimed, Running, Green, Blocked, Failed, and Merged. Queue cards are the
|
|
461
|
+
bounded per-issue hold sample recorded by the latest dispatch; the Queue count
|
|
462
|
+
is the authoritative ready count even when there are more ready issues than
|
|
463
|
+
sampled cards. Run columns show the newest attempt for each issue. Merged keeps
|
|
464
|
+
only the last 24 hours so the board stays operational rather than becoming an
|
|
465
|
+
analytics archive.
|
|
466
|
+
|
|
467
|
+
The board refreshes run, spend, turn, and dispatch values from SQLite every
|
|
468
|
+
second. It refreshes the slower daemon, Herdr, Telegram, and code-graph health
|
|
469
|
+
layer every ten seconds or immediately with `r`. Use arrow keys or `h/j/k/l` to
|
|
470
|
+
select a card, `Enter` to inspect and follow its worker transcript, `u` to run
|
|
471
|
+
the normal unblock workflow, `i` / `p` to open the issue / pull request, `?` for
|
|
472
|
+
help, and `q`, `Esc`, or `Ctrl-C` to go back and quit. Narrow terminals show a
|
|
473
|
+
sliding subset of columns around the selection; terminals below 50×20 get a
|
|
474
|
+
single resize instruction instead of a broken layout. The board is read-only
|
|
475
|
+
except for the explicit `u` action: it never claims work or changes stages.
|
|
476
|
+
|
|
403
477
|
`halt --pane` is **fail-closed**: it exits `0` only when the conductor agent is
|
|
404
478
|
*proven* gone. It writes the recovery pin first, so a failed stop still cannot be
|
|
405
479
|
undone by `herdr-conductor` respawning the agent, and then refuses (nonzero exit,
|
|
@@ -435,16 +509,18 @@ Clear the pin with `omp-conductor release-pane` when you want recovery again.
|
|
|
435
509
|
|
|
436
510
|
Per tick, for the daemon's project:
|
|
437
511
|
|
|
438
|
-
1. **
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
512
|
+
1. **Verify and settle pushed PRs.** For every run in `pushed-pending`, repeat
|
|
513
|
+
the independent head/check verification; green → `pushed-green`, red →
|
|
514
|
+
`failed`, and still pending stays occupied. For every verified
|
|
515
|
+
`pushed-green` run, ask what became of its PR. Merged → `merged`; closed
|
|
516
|
+
without merging → `failed`. Unknown answers leave the row unchanged. This
|
|
517
|
+
maintenance runs even while dispatch is paused or workers are active, so
|
|
518
|
+
status converges on the five-minute tick cadence. It also runs above admission
|
|
519
|
+
so a row settled here frees its issue in the same tick. See
|
|
520
|
+
[what settles a green PR](#what-settles-a-green-pr).
|
|
521
|
+
2. **Paused?** If the pause sentinel exists, the tick claims nothing and returns.
|
|
522
|
+
Settlement has already run, but no queue or admission work occurs. This makes
|
|
523
|
+
`omp-conductor pause` take effect without signalling the process.
|
|
448
524
|
3. **List the queue.** Open issues in `tracker.repo` labelled `queueLabel`.
|
|
449
525
|
4. **Filter and route.** An issue is eligible only if it carries the queue label
|
|
450
526
|
and none of the three state labels (`inProgress`, `blocked`, `failed`). Eligible
|
|
@@ -454,26 +530,31 @@ Per tick, for the daemon's project:
|
|
|
454
530
|
6. **Check spend.** If spend since local midnight has reached `dailySpendUsd`, the
|
|
455
531
|
daemon **pauses itself**, pages at Tier 2, and returns.
|
|
456
532
|
7. **Check capacity.** `maxConcurrentWorkers` minus *live* workers (runs in
|
|
457
|
-
`claimed` or `running`) gives the free slots. A green PR
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
and returns.
|
|
533
|
+
`claimed` or `running`) gives the free slots. A pending or green PR occupies
|
|
534
|
+
its issue but not a slot: its worker is finished, and counting pushed PRs
|
|
535
|
+
would let two completed workers stop the fleet.
|
|
536
|
+
If no slot is free, the tick logs and returns.
|
|
461
537
|
8. **Admit issues** up to the free slots, skipping any issue that already has
|
|
462
|
-
an active run — including a green PR, so a second attempt
|
|
463
|
-
live PR.
|
|
464
|
-
|
|
538
|
+
an active run — including a pending or green PR, so a second attempt cannot
|
|
539
|
+
land on a live PR. Repeated implementation failures consume
|
|
540
|
+
`maxAttemptsPerIssue`; cap kills, daemon orphans and answered blocks consume
|
|
541
|
+
the independent `maxContinuationsPerIssue`. Exhausting either escalates.
|
|
465
542
|
9. **Ask the tracker whether the work already exists.** For each candidate that
|
|
466
543
|
survived step 8 — so at most one API call per free slot, never one per queued
|
|
467
|
-
issue — the daemon asks whether an **open** PR already closes
|
|
468
|
-
|
|
469
|
-
PR
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
544
|
+
issue — the daemon asks whether an **open** PR already closes it. An open PR
|
|
545
|
+
normally holds the issue. One narrow exception permits a routed continuation:
|
|
546
|
+
the PR URL must match the latest terminal run's retained PR exactly. Drafts
|
|
547
|
+
count because their branch can hold the only copy of the work.
|
|
548
|
+
The tracker also finds work missing from a new, moved, restored, or cleared
|
|
549
|
+
store. If the check fails, the candidate is **held**, not admitted, and
|
|
473
550
|
retried next tick: the cost of holding is five minutes, the cost of admitting
|
|
474
551
|
on an unknown is a burned attempt and a duplicate PR. Only that candidate is
|
|
475
552
|
held, so a flaky API cannot stall the rest of the queue.
|
|
476
|
-
10. **
|
|
553
|
+
10. **Record the pass.** Persist ready/routed/admitted counts and group every hold
|
|
554
|
+
under a stable reason code with at most five sample issue numbers. Tracker
|
|
555
|
+
failures mark the summary `DEGRADED`; capacity, sibling, open-PR and budget
|
|
556
|
+
holds remain normal policy state.
|
|
557
|
+
11. **Dispatch** the admitted issues concurrently.
|
|
477
558
|
|
|
478
559
|
Then, per admitted issue:
|
|
479
560
|
|
|
@@ -496,13 +577,15 @@ Then, per admitted issue:
|
|
|
496
577
|
|
|
497
578
|
| Outcome | Labels | Worktree | Escalation |
|
|
498
579
|
| --- | --- | --- | --- |
|
|
580
|
+
| `pushed-pending` | `agent:in-progress` stays while the daemon rechecks GitHub | removed | none |
|
|
499
581
|
| `pushed-green` | `agent:in-progress` stays until the merge closes the issue | removed | none |
|
|
500
582
|
| `blocked` | swapped to `agent:blocked` | removed | Tier 1 |
|
|
501
|
-
| `failed` / `killed` | swapped to `agent:failed` | dirty tree committed to the branch, then
|
|
583
|
+
| `failed` / `killed` | swapped to `agent:failed` | dirty tree committed to the branch, then retained until the PR or issue is terminal | Tier 1 |
|
|
502
584
|
| unexpected error | swapped to `agent:failed` | same | Tier 1 |
|
|
503
585
|
|
|
504
|
-
`pushed-green`
|
|
505
|
-
|
|
586
|
+
`pushed-pending` and `pushed-green` are not the end of the row: later ticks
|
|
587
|
+
verify outstanding checks and settle the PR once it resolves. See
|
|
588
|
+
[what settles a green PR](#what-settles-a-green-pr).
|
|
506
589
|
|
|
507
590
|
Label swaps add the new label before removing the old one: the reverse order
|
|
508
591
|
leaves a window in which the issue carries no state label at all, which is
|
|
@@ -521,6 +604,13 @@ Then, per admitted issue:
|
|
|
521
604
|
run is deliberately *not* salvaged — it stopped on purpose, with turns still
|
|
522
605
|
in hand to commit for itself.
|
|
523
606
|
|
|
607
|
+
Later ticks reap retained failure trees in bounded batches after the tracker
|
|
608
|
+
proves their PR merged/closed or their issue closed, provided no live run or
|
|
609
|
+
queued continuation owns the issue. Cleanup fetches remote refs first and
|
|
610
|
+
keeps any dirty tree or branch with uniquely local commits. Only then does it
|
|
611
|
+
remove the physical tree, prune registrations, and delete the obsolete local
|
|
612
|
+
mirror branch. Unknown tracker, network, repo, or git state is a no-op.
|
|
613
|
+
|
|
524
614
|
### What a restart does to runs that were in flight
|
|
525
615
|
|
|
526
616
|
A `claimed` or `running` row is a promise that a worker process exists, and a
|
|
@@ -538,8 +628,9 @@ label is the crash guard against double-dispatch — and deciding what the dead
|
|
|
538
628
|
worker's remains are worth is the orchestrator's drain-duty judgement, spelled
|
|
539
629
|
out in its brief: an open green PR goes to the merge path, a salvaged sha is a
|
|
540
630
|
continuation hand-off, and a clean orphan has its label released so the next
|
|
541
|
-
tick re-claims it.
|
|
542
|
-
so
|
|
631
|
+
tick re-claims it. Orphans consume `maxContinuationsPerIssue`, not failed
|
|
632
|
+
implementation attempts, so crashes cannot starve the retry needed for a real
|
|
633
|
+
code or CI failure — and a crash loop still escalates.
|
|
543
634
|
|
|
544
635
|
### Deploying a new package onto a busy fleet
|
|
545
636
|
|
|
@@ -548,34 +639,36 @@ this version is installed: startup salvage commits dirty trees before orphaning
|
|
|
548
639
|
rows, and salvage rewrites the mirror's managed `info/exclude` to the package's
|
|
549
640
|
current list before `git add` so a narrowed ignore cannot hide deliverables.
|
|
550
641
|
|
|
551
|
-
It is still disruptive for **in-flight sessions**
|
|
552
|
-
attempt is spent
|
|
642
|
+
It is still disruptive for **in-flight sessions** because the worker process dies
|
|
643
|
+
and the attempt is spent. Update through the lifecycle command rather than
|
|
644
|
+
hand-installing or restarting individual surfaces:
|
|
553
645
|
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
4. `systemctl restart omp-conductor` (or `omp-conductor restart`).
|
|
558
|
-
5. `omp-conductor resume` if you left it paused.
|
|
559
|
-
|
|
560
|
-
If you cannot wait:
|
|
646
|
+
```bash
|
|
647
|
+
omp-conductor upgrade
|
|
648
|
+
```
|
|
561
649
|
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
/ merge / release label). Status prints a `deploy` line while live workers > 0
|
|
567
|
-
so you can see the risk before you restart.
|
|
650
|
+
It pauses new claims, drains workers, installs one pinned release across all
|
|
651
|
+
surfaces, restarts, verifies twice, and resumes only if dispatch was initially
|
|
652
|
+
running. If an install, restart, or verification step fails, dispatch remains
|
|
653
|
+
paused and the command exits nonzero.
|
|
568
654
|
|
|
569
655
|
Do **not** edit files under the running install and expect the daemon to keep
|
|
570
|
-
dispatching — the integrity tripwire pauses and pages.
|
|
571
|
-
the new process records a fresh baseline.
|
|
656
|
+
dispatching — the integrity tripwire pauses and pages. Upgrade by whole release
|
|
657
|
+
so the new process records a fresh baseline.
|
|
572
658
|
|
|
573
659
|
### What settles a green PR
|
|
574
660
|
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
661
|
+
The worker watches CI, then reports the PR URL and the exact remote head SHA it
|
|
662
|
+
observed. The daemon independently reads the PR again and requires it to be open,
|
|
663
|
+
non-draft, still at that head, and backed by a non-empty check rollup in which
|
|
664
|
+
every check succeeded or was skipped. Missing or nonterminal checks become
|
|
665
|
+
`pushed-pending` and are rechecked on later ticks; red or cancelled checks become
|
|
666
|
+
`failed` with a bounded job/log digest. Only verified evidence becomes
|
|
667
|
+
`pushed-green`.
|
|
668
|
+
|
|
669
|
+
What happens after verification is a human's decision, taken minutes to days
|
|
670
|
+
later and never announced to the daemon — so every tick asks the tracker about
|
|
671
|
+
every pushed PR it is still holding:
|
|
579
672
|
|
|
580
673
|
| PR | Row becomes | Why |
|
|
581
674
|
| --- | --- | --- |
|
|
@@ -584,12 +677,12 @@ is still holding, and settles the ones that resolved:
|
|
|
584
677
|
| still open | unchanged | The normal steady state. Its issue must stay occupied, or a second attempt lands on the live PR. |
|
|
585
678
|
| could not be determined | unchanged | A flaky network, a revoked token, a deleted PR. An unknown answer never settles a row; the next tick asks again for free. |
|
|
586
679
|
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
human who closed a PR is already looking at it, so what
|
|
590
|
-
say next
|
|
591
|
-
|
|
592
|
-
|
|
680
|
+
Run history is untouched. A PR closed without merging becomes a concrete failed
|
|
681
|
+
attempt; a merge does not spend failure or continuation budget. A merge normally
|
|
682
|
+
closes the issue, and a human who closed a PR is already looking at it, so what
|
|
683
|
+
an issue's labels should say next remains the orchestrator's drain-duty
|
|
684
|
+
judgement. One unreachable PR costs its own row and nothing else; the rest of
|
|
685
|
+
the sweep still settles.
|
|
593
686
|
|
|
594
687
|
Until this existed, nothing ever revisited a `pushed-green` row: the startup
|
|
595
688
|
reconciler only settles rows that held a process, and `merged` went unwritten. On
|
|
@@ -687,9 +780,10 @@ rest. `0` is a real value (a hard stop), not "unset".
|
|
|
687
780
|
| --- | --- | --- |
|
|
688
781
|
| `maxConcurrentWorkers` | `2` (setup may write `1` on <16 GiB hosts) | Parallel in-process omp sessions inside the daemon PID. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. On hosts under ~16 GiB RAM, prefer `1` so the unit stays out of swap ([host sizing](#host-sizing-and-memory)). Raise it only if you actually have the runners *and* the RAM. |
|
|
689
782
|
| `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total`. |
|
|
690
|
-
| `workerMaxTurns` | `120` |
|
|
783
|
+
| `workerMaxTurns` | `120` | Startup ceiling for each new worker. Catches a session looping without converging; use `omp-conductor extend` to raise one live run without changing this default. |
|
|
691
784
|
| `workerWallClockMs` | `5400000` (90 minutes) | Wall-clock ceiling for one worker. A session that is merely stuck spends no turns, so turns alone cannot detect it. |
|
|
692
|
-
| `maxAttemptsPerIssue` | `2` |
|
|
785
|
+
| `maxAttemptsPerIssue` | `2` | Failed implementation or CI attempts before escalation. Operational stops do not consume this budget, so salvage can continue without stealing the retry needed for a real failure. |
|
|
786
|
+
| `maxContinuationsPerIssue` | `2` | Cap-kill, daemon-orphan and answered-block resumes before escalation. This independently bounds crash/resume loops. |
|
|
693
787
|
|
|
694
788
|
Days are counted from **local midnight**, matching how a human reads "today".
|
|
695
789
|
|
|
@@ -699,8 +793,15 @@ pages at Tier 2**: a loop that is burning money has to halt itself, because
|
|
|
699
793
|
waiting for someone to notice tomorrow is how a runaway becomes expensive.
|
|
700
794
|
Work resumes only after `omp-conductor resume` (or `/conductor resume`).
|
|
701
795
|
|
|
702
|
-
`workerMaxTurns` and `workerWallClockMs` are enforced inside the session driver
|
|
703
|
-
|
|
796
|
+
`workerMaxTurns` and `workerWallClockMs` are enforced inside the session driver.
|
|
797
|
+
The daemon reads a live run's effective turn ceiling at every turn boundary. Use
|
|
798
|
+
`omp-conductor extend <issue> --turns N [--project NAME]` to raise it without
|
|
799
|
+
restarting or reconstructing the session. Extension is monotonic: equal or lower
|
|
800
|
+
values are refused, as are runs whose live controller has already settled. The
|
|
801
|
+
effective value is persisted and shown beside that active run in `status`; editing
|
|
802
|
+
`config.json` changes defaults for future daemon starts, not workers already in
|
|
803
|
+
flight. A cap that fires aborts the run, records it as `killed`, and names the
|
|
804
|
+
ceiling in the escalation.
|
|
704
805
|
|
|
705
806
|
## Worker model
|
|
706
807
|
|
|
@@ -736,10 +837,10 @@ defined" in one call instead of twenty greps.
|
|
|
736
837
|
|
|
737
838
|
### Two things this package does not do for you
|
|
738
839
|
|
|
739
|
-
`omp-conductor` never installs,
|
|
740
|
-
`graphProject` unset, nothing about dispatch, caps
|
|
741
|
-
|
|
742
|
-
`graph-setup` reports them as step 0:
|
|
840
|
+
`omp-conductor` never installs, starts, imports, or depends on the indexer for
|
|
841
|
+
dispatch. With `graphProject` unset, nothing about dispatch, caps, escalation, or
|
|
842
|
+
status changes. A fresh host needs both of these before an index is worth
|
|
843
|
+
anything, and `graph-setup` reports them as step 0:
|
|
743
844
|
|
|
744
845
|
1. **`codebase-memory-mcp` on PATH** — a separate project,
|
|
745
846
|
[DeusData/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp).
|
|
@@ -750,11 +851,11 @@ means a fresh host needs both of these before an index is worth anything, and
|
|
|
750
851
|
|
|
751
852
|
Say yes and the wizard asks for one root, then derives one clone per routed repo
|
|
752
853
|
underneath it (default `~/.cache/conductor-graph/<org>/<repo>`) and writes it to
|
|
753
|
-
each repo's [`graphProject`](#configuration).
|
|
754
|
-
|
|
755
|
-
|
|
756
|
-
|
|
757
|
-
graph` area add it in two prompts — see
|
|
854
|
+
each repo's [`graphProject`](#configuration). The only automatic interaction is
|
|
855
|
+
a bounded, read-only health query; this package never clones, fetches, builds an
|
|
856
|
+
index, or changes systemd. Dispatch, caps and escalation do not depend on graph
|
|
857
|
+
health. On a fleet configured before this key existed, `/conductor setup` and
|
|
858
|
+
the `code graph` area add it in two prompts — see
|
|
758
859
|
[Changing one setting](#changing-one-setting).
|
|
759
860
|
|
|
760
861
|
### Why the clone, and not your checkout or the worktree
|
|
@@ -813,6 +914,18 @@ neither usefully: the indexer resolves its store from `HOME`, systemd's default
|
|
|
813
914
|
user that ran `graph-setup`; the unit sets no `User=`, so check them if that is
|
|
814
915
|
not the account the timer runs as.
|
|
815
916
|
|
|
917
|
+
### Seeing whether the graph is usable
|
|
918
|
+
|
|
919
|
+
When at least one routed repo has `graphProject`, `omp-conductor status` adds a
|
|
920
|
+
`code graph` block. It proves the indexer is on `PATH`, the worker MCP config
|
|
921
|
+
mounts it, every configured clone exists and exactly matches an indexed
|
|
922
|
+
`root_path`, the refresh timer is enabled and active, and the last service run
|
|
923
|
+
succeeded within 45 minutes. A running daemon refreshes this evidence every
|
|
924
|
+
minute and publishes the cached result through `/healthz`; status probes the host
|
|
925
|
+
directly when that cache is unavailable. Every command is read-only, runs with a
|
|
926
|
+
one-second timeout, and graph degradation never changes `/healthz.ok` or blocks
|
|
927
|
+
dispatch. Unconfigured projects omit the block entirely.
|
|
928
|
+
|
|
816
929
|
## Escalation tiers
|
|
817
930
|
|
|
818
931
|
| Tier | Meaning | Raised by | Delivered to |
|
|
@@ -873,7 +986,10 @@ The config lives at `$OMP_CONDUCTOR_HOME/config.json`, or
|
|
|
873
986
|
`~/.omp/conductor/config.json` when that variable is unset. It is written with mode
|
|
874
987
|
`0600` in a directory created `0700`, because it carries chat ids and clone URLs.
|
|
875
988
|
That same directory holds the SQLite store (`conductor.db`), the `paused` sentinel,
|
|
876
|
-
the `sessions/` worker transcripts
|
|
989
|
+
the `sessions/` worker transcripts, the `orchestrator/` session directory,
|
|
990
|
+
`backups/briefs/` for timestamped brief and policy safety copies, and
|
|
991
|
+
`release-policy-blocks.jsonl`, the append-only audit of mechanically rejected
|
|
992
|
+
release/deploy calls.
|
|
877
993
|
|
|
878
994
|
Runtime state lives elsewhere, under `$OMP_CONDUCTOR_RUNTIME_DIR` (default
|
|
879
995
|
`~/.omp/run/daemons/omp-conductor`): `daemon.json`, a mode-`0600` pidfile written
|
|
@@ -909,7 +1025,8 @@ A complete, valid config for one project with two target repos:
|
|
|
909
1025
|
"dailySpendUsd": 25,
|
|
910
1026
|
"workerMaxTurns": 120,
|
|
911
1027
|
"workerWallClockMs": 5400000,
|
|
912
|
-
"maxAttemptsPerIssue": 2
|
|
1028
|
+
"maxAttemptsPerIssue": 2,
|
|
1029
|
+
"maxContinuationsPerIssue": 2
|
|
913
1030
|
},
|
|
914
1031
|
"projects": [
|
|
915
1032
|
{
|
|
@@ -959,6 +1076,7 @@ A complete, valid config for one project with two target repos:
|
|
|
959
1076
|
"merge": "human",
|
|
960
1077
|
"release": "human"
|
|
961
1078
|
},
|
|
1079
|
+
"releasePolicy": "none",
|
|
962
1080
|
"reporting": {
|
|
963
1081
|
"scope": "material"
|
|
964
1082
|
},
|
|
@@ -986,6 +1104,7 @@ Field notes:
|
|
|
986
1104
|
| `escalation.fallbackToIssueComment` | Defaults to `true`. Absent means "yes, still tell me". |
|
|
987
1105
|
| `escalation.orchestrator` | Optional; `"embedded"` (default) or `"external"`. `external` means an orchestrator session already runs elsewhere: the daemon starts none, and tier-1 escalations post as issue comments for that session to drain. Any other value is an error. |
|
|
988
1106
|
| `authority` | Optional; `{ "merge": …, "release": … }`, each `"human"` (default) or `"orchestrator"`. It grants nothing to the daemon — it words the orchestrator's standing orders and the Releases paragraph of the rendered brief, so the config and the prompt cannot disagree about who holds the merge button. Unknown keys and any other value are errors, never folded to the default. |
|
|
1107
|
+
| `releasePolicy` | Optional; `"none"` (default) or `"operator-brief"`. `none` installs a pre-tool-call tripwire in worker, embedded-orchestrator and external-orchestrator sessions. It blocks `git tag`, tag pushes, package publishing, GitHub release creation and recognised deploy commands before execution. `operator-brief` opens that gate only for the procedure in the operator-owned brief. Unknown values are errors. Every rejection is written to `release-policy-blocks.jsonl`; the heartbeat carries that day's count into the daily digest so configured intent and observed behaviour cannot drift silently. This is the mechanical gate; `authority.release` still says who owns the decision. |
|
|
989
1108
|
| `reporting.scope` | Optional; `"material"` (default) or `"escalations"`. Every orchestrator tick appends the matching constraint line to its prompt, re-read from this file each tick — see [Your workflow vs. the package](#your-workflow-vs-the-package). It constrains what the session is told to report; it is not an outbound filter. A config written without the key keeps reporting material events. Any other value is an error, never folded to the default. |
|
|
990
1109
|
| `workspaceRoot` / `mirrorRoot` | Optional; default to `worktrees/` and `mirrors/` under the state directory. `~` is expanded. |
|
|
991
1110
|
|
|
@@ -1000,9 +1119,9 @@ running its loop. A 24/7 omp session with a standing brief and nobody typing int
|
|
|
1000
1119
|
it never gets prompted, so it never runs anything. Installing
|
|
1001
1120
|
`omp plugin install omp-conductor` also installs a heartbeat that prompts it.
|
|
1002
1121
|
|
|
1003
|
-
The heartbeat is **inert unless the session
|
|
1004
|
-
`.conductor-tick.json`**, so
|
|
1005
|
-
|
|
1122
|
+
The heartbeat is **inert unless the session cwd contains
|
|
1123
|
+
`.conductor-tick.json`**, so an ordinary session has no timer. `/conductor setup`
|
|
1124
|
+
writes this file for external orchestration. A manual configuration has this form:
|
|
1006
1125
|
|
|
1007
1126
|
```json
|
|
1008
1127
|
{
|
|
@@ -1199,14 +1318,18 @@ omp-conductor start [--port N] [--project NAME]
|
|
|
1199
1318
|
omp-conductor --version
|
|
1200
1319
|
omp-conductor stop
|
|
1201
1320
|
omp-conductor restart [--port N] [--project NAME]
|
|
1321
|
+
omp-conductor upgrade [--to VERSION] [--project NAME]
|
|
1202
1322
|
omp-conductor status [--project NAME]
|
|
1323
|
+
omp-conductor board [--project NAME]
|
|
1203
1324
|
omp-conductor hold [--project NAME]
|
|
1204
1325
|
omp-conductor halt [--pane] [--project NAME]
|
|
1205
1326
|
omp-conductor arm [--project NAME]
|
|
1206
1327
|
omp-conductor disarm [--project NAME]
|
|
1207
1328
|
omp-conductor release-pane [--project NAME]
|
|
1208
1329
|
omp-conductor tail <issue> [--project NAME]
|
|
1330
|
+
omp-conductor extend <issue> --turns N [--project NAME]
|
|
1209
1331
|
omp-conductor unblock <issue> [--project NAME]
|
|
1332
|
+
omp-conductor friction <escalation-digest|report-noise|report-surprise> --detail TEXT [--issue N] [--project NAME]
|
|
1210
1333
|
omp-conductor daemon [--once] [--port N] [--project NAME]
|
|
1211
1334
|
omp-conductor pause
|
|
1212
1335
|
omp-conductor resume
|
|
@@ -1220,16 +1343,20 @@ omp-conductor help
|
|
|
1220
1343
|
| `start` | Start `herdr-fleet.service` when that optional unit is installed, clearing a previous pane-recovery pin, then spawn the dispatch loop in the background and wait until it answers `GET /healthz` on `:8787`. Without systemd or that unit it keeps the standalone daemon behaviour. It never clears pause or arms ticks. Refuses if a daemon is already live, naming its pid; if the process dies or never serves, it cleans up and quotes the tail of `daemon.log`. |
|
|
1221
1344
|
| `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
|
|
1222
1345
|
| `restart` | Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
|
|
1223
|
-
| `
|
|
1346
|
+
| `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
|
|
1347
|
+
| `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / optional `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, and bounded hold groups; API failures are marked `DEGRADED` so queue starvation cannot look idle. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
|
|
1348
|
+
| `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`: Queue, Claimed, Running, Green, Blocked, Failed, and the last 24 hours of Merged. Refreshes run/spend/turn values every second and slower health every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
|
|
1224
1349
|
| `hold [--project NAME]` | Soft stop: pause claiming **and** disarm ticks. Daemon and pane stay up. Prefer this over `pause` when the intent is "stop the conductor" without killing processes. See [Stop the conductor](#stop-the-conductor-hold--halt). |
|
|
1225
1350
|
| `halt [--pane] [--project NAME]` | `hold`, then stop the dispatch daemon (systemctl-aware). Pane stays up unless `--pane` is passed. `halt --pane` also pins herdr-conductor recovery off for the conductor agent only — it does **not** stop `herdr-fleet.service` or any other herdr session. Fail-closed: exits nonzero unless the agent is proven gone. |
|
|
1226
1351
|
| `arm [--project NAME]` | Proof-gated: send a Telegram challenge and write the arm marker only after your reply appears as a user turn in the orchestrator transcript. Never auto-armed by `resume` / `hold`. |
|
|
1227
1352
|
| `disarm [--project NAME]` | Remove the arm marker so ticks skip. Processes untouched. |
|
|
1228
1353
|
| `release-pane [--project NAME]` | Clear the `halt --pane` recovery pin so herdr-conductor may resume the fleet agent again. |
|
|
1229
1354
|
| `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
|
|
1230
|
-
| `
|
|
1231
|
-
| `
|
|
1232
|
-
| `
|
|
1355
|
+
| `extend <issue> --turns N [--project NAME]` | Monotonically raise that live worker's effective turn ceiling through its owning daemon. The current omp session keeps running; no restart or continuation is created. The daemon persists the new ceiling for `status` and rejects missing, settled, cap-killed, equal, or lower requests instead of implying that an immutable session changed. |
|
|
1356
|
+
| `unblock <issue>` | Remove that issue's `blocked` and `failed` labels so an answered escalation can be claimed again. `agent:in-progress` is never touched. Run history remains intact: blocks consume the independent continuation budget, not failed implementation attempts. The output reports both budgets and warns when either will make the next tick escalate instead of dispatch. Exits `2` when the issue number is missing or malformed. |
|
|
1357
|
+
| `friction <kind> --detail TEXT [--issue N]` | Record one bounded judgment the daemon cannot infer: an escalation belonged in a digest, or a tick report was noise/surprising. The detail is limited to 160 characters. One event never changes policy; three observations inside seven days make the aggregate eligible for one Learning-loop prompt, followed by a seven-day cooldown. |
|
|
1358
|
+
| `daemon` | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. Admitted workers run in a tracked background pool, so settlement and capacity checks remain periodic while they work; shutdown drains the pool before closing the store. This is what `start` launches and what a systemd unit should call. |
|
|
1359
|
+
| `daemon --once` | Run a single tick, wait for workers admitted by that tick, and exit. No HTTP server or pidfile — a drill must not register itself as the daemon, or the next reader believes it and the real daemon's in-flight runs get reconciled as orphans. |
|
|
1233
1360
|
| `--port N` | Accepted by `start`, `restart` and `daemon`. Both `--port 9000` and `--port=9000` work; missing or out of range exits `2` rather than falling back to the default, because probing the wrong endpoint is worse than a hard failure. |
|
|
1234
1361
|
| `--project NAME` | Pick the project to service. One daemon process serves exactly one project; with several configured projects the name is required. |
|
|
1235
1362
|
| `pause` | Stop claiming new work only. The running daemon notices on its next tick; runs already in flight finish. The orchestrator heartbeat keeps ticking if armed — its gate is the arm marker, not this flag. Prefer `hold` to silence both. |
|
|
@@ -1261,17 +1388,52 @@ curl -s localhost:8787/healthz
|
|
|
1261
1388
|
```
|
|
1262
1389
|
|
|
1263
1390
|
```json
|
|
1264
|
-
{
|
|
1391
|
+
{
|
|
1392
|
+
"ok": true,
|
|
1393
|
+
"paused": false,
|
|
1394
|
+
"activeRuns": 1,
|
|
1395
|
+
"project": "demo",
|
|
1396
|
+
"rssBytes": 123456789,
|
|
1397
|
+
"dispatch": {
|
|
1398
|
+
"completedAt": 1786185678000,
|
|
1399
|
+
"ready": 8,
|
|
1400
|
+
"routed": 8,
|
|
1401
|
+
"admitted": 0,
|
|
1402
|
+
"degraded": true,
|
|
1403
|
+
"holds": [
|
|
1404
|
+
{ "reason": "parent-lookup-error", "count": 8, "issues": [321, 320, 318] }
|
|
1405
|
+
]
|
|
1406
|
+
},
|
|
1407
|
+
"codeGraph": {
|
|
1408
|
+
"configured": true,
|
|
1409
|
+
"status": "degraded",
|
|
1410
|
+
"checkedAt": "2026-08-08T13:00:00.000Z",
|
|
1411
|
+
"prerequisites": { "indexer": "present", "mcpMount": "missing" },
|
|
1412
|
+
"repos": [
|
|
1413
|
+
{
|
|
1414
|
+
"name": "api",
|
|
1415
|
+
"path": "/home/fleet/.cache/conductor-graph/acme/api",
|
|
1416
|
+
"clone": "present",
|
|
1417
|
+
"index": "present"
|
|
1418
|
+
}
|
|
1419
|
+
],
|
|
1420
|
+
"timer": { "enabled": "enabled", "active": "active" },
|
|
1421
|
+
"refresh": {
|
|
1422
|
+
"result": "success",
|
|
1423
|
+
"fresh": true,
|
|
1424
|
+
"lastSuccessAt": "2026-08-08T12:50:00.000Z",
|
|
1425
|
+
"ageMs": 600000
|
|
1426
|
+
},
|
|
1427
|
+
"reasons": ["worker MCP configuration does not mount the indexer"]
|
|
1428
|
+
}
|
|
1429
|
+
}
|
|
1265
1430
|
```
|
|
1266
1431
|
|
|
1267
|
-
Any other path or method returns `404`.
|
|
1268
|
-
|
|
1269
|
-
`
|
|
1270
|
-
|
|
1271
|
-
`activeRuns` counts occupied issues — live workers plus green PRs
|
|
1272
|
-
merge — and each of those is settled by the tick once its PR resolves, so the
|
|
1273
|
-
number goes back down on its own. A count that only ever grows is the bug this
|
|
1274
|
-
used to have: read [what settles a green PR](#what-settles-a-green-pr).
|
|
1432
|
+
Any other path or method returns `404`. `ok` reports process liveness only.
|
|
1433
|
+
Nonfatal admission errors and graph degradation keep it `true` so a supervisor
|
|
1434
|
+
does not restart-loop. Inspect `dispatch.degraded` and its bounded reason groups
|
|
1435
|
+
for queue starvation; inspect `codeGraph` for configured graph health.
|
|
1436
|
+
`activeRuns` counts occupied issues — live workers plus green PRs awaiting merge.
|
|
1275
1437
|
|
|
1276
1438
|
## What a worker may and may not do
|
|
1277
1439
|
|
|
@@ -1287,11 +1449,11 @@ dispatcher. The brief is explicit about the boundary:
|
|
|
1287
1449
|
| Run the repo's configured cheap gates, each from its listed `cwd`, over the whole tree. | Run docker or image builds, production builds, browser/e2e suites, or the full test suite on the shared host — CI owns the heavy gates. |
|
|
1288
1450
|
| Review its whole diff, then commit and **push once**. One corrective push if CI is red. | Force-push, `git add -f`, or add AI/co-author attribution. Red twice means stop and report, not push a third time. |
|
|
1289
1451
|
| Open a PR that links the issue, and watch CI to a verdict with `gh pr checks --watch`. | Run `gh pr merge`. **A worker never merges** — that one is absolute, whoever else holds the authority — so PRs land one at a time with a freshness re-check; two workers merging concurrently is how agent PRs clobber each other. Who *may* merge is the [`authority`](#configuration) answer, and it is never the worker. |
|
|
1290
|
-
| Escalate: ambiguity, a cross-repo contract, a needed credential, a product or data-migration decision, a blocking existing test, CI red twice, or most of the wall-clock budget burned. | **
|
|
1452
|
+
| Escalate: ambiguity, a cross-repo contract, a needed credential, a product or data-migration decision, a blocking existing test, CI red twice, or most of the wall-clock budget burned. | **Under the default `releasePolicy: "none"`:** cut a release, push a tag, publish to npm, edit a deployment pin, deploy, or touch infrastructure or secrets. The harness blocks recognised release/deploy tool calls before they run and audits the attempt. A project may deliberately set `operator-brief` only after its operator-owned brief contains the exact release procedure; that opens the tool gate but does not change `authority.release`. |
|
|
1291
1453
|
|
|
1292
|
-
The worker ends with a
|
|
1293
|
-
next). `pushed-green`
|
|
1294
|
-
|
|
1454
|
+
The worker ends with a seven-line evidence report (issue, PR, observed head SHA,
|
|
1455
|
+
state, gates, changed, next). A textual `pushed-green` claim is not success: the
|
|
1456
|
+
daemon repeats the PR/head/check verification before it records that state.
|
|
1295
1457
|
|
|
1296
1458
|
### Worker confinement and the integrity tripwire
|
|
1297
1459
|
|
|
@@ -1303,10 +1465,11 @@ which installs an inline harness extension that blocks `write` / `edit` /
|
|
|
1303
1465
|
`routing.repos` is ever checked out — and the caps still bound *how much* work
|
|
1304
1466
|
happens.
|
|
1305
1467
|
|
|
1306
|
-
|
|
1307
|
-
|
|
1308
|
-
|
|
1309
|
-
|
|
1468
|
+
General shell access is not confined to the worktree. Its argument is an opaque
|
|
1469
|
+
program, so the brief still forbids path escape and the deploy-level answer is a
|
|
1470
|
+
least-privilege worker uid (below). The narrower release-policy tripwire does
|
|
1471
|
+
inspect explicit command shapes such as `git tag`, `npm publish`, and deploy
|
|
1472
|
+
verbs; it blocks those before execution when `releasePolicy` is `none`.
|
|
1310
1473
|
|
|
1311
1474
|
#### Integrity tripwire (package self-hash)
|
|
1312
1475
|
|
|
@@ -1378,25 +1541,27 @@ Known and deliberate in this version:
|
|
|
1378
1541
|
- **No cross-process lock on the mirrors.** Two dispatch loops fetching the same
|
|
1379
1542
|
repo at the same instant can collide on git's ref locks; the run fails and is
|
|
1380
1543
|
retried rather than corrupted.
|
|
1381
|
-
- **
|
|
1382
|
-
|
|
1383
|
-
|
|
1384
|
-
|
|
1385
|
-
|
|
1386
|
-
|
|
1387
|
-
|
|
1388
|
-
|
|
1389
|
-
|
|
1390
|
-
`
|
|
1544
|
+
- **Uniquely local mirror branches are retained.** Terminal runs are reaped
|
|
1545
|
+
automatically only after every commit exists on a remote ref. A failed salvage
|
|
1546
|
+
push deliberately leaves its branch and tree for an operator rather than
|
|
1547
|
+
trading disk hygiene for data loss.
|
|
1548
|
+
- **`stop` is a bounded best-effort drain.** A signal stops new ticks and the
|
|
1549
|
+
daemon waits for its active worker pool before closing the store. The CLI
|
|
1550
|
+
escalates to `SIGKILL` after 10 seconds, so a worker that needs longer is
|
|
1551
|
+
orphaned and salvaged on restart. Use `pause`, wait for `workers 0 / N`, then
|
|
1552
|
+
stop when a clean drain matters. A supervising unit should set
|
|
1553
|
+
`SuccessExitStatus=0 143`, and operators should prefer `omp-conductor stop` /
|
|
1554
|
+
`systemctl stop` over raw `kill`, so `Restart=on-failure` cannot misread a
|
|
1555
|
+
deliberate stop as a crash.
|
|
1391
1556
|
- **A failed orchestrator degrades quietly.** The daemon logs a warning and keeps
|
|
1392
1557
|
running, but tier-1 escalations then land in issue comments — which is exactly the
|
|
1393
1558
|
"nobody reads it until morning" path the orchestrator exists to avoid. The warning
|
|
1394
1559
|
is in `daemon.log`; nothing pages you about it.
|
|
1395
|
-
- **Workers are not terminal panes, so you cannot watch them.**
|
|
1396
|
-
in-process omp session
|
|
1397
|
-
|
|
1398
|
-
|
|
1399
|
-
|
|
1560
|
+
- **Workers are not terminal panes, so you cannot watch them there.** Each
|
|
1561
|
+
worker is an in-process omp session started by `createSession`. The resident
|
|
1562
|
+
daemon tracks workers in a background pool so the five-minute loop keeps
|
|
1563
|
+
settling PRs and checking capacity; shutdown waits for that pool. Herdr still
|
|
1564
|
+
shows exactly one pane (the orchestrator's) regardless of concurrency.
|
|
1400
1565
|
|
|
1401
1566
|
The cap does work. The admission loop (`admitCandidates` in `src/daemon.ts`) computes
|
|
1402
1567
|
`slots = maxConcurrentWorkers - live workers`, admits at most that many issues
|