omp-conductor 0.3.13 → 0.3.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +168 -41
- package/package.json +2 -1
- package/src/brief-upgrade.ts +1 -1
- package/src/briefs/orchestrator.md +3 -2
- package/src/briefs/worker.md +1 -1
- package/src/cli.ts +132 -35
- package/src/confinement.ts +123 -0
- package/src/daemon.ts +55 -2
- package/src/fleet.ts +1100 -0
- package/src/host.ts +90 -0
- package/src/lifecycle.ts +182 -77
- package/src/omp.ts +12 -0
- package/src/orchestrator-tick.ts +1 -1
- package/src/plugin.ts +110 -17
- package/src/tracker/github.ts +25 -1
- package/src/types.ts +10 -0
- package/src/worker.ts +2 -0
- package/systemd/omp-conductor.service.example +54 -0
package/README.md
CHANGED
|
@@ -36,14 +36,15 @@ The package ships three deployables, plus one skill:
|
|
|
36
36
|
|
|
37
37
|
| Deployable | Entry | What it is for |
|
|
38
38
|
| --- | --- | --- |
|
|
39
|
-
| omp plugin | `/conductor` slash command | Inspect and
|
|
39
|
+
| omp plugin | `/conductor` slash command | Inspect and control the fleet from inside an omp session: status, hold/halt, arm/disarm, pause/resume, setup. |
|
|
40
40
|
| Standalone daemon | `omp-conductor` binary | The dispatch loop, managed as a background process (`start` / `stop` / `restart`) with a `/healthz` endpoint for a supervisor. |
|
|
41
41
|
| Orchestrator heartbeat | omp extension, activated by `.conductor-tick.json` | Prompts a 24/7 orchestrator session on a fixed interval so its standing loop actually runs, and marks the session stalled when its prompts stop being consumed. Inert in every other session — including a second session opened in the fleet's own directory. See [Orchestrator tick](#orchestrator-tick). |
|
|
42
42
|
| Onboarding skill | `skill://conductor-onboarding` | Directs an omp session to interview you, read your repos for real CI gates, and tailor `ORCHESTRATOR.md` — then finish through the wizard. Discovered automatically once the plugin is installed. See [Onboarding](#onboarding). |
|
|
43
43
|
|
|
44
44
|
The first two are thin wrappers over the same `daemon.ts`, so the plugin and the
|
|
45
|
-
CLI cannot disagree about what a cap means or where the state lives.
|
|
46
|
-
|
|
45
|
+
CLI cannot disagree about what a cap means or where the state lives. Claiming is
|
|
46
|
+
gated by the pause flag; tick sends are gated by the arm marker — they are not
|
|
47
|
+
the same switch. Prefer `hold` when you want both quiet.
|
|
47
48
|
|
|
48
49
|
## Your workflow vs. the package
|
|
49
50
|
|
|
@@ -108,9 +109,10 @@ session is carrying two versions of your policy.
|
|
|
108
109
|
exactly one value, `"github"`, and every tracker operation shells out to your
|
|
109
110
|
already-authenticated `gh` CLI — the conductor never stores a token of its own.
|
|
110
111
|
Gitea, Jira, and file-based trackers are not supported yet; the seam for them is
|
|
111
|
-
`src/tracker/github.ts`, which implements the whole
|
|
112
|
+
`src/tracker/github.ts`, which implements the whole nine-method `Tracker`
|
|
112
113
|
interface in `src/types.ts` (`listReady`, `addLabel`, `removeLabel`, `comment`,
|
|
113
|
-
`close`, `linkParent`) that a future
|
|
114
|
+
`close`, `linkParent`, `parentOf`, `openCloserFor`, `prState`) that a future
|
|
115
|
+
backend would swap in.
|
|
114
116
|
|
|
115
117
|
You tell the conductor where to look with three keys, all in
|
|
116
118
|
`~/.omp/conductor/config.json` (the [Configuration](#configuration) section has
|
|
@@ -274,7 +276,7 @@ The standing prompt is two layers:
|
|
|
274
276
|
|
|
275
277
|
| Layer | File | Updates how? |
|
|
276
278
|
| --- | --- | --- |
|
|
277
|
-
| Package floor | `src/briefs/orchestrator.md` | Every tick recomposes it into `ORCHESTRATOR.md` from the installed package.
|
|
279
|
+
| Package floor | `src/briefs/orchestrator.md` | Every tick recomposes it into `ORCHESTRATOR.md` from the installed package. Upgrade the package in this host's existing install root + restart is enough. |
|
|
278
280
|
| Fleet policy | `POLICY.md` | Yours. Setup writes the scaffold once; the Learning loop edits only this file. |
|
|
279
281
|
| Composed view | `ORCHESTRATOR.md` | Regenerated from floor + `POLICY.md` on each tick (and at setup). Do not hand-amend it for durable policy. |
|
|
280
282
|
| Worker brief | `src/briefs/worker.md` | Read per run from the package. |
|
|
@@ -346,7 +348,56 @@ The loop ticks every 5 minutes. `omp-conductor stop` shuts the loop down after
|
|
|
346
348
|
the current tick rather than mid-run. When the live process is the MainPID of
|
|
347
349
|
`omp-conductor.service`, stop goes through `systemctl stop` so a unit with
|
|
348
350
|
`Restart=on-failure` cannot bring it straight back; otherwise it is a raw
|
|
349
|
-
`SIGTERM` (then `SIGKILL` after 10 seconds).
|
|
351
|
+
`SIGTERM` (then `SIGKILL` after 10 seconds). An example unit (with
|
|
352
|
+
`SuccessExitStatus=0 143` and `MemoryMax=5G`) ships as
|
|
353
|
+
[`systemd/omp-conductor.service.example`](systemd/omp-conductor.service.example).
|
|
354
|
+
|
|
355
|
+
|
|
356
|
+
### Stop the conductor (hold / halt)
|
|
357
|
+
|
|
358
|
+
Four control planes used to answer "stop" differently. The package verbs:
|
|
359
|
+
|
|
360
|
+
| Verb | Claiming | Tick sends | Dispatch daemon | Conductor pane |
|
|
361
|
+
| --- | --- | --- | --- | --- |
|
|
362
|
+
| `hold` | paused | disarmed | left running | left running |
|
|
363
|
+
| `halt` | paused | disarmed | stopped (systemctl-aware) | left running |
|
|
364
|
+
| `halt --pane` | paused | disarmed | stopped | stopped + recovery pinned off |
|
|
365
|
+
| `pause` | paused | **still armed** | left running | left running |
|
|
366
|
+
|
|
367
|
+
`resume` clears pause only and **never re-arms**. `arm` is proof-gated: it sends a Telegram challenge and writes the arm marker only after your reply appears as a *user* turn in the orchestrator transcript. `halt --pane` targets the configured conductor agent only — it does **not** run `systemctl stop herdr-fleet`.
|
|
368
|
+
|
|
369
|
+
`status` prints a layered header (`dispatch` / `ticks` / `pane` / `recovery` / `herdr` / `daemon`) so a paused fleet cannot hide an armed orchestrator still spending turns.
|
|
370
|
+
|
|
371
|
+
`halt --pane` is **fail-closed**: it exits `0` only when the conductor agent is
|
|
372
|
+
*proven* gone. It writes the recovery pin first, so a failed stop still cannot be
|
|
373
|
+
undone by `herdr-conductor` respawning the agent, and then refuses (nonzero exit,
|
|
374
|
+
message on stderr) on every uncertainty:
|
|
375
|
+
|
|
376
|
+
- no tick config exists at all — `recover.sh` reads only
|
|
377
|
+
`$FLEET_CWD/.conductor-pane-halted`, and without that file the pane's own
|
|
378
|
+
directory is unknown, so the pin would land somewhere recovery never looks and
|
|
379
|
+
the agent would be respawned seconds later. `release-pane` refuses for the
|
|
380
|
+
same reason, and `status` shows `recovery unpinnable` rather than `clear`
|
|
381
|
+
- the tick config does not parse — the agent name would be a guess
|
|
382
|
+
- `herdr agent list` is unreachable, prints nothing, or prints output with no
|
|
383
|
+
explicit `agents` array; only a real `agents: []` means "no agents"
|
|
384
|
+
- an agent row is unreadable — a missing `name`/`pane_id`, or an `agent` field
|
|
385
|
+
present with a non-string value. An *absent* or `null` `agent` is the sticky
|
|
386
|
+
claim herdr reports after the agent exits, and stays a normal answer
|
|
387
|
+
- the configured agent name is not unique, or the claimed pane runs some other agent
|
|
388
|
+
- `pane process-info` fails, or the claim is live `omp` but names no recognizable
|
|
389
|
+
omp foreground PID — "cannot see it" is never reported as "it is stopped"
|
|
390
|
+
- a signal cannot be delivered, or liveness cannot be probed — only `ESRCH`
|
|
391
|
+
("no such process") proves death, so `EPERM` reads as "exists, not ours",
|
|
392
|
+
never as "stopped"
|
|
393
|
+
- the process is still alive after `SIGTERM` then `SIGKILL`
|
|
394
|
+
|
|
395
|
+
The pin is written to the pane's own directory (the one holding
|
|
396
|
+
`.conductor-tick.json`, which is `FLEET_CWD` — the only place `recover.sh` looks),
|
|
397
|
+
including when that tick config is the thing that failed to parse.
|
|
398
|
+
|
|
399
|
+
Clear the pin with `omp-conductor release-pane` when you want recovery again.
|
|
400
|
+
|
|
350
401
|
|
|
351
402
|
## How one tick works
|
|
352
403
|
|
|
@@ -560,6 +611,40 @@ multi-repo request is a human decision about contracts; it is not something to
|
|
|
560
611
|
infer from a label. Sending the issue back costs a label edit; guessing costs a
|
|
561
612
|
bad merge.
|
|
562
613
|
|
|
614
|
+
## Host sizing and memory
|
|
615
|
+
|
|
616
|
+
Workers are **in-process** omp sessions inside the daemon's single PID (plus one
|
|
617
|
+
long-lived orchestrator session). systemd's Memory peak for `omp-conductor.service`
|
|
618
|
+
is therefore daemon + every live worker + the orchestrator + any MCP stdio
|
|
619
|
+
children those sessions mount — not a separate worker process list.
|
|
620
|
+
|
|
621
|
+
On the reference deploy that produced [issue #51](https://github.com/TerrifiedBug/conductor/issues/51):
|
|
622
|
+
|
|
623
|
+
| Shape | Observed |
|
|
624
|
+
| --- | --- |
|
|
625
|
+
| Idle / workers restarting | ~430 MB RSS for the daemon alone |
|
|
626
|
+
| Two workers + orchestrator, busy | **3.2–4.2 GB** Memory peak for the unit; up to ~800 MB swap |
|
|
627
|
+
|
|
628
|
+
That peak is **expected for concurrent SDK sessions**, not evidence of a
|
|
629
|
+
conductor-side leak: the SQLite store is disk-backed, admission state is
|
|
630
|
+
per-tick, and worker sessions are disposed when a run ends. What grows is the
|
|
631
|
+
session heap (conversation + tool output); a single graph-assisted run has been
|
|
632
|
+
measured in the hundreds of thousands of characters of tool output.
|
|
633
|
+
|
|
634
|
+
**Practical guidance**
|
|
635
|
+
|
|
636
|
+
- Prefer **≥16 GiB RAM** for the default `maxConcurrentWorkers: 2`, and do **not**
|
|
637
|
+
co-locate ClickHouse / other multi-GB services beside that fleet on an ≤8 GiB
|
|
638
|
+
box.
|
|
639
|
+
- On hosts under ~16 GiB, set `maxConcurrentWorkers` to **1**. `/conductor setup`
|
|
640
|
+
does this automatically when it can read host RAM.
|
|
641
|
+
- Supervise the daemon with a unit that sets `SuccessExitStatus=0 143` and a
|
|
642
|
+
`MemoryMax=` just above your expected peak. A ready-to-edit example ships as
|
|
643
|
+
[`systemd/omp-conductor.service.example`](systemd/omp-conductor.service.example)
|
|
644
|
+
(`MemoryMax=5G` for the two-worker shape).
|
|
645
|
+
- `omp-conductor status` prints daemon `rss` from `/healthz` when the process is
|
|
646
|
+
up, so you can see pressure without scraping journald.
|
|
647
|
+
|
|
563
648
|
## Caps
|
|
564
649
|
|
|
565
650
|
Caps resolve per project: the global `defaults` block, then the project's own
|
|
@@ -568,7 +653,7 @@ rest. `0` is a real value (a hard stop), not "unset".
|
|
|
568
653
|
|
|
569
654
|
| Cap | Default | What it protects |
|
|
570
655
|
| --- | --- | --- |
|
|
571
|
-
| `maxConcurrentWorkers` | `2` | Parallel omp sessions. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. Raise it only if you actually have the runners. |
|
|
656
|
+
| `maxConcurrentWorkers` | `2` (setup may write `1` on <16 GiB hosts) | Parallel in-process omp sessions inside the daemon PID. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. On hosts under ~16 GiB RAM, prefer `1` so the unit stays out of swap ([host sizing](#host-sizing-and-memory)). Raise it only if you actually have the runners *and* the RAM. |
|
|
572
657
|
| `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total`. |
|
|
573
658
|
| `workerMaxTurns` | `120` | Turn ceiling for one worker. Catches a session looping without converging. |
|
|
574
659
|
| `workerWallClockMs` | `5400000` (90 minutes) | Wall-clock ceiling for one worker. A session that is merely stuck spends no turns, so turns alone cannot detect it. |
|
|
@@ -1082,6 +1167,11 @@ omp-conductor start [--port N] [--project NAME]
|
|
|
1082
1167
|
omp-conductor stop
|
|
1083
1168
|
omp-conductor restart [--port N] [--project NAME]
|
|
1084
1169
|
omp-conductor status [--project NAME]
|
|
1170
|
+
omp-conductor hold [--project NAME]
|
|
1171
|
+
omp-conductor halt [--pane] [--project NAME]
|
|
1172
|
+
omp-conductor arm [--project NAME]
|
|
1173
|
+
omp-conductor disarm [--project NAME]
|
|
1174
|
+
omp-conductor release-pane [--project NAME]
|
|
1085
1175
|
omp-conductor tail <issue> [--project NAME]
|
|
1086
1176
|
omp-conductor unblock <issue> [--project NAME]
|
|
1087
1177
|
omp-conductor daemon [--once] [--port N] [--project NAME]
|
|
@@ -1097,15 +1187,20 @@ omp-conductor help
|
|
|
1097
1187
|
| `start` | Spawn the loop in the background, detached, and wait until it answers `GET /healthz` on `:8787`. Refuses if one is already live, naming its pid. If the process dies or never serves, `start` cleans up after it and quotes the tail of `daemon.log`. |
|
|
1098
1188
|
| `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
|
|
1099
1189
|
| `restart` | Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
|
|
1100
|
-
| `status [--project NAME]` |
|
|
1190
|
+
| `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / `pane` / `recovery` / `herdr` / `daemon`, then the project body (caps, active runs, today's usage). The `daemon` block includes `rss` from `/healthz` when the process is up (workers share that PID — see [host sizing](#host-sizing-and-memory)). While live workers > 0, prints a `deploy` line naming the count so a busy restart is visible before you take it. A `.conductor-stalled` marker in the state directory adds an `orchestrator STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
|
|
1191
|
+
| `hold [--project NAME]` | Soft stop: pause claiming **and** disarm ticks. Daemon and pane stay up. Prefer this over `pause` when the intent is "stop the conductor" without killing processes. See [Stop the conductor](#stop-the-conductor-hold--halt). |
|
|
1192
|
+
| `halt [--pane] [--project NAME]` | `hold`, then stop the dispatch daemon (systemctl-aware). Pane stays up unless `--pane` is passed. `halt --pane` also pins herdr-conductor recovery off for the conductor agent only — it does **not** stop `herdr-fleet.service` or any other herdr session. Fail-closed: exits nonzero unless the agent is proven gone. |
|
|
1193
|
+
| `arm [--project NAME]` | Proof-gated: send a Telegram challenge and write the arm marker only after your reply appears as a user turn in the orchestrator transcript. Never auto-armed by `resume` / `hold`. |
|
|
1194
|
+
| `disarm [--project NAME]` | Remove the arm marker so ticks skip. Processes untouched. |
|
|
1195
|
+
| `release-pane [--project NAME]` | Clear the `halt --pane` recovery pin so herdr-conductor may resume the fleet agent again. |
|
|
1101
1196
|
| `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
|
|
1102
1197
|
| `unblock <issue>` | Remove that issue's `blocked` and `failed` state labels through the tracker, so the next tick can claim it again. This is the supported way back for an escalation you answered: eligibility disqualifies any issue carrying a state label, so an answered issue that keeps one is never re-claimed and the answer is inert. Removing a label the issue does not carry is a no-op, so both are always cleared and neither has to be looked up first. `agent:in-progress` is deliberately not touched — it means a worker process exists, which is not something an answer changes. The run history is left exactly as it is: an answered block still spent a worker, so it still counts toward `maxAttemptsPerIssue`, and the output says how many attempts remain — or warns that the next tick will escalate instead of dispatching, when none do. Exits `2` with `unblock needs an issue number` on a missing or malformed positional. |
|
|
1103
1198
|
| `daemon` | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. This is what `start` launches, and what a systemd unit should call. Writes the pidfile itself, and refuses with `another daemon is alive (pid N); stop it first` rather than becoming a second dispatcher. |
|
|
1104
1199
|
| `daemon --once` | Run a single tick and exit. No HTTP server, and no pidfile — a drill must not register itself as the daemon, or the next reader believes it and the real daemon's in-flight runs get reconciled as orphans. |
|
|
1105
1200
|
| `--port N` | Accepted by `start`, `restart` and `daemon`. Both `--port 9000` and `--port=9000` work; missing or out of range exits `2` rather than falling back to the default, because probing the wrong endpoint is worse than a hard failure. |
|
|
1106
1201
|
| `--project NAME` | Pick the project to service. One daemon process serves exactly one project; with several configured projects the name is required. |
|
|
1107
|
-
| `pause` | Stop claiming new work. The running daemon notices on its next tick; runs already in flight finish. The orchestrator heartbeat keeps ticking — its gate is the arm marker, not this flag. |
|
|
1108
|
-
| `resume` |
|
|
1202
|
+
| `pause` | Stop claiming new work only. The running daemon notices on its next tick; runs already in flight finish. The orchestrator heartbeat keeps ticking if armed — its gate is the arm marker, not this flag. Prefer `hold` to silence both. |
|
|
1203
|
+
| `resume` | Clear pause only — does **not** re-arm. Run `arm` after an inbound Telegram proof to resume ticks. |
|
|
1109
1204
|
| `graph-setup` | Print how to set up the code-graph indexes workers query instead of grepping: a `git clone` for every index-only clone that does not exist yet, the one-shot index command per repo, and a `cbm-reindex.service` + `cbm-reindex.timer` pair generated from the project's own repos and branches. Reads only, so it is safe on a host where you are not root. Exits `1` when no repo in the project has [`graphProject`](#configuration) set, because the fix is a wizard answer rather than a flag. See [Code-graph discovery](#code-graph-discovery). |
|
|
1110
1205
|
| `--write` | Only for `graph-setup`. Writes the refresh script into the state directory and the two units into `/etc/systemd/system`, then prints the exact `systemctl daemon-reload && systemctl enable --now cbm-reindex.timer` to run. It never runs `systemctl` itself and never enables anything: that needs root, and a package that enables system timers behind your back is one you cannot audit by reading its output. |
|
|
1111
1206
|
| `brief-upgrade` | Inspect the package-floor + `POLICY.md` overlay. Reports by default; see [Keeping a brief current](#keeping-a-brief-current). |
|
|
@@ -1116,12 +1211,14 @@ omp-conductor help
|
|
|
1116
1211
|
| `help`, `--help`, `-h` | Print usage. An unknown or missing verb prints it too, and exits `2`. |
|
|
1117
1212
|
|
|
1118
1213
|
Pause is a flag file under the state directory, so it applies to every project and
|
|
1119
|
-
survives a daemon restart.
|
|
1214
|
+
survives a daemon restart. Hold also removes the arm marker the heartbeat reads,
|
|
1215
|
+
so both brains go quiet without killing processes.
|
|
1120
1216
|
|
|
1121
|
-
|
|
1122
|
-
`/conductor
|
|
1123
|
-
|
|
1124
|
-
|
|
1217
|
+
These are available in-session as `/conductor setup`, `/conductor status`,
|
|
1218
|
+
`/conductor hold`, `/conductor halt [--pane]`, `/conductor arm`, `/conductor disarm`,
|
|
1219
|
+
`/conductor release-pane`, `/conductor pause` and `/conductor resume`, each taking
|
|
1220
|
+
an optional project name. Background-process management (`start` / `stop` /
|
|
1221
|
+
`restart`) is CLI-only: the plugin does not start, stop or restart the daemon.
|
|
1125
1222
|
|
|
1126
1223
|
### Health endpoint
|
|
1127
1224
|
|
|
@@ -1162,23 +1259,30 @@ The worker ends with a six-line evidence report (issue, pr, state, gates, change
|
|
|
1162
1259
|
next). `pushed-green` means it watched the checks go green rather than expecting
|
|
1163
1260
|
them to.
|
|
1164
1261
|
|
|
1165
|
-
###
|
|
1262
|
+
### Worker confinement and the integrity tripwire
|
|
1263
|
+
|
|
1264
|
+
A worker session is rooted at its worktree `cwd`. **Structured file tools are
|
|
1265
|
+
gated mechanically:** `runWorker` asks `createSession({ confineToCwd: true })`,
|
|
1266
|
+
which installs an inline harness extension that blocks `write` / `edit` /
|
|
1267
|
+
`read` / `grep` / `glob` when the tool's path resolves outside that worktree
|
|
1268
|
+
(symlink-aware). Target selection was already mechanical — only a repo in
|
|
1269
|
+
`routing.repos` is ever checked out — and the caps still bound *how much* work
|
|
1270
|
+
happens.
|
|
1166
1271
|
|
|
1167
|
-
|
|
1168
|
-
|
|
1169
|
-
|
|
1170
|
-
|
|
1171
|
-
caps, but between them they bound where work starts and how much of it happens,
|
|
1172
|
-
not how far a wandering or prompt-injected session can reach.
|
|
1272
|
+
**`bash` is not gated.** Its argument is an opaque shell string; pretending to
|
|
1273
|
+
parse it would be a false sense of security. The brief still forbids escaping
|
|
1274
|
+
via the shell, and the deploy-level answer is a least-privilege worker uid
|
|
1275
|
+
(below).
|
|
1173
1276
|
|
|
1174
|
-
|
|
1175
|
-
|
|
1176
|
-
|
|
1177
|
-
|
|
1178
|
-
|
|
1179
|
-
|
|
1180
|
-
|
|
1181
|
-
|
|
1277
|
+
#### Integrity tripwire (package self-hash)
|
|
1278
|
+
|
|
1279
|
+
Separately, the conductor watches *itself*. At startup the daemon sha256s every
|
|
1280
|
+
`.ts` and `.md` file of its own installed `src/` — the dispatcher and the briefs
|
|
1281
|
+
both, since rewriting a brief buys more than rewriting the loop — and re-walks
|
|
1282
|
+
that tree on every tick (about 0.6 ms). Any difference at all, changed or added
|
|
1283
|
+
or removed, is read as the package having been modified underneath a running
|
|
1284
|
+
daemon: the tick claims nothing, the fleet is paused, and a tier-2 escalation
|
|
1285
|
+
naming the first few differing paths pages you **once**, not every five minutes.
|
|
1182
1286
|
|
|
1183
1287
|
**A normal deploy never trips it.** The baseline is recorded per daemon process,
|
|
1184
1288
|
so installing a new build and restarting the unit re-records it from the new
|
|
@@ -1187,9 +1291,32 @@ diverge from it. That also means `omp-conductor resume` on its own will not hold
|
|
|
1187
1291
|
— the next tick re-walks, still differs, and pauses again. Put the files back, or
|
|
1188
1292
|
restart onto the build you meant to be running.
|
|
1189
1293
|
|
|
1190
|
-
This
|
|
1191
|
-
|
|
1192
|
-
|
|
1294
|
+
This catches a worker (or human) that still managed to edit the live install —
|
|
1295
|
+
including via `bash` — after the fact. It is detection for the package boundary,
|
|
1296
|
+
not a substitute for the worktree gate or a dedicated uid.
|
|
1297
|
+
|
|
1298
|
+
#### Least-privilege worker uid (deploy)
|
|
1299
|
+
|
|
1300
|
+
The largest remaining win is OS-level: run the daemon (or at least worker
|
|
1301
|
+
sessions, when the harness supports a uid switch) as a user that can write only
|
|
1302
|
+
its worktrees and mirrors. A sketch that matches the reference single-host
|
|
1303
|
+
deploy:
|
|
1304
|
+
|
|
1305
|
+
1. Create a system user, e.g. `conductor-worker`, with home under
|
|
1306
|
+
`/var/lib/conductor-worker` (or similar).
|
|
1307
|
+
2. `chown` the project's `workspaceRoot` and `mirrorRoot` to that user; leave
|
|
1308
|
+
`~/.omp/conductor/config.json` readable only by the operator/daemon account
|
|
1309
|
+
(`0600` as shipped).
|
|
1310
|
+
3. Do **not** put the worker uid in `docker` / `sudoers`, and do not give it the
|
|
1311
|
+
operator's `gh` auth if a narrower deploy token can open PRs in the routed
|
|
1312
|
+
repos alone.
|
|
1313
|
+
4. Point the [example systemd unit](systemd/omp-conductor.service.example)
|
|
1314
|
+
`User=` / `Group=` at that account once the daemon itself should run
|
|
1315
|
+
unprivileged end-to-end.
|
|
1316
|
+
|
|
1317
|
+
Until that uid exists, a root-or-operator daemon still has a mechanical
|
|
1318
|
+
worktree gate on structured tools and an integrity tripwire on its own package —
|
|
1319
|
+
but `bash` plus host credentials remain a prompt-and-deploy problem.
|
|
1193
1320
|
|
|
1194
1321
|
## Limitations
|
|
1195
1322
|
|
|
@@ -1243,13 +1370,13 @@ Known and deliberate in this version:
|
|
|
1243
1370
|
status`, which lists every occupied issue, or follow `daemon.log`.
|
|
1244
1371
|
- **Merges, releases and deploys are human-only, by design.** The conductor
|
|
1245
1372
|
produces green PRs and stops.
|
|
1246
|
-
- **Worker confinement is
|
|
1247
|
-
|
|
1248
|
-
|
|
1249
|
-
|
|
1250
|
-
|
|
1251
|
-
|
|
1252
|
-
|
|
1373
|
+
- **Worker confinement is partial.** Structured `write` / `edit` / `read` /
|
|
1374
|
+
`grep` / `glob` calls are blocked outside the worktree by an inline harness
|
|
1375
|
+
extension (`confineToCwd`). `bash` is not: a shell one-liner can still leave
|
|
1376
|
+
the tree. Prefer a [least-privilege worker uid](#least-privilege-worker-uid-deploy);
|
|
1377
|
+
the [integrity tripwire](#integrity-tripwire-package-self-hash) still pages if
|
|
1378
|
+
the installed package itself changes under a live daemon.
|
|
1379
|
+
|
|
1253
1380
|
|
|
1254
1381
|
## License
|
|
1255
1382
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "omp-conductor",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.15",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
|
|
@@ -23,6 +23,7 @@
|
|
|
23
23
|
"src",
|
|
24
24
|
"!src/**/*.test.ts",
|
|
25
25
|
"skills",
|
|
26
|
+
"systemd",
|
|
26
27
|
"README.md",
|
|
27
28
|
"LICENSE"
|
|
28
29
|
],
|
package/src/brief-upgrade.ts
CHANGED
|
@@ -573,7 +573,7 @@ export function formatBriefStatus(path: string, status: BriefStatus): string {
|
|
|
573
573
|
` floor package template → recomposed into ${status.orchestratorPath} each tick`,
|
|
574
574
|
` policy ${status.policyPath} (Learning loop / operator edits)`,
|
|
575
575
|
"",
|
|
576
|
-
"Protocol updates:
|
|
576
|
+
"Protocol updates: upgrade omp-conductor in this host's existing install root (same package manager), then restart the daemon — no brief-upgrade --apply.",
|
|
577
577
|
"Legacy migrate: omp-conductor brief-upgrade --migrate",
|
|
578
578
|
].join("\n");
|
|
579
579
|
}
|
|
@@ -3,8 +3,9 @@
|
|
|
3
3
|
This file is the **package floor**: duties, tiers, hard boundaries, and the
|
|
4
4
|
Learning loop. It ships inside `omp-conductor` and is re-rendered into your
|
|
5
5
|
workspace on every tick (composed with `POLICY.md` as `ORCHESTRATOR.md` for the
|
|
6
|
-
session). Protocol updates arrive
|
|
7
|
-
|
|
6
|
+
session). Protocol updates arrive when you upgrade the installed `omp-conductor`
|
|
7
|
+
package (same package manager / install root) and restart — you do not
|
|
8
|
+
brief-upgrade the floor.
|
|
8
9
|
|
|
9
10
|
Fleet-specific policy — Releases, Project context, Reporting, Amendments — lives
|
|
10
11
|
in **`POLICY.md`** beside this composed brief. The Learning loop edits
|
package/src/briefs/worker.md
CHANGED
|
@@ -15,7 +15,7 @@ inside your own worktree.
|
|
|
15
15
|
- **Code repo:** {{REPO}}
|
|
16
16
|
- **Your worktree (cwd):** `{{WORKTREE}}`
|
|
17
17
|
- **Your branch:** `{{BRANCH}}` — already created for you off the repo's default
|
|
18
|
-
branch. Never switch branches and never touch a path outside the worktree.
|
|
18
|
+
branch. Never switch branches and never touch a path outside the worktree (write/edit/read/grep/glob are also blocked mechanically outside this checkout; `bash` is still a must-not — do not use it to escape).
|
|
19
19
|
|
|
20
20
|
Read the issue first — it carries the acceptance criteria and any discussion the
|
|
21
21
|
dispatcher did not copy down:
|
package/src/cli.ts
CHANGED
|
@@ -21,12 +21,21 @@ import {
|
|
|
21
21
|
writeMergedBrief,
|
|
22
22
|
} from "./brief-upgrade.ts";
|
|
23
23
|
import { findProject, loadConfig, resolveCaps, stateDir } from "./config.ts";
|
|
24
|
-
import { dbPath,
|
|
24
|
+
import { dbPath, runDaemon, setPaused } from "./daemon.ts";
|
|
25
|
+
import {
|
|
26
|
+
armTicks,
|
|
27
|
+
clearPaneHalt,
|
|
28
|
+
disarmTicks,
|
|
29
|
+
halt,
|
|
30
|
+
haltWithPane,
|
|
31
|
+
hold,
|
|
32
|
+
releaseHold,
|
|
33
|
+
renderStatus,
|
|
34
|
+
} from "./fleet.ts";
|
|
25
35
|
import { formatGraphSetup, graphRepos, writeGraphSetup, type GraphSetupWrite } from "./graph.ts";
|
|
26
36
|
import {
|
|
27
37
|
clearRecord,
|
|
28
38
|
DEFAULT_PORT,
|
|
29
|
-
healthCheck,
|
|
30
39
|
livingDaemon,
|
|
31
40
|
restartDaemon,
|
|
32
41
|
startDaemon,
|
|
@@ -53,6 +62,11 @@ usage:
|
|
|
53
62
|
omp-conductor stop
|
|
54
63
|
omp-conductor restart [--port N] [--project NAME]
|
|
55
64
|
omp-conductor status [--project NAME]
|
|
65
|
+
omp-conductor hold [--project NAME]
|
|
66
|
+
omp-conductor halt [--pane] [--project NAME]
|
|
67
|
+
omp-conductor arm [--project NAME]
|
|
68
|
+
omp-conductor disarm [--project NAME]
|
|
69
|
+
omp-conductor release-pane [--project NAME]
|
|
56
70
|
omp-conductor tail <issue> [--project NAME]
|
|
57
71
|
omp-conductor unblock <issue> [--project NAME]
|
|
58
72
|
omp-conductor daemon [--once] [--port N] [--project NAME]
|
|
@@ -73,8 +87,22 @@ usage:
|
|
|
73
87
|
dirty live worktrees before orphaning those rows — see README
|
|
74
88
|
"Deploying a new package onto a busy fleet". Goes through systemctl
|
|
75
89
|
when the unit owns the live pid.
|
|
76
|
-
status
|
|
77
|
-
daemon
|
|
90
|
+
status layered fleet report: dispatch (running|paused|stopped), ticks
|
|
91
|
+
(armed|disarmed|…), pane, herdr, daemon — then caps and active runs.
|
|
92
|
+
hold soft stop: pause claiming AND disarm ticks. Daemon and pane stay up.
|
|
93
|
+
This is "stop the conductor overnight" without killing processes.
|
|
94
|
+
halt hold, then stop the dispatch daemon (systemctl-aware). Pane stays up
|
|
95
|
+
unless --pane is passed.
|
|
96
|
+
halt --pane
|
|
97
|
+
halt, then pin herdr-conductor recovery off for the conductor agent
|
|
98
|
+
only — does NOT stop herdr-fleet.service or any other herdr session.
|
|
99
|
+
Clear the pin with release-pane when you want recovery again.
|
|
100
|
+
arm proof-gated: send a Telegram challenge and write the arm marker only
|
|
101
|
+
after your reply appears as a user turn in the orchestrator transcript.
|
|
102
|
+
Never auto-armed by resume/hold.
|
|
103
|
+
disarm remove the arm marker so ticks skip. Processes untouched.
|
|
104
|
+
release-pane
|
|
105
|
+
clear the halt --pane recovery pin so herdr-conductor may resume again.
|
|
78
106
|
tail follow the newest run for <issue>: the worker's assistant text and
|
|
79
107
|
the tools it calls, printed as they land. Workers are sessions inside
|
|
80
108
|
the daemon rather than terminals, so this is the only way to watch
|
|
@@ -87,8 +115,8 @@ usage:
|
|
|
87
115
|
cost a worker.
|
|
88
116
|
daemon run the dispatch loop in the foreground; --once runs a single tick
|
|
89
117
|
and exits. This is what \`start\` launches.
|
|
90
|
-
pause stop claiming new work
|
|
91
|
-
resume
|
|
118
|
+
pause stop claiming new work only (ticks keep firing if armed). Prefer hold.
|
|
119
|
+
resume clear pause only — does NOT re-arm. Prefer hold's inverse: resume + arm.
|
|
92
120
|
graph-setup
|
|
93
121
|
print how to set up the code-graph indexes workers query instead of
|
|
94
122
|
grepping: the clone commands for any missing index-only clone, the
|
|
@@ -107,10 +135,18 @@ usage:
|
|
|
107
135
|
help print this text (also --help, -h).
|
|
108
136
|
|
|
109
137
|
Pause is a flag file under the state directory, so it applies to every project
|
|
110
|
-
and survives a daemon restart.
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
138
|
+
and survives a daemon restart. Hold also removes the arm marker the heartbeat
|
|
139
|
+
reads, so both brains go quiet without killing processes. A running daemon is
|
|
140
|
+
tracked by a pidfile under $OMP_CONDUCTOR_RUNTIME_DIR (default
|
|
141
|
+
~/.omp/run/daemons/omp-conductor), written whether it was started in the
|
|
142
|
+
background or in the foreground, and probed for liveness on every read — a
|
|
143
|
+
stale one never blocks a start.
|
|
144
|
+
|
|
145
|
+
Stop the conductor:
|
|
146
|
+
hold no claims, no tick sends (inspectable)
|
|
147
|
+
halt hold + stop dispatch daemon
|
|
148
|
+
halt --pane halt + pin conductor-pane recovery off
|
|
149
|
+
resume && arm clear pause, then prove inbound Telegram before ticks resume`;
|
|
114
150
|
|
|
115
151
|
/** Accepts both `--port 9000` and `--port=9000`; returns undefined when absent. */
|
|
116
152
|
function flag(argv: string[], name: string): string | undefined {
|
|
@@ -146,26 +182,6 @@ function humanDuration(ms: number): string {
|
|
|
146
182
|
return `${Math.floor(h / 24)}d ${String(h % 24).padStart(2, "0")}h`;
|
|
147
183
|
}
|
|
148
184
|
|
|
149
|
-
/**
|
|
150
|
-
* The daemon half of `status`. Kept separate from `formatStatus` because the
|
|
151
|
-
* pidfile and the endpoint are the CLI's business, not the dispatcher's, and
|
|
152
|
-
* because a pid without a `/healthz` answer is a distinct — and interesting —
|
|
153
|
-
* state: the process is up but the loop is not serving.
|
|
154
|
-
*/
|
|
155
|
-
async function daemonSection(): Promise<string> {
|
|
156
|
-
const rec = livingDaemon();
|
|
157
|
-
if (rec === undefined) return "daemon not running";
|
|
158
|
-
const health = await healthCheck(rec.port);
|
|
159
|
-
return [
|
|
160
|
-
"daemon",
|
|
161
|
-
` pid ${rec.pid}`,
|
|
162
|
-
` uptime ${humanDuration(Date.now() - rec.startedAt)}`,
|
|
163
|
-
` port ${rec.port}`,
|
|
164
|
-
...(rec.project === undefined ? [] : [` project ${rec.project}`]),
|
|
165
|
-
` healthz ${health.ok ? `ok ${health.body ?? ""}`.trimEnd() : "unreachable — the process is up but not serving"}`,
|
|
166
|
-
` log ${rec.logFile}`,
|
|
167
|
-
].join("\n");
|
|
168
|
-
}
|
|
169
185
|
|
|
170
186
|
/**
|
|
171
187
|
* The orchestrator half, and the one thing `status` has ever known about the
|
|
@@ -456,9 +472,84 @@ try {
|
|
|
456
472
|
}
|
|
457
473
|
|
|
458
474
|
case "status": {
|
|
459
|
-
const
|
|
475
|
+
const project = flag(argv, "project");
|
|
476
|
+
const text = await renderStatus(project);
|
|
460
477
|
const stalled = stallLine();
|
|
461
|
-
process.stdout.write(`${
|
|
478
|
+
process.stdout.write(`${text}${stalled === undefined ? "\n" : `\n\n${stalled}\n`}`);
|
|
479
|
+
break;
|
|
480
|
+
}
|
|
481
|
+
|
|
482
|
+
case "hold": {
|
|
483
|
+
const r = hold(flag(argv, "project"));
|
|
484
|
+
process.stdout.write(
|
|
485
|
+
`held — claiming paused` +
|
|
486
|
+
`${r.wasPaused ? " (already paused)" : ""}` +
|
|
487
|
+
`; ticks disarmed at ${r.disarmed.path}` +
|
|
488
|
+
`${r.disarmed.wasArmed ? "" : " (was already disarmed)"}\n` +
|
|
489
|
+
`daemon and pane left running; halt to stop the daemon\n`,
|
|
490
|
+
);
|
|
491
|
+
break;
|
|
492
|
+
}
|
|
493
|
+
|
|
494
|
+
case "halt": {
|
|
495
|
+
const project = flag(argv, "project");
|
|
496
|
+
const withPane = argv.includes("--pane");
|
|
497
|
+
if (withPane) {
|
|
498
|
+
const r = await haltWithPane(project);
|
|
499
|
+
const stopLine =
|
|
500
|
+
r.stop.kind === "not-running"
|
|
501
|
+
? "daemon was not running"
|
|
502
|
+
: `daemon stopped — pid ${r.stop.pid}${r.stop.via === "systemctl" ? " (via systemctl)" : ""}`;
|
|
503
|
+
process.stdout.write(
|
|
504
|
+
`halted — claiming paused; ticks disarmed at ${r.hold.disarmed.path}\n` +
|
|
505
|
+
`${stopLine}\n` +
|
|
506
|
+
`pane recovery pinned at ${r.pane.pinPath}\n` +
|
|
507
|
+
`pane stop: ${r.pane.stopped} — ${r.pane.detail}\n` +
|
|
508
|
+
` (conductor agent "${r.pane.agentName}" only — herdr-fleet.service was NOT stopped;\n` +
|
|
509
|
+
` release-pane clears the pin when you want recovery again)\n`,
|
|
510
|
+
);
|
|
511
|
+
} else {
|
|
512
|
+
const r = await halt(project);
|
|
513
|
+
const stopLine =
|
|
514
|
+
r.stop.kind === "not-running"
|
|
515
|
+
? "daemon was not running"
|
|
516
|
+
: `daemon stopped — pid ${r.stop.pid}${r.stop.via === "systemctl" ? " (via systemctl)" : ""}`;
|
|
517
|
+
process.stdout.write(
|
|
518
|
+
`halted — claiming paused; ticks disarmed at ${r.hold.disarmed.path}\n` +
|
|
519
|
+
`${stopLine}\n` +
|
|
520
|
+
`pane left running (pass --pane to stop the conductor agent and pin recovery)\n`,
|
|
521
|
+
);
|
|
522
|
+
}
|
|
523
|
+
break;
|
|
524
|
+
}
|
|
525
|
+
|
|
526
|
+
case "arm": {
|
|
527
|
+
process.stdout.write("arm: sending inbound Telegram challenge…\n");
|
|
528
|
+
const r = await armTicks(flag(argv, "project"));
|
|
529
|
+
process.stdout.write(
|
|
530
|
+
`ARMED — inbound round-trip proved with owner ${r.owner}; ticks are now live.\n` +
|
|
531
|
+
`marker ${r.path}${r.alreadyArmed ? " (replaced previous marker)" : ""}\n`,
|
|
532
|
+
);
|
|
533
|
+
break;
|
|
534
|
+
}
|
|
535
|
+
|
|
536
|
+
case "disarm": {
|
|
537
|
+
const r = disarmTicks(flag(argv, "project"));
|
|
538
|
+
process.stdout.write(
|
|
539
|
+
`disarmed — ticks will be skipped` +
|
|
540
|
+
`${r.wasArmed ? "" : " (was already disarmed)"}\n` +
|
|
541
|
+
`marker ${r.path}\n`,
|
|
542
|
+
);
|
|
543
|
+
break;
|
|
544
|
+
}
|
|
545
|
+
|
|
546
|
+
case "release-pane": {
|
|
547
|
+
const r = clearPaneHalt(flag(argv, "project"));
|
|
548
|
+
process.stdout.write(
|
|
549
|
+
r.wasHalted
|
|
550
|
+
? `pane recovery pin cleared — ${r.path}\nherdr-conductor may resume the fleet agent again\n`
|
|
551
|
+
: `no pane recovery pin at ${r.path}\n`,
|
|
552
|
+
);
|
|
462
553
|
break;
|
|
463
554
|
}
|
|
464
555
|
|
|
@@ -484,12 +575,18 @@ try {
|
|
|
484
575
|
|
|
485
576
|
case "pause":
|
|
486
577
|
setPaused(true);
|
|
487
|
-
process.stdout.write(
|
|
578
|
+
process.stdout.write(
|
|
579
|
+
"paused — no new work will be claimed\n" +
|
|
580
|
+
"note: ticks keep firing if armed; use hold to silence both\n",
|
|
581
|
+
);
|
|
488
582
|
break;
|
|
489
583
|
|
|
490
584
|
case "resume":
|
|
491
|
-
|
|
492
|
-
process.stdout.write(
|
|
585
|
+
releaseHold();
|
|
586
|
+
process.stdout.write(
|
|
587
|
+
"resumed — claiming allowed on the next tick\n" +
|
|
588
|
+
"note: did NOT re-arm; run arm after an inbound Telegram proof to resume ticks\n",
|
|
589
|
+
);
|
|
493
590
|
break;
|
|
494
591
|
|
|
495
592
|
case "graph-setup": {
|