omp-conductor 0.2.2 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +212 -35
- package/package.json +1 -1
- package/skills/conductor-onboarding/SKILL.md +33 -22
- package/src/briefs/orchestrator.md +46 -14
- package/src/briefs/worker.md +9 -2
- package/src/cli.ts +291 -13
- package/src/config.ts +127 -16
- package/src/daemon.ts +474 -50
- package/src/lifecycle.ts +6 -4
- package/src/orchestrator-tick.ts +209 -45
- package/src/plugin.ts +60 -0
- package/src/setup.ts +98 -4
- package/src/store.ts +20 -1
- package/src/tracker/github.ts +94 -2
- package/src/types.ts +56 -3
- package/src/unblock.ts +113 -0
- package/src/worker.ts +14 -0
- package/src/worktree.ts +119 -0
package/README.md
CHANGED
|
@@ -28,8 +28,9 @@ the line.
|
|
|
28
28
|
When a run does get stuck, the first responder is not you. A tier-1 escalation is
|
|
29
29
|
injected into a long-lived **orchestrator session** that can read the issue and the
|
|
30
30
|
run's transcript and then either re-brief the worker or decide the problem genuinely
|
|
31
|
-
needs a human. It never edits product code
|
|
32
|
-
|
|
31
|
+
needs a human. It never edits product code and never pushes a branch; whether it
|
|
32
|
+
may *merge* is a setup answer (`authority`), and it defaults to no. Only tier 2
|
|
33
|
+
pages you directly.
|
|
33
34
|
|
|
34
35
|
The package ships three deployables, plus one skill:
|
|
35
36
|
|
|
@@ -37,7 +38,7 @@ The package ships three deployables, plus one skill:
|
|
|
37
38
|
| --- | --- | --- |
|
|
38
39
|
| omp plugin | `/conductor` slash command | Inspect and arm the conductor from inside an omp session: dry-run the queue, read status, pause, resume. |
|
|
39
40
|
| Standalone daemon | `omp-conductor` binary | The dispatch loop, managed as a background process (`start` / `stop` / `restart`) with a `/healthz` endpoint for a supervisor. |
|
|
40
|
-
| Orchestrator heartbeat | omp extension, activated by `.conductor-tick.json` | Prompts a 24/7 orchestrator session on a fixed interval so its standing loop actually runs. Inert in every other session. See [Orchestrator tick](#orchestrator-tick). |
|
|
41
|
+
| Orchestrator heartbeat | omp extension, activated by `.conductor-tick.json` | Prompts a 24/7 orchestrator session on a fixed interval so its standing loop actually runs, and marks the session stalled when its prompts stop being consumed. Inert in every other session. See [Orchestrator tick](#orchestrator-tick). |
|
|
41
42
|
| Onboarding skill | `skill://conductor-onboarding` | Directs an omp session to interview you, read your repos for real CI gates, and tailor `ORCHESTRATOR.md` — then finish through the wizard. Discovered automatically once the plugin is installed. See [Onboarding](#onboarding). |
|
|
42
43
|
|
|
43
44
|
The first two are thin wrappers over the same `daemon.ts`, so the plugin and the
|
|
@@ -66,7 +67,7 @@ second, explicit confirmation. It is a starting point rather than a contract:
|
|
|
66
67
|
| Section | Whose |
|
|
67
68
|
| --- | --- |
|
|
68
69
|
| Duties (drain, groom, report), escalation tiers, hard boundaries | **Fixed** — they describe how this package already behaves. |
|
|
69
|
-
| Releases | **Yours.**
|
|
70
|
+
| Releases | **Yours.** Its opening paragraph is rendered from the `authority` you answered in setup — by default "humans release, and you do not merge". Everything under it is the procedure, and a delegated session is told not to cut a release until you have written one: what, when, on what proof, and what stays permanently forbidden. |
|
|
70
71
|
| Reporting | **Yours**, seeded from the scope you chose in setup. |
|
|
71
72
|
|
|
72
73
|
Reporting is the one half of that the config also knows about, because the wizard
|
|
@@ -95,7 +96,7 @@ session that ignores its constraint line still reaches you. Scope is a
|
|
|
95
96
|
constraint the model is handed each turn. It is not a gate the model is held to.
|
|
96
97
|
The enforcement roadmap (a tool-call tripwire, and config-versus-behaviour drift
|
|
97
98
|
in the daily digest) is
|
|
98
|
-
[issue #
|
|
99
|
+
[issue #11](https://github.com/TerrifiedBug/conductor/issues/11).
|
|
99
100
|
|
|
100
101
|
Changing the key later does not rewrite an `ORCHESTRATOR.md` you already have:
|
|
101
102
|
the tick line changes, the brief does not. Edit its Reporting section too, or the
|
|
@@ -119,7 +120,7 @@ and create any missing labels):
|
|
|
119
120
|
| Key | Meaning |
|
|
120
121
|
| --- | --- |
|
|
121
122
|
| `tracker.repo` | The **one** `owner/repo` whose issue list is the queue. This is your planning repo — it does not have to contain any code. |
|
|
122
|
-
| `queueLabel` | Open issues in `tracker.repo` carrying this label
|
|
123
|
+
| `queueLabel` | Open issues in `tracker.repo` carrying this label are the work queue. Nothing else is ever read. Required: the wizard pre-fills `ready-for-agent`, but a config that omits the key is rejected, not defaulted. |
|
|
123
124
|
| `routing.repos` + `repo:<name>` labels | Each queued issue must also carry exactly one routing label naming which code repo the work lands in. The conductor cuts the worktree and PR there, from `routing.repos[name].cloneUrl`. An issue with zero or two routing labels is reported as unroutable and skipped — never guessed. |
|
|
124
125
|
|
|
125
126
|
So: one tracker repo supplies the queue, routing labels fan issues out to any
|
|
@@ -182,7 +183,9 @@ So the skill does the part a dialog cannot:
|
|
|
182
183
|
- **Interviews you** on release policy — humans release (the default), the agent
|
|
183
184
|
releases to a named boundary, or the agent releases fully — pressing on the one
|
|
184
185
|
question that makes a delegated release safe: *where does the agent's leg end?*
|
|
185
|
-
|
|
186
|
+
The answer becomes the [`authority`](#configuration) pair `/conductor setup`
|
|
187
|
+
records, and the brief is worded from it. Plus escalation taste, and which of
|
|
188
|
+
the two [`reporting.scope`](#your-workflow-vs-the-package)
|
|
186
189
|
values your answer actually maps to.
|
|
187
190
|
- **Reads your repos instead of asking about them.** It opens each routing repo's
|
|
188
191
|
CI workflows, `package.json` scripts and `Makefile`/`justfile`, then *proposes*
|
|
@@ -326,7 +329,18 @@ Per tick, for the daemon's project:
|
|
|
326
329
|
an active run — including a green PR, so a second attempt can never land on a
|
|
327
330
|
live PR. An issue that has used `maxAttemptsPerIssue` escalates at Tier 1
|
|
328
331
|
instead of being admitted.
|
|
329
|
-
8. **
|
|
332
|
+
8. **Ask the tracker whether the work already exists.** For each candidate that
|
|
333
|
+
survived step 7 — so at most one API call per free slot, never one per queued
|
|
334
|
+
issue — the daemon asks whether an **open** PR already closes the issue. If one
|
|
335
|
+
does, the issue is skipped with its PR named in the log. Drafts count: a draft
|
|
336
|
+
PR's branch still holds the only copy of the work. This is the guard the store
|
|
337
|
+
cannot provide, because a store younger than the PRs (a migration, a wiped or
|
|
338
|
+
relocated state directory, a restore onto a new host) has no row to object
|
|
339
|
+
with. If the check itself fails, the candidate is **held**, not admitted, and
|
|
340
|
+
retried next tick: the cost of holding is five minutes, the cost of admitting
|
|
341
|
+
on an unknown is a burned attempt and a duplicate PR. Only that candidate is
|
|
342
|
+
held, so a flaky API cannot stall the rest of the queue.
|
|
343
|
+
9. **Dispatch** the admitted issues concurrently.
|
|
330
344
|
|
|
331
345
|
Then, per admitted issue:
|
|
332
346
|
|
|
@@ -351,13 +365,26 @@ Then, per admitted issue:
|
|
|
351
365
|
| --- | --- | --- | --- |
|
|
352
366
|
| `pushed-green` | `agent:in-progress` stays until the merge closes the issue | removed | none |
|
|
353
367
|
| `blocked` | swapped to `agent:blocked` | removed | Tier 1 |
|
|
354
|
-
| `failed` / `killed` | swapped to `agent:failed` | **kept** as evidence | Tier 1 |
|
|
355
|
-
| unexpected error | swapped to `agent:failed` |
|
|
368
|
+
| `failed` / `killed` | swapped to `agent:failed` | dirty tree committed to the branch, then **kept** as evidence | Tier 1 |
|
|
369
|
+
| unexpected error | swapped to `agent:failed` | same | Tier 1 |
|
|
356
370
|
|
|
357
371
|
Label swaps add the new label before removing the old one: the reverse order
|
|
358
372
|
leaves a window in which the issue carries no state label at all, which is
|
|
359
373
|
exactly the shape eligibility reads as fresh work.
|
|
360
374
|
|
|
375
|
+
**A non-graceful end salvages the tree first.** A turns-cap kill, a
|
|
376
|
+
wall-clock kill and a crash are all external and unannounced: they land
|
|
377
|
+
mid-edit, and only the run's *branch* is preserved across attempts — the tree
|
|
378
|
+
is removed `--force` by the next one. So before the escalation is written,
|
|
379
|
+
a dirty tree is committed to the run's own branch as
|
|
380
|
+
`wip(#<issue>): attempt <n> killed by <reason> — auto-salvaged` (everything,
|
|
381
|
+
including files git has never seen) and pushed, and the escalation says where
|
|
382
|
+
it went: `WIP committed to <branch> @ <sha>`. A push that is refused leaves
|
|
383
|
+
the commit in this host's mirror and says so; a salvage that fails outright
|
|
384
|
+
says that, loudly, naming the tree that now holds the only copy. A `blocked`
|
|
385
|
+
run is deliberately *not* salvaged — it stopped on purpose, with turns still
|
|
386
|
+
in hand to commit for itself.
|
|
387
|
+
|
|
361
388
|
### What a restart does to runs that were in flight
|
|
362
389
|
|
|
363
390
|
A `claimed` or `running` row is a promise that a worker process exists, and a
|
|
@@ -464,7 +491,7 @@ otherwise looks like a run that was merely unlucky.
|
|
|
464
491
|
| Tier | Meaning | Raised by | Delivered to |
|
|
465
492
|
| --- | --- | --- | --- |
|
|
466
493
|
| 1 | "Not a human's problem yet" — the run is parked and safe. | Unroutable issue, blocked run, failed or killed run, dispatch error, attempts exhausted. | The orchestrator session, as an injected prompt. Falls back to an issue comment when no orchestrator is running, or when it will not accept the injection. |
|
|
467
|
-
| 2 | "The fleet is stopped until you look." | Daily spend cap reached; the project is already paused. | Telegram, when `escalation.telegramChatId` is set and a bot token is readable; otherwise it falls back to the issue comment. |
|
|
494
|
+
| 2 | "The fleet is stopped until you look." | Daily spend cap reached; the installed package changed under the running daemon. Either way the project is already paused. | Telegram, when `escalation.telegramChatId` is set and a bot token is readable; otherwise it falls back to the issue comment. |
|
|
468
495
|
|
|
469
496
|
**The orchestrator** is one persistent, file-backed session per daemon run, resumed
|
|
470
497
|
across restarts so it remembers what it has already handled. Its `cwd` is the state
|
|
@@ -472,9 +499,27 @@ directory, deliberately not a checkout. Delivery resolves when the harness *acce
|
|
|
472
499
|
the prompt, not when the model answers it, so a tick never parks behind a model; an
|
|
473
500
|
injection arriving mid-thought queues as a follow-up instead of interrupting the
|
|
474
501
|
turn in flight. Its standing orders are explicit: re-brief the worker, file or
|
|
475
|
-
comment on issues, or promote to tier 2, and never edit product code
|
|
476
|
-
|
|
477
|
-
|
|
502
|
+
comment on issues, or promote to tier 2, and never edit product code or push a
|
|
503
|
+
branch. Merging is the one line worded from config — see [`authority`](#configuration).
|
|
504
|
+
If it fails to start, the daemon logs a warning and runs on, with tier-1
|
|
505
|
+
escalations degraded to issue comments.
|
|
506
|
+
|
|
507
|
+
**Or no orchestrator at all.** Set `escalation.orchestrator` to `"external"` when
|
|
508
|
+
you already run your own supervising session — a visible TUI session in a pane,
|
|
509
|
+
typically. The daemon then starts none of its own and every tier-1 escalation
|
|
510
|
+
posts as an issue comment, which is what that session drains. One brain, and it
|
|
511
|
+
is the one you can watch.
|
|
512
|
+
|
|
513
|
+
**Answering a tier 1 is only half of it.** A blocked or failed run leaves its state
|
|
514
|
+
label on the issue, and eligibility reads any state label as disqualifying, so an
|
|
515
|
+
answered issue that keeps one is never re-claimed and the answer is inert — nothing
|
|
516
|
+
fails, the issue just stops existing as far as dispatch is concerned.
|
|
517
|
+
[`omp-conductor unblock <issue>`](#cli-reference) is the way back: it clears the
|
|
518
|
+
label through the same tracker the dispatcher writes with. The brief tells the
|
|
519
|
+
orchestrator to run that verb rather than edit the label itself, and that is not a
|
|
520
|
+
formality — orphan detection works by comparing `agent:in-progress` labels against
|
|
521
|
+
live runs, and it is only trustworthy while every state label on the tracker was
|
|
522
|
+
written by this package.
|
|
478
523
|
|
|
479
524
|
Tier 2 borrows the bot token that `omp-telegram` already owns, at
|
|
480
525
|
`~/.omp/agent/telegram/.env` (or `$OMP_TELEGRAM_STATE_DIR/.env`). If you run that
|
|
@@ -488,8 +533,8 @@ issue on every poll, so a ledger in the store — keyed by project, issue, tier
|
|
|
488
533
|
summary — makes a recurring condition page **once** and suppresses the five-minute
|
|
489
534
|
repeats. The marker is recorded only on successful delivery, so a page that could
|
|
490
535
|
not be delivered is retried on the next tick instead of being written off as sent.
|
|
491
|
-
The spend-cap
|
|
492
|
-
only once per day.
|
|
536
|
+
The spend-cap and integrity-tripwire summaries carry the date, so the same
|
|
537
|
+
condition pages again tomorrow but only once per day.
|
|
493
538
|
|
|
494
539
|
If `fallbackToIssueComment` is off and no Telegram transport is configured,
|
|
495
540
|
delivery throws instead of dropping silently. The failure is logged and retried,
|
|
@@ -508,7 +553,10 @@ Runtime state lives elsewhere, under `$OMP_CONDUCTOR_RUNTIME_DIR` (default
|
|
|
508
553
|
atomically, and `daemon.log`, appended across every boot so the previous failure is
|
|
509
554
|
still there when you go looking. It is kept apart from the config directory because
|
|
510
555
|
it is meaningless after a reboot, and the pidfile's liveness is probed on every
|
|
511
|
-
read — a stale one never blocks a `start`.
|
|
556
|
+
read — a stale one never blocks a `start`. Both `start` and a bare `daemon` write
|
|
557
|
+
the pidfile, so a daemon run in the foreground under systemd is as visible to
|
|
558
|
+
`status` as a backgrounded one; `daemon --once` writes nothing, because that drill
|
|
559
|
+
is exactly what the orphan-reconciliation guard reads the pidfile to protect.
|
|
512
560
|
|
|
513
561
|
The file is validated on every read. A malformed config produces one readable error
|
|
514
562
|
listing every fault, and the daemon refuses to start rather than running with half
|
|
@@ -572,7 +620,12 @@ A complete, valid config for one project with two target repos:
|
|
|
572
620
|
"workerModel": "smol",
|
|
573
621
|
"escalation": {
|
|
574
622
|
"telegramChatId": "123456789",
|
|
575
|
-
"fallbackToIssueComment": true
|
|
623
|
+
"fallbackToIssueComment": true,
|
|
624
|
+
"orchestrator": "embedded"
|
|
625
|
+
},
|
|
626
|
+
"authority": {
|
|
627
|
+
"merge": "human",
|
|
628
|
+
"release": "human"
|
|
576
629
|
},
|
|
577
630
|
"reporting": {
|
|
578
631
|
"scope": "material"
|
|
@@ -598,6 +651,8 @@ Field notes:
|
|
|
598
651
|
| `gates` | The exact cheap commands CI also runs, each with the `cwd` it runs from (`cwd` defaults to `.`). Running the real gate locally is what makes an unattended push safe — a subset lets an error outside the source dir reach the runners. |
|
|
599
652
|
| `caps` | Per-project overrides; omit it or pin only the fields you want to change. |
|
|
600
653
|
| `escalation.fallbackToIssueComment` | Defaults to `true`. Absent means "yes, still tell me". |
|
|
654
|
+
| `escalation.orchestrator` | Optional; `"embedded"` (default) or `"external"`. `external` means an orchestrator session already runs elsewhere: the daemon starts none, and tier-1 escalations post as issue comments for that session to drain. Any other value is an error. |
|
|
655
|
+
| `authority` | Optional; `{ "merge": …, "release": … }`, each `"human"` (default) or `"orchestrator"`. It grants nothing to the daemon — it words the orchestrator's standing orders and the Releases paragraph of the rendered brief, so the config and the prompt cannot disagree about who holds the merge button. Unknown keys and any other value are errors, never folded to the default. |
|
|
601
656
|
| `reporting.scope` | Optional; `"material"` (default) or `"escalations"`. Every orchestrator tick appends the matching constraint line to its prompt, re-read from this file each tick — see [Your workflow vs. the package](#your-workflow-vs-the-package). It constrains what the session is told to report; it is not an outbound filter. A config written without the key keeps reporting material events. Any other value is an error, never folded to the default. |
|
|
602
657
|
| `workspaceRoot` / `mirrorRoot` | Optional; default to `worktrees/` and `mirrors/` under the state directory. `~` is expanded. |
|
|
603
658
|
|
|
@@ -614,13 +669,13 @@ it never gets prompted, so it never runs anything. Installing
|
|
|
614
669
|
|
|
615
670
|
The heartbeat is **inert unless the session's cwd contains
|
|
616
671
|
`.conductor-tick.json`**, so it costs an ordinary session nothing. Drop the file
|
|
617
|
-
in the orchestrator's working directory
|
|
672
|
+
in the orchestrator's working directory:
|
|
618
673
|
|
|
619
674
|
```json
|
|
620
675
|
{
|
|
621
676
|
"intervalSeconds": 900,
|
|
622
677
|
"armedFile": "state/armed",
|
|
623
|
-
"accessFile": "/
|
|
678
|
+
"accessFile": "/home/fleet/.omp/agent/telegram/access.json",
|
|
624
679
|
"message": "Run your standing loop from ORCHESTRATOR.md now."
|
|
625
680
|
}
|
|
626
681
|
```
|
|
@@ -630,13 +685,18 @@ in the orchestrator's working directory (on the fleet host, `/root/fleet`):
|
|
|
630
685
|
| `intervalSeconds` | yes | — | Whole seconds between ticks, minimum `60`. A tick costs a full turn of a frontier model, so a sub-minute period is refused rather than obeyed. |
|
|
631
686
|
| `armedFile` | no | none — the gate passes | Path to the arm marker. A tick does nothing while the file is missing. Relative paths resolve against the session cwd, so `state/armed` means `<cwd>/state/armed`. |
|
|
632
687
|
| `accessFile` | no | none — the gate passes | Path to the Telegram bridge's `access.json`. Every tick re-reads it and requires `enabled: true` with exactly one entry in `allowFrom`. Relative paths resolve against the session cwd. **Configure this on any fleet deploy** — see below. |
|
|
633
|
-
| `message` | no | `Tick <ISO timestamp>: run your standing loop from
|
|
688
|
+
| `message` | no | `Tick <ISO timestamp>: re-read <workspaceRoot>/ORCHESTRATOR.md from disk, then run your standing loop from it.`, then the `reporting.scope` line, then the delivery rule | Sent verbatim when set — and then it owns the whole contract: neither the scope line nor the delivery rule is appended to a prompt you wrote yourself. Re-read from disk on **every** tick, so rewording it binds the next heartbeat instead of waiting for a session restart; a re-read that fails — caught mid-edit, removed, or invalid — keeps the value read at session start rather than stopping the heartbeat. `intervalSeconds` is *not* re-read: rescheduling a live timer still needs a restart. The default *orders* the session to re-read its brief, naming the path resolved from the conductor config — a 24/7 session otherwise acts on the copy loaded at its start (resume included), and an amendment added between ticks never binds — and carries the timestamp, which makes two consecutive ticks distinguishable in the session log. |
|
|
634
689
|
|
|
635
690
|
A tick sends one message (`customType` `omp-conductor.tick`, attributed to the
|
|
636
|
-
user): the standing-loop prompt,
|
|
691
|
+
user): the standing-loop prompt, the one constraint line the project's
|
|
637
692
|
[`reporting.scope`](#your-workflow-vs-the-package) resolves to, re-read from the
|
|
638
|
-
conductor config on every tick
|
|
639
|
-
turn
|
|
693
|
+
conductor config on every tick, and a delivery rule. That last line is there
|
|
694
|
+
because end-of-turn text reaches the operator's Telegram only on a turn that
|
|
695
|
+
*began* as an inbound Telegram message: a tick is injected locally, so anything
|
|
696
|
+
the session merely writes at the end of one is read by nobody, and a reportable
|
|
697
|
+
event has to be delivered by an explicit `telegram_send` call the session
|
|
698
|
+
watched succeed. The tick starts a turn if the session is idle; while a turn is
|
|
699
|
+
streaming it is queued as a follow-up and consumed when that turn ends.
|
|
640
700
|
It sends **nothing** when:
|
|
641
701
|
|
|
642
702
|
- `/conductor pause` (or `omp-conductor pause`) holds the pause flag, the same
|
|
@@ -644,7 +704,9 @@ It sends **nothing** when:
|
|
|
644
704
|
- `armedFile` is configured and missing;
|
|
645
705
|
- `accessFile` is configured and the escalation channel is not verifiably up;
|
|
646
706
|
- an earlier tick is still queued. Ticks coalesce rather than stack, so a slow
|
|
647
|
-
turn cannot leave a backlog of heartbeats behind it
|
|
707
|
+
turn cannot leave a backlog of heartbeats behind it — and two coalesced ticks
|
|
708
|
+
in a row are the signal that the session is not slow but wedged, which is
|
|
709
|
+
what the [stall marker](#a-wedged-session-and-the-marker-that-notices) is for.
|
|
648
710
|
|
|
649
711
|
### The escalation channel is a gate, and it fails closed
|
|
650
712
|
|
|
@@ -666,16 +728,91 @@ Leaving `accessFile` unset passes the gate, because an ordinary developer sessio
|
|
|
666
728
|
that happens to have a `.conductor-tick.json` has no bridge to check. It is not an
|
|
667
729
|
off switch for the check: **a fleet deploy always sets it.**
|
|
668
730
|
|
|
669
|
-
Every tick — sent or skipped — is logged with its reason (`
|
|
670
|
-
`escalation channel down`, `tick already pending`) to the omp log.
|
|
671
|
-
deliberately
|
|
672
|
-
|
|
731
|
+
Every tick — sent or skipped — is logged with its reason (`not armed`,
|
|
732
|
+
`escalation channel down`, `tick already pending`) to the omp log. `/conductor
|
|
733
|
+
pause` is deliberately **not** one of the gates: pause stops the *dispatcher*
|
|
734
|
+
claiming work, and the tick drives a different session — one whose duties
|
|
735
|
+
(grooming the queue, draining escalations, reporting) are exactly what stays
|
|
736
|
+
useful while dispatch is stopped. Its own off switch is the arm marker. Skips
|
|
737
|
+
are deliberately silent in the UI: a disarmed fleet would otherwise raise a
|
|
738
|
+
notification every interval, forever. The one exception is a malformed
|
|
739
|
+
`.conductor-tick.json`,
|
|
673
740
|
which notifies once at session start and leaves the heartbeat off; silent failure
|
|
674
741
|
there is the failure mode the heartbeat exists to prevent. A conductor config that
|
|
675
742
|
cannot supply a reporting scope logs `tick reporting scope: using material` once
|
|
676
743
|
per session. The interval does not re-log it, because the file is unlikely to fix
|
|
677
744
|
itself between two ticks.
|
|
678
745
|
|
|
746
|
+
### A wedged session, and the marker that notices
|
|
747
|
+
|
|
748
|
+
Coalescing is also the only wedge detector this package has. On 2026-08-07 the
|
|
749
|
+
dogfood fleet's orchestrator finished a turn, logged `ui.loop-blocked` right
|
|
750
|
+
after an auto-compaction threshold decision, and never started another. The
|
|
751
|
+
process stayed alive, so herdr's recovery — agent listed AND a non-shell
|
|
752
|
+
foreground process — read healthy. The dispatch daemon is a separate process and
|
|
753
|
+
kept working, so `/healthz` was green all night, while the one brain holding
|
|
754
|
+
merge authority sat on a green PR it never merged. A tick injected two minutes
|
|
755
|
+
into the wedge and an operator's Telegram message five minutes later both went
|
|
756
|
+
unconsumed for 23 minutes, until a manual `SIGTERM`. The heartbeat logged `tick
|
|
757
|
+
skipped: tick already pending` throughout, which is exactly what a merely slow
|
|
758
|
+
turn looks like.
|
|
759
|
+
|
|
760
|
+
So the heartbeat counts them. Two consecutive coalesced ticks — a full hour at
|
|
761
|
+
the reference 1800-second interval, generous by construction — mean the last
|
|
762
|
+
prompt was never consumed, and the extension:
|
|
763
|
+
|
|
764
|
+
- writes `<session cwd>/.conductor-stalled`, one line of `<ISO timestamp>
|
|
765
|
+
<diagnosis>`.
|
|
766
|
+
- logs at **error** level: `orchestrator stalled: 2 ticks queued unconsumed —
|
|
767
|
+
the agent loop is not draining; see .conductor-stalled`.
|
|
768
|
+
|
|
769
|
+
Both escapes deliberately leave the session, because a loop that cannot drain
|
|
770
|
+
its queue cannot report on itself — that is the whole failure.
|
|
771
|
+
|
|
772
|
+
**The daemon reads it.** A marker nobody consumes is an artifact, not an alert,
|
|
773
|
+
so the dispatch daemon checks it on its own five-minute tick — and *before* its
|
|
774
|
+
pause check. The orchestrator is a different process and can be wedged while
|
|
775
|
+
the fleet is deliberately paused, which is precisely the state the dogfood
|
|
776
|
+
fleet was in when this happened. One tier-2 page per stall, keyed on the
|
|
777
|
+
marker's own timestamp so a second wedge the same day is not swallowed as a
|
|
778
|
+
repeat, re-armed when the marker clears, and latched only once the page is
|
|
779
|
+
confirmed delivered — an escalation channel that fails on the one tick that
|
|
780
|
+
noticed must not buy permanent silence.
|
|
781
|
+
|
|
782
|
+
It restarts nothing. A wedge lands mid-turn, and no other process can tell a
|
|
783
|
+
half-applied edit from an idle loop; the operator attaches, looks, and decides.
|
|
784
|
+
|
|
785
|
+
**herdr-conductor deliberately does not read it**, though its liveness test
|
|
786
|
+
(agent listed AND a non-shell foreground process) passes straight through a
|
|
787
|
+
wedge. That plugin only runs on `startup`, `pane.exited` and
|
|
788
|
+
`pane.agent_detected`, and a session that stays alive and stops working emits
|
|
789
|
+
none of them — so the check could never fire during the wedge itself. What it
|
|
790
|
+
*would* catch is the recovery afterwards: the marker survives a restart until
|
|
791
|
+
the new session consumes a tick, so every operator SIGTERM-and-resume would
|
|
792
|
+
page about the healthy session they just fixed. Telling those apart needs the
|
|
793
|
+
process start time against the marker's, and herdr's `pane process-info`
|
|
794
|
+
reports pids, not start times. The daemon gives up at most one tick of
|
|
795
|
+
coverage and never cries wolf.
|
|
796
|
+
|
|
797
|
+
The first tick that actually sends clears the counter and deletes the marker,
|
|
798
|
+
and it deletes one it did not write: recovery normally arrives as a fresh
|
|
799
|
+
process resuming the same transcript, so the session doing the clearing is not
|
|
800
|
+
the session that stalled. Nothing else removes the file. Neither the write nor
|
|
801
|
+
the delete can take the heartbeat down — a filesystem error is logged and the
|
|
802
|
+
tick carries on.
|
|
803
|
+
|
|
804
|
+
`omp-conductor status` reads the same marker from the **state directory** and
|
|
805
|
+
prints one more line under the daemon block:
|
|
806
|
+
|
|
807
|
+
```text
|
|
808
|
+
orchestrator STALLED since 2026-08-07T06:27:55.123Z — 2 ticks queued unconsumed — the agent loop is not draining
|
|
809
|
+
```
|
|
810
|
+
|
|
811
|
+
That reading is the reference deploy's convention — the orchestrator session
|
|
812
|
+
runs from `~/.omp/conductor`, which is the state directory — and it is
|
|
813
|
+
one-directional: a line there proves a wedge, and its absence proves nothing,
|
|
814
|
+
least of all on a fleet whose session lives somewhere else.
|
|
815
|
+
|
|
679
816
|
## CLI reference
|
|
680
817
|
|
|
681
818
|
```bash
|
|
@@ -683,6 +820,8 @@ omp-conductor start [--port N] [--project NAME]
|
|
|
683
820
|
omp-conductor stop
|
|
684
821
|
omp-conductor restart [--port N] [--project NAME]
|
|
685
822
|
omp-conductor status [--project NAME]
|
|
823
|
+
omp-conductor tail <issue> [--project NAME]
|
|
824
|
+
omp-conductor unblock <issue> [--project NAME]
|
|
686
825
|
omp-conductor daemon [--once] [--port N] [--project NAME]
|
|
687
826
|
omp-conductor pause
|
|
688
827
|
omp-conductor resume
|
|
@@ -695,12 +834,14 @@ omp-conductor help
|
|
|
695
834
|
| `start` | Spawn the loop in the background, detached, and wait until it answers `GET /healthz` on `:8787`. Refuses if one is already live, naming its pid. If the process dies or never serves, `start` cleans up after it and quotes the tail of `daemon.log`. |
|
|
696
835
|
| `stop` | `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop. |
|
|
697
836
|
| `restart` | `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them — a restart that quietly moved to the default port would leave every existing health check pointing at nothing. |
|
|
698
|
-
| `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. Reads while a daemon in another process writes. |
|
|
699
|
-
| `
|
|
700
|
-
| `
|
|
837
|
+
| `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. A `.conductor-stalled` marker in the state directory adds an `orchestrator STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
|
|
838
|
+
| `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
|
|
839
|
+
| `unblock <issue>` | Remove that issue's `blocked` and `failed` state labels through the tracker, so the next tick can claim it again. This is the supported way back for an escalation you answered: eligibility disqualifies any issue carrying a state label, so an answered issue that keeps one is never re-claimed and the answer is inert. Removing a label the issue does not carry is a no-op, so both are always cleared and neither has to be looked up first. `agent:in-progress` is deliberately not touched — it means a worker process exists, which is not something an answer changes. The run history is left exactly as it is: an answered block still spent a worker, so it still counts toward `maxAttemptsPerIssue`, and the output says how many attempts remain — or warns that the next tick will escalate instead of dispatching, when none do. Exits `2` with `unblock needs an issue number` on a missing or malformed positional. |
|
|
840
|
+
| `daemon` | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. This is what `start` launches, and what a systemd unit should call. Writes the pidfile itself, and refuses with `another daemon is alive (pid N); stop it first` rather than becoming a second dispatcher. |
|
|
841
|
+
| `daemon --once` | Run a single tick and exit. No HTTP server, and no pidfile — a drill must not register itself as the daemon, or the next reader believes it and the real daemon's in-flight runs get reconciled as orphans. |
|
|
701
842
|
| `--port N` | Accepted by `start`, `restart` and `daemon`. Both `--port 9000` and `--port=9000` work; missing or out of range exits `2` rather than falling back to the default, because probing the wrong endpoint is worse than a hard failure. |
|
|
702
843
|
| `--project NAME` | Pick the project to service. One daemon process serves exactly one project; with several configured projects the name is required. |
|
|
703
|
-
| `pause` | Stop claiming new work. The running daemon notices on its next tick; runs already in flight finish. |
|
|
844
|
+
| `pause` | Stop claiming new work. The running daemon notices on its next tick; runs already in flight finish. The orchestrator heartbeat keeps ticking — its gate is the arm marker, not this flag. |
|
|
704
845
|
| `resume` | Allow claiming again. |
|
|
705
846
|
| `brief-upgrade` | Compare a project's `ORCHESTRATOR.md` against the brief this version of the package ships. Reports by default; see [Keeping a brief current](#keeping-a-brief-current). |
|
|
706
847
|
| `--apply` | Only for `brief-upgrade`. Replaces the half above the `YOURS TO EDIT` banner and keeps everything below it, backing the previous file up first. Ignored when the brief cannot be split or the template is unrendered. |
|
|
@@ -741,13 +882,42 @@ dispatcher. The brief is explicit about the boundary:
|
|
|
741
882
|
| Add or update tests for behaviour it introduced. | Suppress a warning, delete an assertion, or special-case an input to make a check pass. |
|
|
742
883
|
| Run the repo's configured cheap gates, each from its listed `cwd`, over the whole tree. | Run docker or image builds, production builds, browser/e2e suites, or the full test suite on the shared host — CI owns the heavy gates. |
|
|
743
884
|
| Review its whole diff, then commit and **push once**. One corrective push if CI is red. | Force-push, `git add -f`, or add AI/co-author attribution. Red twice means stop and report, not push a third time. |
|
|
744
|
-
| Open a PR that links the issue, and watch CI to a verdict with `gh pr checks --watch`. | Run `gh pr merge`. **
|
|
885
|
+
| Open a PR that links the issue, and watch CI to a verdict with `gh pr checks --watch`. | Run `gh pr merge`. **A worker never merges** — that one is absolute, whoever else holds the authority — so PRs land one at a time with a freshness re-check; two workers merging concurrently is how agent PRs clobber each other. Who *may* merge is the [`authority`](#configuration) answer, and it is never the worker. |
|
|
745
886
|
| Escalate: ambiguity, a cross-repo contract, a needed credential, a product or data-migration decision, a blocking existing test, CI red twice, or most of the wall-clock budget burned. | **Cut a release**, push a tag, publish to npm, edit a deployment pin, deploy, or touch infrastructure or secrets — permanently out of scope. Releases are batched and decided outside this loop, so "this needs releasing" is a thing to report, never a task to take on. |
|
|
746
887
|
|
|
747
888
|
The worker ends with a six-line evidence report (issue, pr, state, gates, changed,
|
|
748
889
|
next). `pushed-green` means it watched the checks go green rather than expecting
|
|
749
890
|
them to.
|
|
750
891
|
|
|
892
|
+
### The boundary is prompt text — and one tripwire
|
|
893
|
+
|
|
894
|
+
A worker session gets a `cwd` and nothing else: the harness applies no filesystem
|
|
895
|
+
or network restriction, so "touch any path outside its worktree" is a rule the
|
|
896
|
+
session is asked to keep, not one the process cannot break. Target selection *is*
|
|
897
|
+
mechanical — only a repo in `routing.repos` is ever checked out — and so are the
|
|
898
|
+
caps, but between them they bound where work starts and how much of it happens,
|
|
899
|
+
not how far a wandering or prompt-injected session can reach.
|
|
900
|
+
|
|
901
|
+
The one path that is checked mechanically is the conductor itself. At startup the
|
|
902
|
+
daemon sha256s every `.ts` and `.md` file of its own installed `src/` — the
|
|
903
|
+
dispatcher and the briefs both, since rewriting a brief buys more than rewriting
|
|
904
|
+
the loop — and re-walks that tree on every tick (about 0.6 ms). Any difference at
|
|
905
|
+
all, changed or added or removed, is read as the package having been modified
|
|
906
|
+
underneath a running daemon: the tick claims nothing, the fleet is paused, and a
|
|
907
|
+
tier-2 escalation naming the first few differing paths pages you **once**, not
|
|
908
|
+
every five minutes.
|
|
909
|
+
|
|
910
|
+
**A normal deploy never trips it.** The baseline is recorded per daemon process,
|
|
911
|
+
so installing a new build and restarting the unit re-records it from the new
|
|
912
|
+
files; only a change that lands *while* a daemon is holding the package open can
|
|
913
|
+
diverge from it. That also means `omp-conductor resume` on its own will not hold
|
|
914
|
+
— the next tick re-walks, still differs, and pauses again. Put the files back, or
|
|
915
|
+
restart onto the build you meant to be running.
|
|
916
|
+
|
|
917
|
+
This is detection, not prevention. It catches the worker that wandered, and it
|
|
918
|
+
catches the human who edited the live install "just to test something", which in
|
|
919
|
+
practice is the commoner of the two.
|
|
920
|
+
|
|
751
921
|
## Limitations
|
|
752
922
|
|
|
753
923
|
Known and deliberate in this version:
|
|
@@ -790,12 +960,19 @@ Known and deliberate in this version:
|
|
|
790
960
|
(the orchestrator's) no matter how many workers are running, and no amount of
|
|
791
961
|
`maxConcurrentWorkers` changes that.
|
|
792
962
|
|
|
793
|
-
The cap does work. The admission loop (`src/daemon.ts
|
|
963
|
+
The cap does work. The admission loop (`admitCandidates` in `src/daemon.ts`) computes
|
|
794
964
|
`slots = maxConcurrentWorkers - live workers`, admits at most that many issues
|
|
795
965
|
per tick, and dispatches them together. To see them, read `omp-conductor
|
|
796
966
|
status`, which lists every occupied issue, or follow `daemon.log`.
|
|
797
967
|
- **Merges, releases and deploys are human-only, by design.** The conductor
|
|
798
968
|
produces green PRs and stops.
|
|
969
|
+
- **Worker confinement is behavioural.** `runWorker` hands the session a `cwd` and
|
|
970
|
+
a brief; nothing stops it reading or writing elsewhere on the host, and on a
|
|
971
|
+
single-user deploy that includes the state directory and the host's `gh`
|
|
972
|
+
credentials. The [integrity tripwire](#the-boundary-is-prompt-text--and-one-tripwire)
|
|
973
|
+
turns one case of this into a paused, paged fleet after the fact; the prevention
|
|
974
|
+
half is a deployment concern — run workers as a least-privileged uid whose write
|
|
975
|
+
access ends at its worktree and mirror.
|
|
799
976
|
|
|
800
977
|
## License
|
|
801
978
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "omp-conductor",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.2",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
|
|
@@ -119,16 +119,25 @@ are the orchestrator and a human.** The orchestrator is the right agent for it i
|
|
|
119
119
|
any agent is: it is long-lived, it can see everything that merged since the last
|
|
120
120
|
release, and batching is exactly the judgement a per-issue session cannot make.
|
|
121
121
|
|
|
122
|
-
So frame it as **three** options, in this order, and name the default
|
|
122
|
+
So frame it as **three** options, in this order, and name the default. Each one
|
|
123
|
+
maps to an `authority` answer the wizard asks for directly — the interview
|
|
124
|
+
decides *which answer to give*, and `/conductor setup` is what records it. Never
|
|
125
|
+
write authority into the brief by hand: the config is what words the brief, and
|
|
126
|
+
a hand-edit is a second source that outlives the operator's memory of making it.
|
|
123
127
|
|
|
124
128
|
1. **Humans release.** *(default, and what the package ships)* Work ends at a
|
|
125
129
|
green PR. Merging is a separate human action; releasing is a separate human
|
|
126
130
|
action after that. Neither a worker nor the orchestrator ever tags, pins,
|
|
127
131
|
publishes, or deploys. "This needs releasing" becomes something the
|
|
128
|
-
orchestrator *reports*.
|
|
132
|
+
orchestrator *reports*. → `authority: merge=human, release=human`; answer
|
|
133
|
+
**no** to both wizard confirms.
|
|
129
134
|
2. **The orchestrator releases up to a named boundary.** Delegating part of the
|
|
130
135
|
release to the supervising session, with the stopping line written down.
|
|
131
|
-
|
|
136
|
+
→ usually `merge=orchestrator, release=orchestrator`, with the boundary in
|
|
137
|
+
the procedure rather than in the grant; `merge=orchestrator, release=human`
|
|
138
|
+
when the line falls before the tag.
|
|
139
|
+
3. **The orchestrator releases fully.** → `merge=orchestrator,
|
|
140
|
+
release=orchestrator`.
|
|
132
141
|
|
|
133
142
|
Then ask the question that makes option 2 real:
|
|
134
143
|
|
|
@@ -138,8 +147,8 @@ Not "can it release" — *where does its leg stop*. A boundary you cannot state
|
|
|
138
147
|
one sentence is not a boundary, and an orchestrator with a vague release mandate
|
|
139
148
|
is one that eventually publishes something at 03:00.
|
|
140
149
|
|
|
141
|
-
Give them a worked answer so they can calibrate.
|
|
142
|
-
**the agent's leg ends at the merged
|
|
150
|
+
Give them a worked answer so they can calibrate. One real deployment's answer:
|
|
151
|
+
**the agent's leg ends at the merged version pin.** It may open and land the PR that
|
|
143
152
|
bumps the module image tags, because that artefact is reviewable, reversible by a
|
|
144
153
|
revert, and its correctness is checkable by a named CI check. Deploying that pin
|
|
145
154
|
to prod is **operator territory** — it needs a person who can watch it, and who
|
|
@@ -344,7 +353,8 @@ about gates: an operator describes the release they *remember*, and a release is
|
|
|
344
353
|
the one procedure where being approximately right is worst.
|
|
345
354
|
|
|
346
355
|
Skip this step only for option 1 (humans release). There is nothing to scaffold:
|
|
347
|
-
the
|
|
356
|
+
the rendered paragraph already says humans hold both, and there is no procedure
|
|
357
|
+
under it to write.
|
|
348
358
|
|
|
349
359
|
### Find the release authority
|
|
350
360
|
|
|
@@ -423,7 +433,8 @@ has a safe fleet. Your job is to raise it to *this* fleet.
|
|
|
423
433
|
The file has a hard line in it — an HTML comment banner reading
|
|
424
434
|
`YOURS TO EDIT`. Respect it in both directions:
|
|
425
435
|
|
|
426
|
-
**Above the banner — leave it alone.**
|
|
436
|
+
**Above the banner — leave it alone.** The three duties — drain (unstick what is
|
|
437
|
+
stuck), groom (keep the queue worth draining), report — the
|
|
427
438
|
escalation-tier table, and the hard boundaries describe how the package already
|
|
428
439
|
behaves. Rewriting them makes the brief disagree with the code, and the code wins.
|
|
429
440
|
The evidence rule in particular is not negotiable: *every claim cites evidence — a
|
|
@@ -432,24 +443,24 @@ PR URL, an issue number, or a named check actually read.* "Should be fine",
|
|
|
432
443
|
|
|
433
444
|
Know what is *not* up there, though, because operators expect it to be: the
|
|
434
445
|
orchestrator's own merge and release authority is **not** a hard boundary. It is
|
|
435
|
-
|
|
436
|
-
banner is that a *worker* never merges or releases, and that PRs land
|
|
437
|
-
with a freshness re-check. Delegating a release to the
|
|
438
|
-
either of those, so it needs no negotiation with the
|
|
446
|
+
config — the two `authority` answers — and it defaults to none. What is fixed
|
|
447
|
+
above the banner is that a *worker* never merges or releases, and that PRs land
|
|
448
|
+
one at a time with a freshness re-check. Delegating a release to the
|
|
449
|
+
orchestrator does not touch either of those, so it needs no negotiation with the
|
|
450
|
+
shipped half.
|
|
439
451
|
|
|
440
452
|
**Below the banner — rewrite from the interview.**
|
|
441
453
|
|
|
442
|
-
- **Releases.**
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
the
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
direction or the other.
|
|
454
|
+
- **Releases.** The section's opening paragraph is rendered from the `authority`
|
|
455
|
+
answers, and Duty 1's "the PR is green" branch is rendered from the same
|
|
456
|
+
place. Leave both alone: they are the config speaking, and re-running
|
|
457
|
+
`/conductor setup` is how they change. What you write is everything under that
|
|
458
|
+
paragraph — the procedure you scaffolded in Step 4. If humans-release, there is
|
|
459
|
+
nothing to scaffold and you stop. If they chose a boundary, write the boundary
|
|
460
|
+
as a sentence with an end — *"your leg ends at the merged pin PR; you never
|
|
461
|
+
deploy it"* — then the scaffolded steps, then the forbidden list with its
|
|
462
|
+
citations. Do not restate the merge grant in your own words: a second spelling
|
|
463
|
+
of it is exactly the disagreement the rendered paragraph exists to prevent.
|
|
453
464
|
- **Reporting.** Rewrite it as the one scope they chose, in the second person,
|
|
454
465
|
concretely. Delete the description of the scope they did not choose: it is
|
|
455
466
|
useful in a template and noise in a live prompt. Keep the closing constraint
|