omp-conductor 0.7.1 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -954,6 +954,7 @@ rest. `0` is a real value (a hard stop), not "unset".
954
954
  | Cap | Default | What it protects |
955
955
  | --- | --- | --- |
956
956
  | `maxConcurrentWorkers` | `2` (setup may write `1` on <16 GiB hosts) | Parallel omp sessions, each a child process of the daemon and all inside its cgroup. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. On hosts under ~16 GiB RAM, prefer `1` so the unit stays out of swap ([host sizing](#host-sizing-and-memory)). Raise it only if you actually have the runners *and* the RAM. |
957
+ | `maxConcurrentWorkersPerRepo` | `1` | Max live workers in the **same repo**. The mirror, branch-protection staleness and shared CI egress are all per-repo collision domains, so extra slots should land on other repos. Raise it only when a repo genuinely needs two workers at once. |
957
958
  | `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total`. |
958
959
  | `planUsage` | `null` (unmetered) | Subscription/plan allowance guard: `{ "windowId": "anthropic:7d", "maxUsedFraction": 0.85 }`, or `null` for no plan gate. Independent of `dailySpendUsd` — see [Plan allowance](#plan-allowance-planusage) below. |
959
960
  | `workerMaxTurns` | `120` | Startup ceiling for each new worker. Catches a session looping without converging; use `omp-conductor extend` to raise one live run without changing this default. |
@@ -1373,21 +1374,24 @@ omp-conductor decision resolve <id> --answer "yes, after #132 lands"
1373
1374
  omp-conductor decision withdraw <id> --reason "the release slipped a week"
1374
1375
  ```
1375
1376
 
1376
- **`--resolves-when` is the part that makes a parked question wake up.** Three
1377
+ **`--resolves-when` is the part that makes a parked question wake up.** Six
1377
1378
  conditions, each one something this package can check without asking you:
1378
1379
 
1379
1380
  | Condition | Met when |
1380
1381
  | --- | --- |
1381
1382
  | `pr-merged:<https url>` | `gh` reports that pull request merged. |
1383
+ | `pr-checks-green:<https url>` | Every check on that pull request has a green verdict (a non-empty list, all `success`/`neutral`); a failing or still-pending check is not met. |
1384
+ | `pr-mergeable:<https url>` | The pull request is mergeable (`clean`, not `unknown` or conflicting). |
1382
1385
  | `issue-closed:<number>` | That issue is closed on the tracker. |
1383
1386
  | `npm-version:<pkg>@<version>` | `npm view <pkg>@<version> version` succeeds — the version is published. |
1387
+ | `rate-limit-reset:github` | GraphQL quota on `github` has any remaining capacity again. |
1384
1388
 
1385
1389
  The daemon evaluates them beside each tick, fire-and-forget: a hanging registry
1386
1390
  costs one unevaluated condition, never the tick. A met row is flagged in the next
1387
1391
  digest as `[CONDITION MET — act on this now]`, so the session acts on the answer
1388
1392
  at the moment it becomes actionable rather than when it happens to remember.
1389
1393
 
1390
- Anything else exits `2` and lists the three forms. An unparseable condition on an
1394
+ Anything else exits `2` and lists the six forms. An unparseable condition on an
1391
1395
  existing row is *listed and never treated as met*: a grammar a future release
1392
1396
  adds must not make an old row unloadable, and a question must never be hidden by
1393
1397
  a condition nobody can check.
@@ -1613,6 +1617,7 @@ Field notes:
1613
1617
  | `defaults` | Every `Caps` field. Anything omitted falls back to the built-in default. |
1614
1618
  | `tracker.repo` | `owner/repo`. `tracker.kind` may be omitted; `"github"` is the only accepted value. |
1615
1619
  | `queueLabel` | The one label meaning "a human has signed this off as agent-ready". Matched exactly, case-sensitively. |
1620
+ | `groomBelow` | Optional; default `4`. Routable candidates below this count make the orchestrator's tick prompt say the queue is running low and to groom it (Duty 2). An integer ≥ 1; anything else degrades to the default. |
1616
1621
  | `stateLabels` | Optional; defaults to `agent:in-progress`, `agent:blocked`, `agent:failed`. |
1617
1622
  | `routing.labelPrefix` | Optional; defaults to `repo:`. |
1618
1623
  | `routing.repos` | At least one entry, or nothing can be routed. `name` defaults to the map key, `defaultBranch` to `main`. |
@@ -1710,6 +1715,7 @@ writes this file for external orchestration. A manual configuration has this for
1710
1715
  | Key | Required | Default | Notes |
1711
1716
  | --- | --- | --- | --- |
1712
1717
  | `intervalSeconds` | yes | — | Whole seconds between ticks, minimum `60`. A tick costs a full turn of a frontier model, so a sub-minute period is refused rather than obeyed. |
1718
+ | `budgetSeconds` | no | `600` | Seconds a turn may run before the tick guard refuses its remaining tool calls (#189), and before a queued operator message preempts them. An integer ≥ 60; anything else degrades to the default. |
1713
1719
  | `armedFile` | no | none — the gate passes | Path to the arm marker. A tick does nothing while the file is missing. Relative paths resolve against the session cwd, so `state/armed` means `<cwd>/state/armed`. |
1714
1720
  | `accessFile` | no | none — the gate passes | Path to the Telegram bridge's `access.json`. Every tick re-reads it and requires `enabled: true` with exactly one entry in `allowFrom`. Relative paths resolve against the session cwd. **Configure this on any fleet deploy** — see below. |
1715
1721
  | `message` | no | `Tick <ISO timestamp>: re-read <workspaceRoot>/ORCHESTRATOR.md from disk, then run your standing loop from it.`, then the `reporting.scope` line, then the delivery rule | Sent verbatim when set — and then it owns the whole contract: neither the scope line nor the delivery rule is appended to a prompt you wrote yourself. Re-read from disk on **every** tick, so rewording it binds the next heartbeat instead of waiting for a session restart; a re-read that fails — caught mid-edit, removed, or invalid — keeps the value read at session start rather than stopping the heartbeat. `intervalSeconds` is *not* re-read: rescheduling a live timer still needs a restart. The default *orders* the session to re-read its brief, naming the path resolved from the project's `workspaceRoot`, because a standing prompt drifts out of a long-lived session's context while the file on disk does not. |
@@ -1905,7 +1911,7 @@ least of all on a fleet whose session lives somewhere else.
1905
1911
  omp-conductor start [--port N] [--project NAME]
1906
1912
  omp-conductor --version
1907
1913
  omp-conductor stop
1908
- omp-conductor restart [--port N] [--project NAME]
1914
+ omp-conductor restart [--now] [--timeout SECONDS] [--port N] [--project NAME]
1909
1915
  omp-conductor upgrade [--to VERSION] [--project NAME]
1910
1916
  omp-conductor status [--project NAME]
1911
1917
  omp-conductor ledger [--issue N] [--limit N] [--project NAME]
@@ -1917,7 +1923,7 @@ omp-conductor disarm [--project NAME]
1917
1923
  omp-conductor release-pane [--project NAME]
1918
1924
  omp-conductor tail <issue> [--project NAME]
1919
1925
  omp-conductor extend <issue> --turns N [--project NAME]
1920
- omp-conductor unblock <issue> [--force] [--project NAME]
1926
+ omp-conductor unblock <issue> [--force] [--no-requeue] [--project NAME]
1921
1927
  omp-conductor verb <conductor_*> [--project NAME] [--arg k=v ...]
1922
1928
  omp-conductor friction <escalation-digest|report-noise|report-surprise> --detail TEXT [--issue N] [--project NAME]
1923
1929
  omp-conductor report --text TEXT [--kind material|digest] [--project NAME]
@@ -1926,7 +1932,7 @@ omp-conductor decision resolve <id> --answer TEXT [--project NAME]
1926
1932
  omp-conductor decision withdraw <id> [--reason TEXT] [--project NAME]
1927
1933
  omp-conductor decision list [--project NAME]
1928
1934
  omp-conductor daemon [--once] [--port N] [--project NAME]
1929
- omp-conductor pause
1935
+ omp-conductor pause [--reason TEXT]
1930
1936
  omp-conductor resume
1931
1937
  omp-conductor graph-setup [--project NAME] [--write]
1932
1938
  omp-conductor brief-upgrade [--migrate|--retrofit] [--apply] [--file PATH] [--project NAME]
@@ -1937,9 +1943,9 @@ omp-conductor help
1937
1943
  | --- | --- |
1938
1944
  | `start` | Start `herdr-fleet.service` when that optional unit is installed, clearing a previous pane-recovery pin, then spawn the dispatch loop in the background and wait until it answers `GET /healthz` on `:8787`. Without systemd or that unit it keeps the standalone daemon behaviour. It never clears pause or arms ticks. Refuses if a daemon is already live, naming its pid; if the process dies or never serves, it cleans up and quotes the tail of `daemon.log`. |
1939
1945
  | `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
1940
- | `restart` | Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
1946
+ | `restart [--now] [--timeout SECONDS] [--port N] [--project NAME]` | Drains the fleet by default: pause new claims, wait until live workers reach `0 / N` (bounded by `--timeout SECONDS`, default 1800 = 30 min), restart, then restore the prior dispatch state. Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. `--now` skips the drain and restarts immediately, orphaning any live runs (old behaviour). A drain that hits `--timeout` restarts nothing and leaves dispatch paused — `omp-conductor resume` lifts it, or re-run `restart` to keep waiting. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
1941
1947
  | `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
1942
- | `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, and bounded hold groups; API failures are marked `DEGRADED` so queue starvation cannot look idle. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
1948
+ | `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
1943
1949
  | `ledger [--issue N] [--limit N]` | Every [mediated-verb](#the-mediated-verbs-126) call and how the daemon decided it: the verb, the arguments it was given, allow or refuse, the named refusal reason, and any resulting SHA. Sessions cannot push, open a PR, merge, label or release except through those verbs, so this is the record of what they *tried* as well as what they did — the thing an escalation cites. Reads (`conductor_pr_status`) are deliberately absent: a poll every thirty seconds would bury the refusals. `--issue` narrows to one issue's run; `--limit` defaults to 50. The newest few also appear in `status`. |
1944
1950
  | `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, and the last 24 hours of Merged and History. Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
1945
1951
  | `hold [--project NAME]` | Soft stop: pause claiming **and** disarm ticks. Daemon and pane stay up. Prefer this over `pause` when the intent is "stop the conductor" without killing processes. See [Stop the conductor](#stop-the-conductor-hold--halt). |
@@ -1949,11 +1955,11 @@ omp-conductor help
1949
1955
  | `release-pane [--project NAME]` | Clear the `halt --pane` recovery pin so herdr-conductor may resume the fleet agent again. |
1950
1956
  | `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
1951
1957
  | `extend <issue> --turns N [--project NAME]` | Monotonically raise that live worker's effective turn ceiling through its owning daemon. The current omp session keeps running; no restart or continuation is created. The daemon persists the new ceiling for `status` and rejects missing, settled, cap-killed, equal, or lower requests instead of implying that an immutable session changed. |
1952
- | `unblock <issue> [--force]` | Remove that issue's `blocked` and `failed` labels so an answered escalation can be claimed again. `agent:in-progress` comes off too, but only when the newest recorded run is terminal — that row is the proof no worker still owns the issue, so a live run keeps the label, and so does an issue with no run row at all. Run history remains intact: blocks consume the independent continuation budget, not failed implementation attempts. The output reports both budgets and warns when either will make the next tick escalate instead of dispatch. **Refuses, clearing nothing and exiting `3`, when the newest attempt's work could not be committed and its worktree is the only copy** — re-claiming removes that tree. `--force` records the operator's acceptance on the run row and then clears; the salvage failure stays in history. Exits `2` when the issue number is missing or malformed. |
1958
+ | `unblock <issue> [--force] [--no-requeue]` | Remove that issue's `blocked` and `failed` labels so an answered escalation can be claimed again, and restore the project queue label by default so the dispatcher actually sees it. `agent:in-progress` comes off too, but only when the newest recorded run is terminal — that row is the proof no worker still owns the issue, so a live run keeps the label (and the queue label stays off until that run settles), and so does an issue with no run row at all. Run history remains intact: blocks consume the independent continuation budget, not failed implementation attempts. The output reports both budgets and warns when either will make the next tick escalate instead of dispatch. `--no-requeue` clears the state labels only, leaving the queue label untouched — the case where you are about to close the issue. **Refuses, clearing nothing and exiting `3`, when the newest attempt's work could not be committed and its worktree is the only copy** — re-claiming removes that tree. `--force` records the operator's acceptance on the run row and then clears; the salvage failure stays in history. Exits `2` when the issue number is missing or malformed. |
1953
1959
  | `verb <conductor_*> [--arg k=v ...]` | Run one [mediated verb](#the-mediated-verbs-126) as the orchestrator, from the CLI — the external-orchestrator half of the verb surface. Every argument goes in as a `--arg k=v` string; an orchestrator can merge (`conductor_pr_merge`), label (`conductor_label`), release (`conductor_release`), update a branch (`conductor_pr_update_branch`) or title/body (`conductor_pr_update`), or read PR state (`conductor_pr_status`). The daemon applies the same checks and writes the same ledger rows a session's call would; a missing `--arg` is refused exactly as a missing tool argument is, worker-only verbs (`conductor_push`, `conductor_pr_create`) are refused with `role-not-allowed`, and a refusal exits `3`. An unknown verb exits `2`. |
1954
1960
  | `friction <kind> --detail TEXT [--issue N]` | Record one bounded judgment the daemon cannot infer: an escalation belonged in a digest, or a tick report was noise/surprising. The detail is limited to 160 characters. One event never changes policy; three observations inside seven days make the aggregate eligible for one Learning-loop prompt, followed by a seven-day cooldown. |
1955
1961
  | `report --text TEXT [--kind material|digest]` | Hand a rendered report to the daemon's durable outbox. The text is persisted **before** anything is sent and the command prints a report id; the daemon then owns delivery, retries on a bounded backoff, and records the Telegram message id it actually got back. Delivery is [at-least-once](#report-delivery-the-outbox) — a crash mid-send is retried and the retry says it may be a repeat — so `delivered` is never proof of exactly one message. `--kind digest` is accepted at most once per local day, decided from the ledger rather than from what the session remembers sending; an unknown `--kind` exits `2` rather than quietly becoming a material report. Anything undelivered shows in `status` with its age. |
1956
- | `decision open --question TEXT [--blocks TEXT] [--resolves-when COND]` | Record a question the orchestrator has put to you, and print its id. A question that lives only in a session's context is lost at the next compaction — after which it is either asked twice or dropped silently. `--resolves-when` attaches a machine-checkable condition: `pr-merged:<https url>`, `issue-closed:<n>`, or `npm-version:<pkg>@<version>`; anything else exits `2` listing the three forms. See [The decision ledger](#the-decision-ledger-136). |
1962
+ | `decision open --question TEXT [--blocks TEXT] [--resolves-when COND]` | Record a question the orchestrator has put to you, and print its id. A question that lives only in a session's context is lost at the next compaction — after which it is either asked twice or dropped silently. `--resolves-when` attaches a machine-checkable condition: `pr-merged:<https url>`, `pr-checks-green:<https url>`, `pr-mergeable:<https url>`, `issue-closed:<n>`, `npm-version:<pkg>@<version>`, or `rate-limit-reset:github`; anything else exits `2` listing the six forms. See [The decision ledger](#the-decision-ledger-resolves-when). |
1957
1963
  | `decision resolve <id> --answer TEXT` | Record what you decided. Exits `1` naming the id when it is unknown or no longer open, so a second answer cannot overwrite the first. |
1958
1964
  | `decision withdraw <id> [--reason TEXT]` | Close a question the session stopped needing, with why. Same guard as `resolve`. |
1959
1965
  | `decision list` | Open questions, oldest first: id, age, what each blocks, whether its condition is met, and the question. Prints `no open decisions` when there are none. |
@@ -1961,7 +1967,7 @@ omp-conductor help
1961
1967
  | `daemon --once` | Run a single tick, wait for workers admitted by that tick, and exit. No HTTP server or pidfile — a drill must not register itself as the daemon, or the next reader believes it and the real daemon's in-flight runs get reconciled as orphans. |
1962
1968
  | `--port N` | Accepted by `start`, `restart` and `daemon`. Both `--port 9000` and `--port=9000` work; missing or out of range exits `2` rather than falling back to the default, because probing the wrong endpoint is worse than a hard failure. |
1963
1969
  | `--project NAME` | Pick the project to service. One daemon process serves exactly one project; with several configured projects the name is required. |
1964
- | `pause` | Stop claiming new work only. The running daemon notices on its next tick; runs already in flight finish. The orchestrator heartbeat keeps ticking if armed — its gate is the arm marker, not this flag. Prefer `hold` to silence both. |
1970
+ | `pause [--reason TEXT]` | Stop claiming new work only. The running daemon notices on its next tick; runs already in flight finish. The orchestrator heartbeat keeps ticking if armed — its gate is the arm marker, not this flag. Prefer `hold` to silence both. `--reason TEXT` is recorded in the pause sentinel, which `status` shows as the pause provenance. |
1965
1971
  | `resume` | Clear pause only — does **not** re-arm. Run `arm` after an inbound Telegram proof to resume ticks. |
1966
1972
  | `--version`, `-V`, `version` | Print the installed `omp-conductor` package version and exit `0`. Works from the global binary and npm/plugin install because it reads the package metadata beside the shipped CLI. |
1967
1973
  | `graph-setup` | Print how to set up the code-graph indexes workers query instead of grepping: a `git clone` for every index-only clone that does not exist yet, the one-shot index command per repo, and a `cbm-reindex.service` + `cbm-reindex.timer` pair generated from the project's own repos and branches. Reads only, so it is safe on a host where you are not root. Exits `1` when no repo in the project has [`graphProject`](#configuration) set, because the fix is a wizard answer rather than a flag. See [Code-graph discovery](#code-graph-discovery). |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "omp-conductor",
3
- "version": "0.7.1",
3
+ "version": "0.8.0",
4
4
  "type": "module",
5
5
  "license": "MIT",
6
6
  "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
@@ -282,6 +282,16 @@ Not yours to relax:
282
282
  not the branch. A settled run means nobody else is coming:
283
283
  waiting is not caution, it is a stall.
284
284
 
285
+ **You never sleep, poll, or wait inside a tick.** No `sleep`, no retry loop, no
286
+ "watch this PR until green" — a tool call that exists to pass time is a tool
287
+ call that blocks your operator's messages. Anything that needs waiting for is
288
+ either a watch (`decision open --resolves-when pr-checks-green:<url>`,
289
+ `pr-mergeable:<url>`, `pr-merged:<url>`, `issue-closed:<n>`,
290
+ `npm-version:<pkg>@<version>`, `rate-limit-reset:github`) or a subagent's
291
+ problem. The harness refuses further tool calls once a turn exceeds its budget
292
+ or an operator message is queued — end the turn and let the next tick act on
293
+ `[CONDITION MET]`.
294
+
285
295
  ## Your verb surface
286
296
 
287
297
  You can reach `gh`, and you do not publish with it. Everything below happens
@@ -420,8 +430,9 @@ omp-conductor decision withdraw <id> --reason "<why it stopped mattering>"
420
430
  ```
421
431
 
422
432
  Use `--resolves-when` whenever the answer only becomes actionable once something
423
- observable happens: `pr-merged:<url>`, `issue-closed:<n>`,
424
- `npm-version:<pkg>@<version>`. The daemon checks it for you and flags the row as
433
+ observable happens: `pr-merged:<url>`, `pr-checks-green:<url>`,
434
+ `pr-mergeable:<url>`, `issue-closed:<n>`, `npm-version:<pkg>@<version>`,
435
+ `rate-limit-reset:github`. The daemon checks it for you and flags the row as
425
436
  `[CONDITION MET — act on this now]` in your tick digest, so a parked question
426
437
  wakes up on its own instead of waiting for you to think of it.
427
438
 
package/src/cli.ts CHANGED
@@ -44,6 +44,7 @@ import {
44
44
  startDaemon,
45
45
  stopDaemon,
46
46
  writeRecord,
47
+ type RestartResult,
47
48
  } from "./lifecycle.ts";
48
49
  import { STALL_MARKER_FILE } from "./orchestrator-tick.ts";
49
50
  import { digestDedupeKey } from "./reports.ts";
@@ -63,7 +64,7 @@ import { handleVerbCall, type VerbChannel } from "./verbs/server.ts";
63
64
  import { REPORT_KINDS, VERB_NAMES } from "./types.ts";
64
65
  import type { ProjectConfig, ReportKind } from "./types.ts";
65
66
  import { formatUnblock, unblockIssue } from "./unblock.ts";
66
- import { upgradeConductor } from "./upgrade.ts";
67
+ import { DEFAULT_DEPS, drainAndRestart, upgradeConductor, type UpgradeDeps } from "./upgrade.ts";
67
68
 
68
69
  const FRICTION_FEEDBACK_KINDS = {
69
70
  "escalation-digest": "feedback:escalation-should-digest",
@@ -89,7 +90,7 @@ usage:
89
90
  omp-conductor start [--port N] [--project NAME]
90
91
  omp-conductor --version
91
92
  omp-conductor stop
92
- omp-conductor restart [--port N] [--project NAME]
93
+ omp-conductor restart [--now] [--timeout SECONDS] [--port N] [--project NAME]
93
94
  omp-conductor upgrade [--to VERSION] [--project NAME]
94
95
  omp-conductor board [--project NAME]
95
96
  omp-conductor status [--project NAME]
@@ -101,10 +102,10 @@ usage:
101
102
  omp-conductor release-pane [--project NAME]
102
103
  omp-conductor tail <issue> [--project NAME]
103
104
  omp-conductor extend <issue> --turns N [--project NAME]
104
- omp-conductor unblock <issue> [--force] [--project NAME]
105
+ omp-conductor unblock <issue> [--force] [--no-requeue] [--project NAME]
105
106
  omp-conductor verb <conductor_*> [--project NAME] [--arg k=v ...]
106
107
  omp-conductor daemon [--once] [--port N] [--project NAME]
107
- omp-conductor pause
108
+ omp-conductor pause [--reason TEXT]
108
109
  omp-conductor resume
109
110
  omp-conductor graph-setup [--project NAME] [--write]
110
111
  omp-conductor brief-upgrade [--migrate|--retrofit] [--apply] [--file PATH] [--project NAME]
@@ -126,11 +127,15 @@ usage:
126
127
  stop stop the running daemon. Uses systemctl when the omp-conductor
127
128
  unit owns the process (so Restart=on-failure cannot bring it back);
128
129
  otherwise SIGTERM then SIGKILL.
129
- restart stop then start, keeping the running daemon's port and project
130
- unless a flag overrides them. On boot the new process salvages any
131
- dirty live worktrees before orphaning those rows see README
132
- "Deploying a new package onto a busy fleet". Goes through systemctl
133
- when the unit owns the live pid.
130
+ restart drain the fleet by default: pause new claims, wait until live workers
131
+ reach 0/N (bounded by --timeout SECONDS, default 1800), restart keeping
132
+ the running daemon's port and project unless a flag overrides them,
133
+ then restore the prior dispatch state. --now skips the drain and
134
+ restarts immediately, orphaning any live runs (old behavior). A drain
135
+ that hits --timeout restarts nothing and leaves dispatch paused. On boot
136
+ the new process salvages dirty live worktrees before orphaning those
137
+ rows — see README "Deploying a new package onto a busy fleet". Goes
138
+ through systemctl when the unit owns the live pid.
134
139
  status layered fleet report: dispatch (running|paused|stopped), ticks and
135
140
  next due time, pane, herdr, Telegram bot/API health, daemon, caps
136
141
  and active runs.
@@ -170,7 +175,10 @@ usage:
170
175
  operational-continuation budget, not failed implementation attempts.
171
176
  Refuses when the newest attempt's work could not be committed and
172
177
  its worktree is the only copy: re-claiming removes that tree. Use
173
- --force once you have recovered it or accepted the loss.
178
+ --force once you have recovered it or accepted the loss. Restores
179
+ the project queue label by default so the issue is actually
180
+ dispatchable; --no-requeue clears state labels only (the "about to
181
+ close it" case).
174
182
  verb run one conductor_* verb as the orchestrator, from the CLI: the same
175
183
  checks and the same ledger rows a session's call would get. This is
176
184
  how an external orchestrator merges, labels, releases or reads PR
@@ -202,6 +210,7 @@ usage:
202
210
  daemon run the dispatch loop in the foreground; --once runs a single tick
203
211
  and exits. This is what \`start\` launches.
204
212
  pause stop claiming new work only (ticks keep firing if armed). Prefer hold.
213
+ --reason TEXT is recorded in the pause sentinel (\`status\` shows it).
205
214
  resume clear pause only — does NOT re-arm. Prefer hold's inverse: resume + arm.
206
215
  graph-setup
207
216
  print how to set up the code-graph indexes workers query instead of
@@ -525,23 +534,62 @@ try {
525
534
  }
526
535
 
527
536
  case "restart": {
528
- // Inherit the running daemon's port and project: a restart that quietly
529
- // moved to the default port would leave every existing health check
530
- // pointing at nothing. When the unit owns the live pid, restartDaemon
531
- // goes through systemctl so the replacement stays supervised.
532
- const { previous, record, via } = await restartDaemon({
533
- port: portFlag(argv),
534
- project: flag(argv, "project"),
535
- });
536
- if (previous !== undefined) {
537
+ if (argv.includes("--now")) {
538
+ // Inherit the running daemon's port and project: a restart that quietly
539
+ // moved to the default port would leave every existing health check
540
+ // pointing at nothing. When the unit owns the live pid, restartDaemon
541
+ // goes through systemctl so the replacement stays supervised.
542
+ const { previous, record, via } = await restartDaemon({
543
+ port: portFlag(argv),
544
+ project: flag(argv, "project"),
545
+ });
546
+ if (previous !== undefined) {
547
+ process.stdout.write(
548
+ `stopped — pid ${previous.pid}${via === "systemctl" ? " (via systemctl)" : ""}\n`,
549
+ );
550
+ }
551
+ process.stdout.write(
552
+ `started — pid ${record.pid}, /healthz on :${record.port}` +
553
+ `${record.project === undefined ? "" : `, project ${record.project}`}` +
554
+ `${via === "systemctl" ? " (via systemctl)" : ""}\nlog ${record.logFile}\n`,
555
+ );
556
+ break;
557
+ }
558
+
559
+ // Default: drain the fleet before restarting — pause new claims, wait for
560
+ // live workers to reach 0/N (bounded by --timeout), restart, then restore
561
+ // the prior dispatch state. A timed-out drain restarts nothing and leaves
562
+ // dispatch paused, so it is safe to run while the daemon is wedged.
563
+ const timeoutRaw = flag(argv, "timeout");
564
+ const timeoutSeconds =
565
+ timeoutRaw === undefined ? 1800 : Number.parseInt(timeoutRaw, 10);
566
+ const timeoutMs = (Number.isNaN(timeoutSeconds) ? 1800 : timeoutSeconds) * 1000;
567
+ let result!: RestartResult;
568
+ const deps: UpgradeDeps = {
569
+ ...DEFAULT_DEPS,
570
+ setPaused: (v) => setPaused(v, { source: "restart", reason: "restart, draining" }),
571
+ restartDaemon: async () => {
572
+ result = await restartDaemon({
573
+ port: portFlag(argv),
574
+ project: flag(argv, "project"),
575
+ });
576
+ },
577
+ };
578
+ try {
579
+ await drainAndRestart(deps, { project: flag(argv, "project"), timeoutMs });
580
+ } catch (err) {
581
+ process.stderr.write(`${err instanceof Error ? err.message : String(err)}\n`);
582
+ process.exit(1);
583
+ }
584
+ if (result.previous !== undefined) {
537
585
  process.stdout.write(
538
- `stopped — pid ${previous.pid}${via === "systemctl" ? " (via systemctl)" : ""}\n`,
586
+ `stopped — pid ${result.previous.pid}${result.via === "systemctl" ? " (via systemctl)" : ""}\n`,
539
587
  );
540
588
  }
541
589
  process.stdout.write(
542
- `started — pid ${record.pid}, /healthz on :${record.port}` +
543
- `${record.project === undefined ? "" : `, project ${record.project}`}` +
544
- `${via === "systemctl" ? " (via systemctl)" : ""}\nlog ${record.logFile}\n`,
590
+ `started — pid ${result.record.pid}, /healthz on :${result.record.port}` +
591
+ `${result.record.project === undefined ? "" : `, project ${result.record.project}`}` +
592
+ `${result.via === "systemctl" ? " (via systemctl)" : ""}\nlog ${result.record.logFile}\n`,
545
593
  );
546
594
  break;
547
595
  }
@@ -732,6 +780,7 @@ try {
732
780
  try {
733
781
  const outcome = await unblockIssue(project, makeTracker(project), store, issue, {
734
782
  force: argv.includes("--force"),
783
+ requeue: !argv.includes("--no-requeue"),
735
784
  });
736
785
  process.stdout.write(`${formatUnblock(issue, outcome, project, resolveCaps(project, cfg.defaults))}\n`);
737
786
  // A refusal must not read as success to a script or a board keypress.
@@ -1006,7 +1055,7 @@ try {
1006
1055
  }
1007
1056
 
1008
1057
  case "pause":
1009
- setPaused(true);
1058
+ setPaused(true, { source: "pause", reason: flag(argv, "reason") });
1010
1059
  process.stdout.write(
1011
1060
  "paused — no new work will be claimed\n" +
1012
1061
  "note: ticks keep firing if armed; use hold to silence both\n",
package/src/config.ts CHANGED
@@ -208,6 +208,7 @@ export function resolveCaps(p: ProjectConfig, defaults: Caps): Caps {
208
208
  const o: Partial<Caps> = p.caps ?? {};
209
209
  return {
210
210
  maxConcurrentWorkers: o.maxConcurrentWorkers ?? defaults.maxConcurrentWorkers,
211
+ maxConcurrentWorkersPerRepo: o.maxConcurrentWorkersPerRepo ?? defaults.maxConcurrentWorkersPerRepo,
211
212
  dailySpendUsd: o.dailySpendUsd !== undefined ? o.dailySpendUsd : defaults.dailySpendUsd,
212
213
  planUsage: o.planUsage !== undefined ? o.planUsage : defaults.planUsage,
213
214
  workerMaxTurns: o.workerMaxTurns ?? defaults.workerMaxTurns,
@@ -408,6 +409,15 @@ function normalizeProject(
408
409
  if (nonEmptyString(rawQueueLabel)) queueLabel = rawQueueLabel;
409
410
  else problems.push(`${label}: queueLabel must be a non-empty string — it is the human sign-off gate`);
410
411
 
412
+ // Tolerant like the other optional fields: an integer >= 1 is a grooming
413
+ // trigger, and anything else degrades to absent (the tick default) rather
414
+ // than invalidating the whole project.
415
+ const rawGroomBelow = raw["groomBelow"];
416
+ const groomBelow =
417
+ typeof rawGroomBelow === "number" && Number.isInteger(rawGroomBelow) && rawGroomBelow >= 1
418
+ ? rawGroomBelow
419
+ : undefined;
420
+
411
421
  const routing = raw["routing"] as Raw | undefined;
412
422
  const rawPrefix = routing?.["labelPrefix"];
413
423
  const labelPrefix = typeof rawPrefix === "string" ? rawPrefix : DEFAULT_LABEL_PREFIX;
@@ -434,6 +444,7 @@ function normalizeProject(
434
444
  name,
435
445
  tracker: { kind: "github", repo: trackerRepo },
436
446
  queueLabel,
447
+ ...(groomBelow === undefined ? {} : { groomBelow }),
437
448
  stateLabels: {
438
449
  inProgress: pickString(stateLabels?.["inProgress"], DEFAULT_STATE_LABELS.inProgress),
439
450
  blocked: pickString(stateLabels?.["blocked"], DEFAULT_STATE_LABELS.blocked),
package/src/daemon.ts CHANGED
@@ -34,10 +34,10 @@ import { createReportOutbox, formatOpenReports } from "./reports.ts";
34
34
  import { recordReleaseBlock } from "./release-policy.ts";
35
35
  import { branchName, route } from "./routing.ts";
36
36
  import type { Routed, UnroutableReason } from "./routing.ts";
37
- import { evaluateDecisionConditions, probeNpmVersion } from "./decisions.ts";
37
+ import { evaluateDecisionConditions, probeNpmVersion, probeRateLimitReset } from "./decisions.ts";
38
38
  import { classifyRun, type ClassifyFacts } from "./failure-class.ts";
39
39
  import { dbPath, openStore } from "./store.ts";
40
- import { makeTracker } from "./tracker/github.ts";
40
+ import { makeTracker, type RateLimitStatus } from "./tracker/github.ts";
41
41
  import { RELEASE_SHAPES } from "./types.ts";
42
42
  import type {
43
43
  AdmissionHoldReason,
@@ -323,11 +323,45 @@ export function pausedAt(): number | undefined {
323
323
  }
324
324
  }
325
325
 
326
- export function setPaused(v: boolean): void {
326
+ /**
327
+ * Who paused the fleet and why, read from line 2 of the same sentinel
328
+ * {@link setPaused} writes — `undefined` when the fleet is not paused or the
329
+ * file has no (parseable) line 2. Provenance lives on its own line so line 1
330
+ * stays a pure ISO timestamp that {@link pausedAt} can `Date.parse`; the `armed
331
+ * <ISO> owner=<id>` marker `armTicks` writes is the same one-key-per-line
332
+ * precedent.
333
+ */
334
+ export function pauseProvenance(): { source: string; reason?: string } | undefined {
335
+ const f = join(stateDir(), "paused");
336
+ if (!existsSync(f)) return undefined;
337
+ try {
338
+ const second = readFileSync(f, "utf8").split("\n")[1];
339
+ if (second === undefined) return undefined;
340
+ const match = /^source=(\S+)(?: reason="(.*)")?$/.exec(second.trim());
341
+ if (match === null) return undefined;
342
+ const source = match[1]!; // the regex guarantees group 1 on a match
343
+ const reason = match[2];
344
+ return { source, ...(reason === undefined ? {} : { reason }) };
345
+ } catch {
346
+ // Unreadable sentinel: no provenance to name, and the refuse-everything
347
+ // posture of an unknown pause is unchanged.
348
+ return undefined;
349
+ }
350
+ }
351
+
352
+ export function setPaused(v: boolean, why?: { source: string; reason?: string }): void {
327
353
  const f = join(stateDir(), "paused");
328
354
  if (v) {
329
355
  mkdirSync(dirname(f), { recursive: true });
330
- writeFileSync(f, `${new Date().toISOString()}\n`);
356
+ const line1 = `${new Date().toISOString()}\n`;
357
+ if (why === undefined) {
358
+ writeFileSync(f, line1);
359
+ } else {
360
+ // Quotes are stripped before embedding so a reason cannot break out of
361
+ // the `reason="..."` field of line 2.
362
+ const reason = why.reason === undefined ? "" : ` reason="${why.reason.replaceAll('"', "")}"`;
363
+ writeFileSync(f, `${line1}source=${why.source}${reason}\n`);
364
+ }
331
365
  } else {
332
366
  rmSync(f, { force: true });
333
367
  }
@@ -1757,6 +1791,13 @@ export async function admitCandidates(
1757
1791
  if (list === undefined) activeByIssue.set(run.issue, [run]);
1758
1792
  else list.push(run);
1759
1793
  }
1794
+ // Live worker count per repo, seeded from live runs and incremented as this
1795
+ // same pass admits — so two same-repo candidates can never both clear the
1796
+ // per-repo cap in one tick (#186).
1797
+ const liveByRepo = new Map<string, number>();
1798
+ for (const run of store.liveRuns(project.name)) {
1799
+ liveByRepo.set(run.repo, (liveByRepo.get(run.repo) ?? 0) + 1);
1800
+ }
1760
1801
  const holds: AdmissionHold[] = [];
1761
1802
  const hold = (issue: number, reason: AdmissionHoldReason): void => {
1762
1803
  holds.push({ issue, reason });
@@ -1828,6 +1869,16 @@ export async function admitCandidates(
1828
1869
  }
1829
1870
  }
1830
1871
 
1872
+ // Per-repo concurrency: the mirror, branch-protection staleness and shared
1873
+ // CI egress are all per-repo collision domains, so extra slots should land
1874
+ // on other repos rather than stacking workers into the same one (#186).
1875
+ const liveInRepo = liveByRepo.get(r.repo.name) ?? 0;
1876
+ if (liveInRepo >= caps.maxConcurrentWorkersPerRepo) {
1877
+ hold(issue, "repo-active");
1878
+ log(`#${issue} skipped: ${liveInRepo} live worker(s) already in ${r.repo.name} (cap ${caps.maxConcurrentWorkersPerRepo})`);
1879
+ continue;
1880
+ }
1881
+
1831
1882
  const priorRuns = store.attemptsFor(project.name, issue);
1832
1883
  const failures = store.failuresFor(project.name, issue);
1833
1884
  if (failures >= caps.maxAttemptsPerIssue) {
@@ -1976,6 +2027,7 @@ export async function admitCandidates(
1976
2027
  }
1977
2028
 
1978
2029
  admitted.push({ r, attempt: priorRuns + 1 });
2030
+ liveByRepo.set(r.repo.name, (liveByRepo.get(r.repo.name) ?? 0) + 1);
1979
2031
  if (parent !== undefined) occupiedParents.set(parent, issue);
1980
2032
  }
1981
2033
 
@@ -2083,7 +2135,13 @@ export async function tick(d: Deps, workers?: WorkerPool): Promise<void> {
2083
2135
  for (const expired of d.store.expireDueDecisions(d.project.name, Date.now())) {
2084
2136
  log(`decision ${expired.id} expired unanswered after seven days: ${expired.question}`);
2085
2137
  }
2086
- void evaluateDecisionConditions(d.store, d.project.name, d.tracker, probeNpmVersion, Date.now)
2138
+ void evaluateDecisionConditions(
2139
+ d.store,
2140
+ d.project.name,
2141
+ d.tracker,
2142
+ { npm: probeNpmVersion, rateLimit: probeRateLimitReset },
2143
+ Date.now,
2144
+ )
2087
2145
  .then((met) => {
2088
2146
  for (const decision of met) {
2089
2147
  log(`decision ${decision.id} condition met (${decision.condition ?? "?"}) — surfacing on the next tick`);
@@ -2122,7 +2180,7 @@ export async function tick(d: Deps, workers?: WorkerPool): Promise<void> {
2122
2180
  `ERROR: the installed conductor changed under this daemon — ${integrity.diff.length} file(s) differ ` +
2123
2181
  `(${shown.join(", ")}${integrity.diff.length > shown.length ? ", …" : ""}) — pausing`,
2124
2182
  );
2125
- setPaused(true);
2183
+ setPaused(true, { source: "integrity", reason: "installed package changed under the daemon" });
2126
2184
  if (integrity.page) {
2127
2185
  const delivered = await safeEscalate(d, {
2128
2186
  tier: 2,
@@ -2197,7 +2255,7 @@ export async function tick(d: Deps, workers?: WorkerPool): Promise<void> {
2197
2255
  // operator opted out — turns and wall-clock still brake every run (#46).
2198
2256
  const spent = store.spendSince(project.name, since);
2199
2257
  if (caps.dailySpendUsd !== null && spent >= caps.dailySpendUsd) {
2200
- setPaused(true);
2258
+ setPaused(true, { source: "spend-cap", reason: `daily spend reached $${caps.dailySpendUsd}` });
2201
2259
  await safeEscalate(d, {
2202
2260
  tier: 2,
2203
2261
  project: project.name,
@@ -2403,6 +2461,13 @@ export interface StatusSnapshot {
2403
2461
  * than print a percentage nobody measured (#110).
2404
2462
  */
2405
2463
  planUsage?: PlanUsageStatus;
2464
+ /**
2465
+ * The GitHub API rate-limit budget, when the caller read one. Optional for
2466
+ * the same reason as `planUsage` — the read is I/O and the snapshot is
2467
+ * synchronous — and a broken `gh` must cost one status row, not the report
2468
+ * (#188).
2469
+ */
2470
+ github?: RateLimitStatus;
2406
2471
  }
2407
2472
 
2408
2473
  /** Builds a status reading from an already-open store. Long-lived operator
@@ -2629,7 +2694,7 @@ export async function previewQueue(project?: string): Promise<QueuePreview> {
2629
2694
  */
2630
2695
  export function prepareConductor(): void {
2631
2696
  openStore(dbPath()).close();
2632
- setPaused(true);
2697
+ setPaused(true, { source: "setup" });
2633
2698
  }
2634
2699
 
2635
2700
  /** Bounded per tick: each row costs tracker calls to gather facts for. */