omp-conductor 0.4.3 → 0.4.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -4
- package/package.json +1 -1
- package/src/credentials.ts +176 -2
- package/src/daemon.ts +149 -11
- package/src/failure-class.ts +39 -0
- package/src/plugin.ts +23 -0
- package/src/setup-host.ts +35 -1
- package/src/setup.ts +40 -0
- package/src/store.ts +13 -3
- package/src/tracker/github.ts +26 -9
- package/src/types.ts +14 -4
- package/src/upgrade.ts +2 -0
- package/systemd/omp-conductor.service.example +8 -0
package/README.md
CHANGED
|
@@ -149,6 +149,25 @@ be present. If you run omp, it is.
|
|
|
149
149
|
Also required on the host:
|
|
150
150
|
|
|
151
151
|
- `bun`: the CLI and the daemon run on it (`Bun.serve` backs `/healthz`).
|
|
152
|
+
- **A model credential a *session* can reach.** Sessions run with `$HOME`
|
|
153
|
+
redirected into a per-run tree, so the harness never reads the operator's own
|
|
154
|
+
configuration — that is what keeps a credential-bearing MCP server out of every
|
|
155
|
+
session. It also means a login recorded only in `~/.omp/agent/agent.db` is
|
|
156
|
+
invisible to a worker, which then dies at turn 0 with `No model selected`. Two
|
|
157
|
+
supported shapes:
|
|
158
|
+
- a model API key in the **daemon's** environment (`ANTHROPIC_API_KEY`,
|
|
159
|
+
`OPENAI_API_KEY`, …); these are deliberately not scrubbed, unlike the GitHub
|
|
160
|
+
and npm keys.
|
|
161
|
+
- the harness's **auth broker** — `omp auth-broker serve` plus `omp auth-broker
|
|
162
|
+
token`, or `auth.broker.url` / `auth.broker.token` in
|
|
163
|
+
`~/.omp/agent/config.yml`. The conductor resolves that connection and injects
|
|
164
|
+
it into each session. Refresh tokens stay in the broker and a session receives
|
|
165
|
+
only short-lived access tokens, which is exactly what per-run isolation wants.
|
|
166
|
+
|
|
167
|
+
With neither, `omp-conductor status` reports the hold and **nothing is
|
|
168
|
+
dispatched** — no issue is claimed and no attempt is spent. Copying `agent.db`
|
|
169
|
+
into each run is deliberately not how this works: the harness rotates the OAuth
|
|
170
|
+
credentials in it, so a copy goes stale and can invalidate the original.
|
|
152
171
|
- `gh`, already authenticated: every tracker operation shells out to it, so the
|
|
153
172
|
daemon never handles a GitHub token itself.
|
|
154
173
|
- `git`: mirrors and worktrees.
|
|
@@ -1376,13 +1395,14 @@ persists the class on the row, and performs the one recovery that class names.
|
|
|
1376
1395
|
|
|
1377
1396
|
| Class | Signals | Recovery | Budget |
|
|
1378
1397
|
| --- | --- | --- | --- |
|
|
1398
|
+
| `env-start-failure` | turn 0 plus an explicit harness start error (`No model selected`, a rejected key) | escalate — the session never read the issue | none |
|
|
1379
1399
|
| `settlement-stuck` | a row carrying a PR that has since merged | settle: release the label, mark the row merged | none |
|
|
1380
1400
|
| `merge-conflict` | `pushed-green`, PR open, GitHub reports conflicting | requeue for a rebase continuation | continuation |
|
|
1381
1401
|
| `question` | the worker stopped to ask something (`blocked`) | escalate, carrying the worker's own report as evidence | none |
|
|
1382
1402
|
| `orphan-dirty` | orphaned with a failed salvage and no operator ack | hold — recorded only; the tree is the only copy | none |
|
|
1383
1403
|
| `orphan-clean` | orphaned with nothing uncommitted | requeue | continuation |
|
|
1384
1404
|
| `turn-cap-progress` | at the turn ceiling **with** a PR, head or salvage commit | continue from the branch | continuation |
|
|
1385
|
-
| `turn-cap-spinning` | at the ceiling with no PR and no commits | escalate with the last tool calls the transcript recorded | none |
|
|
1405
|
+
| `turn-cap-spinning` | at the ceiling with no PR and no commits | escalate with the last tool calls the transcript recorded — and the completion path deliberately does **not** requeue it | none |
|
|
1386
1406
|
| `admin-kill` | killed *below* its own ceiling — a restart or a drain | requeue | none |
|
|
1387
1407
|
| `ci-infra` | PR open, every unresolved check cancelled / timed out / stale | re-run the failed jobs | none |
|
|
1388
1408
|
| `ci-deterministic` | PR open, a check genuinely reports `FAILURE` | escalate with the failing check names and links | failed attempt |
|
|
@@ -1394,9 +1414,10 @@ cause nobody has named — the behaviour this exists to end.
|
|
|
1394
1414
|
|
|
1395
1415
|
### The budgets follow the cause
|
|
1396
1416
|
|
|
1397
|
-
`failuresFor` (implementation attempts) excludes `ci-infra`
|
|
1398
|
-
`
|
|
1399
|
-
`settlement-stuck
|
|
1417
|
+
`failuresFor` (implementation attempts) excludes `ci-infra`, `settlement-stuck`
|
|
1418
|
+
and `env-start-failure`; `continuationsFor` excludes `admin-kill`,
|
|
1419
|
+
`settlement-stuck` and `env-start-failure`. An environment fault charges neither:
|
|
1420
|
+
the session never started, so nothing about the issue was attempted. A merge conflict *is* charged as a continuation, because a
|
|
1400
1421
|
rebase is real work — just never as a failed implementation attempt.
|
|
1401
1422
|
|
|
1402
1423
|
An **unclassified** row (every row written before 0.4.3) counts exactly as it
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "omp-conductor",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.5",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
|
package/src/credentials.ts
CHANGED
|
@@ -37,6 +37,7 @@ import {
|
|
|
37
37
|
writeFileSync,
|
|
38
38
|
type Stats,
|
|
39
39
|
} from "node:fs";
|
|
40
|
+
import { homedir } from "node:os";
|
|
40
41
|
import { dirname, join, resolve } from "node:path";
|
|
41
42
|
|
|
42
43
|
import {
|
|
@@ -207,6 +208,159 @@ export function prepareSessionEnvRoot(root: string): SessionEnvRoot {
|
|
|
207
208
|
return paths;
|
|
208
209
|
}
|
|
209
210
|
|
|
211
|
+
/**
|
|
212
|
+
* Environment variables that carry a model credential directly.
|
|
213
|
+
*
|
|
214
|
+
* Deliberately NOT in {@link CREDENTIAL_ENV_KEYS}: those are the ones a session
|
|
215
|
+
* must never inherit. A model credential is the opposite — without one there is
|
|
216
|
+
* no session at all.
|
|
217
|
+
*/
|
|
218
|
+
export const MODEL_ENV_KEYS = [
|
|
219
|
+
"ANTHROPIC_API_KEY",
|
|
220
|
+
"OPENAI_API_KEY",
|
|
221
|
+
"OPENROUTER_API_KEY",
|
|
222
|
+
"XAI_API_KEY",
|
|
223
|
+
"GEMINI_API_KEY",
|
|
224
|
+
"GOOGLE_API_KEY",
|
|
225
|
+
] as const;
|
|
226
|
+
|
|
227
|
+
/** The harness's auth-broker connection, as its own env vars name it. */
|
|
228
|
+
export const BROKER_URL_ENV = "OMP_AUTH_BROKER_URL";
|
|
229
|
+
export const BROKER_TOKEN_ENV = "OMP_AUTH_BROKER_TOKEN";
|
|
230
|
+
|
|
231
|
+
export type ModelCredential =
|
|
232
|
+
/** Already in the daemon's environment, so it survives the redirect untouched. */
|
|
233
|
+
| { kind: "env" }
|
|
234
|
+
/** Resolved from the operator's own home and injected as broker env vars. */
|
|
235
|
+
| { kind: "broker"; url: string; token: string };
|
|
236
|
+
|
|
237
|
+
/**
|
|
238
|
+
* How a session under a redirected `$HOME` is going to authenticate a model —
|
|
239
|
+
* or `undefined` when it cannot, which is a fleet that must not dispatch.
|
|
240
|
+
*
|
|
241
|
+
* Sessions run with `$HOME` pointed at a per-run tree so the harness never reads
|
|
242
|
+
* the operator's configuration; that is what kept a GitHub MCP server carrying
|
|
243
|
+
* its own PAT out of every session. It also hid the harness's *own* credentials,
|
|
244
|
+
* so every worker died at turn 0 with `No model selected` and spent an issue
|
|
245
|
+
* attempt doing it — 27 recoveries on the reference fleet, while the fleet looked
|
|
246
|
+
* busy (#152).
|
|
247
|
+
*
|
|
248
|
+
* The fix is emphatically **not** to copy `<agentDir>/agent.db` into each run.
|
|
249
|
+
* That file holds OAuth credentials the harness rotates, plus threads, jobs and
|
|
250
|
+
* caches; the first run to refresh a token would update its own private copy and
|
|
251
|
+
* strand every later run — and the provider may invalidate the old refresh token
|
|
252
|
+
* as it rotates, so a copy can break the operator's own login too. The harness
|
|
253
|
+
* ships the right answer: an auth broker whose whole point is that *refresh
|
|
254
|
+
* tokens never leave it*, handing clients short-lived access tokens. That is also
|
|
255
|
+
* exactly the shape per-run isolation wants.
|
|
256
|
+
*
|
|
257
|
+
* Precedence mirrors the harness's own resolver, narrowly:
|
|
258
|
+
* 1. broker env vars, or any model API key, already in this process.
|
|
259
|
+
* 2. `auth.broker.url` / `auth.broker.token` in `<agentDir>/config.yml`.
|
|
260
|
+
* 3. `<configRoot>/auth-broker.token`, paired with a URL from 1 or 2.
|
|
261
|
+
*
|
|
262
|
+
* Re-read here rather than imported from the harness because this package must
|
|
263
|
+
* type-check and publish without its optional peer dependency — `omp.ts` is the
|
|
264
|
+
* only file allowed to touch it. `!command` indirection is deliberately not
|
|
265
|
+
* supported: an operator using that has the env-var path, which this honours.
|
|
266
|
+
*/
|
|
267
|
+
export function resolveModelCredential(
|
|
268
|
+
base: Readonly<Record<string, string | undefined>> = process.env,
|
|
269
|
+
agentDir = join(homedir(), ".omp", "agent"),
|
|
270
|
+
configRoot = join(homedir(), ".omp"),
|
|
271
|
+
): ModelCredential | undefined {
|
|
272
|
+
const envUrl = base[BROKER_URL_ENV];
|
|
273
|
+
const envToken = base[BROKER_TOKEN_ENV];
|
|
274
|
+
if ((envUrl ?? "").length > 0 && (envToken ?? "").length > 0) return { kind: "env" };
|
|
275
|
+
if (MODEL_ENV_KEYS.some((key) => (base[key] ?? "").length > 0)) return { kind: "env" };
|
|
276
|
+
|
|
277
|
+
const configured = readBrokerConfig(agentDir);
|
|
278
|
+
const url = (envUrl ?? "").length > 0 ? envUrl : configured.url;
|
|
279
|
+
if (url === undefined || url.length === 0) return undefined;
|
|
280
|
+
const token = configured.token ?? readTokenFile(join(configRoot, "auth-broker.token"));
|
|
281
|
+
if (token === undefined || token.length === 0) return undefined;
|
|
282
|
+
return { kind: "broker", url, token };
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
/**
|
|
286
|
+
* The two `auth.broker.*` values out of the harness's `config.yml`.
|
|
287
|
+
*
|
|
288
|
+
* A deliberately tiny reader rather than a YAML dependency: exactly two scalar
|
|
289
|
+
* keys under one mapping, and anything it cannot understand simply reads as
|
|
290
|
+
* absent — which lands on the same refusal as no broker at all, rather than a
|
|
291
|
+
* parse error during dispatch.
|
|
292
|
+
*/
|
|
293
|
+
function readBrokerConfig(agentDir: string): { url?: string; token?: string } {
|
|
294
|
+
for (const name of ["config.yml", "config.yaml"]) {
|
|
295
|
+
const path = join(agentDir, name);
|
|
296
|
+
if (!existsSync(path)) continue;
|
|
297
|
+
let text: string;
|
|
298
|
+
try {
|
|
299
|
+
text = readFileSync(path, "utf8");
|
|
300
|
+
} catch {
|
|
301
|
+
continue;
|
|
302
|
+
}
|
|
303
|
+
const found = scanBrokerKeys(text);
|
|
304
|
+
if (found.url !== undefined || found.token !== undefined) return found;
|
|
305
|
+
}
|
|
306
|
+
return {};
|
|
307
|
+
}
|
|
308
|
+
|
|
309
|
+
/**
|
|
310
|
+
* `auth.broker.url` / `auth.broker.token` out of a YAML document, in either the
|
|
311
|
+
* dotted or the nested shape the harness's own docs use.
|
|
312
|
+
*
|
|
313
|
+
* An indentation scan rather than a regex or a YAML dependency: two scalar keys
|
|
314
|
+
* under one known path, and anything it cannot understand reads as absent, which
|
|
315
|
+
* lands on the same refusal as no broker at all rather than a parse error during
|
|
316
|
+
* dispatch.
|
|
317
|
+
*/
|
|
318
|
+
function scanBrokerKeys(text: string): { url?: string; token?: string } {
|
|
319
|
+
const out: { url?: string; token?: string } = {};
|
|
320
|
+
/** Enclosing mapping keys, innermost last, by indentation. */
|
|
321
|
+
const path: { indent: number; key: string }[] = [];
|
|
322
|
+
for (const raw of text.split("\n")) {
|
|
323
|
+
const line = raw.replace(/\s+$/, "");
|
|
324
|
+
if (line.trim().length === 0 || line.trim().startsWith("#")) continue;
|
|
325
|
+
const indent = line.length - line.trimStart().length;
|
|
326
|
+
const at = line.indexOf(":");
|
|
327
|
+
if (at < 0) continue;
|
|
328
|
+
const key = line.slice(indent, at).trim();
|
|
329
|
+
const value = line.slice(at + 1);
|
|
330
|
+
while (path.length > 0 && (path.at(-1) as { indent: number }).indent >= indent) path.pop();
|
|
331
|
+
|
|
332
|
+
const dotted = key.split(".");
|
|
333
|
+
const full = [...path.map((p) => p.key), ...dotted];
|
|
334
|
+
if (value.trim().length === 0) {
|
|
335
|
+
path.push({ indent, key: dotted.join(".") });
|
|
336
|
+
continue;
|
|
337
|
+
}
|
|
338
|
+
if (full.length !== 3 || full[0] !== "auth" || full[1] !== "broker") continue;
|
|
339
|
+
const scalar = unquote(value);
|
|
340
|
+
if (scalar.length === 0) continue;
|
|
341
|
+
if (full[2] === "url") out.url ??= scalar;
|
|
342
|
+
else if (full[2] === "token") out.token ??= scalar;
|
|
343
|
+
}
|
|
344
|
+
return out;
|
|
345
|
+
}
|
|
346
|
+
|
|
347
|
+
function unquote(raw: string): string {
|
|
348
|
+
const text = raw.trim();
|
|
349
|
+
if (text.startsWith("!")) return "";
|
|
350
|
+
const quoted = /^(['"])(.*)\1$/.exec(text);
|
|
351
|
+
return (quoted?.[2] ?? text).trim();
|
|
352
|
+
}
|
|
353
|
+
|
|
354
|
+
function readTokenFile(path: string): string | undefined {
|
|
355
|
+
if (!existsSync(path)) return undefined;
|
|
356
|
+
try {
|
|
357
|
+
const text = readFileSync(path, "utf8").trim();
|
|
358
|
+
return text.length === 0 ? undefined : text;
|
|
359
|
+
} catch {
|
|
360
|
+
return undefined;
|
|
361
|
+
}
|
|
362
|
+
}
|
|
363
|
+
|
|
210
364
|
/**
|
|
211
365
|
* The environment one session child runs with.
|
|
212
366
|
*
|
|
@@ -222,7 +376,7 @@ export function prepareSessionEnvRoot(root: string): SessionEnvRoot {
|
|
|
222
376
|
*/
|
|
223
377
|
export function sessionEnv(
|
|
224
378
|
paths: SessionEnvRoot,
|
|
225
|
-
opts: { readToken?: string } = {},
|
|
379
|
+
opts: { readToken?: string; model?: ModelCredential } = {},
|
|
226
380
|
base: Readonly<Record<string, string | undefined>> = process.env,
|
|
227
381
|
): Record<string, string> {
|
|
228
382
|
const env: Record<string, string> = {};
|
|
@@ -244,6 +398,16 @@ export function sessionEnv(
|
|
|
244
398
|
env["NPM_CONFIG_USERCONFIG"] = paths.npmrc;
|
|
245
399
|
env["TMPDIR"] = paths.tmp;
|
|
246
400
|
|
|
401
|
+
// The redirect hides the operator's `config.yml` and token file, so a broker
|
|
402
|
+
// configured *there* has to be handed over explicitly or the session falls back
|
|
403
|
+
// to an empty local store and dies at turn 0 (#152). Env vars are the harness's
|
|
404
|
+
// own highest-precedence shape, and they carry an access-minting token rather
|
|
405
|
+
// than a refresh credential: the broker keeps that.
|
|
406
|
+
if (opts.model?.kind === "broker") {
|
|
407
|
+
env[BROKER_URL_ENV] = opts.model.url;
|
|
408
|
+
env[BROKER_TOKEN_ENV] = opts.model.token;
|
|
409
|
+
}
|
|
410
|
+
|
|
247
411
|
// The one credential a session may be given, and only when an operator
|
|
248
412
|
// configured it. Read-scoped by construction on GitHub's side — this package
|
|
249
413
|
// cannot verify the scope, so the README says plainly that a write-scoped
|
|
@@ -1620,6 +1784,13 @@ export interface BoundaryRequest {
|
|
|
1620
1784
|
denyReadRoots?: readonly string[];
|
|
1621
1785
|
denyReadFiles?: readonly string[];
|
|
1622
1786
|
readToken?: string;
|
|
1787
|
+
/**
|
|
1788
|
+
* How this session authenticates a model. Resolved once by the daemon and
|
|
1789
|
+
* handed down rather than re-read per session, so every child of one tick
|
|
1790
|
+
* agrees — and so a fleet with no credential is refused at admission instead of
|
|
1791
|
+
* discovering it at turn 0 (#152).
|
|
1792
|
+
*/
|
|
1793
|
+
model?: ModelCredential;
|
|
1623
1794
|
/** Where a generated `sandbox-exec` profile is written. */
|
|
1624
1795
|
profilePath?: string;
|
|
1625
1796
|
}
|
|
@@ -1654,7 +1825,10 @@ export function mechanismSatisfies(
|
|
|
1654
1825
|
*/
|
|
1655
1826
|
export function buildSessionBoundary(req: BoundaryRequest): SessionBoundary {
|
|
1656
1827
|
const paths = prepareSessionEnvRoot(req.envRoot);
|
|
1657
|
-
const env = sessionEnv(paths,
|
|
1828
|
+
const env = sessionEnv(paths, {
|
|
1829
|
+
...(req.readToken === undefined ? {} : { readToken: req.readToken }),
|
|
1830
|
+
...(req.model === undefined ? {} : { model: req.model }),
|
|
1831
|
+
});
|
|
1658
1832
|
|
|
1659
1833
|
if (req.isolation === "none") {
|
|
1660
1834
|
return { mechanism: "none", launcher: [], env };
|
package/src/daemon.ts
CHANGED
|
@@ -107,8 +107,10 @@ import {
|
|
|
107
107
|
openRunPr,
|
|
108
108
|
probeHost,
|
|
109
109
|
pushRunBranch,
|
|
110
|
+
resolveModelCredential,
|
|
110
111
|
verifyDaemonAccess,
|
|
111
112
|
type HostProbe,
|
|
113
|
+
type ModelCredential,
|
|
112
114
|
type RunRepoRef,
|
|
113
115
|
type SessionBoundary,
|
|
114
116
|
type SlotPool,
|
|
@@ -198,6 +200,17 @@ interface Deps {
|
|
|
198
200
|
integrity: IntegrityGate;
|
|
199
201
|
stall: StallGate;
|
|
200
202
|
cleanup?: RetainedCleanupCursor;
|
|
203
|
+
/**
|
|
204
|
+
* How sessions authenticate a model, resolved once at startup so every child of
|
|
205
|
+
* one tick agrees — and so a fleet with none is refused at admission instead of
|
|
206
|
+
* discovering it at turn 0 (#152).
|
|
207
|
+
*
|
|
208
|
+
* Optional exactly as {@link Deps.boundary} is: absent means no resolution was
|
|
209
|
+
* made, which is the shape a unit test that does not exercise this gate wants.
|
|
210
|
+
* `runDaemon` always resolves one. `paged` is the page-once latch, so a fleet
|
|
211
|
+
* held for an environment fault says so once rather than every five minutes.
|
|
212
|
+
*/
|
|
213
|
+
model?: { credential?: ModelCredential; paged: boolean };
|
|
201
214
|
/**
|
|
202
215
|
* Reads the connecting uid off a verb socket (#126). Resolved once at startup
|
|
203
216
|
* so the mechanism is logged before the first socket exists; absent on a host
|
|
@@ -903,6 +916,38 @@ export async function collectSettlementFlags(
|
|
|
903
916
|
* One attempt at one issue, from claim to terminal state. Everything is inside
|
|
904
917
|
* a single try/catch so that a bad issue costs its own run and nothing else.
|
|
905
918
|
*/
|
|
919
|
+
/**
|
|
920
|
+
* Whether a turns-cap kill is handed straight back to the queue.
|
|
921
|
+
*
|
|
922
|
+
* A turns cap consumes the independent continuation budget rather than an
|
|
923
|
+
* implementation-failure attempt, so a run that ran out of turns mid-work is
|
|
924
|
+
* requeued and the next tick reattaches its branch (#50).
|
|
925
|
+
*
|
|
926
|
+
* The artifact test is what keeps that from becoming a loop. A run that reached
|
|
927
|
+
* the ceiling with no PR, no observed head and no salvage commit produced
|
|
928
|
+
* nothing to continue *from*, and requeueing it spends another continuation on
|
|
929
|
+
* the same spin — the seven rows on this fleet that burned $109 producing no
|
|
930
|
+
* merged PR. That shape is `turn-cap-spinning`, which #132 says must escalate
|
|
931
|
+
* with evidence instead; requeueing it here would settle the question before the
|
|
932
|
+
* classifier ever saw the row, and the queue label would already be back on.
|
|
933
|
+
*
|
|
934
|
+
* Pure so the rule is testable without a worker session — the same split as
|
|
935
|
+
* {@link settlementFor}.
|
|
936
|
+
*/
|
|
937
|
+
export function shouldContinueAfterTurnsCap(f: {
|
|
938
|
+
killedBy?: string;
|
|
939
|
+
prUrl?: string;
|
|
940
|
+
headSha?: string;
|
|
941
|
+
salvageSha?: string;
|
|
942
|
+
continuation: number;
|
|
943
|
+
maxContinuations: number;
|
|
944
|
+
}): boolean {
|
|
945
|
+
if (f.killedBy !== "turns") return false;
|
|
946
|
+
const hasArtifacts = f.prUrl !== undefined || f.headSha !== undefined || f.salvageSha !== undefined;
|
|
947
|
+
if (!hasArtifacts) return false;
|
|
948
|
+
return hasContinuationBudget(f.continuation, f.maxContinuations);
|
|
949
|
+
}
|
|
950
|
+
|
|
906
951
|
async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
907
952
|
const { project, caps, tracker, store } = d;
|
|
908
953
|
const issue = r.issue.number;
|
|
@@ -1033,6 +1078,10 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
1033
1078
|
denyReadRoots: [...credentialDenyRoots(), project.workspaceRoot],
|
|
1034
1079
|
denyReadFiles: credentialDenyFiles(),
|
|
1035
1080
|
...(credentials.readToken === undefined ? {} : { readToken: credentials.readToken }),
|
|
1081
|
+
// Resolved from the operator's own home at startup, because the session's
|
|
1082
|
+
// redirected $HOME cannot see it (#152). Admission already refused the whole
|
|
1083
|
+
// pass when it is absent, so undefined here is a fleet not dispatching.
|
|
1084
|
+
...(d.model?.credential === undefined ? {} : { model: d.model.credential }),
|
|
1036
1085
|
});
|
|
1037
1086
|
|
|
1038
1087
|
if (boundary.principal !== undefined) {
|
|
@@ -1299,13 +1348,17 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
1299
1348
|
detail: [`${r.issue.title}`, r.issue.url, "", ...salvaged, "", result.report].join("\n"),
|
|
1300
1349
|
});
|
|
1301
1350
|
} else if (state === "failed" || state === "killed") {
|
|
1302
|
-
//
|
|
1303
|
-
//
|
|
1304
|
-
// count includes the segment that just ended.
|
|
1351
|
+
// The row is already `killed`, so this count includes the segment that
|
|
1352
|
+
// just ended.
|
|
1305
1353
|
const continuation = store.continuationsFor(project.name, issue);
|
|
1306
|
-
const continueTurns =
|
|
1307
|
-
result.killedBy
|
|
1308
|
-
|
|
1354
|
+
const continueTurns = shouldContinueAfterTurnsCap({
|
|
1355
|
+
killedBy: result.killedBy,
|
|
1356
|
+
prUrl: result.prUrl,
|
|
1357
|
+
headSha: result.headSha,
|
|
1358
|
+
salvageSha: settlement?.patch?.salvageSha,
|
|
1359
|
+
continuation,
|
|
1360
|
+
maxContinuations: caps.maxContinuationsPerIssue,
|
|
1361
|
+
});
|
|
1309
1362
|
|
|
1310
1363
|
if (continueTurns) {
|
|
1311
1364
|
await tracker.removeLabel(issue, inProgress);
|
|
@@ -1895,7 +1948,7 @@ function planUsageEscalation(project: string, plan: PlanUsageStatus): Escalation
|
|
|
1895
1948
|
* that grows a field has no business breaking these tests.
|
|
1896
1949
|
*/
|
|
1897
1950
|
export async function admitCandidates(
|
|
1898
|
-
d: Pick<Deps, "project" | "caps" | "tracker" | "store" | "escalate" | "usage" | "boundary">,
|
|
1951
|
+
d: Pick<Deps, "project" | "caps" | "tracker" | "store" | "escalate" | "usage" | "boundary" | "model">,
|
|
1899
1952
|
routed: Routed[],
|
|
1900
1953
|
slots: number,
|
|
1901
1954
|
): Promise<AdmissionPass> {
|
|
@@ -1932,6 +1985,48 @@ export async function admitCandidates(
|
|
|
1932
1985
|
return { admitted: [], holds };
|
|
1933
1986
|
}
|
|
1934
1987
|
|
|
1988
|
+
// One layer down from the boundary above: that asks whether this host can run a
|
|
1989
|
+
// session as its own principal, this asks whether a session can do anything at
|
|
1990
|
+
// all once started. Every worker on the reference fleet died at turn 0 with
|
|
1991
|
+
// "No model selected" and consumed an issue attempt for it — three issues in
|
|
1992
|
+
// eleven minutes, while the fleet looked busy (#152).
|
|
1993
|
+
//
|
|
1994
|
+
// Above the tracker calls for the same reason as the boundary: a fleet that
|
|
1995
|
+
// cannot dispatch must not claim an issue it will fail, and must not spend API
|
|
1996
|
+
// budget discovering that.
|
|
1997
|
+
if (d.model !== undefined && d.model.credential === undefined) {
|
|
1998
|
+
for (const r of routed) hold(r.issue.number, "no-model-credential");
|
|
1999
|
+
log(`no model credential holding ${String(routed.length)} candidate(s)`);
|
|
2000
|
+
if (!d.model.paged) {
|
|
2001
|
+
d.model.paged = true;
|
|
2002
|
+
await safeEscalate(d, {
|
|
2003
|
+
tier: 2,
|
|
2004
|
+
project: project.name,
|
|
2005
|
+
issue: NO_ISSUE,
|
|
2006
|
+
summary: `${project.name} cannot dispatch: no model credential is reachable from a worker session`,
|
|
2007
|
+
detail: [
|
|
2008
|
+
"Sessions run with $HOME redirected into a per-run tree, so the harness never reads the",
|
|
2009
|
+
"operator's own configuration — that is what keeps a credential-bearing MCP server out of",
|
|
2010
|
+
"every session. It also means a login recorded only in ~/.omp/agent/agent.db is invisible to",
|
|
2011
|
+
"a worker, which then dies at turn 0 with \"No model selected\".",
|
|
2012
|
+
"",
|
|
2013
|
+
"Two supported shapes, either of which fixes it:",
|
|
2014
|
+
" * a model API key in the daemon's environment (ANTHROPIC_API_KEY, OPENAI_API_KEY, …)",
|
|
2015
|
+
" * the harness's auth broker — `omp auth-broker serve` plus `omp auth-broker token`, or",
|
|
2016
|
+
" auth.broker.url / auth.broker.token in ~/.omp/agent/config.yml. Refresh tokens stay in",
|
|
2017
|
+
" the broker and each session gets a short-lived access token, which is what per-run",
|
|
2018
|
+
" isolation wants.",
|
|
2019
|
+
"",
|
|
2020
|
+
"Copying agent.db into each run is deliberately NOT how this works: the harness rotates the",
|
|
2021
|
+
"OAuth credentials in it, so a copy goes stale and can invalidate the original.",
|
|
2022
|
+
"",
|
|
2023
|
+
"Nothing is dispatched until then. No work has been claimed, and no attempt was spent.",
|
|
2024
|
+
].join("\n"),
|
|
2025
|
+
});
|
|
2026
|
+
}
|
|
2027
|
+
return { admitted: [], holds };
|
|
2028
|
+
}
|
|
2029
|
+
|
|
1935
2030
|
// The plan allowance is a fleet-wide question, so it is asked once per pass
|
|
1936
2031
|
// and answers for every candidate — unlike every gate below it, which is
|
|
1937
2032
|
// per-issue. It sits here rather than beside the spend cap in `tick` for one
|
|
@@ -2891,8 +2986,13 @@ export async function classifyAndRecover(d: Deps): Promise<void> {
|
|
|
2891
2986
|
// base does move under it.
|
|
2892
2987
|
if (run.state === "pushed-green" && cls === "unknown") continue;
|
|
2893
2988
|
|
|
2989
|
+
const retry = run.failureClass !== undefined;
|
|
2894
2990
|
store.updateRun(run.id, { failureClass: cls, recoveryAction: recovery });
|
|
2895
|
-
log(
|
|
2991
|
+
log(
|
|
2992
|
+
retry
|
|
2993
|
+
? `#${run.issue} retrying ${recovery} for ${cls}: ${evidence}`
|
|
2994
|
+
: `#${run.issue} classified ${cls} → ${recovery}: ${evidence}`,
|
|
2995
|
+
);
|
|
2896
2996
|
await recoverRun(d, run, cls, recovery, evidence);
|
|
2897
2997
|
}
|
|
2898
2998
|
}
|
|
@@ -2920,15 +3020,38 @@ async function recoverRun(
|
|
|
2920
3020
|
}
|
|
2921
3021
|
|
|
2922
3022
|
if (recovery === "continue") {
|
|
2923
|
-
//
|
|
2924
|
-
//
|
|
3023
|
+
// Two classes recover by continuing, and only one of them has anything left
|
|
3024
|
+
// to do here.
|
|
3025
|
+
//
|
|
3026
|
+
// `turn-cap-progress` was already handed back by the completion path, which
|
|
3027
|
+
// swapped its labels and left the branch retained. There is nothing to
|
|
3028
|
+
// perform, and writing anything would be actively wrong: overwriting
|
|
3029
|
+
// `lastError` with a rebase brief tells the continuation worker to rebase a
|
|
3030
|
+
// run that simply ran out of turns, and re-swapping labels the completion
|
|
3031
|
+
// path already swapped is a pair of no-op `gh` calls. Record-only, so the
|
|
3032
|
+
// sweep stops re-offering it.
|
|
3033
|
+
if (cls === "turn-cap-progress") {
|
|
3034
|
+
store.updateRun(run.id, { recoveredAt: Date.now() });
|
|
3035
|
+
log(`#${run.issue} already continuing from ${cls}: ${evidence}`);
|
|
3036
|
+
return;
|
|
3037
|
+
}
|
|
3038
|
+
|
|
3039
|
+
// `merge-conflict`: the branch is retained and its PR is open, so #50's
|
|
3040
|
+
// continuation guard admits it and the next tick briefs a rebase.
|
|
3041
|
+
//
|
|
3042
|
+
// The tracker write goes FIRST, and that ordering is the whole retry
|
|
3043
|
+
// contract. Writing `recoveredAt` before the swap took the row out of
|
|
3044
|
+
// `runsNeedingClassification` — which selects on `recoveredAt IS NULL` — so a
|
|
3045
|
+
// refused label swap stranded the row permanently under a log line promising
|
|
3046
|
+
// a retry. That was the defect 0.4.4 claimed to have fixed and did not, for
|
|
3047
|
+
// this one recovery.
|
|
3048
|
+
if (!(await swapToQueue(d, run.issue, inProgress))) return;
|
|
2925
3049
|
store.updateRun(run.id, {
|
|
2926
3050
|
state: "killed",
|
|
2927
3051
|
lastError:
|
|
2928
3052
|
"merge-conflict: base moved under a green PR — rebase, regenerate recorded artifacts, push without force",
|
|
2929
3053
|
recoveredAt: Date.now(),
|
|
2930
3054
|
});
|
|
2931
|
-
if (!(await swapToQueue(d, run.issue, inProgress))) return;
|
|
2932
3055
|
log(`#${run.issue} requeued for a rebase continuation: ${evidence}`);
|
|
2933
3056
|
return;
|
|
2934
3057
|
}
|
|
@@ -3212,6 +3335,19 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
|
|
|
3212
3335
|
// the other thing — the package changing while the daemon that dispatches
|
|
3213
3336
|
// work is holding it open, whether that is a worker that wandered out of its
|
|
3214
3337
|
// worktree or a human editing the live install "just to test something".
|
|
3338
|
+
// Resolved once, here, for the same reason the host probe above is: every
|
|
3339
|
+
// session of every tick must agree, and a fleet with no way to authenticate a
|
|
3340
|
+
// model must refuse at admission rather than burn an attempt per dispatch
|
|
3341
|
+
// (#152).
|
|
3342
|
+
const model: { credential?: ModelCredential; paged: boolean } = { paged: false };
|
|
3343
|
+
const resolvedModel = resolveModelCredential();
|
|
3344
|
+
if (resolvedModel !== undefined) model.credential = resolvedModel;
|
|
3345
|
+
log(
|
|
3346
|
+
resolvedModel === undefined
|
|
3347
|
+
? "model credential: NONE reachable from a session — dispatch is held until an API key or an auth broker is configured"
|
|
3348
|
+
: `model credential: ${resolvedModel.kind === "env" ? "from this daemon's environment" : "auth broker, injected per session"}`,
|
|
3349
|
+
);
|
|
3350
|
+
|
|
3215
3351
|
const integrity: IntegrityGate = { baseline: packageManifest(), paged: false };
|
|
3216
3352
|
log(`package integrity baseline: ${integrity.baseline.size} files under ${import.meta.dir}`);
|
|
3217
3353
|
|
|
@@ -3324,6 +3460,7 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
|
|
|
3324
3460
|
denyReadRoots: [...credentialDenyRoots(), project.workspaceRoot, project.mirrorRoot],
|
|
3325
3461
|
denyReadFiles: credentialDenyFiles(),
|
|
3326
3462
|
...(credentials.readToken === undefined ? {} : { readToken: credentials.readToken }),
|
|
3463
|
+
...(model.credential === undefined ? {} : { model: model.credential }),
|
|
3327
3464
|
});
|
|
3328
3465
|
if (orchestratorBoundary.principal !== undefined) {
|
|
3329
3466
|
const gid = probe.daemonGid;
|
|
@@ -3425,6 +3562,7 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
|
|
|
3425
3562
|
project,
|
|
3426
3563
|
caps,
|
|
3427
3564
|
boundary,
|
|
3565
|
+
model,
|
|
3428
3566
|
tracker,
|
|
3429
3567
|
store,
|
|
3430
3568
|
// Process-wide, so a `status` served off this daemon's own HTTP surface
|
package/src/failure-class.ts
CHANGED
|
@@ -62,6 +62,29 @@ function normalise(state: string): string {
|
|
|
62
62
|
* makes every other reading of the row wrong — that is the #362 case, where a
|
|
63
63
|
* `pushed-green` row outlived its own PR and held the issue out of dispatch.
|
|
64
64
|
*/
|
|
65
|
+
/**
|
|
66
|
+
* Harness start failures, matched on the text it prints.
|
|
67
|
+
*
|
|
68
|
+
* Deliberately a small closed list rather than "anything at turn 0": a run that
|
|
69
|
+
* ended at turn 0 for a reason nobody has named is exactly what `unknown` is for,
|
|
70
|
+
* and quietly declaring it an environment fault would waive an attempt that may
|
|
71
|
+
* have been genuinely spent.
|
|
72
|
+
*/
|
|
73
|
+
const START_FAILURE_SIGNATURES = [
|
|
74
|
+
"no model selected",
|
|
75
|
+
"no model configured",
|
|
76
|
+
"invalid api key",
|
|
77
|
+
"authentication failed",
|
|
78
|
+
"could not load its peer dependency",
|
|
79
|
+
] as const;
|
|
80
|
+
|
|
81
|
+
function startFailure(lastError: string | undefined): string | undefined {
|
|
82
|
+
if (lastError === undefined) return undefined;
|
|
83
|
+
const text = lastError.toLowerCase();
|
|
84
|
+
const hit = START_FAILURE_SIGNATURES.find((signature) => text.includes(signature));
|
|
85
|
+
return hit === undefined ? undefined : lastError.split("\n")[0]?.trim();
|
|
86
|
+
}
|
|
87
|
+
|
|
65
88
|
export function classifyRun(run: RunRecord, facts: ClassifyFacts): Classification {
|
|
66
89
|
const hasArtifacts = run.prUrl !== undefined || run.headSha !== undefined || run.salvageSha !== undefined;
|
|
67
90
|
|
|
@@ -85,6 +108,22 @@ export function classifyRun(run: RunRecord, facts: ClassifyFacts): Classificatio
|
|
|
85
108
|
};
|
|
86
109
|
}
|
|
87
110
|
|
|
111
|
+
// Turn zero with an explicit harness start error is the most classifiable
|
|
112
|
+
// failure there is, and the least deserving of an implementation attempt: the
|
|
113
|
+
// session never got as far as reading the issue. 27 recoveries on the reference
|
|
114
|
+
// fleet were this, logged as `unknown` while the session's own output carried
|
|
115
|
+
// the literal string `No model selected` (#152).
|
|
116
|
+
if ((run.state === "failed" || run.state === "killed") && run.turns === 0) {
|
|
117
|
+
const detail = startFailure(run.lastError);
|
|
118
|
+
if (detail !== undefined) {
|
|
119
|
+
return {
|
|
120
|
+
cls: "env-start-failure",
|
|
121
|
+
recovery: "escalate",
|
|
122
|
+
evidence: `the session never started: ${detail}`,
|
|
123
|
+
};
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
|
|
88
127
|
if (run.state === "blocked") {
|
|
89
128
|
return {
|
|
90
129
|
cls: "question",
|
package/src/plugin.ts
CHANGED
|
@@ -70,6 +70,7 @@ import {
|
|
|
70
70
|
detectTelegram,
|
|
71
71
|
formatGates,
|
|
72
72
|
orchestratorBriefPath,
|
|
73
|
+
relocateBriefIfMoved,
|
|
73
74
|
planLabels,
|
|
74
75
|
renderBriefForProject,
|
|
75
76
|
summariseAmend,
|
|
@@ -1166,9 +1167,20 @@ async function setup(ctx: CommandContext, projectArg: string | undefined): Promi
|
|
|
1166
1167
|
const telegram = detectTelegram();
|
|
1167
1168
|
const nextConfig = buildConfig(answers, existing);
|
|
1168
1169
|
const project = findProject(nextConfig, answers.projectName);
|
|
1170
|
+
// The same project as it is configured right now, so a moved `workspaceRoot`
|
|
1171
|
+
// can be told apart from a fleet that never had a brief. Switching
|
|
1172
|
+
// `credentials.isolation` moves that root (0700 state dir ⇄ shared root), which
|
|
1173
|
+
// relocates both briefs.
|
|
1174
|
+
const previous = existing?.projects.find((p) => p.name === answers.projectName);
|
|
1175
|
+
const relocating =
|
|
1176
|
+
previous !== undefined &&
|
|
1177
|
+
previous.workspaceRoot !== project.workspaceRoot &&
|
|
1178
|
+
existsSync(policyPathForProject(previous)) &&
|
|
1179
|
+
!existsSync(policyPathForProject(project));
|
|
1169
1180
|
if (
|
|
1170
1181
|
project.escalation.orchestrator === "external" &&
|
|
1171
1182
|
!answers.writeOrchestratorBrief &&
|
|
1183
|
+
!relocating &&
|
|
1172
1184
|
(!existsSync(briefPathForProject(project)) || !existsSync(policyPathForProject(project)))
|
|
1173
1185
|
) {
|
|
1174
1186
|
ctx.ui.notify(
|
|
@@ -1231,6 +1243,10 @@ async function setup(ctx: CommandContext, projectArg: string | undefined): Promi
|
|
|
1231
1243
|
answers.writeOrchestratorBrief
|
|
1232
1244
|
? `Writes ${orchestratorBriefPath(answers)}, which is then yours to edit.`
|
|
1233
1245
|
: "",
|
|
1246
|
+
relocating
|
|
1247
|
+
? `Moves your ${POLICY_BRIEF_NAME} to ${policyPathForProject(project)} (the isolation change moves the workspace root) ` +
|
|
1248
|
+
`and recomposes ${ORCHESTRATOR_BRIEF_NAME} beside it. The copy under ${policyPathForProject(previous)} is left in place.`
|
|
1249
|
+
: "",
|
|
1234
1250
|
project.escalation.orchestrator === "external"
|
|
1235
1251
|
? "Dispatch stays paused until the existing arm marker or a new inbound Telegram proof makes the heartbeat live."
|
|
1236
1252
|
: "Dispatch resumes after the smoke succeeds.",
|
|
@@ -1249,6 +1265,13 @@ async function setup(ctx: CommandContext, projectArg: string | undefined): Promi
|
|
|
1249
1265
|
prepareConductor();
|
|
1250
1266
|
const created = await createMissingLabels(answers.trackerRepo, labels);
|
|
1251
1267
|
saveConfig(nextConfig);
|
|
1268
|
+
// Before the optional scaffold write, and deliberately: the scaffold renders a
|
|
1269
|
+
// fresh POLICY.md, so writing it first would replace the policy this is trying
|
|
1270
|
+
// to carry across.
|
|
1271
|
+
const moved = relocateBriefIfMoved(previous, project);
|
|
1272
|
+
if (moved !== undefined) {
|
|
1273
|
+
ctx.ui.notify(`Moved ${POLICY_BRIEF_NAME} to ${moved.to} — the previous copy is still at ${moved.from}.`, "info");
|
|
1274
|
+
}
|
|
1252
1275
|
const briefPath = answers.writeOrchestratorBrief ? writeOrchestratorBrief(answers) : undefined;
|
|
1253
1276
|
const runtimeFiles = writeHostRuntime(runtime);
|
|
1254
1277
|
const smoke = await runSetupSmoke(project.name);
|
package/src/setup-host.ts
CHANGED
|
@@ -18,7 +18,7 @@ import {
|
|
|
18
18
|
TICK_CONFIG_FILE,
|
|
19
19
|
type TickConfig,
|
|
20
20
|
} from "./orchestrator-tick.ts";
|
|
21
|
-
import type
|
|
21
|
+
import { CONDUCTOR_GROUPS, type Caps, type ProjectConfig } from "./types.ts";
|
|
22
22
|
|
|
23
23
|
export const DEFAULT_TICK_INTERVAL_SECONDS = 900;
|
|
24
24
|
export const STAGED_SERVICE_NAME = "omp-conductor.service";
|
|
@@ -125,9 +125,20 @@ export function renderDaemonService(
|
|
|
125
125
|
// daemon for nothing. Without them the probe honestly reports `group-mode`
|
|
126
126
|
// and a configured per-run fleet refuses every dispatch, which is the
|
|
127
127
|
// failure an operator following setup would otherwise hit first.
|
|
128
|
+
//
|
|
129
|
+
// `SupplementaryGroups=` is declared rather than left to initgroups, and that
|
|
130
|
+
// is not belt-and-braces. Supplementary groups are fixed when a process
|
|
131
|
+
// starts, so a daemon that was running when the accounts were provisioned can
|
|
132
|
+
// never join them — and a unit with no `User=` gets no initgroups call at
|
|
133
|
+
// all, so it comes up with `Groups: 0` even after the group database is
|
|
134
|
+
// correct. `probeHost` then reports `none` forever, the wizard cannot offer
|
|
135
|
+
// `uid-pool`, and the fleet quietly keeps the fail-closed value (#153).
|
|
136
|
+
// Declaring the membership makes it a property of the unit instead of a
|
|
137
|
+
// property of how the unit happened to be written.
|
|
128
138
|
...(resolveCredentials(project).isolation === "per-run"
|
|
129
139
|
? [
|
|
130
140
|
`Environment=${systemdQuote(`OMP_CONDUCTOR_SHARED=${sharedRoot()}`)}`,
|
|
141
|
+
`SupplementaryGroups=${CONDUCTOR_GROUPS.daemon} ${CONDUCTOR_GROUPS.runs}`,
|
|
131
142
|
"AmbientCapabilities=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_FOWNER CAP_SETPCAP",
|
|
132
143
|
"CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_FOWNER CAP_SETPCAP",
|
|
133
144
|
]
|
|
@@ -343,11 +354,18 @@ export interface EffectiveUnit {
|
|
|
343
354
|
/** `systemctl show -p AmbientCapabilities`; empty string when unset. */
|
|
344
355
|
ambientCapabilities: string;
|
|
345
356
|
capabilityBoundingSet: string;
|
|
357
|
+
/** `systemctl show -p SupplementaryGroups`; empty string when unset. */
|
|
358
|
+
supplementaryGroups: string;
|
|
346
359
|
/** `systemctl show -p Environment`, newline- or space-joined. */
|
|
347
360
|
environment: string;
|
|
348
361
|
needDaemonReload: boolean;
|
|
349
362
|
}
|
|
350
363
|
|
|
364
|
+
/** Whitespace-separated names as a comparable set; systemd reorders them. */
|
|
365
|
+
function nameSet(text: string): Set<string> {
|
|
366
|
+
return new Set(text.split(/[\s,]+/).map((n) => n.trim()).filter((n) => n.length > 0));
|
|
367
|
+
}
|
|
368
|
+
|
|
351
369
|
/** Capability names as a comparable set: systemd reports them lowercased, `cap_`-prefixed and reordered. */
|
|
352
370
|
function capabilitySet(text: string): Set<string> {
|
|
353
371
|
return new Set(
|
|
@@ -387,6 +405,21 @@ export function unitDriftReason(rendered: string, effective: EffectiveUnit | und
|
|
|
387
405
|
if (effective === undefined) return undefined;
|
|
388
406
|
const problems: string[] = [];
|
|
389
407
|
|
|
408
|
+
// Missing groups only, never extra: an operator may legitimately add their own,
|
|
409
|
+
// and unlike a leftover capability a spare group grants nothing this package
|
|
410
|
+
// relies on. Missing ones are load-bearing — without them the probe resolves to
|
|
411
|
+
// `none` and a configured per-run fleet refuses every dispatch (#153).
|
|
412
|
+
const wantGroups = nameSet(renderedDirective(rendered, "SupplementaryGroups") ?? "");
|
|
413
|
+
const haveGroups = nameSet(effective.supplementaryGroups);
|
|
414
|
+
const missingGroups = [...wantGroups].filter((g) => !haveGroups.has(g));
|
|
415
|
+
if (missingGroups.length > 0) {
|
|
416
|
+
problems.push(
|
|
417
|
+
`SupplementaryGroups is missing ${missingGroups.join(", ")} — supplementary groups are fixed at ` +
|
|
418
|
+
`process start, so the running daemon cannot be a member and the credential boundary probe ` +
|
|
419
|
+
`resolves to none until the unit is reloaded and the service restarted`,
|
|
420
|
+
);
|
|
421
|
+
}
|
|
422
|
+
|
|
390
423
|
const wantAmbient = capabilitySet(renderedDirective(rendered, "AmbientCapabilities") ?? "");
|
|
391
424
|
const haveAmbient = capabilitySet(effective.ambientCapabilities);
|
|
392
425
|
const missingAmbient = [...wantAmbient].filter((c) => !haveAmbient.has(c));
|
|
@@ -440,6 +473,7 @@ export function parseEffectiveUnit(stdout: string): EffectiveUnit {
|
|
|
440
473
|
return {
|
|
441
474
|
ambientCapabilities: values.get("AmbientCapabilities") ?? "",
|
|
442
475
|
capabilityBoundingSet: values.get("CapabilityBoundingSet") ?? "",
|
|
476
|
+
supplementaryGroups: values.get("SupplementaryGroups") ?? "",
|
|
443
477
|
environment: values.get("Environment") ?? "",
|
|
444
478
|
needDaemonReload: (values.get("NeedDaemonReload") ?? "no") === "yes",
|
|
445
479
|
};
|
package/src/setup.ts
CHANGED
|
@@ -808,6 +808,46 @@ export function renderOrchestratorBrief(a: SetupAnswers): string {
|
|
|
808
808
|
* that dialog's answer un-actionable — an operator who says "yes, overwrite it"
|
|
809
809
|
* must get an overwrite.
|
|
810
810
|
*/
|
|
811
|
+
/**
|
|
812
|
+
* Carry a fleet's own `POLICY.md` to a workspace root that just moved.
|
|
813
|
+
*
|
|
814
|
+
* `workspaceRoot` is derived from `credentials.isolation`: `none` puts it under
|
|
815
|
+
* the 0700 state directory, `per-run` under the shared root a slot principal can
|
|
816
|
+
* traverse. So switching a fleet to worker isolation *relocates* the briefs, and
|
|
817
|
+
* the operator's half must go with them.
|
|
818
|
+
*
|
|
819
|
+
* Without this, amending isolation on an external-orchestrator fleet was a
|
|
820
|
+
* deadlock with a data-loss escape hatch. The setup gate refused because neither
|
|
821
|
+
* file existed at the new root; the only way past it was approving a brief write
|
|
822
|
+
* in the same pass, and {@link writeOrchestratorBrief} renders a *fresh* POLICY.md
|
|
823
|
+
* scaffold — silently replacing the operator's release procedure and amendment
|
|
824
|
+
* log, and orphaning the real one at the old root.
|
|
825
|
+
*
|
|
826
|
+
* `ORCHESTRATOR.md` is recomposed rather than copied, because it is derived from
|
|
827
|
+
* the package floor plus that policy on every tick anyway. The source is left in
|
|
828
|
+
* place: this package does not delete an operator's file, and a stale copy under
|
|
829
|
+
* the old root is inert once nothing reads it.
|
|
830
|
+
*
|
|
831
|
+
* Returns the moved path, or undefined when nothing needed moving — the roots
|
|
832
|
+
* match, there was no policy to carry, or the destination already has one.
|
|
833
|
+
*/
|
|
834
|
+
export function relocateBriefIfMoved(
|
|
835
|
+
previous: ProjectConfig | undefined,
|
|
836
|
+
next: ProjectConfig,
|
|
837
|
+
): { from: string; to: string } | undefined {
|
|
838
|
+
if (previous === undefined) return undefined;
|
|
839
|
+
if (previous.workspaceRoot === next.workspaceRoot) return undefined;
|
|
840
|
+
const from = policyPathForProject(previous);
|
|
841
|
+
const to = policyPathForProject(next);
|
|
842
|
+
if (!existsSync(from) || existsSync(to)) return undefined;
|
|
843
|
+
|
|
844
|
+
const policy = readFileSync(from, "utf8");
|
|
845
|
+
mkdirSync(dirname(to), { recursive: true });
|
|
846
|
+
writeFileSync(to, policy);
|
|
847
|
+
writeFileSync(briefPathForProject(next), composeOrchestrator(renderFloorForProject(next), policy));
|
|
848
|
+
return { from, to };
|
|
849
|
+
}
|
|
850
|
+
|
|
811
851
|
export function writeOrchestratorBrief(a: SetupAnswers): string {
|
|
812
852
|
const project = buildProject(a);
|
|
813
853
|
const policyPath = policyPathForProject(project);
|
package/src/store.ts
CHANGED
|
@@ -725,20 +725,30 @@ export function openStore(dbPath: string): Store {
|
|
|
725
725
|
const countFailures = db.query<{ n: number }, [string, number]>(
|
|
726
726
|
`SELECT COUNT(*) AS n FROM runs
|
|
727
727
|
WHERE project = ? AND issue = ? AND state = 'failed'
|
|
728
|
-
AND (failureClass IS NULL OR failureClass NOT IN ('ci-infra', 'settlement-stuck'))`,
|
|
728
|
+
AND (failureClass IS NULL OR failureClass NOT IN ('ci-infra', 'settlement-stuck', 'env-start-failure'))`,
|
|
729
729
|
);
|
|
730
730
|
const countContinuations = db.query<{ n: number }, [string, number]>(
|
|
731
731
|
`SELECT COUNT(*) AS n FROM runs
|
|
732
732
|
WHERE project = ? AND issue = ? AND state IN ('killed', 'orphaned', 'blocked')
|
|
733
|
-
AND (failureClass IS NULL OR failureClass NOT IN ('admin-kill', 'settlement-stuck'))`,
|
|
733
|
+
AND (failureClass IS NULL OR failureClass NOT IN ('admin-kill', 'settlement-stuck', 'env-start-failure'))`,
|
|
734
734
|
);
|
|
735
735
|
// Newest first, and bounded: every row this returns costs `gh` calls to gather
|
|
736
736
|
// facts for, so a fleet with a long unclassified history classifies over
|
|
737
737
|
// several ticks rather than spending one tick's budget on all of it.
|
|
738
|
+
//
|
|
739
|
+
// Classified-but-unrecovered rows are included, not just unclassified ones: a
|
|
740
|
+
// recovery whose tracker write failed has to come back, or the log line saying
|
|
741
|
+
// "retrying next tick" is a lie and the row is stranded with a class and no
|
|
742
|
+
// action. `hold` is excluded because it is *recorded only* by design — its
|
|
743
|
+
// `recoveredAt` stays NULL forever, and re-offering it would spin the sweep.
|
|
738
744
|
const selectUnclassified = db.query<RunRow, [string, number]>(
|
|
739
745
|
`SELECT * FROM runs
|
|
740
|
-
WHERE project = ?
|
|
746
|
+
WHERE project = ?
|
|
741
747
|
AND state IN ('blocked', 'killed', 'failed', 'orphaned', 'pushed-green')
|
|
748
|
+
AND (
|
|
749
|
+
failureClass IS NULL
|
|
750
|
+
OR (recoveredAt IS NULL AND recoveryAction IN ('settle', 'continue', 'requeue', 'rerun-checks'))
|
|
751
|
+
)
|
|
742
752
|
ORDER BY startedAt DESC, rowid DESC
|
|
743
753
|
LIMIT ?`,
|
|
744
754
|
);
|
package/src/tracker/github.ts
CHANGED
|
@@ -113,18 +113,29 @@ interface GhPrVerification {
|
|
|
113
113
|
|
|
114
114
|
/** Carries the captured stderr so callers can classify a failure without
|
|
115
115
|
* re-running the command or parsing the message text of a plain Error. */
|
|
116
|
-
class GhError extends Error {
|
|
116
|
+
export class GhError extends Error {
|
|
117
117
|
readonly argv: string[];
|
|
118
118
|
readonly code: number;
|
|
119
119
|
readonly stderr: string;
|
|
120
|
-
|
|
121
|
-
|
|
120
|
+
/**
|
|
121
|
+
* Whatever the command printed before it failed.
|
|
122
|
+
*
|
|
123
|
+
* Carried because a nonzero exit is not always a failure to *read*: `gh pr
|
|
124
|
+
* checks` exits 1 when a check failed and 8 when one is pending, having
|
|
125
|
+
* already written the complete JSON payload asked for. Discarding stdout here
|
|
126
|
+
* made both CI failure classes unreachable on exactly the pull requests they
|
|
127
|
+
* exist to classify — a red PR answered "no checks" (#132 follow-up).
|
|
128
|
+
*/
|
|
129
|
+
readonly stdout: string;
|
|
130
|
+
|
|
131
|
+
constructor(argv: string[], code: number, stderr: string, stdout = "") {
|
|
122
132
|
const detail = stderr.trim() || "(no stderr)";
|
|
123
133
|
super(`\`gh ${argv.join(" ")}\` exited ${code}: ${detail}`);
|
|
124
134
|
this.name = "GhError";
|
|
125
135
|
this.argv = argv;
|
|
126
136
|
this.code = code;
|
|
127
137
|
this.stderr = stderr;
|
|
138
|
+
this.stdout = stdout;
|
|
128
139
|
}
|
|
129
140
|
}
|
|
130
141
|
|
|
@@ -155,7 +166,7 @@ async function gh(argv: string[], stdin?: string): Promise<string> {
|
|
|
155
166
|
|
|
156
167
|
const signal = proc.signalCode;
|
|
157
168
|
if (code !== 0 || signal) {
|
|
158
|
-
throw new GhError(argv, code, signal ? `${stderr}\nterminated by ${signal}` : stderr);
|
|
169
|
+
throw new GhError(argv, code, signal ? `${stderr}\nterminated by ${signal}` : stderr, stdout);
|
|
159
170
|
}
|
|
160
171
|
return stdout;
|
|
161
172
|
}
|
|
@@ -671,11 +682,17 @@ export function makeTracker(p: ProjectConfig, runGh: typeof gh = gh): Tracker {
|
|
|
671
682
|
if (!PR_URL.test(prUrl)) return [];
|
|
672
683
|
try {
|
|
673
684
|
return checkConclusionsFrom(await runGh(["pr", "checks", prUrl, "--json", "name,state,link"]));
|
|
674
|
-
} catch {
|
|
675
|
-
//
|
|
676
|
-
//
|
|
677
|
-
//
|
|
678
|
-
|
|
685
|
+
} catch (err) {
|
|
686
|
+
// A nonzero exit is the *documented* answer for a PR with a failing or
|
|
687
|
+
// pending check — `gh pr checks` exits 1 and 8 respectively, having
|
|
688
|
+
// already printed the whole payload. Reading only the throw is what made
|
|
689
|
+
// `ci-infra` and `ci-deterministic` unreachable on live data: every red
|
|
690
|
+
// PR answered "no checks", so it classified `unknown` and escalated.
|
|
691
|
+
//
|
|
692
|
+
// So the payload wins when there is one, and only a genuinely unreadable
|
|
693
|
+
// answer falls through to "could not tell".
|
|
694
|
+
const stdout = err instanceof GhError ? err.stdout : "";
|
|
695
|
+
return checkConclusionsFrom(stdout);
|
|
679
696
|
}
|
|
680
697
|
},
|
|
681
698
|
|
package/src/types.ts
CHANGED
|
@@ -877,6 +877,7 @@ export interface Tracker {
|
|
|
877
877
|
* answers instead of re-investigating.
|
|
878
878
|
*/
|
|
879
879
|
export const FAILURE_CLASSES = [
|
|
880
|
+
"env-start-failure",
|
|
880
881
|
"turn-cap-progress",
|
|
881
882
|
"turn-cap-spinning",
|
|
882
883
|
"admin-kill",
|
|
@@ -1005,6 +1006,11 @@ export type AdmissionHoldReason =
|
|
|
1005
1006
|
* dispatch refuses rather than running unprotected under a config that says
|
|
1006
1007
|
* otherwise (#125). */
|
|
1007
1008
|
| "credential-boundary"
|
|
1009
|
+
/** No model credential is reachable from a session's redirected `$HOME`, so
|
|
1010
|
+
* every worker would die at turn 0 with "No model selected" and spend an issue
|
|
1011
|
+
* attempt doing it (#152). Held rather than dispatched: an environment fault
|
|
1012
|
+
* is not a failed implementation. */
|
|
1013
|
+
| "no-model-credential"
|
|
1008
1014
|
| "unroutable:no-repo-label"
|
|
1009
1015
|
| "unroutable:multiple-repo-labels"
|
|
1010
1016
|
| "unroutable:unknown-repo";
|
|
@@ -1299,10 +1305,14 @@ export interface Store {
|
|
|
1299
1305
|
/** Everything an operator still has to care about: pending, sending, failed. */
|
|
1300
1306
|
openReports(project: string): ReportRecord[];
|
|
1301
1307
|
/**
|
|
1302
|
-
* Terminal rows the classification sweep
|
|
1303
|
-
* and bounded (#132)
|
|
1304
|
-
*
|
|
1305
|
-
*
|
|
1308
|
+
* Terminal rows the classification sweep still owes something to, newest first
|
|
1309
|
+
* and bounded (#132): never classified, or classified with a retryable recovery
|
|
1310
|
+
* that has not run. The second half is what makes "retrying next tick" true
|
|
1311
|
+
* when a tracker write fails mid-recovery.
|
|
1312
|
+
*
|
|
1313
|
+
* Bounded because each row costs tracker calls to gather facts for: a fleet
|
|
1314
|
+
* with a long unclassified history works through it over several ticks rather
|
|
1315
|
+
* than spending one tick on all of it.
|
|
1306
1316
|
*/
|
|
1307
1317
|
runsNeedingClassification(project: string, limit?: number): RunRecord[];
|
|
1308
1318
|
/** Unrecovered rows per class, for `status`. Empty when nothing is carrying one. */
|
package/src/upgrade.ts
CHANGED
|
@@ -121,6 +121,14 @@ Environment=PATH=/home/fleet/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin
|
|
|
121
121
|
# launcher omits --bounding-set=-all, the child still ends with every other set
|
|
122
122
|
# empty behind NoNewPrivs, and `status` reports the non-empty CapBnd as a named
|
|
123
123
|
# residual rather than ignoring it.
|
|
124
|
+
# Supplementary groups are fixed when a process starts, so a daemon that was
|
|
125
|
+
# already running when these accounts were provisioned can never join them --
|
|
126
|
+
# and a unit with no User= gets no initgroups call at all, so it comes up with
|
|
127
|
+
# `Groups: 0` even once /etc/group is correct. Declared here so membership is a
|
|
128
|
+
# property of the unit rather than of how it happened to be started; after a
|
|
129
|
+
# `daemon-reload` and a restart, `omp-conductor status` reports the probed
|
|
130
|
+
# mechanism as uid-pool.
|
|
131
|
+
SupplementaryGroups=conductor-daemon conductor-runs
|
|
124
132
|
AmbientCapabilities=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_FOWNER CAP_SETPCAP
|
|
125
133
|
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_FOWNER CAP_SETPCAP
|
|
126
134
|
|