@agentskit/harness 0.8.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,68 @@
1
+ # ADR-0029: Loop resilience, brief pinning, worktree setup, effort routing, and GitHub intake
2
+
3
+ - Status: Accepted
4
+ - Date: 2026-09-12
5
+
6
+ ## Context
7
+
8
+ The 2026-09-11/12 pilot ran the keep-pushing loop (ADR-0027) unattended for ~7 hours and surfaced two production
9
+ defects and, separately, a review of `ColeMurray/background-agents` (an open-source clone of Ramp's Inspect
10
+ background-agent system) suggested transferable ideas already partially covered by this harness's own mechanisms.
11
+
12
+ Measured from `events.ndjson`: `classifyProviderFailure` did not recognise real Claude/Codex usage-limit phrasing
13
+ ("You've hit your session limit", "usage limit reached") or Grok's ("Overloaded", "temporarily limiting requests"),
14
+ so every such failure fell through to `other` and `onProviderFailure` never fired — no cooldown was ever recorded.
15
+ The same gap existed on the review path (`agentskit-review` exit classified without checking the same patterns).
16
+ Result: 19 unclassified `contract.failed` retries across 4 issues and 12 incomplete reviews over 7h, with no cap
17
+ anywhere — nothing counted consecutive failures on an issue or a stage, so a single misclassified error retried
18
+ forever.
19
+
20
+ ## Decision
21
+
22
+ 1. **Failure classification** (bug fix, no schema change): `classifyProviderFailure` (`src/loop/contract.ts`) checks
23
+ a `QUOTA_PATTERN` covering the phrasing above before falling back to the kernel's generic `classifyFailure`, and
24
+ `extractResetsAt` parses a relative (`resets in 3h`) or clock-time (`resets 10:40pm`) reset out of the message.
25
+ `deliver.ts`'s review-incomplete path applies the same classification and marks the **reviewer's own provider id**
26
+ (`ctx.reviewer.provider`, e.g. `codex`) cooling down — not the review-CLI transport id (`review.provider`, e.g.
27
+ `codex-cli`), which `rankModels`/`detectProviders` never look up. This was the second, related pilot bug.
28
+ 2. **Per-issue and per-stage auto-pause** (new `src/loop/resilience-state.ts`, Composition, no kernel change):
29
+ `resilience.maxConsecutiveFailures` (default 3) pauses a single issue — one deduplicated Linear comment, the
30
+ `resilience.pausedLabel` — after that many consecutive `contract.failed`/`worker.dispatch-failed` events; a
31
+ successful dispatch clears the counter. `resilience.stagePauseAfterRuns` does the same for a `loop stage
32
+ tick|deliver` run that *throws* (not a normal idle/ok/blocked report) repeatedly. Both are separate from, and do
33
+ not replace, the existing `blocked`/`stuck` escalation via `linear.excludeLabels` (ADR-0027 §6) — those already
34
+ self-exclude a terminal outcome; this closes the two pre-dispatch paths that had no ceiling at all.
35
+ 3. **Skills pinned into the worker brief** (new `src/loop/skills.ts`): `brief.skills` lists Markdown files, read
36
+ once at dispatch time, sha256-digested, truncated at `brief.maxSkillChars` with a visible note, and embedded in
37
+ a new brief section. A missing file fails the dispatch closed. This is deliberately a separate mechanism from
38
+ the existing Doc Bridge `contract.briefScopes` (path/title pointers resolved per issue) — skills are full pinned
39
+ content with a cryptographic digest recorded in `dispatch.json`, and a handoff never re-reads them, so editing a
40
+ skill file after dispatch cannot affect an in-flight worker.
41
+ 4. **Worktree setup command**: `project.setup.command` (argv, no shell) runs once between `orca worktree create`
42
+ and opening the worker terminal — e.g. `pnpm install --frozen-lockfile` — bounded by `project.setup.timeoutSec`
43
+ (reserved out of the tick time budget so it cannot itself blow the budget). `project.setup.required` (default
44
+ true) routes a failing/timing-out setup through the same worktree-cleanup and consecutive-failure path as any
45
+ other dispatch failure.
46
+ 5. **Reasoning effort per role**: `models.effort.<role>` is rendered into `tui`/`headless` only for a provider that
47
+ declares `providers.<id>.effortFlag`; a provider without one silently ignores it, so the feature is opt-in per
48
+ provider and safe to default-enable. `agentskit-review` has no such flag, so `models.effort.reviewer` is
49
+ recorded (for `loop retro` grouping and future use) without being wired into the review CLI call.
50
+ 6. **GitHub label intake** (new `src/loop/github-intake.ts`): `github.intakeLabel` lets a human ask the loop to
51
+ review a PR it never dispatched (tracked as `pr-<n>`, no Linear issue). It reuses the same checks/review/fix-round
52
+ decisions as a normal dispatch, but every nudge becomes a PR comment (no worker terminal exists) and
53
+ `github.reviewOnly` is pinned `true` in the schema — this loop merges only PRs it dispatched itself, never one it
54
+ was only asked to review, however clean that review comes back.
55
+
56
+ ## Consequences
57
+
58
+ - No kernel change; `pnpm test:boundaries` stays green. All six changes are Composition (`src/loop/*`) or a small
59
+ Adapter addition (`githubOpenPullRequests` gains `label?`, plus `githubLabelRemove`, in `src/adapters/github-cli.ts`).
60
+ - `loop.config.yaml` gains six new top-level/nested blocks (`resilience`, `brief`, `project.setup`, `models.effort`,
61
+ `providers.<id>.effortFlag`, `github`), all with safe defaults (`.prefault({})`) so an existing 0.8.0 config keeps
62
+ its exact prior behaviour unless a project opts in.
63
+ - New CLI surface: `ak-harness loop resume [issue] [--stage tick|deliver]` and `ak-harness loop paused`.
64
+ - `loop doctor` gained a `brief.skills` check; `loop retro`'s `dispatches.byProvider` now groups by
65
+ `provider/model@effort` when an effort was recorded.
66
+ - Explicitly out of scope (from `background-agents`): a remote control plane, sandboxed/remote worktrees
67
+ (Modal/E2B/Daytona), prebuilds/snapshots, a multiplayer web UI, and Slack/GitHub-App bots. This harness's runtime
68
+ stays Orca + local worktrees; none of the above changes that.
@@ -0,0 +1,54 @@
1
+ # ADR-0030: Loop event bus and orchestration hooks, not model-loop middleware
2
+
3
+ - Status: Accepted
4
+ - Date: 2026-09-12
5
+
6
+ ## Context
7
+
8
+ Comparing the loop against frameworks that build a "custom agent harness" in-process (e.g. LangChain's
9
+ `create_agent` + middleware: hooks before/after every model call and every tool call) surfaced a real gap: this
10
+ harness had no deterministic extension point at all. `appendLoopEvent` (`src/loop/tick.ts`) only appends to
11
+ `<stateDir>/events.ndjson` — nothing can react in real time, and nothing can say "stop, don't do this" before a
12
+ consequential action.
13
+
14
+ The gap is real, but the shape of the fix is not "add middleware like LangChain's": this harness orchestrates
15
+ external CLI agents (claude/codex/grok) as opaque processes inside an Orca worktree. There is no in-process
16
+ model↔tool loop to hook into — the worker's own context window, its own tool calls, are invisible to us by design
17
+ (ADR-0027). Building a `before model call` hook here is not possible without controlling that loop, which is
18
+ explicitly out of scope.
19
+
20
+ What we *do* control, end to end, is the orchestration around that opaque worker: whether we dispatch it, whether
21
+ we act on its review, whether we merge its PR. That is the level at which a deterministic extension point is both
22
+ useful and achievable.
23
+
24
+ ## Decision
25
+
26
+ 1. **`src/loop/event-bus.ts`** (Composition): a local, in-process pub/sub over the loop's existing free-form event
27
+ vocabulary (`contract.failed`, `worker.dispatched`, `pr.reviewed`, …) plus a small, fixed set of **orchestration
28
+ lifecycle hooks**: `beforeDispatch`, `afterDispatch`, `beforeReview`, `afterReview`, `beforeMerge`, `afterMerge`,
29
+ `onPause`, `onEscalate`. A `before*` hook may return `{ block: true, reason }` to stop the action; every other
30
+ hook is notification-only. A listener or hook that throws is swallowed (logged as a note, never fatal) — a
31
+ broken local plugin must not take down tick or deliver.
32
+ 2. **Not `kernel/plugins.ts`.** The kernel's plugin registry is tied to `HARNESS_EVENT_TYPES`, the harness's own
33
+ run-lifecycle vocabulary, and carries plugin ids/versions/dependency ordering appropriate for a shared kernel
34
+ contract. The loop's event vocabulary is an open set of strings owned by composition, not the kernel, so
35
+ `event-bus.ts` is a separate, simpler primitive with the same "who's listening" ergonomics rather than a reuse
36
+ of that contract.
37
+ 3. **`plugins.modules`** (`loop.config.yaml`): local `.mjs` files, relative to `project.root`, loaded once at the
38
+ start of `tick`/`deliver`. Same trust level as `agents.registry.yaml` today — files already checked into the
39
+ project's own repo, never fetched over a network. Each module exports `{ id, apply(bus) }`.
40
+ 4. **`appendLoopEvent` gains an optional third `bus` parameter.** Every existing call site (`retro.ts`, tests) is
41
+ unaffected; `tick.ts`/`deliver.ts` pass their bus so every event written to disk is also emitted live.
42
+ 5. **`loop doctor` gains a `plugins.modules` check** using the same `loadLoopPlugins` the runtime uses, so a typo
43
+ or a broken module surfaces before a tick silently drops a plugin.
44
+
45
+ ## Consequences
46
+
47
+ - No kernel change; `pnpm test:boundaries` stays green.
48
+ - `loop.config.yaml` gains one new block (`plugins: { modules: [] }`), empty by default — zero behavior change for
49
+ every existing config.
50
+ - This directly enables the human-approval gate (`delivery.merge.requireHumanApproval`, ADR follow-up in the same
51
+ release) and the PII-scan wiring as bus consumers, without a bespoke mechanism for each.
52
+ - Explicitly **not** delivered: hooking into the worker's own model/tool loop (structural — the worker is an
53
+ opaque CLI), a distributed/external event bus (webhook, message broker) as a first consumer, and dynamic module
54
+ loading from anywhere other than the local repo.
package/docs/LOOP.md CHANGED
@@ -113,6 +113,7 @@ For every issue the loop dispatched (`<stateDir>/issues/<id>/dispatch.json`) and
113
113
  | No PR, idle ≥ `workerIdleTimeoutMin` | one check-in via `terminal send`; idle again after that → **stuck**: lease released, issue → `returnState` + `blocked`, worktree kept |
114
114
  | No PR, terminal gone (> 5 min after dispatch) | **stuck** as above |
115
115
  | PR touches `selfEditPaths` | **held**: one PR comment, no review, no merge |
116
+ | PR touches `secretFilePatterns` (`.env`, `*.pem`, `*.key`, `id_rsa`, `credentials.json`, … by default) | **held**: same as `selfEditPaths` — the loop cannot inspect diff content, only filenames, so this holds on the filename shape alone even if the content is innocuous |
116
117
  | PR conflicting | rebase instruction to the worker, once per head (does not count as a fix round) |
117
118
  | CI red | failing check names to the worker; counts as a fix round |
118
119
  | CI pending / required check missing | wait |
@@ -120,12 +121,66 @@ For every issue the loop dispatched (`<stateDir>/issues/<id>/dispatch.json`) and
120
121
  | Review findings ≥ floor | findings to the worker; counts as a fix round; same head is never re-reviewed |
121
122
  | Fix rounds exhausted (`maxFixRounds`) | **blocked**: Linear comment + label + `returnState`, PR comment, lease released, worktree and PR kept |
122
123
  | Review clean, `merge.auto` | `gh api PUT …/merge` with `sha=<reviewed head>` (GitHub refuses if the head moved) → Linear attach + comment + `doneState`, worktree removed when `cleanupWorktree` |
124
+ | Review clean, `merge.requireHumanApproval` set, no GitHub approval yet | **held**: reuses `pr.reviewDecision` already fetched with the PR snapshot — no extra GitHub call; merges automatically as soon as `reviewDecision` becomes `APPROVED` on a later run |
123
125
  | PR merged outside the loop | same completion path |
124
126
  | PR closed without merge | **abandoned**: lease released, issue → `returnState`, worktree kept |
125
127
 
126
128
  State lives in `<stateDir>/issues/<id>/delivery.json` (reviews per head, fix rounds, nudges) and every decision is
127
129
  appended to `<stateDir>/events.ndjson`. `--dry-run` reports the decision for each issue without touching anything.
128
130
 
131
+ ## GitHub label intake: reviewing PRs the loop never dispatched
132
+
133
+ The loop's normal queue is Linear issues; `github.intakeLabel` (default `loop:review`, set to `null` to disable)
134
+ lets a human ask it to review a PR it had nothing to do with — a contributor's PR, a manual branch, anything —
135
+ without filing a Linear issue for it. Every `deliver` run lists open PRs carrying the label
136
+ (`gh pr list --label <intakeLabel>`) and starts tracking any not seen before as `pr-<n>` under
137
+ `<stateDir>/issues/pr-<n>/intake.json` (`{ pr, headRef, source: 'github-label', addedAt }`); tracking is
138
+ idempotent, so discovery never re-adds a PR it already knows about.
139
+
140
+ An intake PR runs the same checks → review → fix-round decisions as a normal dispatch (see the table above), with
141
+ two differences forced by having no Linear issue and no worker terminal:
142
+
143
+ - Every nudge (conflict, red CI, review findings) is posted as a **PR comment** instead of sent to a worker
144
+ terminal — there is no worker to nudge.
145
+ - `github.reviewOnly` is a fixed guarantee, not a knob (the schema pins it to `true`): a clean review always ends
146
+ in **held**, commented as "merge is human", the label removed, and `finishedAt` recorded — this loop merges only
147
+ PRs it dispatched itself, never one it was only asked to review. If the label is removed on GitHub before the
148
+ loop finishes, it stops tracking the PR the same way (held, no further comments).
149
+
150
+ ## Event bus and orchestration hooks
151
+
152
+ `appendLoopEvent` writes every loop event to `<stateDir>/events.ndjson`, but nothing could react to one *while it
153
+ happens*, and there was no deterministic way to say "don't do this" before a consequential action. `plugins.modules`
154
+ (`loop.config.yaml`, empty by default — zero behavior change until configured) lists local `.mjs` files, relative to
155
+ `project.root` (same trust level as `agents.registry.yaml`: files already in this repo, never fetched over a
156
+ network), loaded once at the start of `tick`/`deliver`. Each exports `{ id, apply(bus) }`:
157
+
158
+ ```js
159
+ export default {
160
+ id: 'slack-notify',
161
+ apply(bus) {
162
+ bus.on('worker.dispatched', (event) => { /* … */ })
163
+ bus.hook('beforeMerge', (payload) => {
164
+ if (isFrozeWindow()) return { block: true, reason: 'release freeze' }
165
+ })
166
+ },
167
+ }
168
+ ```
169
+
170
+ - `bus.on(type | '*', listener)` subscribes to the loop's existing event vocabulary (`contract.failed`,
171
+ `worker.dispatched`, `pr.reviewed`, `provider.cooldown`, `issue.paused`, …) live, in addition to the ndjson log.
172
+ - `bus.hook(name, listener)` subscribes to an **orchestration lifecycle hook**: `beforeDispatch`, `afterDispatch`,
173
+ `beforeReview`, `afterReview`, `beforeMerge`, `afterMerge`, `onPause`, `onEscalate`. A `before*` listener may
174
+ return `{ block: true, reason }` to stop the action (surfaced as a `skipped`/`waiting`/`held` result with the
175
+ reason); every other hook is notification-only. This is deliberately **not** a hook into the worker's own
176
+ model/tool loop — the worker is an opaque external CLI (ADR-0027) and that loop is invisible to us. These hooks
177
+ fire around the orchestration decisions we actually make: dispatch, review, merge.
178
+ - A listener or hook that throws is swallowed (never fatal — a broken plugin must not stop tick or deliver) and,
179
+ for a hook, its error is reported back through `runHook`'s `errors`.
180
+ - `loop doctor` runs a `plugins.modules` check confirming every configured module exists and loads cleanly.
181
+
182
+ See `src/loop/event-bus.ts` for the full API (`createLoopEventBus`, `loadLoopPlugins`).
183
+
129
184
  ## One tick
130
185
 
131
186
  1. **Intake** — `orca linear list-issues` once per configured state, filtered and ordered locally; issues already
@@ -148,6 +203,114 @@ terminal handle, provider/model and lease for the deliver stage.
148
203
 
149
204
  Start from [`loop.config.example.yaml`](../loop.config.example.yaml) at the package root.
150
205
 
206
+ ## Resilience: auto-pause after repeated failures
207
+
208
+ Two failure paths have no natural ceiling elsewhere in the pipeline — contract generation failing on every
209
+ candidate, and a worktree/worker dispatch failing outright — because the issue never gets a worktree, a lease that
210
+ would otherwise expire, or a label that would exclude it from the queue. Left alone, a single misclassified or
211
+ persistent error (a quota message the classifier didn't recognise, a broken `orca worktree create`) retries every
212
+ tick forever. (The 2026-09-11/12 pilot logged 19 such retries across 4 issues in 7h before this existed.)
213
+
214
+ - `resilience.maxConsecutiveFailures` (default 3): after this many **consecutive** `contract.failed` or
215
+ `worker.dispatch-failed` events on the *same* issue, the loop stops retrying it: one deduplicated Linear comment
216
+ explaining why, the `resilience.pausedLabel` (default `loop:paused`), and the issue is skipped locally on every
217
+ later tick regardless of whether that label is in `linear.excludeLabels`. A successful dispatch clears the
218
+ counter. State lives in `<stateDir>/issues/<id>/failures.json` (`loop paused` lists every paused issue).
219
+ - **Resuming** an issue: remove the `loop:paused` label on Linear (the next tick notices via `list-issues` and
220
+ clears the local state itself) or run `ak-harness loop resume <issue>`, which also best-effort removes the label.
221
+ - `resilience.stagePauseAfterRuns` (default 3): a scheduled `loop stage tick|deliver` run that *throws* (a config or
222
+ adapter crash, not a normal idle/ok/blocked report) this many times in a row pauses that stage — `loop stage`
223
+ then short-circuits to a `{"status":"paused", ...}` report instead of running, so a crash loop cannot spend budget
224
+ or provider usage under Orca. `ak-harness loop resume --stage tick|deliver` clears it; a single successful run
225
+ clears it automatically. State lives in `<stateDir>/paused.json`.
226
+
227
+ Neither mechanism touches the existing `blocked`/`stuck` escalations (fix-round exhaustion, an idle worker with no
228
+ PR) — those already label the issue and route it out of the queue via `linear.excludeLabels`.
229
+
230
+ ## PII/secret scanning
231
+
232
+ `security.pii.enabled` (default `false` — enabling it never changes behavior for a project that doesn't need it)
233
+ scans issue text before it enters the orchestrator prompt (`contract.ts`) and the worker brief (`brief.ts`) for
234
+ PII-shaped patterns: emails, common provider API-key prefixes (`sk-…`, `ghp_…`, `AKIA…`, Slack tokens), phone
235
+ numbers, card-number-shaped digit runs (`src/kernel/pii.ts`, pure and dependency-free). `security.pii.action`
236
+ controls what happens on a match:
237
+
238
+ - `redact` (default when enabled): each match is replaced with `[REDACTED:<kind>]` before the text is embedded.
239
+ - `warn`: the text is sent unchanged; a `security.pii-detected` event is still recorded (source `issue-text` or
240
+ `worker-brief`, with the matched kinds and count — never the matched value itself).
241
+ - `block`: the dispatch fails closed instead of ever sending the text anywhere, with a message naming the kinds
242
+ found (not the values). Recorded like any other dispatch failure, so `resilience.maxConsecutiveFailures` still
243
+ applies if it keeps happening.
244
+
245
+ This is a pattern scanner, not a claim of completeness — it catches common shapes, not every possible secret.
246
+
247
+ ## Cost/time circuit breakers
248
+
249
+ The loop cannot count a worker CLI's own model or tool calls — it is an opaque process, not a loop we run
250
+ ourselves — so there is no way to cap "cost" the way an in-process agent harness would. Two proxies close most of
251
+ the gap, both unset (disabled) by default so an existing config is unaffected:
252
+
253
+ - **`delivery.maxDispatchMinutes`**: a hard wall-clock ceiling on one dispatch, independent of idle detection.
254
+ `delivery.workerIdleTimeoutMin` only catches a worker that stopped producing output; this catches one that is
255
+ still active but has run far longer than any real task on the project should. Past the ceiling, `deliver` stops
256
+ nudging/reviewing/merging the issue and escalates it exactly like a stuck worker (Linear comment + label +
257
+ `returnState`, worktree preserved for inspection, lease released) — recorded as a `max-duration.tripped` event.
258
+ - **`resilience.maxUsageDeltaPercent`**: a cost proxy from Orca's own usage reporting. The builder's remaining
259
+ usage percent (`RankedModel.remainingPercent`) is snapshotted at dispatch time (`dispatch.json`'s
260
+ `initialRemainingPercent`); every later `deliver` run compares it against that provider's *current* remaining
261
+ usage. If it dropped by at least this many percentage points while the issue was in flight, the dispatch is
262
+ stopped the same way — recorded as `cost-guard.tripped`. This is deliberately usage-delta, not call-count: it is
263
+ the only per-issue cost signal Orca actually reports for an opaque worker CLI.
264
+
265
+ ## Dynamic outcome progress
266
+
267
+ The contract's outcome list (`brief.ts`) is a static plan frozen before dispatch — it cannot become a live todo
268
+ list without controlling the worker's own loop, which this harness deliberately does not do (ADR-0027). The brief
269
+ documents a lightweight, optional convention instead: as the worker finishes or starts an outcome, it writes
270
+ `progress.json` at the root of its own worktree, e.g. `{"o1": "done", "o2": "in-progress"}` (ids match the
271
+ outcome list). `loop debrief` reads it back best-effort (`readOutcomeProgress`, `src/loop/progress.ts`) — a
272
+ missing, unreadable, or malformed file is never an error, since nothing enforces the worker keeps it current and
273
+ older dispatches never wrote one at all. When present, it shows as `N/M outcome(s) done` per in-flight issue
274
+ instead of a flat "in flight".
275
+
276
+ ## Skills pinned into the worker brief
277
+
278
+ `brief.skills` (default `[]`) lists Markdown files, relative to `project.root`, that every worker brief embeds
279
+ verbatim under a `## Skills (pinned)` section — house conventions the orchestrator's contract can reference but a
280
+ worker starting cold has no other way to see (e.g. `AGENTS.md`, `CLAUDE.md`, `docs/for-agents/INDEX.md`).
281
+
282
+ - Reading and hashing happens once, at dispatch time (`loadPinnedSkills`, `src/loop/skills.ts`): each file is
283
+ sha256-digested and truncated at `brief.maxSkillChars` (default 6000) with a visible `[truncated N chars]` note so
284
+ one large file cannot exhaust the brief budget. A configured path that does not exist or cannot be read **fails
285
+ the dispatch** (fail-closed) rather than silently sending a worker without guidance it was told it would have —
286
+ the same worktree-cleanup and consecutive-failure accounting as any other dispatch failure applies.
287
+ - The rendered brief is persisted to `<stateDir>/issues/<id>/brief.md`, and `dispatch.json` records `briefDigest`
288
+ (hash of the full brief) plus `skills: [{path, digest}]` — enough to prove after the fact exactly which revision
289
+ of a skill file a given worker saw.
290
+ - **Pinning is by design, not by accident:** a handoff (`renderHandoffBrief`) reuses the worktree/branch state, not
291
+ the original brief, and never re-reads `brief.skills` — so editing a skill file after dispatch affects only
292
+ *future* dispatches, never a worker (or its handoff) already in flight.
293
+ - `loop doctor` runs a `brief.skills` check confirming every configured file currently exists and is readable, so a
294
+ typo or a moved file surfaces before the next dispatch fails.
295
+
296
+ ## Worktree setup command
297
+
298
+ A freshly created Orca worktree is a bare checkout — no `node_modules`, no build output, nothing a worker can run
299
+ tests against until it installs dependencies itself, wasting the first several minutes of every dispatch on the
300
+ same shell commands. `project.setup.command` (unset by default; an argv array, e.g.
301
+ `[pnpm, install, --frozen-lockfile]` — no shell, so no `&&`/`|`) runs once in the new worktree between
302
+ `orca worktree create` and opening the worker's terminal.
303
+
304
+ - `project.setup.timeoutSec` (default 600) bounds the run; the loop's per-tick time budget already reserves this
305
+ much time before attempting a dispatch, so a configured setup command cannot itself blow the tick budget.
306
+ - `project.setup.required` (default `true`): a non-zero exit or a timeout removes the just-created worktree, never
307
+ opens a terminal, and fails the dispatch — recorded as a `worker.dispatch-failed` event and counted by the
308
+ per-issue consecutive-failure tracker above, exactly like a contract or worktree-create failure. Set it to
309
+ `false` to have a failing setup only log a note and still hand the worker its terminal.
310
+ - Every run (pass or fail) is recorded as a `worker.setup` event and, when the dispatch succeeds, as `setup:
311
+ {command, exitCode, durationMs, timedOut}` on `dispatch.json` — enough to see in `loop retro` whether a slow or
312
+ flaky setup command is costing more dispatches than it saves.
313
+
151
314
  ## What the doctor checks
152
315
 
153
316
  | Check | Source | Blocking |
@@ -157,6 +320,9 @@ Start from [`loop.config.example.yaml`](../loop.config.example.yaml) at the pack
157
320
  | `routing.<role>` | tiers from `models.<role>` filtered by provider availability | yes — a role with no available provider blocks |
158
321
  | `machine.slots` | `sampleMachine` + `adaptiveConcurrency`, free RAM reserve, WSL cap, running worktrees | no — 0 free slots is a warning, not a failure |
159
322
  | `linear.queue` | `orca linear list-issues` per configured state, filtered and ordered locally | yes — an unreachable Linear blocks |
323
+ | `brief.skills` | existence + readability of each `brief.skills` path under `project.root` | yes when any are unreadable — dispatch would fail closed anyway |
324
+ | `plugins.modules` | each configured module exists and `import()`s without throwing | yes when any fails to load |
325
+ | `mcp.allowlist` | only runs when `mcp.enabled`; builds the default-deny bridge from `mcp.allowTools` and self-tests it (no live MCP server involved — ADR-0028) | warning on an empty allowlist, failed if the allow/deny wiring itself misbehaves |
160
326
 
161
327
 
162
328
  ## Dynamic model routing
@@ -181,11 +347,28 @@ its binary is on PATH, its auth is not known to be missing, no Orca usage window
181
347
 
182
348
  Providers authenticate through their own CLI login (`claude login`, `codex login`, `grok login`); only providers
183
349
  declared `auth: api-key` need an environment variable, and the loop never reads its value. Orca has no per-run model flag; the chosen model is rendered into `providers.<id>.tui` (for example
184
- `codex -m {model} --full-auto`) and launched in the worker terminal.
350
+ `codex -m {model} -s workspace-write -a never`) and launched in the worker terminal. The explicit
351
+ approval policy keeps YOLO runs non-interactive while the workspace sandbox limits changes to the assigned worktree.
185
352
 
186
353
  When a provider runs out of usage the loop records a cooldown in `<stateDir>/provider-cooldowns.json`:
187
354
  `initialMin` doubling up to `maxMin`, never earlier than the reset instant Orca reported.
188
355
 
356
+ ### Reasoning effort per role
357
+
358
+ `models.effort.<role>` (`low | medium | high | xhigh`; defaults: orchestrator/reviewer `high`, builder `medium`,
359
+ watcher `low`) is only applied for a provider that declares `providers.<id>.effortFlag` — a template such as
360
+ `-c model_reasoning_effort={effort}` (codex) or `--reasoning-effort {effort}` (grok); a provider without one
361
+ ignores it entirely, so leaving `effort` at its default is always safe. The flag (with `{effort}` substituted) is
362
+ appended to `tui` as literal text, and appended as its own argv elements (split on whitespace, since headless argv
363
+ is never shell-joined) to `headless`. `agentskit-review` has no reasoning-effort flag, so `models.effort.reviewer`
364
+ is not currently wired into the review CLI call — it is validated and recorded for symmetry and for a future
365
+ reviewer transport that supports it.
366
+
367
+ Whichever effort a dispatched builder actually used is recorded as `effort` on `dispatch.json` and the
368
+ `worker.dispatched` event; `loop retro`'s `dispatches.byProvider` groups by `provider/model@effort` (falling back
369
+ to plain `provider/model` for older events with no effort recorded) so a retro can tell a slow `gpt-5.6-luna@high`
370
+ run from a fast `@medium` one.
371
+
189
372
  ## Machine slots
190
373
 
191
374
  `maxAgents = max(floor, min(adaptiveConcurrency(ceiling), ramBound, wslCap?))` where `ceiling` defaults to
@@ -208,7 +391,7 @@ the published dependency tree. `@agentskit/harness` stays dependency-light (`com
208
391
  | Trigger | `.doc-bridge/index.json` exists under `project.root` |
209
392
  | Knob | `contract.maxContextReferences` (default `6`; `0` disables) |
210
393
  | Behaviour | Query is `"<issue id> <title>"`. Up to N deterministic references are appended to the orchestrator prompt. Missing or malformed index → **no refs** (loop continues). |
211
- | Boundary | No `@agentskit/doc-bridge` import; the adapter only reads the local index contract ([ADR-0003](ADR-0003-doc-bridge-context-binding.md)). Index build/refresh stays with Doc Bridge (`pnpm docs:bridge:index` in repos that use it). |
394
+ | Boundary | No `@agentskit/doc-bridge` import; the adapter only reads the local index contract ([ADR-0003](ADR-0003-doc-bridge-context-binding.md)). Contract resolution rejects indexes older than `contract.docBridgeMaxAgeHours`; exact source freshness remains a Doc Bridge gate. Index build/refresh stays with Doc Bridge (`pnpm docs:bridge:index` in repos that use it). |
212
395
 
213
396
  ### Code Review (`agentskit-review`)
214
397
 
@@ -44,6 +44,7 @@ modules, not a provider dependency.
44
44
  | `src/kernel/learning.ts` | Kernel | Retrospective parsing and human learning promotion | stdlib, `errors` | None |
45
45
  | `src/kernel/status.ts` | Kernel | Deterministic status snapshot and digest | `errors`, `hash`, `block`, `types` (type-only) | None |
46
46
  | `src/kernel/model-policy.ts` | Kernel | Role-to-model binding and validation | `errors`, `hash` | Provider is data, not an SDK |
47
+ | `src/kernel/pii.ts` | Kernel | Deterministic PII/secret pattern scan + redaction over a plain string | None | None |
47
48
  | `src/execution/machine.ts` | Execution support | Machine sampling and adaptive concurrency | stdlib, `types`, `errors` | Host CPU/memory metrics |
48
49
  | `src/execution/coordination.ts` | Execution support | Atomic issue/worktree claims and dispatch ledger | stdlib, `errors`, `hash` | Local state directory only |
49
50
  | `src/kernel/resilience.ts` | Kernel | Failure classification and bounded retry/recovery policy | `errors` | Operation callback supplied by caller |
@@ -87,8 +88,13 @@ modules, not a provider dependency.
87
88
  | `src/loop/doctor.ts` | Composition | Loop readiness report: Orca, providers, routing, slots, workers, queue | adapters, `config`, `cooldown`, `routing`, `slots` | Orca CLI via runner |
88
89
  | `src/loop/contract.ts` | Composition | Task contract schema, orchestrator prompt (issue text as untrusted data), marked-JSON parsing, dispatchability assessment, candidate fallback with auth/quota classification, contract cache | `errors`, `hash`, `resilience`, `doc-bridge`, `config`, `routing` (type-only), `zod` | Headless coding-agent CLI via runner |
89
90
  | `src/loop/brief.ts` | Composition | Worker prompt: frozen contract + repository rules + protected paths + done signal | `contract`, `config`, `linear-orca` (type-only) | None |
90
- | `src/loop/tick.ts` | Composition | One keep-pushing tick: intake, admit (slots + dispatch ledger claim), contract, dispatch into an Orca worktree, Linear transition, escalation, precheck | adapters, `coordination`, `errors`, `hash`, `brief`, `config`, `contract`, `cooldown`, `doctor`, `routing`, `slots` | Orca CLI + Linear via runner; `<stateDir>` files |
91
- | `src/loop/deliver.ts` | Composition | Deliver stage per dispatched issue: PR detection, self-edit hold, conflict/CI/review fix rounds via terminal, review at head, optimistic squash-merge, Linear Done, cleanup, stuck/abandoned escalation | adapters, `coordination`, `errors`, `config`, `cooldown`, `doctor`, `routing`, `tick` | Orca, Linear, GitHub, agentskit-review via runner; `<stateDir>` files |
91
+ | `src/loop/resilience-state.ts` | Composition | Per-issue consecutive-failure counter + pause/resume, and per-stage (`tick`/`deliver`) crash-loop pause/resume | stdlib | `<stateDir>/issues/<id>/failures.json`, `<stateDir>/paused.json` |
92
+ | `src/loop/skills.ts` | Composition | `brief.skills` file loading: sha256 digest, truncation at `maxSkillChars`, fail-closed on a missing file | stdlib (`crypto`) | Local Markdown files under `project.root` |
93
+ | `src/loop/github-intake.ts` | Composition | GitHub label-based intake: discovers open PRs carrying `github.intakeLabel` and tracks them as `pr-<n>` (no Linear issue) | `github-cli` | GitHub via the injected runner; `<stateDir>/issues/pr-<n>/intake.json` |
94
+ | `src/loop/event-bus.ts` | Composition | Local pub/sub over the loop's event vocabulary plus orchestration lifecycle hooks (`beforeDispatch`, `beforeReview`, `beforeMerge`, …); loads local `plugins.modules` files | stdlib (`node:path`, `node:url`, dynamic `import()`) | Local `.mjs` files under `project.root`, never fetched over a network |
95
+ | `src/loop/progress.ts` | Composition | Best-effort read of `progress.json` from a worktree — the worker's optional, dynamic outcome-progress report | stdlib | The worktree filesystem only |
96
+ | `src/loop/tick.ts` | Composition | One keep-pushing tick: intake, admit (slots + dispatch ledger claim), contract, optional worktree setup command, skills pinning, dispatch into an Orca worktree, Linear transition, escalation, per-issue failure tracking, precheck | adapters, `coordination`, `errors`, `brief`, `config`, `contract`, `cooldown`, `doctor`, `resilience-state`, `routing`, `skills`, `slots` | Orca CLI + Linear via runner; `<stateDir>` files |
97
+ | `src/loop/deliver.ts` | Composition | Deliver stage per dispatched issue and per label-intake PR: PR detection, self-edit hold, conflict/CI/review fix rounds via terminal (or PR comment for intake), review at head, optimistic squash-merge (dispatched issues only — intake is review + comment, never merge), Linear Done, cleanup, stuck/abandoned escalation | adapters, `coordination`, `errors`, `config`, `cooldown`, `doctor`, `github-intake`, `routing`, `tick` | Orca, Linear, GitHub, agentskit-review via runner; `<stateDir>` files |
92
98
  | `src/loop/install.ts` | Composition | Orca automation specs (`<prefix>-tick`, `<prefix>-deliver`) with read-only prechecks, idempotent create/edit by name, uninstall, status and the SessionStart hook line | `command`, `orca-cli`, `providers`, `errors`, `config`, `cooldown`, `doctor`, `routing` | Orca automations via runner |
93
99
  | `src/loop/guided-install.ts` | Composition | Interactive install: doctor + environment preflight, dry-run rehearsal, confirmation, install, status; readline IO injected | `command`, `orca-cli`, `config`, `doctor`, `install`, `tick`, stdlib readline | Terminal prompts; Orca via runner |
94
100
  | `src/loop/local-config.ts` | Composition | Per-machine overlay wizard: Linear team members via Orca, queue owner and machine tuning answers, YAML rendering and reload | `orca-cli`, `config`, `yaml` | Orca via runner; writes `loop.config.local.yaml` |
@@ -159,7 +165,7 @@ How these seams are wired into the keep-pushing loop (and what is still only a k
159
165
  | Linear/GitHub/other tracker | `src/adapters/tracking.ts` callback | Caller-owned network mutation | Require idempotency key and explicit tracking authorization. |
160
166
  | Orca CLI (loop) | `src/adapters/orca-cli.ts`, `src/adapters/linear-orca.ts` via `CommandRunner` | Read-only `--json` calls in the doctor; dispatch/mutation arrive in later loop phases | Argv only, never a shell string; every call bounded by a timeout; envelope `ok:false` fails closed. |
161
167
  | Coding-agent CLIs | `src/adapters/providers.ts` | PATH lookup and optional probe command | Env keys are names only; usage comes from Orca, never from provider SDKs. |
162
- | GitHub | `src/adapters/github-cli.ts` via `CommandRunner` | `gh pr view/list`, `gh api PUT …/merge` with `sha=<reviewed head>`, `gh pr comment` | Merge is refused by GitHub when the head moved; every call argv-based and bounded. |
168
+ | GitHub | `src/adapters/github-cli.ts` via `CommandRunner` | `gh pr view/list` (optionally `--label`), `gh api PUT …/merge` with `sha=<reviewed head>`, `gh pr comment`, `gh pr edit --remove-label` | Merge is refused by GitHub when the head moved; every call argv-based and bounded; label intake never calls the merge argv. |
163
169
  | AgentsKit code review | `src/adapters/code-review.ts` via `CommandRunner` | `agentskit-review --pr … --result <file> [--post]` | Exit codes 0/1/2 plus the private result file decide clean/findings/incomplete; the floor is `--block`. |
164
170
  | Process runtime | `src/execution/runtime.ts` | Starts child processes | Execution support; policy and evidence gates remain kernel decisions. |
165
171
  | Docker runtime | `src/execution/runtime.ts` | Starts Docker containers | Optional sandbox selected by config, never a mandatory kernel dependency. |
@@ -9,6 +9,11 @@ project:
9
9
  baseBranch: main
10
10
  root: . # relative to this file
11
11
  stateDir: .codex/loop # ledger, cooldowns, contracts
12
+ setup:
13
+ # command: [pnpm, install, --frozen-lockfile] # argv (no shell), run once in a freshly created worktree
14
+ # before the worker terminal opens; unset = skip
15
+ timeoutSec: 600
16
+ required: true # failing/timing-out setup removes the worktree and counts as a dispatch failure
12
17
 
13
18
  orca:
14
19
  bin: orca # on Linux outside Orca terminals use orca-ide (bare `orca` is the GNOME screen reader)
@@ -24,6 +29,10 @@ linear:
24
29
  person: my-linear-display-name # the ONLY queue this machine drains
25
30
  people: # display name -> Linear user id (orca linear team members …)
26
31
  my-linear-display-name: <linear-user-id>
32
+ rotation:
33
+ enabled: false # set true to advance through owners when the current dispatchable queue drains
34
+ owners: [my-linear-display-name] # ordered display names; the first entry is the initial owner
35
+ advanceWhenEmpty: true # never switch while an active lease remains
27
36
  states: [Todo, Ready] # dispatchable states; anything already started is left alone
28
37
  excludeLabels: [blocked, needs-info]
29
38
  requireLabels: []
@@ -57,6 +66,12 @@ models:
57
66
  maxMin: 240 # …up to 4 h
58
67
  probeBeforeReenable: true
59
68
  exhaustedPercent: 100
69
+ # Reasoning effort requested per role. Only applied for providers below that declare `effortFlag`; others ignore it.
70
+ effort:
71
+ orchestrator: high
72
+ reviewer: high # currently unused: agentskit-review has no reasoning-effort flag (see reviewProvider below)
73
+ builder: medium
74
+ watcher: low
60
75
  providers:
61
76
  claude:
62
77
  bin: claude
@@ -64,12 +79,15 @@ models:
64
79
  envKeys: [ANTHROPIC_API_KEY] # optional API-key fallback
65
80
  tui: "claude --model {model} --permission-mode auto"
66
81
  headless: [claude, -p, "{prompt}", --model, "{model}", --permission-mode, plan, --output-format, text]
82
+ # effortFlag: "--effort {effort}"
67
83
  codex:
68
84
  bin: codex
69
85
  auth: subscription
70
86
  envKeys: [OPENAI_API_KEY]
71
- tui: "codex -m {model} --full-auto"
87
+ # Codex 0.154+ uses explicit approval/sandbox flags; --full-auto is not a valid CLI option.
88
+ tui: "codex -m {model} -s workspace-write -a never"
72
89
  headless: [codex, exec, -m, "{model}", -s, read-only, --skip-git-repo-check, "{prompt}"]
90
+ # effortFlag: "-c model_reasoning_effort={effort}"
73
91
  opencode:
74
92
  bin: opencode
75
93
  auth: none
@@ -81,6 +99,7 @@ models:
81
99
  auth: subscription # `grok login` (OAuth) — no API key needed
82
100
  tui: "grok -m {model}"
83
101
  headless: [grok, -p, "{prompt}", -m, "{model}"]
102
+ # effortFlag: "--reasoning-effort {effort}"
84
103
 
85
104
  machine:
86
105
  floor: 1 # always allow one worker
@@ -110,10 +129,15 @@ delivery:
110
129
  auto: true
111
130
  method: squash
112
131
  requireChecks: true
132
+ requireHumanApproval: false # true: also require a human GitHub review approval before merging, even
133
+ # after a clean loop review + green checks
113
134
  maxFixRounds: 2
114
135
  workerIdleTimeoutMin: 45
136
+ # maxDispatchMinutes: 240 # hard ceiling on one dispatch regardless of activity; unset = disabled
115
137
  handoff: { enabled: true, maxHandoffs: 2, onlyWhenProviderUnavailable: true }
116
138
  selfEditPaths: [loop.config.yaml, ".github/**"] # PRs touching these never auto-merge
139
+ # secretFilePatterns: [...] # default already covers .env, *.pem, *.key, id_rsa, credentials.json, etc.
140
+ # PRs touching a matching filename are held, never reviewed or merged
117
141
  ignoreChecks: [] # advisory check names that never block
118
142
  requiredChecks: [] # empty = every reported check must be green
119
143
  cleanupWorktree: true # remove the Orca worktree after merge
@@ -154,6 +178,35 @@ contract:
154
178
  maxBriefReferences: 4
155
179
  contextProviders: [doc-bridge] # add rag when rag.enabled
156
180
 
181
+ github:
182
+ intakeLabel: loop:review # a PR with this label is reviewed even though the loop never dispatched it;
183
+ # null disables intake entirely
184
+ reviewOnly: true # fixed: intake PRs are always review + comment, never auto-merged
185
+
186
+ plugins:
187
+ modules: [] # e.g. [scripts/loop-plugins/notify.mjs] — local .mjs files (relative to
188
+ # project.root, never fetched over the network) loaded once per tick/deliver;
189
+ # each exports { id, apply(bus) } and can subscribe to loop events or block a
190
+ # beforeDispatch/beforeReview/beforeMerge hook — see docs/LOOP.md
191
+
192
+ security:
193
+ pii:
194
+ enabled: false # scan issue text / worker brief for PII-shaped patterns before embedding them
195
+ action: redact # redact | warn | block
196
+
197
+ resilience:
198
+ maxConsecutiveFailures: 3 # pause an issue after this many consecutive contract/dispatch failures
199
+ pausedLabel: loop:paused # applied to the issue in Linear; remove it (or `loop resume <id>`) to retry
200
+ stagePauseAfterRuns: 3 # pause `loop stage tick|deliver` itself after this many consecutive thrown runs
201
+ # maxUsageDeltaPercent: 40 # stop a dispatch if its provider's remaining usage drops by this many
202
+ # points while the issue is in flight; unset = disabled (cost circuit breaker)
203
+
204
+ brief:
205
+ skills: [] # e.g. [AGENTS.md, CLAUDE.md, docs/for-agents/INDEX.md] — paths relative to project.root,
206
+ # pinned verbatim into every worker brief (sha256-digested in dispatch.json); a
207
+ # missing file fails the dispatch instead of silently sending a worker without it
208
+ maxSkillChars: 6000 # per-file cap; longer files are truncated with a visible note
209
+
157
210
  schedule:
158
211
  tick: "*/5 * * * *"
159
212
  deliver: "*/10 * * * *"
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@agentskit/harness",
3
- "version": "0.8.0",
3
+ "version": "0.10.0",
4
4
  "description": "Portable, evidence-backed development harness for coding agents.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -68,7 +68,7 @@
68
68
  "test:phase-executor": "pnpm typecheck && vitest run --config vitest.config.ts test/phase-executor.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['phase-executor']}))\"",
69
69
  "test:artifacts": "pnpm typecheck && vitest run --config vitest.config.ts test/artifacts.test.ts && node scripts/verify-artifact-cli.mjs",
70
70
  "test:adapters": "pnpm typecheck && vitest run --config vitest.config.ts test/adapters.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['adapters']}))\"",
71
- "test:loop": "pnpm typecheck && vitest run --config vitest.config.ts test/loop.test.ts test/loop-adapters.test.ts test/loop-tick.test.ts test/loop-deliver.test.ts test/loop-install.test.ts test/loop-guided-install.test.ts test/loop-retro.test.ts test/loop-debrief.test.ts test/loop-watch.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['loop-config','loop-providers','loop-routing','loop-doctor','loop-adapters','loop-tick','loop-deliver','loop-install','loop-debrief','loop-watch']}))\"",
71
+ "test:loop": "pnpm typecheck && vitest run --config vitest.config.ts test/loop.test.ts test/loop-adapters.test.ts test/loop-tick.test.ts test/loop-deliver.test.ts test/loop-install.test.ts test/loop-guided-install.test.ts test/loop-retro.test.ts test/loop-debrief.test.ts test/loop-watch.test.ts test/loop-agent-registry.test.ts test/loop-memory.test.ts test/loop-contract-failure.test.ts test/loop-resilience-state.test.ts test/loop-skills.test.ts test/loop-github-intake.test.ts test/loop-event-bus.test.ts test/loop-progress.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['loop-config','loop-providers','loop-routing','loop-doctor','loop-adapters','loop-tick','loop-deliver','loop-install','loop-debrief','loop-watch','loop-agent-registry','loop-memory','loop-contract-failure','loop-resilience-state','loop-skills','loop-github-intake','loop-event-bus','loop-progress']}))\"",
72
72
  "test:review": "pnpm typecheck && vitest run --config vitest.config.ts test/review.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['review-delivery']}))\"",
73
73
  "test:quality": "pnpm typecheck && vitest run --config vitest.config.ts test/quality.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['quality-matrix']}))\"",
74
74
  "test:eval-battery": "pnpm typecheck && vitest run --config vitest.config.ts test/eval-battery.test.ts test/issues-010-014.eval.test.ts test/optimization.test.ts && pnpm build >/dev/null && node scripts/verify-harness-eval-manifest.mjs && node -e \"console.log(JSON.stringify({status:'passed',criteria:['eval-battery','eval-manifest','eval-coverage']}))\"",
@@ -2,7 +2,7 @@
2
2
  "type": "agentskit-harness-release-manifest",
3
3
  "schemaVersion": 1,
4
4
  "package": "@agentskit/harness",
5
- "version": "0.8.0",
5
+ "version": "0.10.0",
6
6
  "channel": "latest",
7
7
  "sourceRevision": "b8cfe596e5f30f2c373b89f00f271079b296feaf",
8
8
  "requiredChecks": [
@@ -32,5 +32,5 @@
32
32
  "pilot-benchmark"
33
33
  ],
34
34
  "nextBaseline": "benchmarks/harness-0.4.0-baseline.json",
35
- "digest": "712347d25fac5a6b0819cf8ba60f11ba9dc37c8d9380fb7df9e834f711509a86"
35
+ "digest": "231258f2636bbc19cd5b202f37c87146f7892cd5b79a41b97e12f4301c4573d4"
36
36
  }