@agentskit/harness 0.12.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +95 -0
- package/README.md +8 -3
- package/capabilities/public-surface.json +83 -83
- package/dist/cli.js +309 -66
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +157 -9
- package/dist/index.js +291 -70
- package/dist/index.js.map +1 -1
- package/docs/ADR-0019-human-decision-attestation.md +9 -4
- package/docs/MODULE-BOUNDARIES.md +5 -4
- package/loop.config.example.yaml +29 -2
- package/package.json +12 -8
- package/release/manifest.json +3 -3
- package/release/notes.md +12 -0
|
@@ -13,7 +13,10 @@ Record `approval.recorded` and `authorization.recorded` events in the existing
|
|
|
13
13
|
hash-chained lifecycle log. Each event contains the decision, resulting state,
|
|
14
14
|
verification digest, source revision, contract hash, and human actor; tracking
|
|
15
15
|
authorization also contains its declared target. Rejections are recorded too,
|
|
16
|
-
so every human terminal decision is auditable.
|
|
16
|
+
so every human terminal decision is auditable. The default tracking mode is
|
|
17
|
+
goal-scoped: one human approval covers the declared tracking effect and emits
|
|
18
|
+
both events. A project may set `tracking.authorization: "separate"` when an
|
|
19
|
+
independent tracking decision is required.
|
|
17
20
|
|
|
18
21
|
The existing freshness and verification-attestation checks run before either
|
|
19
22
|
decision is accepted. The `run.json` fields remain a convenient projection,
|
|
@@ -22,6 +25,8 @@ but the event is the audit record.
|
|
|
22
25
|
## Consequences
|
|
23
26
|
|
|
24
27
|
Consumers can independently prove which verified result was approved and which
|
|
25
|
-
external tracking target was authorized.
|
|
26
|
-
|
|
27
|
-
audit
|
|
28
|
+
external tracking target was authorized. Humans do not need to copy a run ID or
|
|
29
|
+
digest: approval commands resolve the latest pending run, while the identifiers
|
|
30
|
+
remain in the audit record. Legacy logs remain readable and are reported as
|
|
31
|
+
legacy until they contain the new hashed event protocol. This is an audit
|
|
32
|
+
attestation, not a digital signature or external notarization.
|
|
@@ -86,15 +86,16 @@ modules, not a provider dependency.
|
|
|
86
86
|
| `src/loop/routing.ts` | Composition | Tiered role → provider/model selection over provider availability | `model-policy`, `providers` (type-only), `config` | None |
|
|
87
87
|
| `src/loop/cooldown.ts` | Composition | Provider cooldown store with exponential backoff | stdlib, `providers` | `<stateDir>/provider-cooldowns.json` |
|
|
88
88
|
| `src/loop/doctor.ts` | Composition | Loop readiness report: Orca, providers, routing, slots, workers, queue | adapters, `config`, `cooldown`, `routing`, `slots` | Orca CLI via runner |
|
|
89
|
-
| `src/loop/contract.ts` | Composition | Task contract schema, orchestrator prompt (issue text as untrusted data), marked-JSON parsing, dispatchability assessment, candidate fallback with auth/quota classification, contract cache | `errors`, `hash`, `resilience`, `doc-bridge`, `config`, `routing` (type-only), `zod` | Headless coding-agent CLI via runner |
|
|
89
|
+
| `src/loop/contract.ts` | Composition | Task contract schema, orchestrator prompt (issue text as untrusted data), marked-JSON parsing, dispatchability assessment, candidate fallback with auth/quota classification, contract cache | `errors`, `hash`, `resilience`, `doc-bridge`, `config`, `fs-atomic`, `routing` (type-only), `zod` | Headless coding-agent CLI via runner |
|
|
90
90
|
| `src/loop/brief.ts` | Composition | Worker prompt: frozen contract + repository rules + protected paths + done signal | `contract`, `config`, `linear-orca` (type-only) | None |
|
|
91
91
|
| `src/loop/resilience-state.ts` | Composition | Per-issue consecutive-failure counter + pause/resume, and per-stage (`tick`/`deliver`) crash-loop pause/resume | stdlib | `<stateDir>/issues/<id>/failures.json`, `<stateDir>/paused.json` |
|
|
92
92
|
| `src/loop/skills.ts` | Composition | `brief.skills` file loading: sha256 digest, truncation at `maxSkillChars`, fail-closed on a missing file | stdlib (`crypto`) | Local Markdown files under `project.root` |
|
|
93
93
|
| `src/loop/github-intake.ts` | Composition | GitHub label-based intake: discovers open PRs carrying `github.intakeLabel` and tracks them as `pr-<n>` (no Linear issue) | `github-cli` | GitHub via the injected runner; `<stateDir>/issues/pr-<n>/intake.json` |
|
|
94
94
|
| `src/loop/event-bus.ts` | Composition | Local pub/sub over the loop's event vocabulary plus orchestration lifecycle hooks (`beforeDispatch`, `beforeReview`, `beforeMerge`, …); loads local `plugins.modules` files | stdlib (`node:path`, `node:url`, dynamic `import()`) | Local `.mjs` files under `project.root`, never fetched over a network |
|
|
95
95
|
| `src/loop/progress.ts` | Composition | Best-effort read of `progress.json` from a worktree — the worker's optional, dynamic outcome-progress report | stdlib | The worktree filesystem only |
|
|
96
|
-
| `src/loop/
|
|
97
|
-
| `src/loop/
|
|
96
|
+
| `src/loop/fs-atomic.ts` | Composition | Atomic JSON write (temp file + rename) so a reader of `dispatch.json`/`delivery.json`/`contract.json` never observes a truncated file from a crash mid-write or a concurrent read | stdlib | The target file's own directory (temp file created and renamed there) |
|
|
97
|
+
| `src/loop/tick.ts` | Composition | One keep-pushing tick: intake, admit (slots + dispatch ledger claim), contract, optional worktree setup command, skills pinning, dispatch into an Orca worktree, Linear transition, escalation, per-issue failure tracking, precheck | adapters, `coordination`, `errors`, `brief`, `config`, `contract`, `cooldown`, `doctor`, `fs-atomic`, `resilience-state`, `routing`, `skills`, `slots` | Orca CLI + Linear via runner; `<stateDir>` files |
|
|
98
|
+
| `src/loop/deliver.ts` | Composition | Deliver stage per dispatched issue and per label-intake PR: PR detection, self-edit hold, conflict/CI/review fix rounds via terminal (or PR comment for intake), review at head, optimistic squash-merge (dispatched issues only — intake is review + comment, never merge), Linear Done, cleanup, stuck/abandoned escalation | adapters, `coordination`, `errors`, `config`, `cooldown`, `doctor`, `fs-atomic`, `github-intake`, `routing`, `tick` | Orca, Linear, GitHub, agentskit-review via runner; `<stateDir>` files |
|
|
98
99
|
| `src/loop/install.ts` | Composition | Orca automation specs (`<prefix>-tick`, `<prefix>-deliver`) with read-only prechecks, idempotent create/edit by name, uninstall, status and the SessionStart hook line | `command`, `orca-cli`, `providers`, `errors`, `config`, `cooldown`, `doctor`, `routing` | Orca automations via runner |
|
|
99
100
|
| `src/loop/guided-install.ts` | Composition | Interactive install: doctor + environment preflight, dry-run rehearsal, confirmation, install, status; readline IO injected | `command`, `orca-cli`, `config`, `doctor`, `install`, `tick`, stdlib readline | Terminal prompts; Orca via runner |
|
|
100
101
|
| `src/loop/local-config.ts` | Composition | Per-machine overlay wizard: Linear team members via Orca, queue owner and machine tuning answers, YAML rendering and reload | `orca-cli`, `config`, `yaml` | Orca via runner; writes `loop.config.local.yaml` |
|
|
@@ -163,7 +164,7 @@ How these seams are wired into the keep-pushing loop (and what is still only a k
|
|
|
163
164
|
| --- | --- | --- | --- |
|
|
164
165
|
| Doc Bridge | `src/adapters/doc-bridge.ts` | Reads a local index | Keep behind `ContextProvider`; measure context hit/quality separately. |
|
|
165
166
|
| Orca | `src/adapters/orca.ts` | None; produces argv and lifecycle projections only | Keep lease/worktree/issue-lock/SHA planning provider-neutral; execution belongs to the orchestrator. |
|
|
166
|
-
| Linear/GitHub/other tracker | `src/adapters/tracking.ts` callback | Caller-owned network mutation | Require idempotency key
|
|
167
|
+
| Linear/GitHub/other tracker | `src/adapters/tracking.ts` callback | Caller-owned network mutation | Require idempotency key; goal approval covers declared tracking by default, with `tracking.authorization: "separate"` as an explicit opt-out. |
|
|
167
168
|
| Orca CLI (loop) | `src/adapters/orca-cli.ts`, `src/adapters/linear-orca.ts` via `CommandRunner` | Read-only `--json` calls in the doctor; dispatch/mutation arrive in later loop phases | Argv only, never a shell string; every call bounded by a timeout; envelope `ok:false` fails closed. |
|
|
168
169
|
| Coding-agent CLIs | `src/adapters/providers.ts` | PATH lookup and optional probe command | Env keys are names only; usage comes from Orca, never from provider SDKs. |
|
|
169
170
|
| GitHub | `src/adapters/github-cli.ts` via `CommandRunner` | `gh pr view/list` (optionally `--label`), `gh api PUT …/merge` with `sha=<reviewed head>`, `gh pr comment`, `gh pr edit --remove-label` | Merge is refused by GitHub when the head moved; every call argv-based and bounded; label intake never calls the merge argv. |
|
package/loop.config.example.yaml
CHANGED
|
@@ -23,10 +23,33 @@ orca:
|
|
|
23
23
|
minVersion: 1.4.200
|
|
24
24
|
timeoutMs: 20000
|
|
25
25
|
|
|
26
|
+
# Revisão mais rigorosa só nas fatias que pedem, por label. A revisão É o gate quando não há CI, e um
|
|
27
|
+
# contrato que congela evidência não deve ser julgado com o mesmo orçamento de um ajuste de copy.
|
|
28
|
+
# Primeira entrada que casa vence, e só os campos nomeados são trocados.
|
|
29
|
+
# reviewOverrides:
|
|
30
|
+
# - anyLabels: [layer:L2, layer:L3]
|
|
31
|
+
# votes: 2
|
|
32
|
+
# minSeverity: nit
|
|
33
|
+
# reason: caminho crítico do MVP
|
|
34
|
+
|
|
35
|
+
# Suítes já vermelhas na branch base. O harness NÃO roda o verifyCommand — o worker roda, na worktree
|
|
36
|
+
# dele — então tolerar falha conhecida é informação no briefing, não parsing de saída. A issue de
|
|
37
|
+
# rastreamento é obrigatória: quarentena sem dono vira permanente.
|
|
38
|
+
# knownFailures:
|
|
39
|
+
# - path: packages/foo/tests/bar.test.ts
|
|
40
|
+
# issue: ENG-123
|
|
41
|
+
# reason: corrida de escrita em arquivo, pré-existente
|
|
42
|
+
|
|
26
43
|
linear:
|
|
27
44
|
workspaceId: <linear-workspace-id> # orca linear team list --workspace all --json
|
|
28
45
|
teamKey: ENG
|
|
29
|
-
person: my-linear-display-name # the
|
|
46
|
+
person: my-linear-display-name # the queue this machine drains (and, under unassigned ownership, the name it claims with)
|
|
47
|
+
queueOwnership: person # person = the issues assigned to `person`. unassigned = the issues nobody holds,
|
|
48
|
+
# with the assignee as a transient claim the loop writes on dispatch and clears on
|
|
49
|
+
# return, so several machines share one priority-ordered queue.
|
|
50
|
+
# Switching to unassigned REQUIRES clearing the backlog's assignees: with `person`
|
|
51
|
+
# and an unassigned backlog the queue comes back empty and the loop looks healthy
|
|
52
|
+
# while doing nothing.
|
|
30
53
|
people: # display name -> Linear user id (orca linear team members …)
|
|
31
54
|
my-linear-display-name: <linear-user-id>
|
|
32
55
|
rotation:
|
|
@@ -35,7 +58,11 @@ linear:
|
|
|
35
58
|
advanceWhenEmpty: true # never switch while an implementation lease remains
|
|
36
59
|
states: [Todo, Ready] # dispatchable states; anything already started is left alone
|
|
37
60
|
excludeLabels: [blocked, needs-info]
|
|
38
|
-
requireLabels: []
|
|
61
|
+
requireLabels: [] # ALL of these (AND); empty = no constraint
|
|
62
|
+
anyLabels: [] # at least ONE of these (OR) — how a machine declares the slices it
|
|
63
|
+
# drains, e.g. [layer:L2, layer:L3]. requireLabels cannot say this:
|
|
64
|
+
# it demands every label on the same issue, so two layers there
|
|
65
|
+
# match nothing and the queue comes back silently empty.
|
|
39
66
|
projects: [] # empty = all projects
|
|
40
67
|
order: [priority, updatedAt] # urgent first; priority 0 (none) sorts last
|
|
41
68
|
maxQueue: 50
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agentskit/harness",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"description": "Portable, evidence-backed development harness for coding agents.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -38,6 +38,7 @@
|
|
|
38
38
|
"build": "tsup && node -e \"console.log(JSON.stringify({status:'passed',criteria:['package']}))\"",
|
|
39
39
|
"typecheck": "tsc -p tsconfig.json --noEmit",
|
|
40
40
|
"test": "pnpm typecheck && vitest run --config vitest.config.ts test && node -e \"console.log(JSON.stringify({status:'passed',criteria:['package']}))\"",
|
|
41
|
+
"test:coverage": "vitest run --config vitest.config.ts --coverage test",
|
|
41
42
|
"test:contract": "pnpm typecheck && vitest run --config vitest.config.ts test/contract.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['protocol']}))\"",
|
|
42
43
|
"test:extensibility": "pnpm typecheck && vitest run --config vitest.config.ts test/extensibility.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['extensibility']}))\"",
|
|
43
44
|
"test:event-log": "pnpm typecheck && vitest run --config vitest.config.ts test/extensibility.test.ts test/harness.test.ts -t \"ordered append-only lifecycle log|complete typed task through human approval|authorization only when tracking|human rejection\" && node -e \"console.log(JSON.stringify({status:'passed',criteria:['event-log','verification-attestation','approval-attestation']}))\"",
|
|
@@ -67,6 +68,7 @@
|
|
|
67
68
|
"test:boundaries": "pnpm typecheck && vitest run --config vitest.config.ts test/boundaries.test.ts && node scripts/verify-dependency-directions.mjs",
|
|
68
69
|
"test:phase-executor": "pnpm typecheck && vitest run --config vitest.config.ts test/phase-executor.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['phase-executor']}))\"",
|
|
69
70
|
"test:artifacts": "pnpm typecheck && vitest run --config vitest.config.ts test/artifacts.test.ts && node scripts/verify-artifact-cli.mjs",
|
|
71
|
+
"test:loop-watch-sigint": "pnpm typecheck && pnpm build >/dev/null && node scripts/verify-loop-watch-sigint.mjs",
|
|
70
72
|
"test:adapters": "pnpm typecheck && vitest run --config vitest.config.ts test/adapters.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['adapters']}))\"",
|
|
71
73
|
"test:loop": "pnpm typecheck && vitest run --config vitest.config.ts test/loop.test.ts test/loop-adapters.test.ts test/loop-tick.test.ts test/loop-deliver.test.ts test/loop-install.test.ts test/loop-guided-install.test.ts test/loop-retro.test.ts test/loop-debrief.test.ts test/loop-watch.test.ts test/loop-observability.test.ts test/loop-agent-registry.test.ts test/loop-memory.test.ts test/loop-contract-failure.test.ts test/loop-resilience-state.test.ts test/loop-skills.test.ts test/loop-github-intake.test.ts test/loop-event-bus.test.ts test/loop-progress.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['loop-config','loop-providers','loop-routing','loop-doctor','loop-adapters','loop-tick','loop-deliver','loop-install','loop-debrief','loop-watch','loop-observability','loop-agent-registry','loop-memory','loop-contract-failure','loop-resilience-state','loop-skills','loop-github-intake','loop-event-bus','loop-progress']}))\"",
|
|
72
74
|
"test:review": "pnpm typecheck && vitest run --config vitest.config.ts test/review.test.ts && node -e \"console.log(JSON.stringify({status:'passed',criteria:['review-delivery']}))\"",
|
|
@@ -85,18 +87,20 @@
|
|
|
85
87
|
"prepublishOnly": "pnpm typecheck && pnpm build && pnpm test"
|
|
86
88
|
},
|
|
87
89
|
"dependencies": {
|
|
88
|
-
"commander": "^14.0.
|
|
90
|
+
"commander": "^14.0.3",
|
|
89
91
|
"ink": "7.1.1",
|
|
90
92
|
"react": "19.3.0",
|
|
91
|
-
"yaml": "^2.9.
|
|
92
|
-
"zod": "^4.6.
|
|
93
|
+
"yaml": "^2.9.1",
|
|
94
|
+
"zod": "^4.6.5"
|
|
93
95
|
},
|
|
94
96
|
"devDependencies": {
|
|
95
|
-
"@types/node": "^22.
|
|
97
|
+
"@types/node": "^22.20.3",
|
|
96
98
|
"@types/react": "^19.3.0",
|
|
97
|
-
"
|
|
98
|
-
"
|
|
99
|
-
"
|
|
99
|
+
"@vitest/coverage-v8": "4.1.11",
|
|
100
|
+
"ink-testing-library": "4.0.0",
|
|
101
|
+
"tsup": "^8.5.1",
|
|
102
|
+
"typescript": "^5.9.3",
|
|
103
|
+
"vitest": "^4.1.11"
|
|
100
104
|
},
|
|
101
105
|
"license": "MIT",
|
|
102
106
|
"packageManager": "pnpm@10.34.4",
|
package/release/manifest.json
CHANGED
|
@@ -2,9 +2,9 @@
|
|
|
2
2
|
"type": "agentskit-harness-release-manifest",
|
|
3
3
|
"schemaVersion": 1,
|
|
4
4
|
"package": "@agentskit/harness",
|
|
5
|
-
"version": "0.
|
|
5
|
+
"version": "0.13.0",
|
|
6
6
|
"channel": "latest",
|
|
7
|
-
"sourceRevision": "
|
|
7
|
+
"sourceRevision": "7eed0dcb3162a8808a27e38290356cf005d7c94e",
|
|
8
8
|
"requiredChecks": [
|
|
9
9
|
"typecheck",
|
|
10
10
|
"test",
|
|
@@ -32,5 +32,5 @@
|
|
|
32
32
|
"pilot-benchmark"
|
|
33
33
|
],
|
|
34
34
|
"nextBaseline": "benchmarks/harness-0.4.0-baseline.json",
|
|
35
|
-
"digest": "
|
|
35
|
+
"digest": "599670762fa2f8b1c4bd6bf1a0aa9c0dfa40cffbf77781d6cf15ee048cc79000"
|
|
36
36
|
}
|
package/release/notes.md
CHANGED
|
@@ -1,3 +1,15 @@
|
|
|
1
|
+
# 0.13.0 release candidate
|
|
2
|
+
|
|
3
|
+
A full-codebase test-coverage sweep (every module in `src/kernel/`, `src/execution/`, `src/adapters/`, and most of
|
|
4
|
+
`src/loop/` brought to 90%+ statements/branches) that surfaced eight real bugs, each fixed in its own focused PR:
|
|
5
|
+
atomic writes for per-issue state files (`dispatch.json`/`delivery.json`/`contract.json`), a lock around
|
|
6
|
+
`events.ndjson` rotation, PII-scanner coverage for current-format secrets (OpenAI project keys, fine-grained GitHub
|
|
7
|
+
PATs, Google/Stripe keys, PEM blocks, AWS secret keys), a fixed `Ctrl-C` for `loop watch`, raw throws in
|
|
8
|
+
`retro.ts`/`doc-bridge.ts` reclassified as `HarnessError`, two `execution/agent.ts` session-recorder bugs (dead
|
|
9
|
+
code; a swallowed invalid-runtime-result error), an execute-bit check in `findExecutable`, and a `loop debrief
|
|
10
|
+
--issue X` fix for a normal not-yet-dispatched issue. Everything else in this release is test-only. Publication
|
|
11
|
+
remains gated on a merge to `main` through npm Trusted Publishing.
|
|
12
|
+
|
|
1
13
|
# 0.12.0 release candidate
|
|
2
14
|
|
|
3
15
|
Reuses Orca instead of reinventing it, where Orca's own tooling gives a better answer than the harness's own
|