@tachikomagundam/abathur 0.2.2 → 0.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -18
- package/config/genomes/historian.example.jsonc +9 -4
- package/dist/bench/adapter.js +22 -4
- package/dist/bench/fixture.js +8 -4
- package/dist/bench/toy.js +3 -3
- package/dist/commands/status.js +6 -1
- package/dist/core/evolve/run-bench.js +2 -1
- package/dist/test/bench-adapter.test.js +24 -0
- package/dist/test/bench-fixture.test.js +6 -2
- package/dist/test/bench-reporoot.test.js +197 -0
- package/dist/test/historian-grader-integrity.test.js +438 -0
- package/dist/test/historian-grader-io.test.js +277 -1
- package/dist/test/historian-run-scenario.test.js +205 -0
- package/dist/test/opencode.test.js +21 -0
- package/dist/test/status-repo.test.js +72 -0
- package/graders/historian/grader-core.d.mts +43 -0
- package/graders/historian/grader-core.mjs +215 -5
- package/graders/historian/grader-support.d.mts +11 -0
- package/graders/historian/grader-support.mjs +28 -0
- package/graders/historian/grader.mjs +62 -5
- package/graders/historian/mutate.sh +6 -3
- package/graders/historian/run-scenario.sh +39 -3
- package/graders/historian/seed-wrapped.sh +1 -1
- package/package.json +1 -1
- package/plugin/abathur.ts +47 -7
package/README.md
CHANGED
|
@@ -142,7 +142,11 @@ Abathur ships an official opencode plugin adapter. The harness remains the
|
|
|
142
142
|
orchestrator, but once set up an opencode session can also drive the CLI
|
|
143
143
|
directly. Two install routes:
|
|
144
144
|
|
|
145
|
-
|
|
145
|
+
Both routes deliver the tool **and** the `/abathur` slash command; pick by
|
|
146
|
+
taste — A needs no edit of your opencode config and works fully offline, B
|
|
147
|
+
needs no file copying and removes with one deleted line.
|
|
148
|
+
|
|
149
|
+
**Route A — one command (file-copy installer).**
|
|
146
150
|
|
|
147
151
|
```bash
|
|
148
152
|
npm i -g @tachikomagundam/abathur
|
|
@@ -150,7 +154,7 @@ abathur opencode install # copies plugin assets into ~/.config/opencode/
|
|
|
150
154
|
# restart opencode
|
|
151
155
|
```
|
|
152
156
|
|
|
153
|
-
**Route B — config only (
|
|
157
|
+
**Route B — config only (auto-downloaded from npm).** Put the package name in
|
|
154
158
|
your opencode config and restart:
|
|
155
159
|
|
|
156
160
|
```jsonc
|
|
@@ -159,11 +163,16 @@ your opencode config and restart:
|
|
|
159
163
|
|
|
160
164
|
opencode then downloads the package from npm at startup into its own cache
|
|
161
165
|
(`~/.cache/opencode/packages/…`) and loads the package's `./server` export —
|
|
162
|
-
no file copying, no uninstall step (delete the line).
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
166
|
+
no file copying, no uninstall step (delete the line). Since 0.2.3 the plugin
|
|
167
|
+
also self-registers `/abathur` through its `config` hook: slash commands are
|
|
168
|
+
entries in the merged config's `command` map, and a plugin may add one there
|
|
169
|
+
— the upstream `opencode-acp` plugin registers `/acp` the same way (the
|
|
170
|
+
0.2.0–0.2.2 note here claiming plugins cannot register slash commands was
|
|
171
|
+
wrong; corrected). If Route A's copied `commands/abathur.md` exists it stays
|
|
172
|
+
authoritative — the injection never overwrites it, and the name-keyed command
|
|
173
|
+
map means `/abathur` can never appear twice. One honest limit stands for both
|
|
174
|
+
routes: the tool spawns the `abathur` **CLI**, which must exist on `PATH` (or
|
|
175
|
+
via `ABATHUR_BIN`) — keep the `npm i -g` install.
|
|
167
176
|
|
|
168
177
|
Either route gives the session:
|
|
169
178
|
|
|
@@ -176,9 +185,10 @@ Either route gives the session:
|
|
|
176
185
|
so the allowlist limits typos and UX, not capability. `promote` and
|
|
177
186
|
`tombstone` are deliberately NOT reachable through the tool: they are
|
|
178
187
|
human gates and must be run in a terminal.
|
|
179
|
-
- the user command **`/abathur <args…>`**
|
|
180
|
-
|
|
181
|
-
the
|
|
188
|
+
- the user command **`/abathur <args…>`** — a slash command that tells the
|
|
189
|
+
agent to translate its arguments into a tool call and report the exit code.
|
|
190
|
+
Route A provides it as the copied `commands/abathur.md` file; Route B gets
|
|
191
|
+
it from the plugin's `config` hook at startup.
|
|
182
192
|
|
|
183
193
|
Route A ships its own manager commands: `abathur opencode status` shows each
|
|
184
194
|
target's path, installed/packaged sha256, and state (up-to-date / outdated /
|
|
@@ -430,7 +440,10 @@ abathur run --genome toy-smoke --dry-run # 计划 + requires[] 探针,零
|
|
|
430
440
|
Abathur 自带官方 opencode 插件适配器。工装仍是编排者,但配置之后,opencode
|
|
431
441
|
会话也可以直接驱动 CLI。安装有两条路线:
|
|
432
442
|
|
|
433
|
-
|
|
443
|
+
两条路线都交付工具**和** `/abathur` 斜杠命令,按喜好挑选:A 不必改动你的
|
|
444
|
+
opencode 配置、且完全离线可用;B 不复制任何文件、删掉一行即卸载。
|
|
445
|
+
|
|
446
|
+
**路线 A——一条命令(文件复制安装器)。**
|
|
434
447
|
|
|
435
448
|
```bash
|
|
436
449
|
npm i -g @tachikomagundam/abathur
|
|
@@ -438,7 +451,7 @@ abathur opencode install # 把插件资产复制进 ~/.config/opencode/
|
|
|
438
451
|
# 重启 opencode
|
|
439
452
|
```
|
|
440
453
|
|
|
441
|
-
**路线 B
|
|
454
|
+
**路线 B——纯配置(自动从 npm 下载)。** 把包名写进 opencode 配置再
|
|
442
455
|
重启:
|
|
443
456
|
|
|
444
457
|
```jsonc
|
|
@@ -447,10 +460,14 @@ abathur opencode install # 把插件资产复制进 ~/.config/opencode/
|
|
|
447
460
|
|
|
448
461
|
opencode 会在启动时自行从 npm 把包下载到它自己的缓存
|
|
449
462
|
(`~/.cache/opencode/packages/…`)并加载该包的 `./server` 导出——不复制文件,
|
|
450
|
-
也没有 uninstall
|
|
451
|
-
`
|
|
452
|
-
`
|
|
453
|
-
|
|
463
|
+
也没有 uninstall 步骤(删掉那行即可)。自 0.2.3 起,插件还会通过它的
|
|
464
|
+
`config` 钩子自注册 `/abathur`:斜杠命令本质是合并后配置的 `command`
|
|
465
|
+
映射里的条目,插件可以在钩子里添加一条——上游的 `opencode-acp` 插件正是
|
|
466
|
+
这样注册 `/acp` 的(0.2.0–0.2.2 在此声称"插件无法注册斜杠命令"是错的,
|
|
467
|
+
已更正)。若路线 A 复制的 `commands/abathur.md` 存在,它保持权威——注入
|
|
468
|
+
绝不覆盖它,而且命令映射按名字键控,`/abathur` 绝不会出现两份。对两条
|
|
469
|
+
路线同样成立的一个诚实限制:工具 spawn 的是 `abathur` **CLI**,CLI 必须
|
|
470
|
+
在 `PATH`(或 `ABATHUR_BIN`)上可达,`npm i -g` 不能省。
|
|
454
471
|
|
|
455
472
|
任一路线都会让会话获得:
|
|
456
473
|
|
|
@@ -461,8 +478,9 @@ opencode 会在启动时自行从 npm 把包下载到它自己的缓存
|
|
|
461
478
|
bash 等同的权限——`run` 与 `genome` 按设计就会 spawn 变异器/引擎二进制——
|
|
462
479
|
所以允许清单限制的是笔误与体验,而非能力。`promote` 与 `tombstone` 刻意
|
|
463
480
|
不可经由工具触达:它们是人类闸门,必须在终端里运行。
|
|
464
|
-
- 用户命令 **`/abathur
|
|
465
|
-
|
|
481
|
+
- 用户命令 **`/abathur <参数…>`**——斜杠命令,指示智能体把参数
|
|
482
|
+
翻译成工具调用并回报退出码。路线 A 由复制的 `commands/abathur.md`
|
|
483
|
+
文件提供;路线 B 由插件的 `config` 钩子在启动时注册。
|
|
466
484
|
|
|
467
485
|
路线 A 自带管理命令:`abathur opencode status` 逐目标打印路径、已安装/随包
|
|
468
486
|
sha256 与状态
|
|
@@ -8,6 +8,11 @@
|
|
|
8
8
|
//
|
|
9
9
|
// ${ABATHUR_HISTORIAN_REPO} absolute path of the historian bench repo
|
|
10
10
|
// (contains scenarios/, rubric.md, seed_sandbox.sh)
|
|
11
|
+
// — repoPath ONLY: the S4 engine seam delivers the
|
|
12
|
+
// bench's ACTIVE TREE (incumbent repoPath /
|
|
13
|
+
// candidate worktree) to the commands below as the
|
|
14
|
+
// {repoRoot} template var, so unit/seed paths must
|
|
15
|
+
// NOT bake a materialized repo literal here.
|
|
11
16
|
// ${ABATHUR_REPO} absolute path of THIS Abathur checkout
|
|
12
17
|
// (hosts graders/historian/*)
|
|
13
18
|
// ${ABATHUR_WIKI_BASE} canonical wiki root, e.g. http://localhost:3000
|
|
@@ -47,11 +52,11 @@
|
|
|
47
52
|
{ "id": "scenario-08", "path": "scenarios/08-current-state-g5.md", "split": "train" },
|
|
48
53
|
{ "id": "scenario-09", "path": "scenarios/09-timeline-week-groups.md", "split": "val" }
|
|
49
54
|
],
|
|
50
|
-
"seedCommand": "bash ${ABATHUR_REPO}/graders/historian/seed-wrapped.sh
|
|
55
|
+
"seedCommand": "bash ${ABATHUR_REPO}/graders/historian/seed-wrapped.sh {repoRoot}/seed_sandbox.sh",
|
|
51
56
|
"resetCommand": "bash ${ABATHUR_REPO}/graders/historian/reset-sandbox.sh",
|
|
52
|
-
"runCommand": "bash ${ABATHUR_REPO}/graders/historian/run-scenario.sh {unit.id}
|
|
53
|
-
"graderCommand": "node ${ABATHUR_REPO}/graders/historian/grader.mjs {unit.id}
|
|
54
|
-
"agentModel": "
|
|
57
|
+
"runCommand": "bash ${ABATHUR_REPO}/graders/historian/run-scenario.sh {unit.id} {repoRoot}/{unit.path}",
|
|
58
|
+
"graderCommand": "node ${ABATHUR_REPO}/graders/historian/grader.mjs {unit.id} {repoRoot}/{unit.path} ${ABATHUR_WIKI_BASE}",
|
|
59
|
+
"agentModel": "<provider>/<model-id>", // "<provider>/<model-id> as registered in your opencode config"
|
|
55
60
|
// Real agent runs, not unit tests: 1200s caps one stuck scenario without
|
|
56
61
|
// poisoning the generation (timeout => unscored/inconclusive, excluded by
|
|
57
62
|
// the stats layer). seed+reset hooks share a FIXED 60s adapter budget
|
package/dist/bench/adapter.js
CHANGED
|
@@ -10,11 +10,29 @@
|
|
|
10
10
|
// excluded from score distributions (hr discipline), not zeroed.
|
|
11
11
|
import { spawn } from "node:child_process";
|
|
12
12
|
import { cannotAnswer } from "../exit.js";
|
|
13
|
-
|
|
14
|
-
|
|
13
|
+
/**
|
|
14
|
+
* `{repoRoot}` = the bench's ACTIVE TREE: the genome repoPath for the incumbent,
|
|
15
|
+
* the sealed candidate worktree for a candidate (run-loop.ts swaps spec.repoPath
|
|
16
|
+
* per target; adapters resolve it through effectiveRepoPath at construction).
|
|
17
|
+
* Omitting repoRoot keeps `{repoRoot}` an UNKNOWN placeholder — fail-closed
|
|
18
|
+
* exit 2, never a silently half-substituted command.
|
|
19
|
+
*/
|
|
20
|
+
export function unitVars(unit, sandboxDir, repoRoot) {
|
|
21
|
+
const vars = {
|
|
22
|
+
"unit.path": unit.path,
|
|
23
|
+
"unit.id": unit.id,
|
|
24
|
+
sandbox: sandboxDir,
|
|
25
|
+
workdir: sandboxDir,
|
|
26
|
+
};
|
|
27
|
+
if (repoRoot !== undefined)
|
|
28
|
+
vars["repoRoot"] = repoRoot;
|
|
29
|
+
return vars;
|
|
15
30
|
}
|
|
16
|
-
export function sandboxVars(sandboxDir) {
|
|
17
|
-
|
|
31
|
+
export function sandboxVars(sandboxDir, repoRoot) {
|
|
32
|
+
const vars = { sandbox: sandboxDir, workdir: sandboxDir };
|
|
33
|
+
if (repoRoot !== undefined)
|
|
34
|
+
vars["repoRoot"] = repoRoot;
|
|
35
|
+
return vars;
|
|
18
36
|
}
|
|
19
37
|
const PLACEHOLDER = /\{([A-Za-z0-9_.]+)\}/g;
|
|
20
38
|
/**
|
package/dist/bench/fixture.js
CHANGED
|
@@ -32,6 +32,7 @@ import { cannotAnswer, ExitSignal } from "../exit.js";
|
|
|
32
32
|
import { resolveConfigDir } from "../config.js";
|
|
33
33
|
import { fingerprint } from "../core/ids.js";
|
|
34
34
|
import { acquireGenomeLock, Ledger } from "../core/ledger.js";
|
|
35
|
+
import { effectiveRepoPath } from "../core/spec.js";
|
|
35
36
|
import { childStatus, firstLine, inconclusive, parseGraderLine, renderCommand, runChild, sandboxVars, unitVars, ZERO_METRICS, } from "./adapter.js";
|
|
36
37
|
import { buildManifest, gateVal, mirrorSandboxHome, parseRunMeta, sandboxHomeDir, transcriptPathFor, writeManifest, } from "./fixture-support.js";
|
|
37
38
|
import { probeEngines, resolveOpencodeBin } from "./fixture-probe.js";
|
|
@@ -52,7 +53,10 @@ export class FixtureScenariosAdapter {
|
|
|
52
53
|
if (spec.bench.type !== "opencode-fixture-scenarios") {
|
|
53
54
|
cannotAnswer(`fixture adapter: bench.type '${spec.bench.type}' is not "opencode-fixture-scenarios"`, "use ToyBenchAdapter for bench.type 'toy'");
|
|
54
55
|
}
|
|
55
|
-
|
|
56
|
+
// Active-tree resolution follows the run-loop notion (run-loop.ts:142): a
|
|
57
|
+
// concrete repoPath resolves exactly as before; a `${VAR}` env literal
|
|
58
|
+
// resolves at THIS fs boundary or exits 2 naming the variable.
|
|
59
|
+
const root = effectiveRepoPath(spec.repoPath);
|
|
56
60
|
if (!statSync(root, { throwIfNoEntry: false })?.isDirectory()) {
|
|
57
61
|
cannotAnswer(`fixture: repoPath '${spec.repoPath}' is not a readable directory`);
|
|
58
62
|
}
|
|
@@ -84,7 +88,7 @@ export class FixtureScenariosAdapter {
|
|
|
84
88
|
const transcript = transcriptPathFor(sandboxDir, unit.id);
|
|
85
89
|
mkdirSync(path.dirname(transcript), { recursive: true });
|
|
86
90
|
const outcome = await runChild({
|
|
87
|
-
argv: renderCommand(this.spec.bench.runCommand, unitVars(unit, sandboxDir)),
|
|
91
|
+
argv: renderCommand(this.spec.bench.runCommand, unitVars(unit, sandboxDir, this.repoRoot)),
|
|
88
92
|
cwd: sandboxDir,
|
|
89
93
|
timeoutS,
|
|
90
94
|
env: this.sandboxEnv(sandboxDir, unit),
|
|
@@ -109,7 +113,7 @@ export class FixtureScenariosAdapter {
|
|
|
109
113
|
cannotAnswer("fixture adapter: score() called before any reset()/seed()/run() — no sandbox yet", "drive the adapter in reset→seed→run→score order");
|
|
110
114
|
}
|
|
111
115
|
const outcome = await runChild({
|
|
112
|
-
argv: renderCommand(this.spec.bench.graderCommand, unitVars(unit, sandbox)),
|
|
116
|
+
argv: renderCommand(this.spec.bench.graderCommand, unitVars(unit, sandbox, this.repoRoot)),
|
|
113
117
|
cwd: sandbox,
|
|
114
118
|
timeoutS: this.spec.bench.timeoutS,
|
|
115
119
|
env: this.sandboxEnv(sandbox),
|
|
@@ -211,7 +215,7 @@ export class FixtureScenariosAdapter {
|
|
|
211
215
|
if (template === undefined)
|
|
212
216
|
return;
|
|
213
217
|
const outcome = await runChild({
|
|
214
|
-
argv: renderCommand(template, sandboxVars(sandboxDir)),
|
|
218
|
+
argv: renderCommand(template, sandboxVars(sandboxDir, this.repoRoot)),
|
|
215
219
|
cwd: sandboxDir,
|
|
216
220
|
timeoutS: HOOK_TIMEOUT_S,
|
|
217
221
|
env: this.sandboxEnv(sandboxDir),
|
package/dist/bench/toy.js
CHANGED
|
@@ -53,7 +53,7 @@ export class ToyBenchAdapter {
|
|
|
53
53
|
async run(unit, sandboxDir, timeoutS) {
|
|
54
54
|
this.activeSandbox = sandboxDir;
|
|
55
55
|
const outcome = await runChild({
|
|
56
|
-
argv: renderCommand(this.spec.bench.runCommand, unitVars(unit, sandboxDir)),
|
|
56
|
+
argv: renderCommand(this.spec.bench.runCommand, unitVars(unit, sandboxDir, this.repoRoot)),
|
|
57
57
|
cwd: sandboxDir,
|
|
58
58
|
timeoutS,
|
|
59
59
|
...(this.opts.onChild === undefined ? {} : { onChild: this.opts.onChild }),
|
|
@@ -75,7 +75,7 @@ export class ToyBenchAdapter {
|
|
|
75
75
|
cannotAnswer("toy adapter: score() called before any reset()/seed()/run() — no sandbox yet", "drive the adapter in reset→seed→run→score order");
|
|
76
76
|
}
|
|
77
77
|
const outcome = await runChild({
|
|
78
|
-
argv: renderCommand(this.spec.bench.graderCommand, unitVars(unit, sandbox)),
|
|
78
|
+
argv: renderCommand(this.spec.bench.graderCommand, unitVars(unit, sandbox, this.repoRoot)),
|
|
79
79
|
cwd: sandbox,
|
|
80
80
|
timeoutS: this.spec.bench.timeoutS,
|
|
81
81
|
...(this.opts.onChild === undefined ? {} : { onChild: this.opts.onChild }),
|
|
@@ -109,7 +109,7 @@ export class ToyBenchAdapter {
|
|
|
109
109
|
if (template === undefined)
|
|
110
110
|
return;
|
|
111
111
|
const outcome = await runChild({
|
|
112
|
-
argv: renderCommand(template, sandboxVars(sandboxDir)),
|
|
112
|
+
argv: renderCommand(template, sandboxVars(sandboxDir, this.repoRoot)),
|
|
113
113
|
cwd: sandboxDir,
|
|
114
114
|
timeoutS: HOOK_TIMEOUT_S,
|
|
115
115
|
...(this.opts.onChild === undefined ? {} : { onChild: this.opts.onChild }),
|
package/dist/commands/status.js
CHANGED
|
@@ -15,6 +15,7 @@ import { tryGit } from "../util/git.js";
|
|
|
15
15
|
import { readRegistry } from "../core/genome.js";
|
|
16
16
|
import { LEDGER_KIND_GRAFT_IMPORT, decodeGraftImport, listGraftQueue, } from "../core/graft-support.js";
|
|
17
17
|
import { resolveUniqueEntry } from "./run.js";
|
|
18
|
+
import { effectiveRepoPath } from "../core/spec.js";
|
|
18
19
|
const KIND_GENERATION = "generation_complete";
|
|
19
20
|
const KIND_PROMOTE = "promote";
|
|
20
21
|
const KIND_TOMBSTONE = "tombstone";
|
|
@@ -113,7 +114,11 @@ async function runStatus(args) {
|
|
|
113
114
|
renderQueue(queue);
|
|
114
115
|
return EXIT_OK;
|
|
115
116
|
}
|
|
116
|
-
|
|
117
|
+
// Resolve at the fs seam exactly like run/genome/kernel (run-loop.ts:142):
|
|
118
|
+
// feeding the stored UNRESOLVED `${VAR}` literal to git/ledger paths makes
|
|
119
|
+
// Node report the missing cwd as the misleading `spawn git ENOENT`; an unset
|
|
120
|
+
// variable is now a clean exit 2 that names it.
|
|
121
|
+
const repo = effectiveRepoPath(entry.spec.repoPath);
|
|
117
122
|
const records = readLedgerRecords(ledgerPath(repo));
|
|
118
123
|
const gens = generations(records);
|
|
119
124
|
writeStdout(`genome: ${flat(entry.label)} (${entry.fingerprint}) repo: ${flat(repo, 200)}`);
|
|
@@ -4,6 +4,7 @@ import path from "node:path";
|
|
|
4
4
|
import { cannotAnswer } from "../../exit.js";
|
|
5
5
|
import { runId } from "../ids.js";
|
|
6
6
|
import { buildManifest } from "../kernel.js";
|
|
7
|
+
import { effectiveRepoPath } from "../spec.js";
|
|
7
8
|
import { budgetExhausted } from "../stats.js";
|
|
8
9
|
import { ToyBenchAdapter } from "../../bench/toy.js";
|
|
9
10
|
import { FixtureScenariosAdapter } from "../../bench/fixture.js";
|
|
@@ -164,7 +165,7 @@ export async function benchTarget(o) {
|
|
|
164
165
|
spent,
|
|
165
166
|
provenance: provenance ?? { benchType: o.spec.bench.type, versions: [] },
|
|
166
167
|
complete,
|
|
167
|
-
manifest: buildManifest(
|
|
168
|
+
manifest: buildManifest(effectiveRepoPath(o.spec.repoPath), o.spec.kernel.immutableGlobs),
|
|
168
169
|
failures,
|
|
169
170
|
};
|
|
170
171
|
}
|
|
@@ -31,3 +31,27 @@ test("renderCommand: quote-aware argv, placeholders, fail-closed junk (exit 2)",
|
|
|
31
31
|
expectExit2(() => renderCommand(" ", unitVars(unit, "/s")), "empty");
|
|
32
32
|
expectExit2(() => renderCommand('node "oops', unitVars(unit, "/s")), "unbalanced quote");
|
|
33
33
|
});
|
|
34
|
+
// S4 engine seam: {repoRoot} = the bench's ACTIVE TREE, threaded through the
|
|
35
|
+
// shared CommandVars constructors so both adapters resolve it identically.
|
|
36
|
+
// Without a threaded repoRoot the placeholder must stay FAIL-CLOSED (exit 2),
|
|
37
|
+
// exactly like any unknown placeholder — never a half-substituted command.
|
|
38
|
+
test("{repoRoot}: unitVars/sandboxVars carry the active tree into run/grader/hook templates", () => {
|
|
39
|
+
const unit = { id: "u1", path: "scenarios/a.md", split: "train" };
|
|
40
|
+
assert.deepEqual(renderCommand("bash run-scenario.sh {unit.id} {repoRoot}/{unit.path}", unitVars(unit, "/sbx", "/active/tree")), ["bash", "run-scenario.sh", "u1", "/active/tree/scenarios/a.md"]);
|
|
41
|
+
assert.deepEqual(renderCommand("bash seed-wrapped.sh {repoRoot}/seed_sandbox.sh", sandboxVars("/sbx", "/active/tree")), ["bash", "seed-wrapped.sh", "/active/tree/seed_sandbox.sh"]);
|
|
42
|
+
// pre-existing keys untouched by the seam
|
|
43
|
+
assert.deepEqual(unitVars(unit, "/sbx", "/r"), {
|
|
44
|
+
"unit.path": "scenarios/a.md",
|
|
45
|
+
"unit.id": "u1",
|
|
46
|
+
sandbox: "/sbx",
|
|
47
|
+
workdir: "/sbx",
|
|
48
|
+
repoRoot: "/r",
|
|
49
|
+
});
|
|
50
|
+
assert.deepEqual(sandboxVars("/sbx", "/r"), { sandbox: "/sbx", workdir: "/sbx", repoRoot: "/r" });
|
|
51
|
+
});
|
|
52
|
+
test("{repoRoot} unthreaded ⇒ fail-closed exit 2 (unknown placeholder behavior UNCHANGED)", () => {
|
|
53
|
+
const unit = { id: "u1", path: "p", split: "train" };
|
|
54
|
+
expectExit2(() => renderCommand("node x {repoRoot}", unitVars(unit, "/s")), "unknown placeholder");
|
|
55
|
+
expectExit2(() => renderCommand("node x {repoRoot}", sandboxVars("/s")), "unknown placeholder");
|
|
56
|
+
expectExit2(() => renderCommand("node x {repoRoot}", { worktree: "/w", brief: "/b" }), "unknown placeholder");
|
|
57
|
+
});
|
|
@@ -62,9 +62,13 @@ fi
|
|
|
62
62
|
echo "ran $unit"
|
|
63
63
|
echo '{"tokensEst":123,"turns":4}'
|
|
64
64
|
`;
|
|
65
|
+
// Unique sleep duration: node --test runs files concurrently and this test's
|
|
66
|
+
// orphan check scans the whole machine's `ps` output — "sleep 31.7" is the
|
|
67
|
+
// shipped toy genome's hang marker (bench-toy.test.ts); reusing it here made
|
|
68
|
+
// the two files' checks see each other's live sleeps (CI-verified 5/5 collision).
|
|
65
69
|
const BIN_HANG = `#!/usr/bin/env bash
|
|
66
70
|
if [ "\${1:-}" = "--version" ]; then echo "1.2.3"; exit 0; fi
|
|
67
|
-
exec sleep 31
|
|
71
|
+
exec sleep 29.31
|
|
68
72
|
`;
|
|
69
73
|
const BIN_NO_VERSION = `#!/usr/bin/env bash
|
|
70
74
|
if [ "\${1:-}" = "--version" ]; then echo "unsupported flag" >&2; exit 1; fi
|
|
@@ -296,7 +300,7 @@ test("(E) hanging unit: group kill at timeoutS, zero orphans", async (t) => {
|
|
|
296
300
|
assert.equal(run.exitCode, null);
|
|
297
301
|
assert.match(run.note ?? "", /killed after 1s: process group SIGKILL/);
|
|
298
302
|
const ps = spawnSync("ps", ["-eo", "args"], { encoding: "utf8" });
|
|
299
|
-
assert.ok(!ps.stdout.includes("sleep 31
|
|
303
|
+
assert.ok(!ps.stdout.includes("sleep 29.31"), "orphaned sleep survived the group kill");
|
|
300
304
|
});
|
|
301
305
|
// ------------------------------------------------------------------ AC (F)
|
|
302
306
|
test("(F) infra_failed is recorded with a distinct status/note, never a silent 0", async (t) => {
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
// S4 engine seam — the {repoRoot} command variable. The mutation channel's
|
|
2
|
+
// delivery hinge: run/grader/seed/reset templates must resolve {repoRoot} to
|
|
3
|
+
// the bench's ACTIVE TREE — the genome repoPath for the incumbent, the sealed
|
|
4
|
+
// candidate worktree for a candidate (run-loop.ts swaps `spec.repoPath` before
|
|
5
|
+
// benching each candidate; the adapter must follow that same notion, including
|
|
6
|
+
// effectiveRepoPath's env-literal resolution at the FS boundary).
|
|
7
|
+
// Unknown placeholders stay fail-closed (exit 2). Offline: capture scripts
|
|
8
|
+
// record the rendered argv; the fake opencode only answers --version.
|
|
9
|
+
import assert from "node:assert/strict";
|
|
10
|
+
import { chmodSync, cpSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
|
11
|
+
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
|
12
|
+
import * as os from "node:os";
|
|
13
|
+
import path from "node:path";
|
|
14
|
+
import { test } from "node:test";
|
|
15
|
+
import { ExitSignal } from "../exit.js";
|
|
16
|
+
import { parseGenomeSpecDocument } from "../core/spec.js";
|
|
17
|
+
import { FixtureScenariosAdapter } from "../bench/fixture.js";
|
|
18
|
+
// ------------------------------------------------------------------- helpers
|
|
19
|
+
async function freshDir(t, prefix) {
|
|
20
|
+
const dir = await mkdtemp(path.join(os.tmpdir(), `abathur-${prefix}-`));
|
|
21
|
+
t.after(() => rm(dir, { recursive: true, force: true }));
|
|
22
|
+
return dir;
|
|
23
|
+
}
|
|
24
|
+
async function writeScript(dir, name, body) {
|
|
25
|
+
const filePath = path.join(dir, name);
|
|
26
|
+
await writeFile(filePath, body, "utf8");
|
|
27
|
+
chmodSync(filePath, 0o755);
|
|
28
|
+
return filePath;
|
|
29
|
+
}
|
|
30
|
+
/** Records `argv[mode, out, ...rest]`: rest joined per line into `out`; prints the metric/score line. */
|
|
31
|
+
const CAPTURE = `#!/usr/bin/env node
|
|
32
|
+
import { writeFileSync } from "node:fs";
|
|
33
|
+
const [mode, out, ...rest] = process.argv.slice(2);
|
|
34
|
+
writeFileSync(out, rest.join("\\n") + "\\n", "utf8");
|
|
35
|
+
if (mode === "score") {
|
|
36
|
+
process.stdout.write(JSON.stringify({ unit: "one", score: 1, pass: true, metrics: { tokensEst: 3, turns: 1 } }) + "\\n");
|
|
37
|
+
} else {
|
|
38
|
+
process.stdout.write(JSON.stringify({ tokensEst: 3, turns: 1 }) + "\\n");
|
|
39
|
+
}
|
|
40
|
+
`;
|
|
41
|
+
const FAKE_OC = `#!/usr/bin/env bash
|
|
42
|
+
if [ "\${1:-}" = "--version" ]; then echo "1.2.3"; exit 0; fi
|
|
43
|
+
exit 0
|
|
44
|
+
`;
|
|
45
|
+
const UNIT = { id: "one", path: "scenarios/one.md", split: "train" };
|
|
46
|
+
const VAL_UNIT = { id: "v1", path: "scenarios/v1.md", split: "val" };
|
|
47
|
+
async function seamFixture(t, genDir) {
|
|
48
|
+
const root = await freshDir(t, "seam-ad");
|
|
49
|
+
const capPath = await writeScript(root, "cap.mjs", CAPTURE);
|
|
50
|
+
const binDir = await freshDir(t, "seam-bin");
|
|
51
|
+
const binPath = await writeScript(binDir, "oc-fake", FAKE_OC);
|
|
52
|
+
const home = await freshDir(t, "seam-home");
|
|
53
|
+
const configDir = await freshDir(t, "seam-cfg");
|
|
54
|
+
const outDir = path.join(root, "caps");
|
|
55
|
+
mkdirSync(outDir, { recursive: true });
|
|
56
|
+
const out = (name) => path.join(outDir, `${name}.txt`);
|
|
57
|
+
const spec = parseGenomeSpecDocument({
|
|
58
|
+
label: "seam-germ",
|
|
59
|
+
repoPath: genDir,
|
|
60
|
+
bench: {
|
|
61
|
+
type: "opencode-fixture-scenarios",
|
|
62
|
+
units: [UNIT, VAL_UNIT],
|
|
63
|
+
runCommand: `node ${capPath} run ${out("run")} {repoRoot} {unit.path}`,
|
|
64
|
+
seedCommand: `node ${capPath} seed ${out("seed")} {repoRoot}`,
|
|
65
|
+
resetCommand: `node ${capPath} reset ${out("reset")} {repoRoot}`,
|
|
66
|
+
graderCommand: `node ${capPath} score ${out("grader")} {repoRoot}`,
|
|
67
|
+
agentModel: "test/agent",
|
|
68
|
+
timeoutS: 30,
|
|
69
|
+
stats: { halfWidth: 0.15, minEffect: 0.1, nReps: { initial: 1, max: 2 } },
|
|
70
|
+
},
|
|
71
|
+
budget: { maxCandidates: 2, maxModelCalls: 4, maxTokens: 10000, maxWallS: 600 },
|
|
72
|
+
kernel: { immutableGlobs: [] },
|
|
73
|
+
requires: [{ cmd: "node", args: ["--version"], probeExit: 0 }],
|
|
74
|
+
opencodeBinVersion: { minVersion: "1.0.0" },
|
|
75
|
+
}, "<test>");
|
|
76
|
+
const sandbox = path.join(root, "sandbox");
|
|
77
|
+
return {
|
|
78
|
+
spec,
|
|
79
|
+
genDir,
|
|
80
|
+
sandbox,
|
|
81
|
+
home,
|
|
82
|
+
cap: capPath,
|
|
83
|
+
binPath,
|
|
84
|
+
out,
|
|
85
|
+
adapter: (s) => new FixtureScenariosAdapter(s, { env: { HOME: home }, home, configDir, opencodeBin: binPath }),
|
|
86
|
+
};
|
|
87
|
+
}
|
|
88
|
+
/** Minimal genome dir (the fixture adapter only requires repoPath to be a directory). */
|
|
89
|
+
async function genomeTree(t, name = "germ") {
|
|
90
|
+
const genRoot = await freshDir(t, `seam-${name}`);
|
|
91
|
+
const genDir = path.join(genRoot, "germ");
|
|
92
|
+
mkdirSync(path.join(genDir, "scenarios"), { recursive: true });
|
|
93
|
+
writeFileSync(path.join(genDir, "scenarios", "one.md"), "# one\n\n## Brief\ndo it\n", "utf8");
|
|
94
|
+
writeFileSync(path.join(genDir, "scenarios", "v1.md"), "# v1\n\n## Brief\ndo it val\n", "utf8");
|
|
95
|
+
return genDir;
|
|
96
|
+
}
|
|
97
|
+
function readCap(file) {
|
|
98
|
+
return readFileSync(file, "utf8").trim();
|
|
99
|
+
}
|
|
100
|
+
function isExit2Matching(needle) {
|
|
101
|
+
return (cause) => cause instanceof ExitSignal && cause.code === 2 && needle.test(cause.message);
|
|
102
|
+
}
|
|
103
|
+
// --------------------------------------------------------------------- pins
|
|
104
|
+
test("{repoRoot} renders to the incumbent genome repoPath in run + grader + hooks", async (t) => {
|
|
105
|
+
const genDir = await genomeTree(t);
|
|
106
|
+
const f = await seamFixture(t, genDir);
|
|
107
|
+
const adapter = f.adapter(f.spec);
|
|
108
|
+
try {
|
|
109
|
+
await adapter.reset(f.sandbox);
|
|
110
|
+
await adapter.seed(f.sandbox);
|
|
111
|
+
const run = await adapter.run(UNIT, f.sandbox, 30);
|
|
112
|
+
assert.equal(run.status, "ok");
|
|
113
|
+
const score = await adapter.score(UNIT);
|
|
114
|
+
assert.equal(score.kind, "scored");
|
|
115
|
+
assert.equal(readCap(f.out("run")), `${genDir}\nscenarios/one.md`);
|
|
116
|
+
assert.equal(readCap(f.out("grader")), genDir);
|
|
117
|
+
assert.equal(readCap(f.out("seed")), genDir);
|
|
118
|
+
assert.equal(readCap(f.out("reset")), genDir);
|
|
119
|
+
}
|
|
120
|
+
finally {
|
|
121
|
+
adapter.release();
|
|
122
|
+
}
|
|
123
|
+
});
|
|
124
|
+
test("{repoRoot} follows the candidate worktree when spec.repoPath is swapped (run-loop.ts pattern)", async (t) => {
|
|
125
|
+
const genDir = await genomeTree(t);
|
|
126
|
+
const f = await seamFixture(t, genDir);
|
|
127
|
+
// simulate run-loop.ts:279 — benchTarget receives { ...spec, repoPath: worktreePath }
|
|
128
|
+
const worktree = path.join(path.dirname(genDir), "wt");
|
|
129
|
+
cpSync(genDir, worktree, { recursive: true });
|
|
130
|
+
mkdirSync(path.join(worktree, "abathur-notes"), { recursive: true });
|
|
131
|
+
writeFileSync(path.join(worktree, "abathur-notes", "card.md"), "planted run card\n", "utf8");
|
|
132
|
+
const candidate = f.adapter({ ...f.spec, repoPath: worktree });
|
|
133
|
+
try {
|
|
134
|
+
const sandbox = path.join(f.sandbox, "cand");
|
|
135
|
+
await candidate.reset(sandbox);
|
|
136
|
+
await candidate.seed(sandbox);
|
|
137
|
+
const run = await candidate.run(UNIT, sandbox, 30);
|
|
138
|
+
assert.equal(run.status, "ok");
|
|
139
|
+
const score = await candidate.score(UNIT);
|
|
140
|
+
assert.equal(score.kind, "scored");
|
|
141
|
+
assert.equal(readCap(f.out("run")), `${worktree}\nscenarios/one.md`);
|
|
142
|
+
assert.equal(readCap(f.out("grader")), worktree);
|
|
143
|
+
}
|
|
144
|
+
finally {
|
|
145
|
+
candidate.release();
|
|
146
|
+
}
|
|
147
|
+
});
|
|
148
|
+
test("fixture ctor resolves an env-literal repoPath via effectiveRepoPath (run-loop.ts:142 pattern)", async (t) => {
|
|
149
|
+
const genDir = await genomeTree(t, "lit");
|
|
150
|
+
const f = await seamFixture(t, genDir);
|
|
151
|
+
const litSpec = { ...f.spec, repoPath: "${ABATHUR_SEAM_LIT_REPO}" };
|
|
152
|
+
const base = {
|
|
153
|
+
env: { HOME: f.home },
|
|
154
|
+
home: f.home,
|
|
155
|
+
configDir: path.join(f.home, ".config", "abathur"),
|
|
156
|
+
opencodeBin: f.binPath,
|
|
157
|
+
};
|
|
158
|
+
delete process.env["ABATHUR_SEAM_LIT_REPO"];
|
|
159
|
+
try {
|
|
160
|
+
// unset env ⇒ clean exit 2 NAMING the variable (never a garbage-path spawn error)
|
|
161
|
+
assert.throws(() => new FixtureScenariosAdapter(litSpec, base), isExit2Matching(/requires env var ABATHUR_SEAM_LIT_REPO/));
|
|
162
|
+
// set env ⇒ resolves at the FS boundary; {repoRoot} renders to the resolved tree
|
|
163
|
+
process.env["ABATHUR_SEAM_LIT_REPO"] = genDir;
|
|
164
|
+
const adapter = new FixtureScenariosAdapter(litSpec, base);
|
|
165
|
+
try {
|
|
166
|
+
const sandbox = path.join(f.sandbox, "lit");
|
|
167
|
+
await adapter.reset(sandbox);
|
|
168
|
+
await adapter.seed(sandbox);
|
|
169
|
+
const run = await adapter.run(UNIT, sandbox, 30);
|
|
170
|
+
assert.equal(run.status, "ok");
|
|
171
|
+
assert.equal(readCap(f.out("run")), `${genDir}\nscenarios/one.md`);
|
|
172
|
+
}
|
|
173
|
+
finally {
|
|
174
|
+
adapter.release();
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
finally {
|
|
178
|
+
delete process.env["ABATHUR_SEAM_LIT_REPO"];
|
|
179
|
+
}
|
|
180
|
+
});
|
|
181
|
+
test("fail-closed UNCHANGED: unknown placeholder through the adapter is still exit 2", async (t) => {
|
|
182
|
+
const genDir = await genomeTree(t, "unk");
|
|
183
|
+
const f = await seamFixture(t, genDir);
|
|
184
|
+
const bad = {
|
|
185
|
+
...f.spec,
|
|
186
|
+
bench: { ...f.spec.bench, runCommand: `node ${f.cap} run ${f.out("bad")} {nope}` },
|
|
187
|
+
};
|
|
188
|
+
const adapter = f.adapter(bad);
|
|
189
|
+
try {
|
|
190
|
+
await adapter.reset(f.sandbox);
|
|
191
|
+
await adapter.seed(f.sandbox);
|
|
192
|
+
await assert.rejects(() => adapter.run(UNIT, f.sandbox, 30), isExit2Matching(/unknown placeholder/));
|
|
193
|
+
}
|
|
194
|
+
finally {
|
|
195
|
+
adapter.release();
|
|
196
|
+
}
|
|
197
|
+
});
|