chivgent 0.15.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +39 -0
- package/README.zh-CN.md +29 -0
- package/dist/cli-options.d.ts.map +1 -1
- package/dist/cli-options.js +6 -0
- package/dist/cli-options.js.map +1 -1
- package/dist/evals/cli.js +2 -0
- package/dist/evals/cli.js.map +1 -1
- package/dist/evals/graders.d.ts +2 -0
- package/dist/evals/graders.d.ts.map +1 -1
- package/dist/evals/graders.js +70 -0
- package/dist/evals/graders.js.map +1 -1
- package/dist/evals/parse-args.d.ts.map +1 -1
- package/dist/evals/parse-args.js +8 -1
- package/dist/evals/parse-args.js.map +1 -1
- package/dist/evals/report.d.ts.map +1 -1
- package/dist/evals/report.js +3 -2
- package/dist/evals/report.js.map +1 -1
- package/dist/evals/runner.d.ts +30 -1
- package/dist/evals/runner.d.ts.map +1 -1
- package/dist/evals/runner.js +45 -25
- package/dist/evals/runner.js.map +1 -1
- package/dist/evals/task.d.ts.map +1 -1
- package/dist/evals/task.js +35 -1
- package/dist/evals/task.js.map +1 -1
- package/dist/evals/tools.d.ts +5 -0
- package/dist/evals/tools.d.ts.map +1 -0
- package/dist/evals/tools.js +40 -0
- package/dist/evals/tools.js.map +1 -0
- package/dist/providers/usage.d.ts.map +1 -1
- package/dist/providers/usage.js +3 -1
- package/dist/providers/usage.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -498,8 +498,13 @@ nothing about whether this is a good agent. `npm run eval` answers that:
|
|
|
498
498
|
npm run eval # every task in evals/
|
|
499
499
|
npm run eval -- --task rename-symbol --attempts 10
|
|
500
500
|
npm run eval -- --allow-writes --allow-shell --json report.json
|
|
501
|
+
DEEPSEEK_API_KEY=sk-... npm run eval -- --provider deepseek
|
|
501
502
|
```
|
|
502
503
|
|
|
504
|
+
`--api-key` works too, but prefer the environment variable here: npm echoes the
|
|
505
|
+
fully expanded command before running it, so a key on the command line lands in
|
|
506
|
+
the clear in any captured log.
|
|
507
|
+
|
|
503
508
|
```text
|
|
504
509
|
task pass turns tokens tools p50
|
|
505
510
|
find-auth-logic 4/5 2.0 3.4k list_files,search_text,... 6.1s
|
|
@@ -549,6 +554,40 @@ right contents, and an eval that scores results alone would give it full marks.
|
|
|
549
554
|
For a coding agent, tool misuse is the more common and the more interesting
|
|
550
555
|
failure, so tasks assert on how the work was done.
|
|
551
556
|
|
|
557
|
+
A grader naming a tool the task never grants is refused when the task loads.
|
|
558
|
+
`not-used-tool: write_file` on a read-only task cannot fail — it asserts the
|
|
559
|
+
model did not call something it was never given — and its mirror image cannot
|
|
560
|
+
pass. Both are task-definition mistakes, and both are caught before the run
|
|
561
|
+
spends anything. One such grader shipped for two releases before this check
|
|
562
|
+
existed.
|
|
563
|
+
|
|
564
|
+
Three graders exist because a set of tool names cannot answer the question they
|
|
565
|
+
answer. `tool-never-failed` catches a guessed path, which shows up as a failed
|
|
566
|
+
read that a deduplicated name list hides — it is sound only where a failed call
|
|
567
|
+
means the model was wrong, so it fits `read_file` and not `bash`, which reports
|
|
568
|
+
any non-zero exit as an error and is *supposed* to fail on a task that runs a
|
|
569
|
+
failing test. `max-tool-calls` budgets calls rather than turns, because a model
|
|
570
|
+
may issue any number of calls in one turn: "did it follow the imports or read
|
|
571
|
+
the whole project" is a call count. `file-unchanged` compares against the
|
|
572
|
+
fixture, so "don't touch what you weren't asked to touch" becomes a verdict
|
|
573
|
+
rather than an intention — and it needs no second copy of the file in
|
|
574
|
+
`task.json` to drift out of date. The JSON report carries `toolCalls`, every
|
|
575
|
+
call in order with whether it succeeded, alongside the `toolsUsed` set the table
|
|
576
|
+
prints.
|
|
577
|
+
|
|
578
|
+
The nine tasks split into what the agent found and what it changed. Four exist
|
|
579
|
+
because the first real baseline passed 29 of 30 attempts and therefore measured
|
|
580
|
+
nothing: `needle-in-many-files` (40 files with uninformative names, so listing
|
|
581
|
+
gives no signal), `trace-the-default` (a value overridden twice, where stopping
|
|
582
|
+
at the first hop yields a plausible wrong number), `decoy-config` (two
|
|
583
|
+
near-identical modules, one of them dead), and `wrong-test` (a failing test
|
|
584
|
+
whose expectation contradicts the contract documented in the source — bending
|
|
585
|
+
the source to fit it is the trap).
|
|
586
|
+
|
|
587
|
+
A task that scores 0/N or N/N across different models is not measuring anything
|
|
588
|
+
and should be made harder, fixed, or dropped. Passing is not evidence of quality
|
|
589
|
+
when everything passes.
|
|
590
|
+
|
|
552
591
|
## Token usage
|
|
553
592
|
|
|
554
593
|
Providers report what each call cost, and chivgent now keeps it: per turn on
|
package/README.zh-CN.md
CHANGED
|
@@ -477,8 +477,12 @@ npm install -g ./chivgent-0.6.0.tgz
|
|
|
477
477
|
npm run eval # 跑 evals/ 下全部任务
|
|
478
478
|
npm run eval -- --task rename-symbol --attempts 10
|
|
479
479
|
npm run eval -- --allow-writes --allow-shell --json report.json
|
|
480
|
+
DEEPSEEK_API_KEY=sk-... npm run eval -- --provider deepseek
|
|
480
481
|
```
|
|
481
482
|
|
|
483
|
+
`--api-key` 也能用，但这里更推荐环境变量：npm 会在执行前把展开后的完整命令回显出来，
|
|
484
|
+
写在命令行上的 key 会明文出现在任何被采集的日志里。
|
|
485
|
+
|
|
482
486
|
```text
|
|
483
487
|
task pass turns tokens tools p50
|
|
484
488
|
find-auth-logic 4/5 2.0 3.4k list_files,search_text,... 6.1s
|
|
@@ -523,6 +527,31 @@ rename-symbol attempt 2,5: not-used-tool name="write_file" — called write_file
|
|
|
523
527
|
对编码 Agent 来说，**工具误用是比答错更常见、也更值得测的失败**，所以任务不仅判定
|
|
524
528
|
"做对了没有"，还判定"是怎么做的"。
|
|
525
529
|
|
|
530
|
+
**一条判定如果点名了这个任务拿不到的工具，加载任务时就会报错。** 只读任务上的
|
|
531
|
+
`not-used-tool: write_file` 永远不可能失败——它断言模型没调用一个它从没拿到过的工具；
|
|
532
|
+
反过来 `used-tool` 一个拿不到的工具则永远不可能通过。两者都是任务定义写错了，都在花第一
|
|
533
|
+
分钱之前拦下来。这类判定曾经在仓库里躺了两个版本。
|
|
534
|
+
|
|
535
|
+
有三条判定的存在，是因为"用过哪些工具"这个集合回答不了它们回答的问题。
|
|
536
|
+
`tool-never-failed` 抓瞎猜路径——猜错的读必然报错，而去重后的名字集合看不见这件事。
|
|
537
|
+
它只在"失败即出错"的工具上成立：`read_file` 成立，`bash` 不成立——后者对任何非零退出码
|
|
538
|
+
都报错，而在"先跑一次让它挂"的任务上，失败正是对的。`max-tool-calls` 卡的是调用次数
|
|
539
|
+
而不是轮数，因为一轮里可以发起任意多次调用："到底是顺着 import 找过去，还是把整个项目
|
|
540
|
+
读了一遍"，这是个次数问题；
|
|
541
|
+
`file-unchanged` 拿 fixture 原件做字节比较，把"别动没让你动的东西"从愿望变成判定，
|
|
542
|
+
而且不需要在 `task.json` 里再抄一份文件内容（抄一份就会有忘记同步的那天）。
|
|
543
|
+
JSON 报告里多了 `toolCalls`：按调用顺序记录每一次调用和它成功与否，和表格用的
|
|
544
|
+
`toolsUsed` 集合并存。
|
|
545
|
+
|
|
546
|
+
九个任务分成"找到了什么"和"改了什么"两类。其中四个的存在理由是：第一次真实基线
|
|
547
|
+
30 次里过了 29 次，也就是什么都没测出来——`needle-in-many-files`（40 个文件名毫无
|
|
548
|
+
信息量，`list_files` 给不出线索）、`trace-the-default`（一个值被覆盖两次，停在第一跳
|
|
549
|
+
会得到一个看起来很合理的错数字）、`decoy-config`（两个几乎一样的模块，其中一个是死代码）、
|
|
550
|
+
`wrong-test`（挂掉的测试，其期望与源码里写明的契约矛盾，改源码去迎合它就是陷阱）。
|
|
551
|
+
|
|
552
|
+
**一个任务如果在不同模型上都是 0/N 或都是 N/N，它就没在测东西**，应该改难、改对，
|
|
553
|
+
或者删掉。全都过不是质量的证据。
|
|
554
|
+
|
|
526
555
|
## Token 用量
|
|
527
556
|
|
|
528
557
|
Provider 每次调用都会报告代价，chivgent 现在会把它留下来：每轮在 `message_end` 上、
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli-options.d.ts","sourceRoot":"","sources":["../src/cli-options.ts"],"names":[],"mappings":"AAKA,eAAO,MAAM,OAAO,QAC6D,CAAC;AAElF,OAAO,EAEL,gBAAgB,EACjB,MAAM,yBAAyB,CAAC;AAGjC,2CAA2C;AAC3C,MAAM,MAAM,QAAQ,GAAG,MAAM,CAAC;AAE9B,eAAO,MAAM,iBAAiB,IAAI,CAAC;AACnC;;;;GAIG;AACH,eAAO,MAAM,uBAAuB,KAAK,CAAC;AAI1C;;;GAGG;AACH,MAAM,MAAM,cAAc,GAAG,MAAM,CAAC,UAAU,CAAC;AAE/C,MAAM,WAAW,UAAU;IACzB,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,QAAQ,EAAE,QAAQ,CAAC;IAC5B,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,MAAM,EAAE,OAAO,CAAC;IACzB,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;IACxB,0EAA0E;IAC1E,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;IACvB,4DAA4D;IAC5D,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;IAC1B,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC;IACzB,kEAAkE;IAClE,QAAQ,CAAC,eAAe,EAAE,OAAO,CAAC;IAClC,uCAAuC;IACvC,QAAQ,CAAC,YAAY,EAAE,OAAO,CAAC;IAC/B,mEAAmE;IACnE,QAAQ,CAAC,WAAW,EAAE,OAAO,CAAC;IAC9B;;;OAGG;IACH,QAAQ,CAAC,UAAU,EAAE,OAAO,CAAC;IAC7B,8EAA8E;IAC9E,QAAQ,CAAC,UAAU,EAAE,OAAO,CAAC;IAC7B,kEAAkE;IAClE,QAAQ,CAAC,cAAc,EAAE,OAAO,CAAC;IACjC,kEAAkE;IAClE,QAAQ,CAAC,WAAW,EAAE,OAAO,CAAC;IAC9B,8DAA8D;IAC9D,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;IACxB,2EAA2E;IAC3E,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,CAAC;IAC1B,4DAA4D;IAC5D,QAAQ,CAAC,WAAW,EAAE,OAAO,CAAC;IAC9B,yEAAyE;IACzE,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC;IACzB,+CAA+C;IAC/C,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAC;IAC/B,oEAAoE;IACpE,QAAQ,CAAC,UAAU,EAAE,OAAO,CAAC;IAC7B,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;IACvB,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;CAC3B;AAED,wBAAgB,YAAY,CAC1B,IAAI,EAAE,SAAS,MAAM,EAAE,EACvB,WAAW,EAAE,cAAc,EAC3B,QAAQ,GAAE,gBAA0C,GACnD,UAAU,
|
|
1
|
+
{"version":3,"file":"cli-options.d.ts","sourceRoot":"","sources":["../src/cli-options.ts"],"names":[],"mappings":"AAKA,eAAO,MAAM,OAAO,QAC6D,CAAC;AAElF,OAAO,EAEL,gBAAgB,EACjB,MAAM,yBAAyB,CAAC;AAGjC,2CAA2C;AAC3C,MAAM,MAAM,QAAQ,GAAG,MAAM,CAAC;AAE9B,eAAO,MAAM,iBAAiB,IAAI,CAAC;AACnC;;;;GAIG;AACH,eAAO,MAAM,uBAAuB,KAAK,CAAC;AAI1C;;;GAGG;AACH,MAAM,MAAM,cAAc,GAAG,MAAM,CAAC,UAAU,CAAC;AAE/C,MAAM,WAAW,UAAU;IACzB,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,QAAQ,EAAE,QAAQ,CAAC;IAC5B,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,MAAM,EAAE,OAAO,CAAC;IACzB,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;IACxB,0EAA0E;IAC1E,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;IACvB,4DAA4D;IAC5D,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;IAC1B,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC;IACzB,kEAAkE;IAClE,QAAQ,CAAC,eAAe,EAAE,OAAO,CAAC;IAClC,uCAAuC;IACvC,QAAQ,CAAC,YAAY,EAAE,OAAO,CAAC;IAC/B,mEAAmE;IACnE,QAAQ,CAAC,WAAW,EAAE,OAAO,CAAC;IAC9B;;;OAGG;IACH,QAAQ,CAAC,UAAU,EAAE,OAAO,CAAC;IAC7B,8EAA8E;IAC9E,QAAQ,CAAC,UAAU,EAAE,OAAO,CAAC;IAC7B,kEAAkE;IAClE,QAAQ,CAAC,cAAc,EAAE,OAAO,CAAC;IACjC,kEAAkE;IAClE,QAAQ,CAAC,WAAW,EAAE,OAAO,CAAC;IAC9B,8DAA8D;IAC9D,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;IACxB,2EAA2E;IAC3E,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,CAAC;IAC1B,4DAA4D;IAC5D,QAAQ,CAAC,WAAW,EAAE,OAAO,CAAC;IAC9B,yEAAyE;IACzE,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC;IACzB,+CAA+C;IAC/C,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAC;IAC/B,oEAAoE;IACpE,QAAQ,CAAC,UAAU,EAAE,OAAO,CAAC;IAC7B,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;IACvB,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;CAC3B;AAED,wBAAgB,YAAY,CAC1B,IAAI,EAAE,SAAS,MAAM,EAAE,EACvB,WAAW,EAAE,cAAc,EAC3B,QAAQ,GAAE,gBAA0C,GACnD,UAAU,CA0IZ;AAED,wBAAgB,QAAQ,CACtB,QAAQ,GAAE,gBAA0C,GACnD,MAAM,CAuDR"}
|
package/dist/cli-options.js
CHANGED
|
@@ -58,6 +58,12 @@ export function parseCliArgs(argv, environment, registry = defaultProviderRegist
|
|
|
58
58
|
}
|
|
59
59
|
else if (argument === "--api-key") {
|
|
60
60
|
apiKey = readOptionValue(argv, index, "--api-key");
|
|
61
|
+
if (apiKey.length === 0) {
|
|
62
|
+
// An empty key would fall through to the environment and then the
|
|
63
|
+
// stored file, so an unset shell variable would silently run against
|
|
64
|
+
// a different credential than the one named on the command line.
|
|
65
|
+
throw new TypeError("--api-key requires a value.");
|
|
66
|
+
}
|
|
61
67
|
index += 1;
|
|
62
68
|
}
|
|
63
69
|
else if (argument === "--max-turns") {
|
package/dist/cli-options.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli-options.js","sourceRoot":"","sources":["../src/cli-options.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,OAAO,GAAG,aAAa,CAAC,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC;AAC/C,MAAM,eAAe,GAAG,OAAO,CAAC,iBAAiB,CAAmC,CAAC;AAErF,MAAM,CAAC,MAAM,OAAO,GAClB,OAAO,eAAe,CAAC,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,eAAe,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;AAElF,OAAO,EACL,uBAAuB,EACvB,gBAAgB,GACjB,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,sBAAsB,EAAE,MAAM,8BAA8B,CAAC;AAKtE,MAAM,CAAC,MAAM,iBAAiB,GAAG,CAAC,CAAC;AACnC;;;;GAIG;AACH,MAAM,CAAC,MAAM,uBAAuB,GAAG,EAAE,CAAC;AAC1C,MAAM,kBAAkB,GAAG,KAAK,CAAC;AACjC,MAAM,aAAa,GAAG,GAAG,CAAC;AAuD1B,MAAM,UAAU,YAAY,CAC1B,IAAuB,EACvB,WAA2B,EAC3B,QAAQ,GAAqB,uBAAuB;IAEpD,IAAI,QAAQ,GAAa,QAAQ,CAAC;IAClC,IAAI,cAAkC,CAAC;IACvC,IAAI,MAA0B,CAAC;IAC/B,IAAI,aAAa,GAAG,sBAAsB,CAAC;IAC3C,IAAI,UAAU,GAAG,IAAI,CAAC;IACtB,IAAI,QAAQ,GAAG,iBAAiB,CAAC;IACjC,IAAI,MAAM,GAAG,IAAI,CAAC;IAClB,IAAI,KAAK,GAAG,KAAK,CAAC;IAClB,IAAI,IAAI,GAAG,KAAK,CAAC;IACjB,IAAI,OAAO,GAAG,IAAI,CAAC;IACnB,IAAI,MAA0B,CAAC;IAC/B,IAAI,eAAe,GAAG,KAAK,CAAC;IAC5B,IAAI,YAAY,GAAG,KAAK,CAAC;IACzB,IAAI,WAAW,GAAG,KAAK,CAAC;IACxB,IAAI,UAAU,GAAG,KAAK,CAAC;IACvB,IAAI,UAAU,GAAG,IAAI,CAAC;IACtB,IAAI,KAAK,GAAG,KAAK,CAAC;IAClB,IAAI,OAA2B,CAAC;IAChC,IAAI,WAAW,GAAG,KAAK,CAAC;IACxB,IAAI,cAAc,GAAG,KAAK,CAAC;IAC3B,IAAI,WAAW,GAAG,KAAK,CAAC;IACxB,IAAI,gBAAgB,GAAG,KAAK,CAAC;IAC7B,IAAI,IAAI,GAAG,KAAK,CAAC;IACjB,IAAI,OAAO,GAAG,KAAK,CAAC;IACpB,MAAM,WAAW,GAAa,EAAE,CAAC;IAEjC,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC,MAAM,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC;QACpD,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;QAC7B,IAAI,QAAQ,KAAK,QAAQ,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YAC/C,IAAI,GAAG,IAAI,CAAC;QACd,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YACzD,OAAO,GAAG,IAAI,CAAC;QACjB,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,EAAE,CAAC;YACrC,MAAM,KAAK,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,YAAY,CAAC,CAAC;YACzD,QAAQ,GAAG,aAAa,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC;YAC1C,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAClC,cAAc,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,SAAS,CAAC,CAAC;YACzD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,EAAE,CAAC;YACpC,MAAM,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,WAAW,CAAC,CAAC;YACnD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,aAAa,EAAE,CAAC;YACtC,QAAQ,GAAG,aAAa,CAAC,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,aAAa,CAAC,CAAC,CAAC;YACtE,gBAAgB,GAAG,IAAI,CAAC;YACxB,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,aAAa,EAAE,CAAC;YACtC,MAAM,GAAG,KAAK,CAAC;QACjB,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YACvD,KAAK,GAAG,IAAI,CAAC;QACf,CAAC;aAAM,IAAI,QAAQ,KAAK,QAAQ,EAAE,CAAC;YACjC,IAAI,GAAG,IAAI,CAAC;QACd,CAAC;aAAM,IAAI,QAAQ,KAAK,cAAc,EAAE,CAAC;YACvC,OAAO,GAAG,KAAK,CAAC;QAClB,CAAC;aAAM,IAAI,QAAQ,KAAK,UAAU,EAAE,CAAC;YACnC,MAAM,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,UAAU,CAAC,CAAC;YAClD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YAC1D,eAAe,GAAG,IAAI,CAAC;QACzB,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,EAAE,CAAC;YACrC,YAAY,GAAG,IAAI,CAAC;QACtB,CAAC;aAAM,IAAI,QAAQ,KAAK,gBAAgB,EAAE,CAAC;YACzC,WAAW,GAAG,IAAI,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,eAAe,EAAE,CAAC;YACxC,UAAU,GAAG,IAAI,CAAC;QACpB,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAClC,KAAK,GAAG,IAAI,CAAC;QACf,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,EAAE,CAAC;YACpC,OAAO,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,WAAW,CAAC,CAAC;YACpD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,EAAE,CAAC;YACpC,WAAW,GAAG,IAAI,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,iBAAiB,EAAE,CAAC;YAC1C,UAAU,GAAG,KAAK,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,cAAc,EAAE,CAAC;YACvC,cAAc,GAAG,IAAI,CAAC;QACxB,CAAC;aAAM,IAAI,QAAQ,KAAK,gBAAgB,EAAE,CAAC;YACzC,WAAW,GAAG,IAAI,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,kBAAkB,EAAE,CAAC;YAC3C,aAAa,GAAG,kBAAkB,CAChC,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,kBAAkB,CAAC,CACjD,CAAC;YACF,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,iBAAiB,EAAE,CAAC;YAC1C,UAAU,GAAG,KAAK,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,EAAE,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;YACrC,MAAM,IAAI,SAAS,CAAC,mBAAmB,QAAQ,EAAE,CAAC,CAAC;QACrD,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAClC,WAAW,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,MAAM,UAAU,GAAG,QAAQ,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;IAC1C,MAAM,KAAK,GACT,cAAc;QACd,qBAAqB,CAAC,WAAW,EAAE,UAAU,CAAC,YAAY,CAAC;QAC3D,UAAU,CAAC,YAAY,CAAC;IAC1B,MAAM,OAAO,GACX,qBAAqB,CAAC,WAAW,EAAE,UAAU,CAAC,cAAc,CAAC;QAC7D,UAAU,CAAC,cAAc,CAAC;IAE5B,OAAO;QACL,GAAG,CAAC,WAAW,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,WAAW,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACtE,QAAQ;QACR,GAAG,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC;QACzC,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC;QAC7C,QAAQ,EACN,CAAC,WAAW,IAAI,UAAU,CAAC,IAAI,CAAC,gBAAgB;YAC9C,CAAC,CAAC,uBAAuB;YACzB,CAAC,CAAC,QAAQ;QACd,MAAM;QACN,KAAK;QACL,IAAI;QACJ,OAAO;QACP,GAAG,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC;QAC3C,eAAe;QACf,YAAY;QACZ,WAAW;QACX,UAAU;QACV,UAAU;QACV,KAAK;QACL,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC;QAC7C,WAAW;QACX,cAAc;QACd,WAAW;QACX,GAAG,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC;QAC3C,aAAa;QACb,UAAU;QACV,IAAI;QACJ,OAAO;KACR,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,QAAQ,CACtB,QAAQ,GAAqB,uBAAuB;IAEpD,OAAO,YAAY,OAAO;;;;;;;qBAOP,QAAQ,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC;;;uDAGS,iBAAiB,KAAK,uBAAuB;;;;;;;;;;;;;;;;+DAgBrC,sBAAsB;;;;;;;;;;EAUnF,iBAAiB,CAAC,QAAQ,CAAC;;;;;;;;;;;;;;;;;CAiB5B,CAAC;AACF,CAAC;AAED,SAAS,iBAAiB,CAAC,QAA0B;IACnD,MAAM,GAAG,GAAG,QAAQ,CAAC,GAAG,EAAE,CAAC;IAC3B,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,GAAG,CAAC,CAAC;IAC1D,OAAO,GAAG;SACP,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE;QACV,MAAM,UAAU,GAAG,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACpC,MAAM,KAAK,GACT,UAAU,CAAC,YAAY,KAAK,SAAS;YACnC,CAAC,CAAC,gBAAgB;YAClB,CAAC,CAAC,WAAW,UAAU,CAAC,YAAY,EAAE,CAAC;QAC3C,OAAO,KAAK,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,UAAU,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,KAAK,GAAG,CAAC;IAC5E,CAAC,CAAC;SACD,IAAI,CAAC,IAAI,CAAC,CAAC;AAChB,CAAC;AAED,SAAS,eAAe,CACtB,IAAuB,EACvB,KAAa,EACb,MAAc;IAEd,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC;IAC9B,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;QACjD,MAAM,IAAI,SAAS,CAAC,GAAG,MAAM,oBAAoB,CAAC,CAAC;IACrD,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,kBAAkB,CAAC,KAAa;IACvC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC7B,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,kBAAkB,EAAE,CAAC;QACjE,MAAM,IAAI,SAAS,CACjB,mDAAmD,kBAAkB,GAAG,CACzE,CAAC;IACJ,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,aAAa,CAAC,KAAa;IAClC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC7B,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,CAAC,IAAI,MAAM,GAAG,aAAa,EAAE,CAAC;QAC1E,MAAM,IAAI,SAAS,CACjB,4CAA4C,aAAa,GAAG,CAC7D,CAAC;IACJ,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,aAAa,CAAC,KAAa,EAAE,QAA0B;IAC9D,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;QACzB,MAAM,IAAI,SAAS,CACjB,yBAAyB,KAAK,qBAAqB,QAAQ,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAChF,CAAC;IACJ,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,qBAAqB,CAC5B,WAA2B,EAC3B,IAAuB;IAEvB,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,KAAK,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC;QAC/B,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC5C,OAAO,KAAK,CAAC;QACf,CAAC;IACH,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC"}
|
|
1
|
+
{"version":3,"file":"cli-options.js","sourceRoot":"","sources":["../src/cli-options.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,OAAO,GAAG,aAAa,CAAC,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC;AAC/C,MAAM,eAAe,GAAG,OAAO,CAAC,iBAAiB,CAAmC,CAAC;AAErF,MAAM,CAAC,MAAM,OAAO,GAClB,OAAO,eAAe,CAAC,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,eAAe,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;AAElF,OAAO,EACL,uBAAuB,EACvB,gBAAgB,GACjB,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,sBAAsB,EAAE,MAAM,8BAA8B,CAAC;AAKtE,MAAM,CAAC,MAAM,iBAAiB,GAAG,CAAC,CAAC;AACnC;;;;GAIG;AACH,MAAM,CAAC,MAAM,uBAAuB,GAAG,EAAE,CAAC;AAC1C,MAAM,kBAAkB,GAAG,KAAK,CAAC;AACjC,MAAM,aAAa,GAAG,GAAG,CAAC;AAuD1B,MAAM,UAAU,YAAY,CAC1B,IAAuB,EACvB,WAA2B,EAC3B,QAAQ,GAAqB,uBAAuB;IAEpD,IAAI,QAAQ,GAAa,QAAQ,CAAC;IAClC,IAAI,cAAkC,CAAC;IACvC,IAAI,MAA0B,CAAC;IAC/B,IAAI,aAAa,GAAG,sBAAsB,CAAC;IAC3C,IAAI,UAAU,GAAG,IAAI,CAAC;IACtB,IAAI,QAAQ,GAAG,iBAAiB,CAAC;IACjC,IAAI,MAAM,GAAG,IAAI,CAAC;IAClB,IAAI,KAAK,GAAG,KAAK,CAAC;IAClB,IAAI,IAAI,GAAG,KAAK,CAAC;IACjB,IAAI,OAAO,GAAG,IAAI,CAAC;IACnB,IAAI,MAA0B,CAAC;IAC/B,IAAI,eAAe,GAAG,KAAK,CAAC;IAC5B,IAAI,YAAY,GAAG,KAAK,CAAC;IACzB,IAAI,WAAW,GAAG,KAAK,CAAC;IACxB,IAAI,UAAU,GAAG,KAAK,CAAC;IACvB,IAAI,UAAU,GAAG,IAAI,CAAC;IACtB,IAAI,KAAK,GAAG,KAAK,CAAC;IAClB,IAAI,OAA2B,CAAC;IAChC,IAAI,WAAW,GAAG,KAAK,CAAC;IACxB,IAAI,cAAc,GAAG,KAAK,CAAC;IAC3B,IAAI,WAAW,GAAG,KAAK,CAAC;IACxB,IAAI,gBAAgB,GAAG,KAAK,CAAC;IAC7B,IAAI,IAAI,GAAG,KAAK,CAAC;IACjB,IAAI,OAAO,GAAG,KAAK,CAAC;IACpB,MAAM,WAAW,GAAa,EAAE,CAAC;IAEjC,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC,MAAM,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC;QACpD,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;QAC7B,IAAI,QAAQ,KAAK,QAAQ,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YAC/C,IAAI,GAAG,IAAI,CAAC;QACd,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YACzD,OAAO,GAAG,IAAI,CAAC;QACjB,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,EAAE,CAAC;YACrC,MAAM,KAAK,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,YAAY,CAAC,CAAC;YACzD,QAAQ,GAAG,aAAa,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC;YAC1C,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAClC,cAAc,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,SAAS,CAAC,CAAC;YACzD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,EAAE,CAAC;YACpC,MAAM,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,WAAW,CAAC,CAAC;YACnD,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBACxB,kEAAkE;gBAClE,qEAAqE;gBACrE,iEAAiE;gBACjE,MAAM,IAAI,SAAS,CAAC,6BAA6B,CAAC,CAAC;YACrD,CAAC;YACD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,aAAa,EAAE,CAAC;YACtC,QAAQ,GAAG,aAAa,CAAC,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,aAAa,CAAC,CAAC,CAAC;YACtE,gBAAgB,GAAG,IAAI,CAAC;YACxB,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,aAAa,EAAE,CAAC;YACtC,MAAM,GAAG,KAAK,CAAC;QACjB,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YACvD,KAAK,GAAG,IAAI,CAAC;QACf,CAAC;aAAM,IAAI,QAAQ,KAAK,QAAQ,EAAE,CAAC;YACjC,IAAI,GAAG,IAAI,CAAC;QACd,CAAC;aAAM,IAAI,QAAQ,KAAK,cAAc,EAAE,CAAC;YACvC,OAAO,GAAG,KAAK,CAAC;QAClB,CAAC;aAAM,IAAI,QAAQ,KAAK,UAAU,EAAE,CAAC;YACnC,MAAM,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,UAAU,CAAC,CAAC;YAClD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YAC1D,eAAe,GAAG,IAAI,CAAC;QACzB,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,EAAE,CAAC;YACrC,YAAY,GAAG,IAAI,CAAC;QACtB,CAAC;aAAM,IAAI,QAAQ,KAAK,gBAAgB,EAAE,CAAC;YACzC,WAAW,GAAG,IAAI,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,eAAe,EAAE,CAAC;YACxC,UAAU,GAAG,IAAI,CAAC;QACpB,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAClC,KAAK,GAAG,IAAI,CAAC;QACf,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,EAAE,CAAC;YACpC,OAAO,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,WAAW,CAAC,CAAC;YACpD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,WAAW,EAAE,CAAC;YACpC,WAAW,GAAG,IAAI,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,iBAAiB,EAAE,CAAC;YAC1C,UAAU,GAAG,KAAK,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,cAAc,EAAE,CAAC;YACvC,cAAc,GAAG,IAAI,CAAC;QACxB,CAAC;aAAM,IAAI,QAAQ,KAAK,gBAAgB,EAAE,CAAC;YACzC,WAAW,GAAG,IAAI,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,KAAK,kBAAkB,EAAE,CAAC;YAC3C,aAAa,GAAG,kBAAkB,CAChC,eAAe,CAAC,IAAI,EAAE,KAAK,EAAE,kBAAkB,CAAC,CACjD,CAAC;YACF,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,iBAAiB,EAAE,CAAC;YAC1C,UAAU,GAAG,KAAK,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,EAAE,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;YACrC,MAAM,IAAI,SAAS,CAAC,mBAAmB,QAAQ,EAAE,CAAC,CAAC;QACrD,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAClC,WAAW,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,MAAM,UAAU,GAAG,QAAQ,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;IAC1C,MAAM,KAAK,GACT,cAAc;QACd,qBAAqB,CAAC,WAAW,EAAE,UAAU,CAAC,YAAY,CAAC;QAC3D,UAAU,CAAC,YAAY,CAAC;IAC1B,MAAM,OAAO,GACX,qBAAqB,CAAC,WAAW,EAAE,UAAU,CAAC,cAAc,CAAC;QAC7D,UAAU,CAAC,cAAc,CAAC;IAE5B,OAAO;QACL,GAAG,CAAC,WAAW,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,WAAW,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACtE,QAAQ;QACR,GAAG,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC;QACzC,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC;QAC7C,QAAQ,EACN,CAAC,WAAW,IAAI,UAAU,CAAC,IAAI,CAAC,gBAAgB;YAC9C,CAAC,CAAC,uBAAuB;YACzB,CAAC,CAAC,QAAQ;QACd,MAAM;QACN,KAAK;QACL,IAAI;QACJ,OAAO;QACP,GAAG,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC;QAC3C,eAAe;QACf,YAAY;QACZ,WAAW;QACX,UAAU;QACV,UAAU;QACV,KAAK;QACL,GAAG,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC;QAC7C,WAAW;QACX,cAAc;QACd,WAAW;QACX,GAAG,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC;QAC3C,aAAa;QACb,UAAU;QACV,IAAI;QACJ,OAAO;KACR,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,QAAQ,CACtB,QAAQ,GAAqB,uBAAuB;IAEpD,OAAO,YAAY,OAAO;;;;;;;qBAOP,QAAQ,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC;;;uDAGS,iBAAiB,KAAK,uBAAuB;;;;;;;;;;;;;;;;+DAgBrC,sBAAsB;;;;;;;;;;EAUnF,iBAAiB,CAAC,QAAQ,CAAC;;;;;;;;;;;;;;;;;CAiB5B,CAAC;AACF,CAAC;AAED,SAAS,iBAAiB,CAAC,QAA0B;IACnD,MAAM,GAAG,GAAG,QAAQ,CAAC,GAAG,EAAE,CAAC;IAC3B,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,GAAG,CAAC,CAAC;IAC1D,OAAO,GAAG;SACP,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE;QACV,MAAM,UAAU,GAAG,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACpC,MAAM,KAAK,GACT,UAAU,CAAC,YAAY,KAAK,SAAS;YACnC,CAAC,CAAC,gBAAgB;YAClB,CAAC,CAAC,WAAW,UAAU,CAAC,YAAY,EAAE,CAAC;QAC3C,OAAO,KAAK,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,UAAU,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,KAAK,GAAG,CAAC;IAC5E,CAAC,CAAC;SACD,IAAI,CAAC,IAAI,CAAC,CAAC;AAChB,CAAC;AAED,SAAS,eAAe,CACtB,IAAuB,EACvB,KAAa,EACb,MAAc;IAEd,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC;IAC9B,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;QACjD,MAAM,IAAI,SAAS,CAAC,GAAG,MAAM,oBAAoB,CAAC,CAAC;IACrD,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,kBAAkB,CAAC,KAAa;IACvC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC7B,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,kBAAkB,EAAE,CAAC;QACjE,MAAM,IAAI,SAAS,CACjB,mDAAmD,kBAAkB,GAAG,CACzE,CAAC;IACJ,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,aAAa,CAAC,KAAa;IAClC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC7B,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,CAAC,IAAI,MAAM,GAAG,aAAa,EAAE,CAAC;QAC1E,MAAM,IAAI,SAAS,CACjB,4CAA4C,aAAa,GAAG,CAC7D,CAAC;IACJ,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,aAAa,CAAC,KAAa,EAAE,QAA0B;IAC9D,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;QACzB,MAAM,IAAI,SAAS,CACjB,yBAAyB,KAAK,qBAAqB,QAAQ,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAChF,CAAC;IACJ,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,qBAAqB,CAC5B,WAA2B,EAC3B,IAAuB;IAEvB,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,KAAK,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC;QAC/B,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC5C,OAAO,KAAK,CAAC;QACf,CAAC;IACH,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC"}
|
package/dist/evals/cli.js
CHANGED
|
@@ -21,6 +21,8 @@ Options:
|
|
|
21
21
|
--json PATH Also write a JSON report
|
|
22
22
|
--provider NAME Provider to evaluate
|
|
23
23
|
--model MODEL Model to evaluate
|
|
24
|
+
--api-key KEY Credential to evaluate with (else the environment or
|
|
25
|
+
the stored auth.json, as the main CLI resolves it)
|
|
24
26
|
--allow-writes Grant tasks that need to change files
|
|
25
27
|
--allow-shell Grant tasks that need a shell
|
|
26
28
|
-h, --help Show this help
|
package/dist/evals/cli.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../../src/evals/cli.ts"],"names":[],"mappings":";AAEA,OAAO,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAC;AAC7C,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,OAAO,MAAM,cAAc,CAAC;AACnC,OAAO,EAAE,YAAY,EAAE,OAAO,EAAE,MAAM,mBAAmB,CAAC;AAC1D,OAAO,EAAE,sBAAsB,EAAE,MAAM,wBAAwB,CAAC;AAChE,OAAO,EACL,mBAAmB,EACnB,aAAa,EACb,mBAAmB,GACpB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,cAAc,EAAE,WAAW,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AACxE,OAAO,EAAE,sBAAsB,EAAE,OAAO,EAAmB,MAAM,aAAa,CAAC;AAC/E,OAAO,EAAE,SAAS,EAAE,SAAS,EAA8B,MAAM,WAAW,CAAC;AAC7E,OAAO,EAAE,aAAa,EAAoB,MAAM,iBAAiB,CAAC;AAElE,MAAM,IAAI,GAAG,kBAAkB,OAAO
|
|
1
|
+
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../../src/evals/cli.ts"],"names":[],"mappings":";AAEA,OAAO,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAC;AAC7C,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,OAAO,MAAM,cAAc,CAAC;AACnC,OAAO,EAAE,YAAY,EAAE,OAAO,EAAE,MAAM,mBAAmB,CAAC;AAC1D,OAAO,EAAE,sBAAsB,EAAE,MAAM,wBAAwB,CAAC;AAChE,OAAO,EACL,mBAAmB,EACnB,aAAa,EACb,mBAAmB,GACpB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,cAAc,EAAE,WAAW,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AACxE,OAAO,EAAE,sBAAsB,EAAE,OAAO,EAAmB,MAAM,aAAa,CAAC;AAC/E,OAAO,EAAE,SAAS,EAAE,SAAS,EAA8B,MAAM,WAAW,CAAC;AAC7E,OAAO,EAAE,aAAa,EAAoB,MAAM,iBAAiB,CAAC;AAElE,MAAM,IAAI,GAAG,kBAAkB,OAAO;;;;;;;;;;;;;;;;;;;;;CAqBrC,CAAC;AAGF,SAAS,iBAAiB,CAAC,YAAmC;IAC5D,MAAM,QAAQ,GAAG,CAAC,aAAa,CAAC,CAAC;IACjC,IAAI,YAAY,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE,CAAC;QACpC,QAAQ,CAAC,IAAI,CAAC,mBAAmB,CAAC,CAAC;IACrC,CAAC;IACD,IAAI,YAAY,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;QACnC,QAAQ,CAAC,IAAI,CAAC,mBAAmB,CAAC,CAAC;IACrC,CAAC;IACD,OAAO,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC7B,CAAC;AAED,SAAS,WAAW,CAClB,GAAoB,EACpB,MAAyB;IAEzB,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,GAAG,CAAC;IACb,CAAC;IACD,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;IACpD,KAAK,MAAM,IAAI,IAAI,MAAM,EAAE,CAAC;QAC1B,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;YACrB,MAAM,IAAI,SAAS,CACjB,iBAAiB,IAAI,iBAAiB,CAAC,GAAG,KAAK,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAC/D,CAAC;QACJ,CAAC;IACH,CAAC;IACD,OAAO,GAAG,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;AAC1D,CAAC;AAED,KAAK,UAAU,IAAI,CAAC,IAAuB;IACzC,IAAI,OAAoB,CAAC;IACzB,IAAI,CAAC;QACH,OAAO,GAAG,aAAa,CAAC,IAAI,CAAC,CAAC;IAChC,CAAC;IAAC,OAAO,KAAc,EAAE,CAAC;QACxB,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAC9D,CAAC;QACF,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;QACjB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QAC3B,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,KAAsB,CAAC;IAC3B,IAAI,CAAC;QACH,KAAK,GAAG,WAAW,CACjB,MAAM,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,EAChD,OAAO,CAAC,KAAK,CACd,CAAC;IACJ,CAAC;IAAC,OAAO,KAAc,EAAE,CAAC;QACxB,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAC9D,CAAC;QACF,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACvB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,qBAAqB,OAAO,CAAC,SAAS,KAAK,CAAC,CAAC;QAClE,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,UAAU,GAAG,YAAY,CAAC,CAAC,GAAG,OAAO,CAAC,YAAY,CAAC,EAAE,OAAO,CAAC,GAAG,CAAC,CAAC;IACxE,MAAM,GAAG,GAAG,MAAM,sBAAsB,CAAC,UAAU,CAAC,CAAC;IACrD,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;QAC5B,0EAA0E;QAC1E,iDAAiD;QACjD,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,GAAG,IAAI,CAAC,CAAC;QACjC,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,SAAS,GAAG,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC;IAC3C,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,cAAc,UAAU,CAAC,QAAQ,IAAI,UAAU,CAAC,KAAK,IAAI,GAAG,OAAO,KAAK,CAAC,MAAM,eAAe,CAC/F,CAAC;IAEF,MAAM,OAAO,GAAiB,EAAE,CAAC;IACjC,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,IAAI,CAAC;YACH,OAAO,CAAC,IAAI,CACV,MAAM,OAAO,CAAC,IAAI,EAAE;gBAClB,YAAY,EAAE,GAAG,EAAE,CAAC,GAAG;gBACvB,YAAY,EAAE,iBAAiB,CAAC,IAAI,CAAC,YAAY,CAAC;gBAClD,YAAY,EAAE,OAAO,CAAC,YAAY;gBAClC,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC;gBACzE,SAAS,EAAE,CAAC,MAAM,EAAE,EAAE;oBACpB,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,KAAK,IAAI,CAAC,IAAI,IAAI,MAAM,CAAC,OAAO,KAAK,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,IAAI,CACzE,CAAC;gBACJ,CAAC;aACF,CAAC,CACH,CAAC;QACJ,CAAC;QAAC,OAAO,KAAc,EAAE,CAAC;YACxB,IAAI,KAAK,YAAY,sBAAsB,EAAE,CAAC;gBAC5C,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,KAAK,CAAC,OAAO,IAAI,CAAC,CAAC;gBACpD,SAAS;YACX,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,WAAW,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC;IAClD,MAAM,QAAQ,GAAG,cAAc,CAAC,OAAO,CAAC,CAAC;IACzC,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,QAAQ,EAAE,CAAC,CAAC;IACxC,CAAC;IAED,IAAI,OAAO,CAAC,QAAQ,KAAK,SAAS,EAAE,CAAC;QACnC,MAAM,MAAM,GAAG,YAAY,CAAC,OAAO,EAAE;YACnC,QAAQ,EAAE,UAAU,CAAC,QAAQ;YAC7B,KAAK,EAAE,UAAU,CAAC,KAAK,IAAI,SAAS;YACpC,OAAO,EAAE,OAAO;YAChB,SAAS;SACV,CAAC,CAAC;QACH,MAAM,SAAS,CAAC,OAAO,CAAC,QAAQ,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;QAC1E,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,WAAW,OAAO,CAAC,QAAQ,IAAI,CAAC,CAAC;IACxD,CAAC;IAED,yEAAyE;IACzE,OAAO,CAAC,CAAC;AACX,CAAC;AAED,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;KACxB,IAAI,CAAC,CAAC,QAAQ,EAAE,EAAE;IACjB,OAAO,CAAC,QAAQ,GAAG,QAAQ,CAAC;AAC9B,CAAC,CAAC;KACD,KAAK,CAAC,CAAC,KAAc,EAAE,EAAE;IACxB,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,eAAe,CAAC;IACzE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,uBAAuB,OAAO,IAAI,CAAC,CAAC;IACzD,OAAO,CAAC,QAAQ,GAAG,CAAC,CAAC;AACvB,CAAC,CAAC,CAAC"}
|
package/dist/evals/graders.d.ts
CHANGED
|
@@ -3,6 +3,8 @@ import type { AgentEvent } from "../events.js";
|
|
|
3
3
|
export interface AttemptFacts {
|
|
4
4
|
/** The workspace as the attempt left it. */
|
|
5
5
|
readonly workspace: string;
|
|
6
|
+
/** The pristine fixture, for graders that ask what the attempt changed. */
|
|
7
|
+
readonly fixtureDirectory: string;
|
|
6
8
|
readonly finalAnswer: string;
|
|
7
9
|
readonly turnCount: number;
|
|
8
10
|
readonly status: "completed" | "max_turns" | "aborted" | "error";
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"graders.d.ts","sourceRoot":"","sources":["../../src/evals/graders.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACvD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAE/C,MAAM,WAAW,YAAY;IAC3B,4CAA4C;IAC5C,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,MAAM,EAAE,WAAW,GAAG,WAAW,GAAG,SAAS,GAAG,OAAO,CAAC;IACjE,QAAQ,CAAC,MAAM,EAAE,SAAS,UAAU,EAAE,CAAC;CACxC;AAED,MAAM,WAAW,YAAY;IAC3B,QAAQ,CAAC,MAAM,EAAE,OAAO,CAAC;IACzB,sEAAsE;IACtE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,MAAM,MAAM,GAAG,CAAC,KAAK,EAAE,YAAY,KAAK,OAAO,CAAC,YAAY,CAAC,CAAC;
|
|
1
|
+
{"version":3,"file":"graders.d.ts","sourceRoot":"","sources":["../../src/evals/graders.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACvD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAE/C,MAAM,WAAW,YAAY;IAC3B,4CAA4C;IAC5C,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,2EAA2E;IAC3E,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,MAAM,EAAE,WAAW,GAAG,WAAW,GAAG,SAAS,GAAG,OAAO,CAAC;IACjE,QAAQ,CAAC,MAAM,EAAE,SAAS,UAAU,EAAE,CAAC;CACxC;AAED,MAAM,WAAW,YAAY;IAC3B,QAAQ,CAAC,MAAM,EAAE,OAAO,CAAC;IACzB,sEAAsE;IACtE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,MAAM,MAAM,GAAG,CAAC,KAAK,EAAE,YAAY,KAAK,OAAO,CAAC,YAAY,CAAC,CAAC;AAgRpE,eAAO,MAAM,YAAY,EAAE,SAAS,MAAM,EAA2B,CAAC;AAEtE,wBAAgB,YAAY,CAAC,IAAI,EAAE,UAAU,GAAG,MAAM,CAQrD;AAED,wBAAgB,cAAc,CAAC,IAAI,EAAE,UAAU,GAAG,MAAM,CAMvD;AAED,YAAY,EAAE,SAAS,EAAE,CAAC"}
|
package/dist/evals/graders.js
CHANGED
|
@@ -25,12 +25,29 @@ function toolCalls(events) {
|
|
|
25
25
|
.filter((event) => event.type === "tool_execution_end")
|
|
26
26
|
.map((event) => event.toolName);
|
|
27
27
|
}
|
|
28
|
+
function failedToolCalls(events) {
|
|
29
|
+
return events
|
|
30
|
+
.filter((event) => event.type === "tool_execution_end" &&
|
|
31
|
+
event.isError === true)
|
|
32
|
+
.map((event) => event.toolName);
|
|
33
|
+
}
|
|
28
34
|
function successfulToolCalls(events) {
|
|
29
35
|
return events
|
|
30
36
|
.filter((event) => event.type === "tool_execution_end" &&
|
|
31
37
|
event.isError === false)
|
|
32
38
|
.map((event) => event.toolName);
|
|
33
39
|
}
|
|
40
|
+
/** 1-based line where two texts first differ, for an actionable failure. */
|
|
41
|
+
function firstDifferingLine(before, after) {
|
|
42
|
+
const left = before.split("\n");
|
|
43
|
+
const right = after.split("\n");
|
|
44
|
+
for (let index = 0; index < Math.max(left.length, right.length); index += 1) {
|
|
45
|
+
if (left[index] !== right[index]) {
|
|
46
|
+
return index + 1;
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
return 1;
|
|
50
|
+
}
|
|
34
51
|
const factories = {
|
|
35
52
|
"file-contains": (spec) => {
|
|
36
53
|
const file = requireString(spec, "path");
|
|
@@ -125,6 +142,59 @@ const factories = {
|
|
|
125
142
|
: fail(`${name} never returned a successful result`);
|
|
126
143
|
};
|
|
127
144
|
},
|
|
145
|
+
"tool-never-failed": (spec) => {
|
|
146
|
+
// Sound only where a failure means the model was wrong. `bash` sets
|
|
147
|
+
// isError on any non-zero exit, so on a task that runs a failing test —
|
|
148
|
+
// fix-failing-test, wrong-test — the correct behaviour is a failed call
|
|
149
|
+
// and this grader would punish it.
|
|
150
|
+
const name = requireString(spec, "name");
|
|
151
|
+
return async ({ events }) => {
|
|
152
|
+
// A guessed path shows up here: reading a file that is not there is the
|
|
153
|
+
// failure, and the set of tool names a report keeps cannot see it.
|
|
154
|
+
const failures = failedToolCalls(events).filter((tool) => tool === name);
|
|
155
|
+
return failures.length === 0
|
|
156
|
+
? pass
|
|
157
|
+
: fail(`${name} failed ${failures.length} time(s); this task expects it to be called only on paths the model established exist`);
|
|
158
|
+
};
|
|
159
|
+
},
|
|
160
|
+
"max-tool-calls": (spec) => {
|
|
161
|
+
const name = requireString(spec, "name");
|
|
162
|
+
const limit = spec.count;
|
|
163
|
+
if (!Number.isSafeInteger(limit) || limit < 1) {
|
|
164
|
+
throw new Error('max-tool-calls: "count" must be a positive integer.');
|
|
165
|
+
}
|
|
166
|
+
return async ({ events }) => {
|
|
167
|
+
// Turns cannot express this: a model may issue any number of calls in
|
|
168
|
+
// one turn, so "did it read the whole project" is a call count, not a
|
|
169
|
+
// turn count.
|
|
170
|
+
// Failed calls count. A budget that forgave them would let a model
|
|
171
|
+
// spend freely on guessed paths, which is the opposite of the intent.
|
|
172
|
+
const used = toolCalls(events).filter((tool) => tool === name).length;
|
|
173
|
+
return used <= limit
|
|
174
|
+
? pass
|
|
175
|
+
: fail(`called ${name} ${used} times, over the budget of ${limit}`);
|
|
176
|
+
};
|
|
177
|
+
},
|
|
178
|
+
"file-unchanged": (spec) => {
|
|
179
|
+
const file = requireString(spec, "path");
|
|
180
|
+
return async ({ workspace, fixtureDirectory }) => {
|
|
181
|
+
// Compared against the fixture rather than against text copied into
|
|
182
|
+
// task.json, so there is no second copy to keep in sync.
|
|
183
|
+
const before = await readWorkspaceFile(fixtureDirectory, file);
|
|
184
|
+
if (before === undefined) {
|
|
185
|
+
throw new Error(`file-unchanged: ${file} is not in the fixture, so there is nothing to compare against.`);
|
|
186
|
+
}
|
|
187
|
+
const after = await readWorkspaceFile(workspace, file);
|
|
188
|
+
if (after === undefined) {
|
|
189
|
+
return fail(`${file} was deleted; this task expects it untouched`);
|
|
190
|
+
}
|
|
191
|
+
if (after === before) {
|
|
192
|
+
return pass;
|
|
193
|
+
}
|
|
194
|
+
const line = firstDifferingLine(before, after);
|
|
195
|
+
return fail(`${file} was modified at line ${line}; this task expects it untouched`);
|
|
196
|
+
};
|
|
197
|
+
},
|
|
128
198
|
"answer-matches": (spec) => {
|
|
129
199
|
const pattern = requireString(spec, "pattern");
|
|
130
200
|
const flags = typeof spec.flags === "string" ? spec.flags : "i";
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"graders.js","sourceRoot":"","sources":["../../src/evals/graders.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,kBAAkB,CAAC;AAClD,OAAO,IAAI,MAAM,WAAW,CAAC;
|
|
1
|
+
{"version":3,"file":"graders.js","sourceRoot":"","sources":["../../src/evals/graders.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,kBAAkB,CAAC;AAClD,OAAO,IAAI,MAAM,WAAW,CAAC;AAuB7B,MAAM,IAAI,GAAiB,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,CAAC;AAExD,SAAS,IAAI,CAAC,MAAc;IAC1B,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC;AACnC,CAAC;AAED,SAAS,aAAa,CAAC,IAAgB,EAAE,KAAa;IACpD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;IAC1B,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACpD,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,CAAC,IAAI,MAAM,KAAK,+BAA+B,CAAC,CAAC;IAC1E,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,KAAK,UAAU,iBAAiB,CAC9B,SAAiB,EACjB,YAAoB;IAEpB,IAAI,CAAC;QACH,OAAO,MAAM,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,YAAY,CAAC,EAAE,MAAM,CAAC,CAAC;IACpE,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,SAAS,CAAC;IACnB,CAAC;AACH,CAAC;AAED,yEAAyE;AACzE,SAAS,SAAS,CAAC,MAA6B;IAC9C,OAAO,MAAM;SACV,MAAM,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,IAAI,KAAK,oBAAoB,CAAC;SACtD,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAE,KAA8B,CAAC,QAAQ,CAAC,CAAC;AAC9D,CAAC;AAED,SAAS,eAAe,CAAC,MAA6B;IACpD,OAAO,MAAM;SACV,MAAM,CACL,CAAC,KAAK,EAAE,EAAE,CACR,KAAK,CAAC,IAAI,KAAK,oBAAoB;QAClC,KAA8B,CAAC,OAAO,KAAK,IAAI,CACnD;SACA,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAE,KAA8B,CAAC,QAAQ,CAAC,CAAC;AAC9D,CAAC;AAED,SAAS,mBAAmB,CAAC,MAA6B;IACxD,OAAO,MAAM;SACV,MAAM,CACL,CAAC,KAAK,EAAE,EAAE,CACR,KAAK,CAAC,IAAI,KAAK,oBAAoB;QAClC,KAA8B,CAAC,OAAO,KAAK,KAAK,CACpD;SACA,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAE,KAA8B,CAAC,QAAQ,CAAC,CAAC;AAC9D,CAAC;AAED,4EAA4E;AAC5E,SAAS,kBAAkB,CAAC,MAAc,EAAE,KAAa;IACvD,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAChC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAChC,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC;QAC5E,IAAI,IAAI,CAAC,KAAK,CAAC,KAAK,KAAK,CAAC,KAAK,CAAC,EAAE,CAAC;YACjC,OAAO,KAAK,GAAG,CAAC,CAAC;QACnB,CAAC;IACH,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAID,MAAM,SAAS,GAAkC;IAC/C,eAAe,EAAE,CAAC,IAAI,EAAE,EAAE;QACxB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,SAAS,EAAE,EAAE,EAAE;YAC7B,MAAM,QAAQ,GAAG,MAAM,iBAAiB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YAC1D,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;gBAC3B,OAAO,IAAI,CAAC,GAAG,IAAI,iBAAiB,CAAC,CAAC;YACxC,CAAC;YACD,OAAO,QAAQ,CAAC,QAAQ,CAAC,IAAI,CAAC;gBAC5B,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CAAC,GAAG,IAAI,qBAAqB,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAC/D,CAAC,CAAC;IACJ,CAAC;IAED,eAAe,EAAE,CAAC,IAAI,EAAE,EAAE;QACxB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,SAAS,EAAE,EAAE,EAAE;YAC7B,MAAM,QAAQ,GAAG,MAAM,iBAAiB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YAC1D,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;gBAC3B,OAAO,IAAI,CAAC,GAAG,IAAI,iBAAiB,CAAC,CAAC;YACxC,CAAC;YACD,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,IAAI,CAAC,EAAE,CAAC;gBAC7B,OAAO,IAAI,CAAC;YACd,CAAC;YACD,yEAAyE;YACzE,MAAM,IAAI,GACR,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC;YAC/D,OAAO,IAAI,CACT,GAAG,IAAI,mBAAmB,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,YAAY,IAAI,EAAE,CACjE,CAAC;QACJ,CAAC,CAAC;IACJ,CAAC;IAED,aAAa,EAAE,CAAC,IAAI,EAAE,EAAE;QACtB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,SAAS,EAAE,EAAE,EAAE;YAC7B,IAAI,CAAC;gBACH,MAAM,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC,CAAC;gBACvC,OAAO,IAAI,CAAC;YACd,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO,IAAI,CAAC,GAAG,IAAI,kBAAkB,CAAC,CAAC;YACzC,CAAC;QACH,CAAC,CAAC;IACJ,CAAC;IAED,aAAa,EAAE,CAAC,IAAI,EAAE,EAAE;QACtB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,SAAS,EAAE,EAAE,EAAE;YAC7B,IAAI,CAAC;gBACH,MAAM,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC,CAAC;gBACvC,OAAO,IAAI,CAAC,GAAG,IAAI,eAAe,CAAC,CAAC;YACtC,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO,IAAI,CAAC;YACd,CAAC;QACH,CAAC,CAAC;IACJ,CAAC;IAED,aAAa,EAAE,CAAC,IAAI,EAAE,EAAE;QACtB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,SAAS,EAAE,EAAE,EAAE;YAC7B,MAAM,QAAQ,GAAG,MAAM,iBAAiB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YAC1D,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;gBAC3B,OAAO,IAAI,CAAC,GAAG,IAAI,iBAAiB,CAAC,CAAC;YACxC,CAAC;YACD,OAAO,QAAQ,KAAK,IAAI;gBACtB,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CAAC,GAAG,IAAI,qDAAqD,CAAC,CAAC;QACzE,CAAC,CAAC;IACJ,CAAC;IAED,WAAW,EAAE,CAAC,IAAI,EAAE,EAAE;QACpB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE;YAC1B,MAAM,IAAI,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;YAC/B,OAAO,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC;gBACxB,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CACF,gBAAgB,IAAI,YAAY,IAAI,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,GAAG,IAAI,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAChG,CAAC;QACR,CAAC,CAAC;IACJ,CAAC;IAED,eAAe,EAAE,CAAC,IAAI,EAAE,EAAE;QACxB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE;YAC1B,MAAM,IAAI,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;YAC/B,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,KAAK,IAAI,CAAC,CAAC,MAAM,CAAC;YAC1D,OAAO,KAAK,KAAK,CAAC;gBAChB,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CAAC,UAAU,IAAI,IAAI,KAAK,mCAAmC,CAAC,CAAC;QACvE,CAAC,CAAC;IACJ,CAAC;IAED,gBAAgB,EAAE,CAAC,IAAI,EAAE,EAAE;QACzB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE;YAC1B,OAAO,mBAAmB,CAAC,MAAM,CAAC,CAAC,QAAQ,CAAC,IAAI,CAAC;gBAC/C,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CAAC,GAAG,IAAI,qCAAqC,CAAC,CAAC;QACzD,CAAC,CAAC;IACJ,CAAC;IAED,mBAAmB,EAAE,CAAC,IAAI,EAAE,EAAE;QAC5B,oEAAoE;QACpE,wEAAwE;QACxE,wEAAwE;QACxE,mCAAmC;QACnC,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE;YAC1B,wEAAwE;YACxE,mEAAmE;YACnE,MAAM,QAAQ,GAAG,eAAe,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,KAAK,IAAI,CAAC,CAAC;YACzE,OAAO,QAAQ,CAAC,MAAM,KAAK,CAAC;gBAC1B,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CACF,GAAG,IAAI,WAAW,QAAQ,CAAC,MAAM,uFAAuF,CACzH,CAAC;QACR,CAAC,CAAC;IACJ,CAAC;IAED,gBAAgB,EAAE,CAAC,IAAI,EAAE,EAAE;QACzB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC;QACzB,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,KAAK,CAAC,IAAK,KAAgB,GAAG,CAAC,EAAE,CAAC;YAC1D,MAAM,IAAI,KAAK,CAAC,qDAAqD,CAAC,CAAC;QACzE,CAAC;QACD,OAAO,KAAK,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE;YAC1B,sEAAsE;YACtE,sEAAsE;YACtE,cAAc;YACd,mEAAmE;YACnE,sEAAsE;YACtE,MAAM,IAAI,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,KAAK,IAAI,CAAC,CAAC,MAAM,CAAC;YACtE,OAAO,IAAI,IAAK,KAAgB;gBAC9B,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CAAC,UAAU,IAAI,IAAI,IAAI,8BAA8B,KAAe,EAAE,CAAC,CAAC;QAClF,CAAC,CAAC;IACJ,CAAC;IAED,gBAAgB,EAAE,CAAC,IAAI,EAAE,EAAE;QACzB,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,OAAO,KAAK,EAAE,EAAE,SAAS,EAAE,gBAAgB,EAAE,EAAE,EAAE;YAC/C,oEAAoE;YACpE,yDAAyD;YACzD,MAAM,MAAM,GAAG,MAAM,iBAAiB,CAAC,gBAAgB,EAAE,IAAI,CAAC,CAAC;YAC/D,IAAI,MAAM,KAAK,SAAS,EAAE,CAAC;gBACzB,MAAM,IAAI,KAAK,CACb,mBAAmB,IAAI,iEAAiE,CACzF,CAAC;YACJ,CAAC;YACD,MAAM,KAAK,GAAG,MAAM,iBAAiB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YACvD,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;gBACxB,OAAO,IAAI,CAAC,GAAG,IAAI,8CAA8C,CAAC,CAAC;YACrE,CAAC;YACD,IAAI,KAAK,KAAK,MAAM,EAAE,CAAC;gBACrB,OAAO,IAAI,CAAC;YACd,CAAC;YACD,MAAM,IAAI,GAAG,kBAAkB,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;YAC/C,OAAO,IAAI,CACT,GAAG,IAAI,yBAAyB,IAAI,kCAAkC,CACvE,CAAC;QACJ,CAAC,CAAC;IACJ,CAAC;IAED,gBAAgB,EAAE,CAAC,IAAI,EAAE,EAAE;QACzB,MAAM,OAAO,GAAG,aAAa,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;QAC/C,MAAM,KAAK,GAAG,OAAO,IAAI,CAAC,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC;QAChE,MAAM,UAAU,GAAG,IAAI,MAAM,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;QAC9C,OAAO,KAAK,EAAE,EAAE,WAAW,EAAE,EAAE,EAAE;YAC/B,OAAO,UAAU,CAAC,IAAI,CAAC,WAAW,CAAC;gBACjC,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CACF,6BAA6B,OAAO,IAAI,KAAK,KAAK,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAE,CAC9F,CAAC;QACR,CAAC,CAAC;IACJ,CAAC;IAED,iBAAiB,EAAE,CAAC,IAAI,EAAE,EAAE;QAC1B,MAAM,OAAO,GAAG,aAAa,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;QAC/C,MAAM,KAAK,GAAG,OAAO,IAAI,CAAC,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC;QAChE,MAAM,UAAU,GAAG,IAAI,MAAM,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;QAC9C,OAAO,KAAK,EAAE,EAAE,WAAW,EAAE,EAAE,EAAE;YAC/B,OAAO,UAAU,CAAC,IAAI,CAAC,WAAW,CAAC;gBACjC,CAAC,CAAC,IAAI,CAAC,uBAAuB,OAAO,IAAI,KAAK,2BAA2B,CAAC;gBAC1E,CAAC,CAAC,IAAI,CAAC;QACX,CAAC,CAAC;IACJ,CAAC;IAED,iBAAiB,EAAE,CAAC,IAAI,EAAE,EAAE;QAC1B,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC;QACzB,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,KAAK,CAAC,IAAK,KAAgB,GAAG,CAAC,EAAE,CAAC;YAC1D,MAAM,IAAI,KAAK,CAAC,wDAAwD,CAAC,CAAC;QAC5E,CAAC;QACD,OAAO,KAAK,EAAE,EAAE,SAAS,EAAE,EAAE,EAAE;YAC7B,OAAO,SAAS,IAAK,KAAgB;gBACnC,CAAC,CAAC,IAAI;gBACN,CAAC,CAAC,IAAI,CAAC,QAAQ,SAAS,6BAA6B,KAAe,EAAE,CAAC,CAAC;QAC5E,CAAC,CAAC;IACJ,CAAC;CACF,CAAC;AAEF,MAAM,CAAC,MAAM,YAAY,GAAsB,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;AAEtE,MAAM,UAAU,YAAY,CAAC,IAAgB;IAC3C,MAAM,OAAO,GAAG,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACrC,IAAI,OAAO,KAAK,SAAS,EAAE,CAAC;QAC1B,MAAM,IAAI,KAAK,CACb,wBAAwB,IAAI,CAAC,IAAI,mBAAmB,YAAY,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAC/E,CAAC;IACJ,CAAC;IACD,OAAO,OAAO,CAAC,IAAI,CAAC,CAAC;AACvB,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,IAAgB;IAC7C,MAAM,MAAM,GAAG,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC;SAChC,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC,GAAG,KAAK,MAAM,CAAC;SACjC,GAAG,CAAC,CAAC,CAAC,GAAG,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,GAAG,GAAG,IAAI,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE,CAAC;SACxD,IAAI,CAAC,GAAG,CAAC,CAAC;IACb,OAAO,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,IAAI,IAAI,MAAM,EAAE,CAAC;AACpE,CAAC"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"parse-args.d.ts","sourceRoot":"","sources":["../../src/evals/parse-args.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAE5C;;;;;GAKG;AACH,MAAM,WAAW,WAAW;IAC1B,QAAQ,CAAC,KAAK,EAAE,SAAS,MAAM,EAAE,CAAC;IAClC,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,YAAY,EAAE,SAAS,UAAU,EAAE,CAAC;IAC7C,QAAQ,CAAC,YAAY,EAAE,SAAS,MAAM,EAAE,CAAC;IACzC,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;CACxB;
|
|
1
|
+
{"version":3,"file":"parse-args.d.ts","sourceRoot":"","sources":["../../src/evals/parse-args.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAE5C;;;;;GAKG;AACH,MAAM,WAAW,WAAW;IAC1B,QAAQ,CAAC,KAAK,EAAE,SAAS,MAAM,EAAE,CAAC;IAClC,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,YAAY,EAAE,SAAS,UAAU,EAAE,CAAC;IAC7C,QAAQ,CAAC,YAAY,EAAE,SAAS,MAAM,EAAE,CAAC;IACzC,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;CACxB;AAUD,wBAAgB,aAAa,CAAC,IAAI,EAAE,SAAS,MAAM,EAAE,GAAG,WAAW,CA4DlE"}
|
package/dist/evals/parse-args.js
CHANGED
|
@@ -1,3 +1,10 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Options handed straight to the main CLI's parser.
|
|
3
|
+
*
|
|
4
|
+
* --api-key is here because the credential chain names it the highest-priority
|
|
5
|
+
* source: an ephemeral machine gets its key as a flag, not as stored state.
|
|
6
|
+
*/
|
|
7
|
+
const PROVIDER_OPTIONS = new Set(["--provider", "--model", "--api-key"]);
|
|
1
8
|
export function parseEvalArgs(argv) {
|
|
2
9
|
const tasks = [];
|
|
3
10
|
const capabilities = [];
|
|
@@ -44,7 +51,7 @@ export function parseEvalArgs(argv) {
|
|
|
44
51
|
else if (argument === "--allow-shell") {
|
|
45
52
|
capabilities.push("shell");
|
|
46
53
|
}
|
|
47
|
-
else if (argument
|
|
54
|
+
else if (argument !== undefined && PROVIDER_OPTIONS.has(argument)) {
|
|
48
55
|
// Passed through so the Provider registry resolves them exactly as the
|
|
49
56
|
// real CLI does, including environment fallbacks.
|
|
50
57
|
providerArgs.push(argument, value(index, argument));
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"parse-args.js","sourceRoot":"","sources":["../../src/evals/parse-args.ts"],"names":[],"mappings":"AAkBA,MAAM,UAAU,aAAa,CAAC,IAAuB;IACnD,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,MAAM,YAAY,GAAiB,EAAE,CAAC;IACtC,MAAM,YAAY,GAAa,EAAE,CAAC;IAClC,IAAI,QAA4B,CAAC;IACjC,IAAI,SAAS,GAAG,OAAO,CAAC;IACxB,IAAI,QAA4B,CAAC;IACjC,IAAI,IAAI,GAAG,KAAK,CAAC;IAEjB,MAAM,KAAK,GAAG,CAAC,KAAa,EAAE,MAAc,EAAU,EAAE;QACtD,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC;QAC7B,IAAI,IAAI,KAAK,SAAS,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,MAAM,IAAI,SAAS,CAAC,GAAG,MAAM,oBAAoB,CAAC,CAAC;QACrD,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC,CAAC;IAEF,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC,MAAM,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC;QACpD,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;QAC7B,IAAI,QAAQ,KAAK,QAAQ,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YAC/C,IAAI,GAAG,IAAI,CAAC;QACd,CAAC;aAAM,IAAI,QAAQ,KAAK,QAAQ,EAAE,CAAC;YACjC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC,CAAC;YACnC,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,EAAE,CAAC;YACrC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,KAAK,EAAE,YAAY,CAAC,CAAC,CAAC;YAClD,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,CAAC,IAAI,MAAM,GAAG,EAAE,EAAE,CAAC;gBAC/D,MAAM,IAAI,SAAS,CAAC,6CAA6C,CAAC,CAAC;YACrE,CAAC;YACD,QAAQ,GAAG,MAAM,CAAC;YAClB,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,OAAO,EAAE,CAAC;YAChC,SAAS,GAAG,KAAK,CAAC,KAAK,EAAE,OAAO,CAAC,CAAC;YAClC,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,QAAQ,EAAE,CAAC;YACjC,QAAQ,GAAG,KAAK,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC;YAClC,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,gBAAgB,EAAE,CAAC;YACzC,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC9B,CAAC;aAAM,IAAI,QAAQ,KAAK,eAAe,EAAE,CAAC;YACxC,YAAY,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;QAC7B,CAAC;aAAM,IAAI,QAAQ,KAAK,
|
|
1
|
+
{"version":3,"file":"parse-args.js","sourceRoot":"","sources":["../../src/evals/parse-args.ts"],"names":[],"mappings":"AAkBA;;;;;GAKG;AACH,MAAM,gBAAgB,GAAG,IAAI,GAAG,CAAC,CAAC,YAAY,EAAE,SAAS,EAAE,WAAW,CAAC,CAAC,CAAC;AAEzE,MAAM,UAAU,aAAa,CAAC,IAAuB;IACnD,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,MAAM,YAAY,GAAiB,EAAE,CAAC;IACtC,MAAM,YAAY,GAAa,EAAE,CAAC;IAClC,IAAI,QAA4B,CAAC;IACjC,IAAI,SAAS,GAAG,OAAO,CAAC;IACxB,IAAI,QAA4B,CAAC;IACjC,IAAI,IAAI,GAAG,KAAK,CAAC;IAEjB,MAAM,KAAK,GAAG,CAAC,KAAa,EAAE,MAAc,EAAU,EAAE;QACtD,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC;QAC7B,IAAI,IAAI,KAAK,SAAS,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,MAAM,IAAI,SAAS,CAAC,GAAG,MAAM,oBAAoB,CAAC,CAAC;QACrD,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC,CAAC;IAEF,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC,MAAM,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC;QACpD,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;QAC7B,IAAI,QAAQ,KAAK,QAAQ,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YAC/C,IAAI,GAAG,IAAI,CAAC;QACd,CAAC;aAAM,IAAI,QAAQ,KAAK,QAAQ,EAAE,CAAC;YACjC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC,CAAC;YACnC,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,YAAY,EAAE,CAAC;YACrC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,KAAK,EAAE,YAAY,CAAC,CAAC,CAAC;YAClD,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,CAAC,IAAI,MAAM,GAAG,EAAE,EAAE,CAAC;gBAC/D,MAAM,IAAI,SAAS,CAAC,6CAA6C,CAAC,CAAC;YACrE,CAAC;YACD,QAAQ,GAAG,MAAM,CAAC;YAClB,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,OAAO,EAAE,CAAC;YAChC,SAAS,GAAG,KAAK,CAAC,KAAK,EAAE,OAAO,CAAC,CAAC;YAClC,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,QAAQ,EAAE,CAAC;YACjC,QAAQ,GAAG,KAAK,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC;YAClC,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,gBAAgB,EAAE,CAAC;YACzC,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC9B,CAAC;aAAM,IAAI,QAAQ,KAAK,eAAe,EAAE,CAAC;YACxC,YAAY,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;QAC7B,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,IAAI,gBAAgB,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;YACpE,uEAAuE;YACvE,kDAAkD;YAClD,YAAY,CAAC,IAAI,CAAC,QAAQ,EAAE,KAAK,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC,CAAC;YACpD,KAAK,IAAI,CAAC,CAAC;QACb,CAAC;aAAM,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;YAClC,MAAM,IAAI,SAAS,CAAC,mBAAmB,QAAQ,EAAE,CAAC,CAAC;QACrD,CAAC;IACH,CAAC;IAED,OAAO;QACL,KAAK;QACL,GAAG,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,CAAC;QAC/C,SAAS;QACT,GAAG,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,CAAC;QAC/C,YAAY;QACZ,YAAY;QACZ,IAAI;KACL,CAAC;AACJ,CAAC"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"report.d.ts","sourceRoot":"","sources":["../../src/evals/report.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAG9C,MAAM,WAAW,UAAU;IACzB,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;CAC5B;AA4CD;;;;;GAKG;AACH,wBAAgB,WAAW,CAAC,OAAO,EAAE,SAAS,UAAU,EAAE,GAAG,MAAM,CA0ClE;AAED,wEAAwE;AACxE,wBAAgB,cAAc,CAAC,OAAO,EAAE,SAAS,UAAU,EAAE,GAAG,MAAM,CAgBrE;AAED,wBAAgB,YAAY,CAC1B,OAAO,EAAE,SAAS,UAAU,EAAE,EAC9B,IAAI,EAAE,UAAU,GACf,MAAM,
|
|
1
|
+
{"version":3,"file":"report.d.ts","sourceRoot":"","sources":["../../src/evals/report.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAG9C,MAAM,WAAW,UAAU;IACzB,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;CAC5B;AA4CD;;;;;GAKG;AACH,wBAAgB,WAAW,CAAC,OAAO,EAAE,SAAS,UAAU,EAAE,GAAG,MAAM,CA0ClE;AAED,wEAAwE;AACxE,wBAAgB,cAAc,CAAC,OAAO,EAAE,SAAS,UAAU,EAAE,GAAG,MAAM,CAgBrE;AAED,wBAAgB,YAAY,CAC1B,OAAO,EAAE,SAAS,UAAU,EAAE,EAC9B,IAAI,EAAE,UAAU,GACf,MAAM,CAmCR"}
|
package/dist/evals/report.js
CHANGED
|
@@ -53,11 +53,11 @@ export function formatTable(results) {
|
|
|
53
53
|
task: Math.max(4, ...rows.map((row) => row.task.length)),
|
|
54
54
|
pass: Math.max(4, ...rows.map((row) => row.pass.length)),
|
|
55
55
|
turns: Math.max(5, ...rows.map((row) => row.turns.length)),
|
|
56
|
-
tokens: Math.max(
|
|
56
|
+
tokens: Math.max(7, ...rows.map((row) => row.tokens.length)),
|
|
57
57
|
tools: Math.max(5, ...rows.map((row) => row.tools.length)),
|
|
58
58
|
};
|
|
59
59
|
const lines = [
|
|
60
|
-
`${pad("task", widths.task)} ${pad("pass", widths.pass)} ${pad("turns", widths.turns)} ${pad("
|
|
60
|
+
`${pad("task", widths.task)} ${pad("pass", widths.pass)} ${pad("turns", widths.turns)} ${pad("tok/att", widths.tokens)} ${pad("tools", widths.tools)} p50`,
|
|
61
61
|
];
|
|
62
62
|
for (const row of rows) {
|
|
63
63
|
lines.push(`${pad(row.task, widths.task)} ${pad(row.pass, widths.pass)} ${pad(row.turns, widths.turns)} ${pad(row.tokens, widths.tokens)} ${pad(row.tools, widths.tools)} ${row.time}`);
|
|
@@ -107,6 +107,7 @@ export function toJsonReport(results, meta) {
|
|
|
107
107
|
turnCount: attempt.turnCount,
|
|
108
108
|
durationMs: attempt.durationMs,
|
|
109
109
|
toolsUsed: attempt.toolsUsed,
|
|
110
|
+
toolCalls: attempt.toolCalls,
|
|
110
111
|
...(attempt.usage === undefined ? {} : { usage: attempt.usage }),
|
|
111
112
|
failures: attempt.failures,
|
|
112
113
|
})),
|
package/dist/evals/report.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"report.js","sourceRoot":"","sources":["../../src/evals/report.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AASrD,SAAS,MAAM,CAAC,MAAyB;IACvC,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,MAAM,MAAM,GAAG,CAAC,GAAG,MAAM,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACjD,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IAC7C,OAAO,MAAM,CAAC,MAAM,GAAG,CAAC,KAAK,CAAC;QAC5B,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;QACrE,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;AAC5B,CAAC;AAED,SAAS,IAAI,CAAC,MAAyB;IACrC,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,OAAO,MAAM,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,GAAG,KAAK,EAAE,CAAC,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC;AAC3E,CAAC;AAED,iEAAiE;AACjE,SAAS,YAAY,CAAC,MAAkB;IACtC,MAAM,MAAM,GAAG,MAAM,CAAC,QAAQ;SAC3B,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,WAAW,CAAC;SAClD,MAAM,CAAC,CAAC,KAAK,EAAmB,EAAE,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC;IAC3D,OAAO,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,YAAY,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;AAClE,CAAC;AAED,SAAS,WAAW,CAAC,OAA8B;IACjD,OAAO,OAAO,CAAC,MAAM,CACnB,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,CACd,GAAG;QACH,MAAM,CAAC,QAAQ,CAAC,MAAM,CACpB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,CAAC,KAAK,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,WAAW,IAAI,CAAC,CAAC,EACnE,CAAC,CACF,EACH,CAAC,CACF,CAAC;AACJ,CAAC;AAED,SAAS,GAAG,CAAC,KAAa,EAAE,KAAa;IACvC,OAAO,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,GAAG,GAAG,CAAC,MAAM,CAAC,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,CAAC;AAClF,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,WAAW,CAAC,OAA8B;IACxD,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACzB,OAAO,iBAAiB,CAAC;IAC3B,CAAC;IAED,MAAM,IAAI,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QACpC,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,IAAI,EAAE,GAAG,MAAM,CAAC,MAAM,IAAI,MAAM,CAAC,KAAK,EAAE;QACxC,KAAK,EAAE,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;QAC3E,MAAM,EAAE,YAAY,CAAC,MAAM,CAAC;QAC5B,KAAK,EAAE;YACL,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;SACpE,CAAC,IAAI,CAAC,GAAG,CAAC;QACX,IAAI,EAAE,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG;KAC7F,CAAC,CAAC,CAAC;IAEJ,MAAM,MAAM,GAAG;QACb,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACxD,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACxD,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;QAC1D,MAAM,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAC5D,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;KAC3D,CAAC;IAEF,MAAM,KAAK,GAAG;QACZ,GAAG,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,
|
|
1
|
+
{"version":3,"file":"report.js","sourceRoot":"","sources":["../../src/evals/report.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AASrD,SAAS,MAAM,CAAC,MAAyB;IACvC,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,MAAM,MAAM,GAAG,CAAC,GAAG,MAAM,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACjD,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IAC7C,OAAO,MAAM,CAAC,MAAM,GAAG,CAAC,KAAK,CAAC;QAC5B,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;QACrE,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;AAC5B,CAAC;AAED,SAAS,IAAI,CAAC,MAAyB;IACrC,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,OAAO,MAAM,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,GAAG,KAAK,EAAE,CAAC,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC;AAC3E,CAAC;AAED,iEAAiE;AACjE,SAAS,YAAY,CAAC,MAAkB;IACtC,MAAM,MAAM,GAAG,MAAM,CAAC,QAAQ;SAC3B,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,WAAW,CAAC;SAClD,MAAM,CAAC,CAAC,KAAK,EAAmB,EAAE,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC;IAC3D,OAAO,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,YAAY,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;AAClE,CAAC;AAED,SAAS,WAAW,CAAC,OAA8B;IACjD,OAAO,OAAO,CAAC,MAAM,CACnB,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,CACd,GAAG;QACH,MAAM,CAAC,QAAQ,CAAC,MAAM,CACpB,CAAC,KAAK,EAAE,OAAO,EAAE,EAAE,CAAC,KAAK,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,WAAW,IAAI,CAAC,CAAC,EACnE,CAAC,CACF,EACH,CAAC,CACF,CAAC;AACJ,CAAC;AAED,SAAS,GAAG,CAAC,KAAa,EAAE,KAAa;IACvC,OAAO,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,GAAG,GAAG,CAAC,MAAM,CAAC,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,CAAC;AAClF,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,WAAW,CAAC,OAA8B;IACxD,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACzB,OAAO,iBAAiB,CAAC;IAC3B,CAAC;IAED,MAAM,IAAI,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QACpC,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,IAAI,EAAE,GAAG,MAAM,CAAC,MAAM,IAAI,MAAM,CAAC,KAAK,EAAE;QACxC,KAAK,EAAE,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;QAC3E,MAAM,EAAE,YAAY,CAAC,MAAM,CAAC;QAC5B,KAAK,EAAE;YACL,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;SACpE,CAAC,IAAI,CAAC,GAAG,CAAC;QACX,IAAI,EAAE,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG;KAC7F,CAAC,CAAC,CAAC;IAEJ,MAAM,MAAM,GAAG;QACb,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACxD,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACxD,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;QAC1D,MAAM,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAC5D,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;KAC3D,CAAC;IAEF,MAAM,KAAK,GAAG;QACZ,GAAG,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,KAAK,CAAC,OAAO;KAChK,CAAC;IACF,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,KAAK,CAAC,IAAI,CACR,GAAG,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,GAAG,CAAC,KAAK,EAAE,MAAM,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,GAAG,CAAC,GAAG,CAAC,KAAK,EAAE,MAAM,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,IAAI,EAAE,CACjL,CAAC;IACJ,CAAC;IAED,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,CAAC,KAAK,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;IAC3E,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,CAAC,GAAG,GAAG,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;IACrE,MAAM,OAAO,GAAG,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,MAAM,GAAG,KAAK,CAAC,GAAG,GAAG,CAAC,CAAC;IACrE,MAAM,KAAK,GAAG,WAAW,CAAC,OAAO,CAAC,CAAC;IACnC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACf,KAAK,CAAC,IAAI,CACR,YAAY,MAAM,IAAI,KAAK,KAAK,OAAO,KAAK,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,KAAK,YAAY,CAAC,KAAK,CAAC,SAAS,EAAE,CACnG,CAAC;IACF,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC;AACjC,CAAC;AAED,wEAAwE;AACxE,MAAM,UAAU,cAAc,CAAC,OAA8B;IAC3D,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,KAAK,MAAM,MAAM,IAAI,OAAO,EAAE,CAAC;QAC7B,MAAM,QAAQ,GAAG,IAAI,GAAG,EAAoB,CAAC;QAC7C,KAAK,MAAM,OAAO,IAAI,MAAM,CAAC,QAAQ,EAAE,CAAC;YACtC,KAAK,MAAM,OAAO,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;gBACvC,MAAM,QAAQ,GAAG,QAAQ,CAAC,GAAG,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC;gBAC7C,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;gBAC/B,QAAQ,CAAC,GAAG,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;YAClC,CAAC;QACH,CAAC;QACD,KAAK,MAAM,CAAC,MAAM,EAAE,QAAQ,CAAC,IAAI,QAAQ,EAAE,CAAC;YAC1C,KAAK,CAAC,IAAI,CAAC,GAAG,MAAM,CAAC,IAAI,YAAY,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,KAAK,MAAM,EAAE,CAAC,CAAC;QACxE,CAAC;IACH,CAAC;IACD,OAAO,KAAK,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC;AAC3D,CAAC;AAED,MAAM,UAAU,YAAY,CAC1B,OAA8B,EAC9B,IAAgB;IAEhB,OAAO;QACL,aAAa,EAAE,CAAC;QAChB,GAAG,IAAI;QACP,KAAK,EAAE,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;YAC9B,IAAI,EAAE,MAAM,CAAC,IAAI;YACjB,MAAM,EAAE,MAAM,CAAC,MAAM;YACrB,KAAK,EAAE,MAAM,CAAC,KAAK;YACnB,QAAQ,EAAE,MAAM,CAAC,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,GAAG,MAAM,CAAC,KAAK;YAC/D,SAAS,EAAE,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;YACpE,WAAW,EAAE,MAAM,CAAC,QAAQ,CAAC,MAAM,CACjC,CAAC,GAAG,EAAE,OAAO,EAAE,EAAE,CAAC,GAAG,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,WAAW,IAAI,CAAC,CAAC,EAC/D,CAAC,CACF;YACD,gBAAgB,EAAE,MAAM,CACtB,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,UAAU,CAAC,CACrD;YACD,QAAQ,EAAE,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;gBAC1C,OAAO,EAAE,OAAO,CAAC,OAAO;gBACxB,MAAM,EAAE,OAAO,CAAC,MAAM;gBACtB,MAAM,EAAE,OAAO,CAAC,MAAM;gBACtB,SAAS,EAAE,OAAO,CAAC,SAAS;gBAC5B,UAAU,EAAE,OAAO,CAAC,UAAU;gBAC9B,SAAS,EAAE,OAAO,CAAC,SAAS;gBAC5B,SAAS,EAAE,OAAO,CAAC,SAAS;gBAC5B,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC;gBAChE,QAAQ,EAAE,OAAO,CAAC,QAAQ;aAC3B,CAAC,CAAC;SACJ,CAAC,CAAC;QACH,OAAO,EAAE;YACP,MAAM,EAAE,OAAO,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,CAAC,KAAK,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC;YACnE,KAAK,EAAE,OAAO,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,CAAC,GAAG,GAAG,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC;YAC7D,WAAW,EAAE,WAAW,CAAC,OAAO,CAAC;SAClC;KACF,CAAC;AACJ,CAAC"}
|
package/dist/evals/runner.d.ts
CHANGED
|
@@ -2,6 +2,22 @@ import type { LLMClient } from "../llm.js";
|
|
|
2
2
|
import type { UsageTotal } from "../providers/usage.js";
|
|
3
3
|
import { type AttemptFacts } from "./graders.js";
|
|
4
4
|
import type { Capability, Task } from "./task.js";
|
|
5
|
+
/** One tool result, in call order. */
|
|
6
|
+
export interface ToolCallRecord {
|
|
7
|
+
readonly name: string;
|
|
8
|
+
readonly ok: boolean;
|
|
9
|
+
/**
|
|
10
|
+
* What the call was aimed at: the `path` argument, or a truncated `command`.
|
|
11
|
+
*
|
|
12
|
+
* Full arguments are not kept — an edit carries both halves of the change
|
|
13
|
+
* and would bury the report in file contents. But a name and a success flag
|
|
14
|
+
* alone cannot tell a model that read the four files on an import chain from
|
|
15
|
+
* one that read four unrelated files, and on a task built out of 41 files
|
|
16
|
+
* that distinction is the entire measurement. Omitted when the tool takes
|
|
17
|
+
* neither.
|
|
18
|
+
*/
|
|
19
|
+
readonly target?: string;
|
|
20
|
+
}
|
|
5
21
|
export interface AttemptResult {
|
|
6
22
|
readonly attempt: number;
|
|
7
23
|
readonly passed: boolean;
|
|
@@ -9,6 +25,14 @@ export interface AttemptResult {
|
|
|
9
25
|
readonly turnCount: number;
|
|
10
26
|
readonly durationMs: number;
|
|
11
27
|
readonly toolsUsed: readonly string[];
|
|
28
|
+
/**
|
|
29
|
+
* Every call in order, with whether it succeeded.
|
|
30
|
+
*
|
|
31
|
+
* `toolsUsed` is a deduplicated set, so a model that guessed a path, got an
|
|
32
|
+
* error and recovered looks identical there to one that never guessed. This
|
|
33
|
+
* is what makes that question answerable from the report.
|
|
34
|
+
*/
|
|
35
|
+
readonly toolCalls: readonly ToolCallRecord[];
|
|
12
36
|
/** What the attempt cost, when the Provider reported it. */
|
|
13
37
|
readonly usage?: UsageTotal;
|
|
14
38
|
/** One entry per failed grader, in task order. */
|
|
@@ -21,7 +45,12 @@ export interface TaskResult {
|
|
|
21
45
|
readonly total: number;
|
|
22
46
|
}
|
|
23
47
|
export interface RunnerOptions {
|
|
24
|
-
/**
|
|
48
|
+
/**
|
|
49
|
+
* Called once per attempt, so a test can hand out a fresh fake each time.
|
|
50
|
+
* The CLI returns the same client every time on purpose: it holds only
|
|
51
|
+
* readonly config and takes the history per request, so attempts cannot
|
|
52
|
+
* leak into one another through it.
|
|
53
|
+
*/
|
|
25
54
|
readonly createClient: () => LLMClient;
|
|
26
55
|
readonly systemPrompt: string;
|
|
27
56
|
readonly capabilities: readonly Capability[];
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runner.d.ts","sourceRoot":"","sources":["../../src/evals/runner.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AAC3C,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,uBAAuB,CAAC;
|
|
1
|
+
{"version":3,"file":"runner.d.ts","sourceRoot":"","sources":["../../src/evals/runner.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AAC3C,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,uBAAuB,CAAC;AAIxD,OAAO,EAAgC,KAAK,YAAY,EAAE,MAAM,cAAc,CAAC;AAC/E,OAAO,KAAK,EAAE,UAAU,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AAElD,sCAAsC;AACtC,MAAM,WAAW,cAAc;IAC7B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC;IACrB;;;;;;;;;OASG;IACH,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC;CAC1B;AAwBD,MAAM,WAAW,aAAa;IAC5B,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,MAAM,EAAE,OAAO,CAAC;IACzB,QAAQ,CAAC,MAAM,EAAE,YAAY,CAAC,QAAQ,CAAC,CAAC;IACxC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,QAAQ,CAAC,SAAS,EAAE,SAAS,MAAM,EAAE,CAAC;IACtC;;;;;;OAMG;IACH,QAAQ,CAAC,SAAS,EAAE,SAAS,cAAc,EAAE,CAAC;IAC9C,4DAA4D;IAC5D,QAAQ,CAAC,KAAK,CAAC,EAAE,UAAU,CAAC;IAC5B,kDAAkD;IAClD,QAAQ,CAAC,QAAQ,EAAE,SAAS,MAAM,EAAE,CAAC;CACtC;AAED,MAAM,WAAW,UAAU;IACzB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,SAAS,aAAa,EAAE,CAAC;IAC5C,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB;AAED,MAAM,WAAW,aAAa;IAC5B;;;;;OAKG;IACH,QAAQ,CAAC,YAAY,EAAE,MAAM,SAAS,CAAC;IACvC,QAAQ,CAAC,YAAY,EAAE,MAAM,CAAC;IAC9B,QAAQ,CAAC,YAAY,EAAE,SAAS,UAAU,EAAE,CAAC;IAC7C,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,SAAS,CAAC,EAAE,CAAC,MAAM,EAAE,aAAa,EAAE,IAAI,EAAE,IAAI,KAAK,IAAI,CAAC;IACjE,QAAQ,CAAC,MAAM,CAAC,EAAE,WAAW,CAAC;CAC/B;AAED,qBAAa,sBAAuB,SAAQ,KAAK;IACnC,QAAQ,CAAC,IAAI,EAAE,MAAM;IAAE,QAAQ,CAAC,OAAO,EAAE,SAAS,UAAU,EAAE;IAA1E,YAAqB,IAAI,EAAE,MAAM,EAAW,OAAO,EAAE,SAAS,UAAU,EAAE,EAOzE;CACF;AAwHD;;;;;;GAMG;AACH,wBAAsB,OAAO,CAC3B,IAAI,EAAE,IAAI,EACV,OAAO,EAAE,aAAa,GACrB,OAAO,CAAC,UAAU,CAAC,CAoBrB;AAED,wBAAsB,QAAQ,CAC5B,KAAK,EAAE,SAAS,IAAI,EAAE,EACtB,OAAO,EAAE,aAAa,GACrB,OAAO,CAAC,SAAS,UAAU,EAAE,CAAC,CAMhC"}
|
package/dist/evals/runner.js
CHANGED
|
@@ -1,13 +1,27 @@
|
|
|
1
1
|
import { Agent as AgentClass } from "../agent.js";
|
|
2
2
|
import { LocalWorkspace } from "../workspace.js";
|
|
3
|
-
import {
|
|
4
|
-
import { ReadFileTool } from "../tools/read-file.js";
|
|
5
|
-
import { SearchTextTool } from "../tools/search-text.js";
|
|
6
|
-
import { WriteFileTool } from "../tools/write-file.js";
|
|
7
|
-
import { EditFileTool } from "../tools/edit-file.js";
|
|
8
|
-
import { BashTool } from "../tools/bash.js";
|
|
3
|
+
import { toolsFor } from "./tools.js";
|
|
9
4
|
import { createAttemptWorkspace } from "./fixture.js";
|
|
10
5
|
import { createGrader, describeGrader } from "./graders.js";
|
|
6
|
+
const MAX_TARGET_LENGTH = 80;
|
|
7
|
+
/** The one argument worth keeping, short enough to sit in a report. */
|
|
8
|
+
function callTarget(argumentsValue) {
|
|
9
|
+
if (typeof argumentsValue !== "object" || argumentsValue === null) {
|
|
10
|
+
return undefined;
|
|
11
|
+
}
|
|
12
|
+
const record = argumentsValue;
|
|
13
|
+
const value = typeof record.path === "string"
|
|
14
|
+
? record.path
|
|
15
|
+
: typeof record.command === "string"
|
|
16
|
+
? record.command
|
|
17
|
+
: undefined;
|
|
18
|
+
if (value === undefined) {
|
|
19
|
+
return undefined;
|
|
20
|
+
}
|
|
21
|
+
return value.length > MAX_TARGET_LENGTH
|
|
22
|
+
? `${value.slice(0, MAX_TARGET_LENGTH)}…`
|
|
23
|
+
: value;
|
|
24
|
+
}
|
|
11
25
|
export class MissingCapabilityError extends Error {
|
|
12
26
|
task;
|
|
13
27
|
missing;
|
|
@@ -20,20 +34,6 @@ export class MissingCapabilityError extends Error {
|
|
|
20
34
|
this.name = "MissingCapabilityError";
|
|
21
35
|
}
|
|
22
36
|
}
|
|
23
|
-
function toolsFor(capabilities, cwd) {
|
|
24
|
-
const tools = [
|
|
25
|
-
new ListFilesTool(),
|
|
26
|
-
new SearchTextTool(),
|
|
27
|
-
new ReadFileTool(),
|
|
28
|
-
];
|
|
29
|
-
if (capabilities.includes("writes")) {
|
|
30
|
-
tools.push(new WriteFileTool(), new EditFileTool());
|
|
31
|
-
}
|
|
32
|
-
if (capabilities.includes("shell")) {
|
|
33
|
-
tools.push(new BashTool({ cwd }));
|
|
34
|
-
}
|
|
35
|
-
return tools;
|
|
36
|
-
}
|
|
37
37
|
function missingCapabilities(task, granted) {
|
|
38
38
|
return task.capabilities.filter((capability) => !granted.includes(capability));
|
|
39
39
|
}
|
|
@@ -75,6 +75,7 @@ async function runAttempt(task, attempt, options) {
|
|
|
75
75
|
}
|
|
76
76
|
const facts = {
|
|
77
77
|
workspace: workspace.path,
|
|
78
|
+
fixtureDirectory: task.fixtureDirectory,
|
|
78
79
|
finalAnswer,
|
|
79
80
|
turnCount,
|
|
80
81
|
status,
|
|
@@ -92,6 +93,28 @@ async function runAttempt(task, attempt, options) {
|
|
|
92
93
|
failures.push(`${describeGrader(spec)} — ${result.reason}`);
|
|
93
94
|
}
|
|
94
95
|
}
|
|
96
|
+
// The arguments live on the start event, the outcome on the end event;
|
|
97
|
+
// the call id is what joins them.
|
|
98
|
+
const targets = new Map();
|
|
99
|
+
for (const event of events) {
|
|
100
|
+
if (event.type === "tool_execution_start") {
|
|
101
|
+
const target = callTarget(event.arguments);
|
|
102
|
+
if (target !== undefined) {
|
|
103
|
+
targets.set(event.toolCallId, target);
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
const calls = events
|
|
108
|
+
.filter((event) => event.type === "tool_execution_end")
|
|
109
|
+
.map((event) => {
|
|
110
|
+
const end = event;
|
|
111
|
+
const target = targets.get(end.toolCallId);
|
|
112
|
+
return {
|
|
113
|
+
name: end.toolName,
|
|
114
|
+
ok: end.isError === false,
|
|
115
|
+
...(target === undefined ? {} : { target }),
|
|
116
|
+
};
|
|
117
|
+
});
|
|
95
118
|
return {
|
|
96
119
|
attempt,
|
|
97
120
|
passed: failures.length === 0,
|
|
@@ -99,11 +122,8 @@ async function runAttempt(task, attempt, options) {
|
|
|
99
122
|
turnCount,
|
|
100
123
|
durationMs: Date.now() - startedAt,
|
|
101
124
|
...(usage === undefined ? {} : { usage }),
|
|
102
|
-
toolsUsed: [
|
|
103
|
-
|
|
104
|
-
.filter((event) => event.type === "tool_execution_end")
|
|
105
|
-
.map((event) => event.toolName)),
|
|
106
|
-
],
|
|
125
|
+
toolsUsed: [...new Set(calls.map((call) => call.name))],
|
|
126
|
+
toolCalls: calls,
|
|
107
127
|
failures,
|
|
108
128
|
};
|
|
109
129
|
}
|
package/dist/evals/runner.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runner.js","sourceRoot":"","sources":["../../src/evals/runner.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,KAAK,IAAI,UAAU,EAAE,MAAM,aAAa,CAAC;
|
|
1
|
+
{"version":3,"file":"runner.js","sourceRoot":"","sources":["../../src/evals/runner.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,KAAK,IAAI,UAAU,EAAE,MAAM,aAAa,CAAC;AAIlD,OAAO,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAC;AACjD,OAAO,EAAE,QAAQ,EAAE,MAAM,YAAY,CAAC;AACtC,OAAO,EAAE,sBAAsB,EAAE,MAAM,cAAc,CAAC;AACtD,OAAO,EAAE,YAAY,EAAE,cAAc,EAAqB,MAAM,cAAc,CAAC;AAoB/E,MAAM,iBAAiB,GAAG,EAAE,CAAC;AAE7B,uEAAuE;AACvE,SAAS,UAAU,CAAC,cAAuB;IACzC,IAAI,OAAO,cAAc,KAAK,QAAQ,IAAI,cAAc,KAAK,IAAI,EAAE,CAAC;QAClE,OAAO,SAAS,CAAC;IACnB,CAAC;IACD,MAAM,MAAM,GAAG,cAAyC,CAAC;IACzD,MAAM,KAAK,GACT,OAAO,MAAM,CAAC,IAAI,KAAK,QAAQ;QAC7B,CAAC,CAAC,MAAM,CAAC,IAAI;QACb,CAAC,CAAC,OAAO,MAAM,CAAC,OAAO,KAAK,QAAQ;YAClC,CAAC,CAAC,MAAM,CAAC,OAAO;YAChB,CAAC,CAAC,SAAS,CAAC;IAClB,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;QACxB,OAAO,SAAS,CAAC;IACnB,CAAC;IACD,OAAO,KAAK,CAAC,MAAM,GAAG,iBAAiB;QACrC,CAAC,CAAC,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,iBAAiB,CAAC,GAAG;QACzC,CAAC,CAAC,KAAK,CAAC;AACZ,CAAC;AA6CD,MAAM,OAAO,sBAAuB,SAAQ,KAAK;IAC1B,IAAI;IAAmB,OAAO;IAAnD,YAAqB,IAAY,EAAW,OAA8B;QACxE,KAAK,CACH,SAAS,IAAI,WAAW,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,gBAAgB,OAAO;aACjE,GAAG,CAAC,CAAC,UAAU,EAAE,EAAE,CAAC,CAAC,UAAU,KAAK,QAAQ,CAAC,CAAC,CAAC,gBAAgB,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC;aACnF,IAAI,CAAC,GAAG,CAAC,GAAG,CAChB,CAAC;oBALiB,IAAI;uBAAmB,OAAO;QAMjD,IAAI,CAAC,IAAI,GAAG,wBAAwB,CAAC;IACvC,CAAC;CACF;AAED,SAAS,mBAAmB,CAC1B,IAAU,EACV,OAA8B;IAE9B,OAAO,IAAI,CAAC,YAAY,CAAC,MAAM,CAC7B,CAAC,UAAU,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAC9C,CAAC;AACJ,CAAC;AAED,KAAK,UAAU,UAAU,CACvB,IAAU,EACV,OAAe,EACf,OAAsB;IAEtB,MAAM,SAAS,GAAG,MAAM,sBAAsB,CAAC,IAAI,CAAC,gBAAgB,CAAC,CAAC;IACtE,MAAM,MAAM,GAAiB,EAAE,CAAC;IAChC,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;IAE7B,IAAI,CAAC;QACH,MAAM,KAAK,GAAU,IAAI,UAAU,CAAC;YAClC,YAAY,EAAE,OAAO,CAAC,YAAY;YAClC,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,GAAG,EAAE,OAAO,CAAC,YAAY,EAAE;YAC3B,KAAK,EAAE,QAAQ,CAAC,IAAI,CAAC,YAAY,EAAE,SAAS,CAAC,IAAI,CAAC;YAClD,SAAS,EAAE,IAAI,cAAc,CAAC,SAAS,CAAC,IAAI,EAAE;gBAC5C,WAAW,EAAE,IAAI,CAAC,YAAY,CAAC,QAAQ,CAAC,QAAQ,CAAC;aAClD,CAAC;YACF,SAAS,EAAE,KAAK;YAChB,OAAO,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC;SACvC,CAAC,CAAC;QAEH,IAAI,MAAM,GAA2B,OAAO,CAAC;QAC7C,IAAI,WAAW,GAAG,EAAE,CAAC;QACrB,IAAI,SAAS,GAAG,CAAC,CAAC;QAClB,IAAI,KAA6B,CAAC;QAClC,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,EAAE;gBAC1C,GAAG,CAAC,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC;aACpE,CAAC,CAAC;YACH,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC;YACvB,SAAS,GAAG,MAAM,CAAC,SAAS,CAAC;YAC7B,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC;YACrB,WAAW;gBACT,MAAM,CAAC,MAAM,KAAK,WAAW,CAAC,CAAC,CAAC,MAAM,CAAC,YAAY,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;QACrE,CAAC;QAAC,OAAO,KAAc,EAAE,CAAC;YACxB,yEAAyE;YACzE,mEAAmE;YACnE,MAAM,GAAG,OAAO,CAAC;YACjB,WAAW,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACvE,CAAC;QAED,MAAM,KAAK,GAAiB;YAC1B,SAAS,EAAE,SAAS,CAAC,IAAI;YACzB,gBAAgB,EAAE,IAAI,CAAC,gBAAgB;YACvC,WAAW;YACX,SAAS;YACT,MAAM;YACN,MAAM;SACP,CAAC;QAEF,MAAM,QAAQ,GAAa,EAAE,CAAC;QAC9B,IAAI,MAAM,KAAK,WAAW,EAAE,CAAC;YAC3B,QAAQ,CAAC,IAAI,CACX,MAAM,KAAK,WAAW;gBACpB,CAAC,CAAC,WAAW,IAAI,CAAC,QAAQ,oCAAoC;gBAC9D,CAAC,CAAC,gBAAgB,MAAM,KAAK,WAAW,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,EAAE,CACrE,CAAC;QACJ,CAAC;QACD,KAAK,MAAM,IAAI,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;YAChC,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,IAAI,CAAC,CAAC,KAAK,CAAC,CAAC;YAC/C,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC;gBACnB,QAAQ,CAAC,IAAI,CAAC,GAAG,cAAc,CAAC,IAAI,CAAC,MAAM,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC;YAC9D,CAAC;QACH,CAAC;QAED,uEAAuE;QACvE,kCAAkC;QAClC,MAAM,OAAO,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC1C,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;YAC3B,IAAI,KAAK,CAAC,IAAI,KAAK,sBAAsB,EAAE,CAAC;gBAC1C,MAAM,MAAM,GAAG,UAAU,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC;gBAC3C,IAAI,MAAM,KAAK,SAAS,EAAE,CAAC;oBACzB,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,UAAU,EAAE,MAAM,CAAC,CAAC;gBACxC,CAAC;YACH,CAAC;QACH,CAAC;QACD,MAAM,KAAK,GAA8B,MAAM;aAC5C,MAAM,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,IAAI,KAAK,oBAAoB,CAAC;aACtD,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE;YACb,MAAM,GAAG,GAAG,KAIX,CAAC;YACF,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;YAC3C,OAAO;gBACL,IAAI,EAAE,GAAG,CAAC,QAAQ;gBAClB,EAAE,EAAE,GAAG,CAAC,OAAO,KAAK,KAAK;gBACzB,GAAG,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC;aAC5C,CAAC;QACJ,CAAC,CAAC,CAAC;QAEL,OAAO;YACL,OAAO;YACP,MAAM,EAAE,QAAQ,CAAC,MAAM,KAAK,CAAC;YAC7B,MAAM;YACN,SAAS;YACT,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS;YAClC,GAAG,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC;YACzC,SAAS,EAAE,CAAC,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;YACvD,SAAS,EAAE,KAAK;YAChB,QAAQ;SACT,CAAC;IACJ,CAAC;YAAS,CAAC;QACT,MAAM,SAAS,CAAC,OAAO,EAAE,CAAC;IAC5B,CAAC;AACH,CAAC;AAED;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,OAAO,CAC3B,IAAU,EACV,OAAsB;IAEtB,MAAM,OAAO,GAAG,mBAAmB,CAAC,IAAI,EAAE,OAAO,CAAC,YAAY,CAAC,CAAC;IAChE,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACvB,MAAM,IAAI,sBAAsB,CAAC,IAAI,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;IACvD,CAAC;IAED,MAAM,KAAK,GAAG,OAAO,CAAC,QAAQ,IAAI,IAAI,CAAC,QAAQ,CAAC;IAChD,MAAM,QAAQ,GAAoB,EAAE,CAAC;IACrC,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,IAAI,KAAK,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC;QAC/C,MAAM,MAAM,GAAG,MAAM,UAAU,CAAC,IAAI,EAAE,KAAK,EAAE,OAAO,CAAC,CAAC;QACtD,QAAQ,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACtB,OAAO,CAAC,SAAS,EAAE,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;IACpC,CAAC;IAED,OAAO;QACL,IAAI,EAAE,IAAI,CAAC,IAAI;QACf,QAAQ;QACR,MAAM,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,MAAM;QAC3D,KAAK;KACN,CAAC;AACJ,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAC5B,KAAsB,EACtB,OAAsB;IAEtB,MAAM,OAAO,GAAiB,EAAE,CAAC;IACjC,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,OAAO,CAAC,IAAI,CAAC,MAAM,OAAO,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,CAAC;IAC7C,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC"}
|
package/dist/evals/task.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"task.d.ts","sourceRoot":"","sources":["../../src/evals/task.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"task.d.ts","sourceRoot":"","sources":["../../src/evals/task.ts"],"names":[],"mappings":"AAIA,qBAAa,SAAU,SAAQ,KAAK;IAClC,YAAY,OAAO,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE;QAAE,KAAK,CAAC,EAAE,OAAO,CAAA;KAAE,EAMzD;CACF;AAED,yEAAyE;AACzE,MAAM,MAAM,UAAU,GAAG,QAAQ,GAAG,OAAO,CAAC;AAE5C,MAAM,WAAW,UAAU;IACzB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC;CACjC;AAED,MAAM,WAAW,IAAI;IACnB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,YAAY,EAAE,SAAS,UAAU,EAAE,CAAC;IAC7C,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,OAAO,EAAE,SAAS,UAAU,EAAE,CAAC;IACxC,iEAAiE;IACjE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;CAC5B;AAoFD,wBAAgB,SAAS,CACvB,KAAK,EAAE,OAAO,EACd,SAAS,EAAE,MAAM,EACjB,gBAAgB,EAAE,MAAM,GACvB,IAAI,CA8CN;AAED,wBAAsB,QAAQ,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAgB/D;AAED,6DAA6D;AAC7D,wBAAsB,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,IAAI,EAAE,CAAC,CAwBtE"}
|
package/dist/evals/task.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { readFile, readdir, stat } from "node:fs/promises";
|
|
2
2
|
import path from "node:path";
|
|
3
|
+
import { toolNamesFor } from "./tools.js";
|
|
3
4
|
export class TaskError extends Error {
|
|
4
5
|
constructor(message, options) {
|
|
5
6
|
super(message);
|
|
@@ -38,6 +39,37 @@ function readPositiveInteger(value, fallback, maximum, field, name) {
|
|
|
38
39
|
}
|
|
39
40
|
return value;
|
|
40
41
|
}
|
|
42
|
+
/** Grader types whose "name" must be a tool the task actually grants. */
|
|
43
|
+
const TOOL_NAME_GRADERS = new Set([
|
|
44
|
+
"used-tool",
|
|
45
|
+
"not-used-tool",
|
|
46
|
+
"tool-succeeded",
|
|
47
|
+
"tool-never-failed",
|
|
48
|
+
]);
|
|
49
|
+
/**
|
|
50
|
+
* Refuses a grader that names a tool the task never hands to the model.
|
|
51
|
+
*
|
|
52
|
+
* `not-used-tool: write_file` on a read-only task can never fail — it asserts
|
|
53
|
+
* the model did not call something it was never given, so it reads like a
|
|
54
|
+
* check and is worth nothing. The mirror image, `used-tool` on a tool the task
|
|
55
|
+
* withholds, can never pass. Both are task-definition mistakes, and both are
|
|
56
|
+
* caught here rather than at run time so they cost nothing to find.
|
|
57
|
+
*/
|
|
58
|
+
function assertGradersCanFire(name, capabilities, graders) {
|
|
59
|
+
const available = toolNamesFor(capabilities);
|
|
60
|
+
for (const [index, grader] of graders.entries()) {
|
|
61
|
+
if (!TOOL_NAME_GRADERS.has(grader.type)) {
|
|
62
|
+
continue;
|
|
63
|
+
}
|
|
64
|
+
const tool = grader.name;
|
|
65
|
+
if (typeof tool !== "string" || available.includes(tool)) {
|
|
66
|
+
continue;
|
|
67
|
+
}
|
|
68
|
+
throw new TaskError(`${name}: graders[${index}] (${grader.type}) names "${tool}", which this task never grants. ` +
|
|
69
|
+
`Available with capabilities [${capabilities.join(", ")}]: ${available.join(", ")}. ` +
|
|
70
|
+
`Grant the capability or drop the grader.`);
|
|
71
|
+
}
|
|
72
|
+
}
|
|
41
73
|
export function parseTask(value, directory, fixtureDirectory) {
|
|
42
74
|
if (!isRecord(value)) {
|
|
43
75
|
throw new TaskError(`${directory}: task.json must contain a JSON object.`);
|
|
@@ -57,10 +89,12 @@ export function parseTask(value, directory, fixtureDirectory) {
|
|
|
57
89
|
}
|
|
58
90
|
return entry;
|
|
59
91
|
});
|
|
92
|
+
const capabilities = readCapabilities(value.capabilities, name);
|
|
93
|
+
assertGradersCanFire(name, capabilities, graders);
|
|
60
94
|
return {
|
|
61
95
|
name,
|
|
62
96
|
prompt: value.prompt,
|
|
63
|
-
capabilities
|
|
97
|
+
capabilities,
|
|
64
98
|
attempts: readPositiveInteger(value.attempts, DEFAULT_ATTEMPTS, MAX_ATTEMPTS, "attempts", name),
|
|
65
99
|
maxTurns: readPositiveInteger(value.maxTurns, DEFAULT_MAX_TURNS, 100, "maxTurns", name),
|
|
66
100
|
graders,
|
package/dist/evals/task.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"task.js","sourceRoot":"","sources":["../../src/evals/task.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,kBAAkB,CAAC;AAC3D,OAAO,IAAI,MAAM,WAAW,CAAC;
|
|
1
|
+
{"version":3,"file":"task.js","sourceRoot":"","sources":["../../src/evals/task.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,kBAAkB,CAAC;AAC3D,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AAE1C,MAAM,OAAO,SAAU,SAAQ,KAAK;IAClC,YAAY,OAAe,EAAE,OAA6B;QACxD,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,WAAW,CAAC;QACxB,IAAI,OAAO,EAAE,KAAK,KAAK,SAAS,EAAE,CAAC;YACjC,IAAI,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC;QAC7B,CAAC;IACH,CAAC;CACF;AAsBD,MAAM,gBAAgB,GAAG,CAAC,CAAC;AAC3B,MAAM,iBAAiB,GAAG,EAAE,CAAC;AAC7B,MAAM,YAAY,GAAG,EAAE,CAAC;AAExB,SAAS,QAAQ,CAAC,KAAc;IAC9B,OAAO,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC9E,CAAC;AAED,SAAS,gBAAgB,CAAC,KAAc,EAAE,IAAY;IACpD,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;QACxB,OAAO,EAAE,CAAC;IACZ,CAAC;IACD,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC;QAC1B,MAAM,IAAI,SAAS,CAAC,GAAG,IAAI,kCAAkC,CAAC,CAAC;IACjE,CAAC;IACD,OAAO,KAAK,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE;QACzB,IAAI,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,OAAO,EAAE,CAAC;YAC5C,MAAM,IAAI,SAAS,CACjB,GAAG,IAAI,wBAAwB,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,iCAAiC,CACtF,CAAC;QACJ,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC,CAAC,CAAC;AACL,CAAC;AAED,SAAS,mBAAmB,CAC1B,KAAc,EACd,QAAgB,EAChB,OAAe,EACf,KAAa,EACb,IAAY;IAEZ,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;QACxB,OAAO,QAAQ,CAAC;IAClB,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,KAAK,CAAC,IAAK,KAAgB,GAAG,CAAC,IAAK,KAAgB,GAAG,OAAO,EAAE,CAAC;QACzF,MAAM,IAAI,SAAS,CACjB,GAAG,IAAI,KAAK,KAAK,iCAAiC,OAAO,GAAG,CAC7D,CAAC;IACJ,CAAC;IACD,OAAO,KAAe,CAAC;AACzB,CAAC;AAED,yEAAyE;AACzE,MAAM,iBAAiB,GAAG,IAAI,GAAG,CAAC;IAChC,WAAW;IACX,eAAe;IACf,gBAAgB;IAChB,mBAAmB;CACpB,CAAC,CAAC;AAEH;;;;;;;;GAQG;AACH,SAAS,oBAAoB,CAC3B,IAAY,EACZ,YAAmC,EACnC,OAA8B;IAE9B,MAAM,SAAS,GAAG,YAAY,CAAC,YAAY,CAAC,CAAC;IAC7C,KAAK,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,IAAI,OAAO,CAAC,OAAO,EAAE,EAAE,CAAC;QAChD,IAAI,CAAC,iBAAiB,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC;YACxC,SAAS;QACX,CAAC;QACD,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC;QACzB,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,SAAS,CAAC,QAAQ,CAAC,IAAI,CAAC,EAAE,CAAC;YACzD,SAAS;QACX,CAAC;QACD,MAAM,IAAI,SAAS,CACjB,GAAG,IAAI,aAAa,KAAK,MAAM,MAAM,CAAC,IAAI,YAAY,IAAI,mCAAmC;YAC3F,gCAAgC,YAAY,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI;YACrF,0CAA0C,CAC7C,CAAC;IACJ,CAAC;AACH,CAAC;AAED,MAAM,UAAU,SAAS,CACvB,KAAc,EACd,SAAiB,EACjB,gBAAwB;IAExB,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,EAAE,CAAC;QACrB,MAAM,IAAI,SAAS,CAAC,GAAG,SAAS,yCAAyC,CAAC,CAAC;IAC7E,CAAC;IACD,MAAM,IAAI,GACR,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC;QACrD,CAAC,CAAC,KAAK,CAAC,IAAI;QACZ,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAC;IAC/B,IAAI,OAAO,KAAK,CAAC,MAAM,KAAK,QAAQ,IAAI,KAAK,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACzE,MAAM,IAAI,SAAS,CAAC,GAAG,IAAI,sCAAsC,CAAC,CAAC;IACrE,CAAC;IACD,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAChE,MAAM,IAAI,SAAS,CAAC,GAAG,IAAI,oCAAoC,CAAC,CAAC;IACnE,CAAC;IACD,MAAM,OAAO,GAAG,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,EAAE;QACjD,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;YACvD,MAAM,IAAI,SAAS,CAAC,GAAG,IAAI,aAAa,KAAK,iBAAiB,CAAC,CAAC;QAClE,CAAC;QACD,OAAO,KAAmB,CAAC;IAC7B,CAAC,CAAC,CAAC;IAEH,MAAM,YAAY,GAAG,gBAAgB,CAAC,KAAK,CAAC,YAAY,EAAE,IAAI,CAAC,CAAC;IAChE,oBAAoB,CAAC,IAAI,EAAE,YAAY,EAAE,OAAO,CAAC,CAAC;IAElD,OAAO;QACL,IAAI;QACJ,MAAM,EAAE,KAAK,CAAC,MAAM;QACpB,YAAY;QACZ,QAAQ,EAAE,mBAAmB,CAC3B,KAAK,CAAC,QAAQ,EACd,gBAAgB,EAChB,YAAY,EACZ,UAAU,EACV,IAAI,CACL;QACD,QAAQ,EAAE,mBAAmB,CAC3B,KAAK,CAAC,QAAQ,EACd,iBAAiB,EACjB,GAAG,EACH,UAAU,EACV,IAAI,CACL;QACD,OAAO;QACP,gBAAgB;QAChB,SAAS;KACV,CAAC;AACJ,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,SAAiB;IAC9C,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,WAAW,CAAC,CAAC;IAC/C,IAAI,QAAgB,CAAC;IACrB,IAAI,CAAC;QACH,QAAQ,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAC1C,CAAC;IAAC,OAAO,KAAc,EAAE,CAAC;QACxB,MAAM,IAAI,SAAS,CAAC,kBAAkB,IAAI,GAAG,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC;IACnE,CAAC;IAED,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;IAChC,CAAC;IAAC,OAAO,KAAc,EAAE,CAAC;QACxB,MAAM,IAAI,SAAS,CAAC,GAAG,IAAI,qBAAqB,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC;IACtE,CAAC;IACD,OAAO,SAAS,CAAC,MAAM,EAAE,SAAS,EAAE,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC,CAAC;AACvE,CAAC;AAED,6DAA6D;AAC7D,MAAM,CAAC,KAAK,UAAU,SAAS,CAAC,IAAY;IAC1C,IAAI,OAAO,CAAC;IACZ,IAAI,CAAC;QACH,OAAO,GAAG,MAAM,OAAO,CAAC,IAAI,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAC;IACzD,CAAC;IAAC,OAAO,KAAc,EAAE,CAAC;QACxB,MAAM,IAAI,SAAS,CAAC,qCAAqC,IAAI,GAAG,EAAE;YAChE,KAAK,EAAE,KAAK;SACb,CAAC,CAAC;IACL,CAAC;IAED,MAAM,KAAK,GAAW,EAAE,CAAC;IACzB,KAAK,MAAM,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;QACzE,IAAI,CAAC,KAAK,CAAC,WAAW,EAAE,EAAE,CAAC;YACzB,SAAS;QACX,CAAC;QACD,MAAM,SAAS,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,CAAC,CAAC;QAC9C,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,WAAW,CAAC,CAAC,CAAC;QAChD,CAAC;QAAC,MAAM,CAAC;YACP,SAAS;QACX,CAAC;QACD,KAAK,CAAC,IAAI,CAAC,MAAM,QAAQ,CAAC,SAAS,CAAC,CAAC,CAAC;IACxC,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC"}
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
import type { Tool } from "../tools/tool.js";
|
|
2
|
+
import type { Capability } from "./task.js";
|
|
3
|
+
export declare function toolNamesFor(capabilities: readonly Capability[]): readonly string[];
|
|
4
|
+
export declare function toolsFor(capabilities: readonly Capability[], cwd: string): readonly Tool[];
|
|
5
|
+
//# sourceMappingURL=tools.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"tools.d.ts","sourceRoot":"","sources":["../../src/evals/tools.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,kBAAkB,CAAC;AAO7C,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAgB5C,wBAAgB,YAAY,CAC1B,YAAY,EAAE,SAAS,UAAU,EAAE,GAClC,SAAS,MAAM,EAAE,CAKnB;AAED,wBAAgB,QAAQ,CACtB,YAAY,EAAE,SAAS,UAAU,EAAE,EACnC,GAAG,EAAE,MAAM,GACV,SAAS,IAAI,EAAE,CAajB"}
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
import { ListFilesTool } from "../tools/list-files.js";
|
|
2
|
+
import { ReadFileTool } from "../tools/read-file.js";
|
|
3
|
+
import { SearchTextTool } from "../tools/search-text.js";
|
|
4
|
+
import { WriteFileTool } from "../tools/write-file.js";
|
|
5
|
+
import { EditFileTool } from "../tools/edit-file.js";
|
|
6
|
+
import { BashTool } from "../tools/bash.js";
|
|
7
|
+
/**
|
|
8
|
+
* What each capability puts in the model's hands.
|
|
9
|
+
*
|
|
10
|
+
* This lives apart from the runner because task loading needs the same answer:
|
|
11
|
+
* a grader that names a tool the task never grants is either an assertion that
|
|
12
|
+
* cannot fail or one that cannot pass, and both are worth refusing before the
|
|
13
|
+
* run starts. Keeping one list means the two cannot drift.
|
|
14
|
+
*/
|
|
15
|
+
const ALWAYS = ["list_files", "search_text", "read_file"];
|
|
16
|
+
const BY_CAPABILITY = {
|
|
17
|
+
writes: ["write_file", "edit_file"],
|
|
18
|
+
shell: ["bash"],
|
|
19
|
+
};
|
|
20
|
+
export function toolNamesFor(capabilities) {
|
|
21
|
+
return [
|
|
22
|
+
...ALWAYS,
|
|
23
|
+
...capabilities.flatMap((capability) => BY_CAPABILITY[capability]),
|
|
24
|
+
];
|
|
25
|
+
}
|
|
26
|
+
export function toolsFor(capabilities, cwd) {
|
|
27
|
+
const tools = [
|
|
28
|
+
new ListFilesTool(),
|
|
29
|
+
new SearchTextTool(),
|
|
30
|
+
new ReadFileTool(),
|
|
31
|
+
];
|
|
32
|
+
if (capabilities.includes("writes")) {
|
|
33
|
+
tools.push(new WriteFileTool(), new EditFileTool());
|
|
34
|
+
}
|
|
35
|
+
if (capabilities.includes("shell")) {
|
|
36
|
+
tools.push(new BashTool({ cwd }));
|
|
37
|
+
}
|
|
38
|
+
return tools;
|
|
39
|
+
}
|
|
40
|
+
//# sourceMappingURL=tools.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"tools.js","sourceRoot":"","sources":["../../src/evals/tools.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAC;AACvD,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AACrD,OAAO,EAAE,cAAc,EAAE,MAAM,yBAAyB,CAAC;AACzD,OAAO,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAC;AACvD,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AACrD,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAG5C;;;;;;;GAOG;AACH,MAAM,MAAM,GAAG,CAAC,YAAY,EAAE,aAAa,EAAE,WAAW,CAAU,CAAC;AACnE,MAAM,aAAa,GAA0C;IAC3D,MAAM,EAAE,CAAC,YAAY,EAAE,WAAW,CAAC;IACnC,KAAK,EAAE,CAAC,MAAM,CAAC;CAChB,CAAC;AAEF,MAAM,UAAU,YAAY,CAC1B,YAAmC;IAEnC,OAAO;QACL,GAAG,MAAM;QACT,GAAG,YAAY,CAAC,OAAO,CAAC,CAAC,UAAU,EAAE,EAAE,CAAC,aAAa,CAAC,UAAU,CAAC,CAAC;KACnE,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,QAAQ,CACtB,YAAmC,EACnC,GAAW;IAEX,MAAM,KAAK,GAAW;QACpB,IAAI,aAAa,EAAE;QACnB,IAAI,cAAc,EAAE;QACpB,IAAI,YAAY,EAAE;KACnB,CAAC;IACF,IAAI,YAAY,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE,CAAC;QACpC,KAAK,CAAC,IAAI,CAAC,IAAI,aAAa,EAAE,EAAE,IAAI,YAAY,EAAE,CAAC,CAAC;IACtD,CAAC;IACD,IAAI,YAAY,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;QACnC,KAAK,CAAC,IAAI,CAAC,IAAI,QAAQ,CAAC,EAAE,GAAG,EAAE,CAAC,CAAC,CAAC;IACpC,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"usage.d.ts","sourceRoot":"","sources":["../../src/providers/usage.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,WAAW,CAAC;AAQvC;;;;;GAKG;AACH,wBAAgB,wBAAwB,CAAC,KAAK,EAAE,OAAO,GAAG,KAAK,GAAG,SAAS,
|
|
1
|
+
{"version":3,"file":"usage.d.ts","sourceRoot":"","sources":["../../src/providers/usage.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,WAAW,CAAC;AAQvC;;;;;GAKG;AACH,wBAAgB,wBAAwB,CAAC,KAAK,EAAE,OAAO,GAAG,KAAK,GAAG,SAAS,CAiC1E;AAED,gFAAgF;AAChF,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,OAAO,GAAG,KAAK,GAAG,SAAS,CA4BpE;AAED,MAAM,WAAW,UAAU;IACzB,QAAQ,CAAC,KAAK,EAAE,KAAK,CAAC;IACtB;;;OAGG;IACH,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC;CAC5B;AAED,wBAAgB,QAAQ,CACtB,KAAK,EAAE,UAAU,GAAG,SAAS,EAC7B,IAAI,EAAE,KAAK,GAAG,SAAS,GACtB,UAAU,CAuBZ;AAED,0CAA0C;AAC1C,wBAAgB,YAAY,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,CAQnD"}
|
package/dist/providers/usage.js
CHANGED
|
@@ -23,7 +23,9 @@ export function fromChatCompletionsUsage(value) {
|
|
|
23
23
|
const promptDetails = record.prompt_tokens_details;
|
|
24
24
|
const reasoningTokens = positive(details?.reasoning_tokens);
|
|
25
25
|
const cachedInputTokens = positive(promptDetails?.cached_tokens) ??
|
|
26
|
-
// DeepSeek reports cache hits at the top level
|
|
26
|
+
// DeepSeek also reports cache hits at the top level. It populates both,
|
|
27
|
+
// so this branch is a fallback for a Provider that reports only the flat
|
|
28
|
+
// form, not the path DeepSeek actually takes.
|
|
27
29
|
positive(record.prompt_cache_hit_tokens);
|
|
28
30
|
return {
|
|
29
31
|
inputTokens: inputTokens ?? 0,
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"usage.js","sourceRoot":"","sources":["../../src/providers/usage.ts"],"names":[],"mappings":"AAEA,SAAS,QAAQ,CAAC,KAAc;IAC9B,OAAO,OAAO,KAAK,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC;QACtE,CAAC,CAAC,KAAK;QACP,CAAC,CAAC,SAAS,CAAC;AAChB,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,wBAAwB,CAAC,KAAc;IACrD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI,EAAE,CAAC;QAChD,OAAO,SAAS,CAAC;IACnB,CAAC;IACD,MAAM,MAAM,GAAG,KAAgC,CAAC;IAChD,MAAM,WAAW,GAAG,QAAQ,CAAC,MAAM,CAAC,aAAa,CAAC,CAAC;IACnD,MAAM,YAAY,GAAG,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,CAAC;IACxD,IAAI,WAAW,KAAK,SAAS,IAAI,YAAY,KAAK,SAAS,EAAE,CAAC;QAC5D,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,MAAM,OAAO,GAAG,MAAM,CAAC,yBAEV,CAAC;IACd,MAAM,aAAa,GAAG,MAAM,CAAC,qBAEhB,CAAC;IACd,MAAM,eAAe,GAAG,QAAQ,CAAC,OAAO,EAAE,gBAAgB,CAAC,CAAC;IAC5D,MAAM,iBAAiB,GACrB,QAAQ,CAAC,aAAa,EAAE,aAAa,CAAC;QACtC,
|
|
1
|
+
{"version":3,"file":"usage.js","sourceRoot":"","sources":["../../src/providers/usage.ts"],"names":[],"mappings":"AAEA,SAAS,QAAQ,CAAC,KAAc;IAC9B,OAAO,OAAO,KAAK,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC;QACtE,CAAC,CAAC,KAAK;QACP,CAAC,CAAC,SAAS,CAAC;AAChB,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,wBAAwB,CAAC,KAAc;IACrD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI,EAAE,CAAC;QAChD,OAAO,SAAS,CAAC;IACnB,CAAC;IACD,MAAM,MAAM,GAAG,KAAgC,CAAC;IAChD,MAAM,WAAW,GAAG,QAAQ,CAAC,MAAM,CAAC,aAAa,CAAC,CAAC;IACnD,MAAM,YAAY,GAAG,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,CAAC;IACxD,IAAI,WAAW,KAAK,SAAS,IAAI,YAAY,KAAK,SAAS,EAAE,CAAC;QAC5D,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,MAAM,OAAO,GAAG,MAAM,CAAC,yBAEV,CAAC;IACd,MAAM,aAAa,GAAG,MAAM,CAAC,qBAEhB,CAAC;IACd,MAAM,eAAe,GAAG,QAAQ,CAAC,OAAO,EAAE,gBAAgB,CAAC,CAAC;IAC5D,MAAM,iBAAiB,GACrB,QAAQ,CAAC,aAAa,EAAE,aAAa,CAAC;QACtC,wEAAwE;QACxE,yEAAyE;QACzE,8CAA8C;QAC9C,QAAQ,CAAC,MAAM,CAAC,uBAAuB,CAAC,CAAC;IAE3C,OAAO;QACL,WAAW,EAAE,WAAW,IAAI,CAAC;QAC7B,YAAY,EAAE,YAAY,IAAI,CAAC;QAC/B,WAAW,EACT,QAAQ,CAAC,MAAM,CAAC,YAAY,CAAC,IAAI,CAAC,WAAW,IAAI,CAAC,CAAC,GAAG,CAAC,YAAY,IAAI,CAAC,CAAC;QAC3E,GAAG,CAAC,iBAAiB,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,iBAAiB,EAAE,CAAC;QACjE,GAAG,CAAC,eAAe,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,eAAe,EAAE,CAAC;KAC9D,CAAC;AACJ,CAAC;AAED,gFAAgF;AAChF,MAAM,UAAU,kBAAkB,CAAC,KAAc;IAC/C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI,EAAE,CAAC;QAChD,OAAO,SAAS,CAAC;IACnB,CAAC;IACD,MAAM,MAAM,GAAG,KAAgC,CAAC;IAChD,MAAM,WAAW,GAAG,QAAQ,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;IAClD,MAAM,YAAY,GAAG,QAAQ,CAAC,MAAM,CAAC,aAAa,CAAC,CAAC;IACpD,IAAI,WAAW,KAAK,SAAS,IAAI,YAAY,KAAK,SAAS,EAAE,CAAC;QAC5D,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,MAAM,aAAa,GAAG,MAAM,CAAC,qBAEhB,CAAC;IACd,MAAM,YAAY,GAAG,MAAM,CAAC,oBAEf,CAAC;IACd,MAAM,eAAe,GAAG,QAAQ,CAAC,aAAa,EAAE,gBAAgB,CAAC,CAAC;IAClE,MAAM,iBAAiB,GAAG,QAAQ,CAAC,YAAY,EAAE,aAAa,CAAC,CAAC;IAEhE,OAAO;QACL,WAAW,EAAE,WAAW,IAAI,CAAC;QAC7B,YAAY,EAAE,YAAY,IAAI,CAAC;QAC/B,WAAW,EACT,QAAQ,CAAC,MAAM,CAAC,YAAY,CAAC,IAAI,CAAC,WAAW,IAAI,CAAC,CAAC,GAAG,CAAC,YAAY,IAAI,CAAC,CAAC;QAC3E,GAAG,CAAC,iBAAiB,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,iBAAiB,EAAE,CAAC;QACjE,GAAG,CAAC,eAAe,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,eAAe,EAAE,CAAC;KAC9D,CAAC;AACJ,CAAC;AAWD,MAAM,UAAU,QAAQ,CACtB,KAA6B,EAC7B,IAAuB;IAEvB,MAAM,IAAI,GAAG,KAAK,IAAI;QACpB,KAAK,EAAE,EAAE,WAAW,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,WAAW,EAAE,CAAC,EAAE;QAC1D,QAAQ,EAAE,IAAI;KACf,CAAC;IACF,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;QACvB,OAAO,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,QAAQ,EAAE,KAAK,EAAE,CAAC;IAChD,CAAC;IAED,MAAM,MAAM,GACV,CAAC,IAAI,CAAC,KAAK,CAAC,iBAAiB,IAAI,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,iBAAiB,IAAI,CAAC,CAAC,CAAC;IACtE,MAAM,SAAS,GACb,CAAC,IAAI,CAAC,KAAK,CAAC,eAAe,IAAI,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,eAAe,IAAI,CAAC,CAAC,CAAC;IAClE,OAAO;QACL,KAAK,EAAE;YACL,WAAW,EAAE,IAAI,CAAC,KAAK,CAAC,WAAW,GAAG,IAAI,CAAC,WAAW;YACtD,YAAY,EAAE,IAAI,CAAC,KAAK,CAAC,YAAY,GAAG,IAAI,CAAC,YAAY;YACzD,WAAW,EAAE,IAAI,CAAC,KAAK,CAAC,WAAW,GAAG,IAAI,CAAC,WAAW;YACtD,GAAG,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,iBAAiB,EAAE,MAAM,EAAE,CAAC;YACtD,GAAG,CAAC,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,eAAe,EAAE,SAAS,EAAE,CAAC;SAC3D;QACD,QAAQ,EAAE,IAAI,CAAC,QAAQ;KACxB,CAAC;AACJ,CAAC;AAED,0CAA0C;AAC1C,MAAM,UAAU,YAAY,CAAC,MAAc;IACzC,IAAI,MAAM,GAAG,IAAI,EAAE,CAAC;QAClB,OAAO,MAAM,CAAC,MAAM,CAAC,CAAC;IACxB,CAAC;IACD,IAAI,MAAM,GAAG,SAAS,EAAE,CAAC;QACvB,OAAO,GAAG,CAAC,MAAM,GAAG,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC;IAC1C,CAAC;IACD,OAAO,GAAG,CAAC,MAAM,GAAG,SAAS,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC;AAC/C,CAAC"}
|