agent-hitch 0.2.10 → 0.2.12

package/README.md

@@ -31,19 +31,19 @@ compare models, or fix the model to compare harnesses.
 - [x] **Run and recover.** Share resource budgets, run trials in parallel, and preserve completed work through supported recovery paths.
 - [x] **Collect the evidence.** Keep scores, trajectories, verifier evidence, and provenance in one result bundle.
 
-> **Status:** pre-alpha. Capabilities below describe current `dev`; setup and
-> support limits are covered in the [User Guide](https://rsigear.xyz/docs/hitch/).
+> **Status:** pre-alpha. Setup and support limits are covered in the
+> [User Guide](https://rsigear.xyz/docs/hitch/).
 
 ## Quick Start
 
 You need **Node.js 22+**, Git, and model access. Local runs do not require Docker.
 
-**1. Install Hitch and authenticate the harness.** This example pins Codex to
-an explicit version; complete its login flow before continuing.
+**1. Install Hitch and Codex, then authenticate.** Complete the Codex login
+flow before continuing; Hitch will use this installed executable.
 
 ```bash
-npm install --global agent-hitch@0.2.10
-npx --yes @openai/codex@0.92.0 login
+npm install --global agent-hitch @openai/codex
+codex login
 ```
 
 **2. Run from a clean Git repository.** `git status --short` must show no changes
@@ -53,7 +53,7 @@ for worktree mode. To include uncommitted files, choose
 ```bash
 git status --short
 hitch run \
-  --harness codex@version:0.92.0 \
+  --harness codex@installed \
   --workspace-mode worktree \
   --prompt "Summarize this repository and its test commands. Do not modify files." \
   --timeout 5m \
@@ -136,7 +136,7 @@ and resource admission; it is not a speed or cost comparison against native Harb
 
 ## Local and remote model inference
 
-Managed inference is included in current `dev`. Import a complete Hugging Face safetensors checkpoint once, then use the same
+Import a complete Hugging Face safetensors checkpoint once, then use the same
 run and eval commands with a `local/<name>` model. Hitch chooses a pinned
 CPU or CUDA preview runtime, starts the daemon and SGLang service when needed, and
 records the immutable model/runtime/inference identities automatically.
@@ -211,9 +211,10 @@ for benchmark, resource, and portability contracts.
 | OpenCode | ✓ | ✓ | — |
 | DeepSeek Harness | ✓ | ✓ | ✓ |
 
-Use `codex@installed` for a local executable, `codex@version:0.92.0` for an exact
-package, or `codex@commit:COMMIT` with a real upstream commit. Evaluations require
-a portable version or commit reference. See
+Use `codex@installed` for a local executable, `codex@version:VERSION` for an exact
+package, or `codex@commit:COMMIT` with a real upstream commit. Replace `VERSION`
+with the exact published version you want to run. Evaluations require a portable
+version or commit reference. See
 [versions and workspaces](docs/guide/en/versions-and-workspaces.md) for reference
 selection, artifact preparation, and workspace modes.
 

package/README.zh-CN.md

@@ -29,19 +29,19 @@
 - [x] **并行与恢复。** 共享资源预算、并行执行 Trial，在支持的恢复路径中保留已完成工作。
 - [x] **收齐结果证据。** 分数、轨迹、Verifier 证据与来源信息归入同一份结果包。
 
-> **状态：** pre-alpha。以下能力对应当前 `dev`；配置与支持范围见
+> **状态：** pre-alpha。配置与支持范围见
 > [User Guide](https://rsigear.xyz/docs/hitch/zh/)。
 
 ## Quick Start：快速开始
 
 需要 **Node.js 22+**、Git 和模型访问权限。本地运行不需要 Docker。
 
-**1. 安装 Hitch，并完成 Harness 认证。** 示例固定了一个 Codex 版本，
-继续前先完成登录流程。
+**1. 安装 Hitch 和 Codex，并完成认证。** 继续前先完成 Codex 登录流程；
+Hitch 会使用这个已安装的程序。
 
 ```bash
-npm install --global agent-hitch@0.2.10
-npx --yes @openai/codex@0.92.0 login
+npm install --global agent-hitch @openai/codex
+codex login
 ```
 
 **2. 从干净的 Git 仓库运行。** Worktree 模式要求 `git status --short` 没有
@@ -51,7 +51,7 @@ npx --yes @openai/codex@0.92.0 login
 ```bash
 git status --short
 hitch run \
-  --harness codex@version:0.92.0 \
+  --harness codex@installed \
   --workspace-mode worktree \
   --prompt "总结这个仓库及其测试命令，不修改文件。" \
   --timeout 5m \
@@ -131,7 +131,7 @@ Hitch 采用 **Harbor 的任务定义格式**，接入已有 Benchmark 和自定
 
 ## 本地与远程模型推理
 
-当前 `dev` 已包含托管推理。完整的 Hugging Face safetensors checkpoint 只需导入一次，此后继续使用原有
+完整的 Hugging Face safetensors checkpoint 只需导入一次，此后继续使用原有
 run/eval 命令，并把模型写成 `local/<name>`。Hitch 会自动选择固定摘要的 CPU 或
 CUDA preview runtime、按需启动 daemon 和 SGLang 服务，并记录不可变的模型、runtime
 和 inference 身份。
@@ -202,8 +202,9 @@ hitch eval doctor
 | OpenCode | ✓ | ✓ | — |
 | DeepSeek Harness | ✓ | ✓ | ✓ |
 
-使用 `codex@installed` 运行本机程序，使用 `codex@version:0.92.0` 选择确定的
-软件包版本，或把真实上游提交填入 `codex@commit:COMMIT`。评测要求可移植的
+使用 `codex@installed` 运行本机程序，使用 `codex@version:VERSION` 选择确定的
+软件包版本，或把真实上游提交填入 `codex@commit:COMMIT`。将 `VERSION` 替换为
+要运行的精确已发布版本。评测要求可移植的
 版本或提交引用。引用选择、制品准备和工作区模式见
 [固定版本与隔离工作区](docs/guide/zh-CN/versions-and-workspaces.md)。
 

package/dist/src/cli/commands/capabilities.js

@@ -8,6 +8,7 @@ export function capabilitiesCommand(args) {
         trajectory_analysis: "1",
         trajectory_events_page: "1",
         verifier_evidence: "1",
+        verifier_diagnostic_pages: "1",
         training_external_binding: "1",
         exact_policy_tokens: "1",
         training_policy_fencing: "1",

package/dist/src/cli/commands/capabilities.js.map

@@ -1 +1 @@
-{"version":3,"file":"capabilities.js","sourceRoot":"","sources":["../../../../src/cli/commands/capabilities.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAC3D,OAAO,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,iBAAiB,CAAC;AAEzD,MAAM,UAAU,mBAAmB,CAAC,IAAc;IAChD,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtC,YAAY,CAAC,IAAI,CAAC,CAAC;IACnB,MAAM,YAAY,GAAG;QACnB,cAAc,EAAE,cAAc;QAC9B,mBAAmB,EAAE,GAAG;QACxB,sBAAsB,EAAE,GAAG;QAC3B,iBAAiB,EAAE,GAAG;QACtB,yBAAyB,EAAE,GAAG;QAC9B,mBAAmB,EAAE,GAAG;QACxB,uBAAuB,EAAE,GAAG;QAC5B,kBAAkB,EAAE,CAAC,eAAe,CAAC;QACrC,uBAAuB,EAAE,SAAS;QAClC,kBAAkB,EAAE,GAAG;QACvB,gBAAgB,EAAE,GAAG;QACrB,kBAAkB,EAAE,GAAG;QACvB,oBAAoB,EAAE,GAAG;QACzB,8BAA8B,EAAE,GAAG;QACnC,2BAA2B,EAAE,GAAG;QAChC,4BAA4B,EAAE,GAAG;KACzB,CAAC;IACX,IAAI,IAAI,EAAE,CAAC;QACT,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC;QAC1D,OAAO;IACT,CAAC;IACD,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,YAAY,CAAC;QAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,GAAG,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC;AAC9I,CAAC"}
+{"version":3,"file":"capabilities.js","sourceRoot":"","sources":["../../../../src/cli/commands/capabilities.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAC3D,OAAO,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,iBAAiB,CAAC;AAEzD,MAAM,UAAU,mBAAmB,CAAC,IAAc;IAChD,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtC,YAAY,CAAC,IAAI,CAAC,CAAC;IACnB,MAAM,YAAY,GAAG;QACnB,cAAc,EAAE,cAAc;QAC9B,mBAAmB,EAAE,GAAG;QACxB,sBAAsB,EAAE,GAAG;QAC3B,iBAAiB,EAAE,GAAG;QACtB,yBAAyB,EAAE,GAAG;QAC9B,yBAAyB,EAAE,GAAG;QAC9B,mBAAmB,EAAE,GAAG;QACxB,uBAAuB,EAAE,GAAG;QAC5B,kBAAkB,EAAE,CAAC,eAAe,CAAC;QACrC,uBAAuB,EAAE,SAAS;QAClC,kBAAkB,EAAE,GAAG;QACvB,gBAAgB,EAAE,GAAG;QACrB,kBAAkB,EAAE,GAAG;QACvB,oBAAoB,EAAE,GAAG;QACzB,8BAA8B,EAAE,GAAG;QACnC,2BAA2B,EAAE,GAAG;QAChC,4BAA4B,EAAE,GAAG;KACzB,CAAC;IACX,IAAI,IAAI,EAAE,CAAC;QACT,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC;QAC1D,OAAO;IACT,CAAC;IACD,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,YAAY,CAAC;QAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,GAAG,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC;AAC9I,CAAC"}

package/dist/src/cli/commands/verifier.js

@@ -1,10 +1,16 @@
 import { invalidInput } from "../../foundation/index.js";
-import { loadVerifierEvidence } from "../../runs/index.js";
-import { assertNoArgs, takeFlag } from "../arguments.js";
+import { repairVerifierDiagnostics } from "../../evals/index.js";
+import { loadVerifierDiagnosticPage, loadVerifierEvidence, MAX_VERIFIER_DIAGNOSTIC_PAGE_BYTES } from "../../runs/index.js";
+import { assertNoArgs, takeFlag, takeOption } from "../arguments.js";
+const ARTIFACT_NAMES = new Set(["ctrf.json", "test-stdout.txt", "test-stderr.txt", "stdout.txt", "stderr.txt"]);
 export async function verifierCommand(args, root) {
     const action = args.shift();
+    if (action === "artifact")
+        return verifierArtifactCommand(args, root);
+    if (action === "repair")
+        return verifierRepairCommand(args, root);
     if (action !== "inspect")
-        throw invalidInput("verifier requires inspect");
+        throw invalidInput("verifier requires inspect, artifact, or repair");
     const json = takeFlag(args, "--json");
     const runId = args.shift();
     if (!runId || !/^run_[a-f0-9]{32}$/.test(runId)) {
@@ -33,4 +39,49 @@ export async function verifierCommand(args, root) {
     for (const issue of evidence.verifier.issues ?? [])
         process.stdout.write(`  issue: ${issue}\n`);
 }
+async function verifierRepairCommand(args, root) {
+    const json = takeFlag(args, "--json");
+    const source = takeOption(args, "--source");
+    const runId = args.shift();
+    if (!runId || !/^run_[a-f0-9]{32}$/.test(runId)) {
+        throw invalidInput("verifier repair requires a valid run ID");
+    }
+    assertNoArgs(args);
+    const result = await repairVerifierDiagnostics({ root, runId, ...(source === undefined ? {} : { source }) });
+    if (json)
+        process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
+    else
+        process.stdout.write(`${runId}: verifier diagnostics ${result.status.replace("_", " ")}\n`);
+}
+async function verifierArtifactCommand(args, root) {
+    const json = takeFlag(args, "--json");
+    const offsetValue = takeOption(args, "--offset");
+    const limitValue = takeOption(args, "--limit");
+    const expectedSha256 = takeOption(args, "--sha256");
+    const runId = args.shift();
+    const name = args.shift();
+    if (!runId || !/^run_[a-f0-9]{32}$/.test(runId) || !name || !ARTIFACT_NAMES.has(name)) {
+        throw invalidInput("verifier artifact requires a valid run ID and artifact name");
+    }
+    const offset = offsetValue === undefined ? 0 : Number(offsetValue);
+    const maxBytes = limitValue === undefined ? MAX_VERIFIER_DIAGNOSTIC_PAGE_BYTES : Number(limitValue);
+    if (!Number.isSafeInteger(offset) || offset < 0)
+        throw invalidInput("--offset must be a non-negative integer");
+    if (!Number.isSafeInteger(maxBytes) || maxBytes < 4 || maxBytes > MAX_VERIFIER_DIAGNOSTIC_PAGE_BYTES) {
+        throw invalidInput(`--limit must be an integer from 4 to ${MAX_VERIFIER_DIAGNOSTIC_PAGE_BYTES}`);
+    }
+    if (expectedSha256 !== undefined && !/^sha256:[0-9a-f]{64}$/.test(expectedSha256)) {
+        throw invalidInput("--sha256 must be a SHA-256 digest");
+    }
+    assertNoArgs(args);
+    const page = await loadVerifierDiagnosticPage(root, runId, name, {
+        offset,
+        maxBytes,
+        ...(expectedSha256 ? { expectedSha256: expectedSha256 } : {}),
+    });
+    if (json)
+        process.stdout.write(`${JSON.stringify(page, null, 2)}\n`);
+    else
+        process.stdout.write(page.page.text);
+}
 //# sourceMappingURL=verifier.js.map

package/dist/src/cli/commands/verifier.js.map

@@ -1 +1 @@
-{"version":3,"file":"verifier.js","sourceRoot":"","sources":["../../../../src/cli/commands/verifier.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AACzD,OAAO,EAAE,oBAAoB,EAAE,MAAM,qBAAqB,CAAC;AAC3D,OAAO,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,iBAAiB,CAAC;AAEzD,MAAM,CAAC,KAAK,UAAU,eAAe,CAAC,IAAc,EAAE,IAAY;IAChE,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;IAC5B,IAAI,MAAM,KAAK,SAAS;QAAE,MAAM,YAAY,CAAC,2BAA2B,CAAC,CAAC;IAC1E,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtC,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;IAC3B,IAAI,CAAC,KAAK,IAAI,CAAC,oBAAoB,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QAChD,MAAM,YAAY,CAAC,0CAA0C,CAAC,CAAC;IACjE,CAAC;IACD,YAAY,CAAC,IAAI,CAAC,CAAC;IACnB,MAAM,QAAQ,GAAG,MAAM,oBAAoB,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;IACzD,IAAI,IAAI,EAAE,CAAC;QACT,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;QAC/D,OAAO;IACT,CAAC;IACD,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,cAAc,QAAQ,CAAC,QAAQ,CAAC,MAAM,IAAI,CAAC,CAAC;IACzE,IAAI,QAAQ,CAAC,WAAW,EAAE,CAAC;QACzB,MAAM,MAAM,GAAG,QAAQ,CAAC,WAAW,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,YAAY,QAAQ,CAAC,WAAW,CAAC,MAAM,EAAE,CAAC;QAC1G,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,kBAAkB,QAAQ,CAAC,WAAW,CAAC,MAAM,GAAG,MAAM,IAAI,CAAC,CAAC;IACnF,CAAC;IACD,MAAM,WAAW,GAAG,QAAQ,CAAC,QAAQ,CAAC,WAAW,CAAC;IAClD,IAAI,WAAW,EAAE,CAAC;QAChB,MAAM,SAAS,GAAG;YAChB,GAAG,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;YAC/C,GAAG,CAAC,WAAW,CAAC,MAAM,IAAI,EAAE,CAAC;YAC7B,GAAG,CAAC,WAAW,CAAC,MAAM,IAAI,EAAE,CAAC;SAC9B,CAAC;QACF,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,kBAAkB,SAAS,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,iBAAiB,IAAI,CAAC,CAAC;IACzH,CAAC;IACD,KAAK,MAAM,KAAK,IAAI,QAAQ,CAAC,QAAQ,CAAC,MAAM,IAAI,EAAE;QAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,KAAK,IAAI,CAAC,CAAC;AAClG,CAAC"}
+{"version":3,"file":"verifier.js","sourceRoot":"","sources":["../../../../src/cli/commands/verifier.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AACzD,OAAO,EAAE,yBAAyB,EAAE,MAAM,sBAAsB,CAAC;AACjE,OAAO,EAAE,0BAA0B,EAAE,oBAAoB,EAAE,kCAAkC,EAAE,MAAM,qBAAqB,CAAC;AAE3H,OAAO,EAAE,YAAY,EAAE,QAAQ,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AAErE,MAAM,cAAc,GAAG,IAAI,GAAG,CAAC,CAAC,WAAW,EAAE,iBAAiB,EAAE,iBAAiB,EAAE,YAAY,EAAE,YAAY,CAAC,CAAC,CAAC;AAEhH,MAAM,CAAC,KAAK,UAAU,eAAe,CAAC,IAAc,EAAE,IAAY;IAChE,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;IAC5B,IAAI,MAAM,KAAK,UAAU;QAAE,OAAO,uBAAuB,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IACtE,IAAI,MAAM,KAAK,QAAQ;QAAE,OAAO,qBAAqB,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAClE,IAAI,MAAM,KAAK,SAAS;QAAE,MAAM,YAAY,CAAC,gDAAgD,CAAC,CAAC;IAC/F,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtC,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;IAC3B,IAAI,CAAC,KAAK,IAAI,CAAC,oBAAoB,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QAChD,MAAM,YAAY,CAAC,0CAA0C,CAAC,CAAC;IACjE,CAAC;IACD,YAAY,CAAC,IAAI,CAAC,CAAC;IACnB,MAAM,QAAQ,GAAG,MAAM,oBAAoB,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;IACzD,IAAI,IAAI,EAAE,CAAC;QACT,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;QAC/D,OAAO;IACT,CAAC;IACD,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,cAAc,QAAQ,CAAC,QAAQ,CAAC,MAAM,IAAI,CAAC,CAAC;IACzE,IAAI,QAAQ,CAAC,WAAW,EAAE,CAAC;QACzB,MAAM,MAAM,GAAG,QAAQ,CAAC,WAAW,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,YAAY,QAAQ,CAAC,WAAW,CAAC,MAAM,EAAE,CAAC;QAC1G,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,kBAAkB,QAAQ,CAAC,WAAW,CAAC,MAAM,GAAG,MAAM,IAAI,CAAC,CAAC;IACnF,CAAC;IACD,MAAM,WAAW,GAAG,QAAQ,CAAC,QAAQ,CAAC,WAAW,CAAC;IAClD,IAAI,WAAW,EAAE,CAAC;QAChB,MAAM,SAAS,GAAG;YAChB,GAAG,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;YAC/C,GAAG,CAAC,WAAW,CAAC,MAAM,IAAI,EAAE,CAAC;YAC7B,GAAG,CAAC,WAAW,CAAC,MAAM,IAAI,EAAE,CAAC;SAC9B,CAAC;QACF,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,kBAAkB,SAAS,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,iBAAiB,IAAI,CAAC,CAAC;IACzH,CAAC;IACD,KAAK,MAAM,KAAK,IAAI,QAAQ,CAAC,QAAQ,CAAC,MAAM,IAAI,EAAE;QAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,KAAK,IAAI,CAAC,CAAC;AAClG,CAAC;AAED,KAAK,UAAU,qBAAqB,CAAC,IAAc,EAAE,IAAY;IAC/D,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtC,MAAM,MAAM,GAAG,UAAU,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;IAC5C,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;IAC3B,IAAI,CAAC,KAAK,IAAI,CAAC,oBAAoB,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QAChD,MAAM,YAAY,CAAC,yCAAyC,CAAC,CAAC;IAChE,CAAC;IACD,YAAY,CAAC,IAAI,CAAC,CAAC;IACnB,MAAM,MAAM,GAAG,MAAM,yBAAyB,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,CAAC,CAAC;IAC7G,IAAI,IAAI;QAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;;QAClE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,0BAA0B,MAAM,CAAC,MAAM,CAAC,OAAO,CAAC,GAAG,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;AACnG,CAAC;AAED,KAAK,UAAU,uBAAuB,CAAC,IAAc,EAAE,IAAY;IACjE,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtC,MAAM,WAAW,GAAG,UAAU,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;IACjD,MAAM,UAAU,GAAG,UAAU,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;IAC/C,MAAM,cAAc,GAAG,UAAU,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;IACpD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;IAC3B,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;IAC1B,IAAI,CAAC,KAAK,IAAI,CAAC,oBAAoB,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;QACtF,MAAM,YAAY,CAAC,6DAA6D,CAAC,CAAC;IACpF,CAAC;IACD,MAAM,MAAM,GAAG,WAAW,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;IACnE,MAAM,QAAQ,GAAG,UAAU,KAAK,SAAS,CAAC,CAAC,CAAC,kCAAkC,CAAC,CAAC,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC;IACpG,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,CAAC;QAAE,MAAM,YAAY,CAAC,yCAAyC,CAAC,CAAC;IAC/G,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,QAAQ,CAAC,IAAI,QAAQ,GAAG,CAAC,IAAI,QAAQ,GAAG,kCAAkC,EAAE,CAAC;QACrG,MAAM,YAAY,CAAC,wCAAwC,kCAAkC,EAAE,CAAC,CAAC;IACnG,CAAC;IACD,IAAI,cAAc,KAAK,SAAS,IAAI,CAAC,uBAAuB,CAAC,IAAI,CAAC,cAAc,CAAC,EAAE,CAAC;QAClF,MAAM,YAAY,CAAC,mCAAmC,CAAC,CAAC;IAC1D,CAAC;IACD,YAAY,CAAC,IAAI,CAAC,CAAC;IACnB,MAAM,IAAI,GAAG,MAAM,0BAA0B,CAAC,IAAI,EAAE,KAAK,EAAE,IAAyC,EAAE;QACpG,MAAM;QACN,QAAQ;QACR,GAAG,CAAC,cAAc,CAAC,CAAC,CAAC,EAAE,cAAc,EAAE,cAAwB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACxE,CAAC,CAAC;IACH,IAAI,IAAI;QAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;;QAChE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC5C,CAAC"}

package/dist/src/cli/output.js

@@ -130,6 +130,8 @@ Usage:
   hitch images unpin <sha256:image-id>
   hitch trajectory inspect <run-id> [--json]
   hitch verifier inspect <run-id> [--json]
+  hitch verifier artifact <run-id> <name> [--offset <bytes>] [--limit <bytes>] [--sha256 <digest>] [--json]
+  hitch verifier repair <run-id> [--source <eval-relative-trial-dir>] [--json]
   hitch trajectory project <run-id> [--profile analysis] [--max-bytes <bytes>] [--json]
   hitch trajectory events <run-id> [--types <a,b>] [--seq-start <n>] [--seq-end <n>] [--field <path>] [--canonical-sha256 <digest>] [--limit <n>] [--cursor <opaque>] [--max-bytes <bytes>] [--json]
   hitch training register --file <PATH|->

package/dist/src/cli/output.js.map

@@ -1 +1 @@
-{"version":3,"file":"output.js","sourceRoot":"","sources":["../../../src/cli/output.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,KAAK,EAAE,MAAM,wBAAwB,CAAC;AAE3D,MAAM,UAAU,aAAa,CAAC,QAAkH;IAC9I,IAAI,QAAQ,CAAC,QAAQ,CAAC,IAAI,KAAK,QAAQ;QAAE,OAAO,QAAQ,CAAC,QAAQ,CAAC,MAAM,IAAI,EAAE,CAAC;IAC/E,IAAI,QAAQ,CAAC,QAAQ,CAAC,IAAI,KAAK,SAAS;QAAE,OAAO,QAAQ,CAAC,QAAQ,CAAC,OAAO,IAAI,EAAE,CAAC;IACjF,OAAO,QAAQ,CAAC,QAAQ,CAAC,OAAO,IAAI,QAAQ,CAAC,MAAM,CAAC,SAAS,IAAI,EAAE,CAAC;AACtE,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,gBAAgB,CAAC,MAA8O,EAAE,KAAa,EAAE,MAAc;IAClT,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,SAAS,CAAC;QACR,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,YAAY,KAAK,EAAE,CAAC,CAAC;QACzD,IAAI,MAAM,KAAK,OAAO,EAAE,CAAC;YACvB,IAAI,CAAC;gBACH,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,mBAAmB,CAAC,YAAY,KAAK,kBAAkB,WAAW,EAAE,CAAC,CAAC;gBACpG,MAAM,GAAG,GAAG,QAAQ,CAAC,OAAO,CAAC;gBAC7B,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;oBACrE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;oBACjB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,IAAI,CAAC,CAAC;gBACpC,CAAC;gBACD,WAAW,GAAG,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,IAAI,WAAW,CAAC,CAAC;YACnF,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAK,KAA6B,CAAC,MAAM,KAAK,GAAG;oBAAE,MAAM,KAAK,CAAC;YACjE,CAAC;QACH,CAAC;QACD,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;YAClB,IAAI,MAAM,KAAK,MAAM;gBAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;YAC3F,OAAO,MAAM,CAAC,MAAiC,CAAC;QAClD,CAAC;QACD,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,iBAAiB,CAAC,MAA8O,EAAE,MAAc,EAAE,MAAc;IACpT,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,SAAS,CAAC;QACR,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,aAAa,MAAM,EAAE,CAAC,CAAC;QAC3D,IAAI,MAAM,KAAK,OAAO,EAAE,CAAC;YACvB,IAAI,CAAC;gBACH,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,mBAAmB,CAAC,aAAa,MAAM,kBAAkB,WAAW,EAAE,CAAC,CAAC;gBACtG,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;oBAClF,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;oBACjB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,IAAI,CAAC,CAAC;gBACpC,CAAC;gBACD,WAAW,GAAG,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,IAAI,WAAW,CAAC,CAAC;YACnF,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAK,KAA6B,CAAC,MAAM,KAAK,GAAG;oBAAE,MAAM,KAAK,CAAC;YACjE,CAAC;QACH,CAAC;QACD,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;YAClB,IAAI,MAAM,KAAK,MAAM;gBAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;YAC3F,OAAO,MAAM,CAAC,MAAiC,CAAC;QAClD,CAAC;QACD,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,sBAAsB,CAAC,MAAkG,EAAE,MAAc,EAAE,OAAe;IAC9K,SAAS,CAAC;QACR,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,aAAa,MAAM,WAAW,OAAO,EAAE,CAAC,CAAC;QAC7E,IAAI,MAAM,CAAC,MAAM,IAAK,MAAM,CAAC,KAA6B,EAAE,MAAM,KAAK,WAAW,EAAE,CAAC;YACnF,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;YACpE,OAAO,MAAM,CAAC,MAAiC,CAAC;QAClD,CAAC;QACD,MAAM,KAAK,GAAG,MAAM,CAAC,KAAwF,CAAC;QAC9G,IAAI,KAAK,EAAE,MAAM,KAAK,QAAQ,IAAI,KAAK,EAAE,MAAM,KAAK,WAAW,EAAE,CAAC;YAChE,MAAM,IAAI,UAAU,CAAC,OAAO,KAAK,CAAC,KAAK,EAAE,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,mBAAmB,EAAE;gBACzG,IAAI,EAAE,OAAO,KAAK,CAAC,KAAK,EAAE,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,mBAAmB;gBACpF,QAAQ,EAAE,EAAE;aACb,CAAC,CAAC;QACL,CAAC;QACD,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC;AAED,MAAM,UAAU,QAAQ;IACtB,OAAO;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAwHR,CAAC;AACF,CAAC"}
+{"version":3,"file":"output.js","sourceRoot":"","sources":["../../../src/cli/output.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,KAAK,EAAE,MAAM,wBAAwB,CAAC;AAE3D,MAAM,UAAU,aAAa,CAAC,QAAkH;IAC9I,IAAI,QAAQ,CAAC,QAAQ,CAAC,IAAI,KAAK,QAAQ;QAAE,OAAO,QAAQ,CAAC,QAAQ,CAAC,MAAM,IAAI,EAAE,CAAC;IAC/E,IAAI,QAAQ,CAAC,QAAQ,CAAC,IAAI,KAAK,SAAS;QAAE,OAAO,QAAQ,CAAC,QAAQ,CAAC,OAAO,IAAI,EAAE,CAAC;IACjF,OAAO,QAAQ,CAAC,QAAQ,CAAC,OAAO,IAAI,QAAQ,CAAC,MAAM,CAAC,SAAS,IAAI,EAAE,CAAC;AACtE,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,gBAAgB,CAAC,MAA8O,EAAE,KAAa,EAAE,MAAc;IAClT,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,SAAS,CAAC;QACR,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,YAAY,KAAK,EAAE,CAAC,CAAC;QACzD,IAAI,MAAM,KAAK,OAAO,EAAE,CAAC;YACvB,IAAI,CAAC;gBACH,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,mBAAmB,CAAC,YAAY,KAAK,kBAAkB,WAAW,EAAE,CAAC,CAAC;gBACpG,MAAM,GAAG,GAAG,QAAQ,CAAC,OAAO,CAAC;gBAC7B,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;oBACrE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;oBACjB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,IAAI,CAAC,CAAC;gBACpC,CAAC;gBACD,WAAW,GAAG,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,IAAI,WAAW,CAAC,CAAC;YACnF,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAK,KAA6B,CAAC,MAAM,KAAK,GAAG;oBAAE,MAAM,KAAK,CAAC;YACjE,CAAC;QACH,CAAC;QACD,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;YAClB,IAAI,MAAM,KAAK,MAAM;gBAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;YAC3F,OAAO,MAAM,CAAC,MAAiC,CAAC;QAClD,CAAC;QACD,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,iBAAiB,CAAC,MAA8O,EAAE,MAAc,EAAE,MAAc;IACpT,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,SAAS,CAAC;QACR,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,aAAa,MAAM,EAAE,CAAC,CAAC;QAC3D,IAAI,MAAM,KAAK,OAAO,EAAE,CAAC;YACvB,IAAI,CAAC;gBACH,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,mBAAmB,CAAC,aAAa,MAAM,kBAAkB,WAAW,EAAE,CAAC,CAAC;gBACtG,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;oBAClF,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;oBACjB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,IAAI,CAAC,CAAC;gBACpC,CAAC;gBACD,WAAW,GAAG,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,IAAI,WAAW,CAAC,CAAC;YACnF,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAK,KAA6B,CAAC,MAAM,KAAK,GAAG;oBAAE,MAAM,KAAK,CAAC;YACjE,CAAC;QACH,CAAC;QACD,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;YAClB,IAAI,MAAM,KAAK,MAAM;gBAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;YAC3F,OAAO,MAAM,CAAC,MAAiC,CAAC;QAClD,CAAC;QACD,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,sBAAsB,CAAC,MAAkG,EAAE,MAAc,EAAE,OAAe;IAC9K,SAAS,CAAC;QACR,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,aAAa,MAAM,WAAW,OAAO,EAAE,CAAC,CAAC;QAC7E,IAAI,MAAM,CAAC,MAAM,IAAK,MAAM,CAAC,KAA6B,EAAE,MAAM,KAAK,WAAW,EAAE,CAAC;YACnF,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;YACpE,OAAO,MAAM,CAAC,MAAiC,CAAC;QAClD,CAAC;QACD,MAAM,KAAK,GAAG,MAAM,CAAC,KAAwF,CAAC;QAC9G,IAAI,KAAK,EAAE,MAAM,KAAK,QAAQ,IAAI,KAAK,EAAE,MAAM,KAAK,WAAW,EAAE,CAAC;YAChE,MAAM,IAAI,UAAU,CAAC,OAAO,KAAK,CAAC,KAAK,EAAE,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,mBAAmB,EAAE;gBACzG,IAAI,EAAE,OAAO,KAAK,CAAC,KAAK,EAAE,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,mBAAmB;gBACpF,QAAQ,EAAE,EAAE;aACb,CAAC,CAAC;QACL,CAAC;QACD,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC;AACH,CAAC;AAED,MAAM,UAAU,QAAQ;IACtB,OAAO;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CA0HR,CAAC;AACF,CAAC"}

package/dist/src/evals/benchmark-adapter-manifest.js

@@ -86,6 +86,7 @@ export async function loadBenchmarkAdapterManifest(dataset) {
         benchmark: manifest.benchmark,
         adapter: manifest.adapter,
         scoring: manifest.scoring,
+        ...(manifest.raw_metrics === undefined ? {} : { raw_metrics: manifest.raw_metrics }),
         tasks: actualTasks,
     };
     const expected = sha256(canonicalJson(body));
@@ -98,7 +99,7 @@ export function scoreWithinRange(score, definition) {
 }
 function parseManifest(value) {
     const record = object(value, "benchmark adapter manifest");
-    exact(record, ["schema_version", "kind", "benchmark", "adapter", "scoring", "tasks", "dataset_digest"], "benchmark adapter manifest");
+    exact(record, ["schema_version", "kind", "benchmark", "adapter", "scoring", "raw_metrics", "tasks", "dataset_digest"], "benchmark adapter manifest");
     if (record.schema_version !== "1" || record.kind !== "gear-harbor-benchmark")
         throw invalidInput("unsupported benchmark adapter manifest");
     const benchmark = namedRevision(record.benchmark, "benchmark");
@@ -130,10 +131,27 @@ function parseManifest(value) {
         benchmark,
         adapter,
         scoring,
+        ...(record.raw_metrics === undefined ? {} : { raw_metrics: rawMetricRegistry(record.raw_metrics) }),
         tasks,
         dataset_digest: digest(record.dataset_digest, "dataset digest"),
     };
 }
+function rawMetricRegistry(value) {
+    const registry = object(value, "raw_metrics");
+    exact(registry, ["schema_version", "metrics"], "raw_metrics");
+    if (registry.schema_version !== "1" || !Array.isArray(registry.metrics)) {
+        throw invalidInput("unsupported raw metric registry");
+    }
+    const metrics = registry.metrics.map((value, index) => {
+        const metric = object(value, `raw metric ${index}`);
+        identifier(metric.id, `raw metric ${index} id`);
+        return metric;
+    });
+    if (new Set(metrics.map((metric) => metric.id)).size !== metrics.length) {
+        throw invalidInput("duplicate raw metric id");
+    }
+    return { schema_version: "1", metrics };
+}
 function namedRevision(value, label) {
     const record = object(value, label);
     exact(record, ["id", "revision"], label);

package/dist/src/evals/benchmark-adapter-manifest.js.map

@@ -1 +1 @@
-{"version":3,"file":"benchmark-adapter-manifest.js","sourceRoot":"","sources":["../../../src/evals/benchmark-adapter-manifest.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,OAAO,EAAE,MAAM,kBAAkB,CAAC;AAC5D,OAAO,IAAI,MAAM,WAAW,CAAC;AAE7B,OAAO,EAAE,YAAY,EAAE,MAAM,wBAAwB,CAAC;AAEtD,MAAM,MAAM,GAAG,uBAAuB,CAAC;AAwBvC,+EAA+E;AAC/E,MAAM,CAAC,KAAK,UAAU,6BAA6B,CAAC,KAMnD;IACC,MAAM,GAAG,GAAG,CAAC,GAAG,KAAK,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC;IACtC,IAAI,CAAC,GAAG,CAAC,MAAM,IAAI,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,IAAI,KAAK,GAAG,CAAC,MAAM;QAAE,MAAM,YAAY,CAAC,yDAAyD,CAAC,CAAC;IACnI,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE;QACnD,MAAM,KAAK,GAAG,MAAM,CAAC,EAAE,CAAC,CAAC;QACzB,MAAM,SAAS,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,KAAK,CAAC,CAAC;QAChE,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,KAA4B,EAAE,EAAE;YACzE,IAAI,KAAK,CAAC,IAAI,KAAK,QAAQ;gBAAE,MAAM,YAAY,CAAC,6BAA6B,KAAK,EAAE,CAAC,CAAC;YACtF,MAAM,KAAK,CAAC;QACd,CAAC,CAAC,CAAC;QACH,IAAI,CAAC,IAAI,CAAC,WAAW,EAAE,IAAI,IAAI,CAAC,cAAc,EAAE;YAAE,MAAM,YAAY,CAAC,qCAAqC,KAAK,EAAE,CAAC,CAAC;QACnH,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC,SAAS,CAAC,EAAE,CAAC;IAC1E,CAAC,CAAC,CAAC,CAAC;IACJ,MAAM,IAAI,GAAG;QACX,cAAc,EAAE,GAAY;QAC5B,IAAI,EAAE,uBAAgC;QACtC,SAAS,EAAE,KAAK,CAAC,SAAS;QAC1B,OAAO,EAAE,KAAK,CAAC,OAAO;QACtB,OAAO,EAAE,KAAK,CAAC,OAAO;QACtB,KAAK;KACN,CAAC;IACF,OAAO,EAAE,GAAG,IAAI,EAAE,cAAc,EAAE,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;AAClE,CAAC;AAED,gFAAgF;AAChF,MAAM,CAAC,KAAK,UAAU,4BAA4B,CAAC,OAAe;IAChE,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IACnC,IAAI,GAAY,CAAC;IACjB,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,wBAAwB,CAAC,CAAC,CAAC;QACpE,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,IAAI,CAAC,cAAc,EAAE;YAAE,MAAM,YAAY,CAAC,+CAA+C,CAAC,CAAC;QACjH,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,wBAAwB,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC;IACtF,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAK,KAA+B,CAAC,IAAI,KAAK,QAAQ;YAAE,OAAO,IAAI,CAAC;QACpE,IAAI,KAAK,YAAY,WAAW;YAAE,MAAM,YAAY,CAAC,wCAAwC,CAAC,CAAC;QAC/F,MAAM,KAAK,CAAC;IACd,CAAC;IACD,MAAM,QAAQ,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;IACpC,MAAM,WAAW,GAAoD,EAAE,CAAC;IACxE,KAAK,MAAM,IAAI,IAAI,QAAQ,CAAC,KAAK,EAAE,CAAC;QAClC,MAAM,SAAS,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,OAAO,CAAC,CAAC;QAChD,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,KAA4B,EAAE,EAAE;YACzE,IAAI,KAAK,CAAC,IAAI,KAAK,QAAQ;gBAAE,MAAM,YAAY,CAAC,6BAA6B,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;YAC7F,MAAM,KAAK,CAAC;QACd,CAAC,CAAC,CAAC;QACH,IAAI,CAAC,IAAI,CAAC,WAAW,EAAE,IAAI,IAAI,CAAC,cAAc,EAAE;YAAE,MAAM,YAAY,CAAC,qCAAqC,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;QAC1H,MAAM,MAAM,GAAG,MAAM,cAAc,CAAC,SAAS,CAAC,CAAC;QAC/C,IAAI,MAAM,KAAK,IAAI,CAAC,WAAW;YAAE,MAAM,YAAY,CAAC,kCAAkC,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;QACtG,WAAW,CAAC,IAAI,CAAC,EAAE,OAAO,EAAE,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,MAAM,EAAE,CAAC,CAAC;IACnE,CAAC;IACD,MAAM,eAAe,GAAa,EAAE,CAAC;IACrC,KAAK,MAAM,KAAK,IAAI,MAAM,OAAO,CAAC,IAAI,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC;QACjE,IAAI,CAAC,KAAK,CAAC,WAAW,EAAE;YAAE,SAAS;QACnC,IAAI,CAAC;YACH,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC,CAAC;YACvE,IAAI,QAAQ,CAAC,MAAM,EAAE,IAAI,CAAC,QAAQ,CAAC,cAAc,EAAE;gBAAE,eAAe,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACxF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAK,KAA+B,CAAC,IAAI,KAAK,QAAQ;gBAAE,MAAM,KAAK,CAAC;QACtE,CAAC;IACH,CAAC;IACD,IAAI,IAAI,CAAC,SAAS,CAAC,eAAe,CAAC,IAAI,EAAE,CAAC,KAAK,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC;QACvG,MAAM,YAAY,CAAC,gEAAgE,CAAC,CAAC;IACvF,CAAC;IACD,MAAM,IAAI,GAAG;QACX,cAAc,EAAE,QAAQ,CAAC,cAAc;QACvC,IAAI,EAAE,QAAQ,CAAC,IAAI;QACnB,SAAS,EAAE,QAAQ,CAAC,SAAS;QAC7B,OAAO,EAAE,QAAQ,CAAC,OAAO;QACzB,OAAO,EAAE,QAAQ,CAAC,OAAO;QACzB,KAAK,EAAE,WAAW;KACnB,CAAC;IACF,MAAM,QAAQ,GAAG,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,CAAC;IAC7C,IAAI,QAAQ,CAAC,cAAc,KAAK,QAAQ;QAAE,MAAM,YAAY,CAAC,2CAA2C,CAAC,CAAC;IAC1G,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,MAAM,UAAU,gBAAgB,CAAC,KAAa,EAAE,UAAsC;IACpF,OAAO,KAAK,IAAI,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,KAAK,IAAI,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;AACtE,CAAC;AAED,SAAS,aAAa,CAAC,KAAc;IACnC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,EAAE,4BAA4B,CAAC,CAAC;IAC3D,KAAK,CAAC,MAAM,EAAE,CAAC,gBAAgB,EAAE,MAAM,EAAE,WAAW,EAAE,SAAS,EAAE,SAAS,EAAE,OAAO,EAAE,gBAAgB,CAAC,EAAE,4BAA4B,CAAC,CAAC;IACtI,IAAI,MAAM,CAAC,cAAc,KAAK,GAAG,IAAI,MAAM,CAAC,IAAI,KAAK,uBAAuB;QAAE,MAAM,YAAY,CAAC,wCAAwC,CAAC,CAAC;IAC3I,MAAM,SAAS,GAAG,aAAa,CAAC,MAAM,CAAC,SAAS,EAAE,WAAW,CAAC,CAAC;IAC/D,MAAM,aAAa,GAAG,MAAM,CAAC,MAAM,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IACxD,KAAK,CAAC,aAAa,EAAE,CAAC,IAAI,EAAE,UAAU,EAAE,iBAAiB,CAAC,EAAE,SAAS,CAAC,CAAC;IACvE,IAAI,aAAa,CAAC,eAAe,KAAK,4BAA4B;QAAE,MAAM,YAAY,CAAC,+CAA+C,CAAC,CAAC;IACxI,MAAM,OAAO,GAAG,EAAE,EAAE,EAAE,UAAU,CAAC,aAAa,CAAC,EAAE,EAAE,YAAY,CAAC,EAAE,QAAQ,EAAE,iBAAiB,CAAC,aAAa,CAAC,QAAQ,EAAE,kBAAkB,CAAC,EAAE,eAAe,EAAE,aAAa,CAAC,eAAe,EAAW,CAAC;IACrM,MAAM,aAAa,GAAG,MAAM,CAAC,MAAM,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IACxD,KAAK,CAAC,aAAa,EAAE,CAAC,aAAa,EAAE,eAAe,CAAC,EAAE,SAAS,CAAC,CAAC;IAClE,MAAM,OAAO,GAA6B;QACxC,WAAW,EAAE,eAAe,CAAC,aAAa,CAAC,WAAW,EAAE,aAAa,CAAC;QACtE,GAAG,CAAC,aAAa,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,eAAe,CAAC,aAAa,CAAC,aAAa,EAAE,eAAe,CAAC,EAAE,CAAC;KACvI,CAAC;IACF,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,mDAAmD,CAAC,CAAC;IACvI,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE;QAC5C,MAAM,IAAI,GAAG,MAAM,CAAC,GAAG,EAAE,QAAQ,KAAK,EAAE,CAAC,CAAC;QAC1C,KAAK,CAAC,IAAI,EAAE,CAAC,SAAS,EAAE,aAAa,CAAC,EAAE,QAAQ,KAAK,EAAE,CAAC,CAAC;QACzD,OAAO,EAAE,OAAO,EAAE,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,WAAW,EAAE,MAAM,CAAC,IAAI,CAAC,WAAW,EAAE,QAAQ,KAAK,SAAS,CAAC,EAAE,CAAC;IAC1G,CAAC,CAAC,CAAC;IACH,MAAM,GAAG,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;IAC9C,IAAI,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,IAAI,KAAK,GAAG,CAAC,MAAM,IAAI,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,KAAK,IAAI,CAAC,SAAS,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC;QAChG,MAAM,YAAY,CAAC,sDAAsD,CAAC,CAAC;IAC7E,CAAC;IACD,OAAO;QACL,cAAc,EAAE,GAAG;QACnB,IAAI,EAAE,uBAAuB;QAC7B,SAAS;QACT,OAAO;QACP,OAAO;QACP,KAAK;QACL,cAAc,EAAE,MAAM,CAAC,MAAM,CAAC,cAAc,EAAE,gBAAgB,CAAC;KAChE,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,KAAc,EAAE,KAAa;IAClD,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;IACpC,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,UAAU,CAAC,EAAE,KAAK,CAAC,CAAC;IACzC,OAAO,EAAE,EAAE,EAAE,UAAU,CAAC,MAAM,CAAC,EAAE,EAAE,GAAG,KAAK,KAAK,CAAC,EAAE,QAAQ,EAAE,iBAAiB,CAAC,MAAM,CAAC,QAAQ,EAAE,GAAG,KAAK,WAAW,CAAC,EAAE,CAAC;AACzH,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,KAAa;IACpD,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;IACpC,KAAK,CAAC,MAAM,EAAE,CAAC,eAAe,EAAE,WAAW,EAAE,OAAO,EAAE,SAAS,CAAC,EAAE,KAAK,CAAC,CAAC;IACzE,IAAI,MAAM,CAAC,SAAS,KAAK,UAAU,IAAI,MAAM,CAAC,SAAS,KAAK,UAAU;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,uBAAuB,CAAC,CAAC;IAC5H,IAAI,MAAM,CAAC,OAAO,KAAK,iBAAiB;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,qBAAqB,CAAC,CAAC;IAC5F,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,OAAO,IAAI,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;QACjJ,MAAM,YAAY,CAAC,GAAG,KAAK,mBAAmB,CAAC,CAAC;IAClD,CAAC;IACD,MAAM,KAAK,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAW,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAW,CAAU,CAAC;IAC9E,IAAI,KAAK,CAAC,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,oBAAoB,CAAC,CAAC;IAC1E,OAAO,EAAE,aAAa,EAAE,UAAU,CAAC,MAAM,CAAC,aAAa,EAAE,GAAG,KAAK,gBAAgB,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,KAAK,EAAE,OAAO,EAAE,iBAAiB,EAAE,CAAC;AACvJ,CAAC;AAED,KAAK,UAAU,cAAc,CAAC,IAAY;IACxC,MAAM,IAAI,GAAyE,EAAE,CAAC;IACtF,MAAM,KAAK,GAAG,KAAK,EAAE,SAAiB,EAAiB,EAAE;QACvD,KAAK,MAAM,KAAK,IAAI,CAAC,MAAM,OAAO,CAAC,SAAS,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;YACnI,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,KAAK,CAAC,IAAI,CAAC,CAAC;YAClD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACzE,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,CAAC,CAAC;YACnC,IAAI,IAAI,CAAC,cAAc,EAAE;gBAAE,MAAM,YAAY,CAAC,sCAAsC,QAAQ,EAAE,CAAC,CAAC;YAChG,IAAI,IAAI,CAAC,WAAW,EAAE;gBAAE,MAAM,KAAK,CAAC,QAAQ,CAAC,CAAC;iBACzC,IAAI,IAAI,CAAC,MAAM,EAAE;gBAAE,IAAI,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,QAAQ,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;;gBAC/K,MAAM,YAAY,CAAC,2CAA2C,QAAQ,EAAE,CAAC,CAAC;QACjF,CAAC;IACH,CAAC,CAAC;IACF,MAAM,KAAK,CAAC,IAAI,CAAC,CAAC;IAClB,OAAO,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC;AACtC,CAAC;AAED,SAAS,aAAa,CAAC,KAAc;IACnC,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,OAAO,IAAI,KAAK,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC;IAC3E,IAAI,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QACvC,OAAO,IAAI,MAAM,CAAC,OAAO,CAAC,KAAgC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,KAAK,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,aAAa,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,IAAI,aAAa,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC;IACxM,CAAC;IACD,OAAO,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC;AAC/B,CAAC;AAED,SAAS,MAAM,CAAC,KAAa;IAC3B,OAAO,UAAU,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAY,CAAC;AAChF,CAAC;AAED,SAAS,MAAM,CAAC,KAAc,EAAE,KAAa;IAC3C,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,oBAAoB,CAAC,CAAC;IAClH,OAAO,KAAgC,CAAC;AAC1C,CAAC;AAED,SAAS,KAAK,CAAC,MAA+B,EAAE,MAAyB,EAAE,KAAa;IACtF,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,CAAC;IAChC,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC;IACvE,IAAI,KAAK;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,uBAAuB,KAAK,EAAE,CAAC,CAAC;AACxE,CAAC;AAED,SAAS,UAAU,CAAC,KAAc,EAAE,KAAa;IAC/C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,qCAAqC,CAAC,IAAI,CAAC,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,aAAa,CAAC,CAAC;IAC/H,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,iBAAiB,CAAC,KAAc,EAAE,KAAa;IACtD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,IAAI,KAAK,CAAC,MAAM,GAAG,GAAG,IAAI,KAAK,CAAC,WAAW,EAAE,KAAK,QAAQ;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,aAAa,CAAC,CAAC;IACzJ,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,MAAM,CAAC,KAAc;IAC5B,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,oCAAoC,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,KAAK,KAAK,GAAG,IAAI,KAAK,KAAK,IAAI;QAAE,MAAM,YAAY,CAAC,oBAAoB,CAAC,CAAC;IAChK,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,MAAM,CAAC,KAAc,EAAE,KAAa;IAC3C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,aAAa,CAAC,CAAC;IAChG,OAAO,KAAe,CAAC;AACzB,CAAC"}
+{"version":3,"file":"benchmark-adapter-manifest.js","sourceRoot":"","sources":["../../../src/evals/benchmark-adapter-manifest.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,OAAO,EAAE,MAAM,kBAAkB,CAAC;AAC5D,OAAO,IAAI,MAAM,WAAW,CAAC;AAE7B,OAAO,EAAE,YAAY,EAAE,MAAM,wBAAwB,CAAC;AAEtD,MAAM,MAAM,GAAG,uBAAuB,CAAC;AA0BvC,+EAA+E;AAC/E,MAAM,CAAC,KAAK,UAAU,6BAA6B,CAAC,KAMnD;IACC,MAAM,GAAG,GAAG,CAAC,GAAG,KAAK,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC;IACtC,IAAI,CAAC,GAAG,CAAC,MAAM,IAAI,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,IAAI,KAAK,GAAG,CAAC,MAAM;QAAE,MAAM,YAAY,CAAC,yDAAyD,CAAC,CAAC;IACnI,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE;QACnD,MAAM,KAAK,GAAG,MAAM,CAAC,EAAE,CAAC,CAAC;QACzB,MAAM,SAAS,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,KAAK,CAAC,CAAC;QAChE,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,KAA4B,EAAE,EAAE;YACzE,IAAI,KAAK,CAAC,IAAI,KAAK,QAAQ;gBAAE,MAAM,YAAY,CAAC,6BAA6B,KAAK,EAAE,CAAC,CAAC;YACtF,MAAM,KAAK,CAAC;QACd,CAAC,CAAC,CAAC;QACH,IAAI,CAAC,IAAI,CAAC,WAAW,EAAE,IAAI,IAAI,CAAC,cAAc,EAAE;YAAE,MAAM,YAAY,CAAC,qCAAqC,KAAK,EAAE,CAAC,CAAC;QACnH,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC,SAAS,CAAC,EAAE,CAAC;IAC1E,CAAC,CAAC,CAAC,CAAC;IACJ,MAAM,IAAI,GAAG;QACX,cAAc,EAAE,GAAY;QAC5B,IAAI,EAAE,uBAAgC;QACtC,SAAS,EAAE,KAAK,CAAC,SAAS;QAC1B,OAAO,EAAE,KAAK,CAAC,OAAO;QACtB,OAAO,EAAE,KAAK,CAAC,OAAO;QACtB,KAAK;KACN,CAAC;IACF,OAAO,EAAE,GAAG,IAAI,EAAE,cAAc,EAAE,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;AAClE,CAAC;AAED,gFAAgF;AAChF,MAAM,CAAC,KAAK,UAAU,4BAA4B,CAAC,OAAe;IAChE,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IACnC,IAAI,GAAY,CAAC;IACjB,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,wBAAwB,CAAC,CAAC,CAAC;QACpE,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,IAAI,CAAC,cAAc,EAAE;YAAE,MAAM,YAAY,CAAC,+CAA+C,CAAC,CAAC;QACjH,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,wBAAwB,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC;IACtF,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAK,KAA+B,CAAC,IAAI,KAAK,QAAQ;YAAE,OAAO,IAAI,CAAC;QACpE,IAAI,KAAK,YAAY,WAAW;YAAE,MAAM,YAAY,CAAC,wCAAwC,CAAC,CAAC;QAC/F,MAAM,KAAK,CAAC;IACd,CAAC;IACD,MAAM,QAAQ,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;IACpC,MAAM,WAAW,GAAoD,EAAE,CAAC;IACxE,KAAK,MAAM,IAAI,IAAI,QAAQ,CAAC,KAAK,EAAE,CAAC;QAClC,MAAM,SAAS,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,OAAO,CAAC,CAAC;QAChD,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,KAA4B,EAAE,EAAE;YACzE,IAAI,KAAK,CAAC,IAAI,KAAK,QAAQ;gBAAE,MAAM,YAAY,CAAC,6BAA6B,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;YAC7F,MAAM,KAAK,CAAC;QACd,CAAC,CAAC,CAAC;QACH,IAAI,CAAC,IAAI,CAAC,WAAW,EAAE,IAAI,IAAI,CAAC,cAAc,EAAE;YAAE,MAAM,YAAY,CAAC,qCAAqC,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;QAC1H,MAAM,MAAM,GAAG,MAAM,cAAc,CAAC,SAAS,CAAC,CAAC;QAC/C,IAAI,MAAM,KAAK,IAAI,CAAC,WAAW;YAAE,MAAM,YAAY,CAAC,kCAAkC,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;QACtG,WAAW,CAAC,IAAI,CAAC,EAAE,OAAO,EAAE,IAAI,CAAC,OAAO,EAAE,WAAW,EAAE,MAAM,EAAE,CAAC,CAAC;IACnE,CAAC;IACD,MAAM,eAAe,GAAa,EAAE,CAAC;IACrC,KAAK,MAAM,KAAK,IAAI,MAAM,OAAO,CAAC,IAAI,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC;QACjE,IAAI,CAAC,KAAK,CAAC,WAAW,EAAE;YAAE,SAAS;QACnC,IAAI,CAAC;YACH,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC,CAAC;YACvE,IAAI,QAAQ,CAAC,MAAM,EAAE,IAAI,CAAC,QAAQ,CAAC,cAAc,EAAE;gBAAE,eAAe,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACxF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAK,KAA+B,CAAC,IAAI,KAAK,QAAQ;gBAAE,MAAM,KAAK,CAAC;QACtE,CAAC;IACH,CAAC;IACD,IAAI,IAAI,CAAC,SAAS,CAAC,eAAe,CAAC,IAAI,EAAE,CAAC,KAAK,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC;QACvG,MAAM,YAAY,CAAC,gEAAgE,CAAC,CAAC;IACvF,CAAC;IACD,MAAM,IAAI,GAAG;QACX,cAAc,EAAE,QAAQ,CAAC,cAAc;QACvC,IAAI,EAAE,QAAQ,CAAC,IAAI;QACnB,SAAS,EAAE,QAAQ,CAAC,SAAS;QAC7B,OAAO,EAAE,QAAQ,CAAC,OAAO;QACzB,OAAO,EAAE,QAAQ,CAAC,OAAO;QACzB,GAAG,CAAC,QAAQ,CAAC,WAAW,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,QAAQ,CAAC,WAAW,EAAE,CAAC;QACpF,KAAK,EAAE,WAAW;KACnB,CAAC;IACF,MAAM,QAAQ,GAAG,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,CAAC;IAC7C,IAAI,QAAQ,CAAC,cAAc,KAAK,QAAQ;QAAE,MAAM,YAAY,CAAC,2CAA2C,CAAC,CAAC;IAC1G,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,MAAM,UAAU,gBAAgB,CAAC,KAAa,EAAE,UAAsC;IACpF,OAAO,KAAK,IAAI,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,KAAK,IAAI,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;AACtE,CAAC;AAED,SAAS,aAAa,CAAC,KAAc;IACnC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,EAAE,4BAA4B,CAAC,CAAC;IAC3D,KAAK,CAAC,MAAM,EAAE,CAAC,gBAAgB,EAAE,MAAM,EAAE,WAAW,EAAE,SAAS,EAAE,SAAS,EAAE,aAAa,EAAE,OAAO,EAAE,gBAAgB,CAAC,EAAE,4BAA4B,CAAC,CAAC;IACrJ,IAAI,MAAM,CAAC,cAAc,KAAK,GAAG,IAAI,MAAM,CAAC,IAAI,KAAK,uBAAuB;QAAE,MAAM,YAAY,CAAC,wCAAwC,CAAC,CAAC;IAC3I,MAAM,SAAS,GAAG,aAAa,CAAC,MAAM,CAAC,SAAS,EAAE,WAAW,CAAC,CAAC;IAC/D,MAAM,aAAa,GAAG,MAAM,CAAC,MAAM,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IACxD,KAAK,CAAC,aAAa,EAAE,CAAC,IAAI,EAAE,UAAU,EAAE,iBAAiB,CAAC,EAAE,SAAS,CAAC,CAAC;IACvE,IAAI,aAAa,CAAC,eAAe,KAAK,4BAA4B;QAAE,MAAM,YAAY,CAAC,+CAA+C,CAAC,CAAC;IACxI,MAAM,OAAO,GAAG,EAAE,EAAE,EAAE,UAAU,CAAC,aAAa,CAAC,EAAE,EAAE,YAAY,CAAC,EAAE,QAAQ,EAAE,iBAAiB,CAAC,aAAa,CAAC,QAAQ,EAAE,kBAAkB,CAAC,EAAE,eAAe,EAAE,aAAa,CAAC,eAAe,EAAW,CAAC;IACrM,MAAM,aAAa,GAAG,MAAM,CAAC,MAAM,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IACxD,KAAK,CAAC,aAAa,EAAE,CAAC,aAAa,EAAE,eAAe,CAAC,EAAE,SAAS,CAAC,CAAC;IAClE,MAAM,OAAO,GAA6B;QACxC,WAAW,EAAE,eAAe,CAAC,aAAa,CAAC,WAAW,EAAE,aAAa,CAAC;QACtE,GAAG,CAAC,aAAa,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,eAAe,CAAC,aAAa,CAAC,aAAa,EAAE,eAAe,CAAC,EAAE,CAAC;KACvI,CAAC;IACF,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,mDAAmD,CAAC,CAAC;IACvI,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE;QAC5C,MAAM,IAAI,GAAG,MAAM,CAAC,GAAG,EAAE,QAAQ,KAAK,EAAE,CAAC,CAAC;QAC1C,KAAK,CAAC,IAAI,EAAE,CAAC,SAAS,EAAE,aAAa,CAAC,EAAE,QAAQ,KAAK,EAAE,CAAC,CAAC;QACzD,OAAO,EAAE,OAAO,EAAE,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,WAAW,EAAE,MAAM,CAAC,IAAI,CAAC,WAAW,EAAE,QAAQ,KAAK,SAAS,CAAC,EAAE,CAAC;IAC1G,CAAC,CAAC,CAAC;IACH,MAAM,GAAG,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;IAC9C,IAAI,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,IAAI,KAAK,GAAG,CAAC,MAAM,IAAI,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,KAAK,IAAI,CAAC,SAAS,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC;QAChG,MAAM,YAAY,CAAC,sDAAsD,CAAC,CAAC;IAC7E,CAAC;IACD,OAAO;QACL,cAAc,EAAE,GAAG;QACnB,IAAI,EAAE,uBAAuB;QAC7B,SAAS;QACT,OAAO;QACP,OAAO;QACP,GAAG,CAAC,MAAM,CAAC,WAAW,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,iBAAiB,CAAC,MAAM,CAAC,WAAW,CAAC,EAAE,CAAC;QACnG,KAAK;QACL,cAAc,EAAE,MAAM,CAAC,MAAM,CAAC,cAAc,EAAE,gBAAgB,CAAC;KAChE,CAAC;AACJ,CAAC;AAED,SAAS,iBAAiB,CAAC,KAAc;IACvC,MAAM,QAAQ,GAAG,MAAM,CAAC,KAAK,EAAE,aAAa,CAAC,CAAC;IAC9C,KAAK,CAAC,QAAQ,EAAE,CAAC,gBAAgB,EAAE,SAAS,CAAC,EAAE,aAAa,CAAC,CAAC;IAC9D,IAAI,QAAQ,CAAC,cAAc,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;QACxE,MAAM,YAAY,CAAC,iCAAiC,CAAC,CAAC;IACxD,CAAC;IACD,MAAM,OAAO,GAAG,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,EAAE;QACpD,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,EAAE,cAAc,KAAK,EAAE,CAAC,CAAC;QACpD,UAAU,CAAC,MAAM,CAAC,EAAE,EAAE,cAAc,KAAK,KAAK,CAAC,CAAC;QAChD,OAAO,MAAM,CAAC;IAChB,CAAC,CAAC,CAAC;IACH,IAAI,IAAI,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,OAAO,CAAC,MAAM,EAAE,CAAC;QACxE,MAAM,YAAY,CAAC,yBAAyB,CAAC,CAAC;IAChD,CAAC;IACD,OAAO,EAAE,cAAc,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC;AAC1C,CAAC;AAED,SAAS,aAAa,CAAC,KAAc,EAAE,KAAa;IAClD,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;IACpC,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,UAAU,CAAC,EAAE,KAAK,CAAC,CAAC;IACzC,OAAO,EAAE,EAAE,EAAE,UAAU,CAAC,MAAM,CAAC,EAAE,EAAE,GAAG,KAAK,KAAK,CAAC,EAAE,QAAQ,EAAE,iBAAiB,CAAC,MAAM,CAAC,QAAQ,EAAE,GAAG,KAAK,WAAW,CAAC,EAAE,CAAC;AACzH,CAAC;AAED,SAAS,eAAe,CAAC,KAAc,EAAE,KAAa;IACpD,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;IACpC,KAAK,CAAC,MAAM,EAAE,CAAC,eAAe,EAAE,WAAW,EAAE,OAAO,EAAE,SAAS,CAAC,EAAE,KAAK,CAAC,CAAC;IACzE,IAAI,MAAM,CAAC,SAAS,KAAK,UAAU,IAAI,MAAM,CAAC,SAAS,KAAK,UAAU;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,uBAAuB,CAAC,CAAC;IAC5H,IAAI,MAAM,CAAC,OAAO,KAAK,iBAAiB;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,qBAAqB,CAAC,CAAC;IAC5F,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,OAAO,IAAI,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;QACjJ,MAAM,YAAY,CAAC,GAAG,KAAK,mBAAmB,CAAC,CAAC;IAClD,CAAC;IACD,MAAM,KAAK,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAW,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAW,CAAU,CAAC;IAC9E,IAAI,KAAK,CAAC,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,oBAAoB,CAAC,CAAC;IAC1E,OAAO,EAAE,aAAa,EAAE,UAAU,CAAC,MAAM,CAAC,aAAa,EAAE,GAAG,KAAK,gBAAgB,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,KAAK,EAAE,OAAO,EAAE,iBAAiB,EAAE,CAAC;AACvJ,CAAC;AAED,KAAK,UAAU,cAAc,CAAC,IAAY;IACxC,MAAM,IAAI,GAAyE,EAAE,CAAC;IACtF,MAAM,KAAK,GAAG,KAAK,EAAE,SAAiB,EAAiB,EAAE;QACvD,KAAK,MAAM,KAAK,IAAI,CAAC,MAAM,OAAO,CAAC,SAAS,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;YACnI,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,KAAK,CAAC,IAAI,CAAC,CAAC;YAClD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACzE,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,CAAC,CAAC;YACnC,IAAI,IAAI,CAAC,cAAc,EAAE;gBAAE,MAAM,YAAY,CAAC,sCAAsC,QAAQ,EAAE,CAAC,CAAC;YAChG,IAAI,IAAI,CAAC,WAAW,EAAE;gBAAE,MAAM,KAAK,CAAC,QAAQ,CAAC,CAAC;iBACzC,IAAI,IAAI,CAAC,MAAM,EAAE;gBAAE,IAAI,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,QAAQ,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;;gBAC/K,MAAM,YAAY,CAAC,2CAA2C,QAAQ,EAAE,CAAC,CAAC;QACjF,CAAC;IACH,CAAC,CAAC;IACF,MAAM,KAAK,CAAC,IAAI,CAAC,CAAC;IAClB,OAAO,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC;AACtC,CAAC;AAED,SAAS,aAAa,CAAC,KAAc;IACnC,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,OAAO,IAAI,KAAK,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC;IAC3E,IAAI,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QACvC,OAAO,IAAI,MAAM,CAAC,OAAO,CAAC,KAAgC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,KAAK,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,aAAa,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,IAAI,aAAa,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC;IACxM,CAAC;IACD,OAAO,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC;AAC/B,CAAC;AAED,SAAS,MAAM,CAAC,KAAa;IAC3B,OAAO,UAAU,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAY,CAAC;AAChF,CAAC;AAED,SAAS,MAAM,CAAC,KAAc,EAAE,KAAa;IAC3C,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,oBAAoB,CAAC,CAAC;IAClH,OAAO,KAAgC,CAAC;AAC1C,CAAC;AAED,SAAS,KAAK,CAAC,MAA+B,EAAE,MAAyB,EAAE,KAAa;IACtF,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,CAAC;IAChC,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC;IACvE,IAAI,KAAK;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,uBAAuB,KAAK,EAAE,CAAC,CAAC;AACxE,CAAC;AAED,SAAS,UAAU,CAAC,KAAc,EAAE,KAAa;IAC/C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,qCAAqC,CAAC,IAAI,CAAC,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,aAAa,CAAC,CAAC;IAC/H,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,iBAAiB,CAAC,KAAc,EAAE,KAAa;IACtD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,IAAI,KAAK,CAAC,MAAM,GAAG,GAAG,IAAI,KAAK,CAAC,WAAW,EAAE,KAAK,QAAQ;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,aAAa,CAAC,CAAC;IACzJ,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,MAAM,CAAC,KAAc;IAC5B,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,oCAAoC,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,KAAK,KAAK,GAAG,IAAI,KAAK,KAAK,IAAI;QAAE,MAAM,YAAY,CAAC,oBAAoB,CAAC,CAAC;IAChK,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,MAAM,CAAC,KAAc,EAAE,KAAa;IAC3C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC;QAAE,MAAM,YAAY,CAAC,GAAG,KAAK,aAAa,CAAC,CAAC;IAChG,OAAO,KAAe,CAAC;AACzB,CAAC"}

package/dist/src/evals/index.js

@@ -22,6 +22,7 @@ export { recoverLocalDockerEvalLeases } from "./recovery.js";
 export { inspectEval, isControlPlaneEval, listEvals } from "./records.js";
 export { importEvalTrialRun, importEvalTrialRuns, validateEvalTrialReferences } from "./trial-import.js";
 export { captureVerifierDiagnostics, DEFAULT_VERIFIER_DIAGNOSTIC_MAX_BYTES, MAX_VERIFIER_DIAGNOSTIC_MAX_BYTES, persistTrialVerifierDiagnostics, VERIFIER_ARTIFACT_NAMES, VERIFIER_DIAGNOSTICS_INDEX_REF } from "./verifier-artifacts.js";
+export { repairVerifierDiagnostics } from "./verifier-diagnostic-repair.js";
 export { captureVerifierScoreEvidence } from "./verifier-score-artifacts.js";
 export { MAX_VERIFIER_FEEDBACK_BYTES, MAX_VERIFIER_PROCESS_BYTES } from "../domain/index.js";
 export { createEvalProgress, evalTrialKey, mergeEvalProgressTrial, parseEvalTrialRef, readEvalProgress, replaceInvalidEvalProgressTrial, writeEvalProgress } from "./progress.js";

package/dist/src/evals/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/evals/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,6BAA6B,EAAE,uBAAuB,EAAE,8BAA8B,EAAE,uCAAuC,EAAE,SAAS,EAAE,yBAAyB,EAAE,0BAA0B,EAAE,cAAc,EAAE,mBAAmB,EAAE,MAAM,cAAc,CAAC;AACtQ,OAAO,EAAE,6BAA6B,EAAE,4BAA4B,EAAE,gBAAgB,EAAE,MAAM,iCAAiC,CAAC;AAIhI,OAAO,EAAE,aAAa,EAAE,oBAAoB,EAAE,MAAM,aAAa,CAAC;AAClE,OAAO,EAAE,sBAAsB,EAAE,sBAAsB,EAAE,4BAA4B,EAAE,MAAM,qBAAqB,CAAC;AACnH,OAAO,EAAE,sBAAsB,EAAE,MAAM,gCAAgC,CAAC;AACxE,OAAO,EAAE,uBAAuB,EAAE,kBAAkB,EAAE,uBAAuB,EAAE,MAAM,yBAAyB,CAAC;AAC/G,OAAO,EAAE,oBAAoB,EAAE,0BAA0B,EAAE,oBAAoB,EAAE,MAAM,yBAAyB,CAAC;AAEjH,OAAO,EAAE,2BAA2B,EAAE,2BAA2B,EAAE,mBAAmB,EAAE,kBAAkB,EAAE,gBAAgB,EAAE,uBAAuB,EAAE,MAAM,kBAAkB,CAAC;AAEhL,OAAO,EAAE,uBAAuB,EAAE,MAAM,+BAA+B,CAAC;AACxE,OAAO,EAAE,6BAA6B,EAAE,8BAA8B,EAAE,oCAAoC,EAAE,+BAA+B,EAAE,MAAM,qBAAqB,CAAC;AAE3K,OAAO,EAAE,sBAAsB,EAAE,MAAM,yBAAyB,CAAC;AACjE,OAAO,EAAE,sBAAsB,EAAE,2BAA2B,EAAE,MAAM,+BAA+B,CAAC;AACpG,OAAO,EAAE,+BAA+B,EAAE,8BAA8B,EAAE,MAAM,0BAA0B,CAAC;AAI3G,OAAO,EAAE,oCAAoC,EAAE,8BAA8B,EAAE,oBAAoB,EAAE,oBAAoB,EAAE,uBAAuB,EAAE,sBAAsB,EAAE,yBAAyB,EAAE,mBAAmB,EAAE,mBAAmB,EAAE,sBAAsB,EAAE,qBAAqB,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AAErV,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,OAAO,EAAE,uBAAuB,EAAE,kBAAkB,EAAE,qBAAqB,EAAE,MAAM,8BAA8B,CAAC;AAClH,OAAO,EAAE,0CAA0C,EAAE,MAAM,oCAAoC,CAAC;AAChG,OAAO,EAAE,mBAAmB,EAAE,+BAA+B,EAAE,mBAAmB,EAAE,MAAM,sBAAsB,CAAC;AAEjH,OAAO,EAAE,4BAA4B,EAAE,MAAM,4BAA4B,CAAC;AAE1E,OAAO,EAAE,8BAA8B,EAAE,MAAM,yBAAyB,CAAC;AACzE,OAAO,EAAE,4BAA4B,EAAE,MAAM,eAAe,CAAC;AAG7D,OAAO,EAAE,WAAW,EAAE,kBAAkB,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AAE1E,OAAO,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,2BAA2B,EAAE,MAAM,mBAAmB,CAAC;AAEzG,OAAO,EAAE,0BAA0B,EAAE,qCAAqC,EAAE,iCAAiC,EAAE,+BAA+B,EAAE,uBAAuB,EAAE,8BAA8B,EAAE,MAAM,yBAAyB,CAAC;AAEzO,OAAO,EAAE,4BAA4B,EAAE,MAAM,+BAA+B,CAAC;AAC7E,OAAO,EAAE,2BAA2B,EAAE,0BAA0B,EAAE,MAAM,oBAAoB,CAAC;AAE7F,OAAO,EAAE,kBAAkB,EAAE,YAAY,EAAE,sBAAsB,EAAE,iBAAiB,EAAE,gBAAgB,EAAE,+BAA+B,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;AAClL,OAAO,EAAE,SAAS,EAAE,gBAAgB,EAAE,qBAAqB,EAAE,MAAM,YAAY,CAAC;AAEhF,OAAO,EAAE,gBAAgB,EAAE,4BAA4B,EAAE,kBAAkB,EAAE,kBAAkB,EAAE,MAAM,kBAAkB,CAAC;AAE1H,OAAO,EAAE,4BAA4B,EAAE,0BAA0B,EAAE,4BAA4B,EAAE,iCAAiC,EAAE,4BAA4B,EAAE,mCAAmC,EAAE,2BAA2B,EAAE,4BAA4B,EAAE,6BAA6B,EAAE,+BAA+B,EAAE,iCAAiC,EAAE,MAAM,4BAA4B,CAAC;AAGxY,OAAO,EAAE,0BAA0B,EAAE,MAAM,iCAAiC,CAAC;AAC7E,OAAO,EAAE,oBAAoB,EAAE,MAAM,iCAAiC,CAAC;AACvE,OAAO,EAAE,iCAAiC,EAAE,MAAM,2BAA2B,CAAC;AAE9E,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AACpD,OAAO,EAAE,qBAAqB,EAAE,kCAAkC,EAAE,MAAM,0BAA0B,CAAC;AACrG,OAAO,EAAE,uBAAuB,EAAE,4BAA4B,EAAE,sBAAsB,EAAE,qBAAqB,EAAE,MAAM,yBAAyB,CAAC;AAE/I,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AAEnE,OAAO,EAAE,oCAAoC,EAAE,MAAM,iCAAiC,CAAC;AAEvF,OAAO,EAAE,yBAAyB,EAAE,MAAM,wBAAwB,CAAC;AAEnE,OAAO,EAAE,8BAA8B,EAAE,MAAM,kCAAkC,CAAC;AAClF,OAAO,EAAE,mBAAmB,EAAE,MAAM,mBAAmB,CAAC;AACxD,OAAO,EAAE,uBAAuB,EAAE,uBAAuB,EAAE,uBAAuB,EAAE,+BAA+B,EAAE,MAAM,uBAAuB,CAAC;AACnJ,OAAO,EAAE,wBAAwB,EAAE,MAAM,oBAAoB,CAAC;AAE9D,OAAO,EAAE,gBAAgB,EAAE,8BAA8B,EAAE,gBAAgB,EAAE,2BAA2B,EAAE,MAAM,oBAAoB,CAAC;AACrI,OAAO,EAAE,iCAAiC,EAAE,MAAM,uBAAuB,CAAC;AAC1E,OAAO,EAAE,kBAAkB,EAAE,sBAAsB,EAAE,MAAM,yBAAyB,CAAC;AACrF,OAAO,EAAE,wBAAwB,EAAE,uBAAuB,EAAE,MAAM,+BAA+B,CAAC;AAClG,OAAO,EAAE,0BAA0B,EAAE,MAAM,6BAA6B,CAAC;AACzE,OAAO,EAAE,2BAA2B,EAAE,iCAAiC,EAAE,kCAAkC,EAAE,MAAM,sBAAsB,CAAC;AAC1I,OAAO,EAAE,8BAA8B,EAAE,kCAAkC,EAAE,MAAM,6BAA6B,CAAC;AAEjH,OAAO,EAAE,mBAAmB,EAAE,MAAM,mBAAmB,CAAC;AACxD,OAAO,EAAE,qBAAqB,EAAE,MAAM,gCAAgC,CAAC;AACvE,OAAO,EAAE,0BAA0B,EAAE,2BAA2B,EAAE,uBAAuB,EAAE,MAAM,8BAA8B,CAAC;AAChI,OAAO,EAAE,iBAAiB,EAAE,MAAM,uBAAuB,CAAC;AAC1D,OAAO,EAAE,wBAAwB,EAAE,MAAM,6BAA6B,CAAC;AACvE,OAAO,EAAE,gCAAgC,EAAE,4BAA4B,EAAE,MAAM,iCAAiC,CAAC;AAEjH,OAAO,EAAE,0BAA0B,EAAE,4BAA4B,EAAE,MAAM,8BAA8B,CAAC;AACxG,OAAO,EAAE,2BAA2B,EAAE,MAAM,mBAAmB,CAAC;AAChE,OAAO,EAAE,wBAAwB,EAAE,MAAM,2BAA2B,CAAC;AACrE,OAAO,EAAE,mCAAmC,EAAE,MAAM,uBAAuB,CAAC;AAC5E,OAAO,EAAE,0BAA0B,EAAE,yBAAyB,EAAE,gCAAgC,EAAE,MAAM,uBAAuB,CAAC;AAEhI,OAAO,EAAE,6BAA6B,EAAE,6BAA6B,EAAE,6BAA6B,EAAE,4BAA4B,EAAE,MAAM,iCAAiC,CAAC;AAC5K,OAAO,EAAE,iCAAiC,EAAE,+BAA+B,EAAE,MAAM,4BAA4B,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/evals/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,6BAA6B,EAAE,uBAAuB,EAAE,8BAA8B,EAAE,uCAAuC,EAAE,SAAS,EAAE,yBAAyB,EAAE,0BAA0B,EAAE,cAAc,EAAE,mBAAmB,EAAE,MAAM,cAAc,CAAC;AACtQ,OAAO,EAAE,6BAA6B,EAAE,4BAA4B,EAAE,gBAAgB,EAAE,MAAM,iCAAiC,CAAC;AAIhI,OAAO,EAAE,aAAa,EAAE,oBAAoB,EAAE,MAAM,aAAa,CAAC;AAClE,OAAO,EAAE,sBAAsB,EAAE,sBAAsB,EAAE,4BAA4B,EAAE,MAAM,qBAAqB,CAAC;AACnH,OAAO,EAAE,sBAAsB,EAAE,MAAM,gCAAgC,CAAC;AACxE,OAAO,EAAE,uBAAuB,EAAE,kBAAkB,EAAE,uBAAuB,EAAE,MAAM,yBAAyB,CAAC;AAC/G,OAAO,EAAE,oBAAoB,EAAE,0BAA0B,EAAE,oBAAoB,EAAE,MAAM,yBAAyB,CAAC;AAEjH,OAAO,EAAE,2BAA2B,EAAE,2BAA2B,EAAE,mBAAmB,EAAE,kBAAkB,EAAE,gBAAgB,EAAE,uBAAuB,EAAE,MAAM,kBAAkB,CAAC;AAEhL,OAAO,EAAE,uBAAuB,EAAE,MAAM,+BAA+B,CAAC;AACxE,OAAO,EAAE,6BAA6B,EAAE,8BAA8B,EAAE,oCAAoC,EAAE,+BAA+B,EAAE,MAAM,qBAAqB,CAAC;AAE3K,OAAO,EAAE,sBAAsB,EAAE,MAAM,yBAAyB,CAAC;AACjE,OAAO,EAAE,sBAAsB,EAAE,2BAA2B,EAAE,MAAM,+BAA+B,CAAC;AACpG,OAAO,EAAE,+BAA+B,EAAE,8BAA8B,EAAE,MAAM,0BAA0B,CAAC;AAI3G,OAAO,EAAE,oCAAoC,EAAE,8BAA8B,EAAE,oBAAoB,EAAE,oBAAoB,EAAE,uBAAuB,EAAE,sBAAsB,EAAE,yBAAyB,EAAE,mBAAmB,EAAE,mBAAmB,EAAE,sBAAsB,EAAE,qBAAqB,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AAErV,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,OAAO,EAAE,uBAAuB,EAAE,kBAAkB,EAAE,qBAAqB,EAAE,MAAM,8BAA8B,CAAC;AAClH,OAAO,EAAE,0CAA0C,EAAE,MAAM,oCAAoC,CAAC;AAChG,OAAO,EAAE,mBAAmB,EAAE,+BAA+B,EAAE,mBAAmB,EAAE,MAAM,sBAAsB,CAAC;AAEjH,OAAO,EAAE,4BAA4B,EAAE,MAAM,4BAA4B,CAAC;AAE1E,OAAO,EAAE,8BAA8B,EAAE,MAAM,yBAAyB,CAAC;AACzE,OAAO,EAAE,4BAA4B,EAAE,MAAM,eAAe,CAAC;AAG7D,OAAO,EAAE,WAAW,EAAE,kBAAkB,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AAE1E,OAAO,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,2BAA2B,EAAE,MAAM,mBAAmB,CAAC;AAEzG,OAAO,EAAE,0BAA0B,EAAE,qCAAqC,EAAE,iCAAiC,EAAE,+BAA+B,EAAE,uBAAuB,EAAE,8BAA8B,EAAE,MAAM,yBAAyB,CAAC;AAEzO,OAAO,EAAE,yBAAyB,EAAE,MAAM,iCAAiC,CAAC;AAE5E,OAAO,EAAE,4BAA4B,EAAE,MAAM,+BAA+B,CAAC;AAC7E,OAAO,EAAE,2BAA2B,EAAE,0BAA0B,EAAE,MAAM,oBAAoB,CAAC;AAE7F,OAAO,EAAE,kBAAkB,EAAE,YAAY,EAAE,sBAAsB,EAAE,iBAAiB,EAAE,gBAAgB,EAAE,+BAA+B,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;AAClL,OAAO,EAAE,SAAS,EAAE,gBAAgB,EAAE,qBAAqB,EAAE,MAAM,YAAY,CAAC;AAEhF,OAAO,EAAE,gBAAgB,EAAE,4BAA4B,EAAE,kBAAkB,EAAE,kBAAkB,EAAE,MAAM,kBAAkB,CAAC;AAE1H,OAAO,EAAE,4BAA4B,EAAE,0BAA0B,EAAE,4BAA4B,EAAE,iCAAiC,EAAE,4BAA4B,EAAE,mCAAmC,EAAE,2BAA2B,EAAE,4BAA4B,EAAE,6BAA6B,EAAE,+BAA+B,EAAE,iCAAiC,EAAE,MAAM,4BAA4B,CAAC;AAGxY,OAAO,EAAE,0BAA0B,EAAE,MAAM,iCAAiC,CAAC;AAC7E,OAAO,EAAE,oBAAoB,EAAE,MAAM,iCAAiC,CAAC;AACvE,OAAO,EAAE,iCAAiC,EAAE,MAAM,2BAA2B,CAAC;AAE9E,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AACpD,OAAO,EAAE,qBAAqB,EAAE,kCAAkC,EAAE,MAAM,0BAA0B,CAAC;AACrG,OAAO,EAAE,uBAAuB,EAAE,4BAA4B,EAAE,sBAAsB,EAAE,qBAAqB,EAAE,MAAM,yBAAyB,CAAC;AAE/I,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AAEnE,OAAO,EAAE,oCAAoC,EAAE,MAAM,iCAAiC,CAAC;AAEvF,OAAO,EAAE,yBAAyB,EAAE,MAAM,wBAAwB,CAAC;AAEnE,OAAO,EAAE,8BAA8B,EAAE,MAAM,kCAAkC,CAAC;AAClF,OAAO,EAAE,mBAAmB,EAAE,MAAM,mBAAmB,CAAC;AACxD,OAAO,EAAE,uBAAuB,EAAE,uBAAuB,EAAE,uBAAuB,EAAE,+BAA+B,EAAE,MAAM,uBAAuB,CAAC;AACnJ,OAAO,EAAE,wBAAwB,EAAE,MAAM,oBAAoB,CAAC;AAE9D,OAAO,EAAE,gBAAgB,EAAE,8BAA8B,EAAE,gBAAgB,EAAE,2BAA2B,EAAE,MAAM,oBAAoB,CAAC;AACrI,OAAO,EAAE,iCAAiC,EAAE,MAAM,uBAAuB,CAAC;AAC1E,OAAO,EAAE,kBAAkB,EAAE,sBAAsB,EAAE,MAAM,yBAAyB,CAAC;AACrF,OAAO,EAAE,wBAAwB,EAAE,uBAAuB,EAAE,MAAM,+BAA+B,CAAC;AAClG,OAAO,EAAE,0BAA0B,EAAE,MAAM,6BAA6B,CAAC;AACzE,OAAO,EAAE,2BAA2B,EAAE,iCAAiC,EAAE,kCAAkC,EAAE,MAAM,sBAAsB,CAAC;AAC1I,OAAO,EAAE,8BAA8B,EAAE,kCAAkC,EAAE,MAAM,6BAA6B,CAAC;AAEjH,OAAO,EAAE,mBAAmB,EAAE,MAAM,mBAAmB,CAAC;AACxD,OAAO,EAAE,qBAAqB,EAAE,MAAM,gCAAgC,CAAC;AACvE,OAAO,EAAE,0BAA0B,EAAE,2BAA2B,EAAE,uBAAuB,EAAE,MAAM,8BAA8B,CAAC;AAChI,OAAO,EAAE,iBAAiB,EAAE,MAAM,uBAAuB,CAAC;AAC1D,OAAO,EAAE,wBAAwB,EAAE,MAAM,6BAA6B,CAAC;AACvE,OAAO,EAAE,gCAAgC,EAAE,4BAA4B,EAAE,MAAM,iCAAiC,CAAC;AAEjH,OAAO,EAAE,0BAA0B,EAAE,4BAA4B,EAAE,MAAM,8BAA8B,CAAC;AACxG,OAAO,EAAE,2BAA2B,EAAE,MAAM,mBAAmB,CAAC;AAChE,OAAO,EAAE,wBAAwB,EAAE,MAAM,2BAA2B,CAAC;AACrE,OAAO,EAAE,mCAAmC,EAAE,MAAM,uBAAuB,CAAC;AAC5E,OAAO,EAAE,0BAA0B,EAAE,yBAAyB,EAAE,gCAAgC,EAAE,MAAM,uBAAuB,CAAC;AAEhI,OAAO,EAAE,6BAA6B,EAAE,6BAA6B,EAAE,6BAA6B,EAAE,4BAA4B,EAAE,MAAM,iCAAiC,CAAC;AAC5K,OAAO,EAAE,iCAAiC,EAAE,+BAA+B,EAAE,MAAM,4BAA4B,CAAC"}