@x-otto/eval 0.0.1-alpha.0 → 0.0.1-alpha.2

package/README.md

@@ -1,119 +1,131 @@
 # @x-otto/eval
 
-otto 的自建业务基准（RFC-316 L2 层）：跑一套固定任务，判分，与基线对比，报出回归。
+Otto's in-house business benchmark: run a fixed suite of tasks, score them, compare against a baseline, and report regressions.
 
-## 为什么它长这样
+## Why It's Built This Way
 
-上一代 `@x-otto/eval` 因**零生产接线**被整包删除（RFC-067 M115-03）——代码没错，是没人跑、
-没有东西消费它的输出。所以这一版的第一原则是**先建消费回路，再堆 case**：
+An earlier generation of `@x-otto/eval` was removed in its entirety because it had **zero production wiring** — the
+code wasn't broken, nobody was running it and nothing consumed its output. So this version's first principle is
+**build the consumption loop first, then pile on cases**:
 
-- 报告是给人看的产物，回归判定直接映射到进程退出码（`run` 检出回归即 exit 1，可当门禁）。
-- 全部逻辑只走 otto 的**公开入口**（headless CLI 的 JSON 输出、trace JSONL），
-  引擎里没有一行「为评测而存在」的分支。
+- Reports are artifacts meant for humans to read, and regression verdicts map directly to the process exit code
+  (`run` returning a regression means exit 1, so it can serve as a gate).
+- All logic goes exclusively through otto's **public entry points** (the headless CLI's JSON output, trace JSONL) —
+  there is not a single "exists only for eval" branch inside the engine itself.
 
-## 用法
+## Usage
 
 ```bash
-# 跑整套 case，与基线对比（检出回归 → exit 1）
+# Run the full case suite, compare against the baseline (regression detected → exit 1)
 node packages/eval/dist/cli.js run
 
-# 把当前表现固化为基线（必须给理由）
+# Lock in current behavior as the new baseline (a reason is required)
 node packages/eval/dist/cli.js baseline update --reason "M1 initial baseline"
 
-# 查看已有基线（按 模型 × 套件版本 分键）
+# View existing baselines (keyed by model × suite version)
 node packages/eval/dist/cli.js baseline list
 
-# 少跑几轮做冒烟
+# Run fewer rounds for a smoke test
 node packages/eval/dist/cli.js run --runs 1
 
-# 从一个真实会话提炼 case 骨架（见下「case 从哪来」）
+# Extract a case skeleton from a real session (see "Where cases come from" below)
 node packages/eval/dist/cli.js extract <sessionId>
 ```
 
-## 两条评测轴
+## Two Evaluation Axes
 
-| | Tier B（能力） | Tier A（引擎） |
+| | Tier B (Capability) | Tier A (Engine) |
 | -- | -- | -- |
-| 测什么 | agent 含模型的能力变强/变弱 | 引擎有没有被写坏 |
-| 成本 | 真实 token | 零 |
-| 确定性 | 统计性（n≥3） | 确定性 |
-| 入口 | `cli.js run` | `tier-a-cli.js run` |
-| 适合 | nightly / 大改动后 | pre-push 门禁 |
+| What it tests | Whether the model-inclusive agent's capability improved/regressed | Whether the engine itself was broken |
+| Cost | Real tokens | Zero |
+| Determinism | Statistical (n≥3) | Deterministic |
+| Entry point | `cli.js run` | `tier-a-cli.js run` |
+| Suited for | Nightly / after major changes | pre-push gate |
 
-Tier A 把录制好的模型输出回放给**真实引擎 + 真实 fs 工具**，模型侧被完全固定，
-引擎侧的行为漂移会落在三个断言面上：工具调用序列、文件产物 sha256、turn 终态。
-合法的引擎行为变更走显式重录：`tier-a-cli.js run --update`（并在提交里说明理由）。
+Tier A replays recorded model output against a **real engine + real filesystem tools**. The model side is completely
+fixed, so any drift in engine behavior lands on one of three assertion surfaces: tool call sequence, file artifact
+sha256, and turn final state. Legitimate engine behavior changes go through explicit re-recording:
+`tier-a-cli.js run --update` (with the reason documented in the commit).
 
 ```bash
-node packages/eval/dist/tier-a-cli.js run       # 秒级门禁，mismatch → exit 1
-OTTO_PREPUSH_EVAL=1 git push                    # 挂进 pre-push
-bash scripts/eval-nightly.sh                    # Tier A + Tier B 一起跑（nightly 用）
+node packages/eval/dist/tier-a-cli.js run       # second-level gate, mismatch → exit 1
+OTTO_PREPUSH_EVAL=1 git push                    # wired into pre-push
+bash scripts/eval-nightly.sh                    # runs Tier A + Tier B together (for nightly use)
 ```
 
-## case 从哪来
+## Where Cases Come From
 
-三个渠道，优先级从高到低：
+Three sources, in priority order:
 
-1. **真实事故**：复盘完顺手固化成 case，等于免费买了永久保险。
-2. **真实会话提炼**：`extract <sessionId>` 从 trace 生成骨架（prompt 原文、工具序列、
-   触碰过的文件清单）。刻意**不**自动生成判据、**不**自动脱敏——自动生成的判据只会是
-   "输出像上次一样"（那是快照不是判据），不可靠的脱敏比明示风险更危险。骨架里带 TODO
-   和脱敏清单，人工补完才可用（生成出来是 `judge: liveness`，不进质量分）。
-3. 手工新写。
+1. **Real incidents**: after a postmortem, turn it into a case on the spot — that's essentially permanent insurance
+   bought for free.
+2. **Extracted from real sessions**: `extract <sessionId>` generates a skeleton from a trace (the original prompt,
+   tool call sequence, list of touched files). Judgment criteria and redaction are deliberately **not**
+   auto-generated — an auto-generated judgment criterion would only ever be "output looks like last time" (that's a
+   snapshot, not a judgment), and unreliable auto-redaction is more dangerous than an explicit risk. The skeleton
+   ships with TODOs and a redaction checklist; it's only usable after manual completion (as generated it defaults to
+   `judge: liveness`, which does not count toward the quality score).
+3. Hand-written from scratch.
 
-默认 case 根目录是 `<cwd>/evals`，可用 `--evals-root` 指定。
+The default case root directory is `<cwd>/evals`, overridable with `--evals-root`.
 
-## case 怎么写
+## Writing a Case
 
-一个 case = `evals/cases/<id>/` 目录：
+One case = one `evals/cases/<id>/` directory:
 
 ```
 evals/cases/my-case/
-  case.yaml      # prompt、判分方式、轮数、超时、token 预算
-  fixture/       # 可选：工作区初始状态（每次运行复制到临时目录）
-  verify.sh      # judge=script 时的判分脚本，exit 0 = pass
+  case.yaml      # prompt, scoring method, run count, timeout, token budget
+  fixture/       # optional: initial workspace state (copied to a temp dir on each run)
+  verify.sh      # scoring script when judge=script, exit 0 = pass
 ```
 
 ```yaml
 prompt: |
   Fix the failing test in this directory.
 judge: script          # script | golden | liveness
-runs: 3                # 单次运行不作结论（RFC-316 规则 3）
+runs: 3                # a single run is not conclusive
 timeoutMs: 240000
-maxTokens: 200000      # 成本护栏：超限即停后续运行
-golden:                # 可与 script 并存，两个判据都要满足
+maxTokens: 200000      # cost guardrail: stops subsequent runs once exceeded
+golden:                # can coexist with script; both criteria must pass
   toolSequence: [read, edit]
 ```
 
-## 判分优先级（诚实边界）
+## Scoring Priority (Honest Boundaries)
 
 ```
-可执行判据（verify.sh exit 0）  >  golden 断言  >  liveness（跑完没崩）
+executable criteria (verify.sh exit 0)  >  golden assertions  >  liveness (ran without crashing)
 ```
 
-`liveness` 的 case **不进质量分**——没有判据就说没有，不许用「跑完了」冒充「做对了」。
-报告底部会明确列出有多少 case 属于这一档。
+`liveness` cases **do not count toward the quality score** — if there's no judgment criterion, say so; don't pass off
+"it ran" as "it did the right thing". The report footer explicitly lists how many cases fall into this tier.
 
-`script` 与 `golden` 是正交判据，可同时使用：前者判产物对不对，后者判过程守不守规矩
-（例如"改文件前必须先读"）。只有其一时，两类回归各有一半会蒙混过关。
+`script` and `golden` are orthogonal criteria and can be used together: the former judges whether the output is
+correct, the latter judges whether the process followed the rules (e.g. "must read the file before editing it").
+Using only one of them lets half of each category of regression slip through unnoticed.
 
-## 三态与基线
+## Four States and Baselines
 
-- 每次运行落 `pass` / `fail` / `infra_error` 三态之一。**限流、网络错误、供应商 5xx 归
-  `infra_error`，不进 pass 率**——否则一次 429 会伪装成能力回归。infra 占比超 20% 时整次
-  运行标 invalid，拒绝参与基线对比。
-- 基线按 **`(modelId, suiteVersion)`** 分键。换模型 = 换基线键，宁可报「没有基线」，
-  也不拿 A 模型的分数去审判 B 模型。改 prompt 或判据会改变 `suiteVersion`，同样另起一条曲线。
-- 回归判定两个信号：套件 pass 率相对基线跌超 15 个百分点，或某个 case 从「全 pass」翻到
-  「全 fail」（case flip）。n=3 的统计功效只够抓这个量级的回归，抓不到细微退化——
-  这是首期接受的边界，不假装有统计显著性。
+- Every run lands in one of four states: `pass` / `fail` / `infra_error` / `skipped_budget`. **Rate limits, network
+  errors, and provider 5xx responses all count as `infra_error` and do not count toward the pass rate** — otherwise a
+  single 429 could masquerade as a capability regression. If the infra ratio exceeds 20%, the whole run is flagged
+  invalid and excluded from baseline comparison. `skipped_budget` (runs skipped due to budget exhaustion) likewise
+  does not count toward the pass rate, and is also excluded from the infra-ratio denominator — it's a local guardrail
+  proactively skipping, not a provider/network failure.
+- Baselines are keyed by **`(modelId, suiteVersion)`**. Switching models means switching baseline keys — it's better
+  to report "no baseline available" than to judge model B against model A's scores. Changing the prompt or scoring
+  criteria bumps `suiteVersion`, which likewise starts a new curve.
+- Regression detection relies on two signals: the suite pass rate drops more than 15 percentage points relative to
+  the baseline, or a case flips from "always passing" to "always failing" (a case flip). With n=3, the statistical
+  power is only sufficient to catch regressions at this magnitude — it cannot catch subtle degradation. This is an
+  accepted boundary for the first iteration, not a claim of statistical significance.
 
-## 测试
+## Testing
 
 ```bash
 npx vitest run packages/eval/tests
 ```
 
-`consumption-loop.e2e.test.ts` 是这套体系的自证：它用真实 case 目录、真实 verify 脚本、
-真实工作区隔离，模拟「变更前 / 变更后」两次运行，验证整条链路确实能把引入的退化报出来，
-并在没有退化时保持安静。
+`consumption-loop.e2e.test.ts` is this system's self-proof: it uses a real case directory, a real verify script, and
+real workspace isolation to simulate a "before change / after change" pair of runs, verifying that the whole pipeline
+does report an introduced degradation, and stays quiet when there is none.

package/dist/cli.js

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-import{C as e,S as t,T as n,a as r,c as i,l as a,o,p as s,r as c,s as l,t as u}from"./extract-B65xQZqv.js";import{mkdirSync as d,writeFileSync as f}from"node:fs";import{join as p,resolve as m}from"node:path";import{flushEnvWarnings as h}from"@x-otto/env";function g(e){let t=[],n={evalsRoot:m(process.cwd(),`evals`),reason:``,saveResult:!0};for(let r=0;r<e.length;r++){let i=e[r];i===`--evals-root`?n.evalsRoot=m(e[++r]??``):i===`--runs`?n.runs=Number(e[++r]):i===`--reason`?n.reason=e[++r]??``:i===`--no-save`?n.saveResult=!1:i!==void 0&&t.push(i)}return{command:t.join(` `)||`help`,options:n}}function _(e,t){let n=p(e,`results`);d(n,{recursive:!0});let r=p(n,`${t.startedAt.replace(/[:.]/g,`-`)}-${t.modelId.replace(/[^\w.-]/g,`_`)}.json`);return f(r,`${JSON.stringify(t,null,2)}\n`,`utf8`),r}async function v(){h(e=>process.stderr.write(`${e}\n`));let{command:d,options:f}=g(process.argv.slice(2)),m=p(f.evalsRoot,`cases`),v=p(f.evalsRoot,`baseline.json`);if(d===`baseline list`){let e=l(v);if(e.entries.length===0)return process.stdout.write(`no baselines recorded
+import{C as e,S as t,T as n,a as r,c as i,l as a,o,p as s,r as c,s as l,t as u}from"./extract-CJuoP2bV.js";import{mkdirSync as d,writeFileSync as f}from"node:fs";import{join as p,resolve as m}from"node:path";import{flushEnvWarnings as h}from"@x-otto/env";function g(e){let t=[],n={evalsRoot:m(process.cwd(),`evals`),reason:``,saveResult:!0};for(let r=0;r<e.length;r++){let i=e[r];i===`--evals-root`?n.evalsRoot=m(e[++r]??``):i===`--runs`?n.runs=Number(e[++r]):i===`--reason`?n.reason=e[++r]??``:i===`--no-save`?n.saveResult=!1:i!==void 0&&t.push(i)}return{command:t.join(` `)||`help`,options:n}}function _(e,t){let n=p(e,`results`);d(n,{recursive:!0});let r=p(n,`${t.startedAt.replace(/[:.]/g,`-`)}-${t.modelId.replace(/[^\w.-]/g,`_`)}.json`);return f(r,`${JSON.stringify(t,null,2)}\n`,`utf8`),r}async function v(){h(e=>process.stderr.write(`${e}\n`));let{command:d,options:f}=g(process.argv.slice(2)),m=p(f.evalsRoot,`cases`),v=p(f.evalsRoot,`baseline.json`);if(d===`baseline list`){let e=l(v);if(e.entries.length===0)return process.stdout.write(`no baselines recorded
 `),0;for(let t of e.entries)process.stdout.write(`${t.modelId}  suite=${t.suiteVersion}  pass=${t.overallPassRate===void 0?`—`:`${(t.overallPassRate*100).toFixed(0)}%`}  ${t.recordedAt}  (${t.reason})\n`);return 0}if(d.startsWith(`extract`)){let e=d.split(/\s+/)[1];if(!e)return process.stderr.write(`usage: otto-eval extract <sessionId> [--evals-root <dir>]
 `),2;let{caseDir:t,skeleton:n}=u(e,m);return process.stdout.write(`case skeleton created: ${t}\n  turns=${n.turns} toolCalls=${n.toolSequence.length} touchedPaths=${n.touchedPaths.length}\nnext: read ${t}/EXTRACTION.md — the case is NOT usable until you add a judge and pass the sanitisation checklist.\n`),0}if(d!==`run`&&d!==`baseline update`)return process.stdout.write(`usage:
   otto-eval run [--evals-root <dir>] [--runs <n>] [--no-save]

package/dist/{extract-B65xQZqv.js → extract-CJuoP2bV.js}

@@ -1,5 +1,5 @@
 import{t as e}from"./workspace-C_kTWfF8.js";import{createHash as t}from"node:crypto";import{existsSync as n,mkdirSync as r,readFileSync as i,readdirSync as a,statSync as o,writeFileSync as s}from"node:fs";import{dirname as c,join as l}from"node:path";import{parse as u}from"yaml";import{spawn as d,spawnSync as f}from"node:child_process";import{OTTO_SESSIONS_DIR as p,normalizeEnv as m,normalizeEnvFraction as h}from"@x-otto/env";const g=[`script`,`golden`,`liveness`];function _(e,t,n){let r=e[t];if(typeof r!=`string`||r.trim()===``)throw Error(`case '${n}': field '${t}' must be a non-empty string`);return r}function v(e,t,n,r){let i=e[t];if(i===void 0)return r;if(typeof i!=`number`||!Number.isInteger(i)||i<=0)throw Error(`case '${n}': field '${t}' must be a positive integer`);return i}function y(e,t){if(typeof e!=`object`||!e)throw Error(`case '${t}': judge='golden' requires a 'golden' mapping`);let n=e,r={},i=n.answerContains;if(i!==void 0){if(!Array.isArray(i)||i.some(e=>typeof e!=`string`))throw Error(`case '${t}': golden.answerContains must be a string array`);Object.assign(r,{answerContains:i})}let a=n.toolSequence;if(a!==void 0){if(!Array.isArray(a)||a.some(e=>typeof e!=`string`))throw Error(`case '${t}': golden.toolSequence must be a string array`);Object.assign(r,{toolSequence:a})}if(!r.answerContains&&!r.toolSequence)throw Error(`case '${t}': judge='golden' requires at least one of golden.answerContains / golden.toolSequence`);return r}function b(e){let t=e.split(`/`).filter(Boolean).at(-1)??e,r=l(e,`case.yaml`);if(!n(r))throw Error(`case '${t}': missing case.yaml at ${r}`);let a=u(i(r,`utf8`));if(typeof a!=`object`||!a)throw Error(`case '${t}': case.yaml must contain a mapping`);let o=a,s=_(o,`prompt`,t),c=_(o,`judge`,t);if(!g.includes(c))throw Error(`case '${t}': judge must be one of ${g.join(` | `)}`);let d=c,f={id:t,prompt:s,judge:d,runs:v(o,`runs`,t,3),timeoutMs:v(o,`timeoutMs`,t,3e5),dir:e,hasFixture:n(l(e,`fixture`))};if(o.maxTokens!==void 0&&Object.assign(f,{maxTokens:v(o,`maxTokens`,t,0)}),d===`script`){let r=o.verifyScript===void 0?`verify.sh`:_(o,`verifyScript`,t);if(!n(l(e,r)))throw Error(`case '${t}': judge='script' but verify script '${r}' not found`);Object.assign(f,{verifyScript:r})}return(d===`golden`||o.golden!==void 0)&&Object.assign(f,{golden:y(o.golden,t)}),f}function x(e){if(!n(e))throw Error(`cases root not found: ${e}`);return a(e).filter(e=>!e.startsWith(`.`)).map(t=>l(e,t)).filter(e=>o(e).isDirectory()).sort().map(b)}function S(e){let n=e.map(e=>({id:e.id,prompt:e.prompt,judge:e.judge,golden:e.golden??null}));return t(`sha256`).update(JSON.stringify(n)).digest(`hex`).slice(0,12)}function C(e){let t=e.toLowerCase();return[`rate limit`,`rate_limit`,`429`,`overloaded`,`econnreset`,`etimedout`,`enotfound`,`socket hang up`,`service unavailable`,`internal server error`,`bad gateway`,`no model configured`,`command not found`,`enoent`].some(e=>t.includes(e))?`infra`:`failed`}function w(e={}){let t=e.bin??`otto`;return{invoke(n){return new Promise(r=>{let i=d(t,[`--json`,n.prompt],{cwd:n.cwd,env:{...process.env,...e.env},stdio:[`ignore`,`pipe`,`pipe`]}),a=``,o=``,s=!1,c=e=>{s||(s=!0,clearTimeout(l),r(e))},l=setTimeout(()=>{i.kill(`SIGKILL`),c({kind:`failed`,detail:`timeout after ${n.timeoutMs}ms`})},n.timeoutMs);i.stdout.on(`data`,e=>{a+=e.toString()}),i.stderr.on(`data`,e=>{o+=e.toString()}),i.on(`error`,e=>{c({kind:`infra`,detail:`spawn failed: ${e.message}`})}),i.on(`close`,e=>{if(e!==0){c({kind:C(o),detail:`exit ${e}: ${o.trim().slice(0,500)}`});return}try{let e=JSON.parse(a);if(typeof e.sessionId!=`string`){c({kind:`failed`,detail:`json output missing sessionId`});return}c({kind:`ok`,result:e})}catch{c({kind:`failed`,detail:`unparsable json output: ${a.slice(0,300)}`})}})})}}}function T(e,t){let n=e?.[t];return typeof n==`number`&&Number.isFinite(n)?n:0}function E(e){return l(p,`trace`,`${e}.jsonl`)}function D(e){let t=[];return{metrics:O(e,t),toolNames:t}}function O(e,t){let n={turns:0,toolCalls:0,toolErrors:0,inputTokens:0,outputTokens:0,cacheReadTokens:0,toolDurationMs:0};for(let r of e.split(`
-`)){if(r.trim()===``)continue;let e;try{e=JSON.parse(r)}catch{continue}if(e.node===`tool.call.end`){n.toolCalls++,n.toolDurationMs+=T(e.payload,`durationMs`),e.payload?.errorCategory!==void 0&&n.toolErrors++;let r=e.payload?.name;typeof r==`string`&&t?.push(r);continue}if(e.node===`turn.end`){n.turns++;let t=e.payload?.usage;if(typeof t==`object`&&t){let e=t;n.inputTokens+=T(e,`inputTokens`),n.outputTokens+=T(e,`outputTokens`),n.cacheReadTokens+=T(e,`cacheReadTokens`)}let r=e.payload?.model;typeof r==`string`&&r!==``&&(n.modelId=r)}}return n}function k(e){let t=E(e);if(n(t))return D(i(t,`utf8`))}function A(e){if(e.length===0)return;let t=e.reduce((e,t)=>({turns:e.turns+t.turns,toolCalls:e.toolCalls+t.toolCalls,toolErrors:e.toolErrors+t.toolErrors,inputTokens:e.inputTokens+t.inputTokens,outputTokens:e.outputTokens+t.outputTokens,cacheReadTokens:e.cacheReadTokens+t.cacheReadTokens,toolDurationMs:e.toolDurationMs+t.toolDurationMs,modelId:t.modelId??e.modelId}),{turns:0,toolCalls:0,toolErrors:0,inputTokens:0,outputTokens:0,cacheReadTokens:0,toolDurationMs:0}),n=e.length,r={turns:t.turns/n,toolCalls:t.toolCalls/n,toolErrors:t.toolErrors/n,inputTokens:t.inputTokens/n,outputTokens:t.outputTokens/n,cacheReadTokens:t.cacheReadTokens/n,toolDurationMs:t.toolDurationMs/n};return t.modelId!==void 0&&(r.modelId=t.modelId),r}function j(e,t){let n=0;for(let r of e)n<t.length&&r===t[n]&&n++;return n===t.length}function M(e,t,n){let r=[];if(e.answerContains){let n=t.toLowerCase();for(let t of e.answerContains)n.includes(t.toLowerCase())||r.push(`answer missing '${t}'`)}return e.toolSequence&&!j(n,e.toolSequence)&&r.push(`tool sequence [${e.toolSequence.join(`, `)}] not satisfied by [${n.join(`, `)}]`),r.length===0?{passed:!0,detail:`golden assertions satisfied`}:{passed:!1,detail:r.join(`; `)}}function N(e,t){let n=f(`bash`,[t],{cwd:e.workspaceDir,encoding:`utf8`,timeout:12e4,env:{...process.env,OTTO_EVAL_RESPONSE:e.response}});if(n.error)return{passed:!1,detail:`verify script error: ${n.error.message}`};let r=`${(n.stdout??``).trim()}${n.stderr?` | ${n.stderr.trim()}`:``}`.slice(0,500);return{passed:n.status===0,detail:r||`exit ${n.status}`}}function P(e){let{spec:t}=e,n=[];if(t.judge===`script`&&n.push(N(e,l(t.dir,t.verifyScript??`verify.sh`))),t.golden&&n.push(M(t.golden,e.response,e.toolNames)),n.length===0)return{passed:!0,detail:`liveness only (no quality judgement)`};let r=n.filter(e=>!e.passed);return r.length===0?{passed:!0,detail:n.map(e=>e.detail).join(`; `)}:{passed:!1,detail:r.map(e=>e.detail).join(`; `)}}const F=h(process.env.OTTO_EVAL_INFRA_ERROR_THRESHOLD,.2);function I(e){return e?e.inputTokens+e.outputTokens:0}async function L(t,n){let r=n.now??(()=>Date.now()),i=n.readTrace??k,a=[],o=0,s=0,c=0,l=()=>{if(t.maxTokens===void 0)return!1;let e=s>0?o/s:t.maxTokens/t.runs;return o+c*e>=t.maxTokens};for(let u=0;u<t.runs;u++){if(l()){a.push({outcome:`infra_error`,wallMs:0,detail:`budget exhausted: ${o} measured tokens${c>0?` + ${c} unmeasured run(s)`:``} vs limit ${t.maxTokens}, remaining runs skipped`});continue}let d=e(t),f=r();try{let e=await n.invoker.invoke({prompt:t.prompt,cwd:d.dir,timeoutMs:t.timeoutMs}),l=r()-f;if(e.kind===`infra`){c++,a.push({outcome:`infra_error`,wallMs:l,detail:e.detail});continue}if(e.kind===`failed`){c++,a.push({outcome:`fail`,wallMs:l,detail:e.detail});continue}let u=i(e.result.sessionId),p=P({spec:t,workspaceDir:d.dir,response:e.result.response,toolNames:u?.toolNames??[]});u?(o+=I(u.metrics),s++):c++;let m={outcome:p.passed?`pass`:`fail`,wallMs:l,sessionId:e.result.sessionId,detail:p.detail};u&&(m.metrics=u.metrics),a.push(m)}finally{d.dispose()}let p=a.at(-1);p&&n.onRunComplete?.(t.id,u,p)}return R(t,a)}function R(e,t){let n=t.filter(e=>e.outcome===`infra_error`).length,r=t.length-n,i=t.filter(e=>e.outcome===`pass`).length,a=e.judge!==`liveness`,o={caseId:e.id,judge:e.judge,runs:t,passed:i,effective:r,infraErrors:n,qualityScored:a};return r>0&&Object.assign(o,{passRate:i/r}),o}async function z(e,t,n){let r=n.now??(()=>Date.now()),i=new Date(r()).toISOString(),a=[];for(let t of e)a.push(await L(t,n));return B(a,t,i)}function B(e,t,n){let r=e.filter(e=>e.qualityScored&&e.passRate!==void 0),i=e.reduce((e,t)=>e+t.runs.length,0),a=e.reduce((e,t)=>e+t.infraErrors,0),o=i===0?0:a/i,s=A(e.flatMap(e=>e.runs.map(e=>e.metrics).filter(e=>e!==void 0))),c={suiteVersion:t,modelId:s?.modelId??`unknown`,startedAt:n,cases:e,scoredCases:e.filter(e=>e.qualityScored).length,unscoredCases:e.filter(e=>!e.qualityScored).length,infraErrorRate:o,invalid:o>F};if(r.length>0){let e=r.reduce((e,t)=>e+(t.passRate??0),0);Object.assign(c,{overallPassRate:e/r.length})}return s&&Object.assign(c,{aggregateMetrics:s}),c}const V=m(process.env.OTTO_EVAL_REGRESSION_PP,15);function H(e){return n(e)?{entries:JSON.parse(i(e,`utf8`)).entries??[]}:{entries:[]}}function U(e,t){r(c(e),{recursive:!0}),s(e,`${JSON.stringify(t,null,2)}\n`,`utf8`)}function W(e,t,n){return e.entries.find(e=>e.modelId===t&&e.suiteVersion===n)}function G(e,t,n){let r={};for(let e of t.cases)r[e.caseId]=e.passRate;let i={modelId:t.modelId,suiteVersion:t.suiteVersion,recordedAt:t.startedAt,reason:n,casePassRates:r};return t.overallPassRate!==void 0&&Object.assign(i,{overallPassRate:t.overallPassRate}),{entries:[...e.entries.filter(e=>!(e.modelId===i.modelId&&e.suiteVersion===i.suiteVersion)),i]}}function K(e,t){if(e.invalid)return{status:`invalid_run`,findings:[`infra_error rate ${(e.infraErrorRate*100).toFixed(1)}% exceeds threshold — run not comparable`],flippedCases:[]};if(!t)return{status:`no_baseline`,findings:[`no baseline for (model=${e.modelId}, suite=${e.suiteVersion}) — record one with \`baseline update\``],flippedCases:[]};let n=[],r=[];for(let i of e.cases){if(!i.qualityScored)continue;let e=t.casePassRates[i.caseId];if(e===1&&i.passRate===0){r.push(i.caseId),n.push(`case flip: '${i.caseId}' was 100% pass, now 0%`);continue}i.passRate===void 0&&e!==void 0&&n.push(`'${i.caseId}' produced no comparable runs (all infra errors) — excluded from the suite average`)}let i;e.overallPassRate!==void 0&&t.overallPassRate!==void 0&&(i=(e.overallPassRate-t.overallPassRate)*100,i<-V&&n.push(`suite pass rate dropped ${Math.abs(i).toFixed(1)}pp (${(t.overallPassRate*100).toFixed(1)}% → ${(e.overallPassRate*100).toFixed(1)}%)`));let a={status:r.length>0||i!==void 0&&i<-V?`regressed`:`ok`,findings:n.length>0?n:[`no regression detected`],flippedCases:r};return i!==void 0&&Object.assign(a,{passRateDeltaPp:i}),a}function q(e){return e===void 0?`—`:`${(e*100).toFixed(0)}%`}function J(e,t){let n=[],r=t.status===`regressed`?`REGRESSED`:t.status===`ok`?`OK`:t.status===`invalid_run`?`INVALID (infra errors)`:`NO BASELINE`;n.push(`otto eval — ${r}`),n.push(`model=${e.modelId}  suite=${e.suiteVersion}  at=${e.startedAt}`),n.push(``),n.push(`case                              judge     pass   runs  infra`);for(let t of e.cases){let e=t.caseId.padEnd(33).slice(0,33),r=t.judge.padEnd(9),i=(t.qualityScored?q(t.passRate):`n/a`).padStart(5),a=`${t.passed}/${t.effective}`.padStart(6),o=String(t.infraErrors).padStart(6);n.push(`${e} ${r} ${i} ${a} ${o}`)}if(n.push(``),n.push(`suite pass rate: ${q(e.overallPassRate)}  (scored ${e.scoredCases}, liveness-only ${e.unscoredCases})`),t.passRateDeltaPp!==void 0){let e=t.passRateDeltaPp>=0?`+`:``;n.push(`vs baseline:     ${e}${t.passRateDeltaPp.toFixed(1)}pp`)}n.push(`infra errors:    ${(e.infraErrorRate*100).toFixed(1)}% of runs`);let i=e.aggregateMetrics;i&&n.push(`avg per run:     ${i.turns.toFixed(1)} turns · ${i.toolCalls.toFixed(1)} tools (${i.toolErrors.toFixed(1)} err) · ${Math.round(i.inputTokens+i.outputTokens)} tokens`),n.push(``);for(let e of t.findings)n.push(`- ${e}`);return e.unscoredCases>0&&(n.push(``),n.push(`note: ${e.unscoredCases} case(s) have no machine judgement (liveness only) and do not affect the suite pass rate.`)),n.join(`
+`)){if(r.trim()===``)continue;let e;try{e=JSON.parse(r)}catch{continue}if(e.node===`tool.call.end`){n.toolCalls++,n.toolDurationMs+=T(e.payload,`durationMs`),e.payload?.errorCategory!==void 0&&n.toolErrors++;let r=e.payload?.name;typeof r==`string`&&t?.push(r);continue}if(e.node===`turn.end`){n.turns++;let t=e.payload?.usage;if(typeof t==`object`&&t){let e=t;n.inputTokens+=T(e,`inputTokens`),n.outputTokens+=T(e,`outputTokens`),n.cacheReadTokens+=T(e,`cacheReadTokens`)}let r=e.payload?.model;typeof r==`string`&&r!==``&&(n.modelId=r)}}return n}function k(e){let t=E(e);if(n(t))return D(i(t,`utf8`))}function A(e){if(e.length===0)return;let t=e.reduce((e,t)=>({turns:e.turns+t.turns,toolCalls:e.toolCalls+t.toolCalls,toolErrors:e.toolErrors+t.toolErrors,inputTokens:e.inputTokens+t.inputTokens,outputTokens:e.outputTokens+t.outputTokens,cacheReadTokens:e.cacheReadTokens+t.cacheReadTokens,toolDurationMs:e.toolDurationMs+t.toolDurationMs,modelId:t.modelId??e.modelId}),{turns:0,toolCalls:0,toolErrors:0,inputTokens:0,outputTokens:0,cacheReadTokens:0,toolDurationMs:0}),n=e.length,r={turns:t.turns/n,toolCalls:t.toolCalls/n,toolErrors:t.toolErrors/n,inputTokens:t.inputTokens/n,outputTokens:t.outputTokens/n,cacheReadTokens:t.cacheReadTokens/n,toolDurationMs:t.toolDurationMs/n};return t.modelId!==void 0&&(r.modelId=t.modelId),r}function j(e,t){let n=0;for(let r of e)n<t.length&&r===t[n]&&n++;return n===t.length}function M(e,t,n){let r=[];if(e.answerContains){let n=t.toLowerCase();for(let t of e.answerContains)n.includes(t.toLowerCase())||r.push(`answer missing '${t}'`)}return e.toolSequence&&!j(n,e.toolSequence)&&r.push(`tool sequence [${e.toolSequence.join(`, `)}] not satisfied by [${n.join(`, `)}]`),r.length===0?{passed:!0,detail:`golden assertions satisfied`}:{passed:!1,detail:r.join(`; `)}}function N(e,t){let n=f(`bash`,[t],{cwd:e.workspaceDir,encoding:`utf8`,timeout:12e4,env:{...process.env,OTTO_EVAL_RESPONSE:e.response}});if(n.error)return{passed:!1,detail:`verify script error: ${n.error.message}`};let r=`${(n.stdout??``).trim()}${n.stderr?` | ${n.stderr.trim()}`:``}`.slice(0,500);return{passed:n.status===0,detail:r||`exit ${n.status}`}}function P(e){let{spec:t}=e,n=[];if(t.judge===`script`&&n.push(N(e,l(t.dir,t.verifyScript??`verify.sh`))),t.golden&&n.push(M(t.golden,e.response,e.toolNames)),n.length===0)return{passed:!0,detail:`liveness only (no quality judgement)`};let r=n.filter(e=>!e.passed);return r.length===0?{passed:!0,detail:n.map(e=>e.detail).join(`; `)}:{passed:!1,detail:r.map(e=>e.detail).join(`; `)}}const F=h(process.env.OTTO_EVAL_INFRA_ERROR_THRESHOLD,.2);function I(e){return e?e.inputTokens+e.outputTokens:0}async function L(t,n){let r=n.now??(()=>Date.now()),i=n.readTrace??k,a=[],o=0,s=0,c=0,l=()=>{if(t.maxTokens===void 0)return!1;let e=s>0?o/s:t.maxTokens/t.runs;return o+c*e>=t.maxTokens};for(let u=0;u<t.runs;u++){if(l()){a.push({outcome:`skipped_budget`,wallMs:0,detail:`budget exhausted: ${o} measured tokens${c>0?` + ${c} unmeasured run(s)`:``} vs limit ${t.maxTokens}, remaining runs skipped`});continue}let d=e(t),f=r();try{let e=await n.invoker.invoke({prompt:t.prompt,cwd:d.dir,timeoutMs:t.timeoutMs}),l=r()-f;if(e.kind===`infra`){c++,a.push({outcome:`infra_error`,wallMs:l,detail:e.detail});continue}if(e.kind===`failed`){c++,a.push({outcome:`fail`,wallMs:l,detail:e.detail});continue}let u=i(e.result.sessionId),p=P({spec:t,workspaceDir:d.dir,response:e.result.response,toolNames:u?.toolNames??[]});u?(o+=I(u.metrics),s++):c++;let m={outcome:p.passed?`pass`:`fail`,wallMs:l,sessionId:e.result.sessionId,detail:p.detail};u&&(m.metrics=u.metrics),a.push(m)}finally{d.dispose()}let p=a.at(-1);p&&n.onRunComplete?.(t.id,u,p)}return R(t,a)}function R(e,t){let n=t.filter(e=>e.outcome===`infra_error`).length,r=t.filter(e=>e.outcome===`skipped_budget`).length,i=t.length-n-r,a=t.filter(e=>e.outcome===`pass`).length,o=e.judge!==`liveness`,s={caseId:e.id,judge:e.judge,runs:t,passed:a,effective:i,infraErrors:n,skippedBudget:r,qualityScored:o};return i>0&&Object.assign(s,{passRate:a/i}),s}async function z(e,t,n){let r=n.now??(()=>Date.now()),i=new Date(r()).toISOString(),a=[];for(let t of e)a.push(await L(t,n));return B(a,t,i)}function B(e,t,n){let r=e.filter(e=>e.qualityScored&&e.passRate!==void 0),i=e.reduce((e,t)=>e+t.skippedBudget,0),a=e.reduce((e,t)=>e+t.runs.length,0)-i,o=e.reduce((e,t)=>e+t.infraErrors,0),s=a===0?0:o/a,c=A(e.flatMap(e=>e.runs.map(e=>e.metrics).filter(e=>e!==void 0))),l={suiteVersion:t,modelId:c?.modelId??`unknown`,startedAt:n,cases:e,scoredCases:e.filter(e=>e.qualityScored).length,unscoredCases:e.filter(e=>!e.qualityScored).length,infraErrorRate:s,skippedBudgetTotal:i,invalid:s>F};if(r.length>0){let e=r.reduce((e,t)=>e+(t.passRate??0),0);Object.assign(l,{overallPassRate:e/r.length})}return c&&Object.assign(l,{aggregateMetrics:c}),l}const V=m(process.env.OTTO_EVAL_REGRESSION_PP,15);function H(e){return n(e)?{entries:JSON.parse(i(e,`utf8`)).entries??[]}:{entries:[]}}function U(e,t){r(c(e),{recursive:!0}),s(e,`${JSON.stringify(t,null,2)}\n`,`utf8`)}function W(e,t,n){return e.entries.find(e=>e.modelId===t&&e.suiteVersion===n)}function G(e,t,n){let r={};for(let e of t.cases)r[e.caseId]=e.passRate;let i={modelId:t.modelId,suiteVersion:t.suiteVersion,recordedAt:t.startedAt,reason:n,casePassRates:r};return t.overallPassRate!==void 0&&Object.assign(i,{overallPassRate:t.overallPassRate}),{entries:[...e.entries.filter(e=>!(e.modelId===i.modelId&&e.suiteVersion===i.suiteVersion)),i]}}function K(e,t){if(e.invalid)return{status:`invalid_run`,findings:[`infra_error rate ${(e.infraErrorRate*100).toFixed(1)}% exceeds threshold — run not comparable`],flippedCases:[]};if(!t)return{status:`no_baseline`,findings:[`no baseline for (model=${e.modelId}, suite=${e.suiteVersion}) — record one with \`baseline update\``],flippedCases:[]};let n=[],r=[];for(let i of e.cases){if(!i.qualityScored)continue;let e=t.casePassRates[i.caseId];if(e===1&&i.passRate===0){r.push(i.caseId),n.push(`case flip: '${i.caseId}' was 100% pass, now 0%`);continue}i.passRate===void 0&&e!==void 0&&n.push(`'${i.caseId}' produced no comparable runs (all infra errors) — excluded from the suite average`)}let i;e.overallPassRate!==void 0&&t.overallPassRate!==void 0&&(i=(e.overallPassRate-t.overallPassRate)*100,i<-V&&n.push(`suite pass rate dropped ${Math.abs(i).toFixed(1)}pp (${(t.overallPassRate*100).toFixed(1)}% → ${(e.overallPassRate*100).toFixed(1)}%)`));let a={status:r.length>0||i!==void 0&&i<-V?`regressed`:`ok`,findings:n.length>0?n:[`no regression detected`],flippedCases:r};return i!==void 0&&Object.assign(a,{passRateDeltaPp:i}),a}function q(e){return e===void 0?`—`:`${(e*100).toFixed(0)}%`}function J(e,t){let n=[],r=t.status===`regressed`?`REGRESSED`:t.status===`ok`?`OK`:t.status===`invalid_run`?`INVALID (infra errors)`:`NO BASELINE`;n.push(`otto eval — ${r}`),n.push(`model=${e.modelId}  suite=${e.suiteVersion}  at=${e.startedAt}`),n.push(``),n.push(`case                              judge     pass   runs  infra  skip`);for(let t of e.cases){let e=t.caseId.padEnd(33).slice(0,33),r=t.judge.padEnd(9),i=(t.qualityScored?q(t.passRate):`n/a`).padStart(5),a=`${t.passed}/${t.effective}`.padStart(6),o=String(t.infraErrors).padStart(6),s=String(t.skippedBudget).padStart(5);n.push(`${e} ${r} ${i} ${a} ${o} ${s}`)}if(n.push(``),n.push(`suite pass rate: ${q(e.overallPassRate)}  (scored ${e.scoredCases}, liveness-only ${e.unscoredCases})`),t.passRateDeltaPp!==void 0){let e=t.passRateDeltaPp>=0?`+`:``;n.push(`vs baseline:     ${e}${t.passRateDeltaPp.toFixed(1)}pp`)}n.push(`infra errors:    ${(e.infraErrorRate*100).toFixed(1)}% of runs`),e.skippedBudgetTotal>0&&n.push(`skipped (budget): ${e.skippedBudgetTotal} run(s) — suite incomplete for those cases`);let i=e.aggregateMetrics;i&&n.push(`avg per run:     ${i.turns.toFixed(1)} turns · ${i.toolCalls.toFixed(1)} tools (${i.toolErrors.toFixed(1)} err) · ${Math.round(i.inputTokens+i.outputTokens)} tokens`),n.push(``);for(let e of t.findings)n.push(`- ${e}`);return e.unscoredCases>0&&(n.push(``),n.push(`note: ${e.unscoredCases} case(s) have no machine judgement (liveness only) and do not affect the suite pass rate.`)),n.join(`
 `)}const Y=[`path`,`filePath`,`file_path`];function X(e,t){let n,r=[],i=new Set,a=0;for(let e of t.split(`
 `)){if(e.trim()===``)continue;let t;try{t=JSON.parse(e)}catch{continue}if(t.node===`prompt.before`&&n===void 0){let e=t.payload?.text;typeof e==`string`&&e.trim()!==``&&(n=e);continue}if(t.node===`tool.call.start`){let e=t.payload?.name;typeof e==`string`&&r.push(e);let n=t.payload?.arguments;if(typeof n==`object`&&n)for(let e of Y){let t=n[e];typeof t==`string`&&t!==``&&i.add(t)}continue}t.node===`turn.end`&&a++}return{sessionId:e,toolSequence:r,touchedPaths:[...i].sort(),turns:a,...n===void 0?{}:{prompt:n}}}function Z(e){let t=e.prompt===void 0?`prompt: |
   TODO: trace 里没有 prompt.before 文本，请手工填写`:`prompt: |\n${e.prompt.split(`
@@ -36,4 +36,4 @@ ${t}
 - [ ] 路径不含本机用户名等机器指纹（用相对路径）
 - [ ] verify 判据已由人工编写（不要用"输出像上次一样"的快照当判据）
 `}function $(e,t,i=ee){let a=i(e);if(a===void 0)throw Error(`no trace found for session ${e} (looked at ${E(e)})`);let o=X(e,a),c=l(t,`extracted-${e.slice(0,8)}`);if(n(c))throw Error(`${c} already exists — refusing to overwrite (delete it first if intended)`);return r(l(c,`fixture`),{recursive:!0}),s(l(c,`case.yaml`),Z(o),`utf8`),s(l(c,`EXTRACTION.md`),Q(o),`utf8`),{caseDir:c,skeleton:o}}function ee(e){let t=E(e);return n(t)?i(t,`utf8`):void 0}export{S as C,w as S,x as T,A as _,W as a,E as b,U as c,B as d,L as f,P as g,j as h,V as i,G as l,R as m,X as n,K as o,z as p,J as r,H as s,$ as t,F as u,D as v,b as w,C as x,k as y};
-//# sourceMappingURL=extract-B65xQZqv.js.map
+//# sourceMappingURL=extract-CJuoP2bV.js.map

package/dist/extract-CJuoP2bV.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"extract-CJuoP2bV.js","names":["parseYaml","renderReport"],"sources":["../src/case-loader.ts","../src/otto-invoker.ts","../src/trace-metrics.ts","../src/verify.ts","../src/runner.ts","../src/baseline.ts","../src/report.ts","../src/extract.ts"],"sourcesContent":["/**\n * case-loader.ts —— 读取并校验 `evals/cases/<id>/case.yaml`（RFC-316 D4）。\n *\n * 校验失败一律抛错而非静默降级：一个格式错的 case 静默变成 liveness case，\n * 会让 scorecard 悄悄失去质量信号——这正是 RFC-316 规则 2 要防的「假质量分」。\n */\n\nimport { createHash } from 'node:crypto'\nimport { readFileSync, readdirSync, existsSync, statSync } from 'node:fs'\nimport { join } from 'node:path'\nimport { parse as parseYaml } from 'yaml'\n\nimport type { CaseSpec, GoldenSpec, JudgeKind } from './types'\n\nconst DEFAULT_RUNS = 3\nconst DEFAULT_TIMEOUT_MS = 300_000\nconst DEFAULT_VERIFY_SCRIPT = 'verify.sh'\n\nconst JUDGE_KINDS: readonly JudgeKind[] = ['script', 'golden', 'liveness']\n\nfunction requireString(raw: Record<string, unknown>, key: string, caseId: string): string {\n  const value = raw[key]\n  if (typeof value !== 'string' || value.trim() === '') {\n    throw new Error(`case '${caseId}': field '${key}' must be a non-empty string`)\n  }\n  return value\n}\n\nfunction optionalPositiveInt(\n  raw: Record<string, unknown>,\n  key: string,\n  caseId: string,\n  fallback: number,\n): number {\n  const value = raw[key]\n  if (value === undefined) {\n    return fallback\n  }\n  if (typeof value !== 'number' || !Number.isInteger(value) || value <= 0) {\n    throw new Error(`case '${caseId}': field '${key}' must be a positive integer`)\n  }\n  return value\n}\n\nfunction parseGolden(raw: unknown, caseId: string): GoldenSpec {\n  if (typeof raw !== 'object' || raw === null) {\n    throw new Error(`case '${caseId}': judge='golden' requires a 'golden' mapping`)\n  }\n  const record = raw as Record<string, unknown>\n  const golden: GoldenSpec = {}\n  const answerContains = record['answerContains']\n  if (answerContains !== undefined) {\n    if (!Array.isArray(answerContains) || answerContains.some((x) => typeof x !== 'string')) {\n      throw new Error(`case '${caseId}': golden.answerContains must be a string array`)\n    }\n    Object.assign(golden, { answerContains: answerContains as string[] })\n  }\n  const toolSequence = record['toolSequence']\n  if (toolSequence !== undefined) {\n    if (!Array.isArray(toolSequence) || toolSequence.some((x) => typeof x !== 'string')) {\n      throw new Error(`case '${caseId}': golden.toolSequence must be a string array`)\n    }\n    Object.assign(golden, { toolSequence: toolSequence as string[] })\n  }\n  if (!golden.answerContains && !golden.toolSequence) {\n    throw new Error(\n      `case '${caseId}': judge='golden' requires at least one of golden.answerContains / golden.toolSequence`,\n    )\n  }\n  return golden\n}\n\n/** 解析单个 case 目录。`dir` 必须已存在且含 case.yaml。 */\nexport function loadCase(dir: string): CaseSpec {\n  const caseId = dir.split('/').filter(Boolean).at(-1) ?? dir\n  const yamlPath = join(dir, 'case.yaml')\n  if (!existsSync(yamlPath)) {\n    throw new Error(`case '${caseId}': missing case.yaml at ${yamlPath}`)\n  }\n\n  const parsed: unknown = parseYaml(readFileSync(yamlPath, 'utf8'))\n  if (typeof parsed !== 'object' || parsed === null) {\n    throw new Error(`case '${caseId}': case.yaml must contain a mapping`)\n  }\n  const raw = parsed as Record<string, unknown>\n\n  const prompt = requireString(raw, 'prompt', caseId)\n  const judgeRaw = requireString(raw, 'judge', caseId)\n  if (!JUDGE_KINDS.includes(judgeRaw as JudgeKind)) {\n    throw new Error(`case '${caseId}': judge must be one of ${JUDGE_KINDS.join(' | ')}`)\n  }\n  const judge = judgeRaw as JudgeKind\n\n  const spec: CaseSpec = {\n    id: caseId,\n    prompt,\n    judge,\n    runs: optionalPositiveInt(raw, 'runs', caseId, DEFAULT_RUNS),\n    timeoutMs: optionalPositiveInt(raw, 'timeoutMs', caseId, DEFAULT_TIMEOUT_MS),\n    dir,\n    hasFixture: existsSync(join(dir, 'fixture')),\n  }\n\n  if (raw['maxTokens'] !== undefined) {\n    Object.assign(spec, { maxTokens: optionalPositiveInt(raw, 'maxTokens', caseId, 0) })\n  }\n\n  if (judge === 'script') {\n    const script =\n      raw['verifyScript'] === undefined\n        ? DEFAULT_VERIFY_SCRIPT\n        : requireString(raw, 'verifyScript', caseId)\n    if (!existsSync(join(dir, script))) {\n      throw new Error(`case '${caseId}': judge='script' but verify script '${script}' not found`)\n    }\n    Object.assign(spec, { verifyScript: script })\n  }\n\n  // golden 断言与 script 判据正交：judge='script' 的 case 也可附加 golden（工具纪律 + 产物正确性\n  // 两个判据都要满足）。judge='golden' 时 golden 必填，其余情况可选。\n  if (judge === 'golden' || raw['golden'] !== undefined) {\n    Object.assign(spec, { golden: parseGolden(raw['golden'], caseId) })\n  }\n\n  return spec\n}\n\n/** 扫描 case 根目录（`evals/cases/`），按 id 排序返回全部 case。 */\nexport function loadSuite(casesRoot: string): readonly CaseSpec[] {\n  if (!existsSync(casesRoot)) {\n    throw new Error(`cases root not found: ${casesRoot}`)\n  }\n  const dirs = readdirSync(casesRoot)\n    .filter((name) => !name.startsWith('.'))\n    .map((name) => join(casesRoot, name))\n    .filter((path) => statSync(path).isDirectory())\n    .sort()\n  return dirs.map(loadCase)\n}\n\n/**\n * 套件版本 = 全部 case 判分相关字段的内容指纹（基线分键用）。\n *\n * 只纳入**影响判分口径**的字段：case 集合、prompt、judge、golden。改超时/runs 不算换套件\n * （不影响可比性），改 prompt 或判据则必须换基线键——否则会把两套不同题目的分数当同一条曲线看。\n */\nexport function computeSuiteVersion(cases: readonly CaseSpec[]): string {\n  const canonical = cases.map((c) => ({\n    id: c.id,\n    prompt: c.prompt,\n    judge: c.judge,\n    golden: c.golden ?? null,\n  }))\n  return createHash('sha256').update(JSON.stringify(canonical)).digest('hex').slice(0, 12)\n}\n","/**\n * otto-invoker.ts —— 调用 headless otto 的唯一出口（RFC-316 规则 6：只走公开入口）。\n *\n * 端口化的理由：runner 的编排逻辑（n 次运行 / 预算护栏 / 三态统计）必须能在**不起真实模型**\n * 的情况下被单测，否则评测体系自身没有回归保护。真实实现 spawn `otto --json`，测试注入 fake。\n */\n\nimport { spawn } from 'node:child_process'\n\n/** headless `otto --json` 的输出契约（packages/cli/src/modes/run-modes.ts JsonModeResult）。 */\nexport interface OttoJsonResult {\n  sessionId: string\n  status: string\n  messageCount: number\n  response: string\n}\n\nexport interface InvokeRequest {\n  prompt: string\n  /** agent 的工作目录。 */\n  cwd: string\n  timeoutMs: number\n}\n\nexport type InvokeResponse =\n  | { kind: 'ok'; result: OttoJsonResult }\n  /** 进程跑起来了但任务失败（非零退出、输出不可解析）——算 fail。 */\n  | { kind: 'failed'; detail: string }\n  /**\n   * 基础设施问题（限流、网络、供应商 5xx、二进制缺失）——算 infra_error，不进 pass 率\n   * （RFC-316 D6 规则 5：防限流假回归）。\n   */\n  | { kind: 'infra'; detail: string }\n\n/** 调用 otto 的端口。 */\nexport interface OttoInvoker {\n  invoke(request: InvokeRequest): Promise<InvokeResponse>\n}\n\n/**\n * 判定一段 stderr 是否为基础设施故障而非任务失败。\n *\n * 保守取向：只认明确的传输层/额度层信号。误判成 infra 会让真实回归被藏起来，\n * 比误判成 fail 更危险——所以这里宁可漏放。\n */\nexport function classifyFailure(stderr: string): 'infra' | 'failed' {\n  const text = stderr.toLowerCase()\n  const infraSignals = [\n    'rate limit',\n    'rate_limit',\n    '429',\n    'overloaded',\n    'econnreset',\n    'etimedout',\n    'enotfound',\n    'socket hang up',\n    'service unavailable',\n    'internal server error',\n    'bad gateway',\n    'no model configured',\n    'command not found',\n    'enoent',\n  ]\n  return infraSignals.some((signal) => text.includes(signal)) ? 'infra' : 'failed'\n}\n\nexport interface SpawnInvokerOptions {\n  /** otto 可执行文件（默认 'otto'）。 */\n  bin?: string\n  /** 附加环境变量（如指定模型）。 */\n  env?: Readonly<Record<string, string>>\n}\n\n/** 真实实现：spawn `otto --json \"<prompt>\"`。 */\nexport function createSpawnInvoker(options: SpawnInvokerOptions = {}): OttoInvoker {\n  const bin = options.bin ?? 'otto'\n  return {\n    invoke(request) {\n      return new Promise<InvokeResponse>((resolve) => {\n        const child = spawn(bin, ['--json', request.prompt], {\n          cwd: request.cwd,\n          env: { ...process.env, ...options.env },\n          stdio: ['ignore', 'pipe', 'pipe'],\n        })\n        let stdout = ''\n        let stderr = ''\n        let settled = false\n        const finish = (response: InvokeResponse): void => {\n          if (settled) {\n            return\n          }\n          settled = true\n          clearTimeout(timer)\n          resolve(response)\n        }\n        const timer = setTimeout(() => {\n          child.kill('SIGKILL')\n          finish({ kind: 'failed', detail: `timeout after ${request.timeoutMs}ms` })\n        }, request.timeoutMs)\n\n        child.stdout.on('data', (chunk: Buffer) => {\n          stdout += chunk.toString()\n        })\n        child.stderr.on('data', (chunk: Buffer) => {\n          stderr += chunk.toString()\n        })\n        child.on('error', (error) => {\n          finish({ kind: 'infra', detail: `spawn failed: ${error.message}` })\n        })\n        child.on('close', (code) => {\n          if (code !== 0) {\n            const kind = classifyFailure(stderr)\n            finish({ kind, detail: `exit ${code}: ${stderr.trim().slice(0, 500)}` })\n            return\n          }\n          try {\n            const parsed = JSON.parse(stdout) as OttoJsonResult\n            if (typeof parsed.sessionId !== 'string') {\n              finish({ kind: 'failed', detail: 'json output missing sessionId' })\n              return\n            }\n            finish({ kind: 'ok', result: parsed })\n          } catch {\n            finish({ kind: 'failed', detail: `unparsable json output: ${stdout.slice(0, 300)}` })\n          }\n        })\n      })\n    },\n  }\n}\n","/**\n * trace-metrics.ts —— 从 trace JSONL 聚合运行指标（RFC-316 G3 / L3 层）。\n *\n * 数据源是 append-only 的 `OTTO_SESSIONS_DIR/trace/<sessionId>.jsonl`，otto 每个会话都会写。\n * 这一层是「零成本指标」：不额外跑任何东西，评测跑完顺手把成本/步数/工具成功率读出来。\n *\n * 只读两类锚点（其余事件忽略，容忍跨版本 trace）：\n * - `tool.call.end`：工具名 / 耗时 / errorCategory\n * - `turn.end`：usage（token）/ model\n */\n\nimport { readFileSync, existsSync } from 'node:fs'\nimport { join } from 'node:path'\nimport { OTTO_SESSIONS_DIR } from '@x-otto/env'\n\nimport type { RunMetrics } from './types'\n\ninterface RawTraceEvent {\n  kind?: string\n  node?: string\n  payload?: Record<string, unknown>\n}\n\nfunction readNumber(payload: Record<string, unknown> | undefined, key: string): number {\n  const value = payload?.[key]\n  return typeof value === 'number' && Number.isFinite(value) ? value : 0\n}\n\n/** trace 文件的默认路径。 */\nexport function traceFilePath(sessionId: string): string {\n  return join(OTTO_SESSIONS_DIR, 'trace', `${sessionId}.jsonl`)\n}\n\n/** 一次运行从 trace 还原出的全部可判分信息。 */\nexport interface TraceRun {\n  metrics: RunMetrics\n  /** 工具调用名序列（按 tool.call.end 发生顺序），golden.toolSequence 判据的输入。 */\n  toolNames: readonly string[]\n}\n\n/**\n * 解析 trace 内容为指标 + 工具序列。传内容而非路径，便于单测与远端 trace 复用。\n * 坏行直接跳过——trace 是 append-only 的，进程被杀时最后一行可能截断，这是正常状态。\n */\nexport function parseTraceRun(content: string): TraceRun {\n  const toolNames: string[] = []\n  const metrics = parseTraceMetrics(content, toolNames)\n  return { metrics, toolNames }\n}\n\nfunction parseTraceMetrics(content: string, toolNamesOut?: string[]): RunMetrics {\n  const metrics: RunMetrics = {\n    turns: 0,\n    toolCalls: 0,\n    toolErrors: 0,\n    inputTokens: 0,\n    outputTokens: 0,\n    cacheReadTokens: 0,\n    toolDurationMs: 0,\n  }\n\n  for (const line of content.split('\\n')) {\n    if (line.trim() === '') {\n      continue\n    }\n    let event: RawTraceEvent\n    try {\n      event = JSON.parse(line) as RawTraceEvent\n    } catch {\n      continue\n    }\n\n    if (event.node === 'tool.call.end') {\n      metrics.toolCalls++\n      metrics.toolDurationMs += readNumber(event.payload, 'durationMs')\n      if (event.payload?.['errorCategory'] !== undefined) {\n        metrics.toolErrors++\n      }\n      const name = event.payload?.['name']\n      if (typeof name === 'string') {\n        toolNamesOut?.push(name)\n      }\n      continue\n    }\n\n    if (event.node === 'turn.end') {\n      metrics.turns++\n      const usage = event.payload?.['usage']\n      if (typeof usage === 'object' && usage !== null) {\n        const u = usage as Record<string, unknown>\n        metrics.inputTokens += readNumber(u, 'inputTokens')\n        metrics.outputTokens += readNumber(u, 'outputTokens')\n        metrics.cacheReadTokens += readNumber(u, 'cacheReadTokens')\n      }\n      const model = event.payload?.['model']\n      if (typeof model === 'string' && model !== '') {\n        metrics.modelId = model\n      }\n    }\n  }\n\n  return metrics\n}\n\n/** 读某个会话的 trace 并聚合；文件不存在返回 undefined（trace 可能被关闭）。 */\nexport function readTraceRun(sessionId: string): TraceRun | undefined {\n  const path = traceFilePath(sessionId)\n  if (!existsSync(path)) {\n    return undefined\n  }\n  return parseTraceRun(readFileSync(path, 'utf8'))\n}\n\n/** 多次运行的指标均值（scorecard 汇总用）。 */\nexport function averageMetrics(samples: readonly RunMetrics[]): RunMetrics | undefined {\n  if (samples.length === 0) {\n    return undefined\n  }\n  const sum = samples.reduce<RunMetrics>(\n    (acc, m) => ({\n      turns: acc.turns + m.turns,\n      toolCalls: acc.toolCalls + m.toolCalls,\n      toolErrors: acc.toolErrors + m.toolErrors,\n      inputTokens: acc.inputTokens + m.inputTokens,\n      outputTokens: acc.outputTokens + m.outputTokens,\n      cacheReadTokens: acc.cacheReadTokens + m.cacheReadTokens,\n      toolDurationMs: acc.toolDurationMs + m.toolDurationMs,\n      modelId: m.modelId ?? acc.modelId,\n    }),\n    {\n      turns: 0,\n      toolCalls: 0,\n      toolErrors: 0,\n      inputTokens: 0,\n      outputTokens: 0,\n      cacheReadTokens: 0,\n      toolDurationMs: 0,\n    },\n  )\n  const n = samples.length\n  const averaged: RunMetrics = {\n    turns: sum.turns / n,\n    toolCalls: sum.toolCalls / n,\n    toolErrors: sum.toolErrors / n,\n    inputTokens: sum.inputTokens / n,\n    outputTokens: sum.outputTokens / n,\n    cacheReadTokens: sum.cacheReadTokens / n,\n    toolDurationMs: sum.toolDurationMs / n,\n  }\n  if (sum.modelId !== undefined) {\n    averaged.modelId = sum.modelId\n  }\n  return averaged\n}\n","/**\n * verify.ts —— 判分（RFC-316 D3 判分优先级的执行处）。\n *\n * 优先级：可执行判据（script exit 0） > golden 断言 > liveness（跑完没崩）。\n * **liveness 永不产生质量分**——这是继承已删 @x-otto/eval scorecard 的诚实边界：\n * 没有判据就老实说没有，不许用「跑完了」冒充「做对了」。\n */\n\nimport { spawnSync } from 'node:child_process'\nimport { join } from 'node:path'\n\nimport type { CaseSpec, GoldenSpec } from './types'\n\nexport interface VerifyInput {\n  spec: CaseSpec\n  /** agent 运行后的工作区（判分脚本的 cwd）。 */\n  workspaceDir: string\n  /** headless 输出的最终回答。 */\n  response: string\n  /** 本次运行的工具调用序列（按发生顺序），无 trace 时为空。 */\n  toolNames: readonly string[]\n}\n\nexport interface VerifyOutput {\n  passed: boolean\n  detail: string\n}\n\n/** 子序列匹配：期望序列的每个工具名按顺序出现即可，允许中间夹杂其他调用。 */\nexport function matchesToolSubsequence(\n  actual: readonly string[],\n  expected: readonly string[],\n): boolean {\n  let cursor = 0\n  for (const name of actual) {\n    if (cursor < expected.length && name === expected[cursor]) {\n      cursor++\n    }\n  }\n  return cursor === expected.length\n}\n\nfunction verifyGolden(\n  golden: GoldenSpec,\n  response: string,\n  toolNames: readonly string[],\n): VerifyOutput {\n  const misses: string[] = []\n  if (golden.answerContains) {\n    const haystack = response.toLowerCase()\n    for (const needle of golden.answerContains) {\n      if (!haystack.includes(needle.toLowerCase())) {\n        misses.push(`answer missing '${needle}'`)\n      }\n    }\n  }\n  if (golden.toolSequence && !matchesToolSubsequence(toolNames, golden.toolSequence)) {\n    misses.push(\n      `tool sequence [${golden.toolSequence.join(', ')}] not satisfied by [${toolNames.join(', ')}]`,\n    )\n  }\n  return misses.length === 0\n    ? { passed: true, detail: 'golden assertions satisfied' }\n    : { passed: false, detail: misses.join('; ') }\n}\n\nfunction runVerifyScript(input: VerifyInput, script: string): VerifyOutput {\n  const proc = spawnSync('bash', [script], {\n    cwd: input.workspaceDir,\n    encoding: 'utf8',\n    timeout: 120_000,\n    env: { ...process.env, OTTO_EVAL_RESPONSE: input.response },\n  })\n  if (proc.error) {\n    return { passed: false, detail: `verify script error: ${proc.error.message}` }\n  }\n  const detail =\n    `${(proc.stdout ?? '').trim()}${proc.stderr ? ` | ${proc.stderr.trim()}` : ''}`.slice(0, 500)\n  return { passed: proc.status === 0, detail: detail || `exit ${proc.status}` }\n}\n\n/**\n * 执行判分。script 判据在工作区内跑，exit 0 = pass。\n *\n * script + golden 可**同时**存在且必须都满足：判分方式表达的是\"这个 case 靠什么下结论\"，\n * 而工具纪律（golden.toolSequence）与产物正确性（script）是**两个正交的判据**——\n * 只有前者会让\"读了但改错\"蒙混过关，只有后者会让\"盲改碰对\"蒙混过关。\n */\nexport function verifyRun(input: VerifyInput): VerifyOutput {\n  const { spec } = input\n  const parts: VerifyOutput[] = []\n\n  if (spec.judge === 'script') {\n    parts.push(runVerifyScript(input, join(spec.dir, spec.verifyScript ?? 'verify.sh')))\n  }\n  if (spec.golden) {\n    parts.push(verifyGolden(spec.golden, input.response, input.toolNames))\n  }\n\n  if (parts.length === 0) {\n    return { passed: true, detail: 'liveness only (no quality judgement)' }\n  }\n\n  const failed = parts.filter((p) => !p.passed)\n  return failed.length === 0\n    ? { passed: true, detail: parts.map((p) => p.detail).join('; ') }\n    : { passed: false, detail: failed.map((p) => p.detail).join('; ') }\n}\n","/**\n * runner.ts —— 评测编排（RFC-316 M1 核心链路）。\n *\n * 一个 case 跑 n 次 → 四态统计（pass/fail/infra_error/skipped_budget）→ 汇总成 CaseResult。\n * 三条不可削弱的纪律都落在这里：\n * - **infra_error 不进 pass 率**（规则 5）：限流/网络故障不许伪装成能力回归。\n * - **预算护栏**（规则 4）：单 case token 超限即停后续运行并标 incomplete，不静默烧钱。\n * - **liveness 不产生质量分**（规则 2）：qualityScored=false 的 case 不进套件 pass 率。\n */\n\nimport { normalizeEnvFraction } from '@x-otto/env'\n\nimport { createWorkspace } from './workspace'\nimport { verifyRun } from './verify'\nimport { readTraceRun, averageMetrics } from './trace-metrics'\n\nimport type { OttoInvoker } from './otto-invoker'\nimport type { TraceRun } from './trace-metrics'\nimport type { CaseResult, CaseSpec, RunMetrics, RunResult, Scorecard } from './types'\n\n/**\n * infra_error 占比超过此阈值时整次运行标 invalid（RFC-316 D6 规则 5）。\n * 硬编码审计 P1：不同 CI 环境对限流/网络故障的容忍度不同，经 `OTTO_EVAL_INFRA_ERROR_THRESHOLD`\n * 覆盖（比例语义，(0,1] 区间，normalizeEnvFraction 越界回退默认）。\n */\nexport const INFRA_ERROR_INVALID_THRESHOLD = normalizeEnvFraction(\n  process.env['OTTO_EVAL_INFRA_ERROR_THRESHOLD'],\n  0.2,\n)\n\nexport interface RunnerDeps {\n  invoker: OttoInvoker\n  /** 读 trace 的端口（默认读真实 trace 文件；测试注入 fake）。 */\n  readTrace?: (sessionId: string) => TraceRun | undefined\n  /** 当前时刻（毫秒），确定性测试用。 */\n  now?: () => number\n  /** 每次运行结束的进度回调（CLI 打点用）。 */\n  onRunComplete?: (caseId: string, index: number, result: RunResult) => void\n}\n\nfunction totalTokens(metrics: RunMetrics | undefined): number {\n  if (!metrics) {\n    return 0\n  }\n  return metrics.inputTokens + metrics.outputTokens\n}\n\n/** 跑单个 case 的 n 次运行。 */\nexport async function runCase(spec: CaseSpec, deps: RunnerDeps): Promise<CaseResult> {\n  const now = deps.now ?? (() => Date.now())\n  const readTrace = deps.readTrace ?? readTraceRun\n  const runs: RunResult[] = []\n  let spentTokens = 0\n  /** 已成功计量的运行次数（用于估算不可计量运行的开销）。 */\n  let measuredRuns = 0\n  /**\n   * 无法计量 token 的运行次数：trace 缺失，或 fail/timeout 路径拿不到 usage。\n   *\n   * 审计修订：早期实现只在「成功且有 trace」时累加 spentTokens，于是失败运行被当作\n   * **零消耗**——但它们同样调用了模型、同样烧钱。一个\"每次都超时\"的 case 因此可以\n   * 无限重试而预算护栏一次都不触发。护栏若只在一切正常时生效，就不是护栏。\n   *\n   * 估算口径：有实测样本时按**已观测均值**计价（最贴近现实）；一个样本都没有时按\n   * `maxTokens/runs` 均摊——那正是用户声明 `runs` 次时的预期单价，此时跑满 runs 次\n   * 属于预期内，不该被拦。\n   */\n  let unmeasuredRuns = 0\n\n  const budgetExceeded = (): boolean => {\n    if (spec.maxTokens === undefined) {\n      return false\n    }\n    const assumedPerRun = measuredRuns > 0 ? spentTokens / measuredRuns : spec.maxTokens / spec.runs\n    return spentTokens + unmeasuredRuns * assumedPerRun >= spec.maxTokens\n  }\n\n  for (let index = 0; index < spec.runs; index++) {\n    if (budgetExceeded()) {\n      // RFC-353 S9: 预算耗尽是本地跳过决策，不是 infra 故障——用 skipped_budget 终态\n      // 区分\"未尝试\"与\"provider/网络不可靠\"，避免膨胀 infraErrorRate 误判套件失效。\n      runs.push({\n        outcome: 'skipped_budget',\n        wallMs: 0,\n        detail: `budget exhausted: ${spentTokens} measured tokens${\n          unmeasuredRuns > 0 ? ` + ${unmeasuredRuns} unmeasured run(s)` : ''\n        } vs limit ${spec.maxTokens}, remaining runs skipped`,\n      })\n      continue\n    }\n\n    const workspace = createWorkspace(spec)\n    const startedAt = now()\n    try {\n      const response = await deps.invoker.invoke({\n        prompt: spec.prompt,\n        cwd: workspace.dir,\n        timeoutMs: spec.timeoutMs,\n      })\n      const wallMs = now() - startedAt\n\n      if (response.kind === 'infra') {\n        // infra 故障（限流/网络）通常在模型真正产出前就断了，不计入已花费；\n        // 但也拿不到用量证据，故记为不可计量，让护栏保守对待。\n        unmeasuredRuns++\n        runs.push({ outcome: 'infra_error', wallMs, detail: response.detail })\n        continue\n      }\n      if (response.kind === 'failed') {\n        // 失败/超时同样烧了 token（模型已被调用），只是拿不到 usage —— 必须计入护栏，\n        // 否则\"每次都失败\"的 case 会无限重试烧钱。\n        unmeasuredRuns++\n        runs.push({ outcome: 'fail', wallMs, detail: response.detail })\n        continue\n      }\n\n      const trace = readTrace(response.result.sessionId)\n      const verdict = verifyRun({\n        spec,\n        workspaceDir: workspace.dir,\n        response: response.result.response,\n        toolNames: trace?.toolNames ?? [],\n      })\n      if (trace) {\n        spentTokens += totalTokens(trace.metrics)\n        measuredRuns++\n      } else {\n        // 成功但无 trace（trace 被关闭/文件缺失）：同样计量不到，不能当作零消耗。\n        unmeasuredRuns++\n      }\n\n      const result: RunResult = {\n        outcome: verdict.passed ? 'pass' : 'fail',\n        wallMs,\n        sessionId: response.result.sessionId,\n        detail: verdict.detail,\n      }\n      if (trace) {\n        result.metrics = trace.metrics\n      }\n      runs.push(result)\n    } finally {\n      workspace.dispose()\n    }\n\n    const last = runs.at(-1)\n    if (last) {\n      deps.onRunComplete?.(spec.id, index, last)\n    }\n  }\n\n  return summarizeCase(spec, runs)\n}\n\n/** 把 n 次运行汇总成 CaseResult（四态统计的唯一实现处）。 */\nexport function summarizeCase(spec: CaseSpec, runs: readonly RunResult[]): CaseResult {\n  const infraErrors = runs.filter((r) => r.outcome === 'infra_error').length\n  const skippedBudget = runs.filter((r) => r.outcome === 'skipped_budget').length\n  const effective = runs.length - infraErrors - skippedBudget\n  const passed = runs.filter((r) => r.outcome === 'pass').length\n  const qualityScored = spec.judge !== 'liveness'\n\n  const result: CaseResult = {\n    caseId: spec.id,\n    judge: spec.judge,\n    runs,\n    passed,\n    effective,\n    infraErrors,\n    skippedBudget,\n    qualityScored,\n  }\n  if (effective > 0) {\n    Object.assign(result, { passRate: passed / effective })\n  }\n  return result\n}\n\n/** 跑整套 case，产出 scorecard。 */\nexport async function runSuite(\n  cases: readonly CaseSpec[],\n  suiteVersion: string,\n  deps: RunnerDeps,\n): Promise<Scorecard> {\n  const now = deps.now ?? (() => Date.now())\n  const startedAt = new Date(now()).toISOString()\n  const results: CaseResult[] = []\n  for (const spec of cases) {\n    results.push(await runCase(spec, deps))\n  }\n  return buildScorecard(results, suiteVersion, startedAt)\n}\n\n/** 聚合 case 结果为 scorecard（纯函数，便于对拍手算值）。 */\nexport function buildScorecard(\n  cases: readonly CaseResult[],\n  suiteVersion: string,\n  startedAt: string,\n): Scorecard {\n  const scored = cases.filter((c) => c.qualityScored && c.passRate !== undefined)\n  const totalSkippedBudget = cases.reduce((sum, c) => sum + c.skippedBudget, 0)\n  // RFC-353 S9: infraErrorRate 分母排除 skipped_budget——它们不是\"发起的运行\"（本地\n  // 预算护栏主动跳过，未真正调用模型/触达 provider），混入分母会稀释 infraErrorRate\n  // 掩盖真实的 infra 不稳定信号。\n  const totalRuns = cases.reduce((sum, c) => sum + c.runs.length, 0) - totalSkippedBudget\n  const totalInfra = cases.reduce((sum, c) => sum + c.infraErrors, 0)\n  const infraErrorRate = totalRuns === 0 ? 0 : totalInfra / totalRuns\n\n  const allMetrics = cases.flatMap((c) =>\n    c.runs.map((r) => r.metrics).filter((m): m is RunMetrics => m !== undefined),\n  )\n  const aggregate = averageMetrics(allMetrics)\n\n  const scorecard: Scorecard = {\n    suiteVersion,\n    modelId: aggregate?.modelId ?? 'unknown',\n    startedAt,\n    cases,\n    scoredCases: cases.filter((c) => c.qualityScored).length,\n    unscoredCases: cases.filter((c) => !c.qualityScored).length,\n    infraErrorRate,\n    skippedBudgetTotal: totalSkippedBudget,\n    invalid: infraErrorRate > INFRA_ERROR_INVALID_THRESHOLD,\n  }\n  if (scored.length > 0) {\n    const sum = scored.reduce((acc, c) => acc + (c.passRate ?? 0), 0)\n    Object.assign(scorecard, { overallPassRate: sum / scored.length })\n  }\n  if (aggregate) {\n    Object.assign(scorecard, { aggregateMetrics: aggregate })\n  }\n  return scorecard\n}\n","/**\n * baseline.ts —— 基线存储与回归判定（RFC-316 D6 规则 3/4）。\n *\n * 两条承重规则：\n * - **基线按 (modelId, suiteVersion) 分键**：模型升级产生新条目而非覆盖，跨模型对比必须显式。\n *   没有这条，换一次模型就会把曲线接错——之后所有\"变强/变弱\"结论都是噪声。\n * - **阈值显式且诚实**：n=3 的统计功效只够抓大回归，故只用两个粗信号（套件 pass 率跌幅、\n *   case flip），不假装有统计显著性。\n */\n\nimport { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'\nimport { dirname } from 'node:path'\n\nimport { normalizeEnv } from '@x-otto/env'\n\nimport type { BaselineEntry, BaselineFile, RegressionVerdict, Scorecard } from './types'\n\n/**\n * 套件 pass 率相对基线的下降超过此百分点即判回归（RFC-316 D6 规则 4）。\n * 硬编码审计 P1：不同 CI 环境的回归门禁松紧不同，经 `OTTO_EVAL_REGRESSION_PP` 覆盖\n * （百分点整数语义，normalizeEnv 非正数回退默认）。\n */\nexport const PASS_RATE_REGRESSION_PP = normalizeEnv(process.env['OTTO_EVAL_REGRESSION_PP'], 15)\n\nexport function loadBaselines(path: string): BaselineFile {\n  if (!existsSync(path)) {\n    return { entries: [] }\n  }\n  const parsed = JSON.parse(readFileSync(path, 'utf8')) as BaselineFile\n  return { entries: parsed.entries ?? [] }\n}\n\nexport function saveBaselines(path: string, file: BaselineFile): void {\n  mkdirSync(dirname(path), { recursive: true })\n  writeFileSync(path, `${JSON.stringify(file, null, 2)}\\n`, 'utf8')\n}\n\n/** 按 (modelId, suiteVersion) 查基线。 */\nexport function findBaseline(\n  file: BaselineFile,\n  modelId: string,\n  suiteVersion: string,\n): BaselineEntry | undefined {\n  return file.entries.find((e) => e.modelId === modelId && e.suiteVersion === suiteVersion)\n}\n\n/** 把一次运行固化成基线条目（同键覆盖，异键新增）。 */\nexport function upsertBaseline(\n  file: BaselineFile,\n  scorecard: Scorecard,\n  reason: string,\n): BaselineFile {\n  const casePassRates: Record<string, number | undefined> = {}\n  for (const c of scorecard.cases) {\n    casePassRates[c.caseId] = c.passRate\n  }\n\n  const entry: BaselineEntry = {\n    modelId: scorecard.modelId,\n    suiteVersion: scorecard.suiteVersion,\n    recordedAt: scorecard.startedAt,\n    reason,\n    casePassRates,\n  }\n  if (scorecard.overallPassRate !== undefined) {\n    Object.assign(entry, { overallPassRate: scorecard.overallPassRate })\n  }\n\n  const kept = file.entries.filter(\n    (e) => !(e.modelId === entry.modelId && e.suiteVersion === entry.suiteVersion),\n  )\n  return { entries: [...kept, entry] }\n}\n\n/** 回归判定：套件 pass 率跌幅 + case flip 双信号。 */\nexport function judgeRegression(\n  scorecard: Scorecard,\n  baseline: BaselineEntry | undefined,\n): RegressionVerdict {\n  if (scorecard.invalid) {\n    return {\n      status: 'invalid_run',\n      findings: [\n        `infra_error rate ${(scorecard.infraErrorRate * 100).toFixed(1)}% exceeds threshold — run not comparable`,\n      ],\n      flippedCases: [],\n    }\n  }\n  if (!baseline) {\n    return {\n      status: 'no_baseline',\n      findings: [\n        `no baseline for (model=${scorecard.modelId}, suite=${scorecard.suiteVersion}) — record one with \\`baseline update\\``,\n      ],\n      flippedCases: [],\n    }\n  }\n\n  const findings: string[] = []\n  const flipped: string[] = []\n\n  for (const c of scorecard.cases) {\n    if (!c.qualityScored) {\n      continue\n    }\n    const before = baseline.casePassRates[c.caseId]\n    if (before === 1 && c.passRate === 0) {\n      flipped.push(c.caseId)\n      findings.push(`case flip: '${c.caseId}' was 100% pass, now 0%`)\n      continue\n    }\n    // 一个 case 的运行全是 infra_error 时它没有 pass 率，会**静默退出**套件均值——\n    // 于是\"某题目彻底跑不起来\"看上去和\"套件表现不变\"一模一样。这里显式点名，\n    // 不判回归（确实无从判断），但绝不让它无声消失。\n    if (c.passRate === undefined && before !== undefined) {\n      findings.push(\n        `'${c.caseId}' produced no comparable runs (all infra errors) — excluded from the suite average`,\n      )\n    }\n  }\n\n  let deltaPp: number | undefined\n  if (scorecard.overallPassRate !== undefined && baseline.overallPassRate !== undefined) {\n    deltaPp = (scorecard.overallPassRate - baseline.overallPassRate) * 100\n    if (deltaPp < -PASS_RATE_REGRESSION_PP) {\n      findings.push(\n        `suite pass rate dropped ${Math.abs(deltaPp).toFixed(1)}pp ` +\n          `(${(baseline.overallPassRate * 100).toFixed(1)}% → ${(scorecard.overallPassRate * 100).toFixed(1)}%)`,\n      )\n    }\n  }\n\n  const regressed =\n    flipped.length > 0 || (deltaPp !== undefined && deltaPp < -PASS_RATE_REGRESSION_PP)\n\n  const verdict: RegressionVerdict = {\n    status: regressed ? 'regressed' : 'ok',\n    findings: findings.length > 0 ? findings : ['no regression detected'],\n    flippedCases: flipped,\n  }\n  if (deltaPp !== undefined) {\n    Object.assign(verdict, { passRateDeltaPp: deltaPp })\n  }\n  return verdict\n}\n","/**\n * report.ts —— scorecard + 回归判定的人类可读渲染（RFC-316 D6 消费回路的出口）。\n *\n * 报告是评测体系唯一被人真正阅读的产物——@x-otto/eval 上一次死于「没人读」，\n * 所以这里刻意把三件事放在最显眼处：回归结论、逐 case pass 率、诚实边界（未评质量的 case 数）。\n */\n\nimport type { RegressionVerdict, Scorecard } from './types'\n\nfunction pct(value: number | undefined): string {\n  return value === undefined ? '—' : `${(value * 100).toFixed(0)}%`\n}\n\nexport function renderReport(scorecard: Scorecard, verdict: RegressionVerdict): string {\n  const lines: string[] = []\n\n  const headline =\n    verdict.status === 'regressed'\n      ? 'REGRESSED'\n      : verdict.status === 'ok'\n        ? 'OK'\n        : verdict.status === 'invalid_run'\n          ? 'INVALID (infra errors)'\n          : 'NO BASELINE'\n\n  lines.push(`otto eval — ${headline}`)\n  lines.push(\n    `model=${scorecard.modelId}  suite=${scorecard.suiteVersion}  at=${scorecard.startedAt}`,\n  )\n  lines.push('')\n\n  lines.push('case                              judge     pass   runs  infra  skip')\n  for (const c of scorecard.cases) {\n    const id = c.caseId.padEnd(33).slice(0, 33)\n    const judge = c.judge.padEnd(9)\n    const rate = (c.qualityScored ? pct(c.passRate) : 'n/a').padStart(5)\n    const runs = `${c.passed}/${c.effective}`.padStart(6)\n    const infra = String(c.infraErrors).padStart(6)\n    const skip = String(c.skippedBudget).padStart(5)\n    lines.push(`${id} ${judge} ${rate} ${runs} ${infra} ${skip}`)\n  }\n  lines.push('')\n\n  lines.push(\n    `suite pass rate: ${pct(scorecard.overallPassRate)}  (scored ${scorecard.scoredCases}, liveness-only ${scorecard.unscoredCases})`,\n  )\n  if (verdict.passRateDeltaPp !== undefined) {\n    const sign = verdict.passRateDeltaPp >= 0 ? '+' : ''\n    lines.push(`vs baseline:     ${sign}${verdict.passRateDeltaPp.toFixed(1)}pp`)\n  }\n  lines.push(`infra errors:    ${(scorecard.infraErrorRate * 100).toFixed(1)}% of runs`)\n  if (scorecard.skippedBudgetTotal > 0) {\n    // RFC-353 S9: 预算耗尽跳过的运行不是 infra 故障，单独一行呈现——提醒读者本次\n    // 评测有 case 未完整跑满 runs 次，结果基于更少样本量。\n    lines.push(`skipped (budget): ${scorecard.skippedBudgetTotal} run(s) — suite incomplete for those cases`)\n  }\n\n  const m = scorecard.aggregateMetrics\n  if (m) {\n    lines.push(\n      `avg per run:     ${m.turns.toFixed(1)} turns · ${m.toolCalls.toFixed(1)} tools ` +\n        `(${m.toolErrors.toFixed(1)} err) · ${Math.round(m.inputTokens + m.outputTokens)} tokens`,\n    )\n  }\n\n  lines.push('')\n  for (const finding of verdict.findings) {\n    lines.push(`- ${finding}`)\n  }\n\n  if (scorecard.unscoredCases > 0) {\n    lines.push('')\n    lines.push(\n      `note: ${scorecard.unscoredCases} case(s) have no machine judgement (liveness only) and do not affect the suite pass rate.`,\n    )\n  }\n\n  return lines.join('\\n')\n}\n","/**\n * extract.ts —— trace 提炼器（RFC-316 D4 来源②：真实会话 → case 骨架）。\n *\n * case 库能否持续生长，决定这套评测第二次会不会死。真实事故/真实任务是最好的 case 来源，\n * 但从零手写 case 的摩擦太大——本模块把一个真实会话一键转成**骨架**：prompt 原文、\n * 工具调用序列（golden.toolSequence 的现成候选）、触碰过的文件清单（fixture 提示）。\n *\n * 两条刻意的\"不做\"：\n * - **不自动生成判据**：判据是 case 的灵魂，必须人写（RFC-316 D3 诚实边界——自动生成的\n *   判据只会是\"输出像上次一样\"，那是快照不是判据）。骨架里留 TODO 与建议。\n * - **不自动脱敏**：不可靠的脱敏比明示风险更危险（对齐 RFC-200 录制红线）。骨架自带\n *   脱敏检查清单，入库前人工过一遍是硬规则（RFC-316 规则 7）。\n */\n\nimport { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'\nimport { join } from 'node:path'\n\nimport { traceFilePath } from './trace-metrics'\n\ninterface RawTraceEvent {\n  node?: string\n  payload?: Record<string, unknown>\n}\n\n/** 从 trace 还原的会话骨架素材。 */\nexport interface SessionSkeleton {\n  sessionId: string\n  /** 首条用户 prompt 原文（prompt.before 的 text）。 */\n  prompt?: string\n  /** 工具调用序列（名称，按发生顺序）。 */\n  toolSequence: readonly string[]\n  /** 工具参数里出现过的文件路径（write/edit/read 的 path 类参数）——fixture 需要什么的线索。 */\n  touchedPaths: readonly string[]\n  /** LLM 往返数（turn.end 计数）。 */\n  turns: number\n}\n\nconst PATH_ARGUMENT_KEYS = ['path', 'filePath', 'file_path'] as const\n\n/** 解析 trace 内容为骨架素材（纯函数，可测）。 */\nexport function parseSessionSkeleton(sessionId: string, content: string): SessionSkeleton {\n  let prompt: string | undefined\n  const toolSequence: string[] = []\n  const touchedPaths = new Set<string>()\n  let turns = 0\n\n  for (const line of content.split('\\n')) {\n    if (line.trim() === '') {\n      continue\n    }\n    let event: RawTraceEvent\n    try {\n      event = JSON.parse(line) as RawTraceEvent\n    } catch {\n      continue\n    }\n\n    if (event.node === 'prompt.before' && prompt === undefined) {\n      const text = event.payload?.['text']\n      if (typeof text === 'string' && text.trim() !== '') {\n        prompt = text\n      }\n      continue\n    }\n\n    if (event.node === 'tool.call.start') {\n      const name = event.payload?.['name']\n      if (typeof name === 'string') {\n        toolSequence.push(name)\n      }\n      const args = event.payload?.['arguments']\n      if (typeof args === 'object' && args !== null) {\n        for (const key of PATH_ARGUMENT_KEYS) {\n          const value = (args as Record<string, unknown>)[key]\n          if (typeof value === 'string' && value !== '') {\n            touchedPaths.add(value)\n          }\n        }\n      }\n      continue\n    }\n\n    if (event.node === 'turn.end') {\n      turns++\n    }\n  }\n\n  return {\n    sessionId,\n    toolSequence,\n    touchedPaths: [...touchedPaths].sort(),\n    turns,\n    ...(prompt !== undefined ? { prompt } : {}),\n  }\n}\n\nconst SANITIZE_CHECKLIST = `## 脱敏检查清单（入库前必须逐项人工核对 — RFC-316 规则 7）\n\n- [ ] prompt 里没有 API key / token / 密码 / 内部主机名\n- [ ] prompt 里没有真实用户数据（邮箱/手机号/姓名）\n- [ ] fixture 文件里没有上述任何内容\n- [ ] 路径不含本机用户名等机器指纹（用相对路径）\n- [ ] verify 判据已由人工编写（不要用\"输出像上次一样\"的快照当判据）\n`\n\nfunction renderCaseYaml(skeleton: SessionSkeleton): string {\n  const promptBlock =\n    skeleton.prompt !== undefined\n      ? `prompt: |\\n${skeleton.prompt\n          .split('\\n')\n          .map((line) => `  ${line}`)\n          .join('\\n')}`\n      : 'prompt: |\\n  TODO: trace 里没有 prompt.before 文本，请手工填写'\n\n  const toolHint =\n    skeleton.toolSequence.length > 0\n      ? `# 实际会话的工具序列（golden.toolSequence 候选，按需裁剪）：\\n# ${skeleton.toolSequence.join(' → ')}\\n`\n      : ''\n\n  return `# 由 otto-eval extract 生成的骨架（会话 ${skeleton.sessionId}）。\n# TODO：本 case 尚不可用——judge 是 liveness 占位，必须人工补判据后才能进套件质量分。\n${promptBlock}\njudge: liveness  # TODO: 改成 script（写 verify.sh）或 golden（写断言）\n${toolHint}runs: 3\ntimeoutMs: 240000\nmaxTokens: 200000\n`\n}\n\nfunction renderReport(skeleton: SessionSkeleton): string {\n  const paths =\n    skeleton.touchedPaths.length > 0\n      ? skeleton.touchedPaths.map((p) => `- \\`${p}\\``).join('\\n')\n      : '- （无 —— 会话没有文件类工具调用）'\n\n  return `# 提炼报告 — 会话 ${skeleton.sessionId}\n\n- LLM 往返：${skeleton.turns}\n- 工具调用：${skeleton.toolSequence.length} 次\n\n## 会话触碰过的文件（fixture 需要复现哪些初始状态的线索）\n\n${paths}\n\n## 下一步（按序完成后删除本文件）\n\n1. 在 fixture/ 里构造这些文件的**任务开始前**状态（trace 只有调用记录，初始内容需要你还原）\n2. 写判据：verify.sh（首选，可执行判据）或 case.yaml 里的 golden 断言\n3. 把 judge 从 liveness 改成 script/golden\n4. 过一遍下面的脱敏清单\n\n${SANITIZE_CHECKLIST}`\n}\n\nexport interface ExtractResult {\n  caseDir: string\n  skeleton: SessionSkeleton\n}\n\n/** 从会话 trace 生成 case 骨架目录。目标已存在时拒绝（不覆盖人工编辑过的 case）。 */\nexport function extractCase(\n  sessionId: string,\n  casesRoot: string,\n  readTraceContent: (sessionId: string) => string | undefined = defaultReadTrace,\n): ExtractResult {\n  const content = readTraceContent(sessionId)\n  if (content === undefined) {\n    throw new Error(\n      `no trace found for session ${sessionId} (looked at ${traceFilePath(sessionId)})`,\n    )\n  }\n\n  const skeleton = parseSessionSkeleton(sessionId, content)\n  const caseDir = join(casesRoot, `extracted-${sessionId.slice(0, 8)}`)\n  if (existsSync(caseDir)) {\n    throw new Error(\n      `${caseDir} already exists — refusing to overwrite (delete it first if intended)`,\n    )\n  }\n\n  mkdirSync(join(caseDir, 'fixture'), { recursive: true })\n  writeFileSync(join(caseDir, 'case.yaml'), renderCaseYaml(skeleton), 'utf8')\n  writeFileSync(join(caseDir, 'EXTRACTION.md'), renderReport(skeleton), 'utf8')\n\n  return { caseDir, skeleton }\n}\n\nfunction defaultReadTrace(sessionId: string): string | undefined {\n  const path = traceFilePath(sessionId)\n  return existsSync(path) ? readFileSync(path, 'utf8') : undefined\n}\n"],"mappings":"8aAcA,MAIM,EAAoC,CAAC,SAAU,SAAU,WAAW,CAE1E,SAAS,EAAc,EAA8B,EAAa,EAAwB,CACxF,IAAM,EAAQ,EAAI,GAClB,GAAI,OAAO,GAAU,UAAY,EAAM,MAAM,GAAK,GAChD,MAAU,MAAM,SAAS,EAAO,YAAY,EAAI,8BAA8B,CAEhF,OAAO,EAGT,SAAS,EACP,EACA,EACA,EACA,EACQ,CACR,IAAM,EAAQ,EAAI,GAClB,GAAI,IAAU,IAAA,GACZ,OAAO,EAET,GAAI,OAAO,GAAU,UAAY,CAAC,OAAO,UAAU,EAAM,EAAI,GAAS,EACpE,MAAU,MAAM,SAAS,EAAO,YAAY,EAAI,8BAA8B,CAEhF,OAAO,EAGT,SAAS,EAAY,EAAc,EAA4B,CAC7D,GAAI,OAAO,GAAQ,WAAY,EAC7B,MAAU,MAAM,SAAS,EAAO,+CAA+C,CAEjF,IAAM,EAAS,EACT,EAAqB,EAAE,CACvB,EAAiB,EAAO,eAC9B,GAAI,IAAmB,IAAA,GAAW,CAChC,GAAI,CAAC,MAAM,QAAQ,EAAe,EAAI,EAAe,KAAM,GAAM,OAAO,GAAM,SAAS,CACrF,MAAU,MAAM,SAAS,EAAO,iDAAiD,CAEnF,OAAO,OAAO,EAAQ,CAAkB,iBAA4B,CAAC,CAEvE,IAAM,EAAe,EAAO,aAC5B,GAAI,IAAiB,IAAA,GAAW,CAC9B,GAAI,CAAC,MAAM,QAAQ,EAAa,EAAI,EAAa,KAAM,GAAM,OAAO,GAAM,SAAS,CACjF,MAAU,MAAM,SAAS,EAAO,+CAA+C,CAEjF,OAAO,OAAO,EAAQ,CAAgB,eAA0B,CAAC,CAEnE,GAAI,CAAC,EAAO,gBAAkB,CAAC,EAAO,aACpC,MAAU,MACR,SAAS,EAAO,wFACjB,CAEH,OAAO,EAIT,SAAgB,EAAS,EAAuB,CAC9C,IAAM,EAAS,EAAI,MAAM,IAAI,CAAC,OAAO,QAAQ,CAAC,GAAG,GAAG,EAAI,EAClD,EAAW,EAAK,EAAK,YAAY,CACvC,GAAI,CAAC,EAAW,EAAS,CACvB,MAAU,MAAM,SAAS,EAAO,0BAA0B,IAAW,CAGvE,IAAM,EAAkBA,EAAU,EAAa,EAAU,OAAO,CAAC,CACjE,GAAI,OAAO,GAAW,WAAY,EAChC,MAAU,MAAM,SAAS,EAAO,qCAAqC,CAEvE,IAAM,EAAM,EAEN,EAAS,EAAc,EAAK,SAAU,EAAO,CAC7C,EAAW,EAAc,EAAK,QAAS,EAAO,CACpD,GAAI,CAAC,EAAY,SAAS,EAAsB,CAC9C,MAAU,MAAM,SAAS,EAAO,0BAA0B,EAAY,KAAK,MAAM,GAAG,CAEtF,IAAM,EAAQ,EAER,EAAiB,CACrB,GAAI,EACJ,SACA,QACA,KAAM,EAAoB,EAAK,OAAQ,EAAQ,EAAa,CAC5D,UAAW,EAAoB,EAAK,YAAa,EAAQ,IAAmB,CAC5E,MACA,WAAY,EAAW,EAAK,EAAK,UAAU,CAAC,CAC7C,CAMD,GAJI,EAAI,YAAiB,IAAA,IACvB,OAAO,OAAO,EAAM,CAAE,UAAW,EAAoB,EAAK,YAAa,EAAQ,EAAE,CAAE,CAAC,CAGlF,IAAU,SAAU,CACtB,IAAM,EACJ,EAAI,eAAoB,IAAA,GACpB,YACA,EAAc,EAAK,eAAgB,EAAO,CAChD,GAAI,CAAC,EAAW,EAAK,EAAK,EAAO,CAAC,CAChC,MAAU,MAAM,SAAS,EAAO,uCAAuC,EAAO,aAAa,CAE7F,OAAO,OAAO,EAAM,CAAE,aAAc,EAAQ,CAAC,CAS/C,OAJI,IAAU,UAAY,EAAI,SAAc,IAAA,KAC1C,OAAO,OAAO,EAAM,CAAE,OAAQ,EAAY,EAAI,OAAW,EAAO,CAAE,CAAC,CAG9D,EAIT,SAAgB,EAAU,EAAwC,CAChE,GAAI,CAAC,EAAW,EAAU,CACxB,MAAU,MAAM,yBAAyB,IAAY,CAOvD,OALa,EAAY,EAAU,CAChC,OAAQ,GAAS,CAAC,EAAK,WAAW,IAAI,CAAC,CACvC,IAAK,GAAS,EAAK,EAAW,EAAK,CAAC,CACpC,OAAQ,GAAS,EAAS,EAAK,CAAC,aAAa,CAAC,CAC9C,MAAM,CACG,IAAI,EAAS,CAS3B,SAAgB,EAAoB,EAAoC,CACtE,IAAM,EAAY,EAAM,IAAK,IAAO,CAClC,GAAI,EAAE,GACN,OAAQ,EAAE,OACV,MAAO,EAAE,MACT,OAAQ,EAAE,QAAU,KACrB,EAAE,CACH,OAAO,EAAW,SAAS,CAAC,OAAO,KAAK,UAAU,EAAU,CAAC,CAAC,OAAO,MAAM,CAAC,MAAM,EAAG,GAAG,CC5G1F,SAAgB,EAAgB,EAAoC,CAClE,IAAM,EAAO,EAAO,aAAa,CAiBjC,MAhBqB,CACnB,aACA,aACA,MACA,aACA,aACA,YACA,YACA,iBACA,sBACA,wBACA,cACA,sBACA,oBACA,SACD,CACmB,KAAM,GAAW,EAAK,SAAS,EAAO,CAAC,CAAG,QAAU,SAW1E,SAAgB,EAAmB,EAA+B,EAAE,CAAe,CACjF,IAAM,EAAM,EAAQ,KAAO,OAC3B,MAAO,CACL,OAAO,EAAS,CACd,OAAO,IAAI,QAAyB,GAAY,CAC9C,IAAM,EAAQ,EAAM,EAAK,CAAC,SAAU,EAAQ,OAAO,CAAE,CACnD,IAAK,EAAQ,IACb,IAAK,CAAE,GAAG,QAAQ,IAAK,GAAG,EAAQ,IAAK,CACvC,MAAO,CAAC,SAAU,OAAQ,OAAO,CAClC,CAAC,CACE,EAAS,GACT,EAAS,GACT,EAAU,GACR,EAAU,GAAmC,CAC7C,IAGJ,EAAU,GACV,aAAa,EAAM,CACnB,EAAQ,EAAS,GAEb,EAAQ,eAAiB,CAC7B,EAAM,KAAK,UAAU,CACrB,EAAO,CAAE,KAAM,SAAU,OAAQ,iBAAiB,EAAQ,UAAU,IAAK,CAAC,EACzE,EAAQ,UAAU,CAErB,EAAM,OAAO,GAAG,OAAS,GAAkB,CACzC,GAAU,EAAM,UAAU,EAC1B,CACF,EAAM,OAAO,GAAG,OAAS,GAAkB,CACzC,GAAU,EAAM,UAAU,EAC1B,CACF,EAAM,GAAG,QAAU,GAAU,CAC3B,EAAO,CAAE,KAAM,QAAS,OAAQ,iBAAiB,EAAM,UAAW,CAAC,EACnE,CACF,EAAM,GAAG,QAAU,GAAS,CAC1B,GAAI,IAAS,EAAG,CAEd,EAAO,CAAE,KADI,EAAgB,EAAO,CACrB,OAAQ,QAAQ,EAAK,IAAI,EAAO,MAAM,CAAC,MAAM,EAAG,IAAI,GAAI,CAAC,CACxE,OAEF,GAAI,CACF,IAAM,EAAS,KAAK,MAAM,EAAO,CACjC,GAAI,OAAO,EAAO,WAAc,SAAU,CACxC,EAAO,CAAE,KAAM,SAAU,OAAQ,gCAAiC,CAAC,CACnE,OAEF,EAAO,CAAE,KAAM,KAAM,OAAQ,EAAQ,CAAC,MAChC,CACN,EAAO,CAAE,KAAM,SAAU,OAAQ,2BAA2B,EAAO,MAAM,EAAG,IAAI,GAAI,CAAC,GAEvF,EACF,EAEL,CCzGH,SAAS,EAAW,EAA8C,EAAqB,CACrF,IAAM,EAAQ,IAAU,GACxB,OAAO,OAAO,GAAU,UAAY,OAAO,SAAS,EAAM,CAAG,EAAQ,EAIvE,SAAgB,EAAc,EAA2B,CACvD,OAAO,EAAK,EAAmB,QAAS,GAAG,EAAU,QAAQ,CAc/D,SAAgB,EAAc,EAA2B,CACvD,IAAM,EAAsB,EAAE,CAE9B,MAAO,CAAE,QADO,EAAkB,EAAS,EAAU,CACnC,YAAW,CAG/B,SAAS,EAAkB,EAAiB,EAAqC,CAC/E,IAAM,EAAsB,CAC1B,MAAO,EACP,UAAW,EACX,WAAY,EACZ,YAAa,EACb,aAAc,EACd,gBAAiB,EACjB,eAAgB,EACjB,CAED,IAAK,IAAM,KAAQ,EAAQ,MAAM;EAAK,CAAE,CACtC,GAAI,EAAK,MAAM,GAAK,GAClB,SAEF,IAAI,EACJ,GAAI,CACF,EAAQ,KAAK,MAAM,EAAK,MAClB,CACN,SAGF,GAAI,EAAM,OAAS,gBAAiB,CAClC,EAAQ,YACR,EAAQ,gBAAkB,EAAW,EAAM,QAAS,aAAa,CAC7D,EAAM,SAAU,gBAAqB,IAAA,IACvC,EAAQ,aAEV,IAAM,EAAO,EAAM,SAAU,KACzB,OAAO,GAAS,UAClB,GAAc,KAAK,EAAK,CAE1B,SAGF,GAAI,EAAM,OAAS,WAAY,CAC7B,EAAQ,QACR,IAAM,EAAQ,EAAM,SAAU,MAC9B,GAAI,OAAO,GAAU,UAAY,EAAgB,CAC/C,IAAM,EAAI,EACV,EAAQ,aAAe,EAAW,EAAG,cAAc,CACnD,EAAQ,cAAgB,EAAW,EAAG,eAAe,CACrD,EAAQ,iBAAmB,EAAW,EAAG,kBAAkB,CAE7D,IAAM,EAAQ,EAAM,SAAU,MAC1B,OAAO,GAAU,UAAY,IAAU,KACzC,EAAQ,QAAU,IAKxB,OAAO,EAIT,SAAgB,EAAa,EAAyC,CACpE,IAAM,EAAO,EAAc,EAAU,CAChC,KAAW,EAAK,CAGrB,OAAO,EAAc,EAAa,EAAM,OAAO,CAAC,CAIlD,SAAgB,EAAe,EAAwD,CACrF,GAAI,EAAQ,SAAW,EACrB,OAEF,IAAM,EAAM,EAAQ,QACjB,EAAK,KAAO,CACX,MAAO,EAAI,MAAQ,EAAE,MACrB,UAAW,EAAI,UAAY,EAAE,UAC7B,WAAY,EAAI,WAAa,EAAE,WAC/B,YAAa,EAAI,YAAc,EAAE,YACjC,aAAc,EAAI,aAAe,EAAE,aACnC,gBAAiB,EAAI,gBAAkB,EAAE,gBACzC,eAAgB,EAAI,eAAiB,EAAE,eACvC,QAAS,EAAE,SAAW,EAAI,QAC3B,EACD,CACE,MAAO,EACP,UAAW,EACX,WAAY,EACZ,YAAa,EACb,aAAc,EACd,gBAAiB,EACjB,eAAgB,EACjB,CACF,CACK,EAAI,EAAQ,OACZ,EAAuB,CAC3B,MAAO,EAAI,MAAQ,EACnB,UAAW,EAAI,UAAY,EAC3B,WAAY,EAAI,WAAa,EAC7B,YAAa,EAAI,YAAc,EAC/B,aAAc,EAAI,aAAe,EACjC,gBAAiB,EAAI,gBAAkB,EACvC,eAAgB,EAAI,eAAiB,EACtC,CAID,OAHI,EAAI,UAAY,IAAA,KAClB,EAAS,QAAU,EAAI,SAElB,EC3HT,SAAgB,EACd,EACA,EACS,CACT,IAAI,EAAS,EACb,IAAK,IAAM,KAAQ,EACb,EAAS,EAAS,QAAU,IAAS,EAAS,IAChD,IAGJ,OAAO,IAAW,EAAS,OAG7B,SAAS,EACP,EACA,EACA,EACc,CACd,IAAM,EAAmB,EAAE,CAC3B,GAAI,EAAO,eAAgB,CACzB,IAAM,EAAW,EAAS,aAAa,CACvC,IAAK,IAAM,KAAU,EAAO,eACrB,EAAS,SAAS,EAAO,aAAa,CAAC,EAC1C,EAAO,KAAK,mBAAmB,EAAO,GAAG,CAS/C,OALI,EAAO,cAAgB,CAAC,EAAuB,EAAW,EAAO,aAAa,EAChF,EAAO,KACL,kBAAkB,EAAO,aAAa,KAAK,KAAK,CAAC,sBAAsB,EAAU,KAAK,KAAK,CAAC,GAC7F,CAEI,EAAO,SAAW,EACrB,CAAE,OAAQ,GAAM,OAAQ,8BAA+B,CACvD,CAAE,OAAQ,GAAO,OAAQ,EAAO,KAAK,KAAK,CAAE,CAGlD,SAAS,EAAgB,EAAoB,EAA8B,CACzE,IAAM,EAAO,EAAU,OAAQ,CAAC,EAAO,CAAE,CACvC,IAAK,EAAM,aACX,SAAU,OACV,QAAS,KACT,IAAK,CAAE,GAAG,QAAQ,IAAK,mBAAoB,EAAM,SAAU,CAC5D,CAAC,CACF,GAAI,EAAK,MACP,MAAO,CAAE,OAAQ,GAAO,OAAQ,wBAAwB,EAAK,MAAM,UAAW,CAEhF,IAAM,EACJ,IAAI,EAAK,QAAU,IAAI,MAAM,GAAG,EAAK,OAAS,MAAM,EAAK,OAAO,MAAM,GAAK,KAAK,MAAM,EAAG,IAAI,CAC/F,MAAO,CAAE,OAAQ,EAAK,SAAW,EAAG,OAAQ,GAAU,QAAQ,EAAK,SAAU,CAU/E,SAAgB,EAAU,EAAkC,CAC1D,GAAM,CAAE,QAAS,EACX,EAAwB,EAAE,CAShC,GAPI,EAAK,QAAU,UACjB,EAAM,KAAK,EAAgB,EAAO,EAAK,EAAK,IAAK,EAAK,cAAgB,YAAY,CAAC,CAAC,CAElF,EAAK,QACP,EAAM,KAAK,EAAa,EAAK,OAAQ,EAAM,SAAU,EAAM,UAAU,CAAC,CAGpE,EAAM,SAAW,EACnB,MAAO,CAAE,OAAQ,GAAM,OAAQ,uCAAwC,CAGzE,IAAM,EAAS,EAAM,OAAQ,GAAM,CAAC,EAAE,OAAO,CAC7C,OAAO,EAAO,SAAW,EACrB,CAAE,OAAQ,GAAM,OAAQ,EAAM,IAAK,GAAM,EAAE,OAAO,CAAC,KAAK,KAAK,CAAE,CAC/D,CAAE,OAAQ,GAAO,OAAQ,EAAO,IAAK,GAAM,EAAE,OAAO,CAAC,KAAK,KAAK,CAAE,CCjFvE,MAAa,EAAgC,EAC3C,QAAQ,IAAI,gCACZ,GACD,CAYD,SAAS,EAAY,EAAyC,CAI5D,OAHK,EAGE,EAAQ,YAAc,EAAQ,aAF5B,EAMX,eAAsB,EAAQ,EAAgB,EAAuC,CACnF,IAAM,EAAM,EAAK,UAAc,KAAK,KAAK,EACnC,EAAY,EAAK,WAAa,EAC9B,EAAoB,EAAE,CACxB,EAAc,EAEd,EAAe,EAYf,EAAiB,EAEf,MAAgC,CACpC,GAAI,EAAK,YAAc,IAAA,GACrB,MAAO,GAET,IAAM,EAAgB,EAAe,EAAI,EAAc,EAAe,EAAK,UAAY,EAAK,KAC5F,OAAO,EAAc,EAAiB,GAAiB,EAAK,WAG9D,IAAK,IAAI,EAAQ,EAAG,EAAQ,EAAK,KAAM,IAAS,CAC9C,GAAI,GAAgB,CAAE,CAGpB,EAAK,KAAK,CACR,QAAS,iBACT,OAAQ,EACR,OAAQ,qBAAqB,EAAY,kBACvC,EAAiB,EAAI,MAAM,EAAe,oBAAsB,GACjE,YAAY,EAAK,UAAU,0BAC7B,CAAC,CACF,SAGF,IAAM,EAAY,EAAgB,EAAK,CACjC,EAAY,GAAK,CACvB,GAAI,CACF,IAAM,EAAW,MAAM,EAAK,QAAQ,OAAO,CACzC,OAAQ,EAAK,OACb,IAAK,EAAU,IACf,UAAW,EAAK,UACjB,CAAC,CACI,EAAS,GAAK,CAAG,EAEvB,GAAI,EAAS,OAAS,QAAS,CAG7B,IACA,EAAK,KAAK,CAAE,QAAS,cAAe,SAAQ,OAAQ,EAAS,OAAQ,CAAC,CACtE,SAEF,GAAI,EAAS,OAAS,SAAU,CAG9B,IACA,EAAK,KAAK,CAAE,QAAS,OAAQ,SAAQ,OAAQ,EAAS,OAAQ,CAAC,CAC/D,SAGF,IAAM,EAAQ,EAAU,EAAS,OAAO,UAAU,CAC5C,EAAU,EAAU,CACxB,OACA,aAAc,EAAU,IACxB,SAAU,EAAS,OAAO,SAC1B,UAAW,GAAO,WAAa,EAAE,CAClC,CAAC,CACE,GACF,GAAe,EAAY,EAAM,QAAQ,CACzC,KAGA,IAGF,IAAM,EAAoB,CACxB,QAAS,EAAQ,OAAS,OAAS,OACnC,SACA,UAAW,EAAS,OAAO,UAC3B,OAAQ,EAAQ,OACjB,CACG,IACF,EAAO,QAAU,EAAM,SAEzB,EAAK,KAAK,EAAO,QACT,CACR,EAAU,SAAS,CAGrB,IAAM,EAAO,EAAK,GAAG,GAAG,CACpB,GACF,EAAK,gBAAgB,EAAK,GAAI,EAAO,EAAK,CAI9C,OAAO,EAAc,EAAM,EAAK,CAIlC,SAAgB,EAAc,EAAgB,EAAwC,CACpF,IAAM,EAAc,EAAK,OAAQ,GAAM,EAAE,UAAY,cAAc,CAAC,OAC9D,EAAgB,EAAK,OAAQ,GAAM,EAAE,UAAY,iBAAiB,CAAC,OACnE,EAAY,EAAK,OAAS,EAAc,EACxC,EAAS,EAAK,OAAQ,GAAM,EAAE,UAAY,OAAO,CAAC,OAClD,EAAgB,EAAK,QAAU,WAE/B,EAAqB,CACzB,OAAQ,EAAK,GACb,MAAO,EAAK,MACZ,OACA,SACA,YACA,cACA,gBACA,gBACD,CAID,OAHI,EAAY,GACd,OAAO,OAAO,EAAQ,CAAE,SAAU,EAAS,EAAW,CAAC,CAElD,EAIT,eAAsB,EACpB,EACA,EACA,EACoB,CACpB,IAAM,EAAM,EAAK,UAAc,KAAK,KAAK,EACnC,EAAY,IAAI,KAAK,GAAK,CAAC,CAAC,aAAa,CACzC,EAAwB,EAAE,CAChC,IAAK,IAAM,KAAQ,EACjB,EAAQ,KAAK,MAAM,EAAQ,EAAM,EAAK,CAAC,CAEzC,OAAO,EAAe,EAAS,EAAc,EAAU,CAIzD,SAAgB,EACd,EACA,EACA,EACW,CACX,IAAM,EAAS,EAAM,OAAQ,GAAM,EAAE,eAAiB,EAAE,WAAa,IAAA,GAAU,CACzE,EAAqB,EAAM,QAAQ,EAAK,IAAM,EAAM,EAAE,cAAe,EAAE,CAIvE,EAAY,EAAM,QAAQ,EAAK,IAAM,EAAM,EAAE,KAAK,OAAQ,EAAE,CAAG,EAC/D,EAAa,EAAM,QAAQ,EAAK,IAAM,EAAM,EAAE,YAAa,EAAE,CAC7D,EAAiB,IAAc,EAAI,EAAI,EAAa,EAKpD,EAAY,EAHC,EAAM,QAAS,GAChC,EAAE,KAAK,IAAK,GAAM,EAAE,QAAQ,CAAC,OAAQ,GAAuB,IAAM,IAAA,GAAU,CAC7E,CAC2C,CAEtC,EAAuB,CAC3B,eACA,QAAS,GAAW,SAAW,UAC/B,YACA,QACA,YAAa,EAAM,OAAQ,GAAM,EAAE,cAAc,CAAC,OAClD,cAAe,EAAM,OAAQ,GAAM,CAAC,EAAE,cAAc,CAAC,OACrD,iBACA,mBAAoB,EACpB,QAAS,EAAiB,EAC3B,CACD,GAAI,EAAO,OAAS,EAAG,CACrB,IAAM,EAAM,EAAO,QAAQ,EAAK,IAAM,GAAO,EAAE,UAAY,GAAI,EAAE,CACjE,OAAO,OAAO,EAAW,CAAE,gBAAiB,EAAM,EAAO,OAAQ,CAAC,CAKpE,OAHI,GACF,OAAO,OAAO,EAAW,CAAE,iBAAkB,EAAW,CAAC,CAEpD,EChNT,MAAa,EAA0B,EAAa,QAAQ,IAAI,wBAA4B,GAAG,CAE/F,SAAgB,EAAc,EAA4B,CAKxD,OAJK,EAAW,EAAK,CAId,CAAE,QADM,KAAK,MAAM,EAAa,EAAM,OAAO,CAAC,CAC5B,SAAW,EAAE,CAAE,CAH/B,CAAE,QAAS,EAAE,CAAE,CAM1B,SAAgB,EAAc,EAAc,EAA0B,CACpE,EAAU,EAAQ,EAAK,CAAE,CAAE,UAAW,GAAM,CAAC,CAC7C,EAAc,EAAM,GAAG,KAAK,UAAU,EAAM,KAAM,EAAE,CAAC,IAAK,OAAO,CAInE,SAAgB,EACd,EACA,EACA,EAC2B,CAC3B,OAAO,EAAK,QAAQ,KAAM,GAAM,EAAE,UAAY,GAAW,EAAE,eAAiB,EAAa,CAI3F,SAAgB,EACd,EACA,EACA,EACc,CACd,IAAM,EAAoD,EAAE,CAC5D,IAAK,IAAM,KAAK,EAAU,MACxB,EAAc,EAAE,QAAU,EAAE,SAG9B,IAAM,EAAuB,CAC3B,QAAS,EAAU,QACnB,aAAc,EAAU,aACxB,WAAY,EAAU,UACtB,SACA,gBACD,CAQD,OAPI,EAAU,kBAAoB,IAAA,IAChC,OAAO,OAAO,EAAO,CAAE,gBAAiB,EAAU,gBAAiB,CAAC,CAM/D,CAAE,QAAS,CAAC,GAHN,EAAK,QAAQ,OACvB,GAAM,EAAE,EAAE,UAAY,EAAM,SAAW,EAAE,eAAiB,EAAM,cAClE,CAC2B,EAAM,CAAE,CAItC,SAAgB,EACd,EACA,EACmB,CACnB,GAAI,EAAU,QACZ,MAAO,CACL,OAAQ,cACR,SAAU,CACR,qBAAqB,EAAU,eAAiB,KAAK,QAAQ,EAAE,CAAC,0CACjE,CACD,aAAc,EAAE,CACjB,CAEH,GAAI,CAAC,EACH,MAAO,CACL,OAAQ,cACR,SAAU,CACR,0BAA0B,EAAU,QAAQ,UAAU,EAAU,aAAa,yCAC9E,CACD,aAAc,EAAE,CACjB,CAGH,IAAM,EAAqB,EAAE,CACvB,EAAoB,EAAE,CAE5B,IAAK,IAAM,KAAK,EAAU,MAAO,CAC/B,GAAI,CAAC,EAAE,cACL,SAEF,IAAM,EAAS,EAAS,cAAc,EAAE,QACxC,GAAI,IAAW,GAAK,EAAE,WAAa,EAAG,CACpC,EAAQ,KAAK,EAAE,OAAO,CACtB,EAAS,KAAK,eAAe,EAAE,OAAO,yBAAyB,CAC/D,SAKE,EAAE,WAAa,IAAA,IAAa,IAAW,IAAA,IACzC,EAAS,KACP,IAAI,EAAE,OAAO,oFACd,CAIL,IAAI,EACA,EAAU,kBAAoB,IAAA,IAAa,EAAS,kBAAoB,IAAA,KAC1E,GAAW,EAAU,gBAAkB,EAAS,iBAAmB,IAC/D,EAAU,CAAC,GACb,EAAS,KACP,2BAA2B,KAAK,IAAI,EAAQ,CAAC,QAAQ,EAAE,CAAC,OACjD,EAAS,gBAAkB,KAAK,QAAQ,EAAE,CAAC,OAAO,EAAU,gBAAkB,KAAK,QAAQ,EAAE,CAAC,IACtG,EAOL,IAAM,EAA6B,CACjC,OAHA,EAAQ,OAAS,GAAM,IAAY,IAAA,IAAa,EAAU,CAAC,EAGvC,YAAc,KAClC,SAAU,EAAS,OAAS,EAAI,EAAW,CAAC,yBAAyB,CACrE,aAAc,EACf,CAID,OAHI,IAAY,IAAA,IACd,OAAO,OAAO,EAAS,CAAE,gBAAiB,EAAS,CAAC,CAE/C,ECtIT,SAAS,EAAI,EAAmC,CAC9C,OAAO,IAAU,IAAA,GAAY,IAAM,IAAI,EAAQ,KAAK,QAAQ,EAAE,CAAC,GAGjE,SAAgBC,EAAa,EAAsB,EAAoC,CACrF,IAAM,EAAkB,EAAE,CAEpB,EACJ,EAAQ,SAAW,YACf,YACA,EAAQ,SAAW,KACjB,KACA,EAAQ,SAAW,cACjB,yBACA,cAEV,EAAM,KAAK,eAAe,IAAW,CACrC,EAAM,KACJ,SAAS,EAAU,QAAQ,UAAU,EAAU,aAAa,OAAO,EAAU,YAC9E,CACD,EAAM,KAAK,GAAG,CAEd,EAAM,KAAK,uEAAuE,CAClF,IAAK,IAAM,KAAK,EAAU,MAAO,CAC/B,IAAM,EAAK,EAAE,OAAO,OAAO,GAAG,CAAC,MAAM,EAAG,GAAG,CACrC,EAAQ,EAAE,MAAM,OAAO,EAAE,CACzB,GAAQ,EAAE,cAAgB,EAAI,EAAE,SAAS,CAAG,OAAO,SAAS,EAAE,CAC9D,EAAO,GAAG,EAAE,OAAO,GAAG,EAAE,YAAY,SAAS,EAAE,CAC/C,EAAQ,OAAO,EAAE,YAAY,CAAC,SAAS,EAAE,CACzC,EAAO,OAAO,EAAE,cAAc,CAAC,SAAS,EAAE,CAChD,EAAM,KAAK,GAAG,EAAG,GAAG,EAAM,GAAG,EAAK,GAAG,EAAK,GAAG,EAAM,GAAG,IAAO,CAO/D,GALA,EAAM,KAAK,GAAG,CAEd,EAAM,KACJ,oBAAoB,EAAI,EAAU,gBAAgB,CAAC,YAAY,EAAU,YAAY,kBAAkB,EAAU,cAAc,GAChI,CACG,EAAQ,kBAAoB,IAAA,GAAW,CACzC,IAAM,EAAO,EAAQ,iBAAmB,EAAI,IAAM,GAClD,EAAM,KAAK,oBAAoB,IAAO,EAAQ,gBAAgB,QAAQ,EAAE,CAAC,IAAI,CAE/E,EAAM,KAAK,qBAAqB,EAAU,eAAiB,KAAK,QAAQ,EAAE,CAAC,WAAW,CAClF,EAAU,mBAAqB,GAGjC,EAAM,KAAK,qBAAqB,EAAU,mBAAmB,4CAA4C,CAG3G,IAAM,EAAI,EAAU,iBAChB,GACF,EAAM,KACJ,oBAAoB,EAAE,MAAM,QAAQ,EAAE,CAAC,WAAW,EAAE,UAAU,QAAQ,EAAE,CAAC,UACnE,EAAE,WAAW,QAAQ,EAAE,CAAC,UAAU,KAAK,MAAM,EAAE,YAAc,EAAE,aAAa,CAAC,SACpF,CAGH,EAAM,KAAK,GAAG,CACd,IAAK,IAAM,KAAW,EAAQ,SAC5B,EAAM,KAAK,KAAK,IAAU,CAU5B,OAPI,EAAU,cAAgB,IAC5B,EAAM,KAAK,GAAG,CACd,EAAM,KACJ,SAAS,EAAU,cAAc,2FAClC,EAGI,EAAM,KAAK;EAAK,CCxCzB,MAAM,EAAqB,CAAC,OAAQ,WAAY,YAAY,CAG5D,SAAgB,EAAqB,EAAmB,EAAkC,CACxF,IAAI,EACE,EAAyB,EAAE,CAC3B,EAAe,IAAI,IACrB,EAAQ,EAEZ,IAAK,IAAM,KAAQ,EAAQ,MAAM;EAAK,CAAE,CACtC,GAAI,EAAK,MAAM,GAAK,GAClB,SAEF,IAAI,EACJ,GAAI,CACF,EAAQ,KAAK,MAAM,EAAK,MAClB,CACN,SAGF,GAAI,EAAM,OAAS,iBAAmB,IAAW,IAAA,GAAW,CAC1D,IAAM,EAAO,EAAM,SAAU,KACzB,OAAO,GAAS,UAAY,EAAK,MAAM,GAAK,KAC9C,EAAS,GAEX,SAGF,GAAI,EAAM,OAAS,kBAAmB,CACpC,IAAM,EAAO,EAAM,SAAU,KACzB,OAAO,GAAS,UAClB,EAAa,KAAK,EAAK,CAEzB,IAAM,EAAO,EAAM,SAAU,UAC7B,GAAI,OAAO,GAAS,UAAY,EAC9B,IAAK,IAAM,KAAO,EAAoB,CACpC,IAAM,EAAS,EAAiC,GAC5C,OAAO,GAAU,UAAY,IAAU,IACzC,EAAa,IAAI,EAAM,CAI7B,SAGE,EAAM,OAAS,YACjB,IAIJ,MAAO,CACL,YACA,eACA,aAAc,CAAC,GAAG,EAAa,CAAC,MAAM,CACtC,QACA,GAAI,IAAW,IAAA,GAAyB,EAAE,CAAf,CAAE,SAAQ,CACtC,CAYH,SAAS,EAAe,EAAmC,CACzD,IAAM,EACJ,EAAS,SAAW,IAAA,GAKhB;0CAJA,cAAc,EAAS,OACpB,MAAM;EAAK,CACX,IAAK,GAAS,KAAK,IAAO,CAC1B,KAAK;EAAK,GAGb,EACJ,EAAS,aAAa,OAAS,EAC3B,gDAAgD,EAAS,aAAa,KAAK,MAAM,CAAC,IAClF,GAEN,MAAO,kCAAkC,EAAS,UAAU;;EAE5D,EAAY;;EAEZ,EAAS;;;EAMX,SAAS,EAAa,EAAmC,CACvD,IAAM,EACJ,EAAS,aAAa,OAAS,EAC3B,EAAS,aAAa,IAAK,GAAM,OAAO,EAAE,IAAI,CAAC,KAAK;EAAK,CACzD,uBAEN,MAAO,eAAe,EAAS,UAAU;;WAEhC,EAAS,MAAM;SACjB,EAAS,aAAa,OAAO;;;;EAIpC,EAAM;;;;;;;;;;;;;;;;EAkBR,SAAgB,EACd,EACA,EACA,EAA8D,GAC/C,CACf,IAAM,EAAU,EAAiB,EAAU,CAC3C,GAAI,IAAY,IAAA,GACd,MAAU,MACR,8BAA8B,EAAU,cAAc,EAAc,EAAU,CAAC,GAChF,CAGH,IAAM,EAAW,EAAqB,EAAW,EAAQ,CACnD,EAAU,EAAK,EAAW,aAAa,EAAU,MAAM,EAAG,EAAE,GAAG,CACrE,GAAI,EAAW,EAAQ,CACrB,MAAU,MACR,GAAG,EAAQ,uEACZ,CAOH,OAJA,EAAU,EAAK,EAAS,UAAU,CAAE,CAAE,UAAW,GAAM,CAAC,CACxD,EAAc,EAAK,EAAS,YAAY,CAAE,EAAe,EAAS,CAAE,OAAO,CAC3E,EAAc,EAAK,EAAS,gBAAgB,CAAE,EAAa,EAAS,CAAE,OAAO,CAEtE,CAAE,UAAS,WAAU,CAG9B,SAAS,GAAiB,EAAuC,CAC/D,IAAM,EAAO,EAAc,EAAU,CACrC,OAAO,EAAW,EAAK,CAAG,EAAa,EAAM,OAAO,CAAG,IAAA"}

package/dist/index.d.ts

@@ -7,8 +7,10 @@
  */
 /** 判分方式（RFC-316 D3 判分优先级的落地形态）。 */
 type JudgeKind = /** 可执行判据：verify 脚本 exit 0 = pass。coding 任务首选。 */'script' /** golden 断言：最终输出包含/等于期望，或工具调用序列匹配。 */ | 'golden' /** 无机器判据：只评 liveness（跑完没崩），不进质量分。 */ | 'liveness';
-/** 单次运行的三态结果（RFC-316 D6 规则 5：infra_error 不进 pass 率）。 */
-type RunOutcome = 'pass' | 'fail' | 'infra_error';
+/** 单次运行的四态结果（RFC-316 D6 规则 5：infra_error 不进 pass 率）。
+ *  RFC-353 S9 新增 'skipped_budget'：预算耗尽跳过的运行——"未尝试"而非"infra 故障"，
+ *  必须与真实 infra 故障区分（否则本地预算决策会污染 infraErrorRate、误判套件失效）。 */
+type RunOutcome = 'pass' | 'fail' | 'infra_error' | 'skipped_budget';
 /** golden 断言配置——两条判据可同时给，均满足才算 pass。 */
 interface GoldenSpec {
   /** 最终回答须包含的子串（大小写不敏感）。 */
@@ -73,10 +75,13 @@ interface CaseResult {
   runs: readonly RunResult[];
   /** pass 次数。 */
   passed: number;
-  /** 参与统计的有效次数（= runs 数 - infra_error 数）。 */
+  /** 参与统计的有效次数（= runs 数 - infra_error 数 - skipped_budget 数）。 */
   effective: number;
   /** infra_error 次数。 */
   infraErrors: number;
+  /** RFC-353 S9: 预算耗尽跳过的次数——不进 effective/passRate/infraErrorRate，
+   *  在 scorecard 中单独呈现为"未完成评测"，不伪装成 infra 故障。 */
+  skippedBudget: number;
   /** pass 率（passed/effective）；effective=0 时 undefined。 */
   passRate?: number;
   /** judge='liveness' 时为 true——不进质量分（RFC-316 规则 2 诚实边界）。 */
@@ -99,6 +104,9 @@ interface Scorecard {
   overallPassRate?: number;
   /** infra_error 占全部运行次数的比例。 */
   infraErrorRate: number;
+  /** RFC-353 S9: 全部 case 累计的 skipped_budget 次数——预算耗尽跳过的运行，
+   *  提示"未完整评测"，不计入 infraErrorRate/invalid 判定。 */
+  skippedBudgetTotal: number;
   /** infraErrorRate 超阈值时为 true——本次运行不参与基线对比（RFC-316 D6 规则 5）。 */
   invalid: boolean;
   /** 聚合的运行指标（所有有效运行的均值）。 */
@@ -269,7 +277,7 @@ interface RunnerDeps {
 }
 /** 跑单个 case 的 n 次运行。 */
 declare function runCase(spec: CaseSpec, deps: RunnerDeps): Promise<CaseResult>;
-/** 把 n 次运行汇总成 CaseResult（三态统计的唯一实现处）。 */
+/** 把 n 次运行汇总成 CaseResult（四态统计的唯一实现处）。 */
 declare function summarizeCase(spec: CaseSpec, runs: readonly RunResult[]): CaseResult;
 /** 跑整套 case，产出 scorecard。 */
 declare function runSuite(cases: readonly CaseSpec[], suiteVersion: string, deps: RunnerDeps): Promise<Scorecard>;

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","names":[],"sources":["../src/types.ts","../src/case-loader.ts","../src/workspace.ts","../src/otto-invoker.ts","../src/trace-metrics.ts","../src/verify.ts","../src/runner.ts","../src/baseline.ts","../src/report.ts","../src/extract.ts","../src/tier-a.ts"],"mappings":";;AAQA;;;;;AASA;AAAA,KATY,SAAA;;KASA,UAAA;AAWZ;AAAA,UARiB,UAAA;;EAEf,cAAA;EAQA;EANA,YAAA;AAAA;;UAIe,QAAA;EAUf;EARA,EAAA;EAUA;EARA,MAAA;EAYA;EAVA,KAAA,EAAO,SAAA;EAcP;EAZA,YAAA;EAYU;EAVV,MAAA,GAAS,UAAA;EAcgB;EAZzB,IAAA;EAYyB;EAVzB,SAAA;EAcA;EAZA,SAAA;EAeA;EAbA,GAAA;EAeA;EAbA,UAAA;AAAA;;UAIe,UAAA;EAiBA;EAff,KAAA;;EAEA,SAAA;EAcA;EAZA,UAAA;EACA,WAAA;EACA,YAAA;EACA,eAAA;EAgBA;EAdA,OAAA;EAcoB;EAZpB,cAAA;AAAA;;UAIe,SAAA;EACf,OAAA,EAAS,UAAA;EAYT;EAVA,MAAA;EAWO;EATP,SAAA;EAUe;EARf,MAAA;EACA,OAAA,GAAU,UAAA;AAAA;;UAIK,UAAA;EACf,MAAA;EACA,KAAA,EAAO,SAAA;EACP,IAAA,WAAe,SAAA;EAcS;EAZxB,MAAA;EA+B6B;EA7B7B,SAAA;EAcA;EAZA,WAAA;EAeA;EAbA,QAAA;EAeA;EAbA,aAAA;AAAA;;UAIe,SAAA;EAmBf;EAjBA,YAAA;EAiB6B;EAf7B,OAAA;EAmBe;EAjBf,SAAA;EACA,KAAA,WAAgB,UAAA;EAwBO;EAtBvB,WAAA;EAgBA;EAdA,aAAA;EAiBA;EAfA,eAAA;EAkBA;EAhBA,cAAA;EAgBwB;EAdxB,OAAA;EAc8B;EAZ9B,gBAAA,GAAmB,UAAA;AAAA;;UAIJ,aAAA;EACf,OAAA;EACA,YAAA;EACA,UAAA;;EAEA,MAAA;EACA,eAAA;EAaA;EAXA,aAAA,EAAe,QAAA,CAAS,MAAA;AAAA;AAAA,UAGT,YAAA;EACf,OAAA,WAAkB,aAAA;AAAA;;UAIH,iBAAA;EACf,MAAA;ECjEsB;EDmEtB,QAAA;ECnEuB;EDqEvB,eAAA;ECdc;EDgBd,YAAA;AAAA;;;;iBCvEc,QAAA,CAAS,GAAA,WAAc,QAAA;;iBAuDvB,SAAA,CAAU,SAAA,oBAA6B,QAAA;AD5GvD;;;;;AAQA;AARA,iBC8HgB,mBAAA,CAAoB,KAAA,WAAgB,QAAA;;;UCpInC,aAAA;EFGK;EEDpB,GAAA;EFIe;EEFf,OAAA;AAAA;;iBAIc,eAAA,CAAgB,IAAA,EAAM,QAAA,GAAW,aAAA;;;;AFdjD;;;;;AASA;AAAA,UGPiB,cAAA;EACf,SAAA;EACA,MAAA;EACA,YAAA;EACA,QAAA;AAAA;AAAA,UAGe,aAAA;EACf,MAAA;EHMY;EGJZ,GAAA;EACA,SAAA;AAAA;AAAA,KAGU,cAAA;EACN,IAAA;EAAY,MAAA,EAAQ,cAAA;AAAA;EAEpB,IAAA;EAAgB,MAAA;AAAA;;;;;;EAKhB,IAAA;EAAe,MAAA;AAAA;;UAGJ,WAAA;EACf,MAAA,CAAO,OAAA,EAAS,aAAA,GAAgB,OAAA,CAAQ,cAAA;AAAA;;;;;;;iBAS1B,eAAA,CAAgB,MAAA;AAAA,UAqBf,mBAAA;EHGA;EGDf,GAAA;;EAEA,GAAA,GAAM,QAAA,CAAS,MAAA;AAAA;;iBAID,kBAAA,CAAmB,OAAA,GAAS,mBAAA,GAA2B,WAAA;;;AHtDvE;AAAA,iBISgB,aAAA,CAAc,SAAA;;UAKb,QAAA;EACf,OAAA,EAAS,UAAA;EJPM;EISf,SAAA;AAAA;;;;;iBAOc,aAAA,CAAc,OAAA,WAAkB,QAAA;;iBA6DhC,YAAA,CAAa,SAAA,WAAoB,QAAA;;iBASjC,cAAA,CAAe,OAAA,WAAkB,UAAA,KAAe,UAAA;;;UCrG/C,WAAA;EACf,IAAA,EAAM,QAAA;ELGc;EKDpB,YAAA;ELIyB;EKFzB,QAAA;ELIA;EKFA,SAAA;AAAA;AAAA,UAGe,YAAA;EACf,MAAA;EACA,MAAA;AAAA;;iBAIc,sBAAA,CACd,MAAA,qBACA,QAAA;;;;;;;;iBAyDc,SAAA,CAAU,KAAA,EAAO,WAAA,GAAc,YAAA;;;;;;;AL5D/C;cMHa,6BAAA;AAAA,UAKI,UAAA;EACf,OAAA,EAAS,WAAA;ENDT;EMGA,SAAA,IAAa,SAAA,aAAsB,QAAA;ENCnC;EMCA,GAAA;ENCA;EMCA,aAAA,IAAiB,MAAA,UAAgB,KAAA,UAAe,MAAA,EAAQ,SAAA;AAAA;;iBAWpC,OAAA,CAAQ,IAAA,EAAM,QAAA,EAAU,IAAA,EAAM,UAAA,GAAa,OAAA,CAAQ,UAAA;;iBAwGzD,aAAA,CAAc,IAAA,EAAM,QAAA,EAAU,IAAA,WAAe,SAAA,KAAc,UAAA;;iBAsBrD,QAAA,CACpB,KAAA,WAAgB,QAAA,IAChB,YAAA,UACA,IAAA,EAAM,UAAA,GACL,OAAA,CAAQ,SAAA;;iBAWK,cAAA,CACd,KAAA,WAAgB,UAAA,IAChB,YAAA,UACA,SAAA,WACC,SAAA;;;;AN7KH;;;;cOEa,uBAAA;AAAA,iBAEG,aAAA,CAAc,IAAA,WAAe,YAAA;AAAA,iBAQ7B,aAAA,CAAc,IAAA,UAAc,IAAA,EAAM,YAAA;;iBAMlC,YAAA,CACd,IAAA,EAAM,YAAA,EACN,OAAA,UACA,YAAA,WACC,aAAA;;iBAKa,cAAA,CACd,IAAA,EAAM,YAAA,EACN,SAAA,EAAW,SAAA,EACX,MAAA,WACC,YAAA;;iBAwBa,eAAA,CACd,SAAA,EAAW,SAAA,EACX,QAAA,EAAU,aAAA,eACT,iBAAA;;;iBCjEa,YAAA,CAAa,SAAA,EAAW,SAAA,EAAW,OAAA,EAAS,iBAAA;;;;ARL5D;;;;;AASA;;;;;AAGA;;;USKiB,eAAA;EACf,SAAA;ETEe;ESAf,MAAA;;EAEA,YAAA;ETAA;ESEA,YAAA;ETEA;ESAA,KAAA;AAAA;;iBAMc,oBAAA,CAAqB,SAAA,UAAmB,OAAA,WAAkB,eAAA;AAAA,UAkHzD,aAAA;EACf,OAAA;EACA,QAAA,EAAU,eAAA;AAAA;;iBAII,WAAA,CACd,SAAA,UACA,SAAA,UACA,gBAAA,IAAmB,SAAA,kCAClB,aAAA;;;;AT5JH;;;;;AASA;;;;;AAGA;;;;;UUeiB,aAAA;EACf,EAAA;;EAEA,MAAA;EVRA;EUUA,KAAA;EVNA;EUQA,SAAA;EACA,GAAA;EACA,UAAA;AAAA;AAAA,UAGe,cAAA;EACf,IAAA;EACA,SAAA,EAAW,MAAA;AAAA;;UAII,WAAA;EACf,UAAA;EACA,SAAA,WAAoB,cAAA;EVHK;EUKzB,SAAA,EAAW,QAAA,CAAS,MAAA;EACpB,OAAA;IACE,UAAA;IACA,QAAA;EAAA;AAAA;AAAA,UAIa,YAAA;EACf,UAAA;EACA,OAAA;EACA,OAAA;AAAA;;iBAMc,YAAA,CAAa,GAAA,WAAc,aAAA;AAAA,iBAmD3B,iBAAA,CAAkB,cAAA,oBAAkC,aAAA;;iBAyG9C,WAAA,CAAY,QAAA,EAAU,aAAA,GAAgB,OAAA,CAAQ,WAAA;AAAA,iBA+DpD,UAAA,CAAW,QAAA,EAAU,aAAA;AAAA,iBAIrB,WAAA,CAAY,QAAA,EAAU,aAAA,EAAe,MAAA,EAAQ,WAAA;;iBAK7C,eAAA,CAAgB,QAAA,EAAU,aAAA,EAAe,MAAA,EAAQ,WAAA,GAAc,YAAA"}
+{"version":3,"file":"index.d.ts","names":[],"sources":["../src/types.ts","../src/case-loader.ts","../src/workspace.ts","../src/otto-invoker.ts","../src/trace-metrics.ts","../src/verify.ts","../src/runner.ts","../src/baseline.ts","../src/report.ts","../src/extract.ts","../src/tier-a.ts"],"mappings":";;AAQA;;;;;AAWA;AAAA,KAXY,SAAA;;;AAsBZ;KAXY,UAAA;;UAGK,UAAA;EAUf;EARA,cAAA;EAYA;EAVA,YAAA;AAAA;;UAIe,QAAA;EAYf;EAVA,EAAA;EAcA;EAZA,MAAA;EAgBA;EAdA,KAAA,EAAO,SAAA;EAcG;EAZV,YAAA;EAgByB;EAdzB,MAAA,GAAS,UAAA;EAcgB;EAZzB,IAAA;EAgBA;EAdA,SAAA;EAiBA;EAfA,SAAA;EAiBA;EAfA,GAAA;EAmBA;EAjBA,UAAA;AAAA;AAqBF;AAAA,UAjBiB,UAAA;;EAEf,KAAA;EAgBA;EAdA,SAAA;EAgBA;EAdA,UAAA;EACA,WAAA;EACA,YAAA;EACA,eAAA;EAgBoB;EAdpB,OAAA;EAkBe;EAhBf,cAAA;AAAA;;UAIe,SAAA;EACf,OAAA,EAAS,UAAA;EAaF;EAXP,MAAA;EAYe;EAVf,SAAA;EAcA;EAZA,MAAA;EACA,OAAA,GAAU,UAAA;AAAA;;UAIK,UAAA;EACf,MAAA;EACA,KAAA,EAAO,SAAA;EACP,IAAA,WAAe,SAAA;;EAEf,MAAA;EAiBA;EAfA,SAAA;EAmBA;EAjBA,WAAA;EAkBgB;;EAfhB,aAAA;EAqBA;EAnBA,QAAA;EAwBA;EAtBA,aAAA;AAAA;;UAIe,SAAA;EAsBc;EApB7B,YAAA;EAwB4B;EAtB5B,OAAA;EA8BuB;EA5BvB,SAAA;EACA,KAAA,WAAgB,UAAA;EAsBhB;EApBA,WAAA;EAuBA;EArBA,aAAA;EAuBe;EArBf,eAAA;EAqB8B;EAnB9B,cAAA;EAsBe;;EAnBf,kBAAA;EAoBA;EAlBA,OAAA;EAsBe;EApBf,gBAAA,GAAmB,UAAA;AAAA;;UAIJ,aAAA;EACf,OAAA;EACA,YAAA;EACA,UAAA;EAoBY;EAlBZ,MAAA;EACA,eAAA;;EAEA,aAAA,EAAe,QAAA,CAAS,MAAA;AAAA;AAAA,UAGT,YAAA;EACf,OAAA,WAAkB,aAAA;AAAA;;UAIH,iBAAA;EACf,MAAA;;EAEA,QAAA;ECpB6D;EDsB7D,eAAA;ECJiC;EDMjC,YAAA;AAAA;;;;iBC/Ec,QAAA,CAAS,GAAA,WAAc,QAAA;;iBAuDvB,SAAA,CAAU,SAAA,oBAA6B,QAAA;AD1GvD;;;;;AAQA;AARA,iBC4HgB,mBAAA,CAAoB,KAAA,WAAgB,QAAA;;;UCpInC,aAAA;EFKK;EEHpB,GAAA;EFMe;EEJf,OAAA;AAAA;;iBAIc,eAAA,CAAgB,IAAA,EAAM,QAAA,GAAW,aAAA;;;;AFdjD;;;;;AAWA;AAAA,UGTiB,cAAA;EACf,SAAA;EACA,MAAA;EACA,YAAA;EACA,QAAA;AAAA;AAAA,UAGe,aAAA;EACf,MAAA;EHQY;EGNZ,GAAA;EACA,SAAA;AAAA;AAAA,KAGU,cAAA;EACN,IAAA;EAAY,MAAA,EAAQ,cAAA;AAAA;EAEpB,IAAA;EAAgB,MAAA;AAAA;;;;;;EAKhB,IAAA;EAAe,MAAA;AAAA;;UAGJ,WAAA;EACf,MAAA,CAAO,OAAA,EAAS,aAAA,GAAgB,OAAA,CAAQ,cAAA;AAAA;;;;;;;iBAS1B,eAAA,CAAgB,MAAA;AAAA,UAqBf,mBAAA;EHKA;EGHf,GAAA;;EAEA,GAAA,GAAM,QAAA,CAAS,MAAA;AAAA;;iBAID,kBAAA,CAAmB,OAAA,GAAS,mBAAA,GAA2B,WAAA;;;AHpDvE;AAAA,iBIOgB,aAAA,CAAc,SAAA;;UAKb,QAAA;EACf,OAAA,EAAS,UAAA;EJLM;EIOf,SAAA;AAAA;;;;;iBAOc,aAAA,CAAc,OAAA,WAAkB,QAAA;;iBA6DhC,YAAA,CAAa,SAAA,WAAoB,QAAA;;iBASjC,cAAA,CAAe,OAAA,WAAkB,UAAA,KAAe,UAAA;;;UCrG/C,WAAA;EACf,IAAA,EAAM,QAAA;ELKc;EKHpB,YAAA;ELMyB;EKJzB,QAAA;ELMA;EKJA,SAAA;AAAA;AAAA,UAGe,YAAA;EACf,MAAA;EACA,MAAA;AAAA;;iBAIc,sBAAA,CACd,MAAA,qBACA,QAAA;;;;;;;;iBAyDc,SAAA,CAAU,KAAA,EAAO,WAAA,GAAc,YAAA;;;;;;;AL1D/C;cMLa,6BAAA;AAAA,UAKI,UAAA;EACf,OAAA,EAAS,WAAA;ENCT;EMCA,SAAA,IAAa,SAAA,aAAsB,QAAA;ENGnC;EMDA,GAAA;ENGA;EMDA,aAAA,IAAiB,MAAA,UAAgB,KAAA,UAAe,MAAA,EAAQ,SAAA;AAAA;;iBAWpC,OAAA,CAAQ,IAAA,EAAM,QAAA,EAAU,IAAA,EAAM,UAAA,GAAa,OAAA,CAAQ,UAAA;;iBA0GzD,aAAA,CAAc,IAAA,EAAM,QAAA,EAAU,IAAA,WAAe,SAAA,KAAc,UAAA;;iBAwBrD,QAAA,CACpB,KAAA,WAAgB,QAAA,IAChB,YAAA,UACA,IAAA,EAAM,UAAA,GACL,OAAA,CAAQ,SAAA;;iBAWK,cAAA,CACd,KAAA,WAAgB,UAAA,IAChB,YAAA,UACA,SAAA,WACC,SAAA;;;;AN/KH;;;;cOAa,uBAAA;AAAA,iBAEG,aAAA,CAAc,IAAA,WAAe,YAAA;AAAA,iBAQ7B,aAAA,CAAc,IAAA,UAAc,IAAA,EAAM,YAAA;;iBAMlC,YAAA,CACd,IAAA,EAAM,YAAA,EACN,OAAA,UACA,YAAA,WACC,aAAA;;iBAKa,cAAA,CACd,IAAA,EAAM,YAAA,EACN,SAAA,EAAW,SAAA,EACX,MAAA,WACC,YAAA;;iBAwBa,eAAA,CACd,SAAA,EAAW,SAAA,EACX,QAAA,EAAU,aAAA,eACT,iBAAA;;;iBCjEa,YAAA,CAAa,SAAA,EAAW,SAAA,EAAW,OAAA,EAAS,iBAAA;;;;ARL5D;;;;;AAWA;;;;;AAGA;;;USGiB,eAAA;EACf,SAAA;ETIe;ESFf,MAAA;;EAEA,YAAA;ETEA;ESAA,YAAA;ETIA;ESFA,KAAA;AAAA;;iBAMc,oBAAA,CAAqB,SAAA,UAAmB,OAAA,WAAkB,eAAA;AAAA,UAkHzD,aAAA;EACf,OAAA;EACA,QAAA,EAAU,eAAA;AAAA;;iBAII,WAAA,CACd,SAAA,UACA,SAAA,UACA,gBAAA,IAAmB,SAAA,kCAClB,aAAA;;;;AT5JH;;;;;AAWA;;;;;AAGA;;;;;UUaiB,aAAA;EACf,EAAA;;EAEA,MAAA;EVNA;EUQA,KAAA;EVJA;EUMA,SAAA;EACA,GAAA;EACA,UAAA;AAAA;AAAA,UAGe,cAAA;EACf,IAAA;EACA,SAAA,EAAW,MAAA;AAAA;;UAII,WAAA;EACf,UAAA;EACA,SAAA,WAAoB,cAAA;EVDK;EUGzB,SAAA,EAAW,QAAA,CAAS,MAAA;EACpB,OAAA;IACE,UAAA;IACA,QAAA;EAAA;AAAA;AAAA,UAIa,YAAA;EACf,UAAA;EACA,OAAA;EACA,OAAA;AAAA;;iBAMc,YAAA,CAAa,GAAA,WAAc,aAAA;AAAA,iBAmD3B,iBAAA,CAAkB,cAAA,oBAAkC,aAAA;;iBAyG9C,WAAA,CAAY,QAAA,EAAU,aAAA,GAAgB,OAAA,CAAQ,WAAA;AAAA,iBA+DpD,UAAA,CAAW,QAAA,EAAU,aAAA;AAAA,iBAIrB,WAAA,CAAY,QAAA,EAAU,aAAA,EAAe,MAAA,EAAQ,WAAA;;iBAK7C,eAAA,CAAgB,QAAA,EAAU,aAAA,EAAe,MAAA,EAAQ,WAAA,GAAc,YAAA"}

package/dist/index.js

@@ -1 +1 @@
-import{C as e,S as t,T as n,_ as r,a as i,b as a,c as o,d as s,f as c,g as l,h as u,i as d,l as f,m as p,n as m,o as h,p as g,r as _,s as v,t as y,u as b,v as x,w as S,x as C,y as w}from"./extract-B65xQZqv.js";import{t as T}from"./workspace-C_kTWfF8.js";import{a as E,i as D,n as O,o as k,r as A,t as j}from"./tier-a-DvMUFMGy.js";export{b as INFRA_ERROR_INVALID_THRESHOLD,d as PASS_RATE_REGRESSION_PP,r as averageMetrics,s as buildScorecard,C as classifyFailure,j as compareToGolden,e as computeSuiteVersion,t as createSpawnInvoker,T as createWorkspace,y as extractCase,i as findBaseline,O as goldenPath,h as judgeRegression,v as loadBaselines,S as loadCase,A as loadScenario,D as loadScenarioSuite,n as loadSuite,u as matchesToolSubsequence,m as parseSessionSkeleton,x as parseTraceRun,w as readTraceRun,_ as renderReport,c as runCase,E as runScenario,g as runSuite,o as saveBaselines,p as summarizeCase,a as traceFilePath,f as upsertBaseline,l as verifyRun,k as writeGolden};
+import{C as e,S as t,T as n,_ as r,a as i,b as a,c as o,d as s,f as c,g as l,h as u,i as d,l as f,m as p,n as m,o as h,p as g,r as _,s as v,t as y,u as b,v as x,w as S,x as C,y as w}from"./extract-CJuoP2bV.js";import{t as T}from"./workspace-C_kTWfF8.js";import{a as E,i as D,n as O,o as k,r as A,t as j}from"./tier-a-DvMUFMGy.js";export{b as INFRA_ERROR_INVALID_THRESHOLD,d as PASS_RATE_REGRESSION_PP,r as averageMetrics,s as buildScorecard,C as classifyFailure,j as compareToGolden,e as computeSuiteVersion,t as createSpawnInvoker,T as createWorkspace,y as extractCase,i as findBaseline,O as goldenPath,h as judgeRegression,v as loadBaselines,S as loadCase,A as loadScenario,D as loadScenarioSuite,n as loadSuite,u as matchesToolSubsequence,m as parseSessionSkeleton,x as parseTraceRun,w as readTraceRun,_ as renderReport,c as runCase,E as runScenario,g as runSuite,o as saveBaselines,p as summarizeCase,a as traceFilePath,f as upsertBaseline,l as verifyRun,k as writeGolden};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@x-otto/eval",
-  "version": "0.0.1-alpha.0",
+  "version": "0.0.1-alpha.2",
   "bin": {
     "otto-eval": "./dist/cli.js",
     "otto-eval-tier-a": "./dist/tier-a-cli.js"
@@ -18,13 +18,13 @@
     }
   },
   "dependencies": {
-    "@x-otto/agent": "0.0.1-alpha.0",
-    "@x-otto/env": "0.1.0-alpha.1",
-    "@x-otto/interchange": "0.1.0-alpha.1",
-    "@x-otto/provider": "0.1.0-alpha.1",
-    "@x-otto/shared": "0.1.0-alpha.1",
-    "@x-otto/tools": "0.0.1-alpha.0",
-    "yaml": "2.8.3"
+    "yaml": "2.8.3",
+    "@x-otto/agent": "0.0.1-alpha.2",
+    "@x-otto/interchange": "0.1.0-alpha.3",
+    "@x-otto/env": "0.1.0-alpha.6",
+    "@x-otto/shared": "0.1.0-alpha.6",
+    "@x-otto/provider": "0.1.0-alpha.5",
+    "@x-otto/tools": "0.0.1-alpha.2"
   },
   "devDependencies": {
     "typescript": "6.0.2",

package/dist/extract-B65xQZqv.js.map

@@ -1 +0,0 @@
-{"version":3,"file":"extract-B65xQZqv.js","names":["parseYaml","renderReport"],"sources":["../src/case-loader.ts","../src/otto-invoker.ts","../src/trace-metrics.ts","../src/verify.ts","../src/runner.ts","../src/baseline.ts","../src/report.ts","../src/extract.ts"],"sourcesContent":["/**\n * case-loader.ts —— 读取并校验 `evals/cases/<id>/case.yaml`（RFC-316 D4）。\n *\n * 校验失败一律抛错而非静默降级：一个格式错的 case 静默变成 liveness case，\n * 会让 scorecard 悄悄失去质量信号——这正是 RFC-316 规则 2 要防的「假质量分」。\n */\n\nimport { createHash } from 'node:crypto'\nimport { readFileSync, readdirSync, existsSync, statSync } from 'node:fs'\nimport { join } from 'node:path'\nimport { parse as parseYaml } from 'yaml'\n\nimport type { CaseSpec, GoldenSpec, JudgeKind } from './types'\n\nconst DEFAULT_RUNS = 3\nconst DEFAULT_TIMEOUT_MS = 300_000\nconst DEFAULT_VERIFY_SCRIPT = 'verify.sh'\n\nconst JUDGE_KINDS: readonly JudgeKind[] = ['script', 'golden', 'liveness']\n\nfunction requireString(raw: Record<string, unknown>, key: string, caseId: string): string {\n  const value = raw[key]\n  if (typeof value !== 'string' || value.trim() === '') {\n    throw new Error(`case '${caseId}': field '${key}' must be a non-empty string`)\n  }\n  return value\n}\n\nfunction optionalPositiveInt(\n  raw: Record<string, unknown>,\n  key: string,\n  caseId: string,\n  fallback: number,\n): number {\n  const value = raw[key]\n  if (value === undefined) {\n    return fallback\n  }\n  if (typeof value !== 'number' || !Number.isInteger(value) || value <= 0) {\n    throw new Error(`case '${caseId}': field '${key}' must be a positive integer`)\n  }\n  return value\n}\n\nfunction parseGolden(raw: unknown, caseId: string): GoldenSpec {\n  if (typeof raw !== 'object' || raw === null) {\n    throw new Error(`case '${caseId}': judge='golden' requires a 'golden' mapping`)\n  }\n  const record = raw as Record<string, unknown>\n  const golden: GoldenSpec = {}\n  const answerContains = record['answerContains']\n  if (answerContains !== undefined) {\n    if (!Array.isArray(answerContains) || answerContains.some((x) => typeof x !== 'string')) {\n      throw new Error(`case '${caseId}': golden.answerContains must be a string array`)\n    }\n    Object.assign(golden, { answerContains: answerContains as string[] })\n  }\n  const toolSequence = record['toolSequence']\n  if (toolSequence !== undefined) {\n    if (!Array.isArray(toolSequence) || toolSequence.some((x) => typeof x !== 'string')) {\n      throw new Error(`case '${caseId}': golden.toolSequence must be a string array`)\n    }\n    Object.assign(golden, { toolSequence: toolSequence as string[] })\n  }\n  if (!golden.answerContains && !golden.toolSequence) {\n    throw new Error(\n      `case '${caseId}': judge='golden' requires at least one of golden.answerContains / golden.toolSequence`,\n    )\n  }\n  return golden\n}\n\n/** 解析单个 case 目录。`dir` 必须已存在且含 case.yaml。 */\nexport function loadCase(dir: string): CaseSpec {\n  const caseId = dir.split('/').filter(Boolean).at(-1) ?? dir\n  const yamlPath = join(dir, 'case.yaml')\n  if (!existsSync(yamlPath)) {\n    throw new Error(`case '${caseId}': missing case.yaml at ${yamlPath}`)\n  }\n\n  const parsed: unknown = parseYaml(readFileSync(yamlPath, 'utf8'))\n  if (typeof parsed !== 'object' || parsed === null) {\n    throw new Error(`case '${caseId}': case.yaml must contain a mapping`)\n  }\n  const raw = parsed as Record<string, unknown>\n\n  const prompt = requireString(raw, 'prompt', caseId)\n  const judgeRaw = requireString(raw, 'judge', caseId)\n  if (!JUDGE_KINDS.includes(judgeRaw as JudgeKind)) {\n    throw new Error(`case '${caseId}': judge must be one of ${JUDGE_KINDS.join(' | ')}`)\n  }\n  const judge = judgeRaw as JudgeKind\n\n  const spec: CaseSpec = {\n    id: caseId,\n    prompt,\n    judge,\n    runs: optionalPositiveInt(raw, 'runs', caseId, DEFAULT_RUNS),\n    timeoutMs: optionalPositiveInt(raw, 'timeoutMs', caseId, DEFAULT_TIMEOUT_MS),\n    dir,\n    hasFixture: existsSync(join(dir, 'fixture')),\n  }\n\n  if (raw['maxTokens'] !== undefined) {\n    Object.assign(spec, { maxTokens: optionalPositiveInt(raw, 'maxTokens', caseId, 0) })\n  }\n\n  if (judge === 'script') {\n    const script =\n      raw['verifyScript'] === undefined\n        ? DEFAULT_VERIFY_SCRIPT\n        : requireString(raw, 'verifyScript', caseId)\n    if (!existsSync(join(dir, script))) {\n      throw new Error(`case '${caseId}': judge='script' but verify script '${script}' not found`)\n    }\n    Object.assign(spec, { verifyScript: script })\n  }\n\n  // golden 断言与 script 判据正交：judge='script' 的 case 也可附加 golden（工具纪律 + 产物正确性\n  // 两个判据都要满足）。judge='golden' 时 golden 必填，其余情况可选。\n  if (judge === 'golden' || raw['golden'] !== undefined) {\n    Object.assign(spec, { golden: parseGolden(raw['golden'], caseId) })\n  }\n\n  return spec\n}\n\n/** 扫描 case 根目录（`evals/cases/`），按 id 排序返回全部 case。 */\nexport function loadSuite(casesRoot: string): readonly CaseSpec[] {\n  if (!existsSync(casesRoot)) {\n    throw new Error(`cases root not found: ${casesRoot}`)\n  }\n  const dirs = readdirSync(casesRoot)\n    .filter((name) => !name.startsWith('.'))\n    .map((name) => join(casesRoot, name))\n    .filter((path) => statSync(path).isDirectory())\n    .sort()\n  return dirs.map(loadCase)\n}\n\n/**\n * 套件版本 = 全部 case 判分相关字段的内容指纹（基线分键用）。\n *\n * 只纳入**影响判分口径**的字段：case 集合、prompt、judge、golden。改超时/runs 不算换套件\n * （不影响可比性），改 prompt 或判据则必须换基线键——否则会把两套不同题目的分数当同一条曲线看。\n */\nexport function computeSuiteVersion(cases: readonly CaseSpec[]): string {\n  const canonical = cases.map((c) => ({\n    id: c.id,\n    prompt: c.prompt,\n    judge: c.judge,\n    golden: c.golden ?? null,\n  }))\n  return createHash('sha256').update(JSON.stringify(canonical)).digest('hex').slice(0, 12)\n}\n","/**\n * otto-invoker.ts —— 调用 headless otto 的唯一出口（RFC-316 规则 6：只走公开入口）。\n *\n * 端口化的理由：runner 的编排逻辑（n 次运行 / 预算护栏 / 三态统计）必须能在**不起真实模型**\n * 的情况下被单测，否则评测体系自身没有回归保护。真实实现 spawn `otto --json`，测试注入 fake。\n */\n\nimport { spawn } from 'node:child_process'\n\n/** headless `otto --json` 的输出契约（packages/cli/src/modes/run-modes.ts JsonModeResult）。 */\nexport interface OttoJsonResult {\n  sessionId: string\n  status: string\n  messageCount: number\n  response: string\n}\n\nexport interface InvokeRequest {\n  prompt: string\n  /** agent 的工作目录。 */\n  cwd: string\n  timeoutMs: number\n}\n\nexport type InvokeResponse =\n  | { kind: 'ok'; result: OttoJsonResult }\n  /** 进程跑起来了但任务失败（非零退出、输出不可解析）——算 fail。 */\n  | { kind: 'failed'; detail: string }\n  /**\n   * 基础设施问题（限流、网络、供应商 5xx、二进制缺失）——算 infra_error，不进 pass 率\n   * （RFC-316 D6 规则 5：防限流假回归）。\n   */\n  | { kind: 'infra'; detail: string }\n\n/** 调用 otto 的端口。 */\nexport interface OttoInvoker {\n  invoke(request: InvokeRequest): Promise<InvokeResponse>\n}\n\n/**\n * 判定一段 stderr 是否为基础设施故障而非任务失败。\n *\n * 保守取向：只认明确的传输层/额度层信号。误判成 infra 会让真实回归被藏起来，\n * 比误判成 fail 更危险——所以这里宁可漏放。\n */\nexport function classifyFailure(stderr: string): 'infra' | 'failed' {\n  const text = stderr.toLowerCase()\n  const infraSignals = [\n    'rate limit',\n    'rate_limit',\n    '429',\n    'overloaded',\n    'econnreset',\n    'etimedout',\n    'enotfound',\n    'socket hang up',\n    'service unavailable',\n    'internal server error',\n    'bad gateway',\n    'no model configured',\n    'command not found',\n    'enoent',\n  ]\n  return infraSignals.some((signal) => text.includes(signal)) ? 'infra' : 'failed'\n}\n\nexport interface SpawnInvokerOptions {\n  /** otto 可执行文件（默认 'otto'）。 */\n  bin?: string\n  /** 附加环境变量（如指定模型）。 */\n  env?: Readonly<Record<string, string>>\n}\n\n/** 真实实现：spawn `otto --json \"<prompt>\"`。 */\nexport function createSpawnInvoker(options: SpawnInvokerOptions = {}): OttoInvoker {\n  const bin = options.bin ?? 'otto'\n  return {\n    invoke(request) {\n      return new Promise<InvokeResponse>((resolve) => {\n        const child = spawn(bin, ['--json', request.prompt], {\n          cwd: request.cwd,\n          env: { ...process.env, ...options.env },\n          stdio: ['ignore', 'pipe', 'pipe'],\n        })\n        let stdout = ''\n        let stderr = ''\n        let settled = false\n        const finish = (response: InvokeResponse): void => {\n          if (settled) {\n            return\n          }\n          settled = true\n          clearTimeout(timer)\n          resolve(response)\n        }\n        const timer = setTimeout(() => {\n          child.kill('SIGKILL')\n          finish({ kind: 'failed', detail: `timeout after ${request.timeoutMs}ms` })\n        }, request.timeoutMs)\n\n        child.stdout.on('data', (chunk: Buffer) => {\n          stdout += chunk.toString()\n        })\n        child.stderr.on('data', (chunk: Buffer) => {\n          stderr += chunk.toString()\n        })\n        child.on('error', (error) => {\n          finish({ kind: 'infra', detail: `spawn failed: ${error.message}` })\n        })\n        child.on('close', (code) => {\n          if (code !== 0) {\n            const kind = classifyFailure(stderr)\n            finish({ kind, detail: `exit ${code}: ${stderr.trim().slice(0, 500)}` })\n            return\n          }\n          try {\n            const parsed = JSON.parse(stdout) as OttoJsonResult\n            if (typeof parsed.sessionId !== 'string') {\n              finish({ kind: 'failed', detail: 'json output missing sessionId' })\n              return\n            }\n            finish({ kind: 'ok', result: parsed })\n          } catch {\n            finish({ kind: 'failed', detail: `unparsable json output: ${stdout.slice(0, 300)}` })\n          }\n        })\n      })\n    },\n  }\n}\n","/**\n * trace-metrics.ts —— 从 trace JSONL 聚合运行指标（RFC-316 G3 / L3 层）。\n *\n * 数据源是 append-only 的 `OTTO_SESSIONS_DIR/trace/<sessionId>.jsonl`，otto 每个会话都会写。\n * 这一层是「零成本指标」：不额外跑任何东西，评测跑完顺手把成本/步数/工具成功率读出来。\n *\n * 只读两类锚点（其余事件忽略，容忍跨版本 trace）：\n * - `tool.call.end`：工具名 / 耗时 / errorCategory\n * - `turn.end`：usage（token）/ model\n */\n\nimport { readFileSync, existsSync } from 'node:fs'\nimport { join } from 'node:path'\nimport { OTTO_SESSIONS_DIR } from '@x-otto/env'\n\nimport type { RunMetrics } from './types'\n\ninterface RawTraceEvent {\n  kind?: string\n  node?: string\n  payload?: Record<string, unknown>\n}\n\nfunction readNumber(payload: Record<string, unknown> | undefined, key: string): number {\n  const value = payload?.[key]\n  return typeof value === 'number' && Number.isFinite(value) ? value : 0\n}\n\n/** trace 文件的默认路径。 */\nexport function traceFilePath(sessionId: string): string {\n  return join(OTTO_SESSIONS_DIR, 'trace', `${sessionId}.jsonl`)\n}\n\n/** 一次运行从 trace 还原出的全部可判分信息。 */\nexport interface TraceRun {\n  metrics: RunMetrics\n  /** 工具调用名序列（按 tool.call.end 发生顺序），golden.toolSequence 判据的输入。 */\n  toolNames: readonly string[]\n}\n\n/**\n * 解析 trace 内容为指标 + 工具序列。传内容而非路径，便于单测与远端 trace 复用。\n * 坏行直接跳过——trace 是 append-only 的，进程被杀时最后一行可能截断，这是正常状态。\n */\nexport function parseTraceRun(content: string): TraceRun {\n  const toolNames: string[] = []\n  const metrics = parseTraceMetrics(content, toolNames)\n  return { metrics, toolNames }\n}\n\nfunction parseTraceMetrics(content: string, toolNamesOut?: string[]): RunMetrics {\n  const metrics: RunMetrics = {\n    turns: 0,\n    toolCalls: 0,\n    toolErrors: 0,\n    inputTokens: 0,\n    outputTokens: 0,\n    cacheReadTokens: 0,\n    toolDurationMs: 0,\n  }\n\n  for (const line of content.split('\\n')) {\n    if (line.trim() === '') {\n      continue\n    }\n    let event: RawTraceEvent\n    try {\n      event = JSON.parse(line) as RawTraceEvent\n    } catch {\n      continue\n    }\n\n    if (event.node === 'tool.call.end') {\n      metrics.toolCalls++\n      metrics.toolDurationMs += readNumber(event.payload, 'durationMs')\n      if (event.payload?.['errorCategory'] !== undefined) {\n        metrics.toolErrors++\n      }\n      const name = event.payload?.['name']\n      if (typeof name === 'string') {\n        toolNamesOut?.push(name)\n      }\n      continue\n    }\n\n    if (event.node === 'turn.end') {\n      metrics.turns++\n      const usage = event.payload?.['usage']\n      if (typeof usage === 'object' && usage !== null) {\n        const u = usage as Record<string, unknown>\n        metrics.inputTokens += readNumber(u, 'inputTokens')\n        metrics.outputTokens += readNumber(u, 'outputTokens')\n        metrics.cacheReadTokens += readNumber(u, 'cacheReadTokens')\n      }\n      const model = event.payload?.['model']\n      if (typeof model === 'string' && model !== '') {\n        metrics.modelId = model\n      }\n    }\n  }\n\n  return metrics\n}\n\n/** 读某个会话的 trace 并聚合；文件不存在返回 undefined（trace 可能被关闭）。 */\nexport function readTraceRun(sessionId: string): TraceRun | undefined {\n  const path = traceFilePath(sessionId)\n  if (!existsSync(path)) {\n    return undefined\n  }\n  return parseTraceRun(readFileSync(path, 'utf8'))\n}\n\n/** 多次运行的指标均值（scorecard 汇总用）。 */\nexport function averageMetrics(samples: readonly RunMetrics[]): RunMetrics | undefined {\n  if (samples.length === 0) {\n    return undefined\n  }\n  const sum = samples.reduce<RunMetrics>(\n    (acc, m) => ({\n      turns: acc.turns + m.turns,\n      toolCalls: acc.toolCalls + m.toolCalls,\n      toolErrors: acc.toolErrors + m.toolErrors,\n      inputTokens: acc.inputTokens + m.inputTokens,\n      outputTokens: acc.outputTokens + m.outputTokens,\n      cacheReadTokens: acc.cacheReadTokens + m.cacheReadTokens,\n      toolDurationMs: acc.toolDurationMs + m.toolDurationMs,\n      modelId: m.modelId ?? acc.modelId,\n    }),\n    {\n      turns: 0,\n      toolCalls: 0,\n      toolErrors: 0,\n      inputTokens: 0,\n      outputTokens: 0,\n      cacheReadTokens: 0,\n      toolDurationMs: 0,\n    },\n  )\n  const n = samples.length\n  const averaged: RunMetrics = {\n    turns: sum.turns / n,\n    toolCalls: sum.toolCalls / n,\n    toolErrors: sum.toolErrors / n,\n    inputTokens: sum.inputTokens / n,\n    outputTokens: sum.outputTokens / n,\n    cacheReadTokens: sum.cacheReadTokens / n,\n    toolDurationMs: sum.toolDurationMs / n,\n  }\n  if (sum.modelId !== undefined) {\n    averaged.modelId = sum.modelId\n  }\n  return averaged\n}\n","/**\n * verify.ts —— 判分（RFC-316 D3 判分优先级的执行处）。\n *\n * 优先级：可执行判据（script exit 0） > golden 断言 > liveness（跑完没崩）。\n * **liveness 永不产生质量分**——这是继承已删 @x-otto/eval scorecard 的诚实边界：\n * 没有判据就老实说没有，不许用「跑完了」冒充「做对了」。\n */\n\nimport { spawnSync } from 'node:child_process'\nimport { join } from 'node:path'\n\nimport type { CaseSpec, GoldenSpec } from './types'\n\nexport interface VerifyInput {\n  spec: CaseSpec\n  /** agent 运行后的工作区（判分脚本的 cwd）。 */\n  workspaceDir: string\n  /** headless 输出的最终回答。 */\n  response: string\n  /** 本次运行的工具调用序列（按发生顺序），无 trace 时为空。 */\n  toolNames: readonly string[]\n}\n\nexport interface VerifyOutput {\n  passed: boolean\n  detail: string\n}\n\n/** 子序列匹配：期望序列的每个工具名按顺序出现即可，允许中间夹杂其他调用。 */\nexport function matchesToolSubsequence(\n  actual: readonly string[],\n  expected: readonly string[],\n): boolean {\n  let cursor = 0\n  for (const name of actual) {\n    if (cursor < expected.length && name === expected[cursor]) {\n      cursor++\n    }\n  }\n  return cursor === expected.length\n}\n\nfunction verifyGolden(\n  golden: GoldenSpec,\n  response: string,\n  toolNames: readonly string[],\n): VerifyOutput {\n  const misses: string[] = []\n  if (golden.answerContains) {\n    const haystack = response.toLowerCase()\n    for (const needle of golden.answerContains) {\n      if (!haystack.includes(needle.toLowerCase())) {\n        misses.push(`answer missing '${needle}'`)\n      }\n    }\n  }\n  if (golden.toolSequence && !matchesToolSubsequence(toolNames, golden.toolSequence)) {\n    misses.push(\n      `tool sequence [${golden.toolSequence.join(', ')}] not satisfied by [${toolNames.join(', ')}]`,\n    )\n  }\n  return misses.length === 0\n    ? { passed: true, detail: 'golden assertions satisfied' }\n    : { passed: false, detail: misses.join('; ') }\n}\n\nfunction runVerifyScript(input: VerifyInput, script: string): VerifyOutput {\n  const proc = spawnSync('bash', [script], {\n    cwd: input.workspaceDir,\n    encoding: 'utf8',\n    timeout: 120_000,\n    env: { ...process.env, OTTO_EVAL_RESPONSE: input.response },\n  })\n  if (proc.error) {\n    return { passed: false, detail: `verify script error: ${proc.error.message}` }\n  }\n  const detail =\n    `${(proc.stdout ?? '').trim()}${proc.stderr ? ` | ${proc.stderr.trim()}` : ''}`.slice(0, 500)\n  return { passed: proc.status === 0, detail: detail || `exit ${proc.status}` }\n}\n\n/**\n * 执行判分。script 判据在工作区内跑，exit 0 = pass。\n *\n * script + golden 可**同时**存在且必须都满足：判分方式表达的是\"这个 case 靠什么下结论\"，\n * 而工具纪律（golden.toolSequence）与产物正确性（script）是**两个正交的判据**——\n * 只有前者会让\"读了但改错\"蒙混过关，只有后者会让\"盲改碰对\"蒙混过关。\n */\nexport function verifyRun(input: VerifyInput): VerifyOutput {\n  const { spec } = input\n  const parts: VerifyOutput[] = []\n\n  if (spec.judge === 'script') {\n    parts.push(runVerifyScript(input, join(spec.dir, spec.verifyScript ?? 'verify.sh')))\n  }\n  if (spec.golden) {\n    parts.push(verifyGolden(spec.golden, input.response, input.toolNames))\n  }\n\n  if (parts.length === 0) {\n    return { passed: true, detail: 'liveness only (no quality judgement)' }\n  }\n\n  const failed = parts.filter((p) => !p.passed)\n  return failed.length === 0\n    ? { passed: true, detail: parts.map((p) => p.detail).join('; ') }\n    : { passed: false, detail: failed.map((p) => p.detail).join('; ') }\n}\n","/**\n * runner.ts —— 评测编排（RFC-316 M1 核心链路）。\n *\n * 一个 case 跑 n 次 → 三态统计（pass/fail/infra_error）→ 汇总成 CaseResult。\n * 三条不可削弱的纪律都落在这里：\n * - **infra_error 不进 pass 率**（规则 5）：限流/网络故障不许伪装成能力回归。\n * - **预算护栏**（规则 4）：单 case token 超限即停后续运行并标 incomplete，不静默烧钱。\n * - **liveness 不产生质量分**（规则 2）：qualityScored=false 的 case 不进套件 pass 率。\n */\n\nimport { normalizeEnvFraction } from '@x-otto/env'\n\nimport { createWorkspace } from './workspace'\nimport { verifyRun } from './verify'\nimport { readTraceRun, averageMetrics } from './trace-metrics'\n\nimport type { OttoInvoker } from './otto-invoker'\nimport type { TraceRun } from './trace-metrics'\nimport type { CaseResult, CaseSpec, RunMetrics, RunResult, Scorecard } from './types'\n\n/**\n * infra_error 占比超过此阈值时整次运行标 invalid（RFC-316 D6 规则 5）。\n * 硬编码审计 P1：不同 CI 环境对限流/网络故障的容忍度不同，经 `OTTO_EVAL_INFRA_ERROR_THRESHOLD`\n * 覆盖（比例语义，(0,1] 区间，normalizeEnvFraction 越界回退默认）。\n */\nexport const INFRA_ERROR_INVALID_THRESHOLD = normalizeEnvFraction(\n  process.env['OTTO_EVAL_INFRA_ERROR_THRESHOLD'],\n  0.2,\n)\n\nexport interface RunnerDeps {\n  invoker: OttoInvoker\n  /** 读 trace 的端口（默认读真实 trace 文件；测试注入 fake）。 */\n  readTrace?: (sessionId: string) => TraceRun | undefined\n  /** 当前时刻（毫秒），确定性测试用。 */\n  now?: () => number\n  /** 每次运行结束的进度回调（CLI 打点用）。 */\n  onRunComplete?: (caseId: string, index: number, result: RunResult) => void\n}\n\nfunction totalTokens(metrics: RunMetrics | undefined): number {\n  if (!metrics) {\n    return 0\n  }\n  return metrics.inputTokens + metrics.outputTokens\n}\n\n/** 跑单个 case 的 n 次运行。 */\nexport async function runCase(spec: CaseSpec, deps: RunnerDeps): Promise<CaseResult> {\n  const now = deps.now ?? (() => Date.now())\n  const readTrace = deps.readTrace ?? readTraceRun\n  const runs: RunResult[] = []\n  let spentTokens = 0\n  /** 已成功计量的运行次数（用于估算不可计量运行的开销）。 */\n  let measuredRuns = 0\n  /**\n   * 无法计量 token 的运行次数：trace 缺失，或 fail/timeout 路径拿不到 usage。\n   *\n   * 审计修订：早期实现只在「成功且有 trace」时累加 spentTokens，于是失败运行被当作\n   * **零消耗**——但它们同样调用了模型、同样烧钱。一个\"每次都超时\"的 case 因此可以\n   * 无限重试而预算护栏一次都不触发。护栏若只在一切正常时生效，就不是护栏。\n   *\n   * 估算口径：有实测样本时按**已观测均值**计价（最贴近现实）；一个样本都没有时按\n   * `maxTokens/runs` 均摊——那正是用户声明 `runs` 次时的预期单价，此时跑满 runs 次\n   * 属于预期内，不该被拦。\n   */\n  let unmeasuredRuns = 0\n\n  const budgetExceeded = (): boolean => {\n    if (spec.maxTokens === undefined) {\n      return false\n    }\n    const assumedPerRun = measuredRuns > 0 ? spentTokens / measuredRuns : spec.maxTokens / spec.runs\n    return spentTokens + unmeasuredRuns * assumedPerRun >= spec.maxTokens\n  }\n\n  for (let index = 0; index < spec.runs; index++) {\n    if (budgetExceeded()) {\n      runs.push({\n        outcome: 'infra_error',\n        wallMs: 0,\n        detail: `budget exhausted: ${spentTokens} measured tokens${\n          unmeasuredRuns > 0 ? ` + ${unmeasuredRuns} unmeasured run(s)` : ''\n        } vs limit ${spec.maxTokens}, remaining runs skipped`,\n      })\n      continue\n    }\n\n    const workspace = createWorkspace(spec)\n    const startedAt = now()\n    try {\n      const response = await deps.invoker.invoke({\n        prompt: spec.prompt,\n        cwd: workspace.dir,\n        timeoutMs: spec.timeoutMs,\n      })\n      const wallMs = now() - startedAt\n\n      if (response.kind === 'infra') {\n        // infra 故障（限流/网络）通常在模型真正产出前就断了，不计入已花费；\n        // 但也拿不到用量证据，故记为不可计量，让护栏保守对待。\n        unmeasuredRuns++\n        runs.push({ outcome: 'infra_error', wallMs, detail: response.detail })\n        continue\n      }\n      if (response.kind === 'failed') {\n        // 失败/超时同样烧了 token（模型已被调用），只是拿不到 usage —— 必须计入护栏，\n        // 否则\"每次都失败\"的 case 会无限重试烧钱。\n        unmeasuredRuns++\n        runs.push({ outcome: 'fail', wallMs, detail: response.detail })\n        continue\n      }\n\n      const trace = readTrace(response.result.sessionId)\n      const verdict = verifyRun({\n        spec,\n        workspaceDir: workspace.dir,\n        response: response.result.response,\n        toolNames: trace?.toolNames ?? [],\n      })\n      if (trace) {\n        spentTokens += totalTokens(trace.metrics)\n        measuredRuns++\n      } else {\n        // 成功但无 trace（trace 被关闭/文件缺失）：同样计量不到，不能当作零消耗。\n        unmeasuredRuns++\n      }\n\n      const result: RunResult = {\n        outcome: verdict.passed ? 'pass' : 'fail',\n        wallMs,\n        sessionId: response.result.sessionId,\n        detail: verdict.detail,\n      }\n      if (trace) {\n        result.metrics = trace.metrics\n      }\n      runs.push(result)\n    } finally {\n      workspace.dispose()\n    }\n\n    const last = runs.at(-1)\n    if (last) {\n      deps.onRunComplete?.(spec.id, index, last)\n    }\n  }\n\n  return summarizeCase(spec, runs)\n}\n\n/** 把 n 次运行汇总成 CaseResult（三态统计的唯一实现处）。 */\nexport function summarizeCase(spec: CaseSpec, runs: readonly RunResult[]): CaseResult {\n  const infraErrors = runs.filter((r) => r.outcome === 'infra_error').length\n  const effective = runs.length - infraErrors\n  const passed = runs.filter((r) => r.outcome === 'pass').length\n  const qualityScored = spec.judge !== 'liveness'\n\n  const result: CaseResult = {\n    caseId: spec.id,\n    judge: spec.judge,\n    runs,\n    passed,\n    effective,\n    infraErrors,\n    qualityScored,\n  }\n  if (effective > 0) {\n    Object.assign(result, { passRate: passed / effective })\n  }\n  return result\n}\n\n/** 跑整套 case，产出 scorecard。 */\nexport async function runSuite(\n  cases: readonly CaseSpec[],\n  suiteVersion: string,\n  deps: RunnerDeps,\n): Promise<Scorecard> {\n  const now = deps.now ?? (() => Date.now())\n  const startedAt = new Date(now()).toISOString()\n  const results: CaseResult[] = []\n  for (const spec of cases) {\n    results.push(await runCase(spec, deps))\n  }\n  return buildScorecard(results, suiteVersion, startedAt)\n}\n\n/** 聚合 case 结果为 scorecard（纯函数，便于对拍手算值）。 */\nexport function buildScorecard(\n  cases: readonly CaseResult[],\n  suiteVersion: string,\n  startedAt: string,\n): Scorecard {\n  const scored = cases.filter((c) => c.qualityScored && c.passRate !== undefined)\n  const totalRuns = cases.reduce((sum, c) => sum + c.runs.length, 0)\n  const totalInfra = cases.reduce((sum, c) => sum + c.infraErrors, 0)\n  const infraErrorRate = totalRuns === 0 ? 0 : totalInfra / totalRuns\n\n  const allMetrics = cases.flatMap((c) =>\n    c.runs.map((r) => r.metrics).filter((m): m is RunMetrics => m !== undefined),\n  )\n  const aggregate = averageMetrics(allMetrics)\n\n  const scorecard: Scorecard = {\n    suiteVersion,\n    modelId: aggregate?.modelId ?? 'unknown',\n    startedAt,\n    cases,\n    scoredCases: cases.filter((c) => c.qualityScored).length,\n    unscoredCases: cases.filter((c) => !c.qualityScored).length,\n    infraErrorRate,\n    invalid: infraErrorRate > INFRA_ERROR_INVALID_THRESHOLD,\n  }\n  if (scored.length > 0) {\n    const sum = scored.reduce((acc, c) => acc + (c.passRate ?? 0), 0)\n    Object.assign(scorecard, { overallPassRate: sum / scored.length })\n  }\n  if (aggregate) {\n    Object.assign(scorecard, { aggregateMetrics: aggregate })\n  }\n  return scorecard\n}\n","/**\n * baseline.ts —— 基线存储与回归判定（RFC-316 D6 规则 3/4）。\n *\n * 两条承重规则：\n * - **基线按 (modelId, suiteVersion) 分键**：模型升级产生新条目而非覆盖，跨模型对比必须显式。\n *   没有这条，换一次模型就会把曲线接错——之后所有\"变强/变弱\"结论都是噪声。\n * - **阈值显式且诚实**：n=3 的统计功效只够抓大回归，故只用两个粗信号（套件 pass 率跌幅、\n *   case flip），不假装有统计显著性。\n */\n\nimport { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'\nimport { dirname } from 'node:path'\n\nimport { normalizeEnv } from '@x-otto/env'\n\nimport type { BaselineEntry, BaselineFile, RegressionVerdict, Scorecard } from './types'\n\n/**\n * 套件 pass 率相对基线的下降超过此百分点即判回归（RFC-316 D6 规则 4）。\n * 硬编码审计 P1：不同 CI 环境的回归门禁松紧不同，经 `OTTO_EVAL_REGRESSION_PP` 覆盖\n * （百分点整数语义，normalizeEnv 非正数回退默认）。\n */\nexport const PASS_RATE_REGRESSION_PP = normalizeEnv(process.env['OTTO_EVAL_REGRESSION_PP'], 15)\n\nexport function loadBaselines(path: string): BaselineFile {\n  if (!existsSync(path)) {\n    return { entries: [] }\n  }\n  const parsed = JSON.parse(readFileSync(path, 'utf8')) as BaselineFile\n  return { entries: parsed.entries ?? [] }\n}\n\nexport function saveBaselines(path: string, file: BaselineFile): void {\n  mkdirSync(dirname(path), { recursive: true })\n  writeFileSync(path, `${JSON.stringify(file, null, 2)}\\n`, 'utf8')\n}\n\n/** 按 (modelId, suiteVersion) 查基线。 */\nexport function findBaseline(\n  file: BaselineFile,\n  modelId: string,\n  suiteVersion: string,\n): BaselineEntry | undefined {\n  return file.entries.find((e) => e.modelId === modelId && e.suiteVersion === suiteVersion)\n}\n\n/** 把一次运行固化成基线条目（同键覆盖，异键新增）。 */\nexport function upsertBaseline(\n  file: BaselineFile,\n  scorecard: Scorecard,\n  reason: string,\n): BaselineFile {\n  const casePassRates: Record<string, number | undefined> = {}\n  for (const c of scorecard.cases) {\n    casePassRates[c.caseId] = c.passRate\n  }\n\n  const entry: BaselineEntry = {\n    modelId: scorecard.modelId,\n    suiteVersion: scorecard.suiteVersion,\n    recordedAt: scorecard.startedAt,\n    reason,\n    casePassRates,\n  }\n  if (scorecard.overallPassRate !== undefined) {\n    Object.assign(entry, { overallPassRate: scorecard.overallPassRate })\n  }\n\n  const kept = file.entries.filter(\n    (e) => !(e.modelId === entry.modelId && e.suiteVersion === entry.suiteVersion),\n  )\n  return { entries: [...kept, entry] }\n}\n\n/** 回归判定：套件 pass 率跌幅 + case flip 双信号。 */\nexport function judgeRegression(\n  scorecard: Scorecard,\n  baseline: BaselineEntry | undefined,\n): RegressionVerdict {\n  if (scorecard.invalid) {\n    return {\n      status: 'invalid_run',\n      findings: [\n        `infra_error rate ${(scorecard.infraErrorRate * 100).toFixed(1)}% exceeds threshold — run not comparable`,\n      ],\n      flippedCases: [],\n    }\n  }\n  if (!baseline) {\n    return {\n      status: 'no_baseline',\n      findings: [\n        `no baseline for (model=${scorecard.modelId}, suite=${scorecard.suiteVersion}) — record one with \\`baseline update\\``,\n      ],\n      flippedCases: [],\n    }\n  }\n\n  const findings: string[] = []\n  const flipped: string[] = []\n\n  for (const c of scorecard.cases) {\n    if (!c.qualityScored) {\n      continue\n    }\n    const before = baseline.casePassRates[c.caseId]\n    if (before === 1 && c.passRate === 0) {\n      flipped.push(c.caseId)\n      findings.push(`case flip: '${c.caseId}' was 100% pass, now 0%`)\n      continue\n    }\n    // 一个 case 的运行全是 infra_error 时它没有 pass 率，会**静默退出**套件均值——\n    // 于是\"某题目彻底跑不起来\"看上去和\"套件表现不变\"一模一样。这里显式点名，\n    // 不判回归（确实无从判断），但绝不让它无声消失。\n    if (c.passRate === undefined && before !== undefined) {\n      findings.push(\n        `'${c.caseId}' produced no comparable runs (all infra errors) — excluded from the suite average`,\n      )\n    }\n  }\n\n  let deltaPp: number | undefined\n  if (scorecard.overallPassRate !== undefined && baseline.overallPassRate !== undefined) {\n    deltaPp = (scorecard.overallPassRate - baseline.overallPassRate) * 100\n    if (deltaPp < -PASS_RATE_REGRESSION_PP) {\n      findings.push(\n        `suite pass rate dropped ${Math.abs(deltaPp).toFixed(1)}pp ` +\n          `(${(baseline.overallPassRate * 100).toFixed(1)}% → ${(scorecard.overallPassRate * 100).toFixed(1)}%)`,\n      )\n    }\n  }\n\n  const regressed =\n    flipped.length > 0 || (deltaPp !== undefined && deltaPp < -PASS_RATE_REGRESSION_PP)\n\n  const verdict: RegressionVerdict = {\n    status: regressed ? 'regressed' : 'ok',\n    findings: findings.length > 0 ? findings : ['no regression detected'],\n    flippedCases: flipped,\n  }\n  if (deltaPp !== undefined) {\n    Object.assign(verdict, { passRateDeltaPp: deltaPp })\n  }\n  return verdict\n}\n","/**\n * report.ts —— scorecard + 回归判定的人类可读渲染（RFC-316 D6 消费回路的出口）。\n *\n * 报告是评测体系唯一被人真正阅读的产物——@x-otto/eval 上一次死于「没人读」，\n * 所以这里刻意把三件事放在最显眼处：回归结论、逐 case pass 率、诚实边界（未评质量的 case 数）。\n */\n\nimport type { RegressionVerdict, Scorecard } from './types'\n\nfunction pct(value: number | undefined): string {\n  return value === undefined ? '—' : `${(value * 100).toFixed(0)}%`\n}\n\nexport function renderReport(scorecard: Scorecard, verdict: RegressionVerdict): string {\n  const lines: string[] = []\n\n  const headline =\n    verdict.status === 'regressed'\n      ? 'REGRESSED'\n      : verdict.status === 'ok'\n        ? 'OK'\n        : verdict.status === 'invalid_run'\n          ? 'INVALID (infra errors)'\n          : 'NO BASELINE'\n\n  lines.push(`otto eval — ${headline}`)\n  lines.push(\n    `model=${scorecard.modelId}  suite=${scorecard.suiteVersion}  at=${scorecard.startedAt}`,\n  )\n  lines.push('')\n\n  lines.push('case                              judge     pass   runs  infra')\n  for (const c of scorecard.cases) {\n    const id = c.caseId.padEnd(33).slice(0, 33)\n    const judge = c.judge.padEnd(9)\n    const rate = (c.qualityScored ? pct(c.passRate) : 'n/a').padStart(5)\n    const runs = `${c.passed}/${c.effective}`.padStart(6)\n    const infra = String(c.infraErrors).padStart(6)\n    lines.push(`${id} ${judge} ${rate} ${runs} ${infra}`)\n  }\n  lines.push('')\n\n  lines.push(\n    `suite pass rate: ${pct(scorecard.overallPassRate)}  (scored ${scorecard.scoredCases}, liveness-only ${scorecard.unscoredCases})`,\n  )\n  if (verdict.passRateDeltaPp !== undefined) {\n    const sign = verdict.passRateDeltaPp >= 0 ? '+' : ''\n    lines.push(`vs baseline:     ${sign}${verdict.passRateDeltaPp.toFixed(1)}pp`)\n  }\n  lines.push(`infra errors:    ${(scorecard.infraErrorRate * 100).toFixed(1)}% of runs`)\n\n  const m = scorecard.aggregateMetrics\n  if (m) {\n    lines.push(\n      `avg per run:     ${m.turns.toFixed(1)} turns · ${m.toolCalls.toFixed(1)} tools ` +\n        `(${m.toolErrors.toFixed(1)} err) · ${Math.round(m.inputTokens + m.outputTokens)} tokens`,\n    )\n  }\n\n  lines.push('')\n  for (const finding of verdict.findings) {\n    lines.push(`- ${finding}`)\n  }\n\n  if (scorecard.unscoredCases > 0) {\n    lines.push('')\n    lines.push(\n      `note: ${scorecard.unscoredCases} case(s) have no machine judgement (liveness only) and do not affect the suite pass rate.`,\n    )\n  }\n\n  return lines.join('\\n')\n}\n","/**\n * extract.ts —— trace 提炼器（RFC-316 D4 来源②：真实会话 → case 骨架）。\n *\n * case 库能否持续生长，决定这套评测第二次会不会死。真实事故/真实任务是最好的 case 来源，\n * 但从零手写 case 的摩擦太大——本模块把一个真实会话一键转成**骨架**：prompt 原文、\n * 工具调用序列（golden.toolSequence 的现成候选）、触碰过的文件清单（fixture 提示）。\n *\n * 两条刻意的\"不做\"：\n * - **不自动生成判据**：判据是 case 的灵魂，必须人写（RFC-316 D3 诚实边界——自动生成的\n *   判据只会是\"输出像上次一样\"，那是快照不是判据）。骨架里留 TODO 与建议。\n * - **不自动脱敏**：不可靠的脱敏比明示风险更危险（对齐 RFC-200 录制红线）。骨架自带\n *   脱敏检查清单，入库前人工过一遍是硬规则（RFC-316 规则 7）。\n */\n\nimport { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'\nimport { join } from 'node:path'\n\nimport { traceFilePath } from './trace-metrics'\n\ninterface RawTraceEvent {\n  node?: string\n  payload?: Record<string, unknown>\n}\n\n/** 从 trace 还原的会话骨架素材。 */\nexport interface SessionSkeleton {\n  sessionId: string\n  /** 首条用户 prompt 原文（prompt.before 的 text）。 */\n  prompt?: string\n  /** 工具调用序列（名称，按发生顺序）。 */\n  toolSequence: readonly string[]\n  /** 工具参数里出现过的文件路径（write/edit/read 的 path 类参数）——fixture 需要什么的线索。 */\n  touchedPaths: readonly string[]\n  /** LLM 往返数（turn.end 计数）。 */\n  turns: number\n}\n\nconst PATH_ARGUMENT_KEYS = ['path', 'filePath', 'file_path'] as const\n\n/** 解析 trace 内容为骨架素材（纯函数，可测）。 */\nexport function parseSessionSkeleton(sessionId: string, content: string): SessionSkeleton {\n  let prompt: string | undefined\n  const toolSequence: string[] = []\n  const touchedPaths = new Set<string>()\n  let turns = 0\n\n  for (const line of content.split('\\n')) {\n    if (line.trim() === '') {\n      continue\n    }\n    let event: RawTraceEvent\n    try {\n      event = JSON.parse(line) as RawTraceEvent\n    } catch {\n      continue\n    }\n\n    if (event.node === 'prompt.before' && prompt === undefined) {\n      const text = event.payload?.['text']\n      if (typeof text === 'string' && text.trim() !== '') {\n        prompt = text\n      }\n      continue\n    }\n\n    if (event.node === 'tool.call.start') {\n      const name = event.payload?.['name']\n      if (typeof name === 'string') {\n        toolSequence.push(name)\n      }\n      const args = event.payload?.['arguments']\n      if (typeof args === 'object' && args !== null) {\n        for (const key of PATH_ARGUMENT_KEYS) {\n          const value = (args as Record<string, unknown>)[key]\n          if (typeof value === 'string' && value !== '') {\n            touchedPaths.add(value)\n          }\n        }\n      }\n      continue\n    }\n\n    if (event.node === 'turn.end') {\n      turns++\n    }\n  }\n\n  return {\n    sessionId,\n    toolSequence,\n    touchedPaths: [...touchedPaths].sort(),\n    turns,\n    ...(prompt !== undefined ? { prompt } : {}),\n  }\n}\n\nconst SANITIZE_CHECKLIST = `## 脱敏检查清单（入库前必须逐项人工核对 — RFC-316 规则 7）\n\n- [ ] prompt 里没有 API key / token / 密码 / 内部主机名\n- [ ] prompt 里没有真实用户数据（邮箱/手机号/姓名）\n- [ ] fixture 文件里没有上述任何内容\n- [ ] 路径不含本机用户名等机器指纹（用相对路径）\n- [ ] verify 判据已由人工编写（不要用\"输出像上次一样\"的快照当判据）\n`\n\nfunction renderCaseYaml(skeleton: SessionSkeleton): string {\n  const promptBlock =\n    skeleton.prompt !== undefined\n      ? `prompt: |\\n${skeleton.prompt\n          .split('\\n')\n          .map((line) => `  ${line}`)\n          .join('\\n')}`\n      : 'prompt: |\\n  TODO: trace 里没有 prompt.before 文本，请手工填写'\n\n  const toolHint =\n    skeleton.toolSequence.length > 0\n      ? `# 实际会话的工具序列（golden.toolSequence 候选，按需裁剪）：\\n# ${skeleton.toolSequence.join(' → ')}\\n`\n      : ''\n\n  return `# 由 otto-eval extract 生成的骨架（会话 ${skeleton.sessionId}）。\n# TODO：本 case 尚不可用——judge 是 liveness 占位，必须人工补判据后才能进套件质量分。\n${promptBlock}\njudge: liveness  # TODO: 改成 script（写 verify.sh）或 golden（写断言）\n${toolHint}runs: 3\ntimeoutMs: 240000\nmaxTokens: 200000\n`\n}\n\nfunction renderReport(skeleton: SessionSkeleton): string {\n  const paths =\n    skeleton.touchedPaths.length > 0\n      ? skeleton.touchedPaths.map((p) => `- \\`${p}\\``).join('\\n')\n      : '- （无 —— 会话没有文件类工具调用）'\n\n  return `# 提炼报告 — 会话 ${skeleton.sessionId}\n\n- LLM 往返：${skeleton.turns}\n- 工具调用：${skeleton.toolSequence.length} 次\n\n## 会话触碰过的文件（fixture 需要复现哪些初始状态的线索）\n\n${paths}\n\n## 下一步（按序完成后删除本文件）\n\n1. 在 fixture/ 里构造这些文件的**任务开始前**状态（trace 只有调用记录，初始内容需要你还原）\n2. 写判据：verify.sh（首选，可执行判据）或 case.yaml 里的 golden 断言\n3. 把 judge 从 liveness 改成 script/golden\n4. 过一遍下面的脱敏清单\n\n${SANITIZE_CHECKLIST}`\n}\n\nexport interface ExtractResult {\n  caseDir: string\n  skeleton: SessionSkeleton\n}\n\n/** 从会话 trace 生成 case 骨架目录。目标已存在时拒绝（不覆盖人工编辑过的 case）。 */\nexport function extractCase(\n  sessionId: string,\n  casesRoot: string,\n  readTraceContent: (sessionId: string) => string | undefined = defaultReadTrace,\n): ExtractResult {\n  const content = readTraceContent(sessionId)\n  if (content === undefined) {\n    throw new Error(\n      `no trace found for session ${sessionId} (looked at ${traceFilePath(sessionId)})`,\n    )\n  }\n\n  const skeleton = parseSessionSkeleton(sessionId, content)\n  const caseDir = join(casesRoot, `extracted-${sessionId.slice(0, 8)}`)\n  if (existsSync(caseDir)) {\n    throw new Error(\n      `${caseDir} already exists — refusing to overwrite (delete it first if intended)`,\n    )\n  }\n\n  mkdirSync(join(caseDir, 'fixture'), { recursive: true })\n  writeFileSync(join(caseDir, 'case.yaml'), renderCaseYaml(skeleton), 'utf8')\n  writeFileSync(join(caseDir, 'EXTRACTION.md'), renderReport(skeleton), 'utf8')\n\n  return { caseDir, skeleton }\n}\n\nfunction defaultReadTrace(sessionId: string): string | undefined {\n  const path = traceFilePath(sessionId)\n  return existsSync(path) ? readFileSync(path, 'utf8') : undefined\n}\n"],"mappings":"8aAcA,MAIM,EAAoC,CAAC,SAAU,SAAU,WAAW,CAE1E,SAAS,EAAc,EAA8B,EAAa,EAAwB,CACxF,IAAM,EAAQ,EAAI,GAClB,GAAI,OAAO,GAAU,UAAY,EAAM,MAAM,GAAK,GAChD,MAAU,MAAM,SAAS,EAAO,YAAY,EAAI,8BAA8B,CAEhF,OAAO,EAGT,SAAS,EACP,EACA,EACA,EACA,EACQ,CACR,IAAM,EAAQ,EAAI,GAClB,GAAI,IAAU,IAAA,GACZ,OAAO,EAET,GAAI,OAAO,GAAU,UAAY,CAAC,OAAO,UAAU,EAAM,EAAI,GAAS,EACpE,MAAU,MAAM,SAAS,EAAO,YAAY,EAAI,8BAA8B,CAEhF,OAAO,EAGT,SAAS,EAAY,EAAc,EAA4B,CAC7D,GAAI,OAAO,GAAQ,WAAY,EAC7B,MAAU,MAAM,SAAS,EAAO,+CAA+C,CAEjF,IAAM,EAAS,EACT,EAAqB,EAAE,CACvB,EAAiB,EAAO,eAC9B,GAAI,IAAmB,IAAA,GAAW,CAChC,GAAI,CAAC,MAAM,QAAQ,EAAe,EAAI,EAAe,KAAM,GAAM,OAAO,GAAM,SAAS,CACrF,MAAU,MAAM,SAAS,EAAO,iDAAiD,CAEnF,OAAO,OAAO,EAAQ,CAAkB,iBAA4B,CAAC,CAEvE,IAAM,EAAe,EAAO,aAC5B,GAAI,IAAiB,IAAA,GAAW,CAC9B,GAAI,CAAC,MAAM,QAAQ,EAAa,EAAI,EAAa,KAAM,GAAM,OAAO,GAAM,SAAS,CACjF,MAAU,MAAM,SAAS,EAAO,+CAA+C,CAEjF,OAAO,OAAO,EAAQ,CAAgB,eAA0B,CAAC,CAEnE,GAAI,CAAC,EAAO,gBAAkB,CAAC,EAAO,aACpC,MAAU,MACR,SAAS,EAAO,wFACjB,CAEH,OAAO,EAIT,SAAgB,EAAS,EAAuB,CAC9C,IAAM,EAAS,EAAI,MAAM,IAAI,CAAC,OAAO,QAAQ,CAAC,GAAG,GAAG,EAAI,EAClD,EAAW,EAAK,EAAK,YAAY,CACvC,GAAI,CAAC,EAAW,EAAS,CACvB,MAAU,MAAM,SAAS,EAAO,0BAA0B,IAAW,CAGvE,IAAM,EAAkBA,EAAU,EAAa,EAAU,OAAO,CAAC,CACjE,GAAI,OAAO,GAAW,WAAY,EAChC,MAAU,MAAM,SAAS,EAAO,qCAAqC,CAEvE,IAAM,EAAM,EAEN,EAAS,EAAc,EAAK,SAAU,EAAO,CAC7C,EAAW,EAAc,EAAK,QAAS,EAAO,CACpD,GAAI,CAAC,EAAY,SAAS,EAAsB,CAC9C,MAAU,MAAM,SAAS,EAAO,0BAA0B,EAAY,KAAK,MAAM,GAAG,CAEtF,IAAM,EAAQ,EAER,EAAiB,CACrB,GAAI,EACJ,SACA,QACA,KAAM,EAAoB,EAAK,OAAQ,EAAQ,EAAa,CAC5D,UAAW,EAAoB,EAAK,YAAa,EAAQ,IAAmB,CAC5E,MACA,WAAY,EAAW,EAAK,EAAK,UAAU,CAAC,CAC7C,CAMD,GAJI,EAAI,YAAiB,IAAA,IACvB,OAAO,OAAO,EAAM,CAAE,UAAW,EAAoB,EAAK,YAAa,EAAQ,EAAE,CAAE,CAAC,CAGlF,IAAU,SAAU,CACtB,IAAM,EACJ,EAAI,eAAoB,IAAA,GACpB,YACA,EAAc,EAAK,eAAgB,EAAO,CAChD,GAAI,CAAC,EAAW,EAAK,EAAK,EAAO,CAAC,CAChC,MAAU,MAAM,SAAS,EAAO,uCAAuC,EAAO,aAAa,CAE7F,OAAO,OAAO,EAAM,CAAE,aAAc,EAAQ,CAAC,CAS/C,OAJI,IAAU,UAAY,EAAI,SAAc,IAAA,KAC1C,OAAO,OAAO,EAAM,CAAE,OAAQ,EAAY,EAAI,OAAW,EAAO,CAAE,CAAC,CAG9D,EAIT,SAAgB,EAAU,EAAwC,CAChE,GAAI,CAAC,EAAW,EAAU,CACxB,MAAU,MAAM,yBAAyB,IAAY,CAOvD,OALa,EAAY,EAAU,CAChC,OAAQ,GAAS,CAAC,EAAK,WAAW,IAAI,CAAC,CACvC,IAAK,GAAS,EAAK,EAAW,EAAK,CAAC,CACpC,OAAQ,GAAS,EAAS,EAAK,CAAC,aAAa,CAAC,CAC9C,MAAM,CACG,IAAI,EAAS,CAS3B,SAAgB,EAAoB,EAAoC,CACtE,IAAM,EAAY,EAAM,IAAK,IAAO,CAClC,GAAI,EAAE,GACN,OAAQ,EAAE,OACV,MAAO,EAAE,MACT,OAAQ,EAAE,QAAU,KACrB,EAAE,CACH,OAAO,EAAW,SAAS,CAAC,OAAO,KAAK,UAAU,EAAU,CAAC,CAAC,OAAO,MAAM,CAAC,MAAM,EAAG,GAAG,CC5G1F,SAAgB,EAAgB,EAAoC,CAClE,IAAM,EAAO,EAAO,aAAa,CAiBjC,MAhBqB,CACnB,aACA,aACA,MACA,aACA,aACA,YACA,YACA,iBACA,sBACA,wBACA,cACA,sBACA,oBACA,SACD,CACmB,KAAM,GAAW,EAAK,SAAS,EAAO,CAAC,CAAG,QAAU,SAW1E,SAAgB,EAAmB,EAA+B,EAAE,CAAe,CACjF,IAAM,EAAM,EAAQ,KAAO,OAC3B,MAAO,CACL,OAAO,EAAS,CACd,OAAO,IAAI,QAAyB,GAAY,CAC9C,IAAM,EAAQ,EAAM,EAAK,CAAC,SAAU,EAAQ,OAAO,CAAE,CACnD,IAAK,EAAQ,IACb,IAAK,CAAE,GAAG,QAAQ,IAAK,GAAG,EAAQ,IAAK,CACvC,MAAO,CAAC,SAAU,OAAQ,OAAO,CAClC,CAAC,CACE,EAAS,GACT,EAAS,GACT,EAAU,GACR,EAAU,GAAmC,CAC7C,IAGJ,EAAU,GACV,aAAa,EAAM,CACnB,EAAQ,EAAS,GAEb,EAAQ,eAAiB,CAC7B,EAAM,KAAK,UAAU,CACrB,EAAO,CAAE,KAAM,SAAU,OAAQ,iBAAiB,EAAQ,UAAU,IAAK,CAAC,EACzE,EAAQ,UAAU,CAErB,EAAM,OAAO,GAAG,OAAS,GAAkB,CACzC,GAAU,EAAM,UAAU,EAC1B,CACF,EAAM,OAAO,GAAG,OAAS,GAAkB,CACzC,GAAU,EAAM,UAAU,EAC1B,CACF,EAAM,GAAG,QAAU,GAAU,CAC3B,EAAO,CAAE,KAAM,QAAS,OAAQ,iBAAiB,EAAM,UAAW,CAAC,EACnE,CACF,EAAM,GAAG,QAAU,GAAS,CAC1B,GAAI,IAAS,EAAG,CAEd,EAAO,CAAE,KADI,EAAgB,EAAO,CACrB,OAAQ,QAAQ,EAAK,IAAI,EAAO,MAAM,CAAC,MAAM,EAAG,IAAI,GAAI,CAAC,CACxE,OAEF,GAAI,CACF,IAAM,EAAS,KAAK,MAAM,EAAO,CACjC,GAAI,OAAO,EAAO,WAAc,SAAU,CACxC,EAAO,CAAE,KAAM,SAAU,OAAQ,gCAAiC,CAAC,CACnE,OAEF,EAAO,CAAE,KAAM,KAAM,OAAQ,EAAQ,CAAC,MAChC,CACN,EAAO,CAAE,KAAM,SAAU,OAAQ,2BAA2B,EAAO,MAAM,EAAG,IAAI,GAAI,CAAC,GAEvF,EACF,EAEL,CCzGH,SAAS,EAAW,EAA8C,EAAqB,CACrF,IAAM,EAAQ,IAAU,GACxB,OAAO,OAAO,GAAU,UAAY,OAAO,SAAS,EAAM,CAAG,EAAQ,EAIvE,SAAgB,EAAc,EAA2B,CACvD,OAAO,EAAK,EAAmB,QAAS,GAAG,EAAU,QAAQ,CAc/D,SAAgB,EAAc,EAA2B,CACvD,IAAM,EAAsB,EAAE,CAE9B,MAAO,CAAE,QADO,EAAkB,EAAS,EAAU,CACnC,YAAW,CAG/B,SAAS,EAAkB,EAAiB,EAAqC,CAC/E,IAAM,EAAsB,CAC1B,MAAO,EACP,UAAW,EACX,WAAY,EACZ,YAAa,EACb,aAAc,EACd,gBAAiB,EACjB,eAAgB,EACjB,CAED,IAAK,IAAM,KAAQ,EAAQ,MAAM;EAAK,CAAE,CACtC,GAAI,EAAK,MAAM,GAAK,GAClB,SAEF,IAAI,EACJ,GAAI,CACF,EAAQ,KAAK,MAAM,EAAK,MAClB,CACN,SAGF,GAAI,EAAM,OAAS,gBAAiB,CAClC,EAAQ,YACR,EAAQ,gBAAkB,EAAW,EAAM,QAAS,aAAa,CAC7D,EAAM,SAAU,gBAAqB,IAAA,IACvC,EAAQ,aAEV,IAAM,EAAO,EAAM,SAAU,KACzB,OAAO,GAAS,UAClB,GAAc,KAAK,EAAK,CAE1B,SAGF,GAAI,EAAM,OAAS,WAAY,CAC7B,EAAQ,QACR,IAAM,EAAQ,EAAM,SAAU,MAC9B,GAAI,OAAO,GAAU,UAAY,EAAgB,CAC/C,IAAM,EAAI,EACV,EAAQ,aAAe,EAAW,EAAG,cAAc,CACnD,EAAQ,cAAgB,EAAW,EAAG,eAAe,CACrD,EAAQ,iBAAmB,EAAW,EAAG,kBAAkB,CAE7D,IAAM,EAAQ,EAAM,SAAU,MAC1B,OAAO,GAAU,UAAY,IAAU,KACzC,EAAQ,QAAU,IAKxB,OAAO,EAIT,SAAgB,EAAa,EAAyC,CACpE,IAAM,EAAO,EAAc,EAAU,CAChC,KAAW,EAAK,CAGrB,OAAO,EAAc,EAAa,EAAM,OAAO,CAAC,CAIlD,SAAgB,EAAe,EAAwD,CACrF,GAAI,EAAQ,SAAW,EACrB,OAEF,IAAM,EAAM,EAAQ,QACjB,EAAK,KAAO,CACX,MAAO,EAAI,MAAQ,EAAE,MACrB,UAAW,EAAI,UAAY,EAAE,UAC7B,WAAY,EAAI,WAAa,EAAE,WAC/B,YAAa,EAAI,YAAc,EAAE,YACjC,aAAc,EAAI,aAAe,EAAE,aACnC,gBAAiB,EAAI,gBAAkB,EAAE,gBACzC,eAAgB,EAAI,eAAiB,EAAE,eACvC,QAAS,EAAE,SAAW,EAAI,QAC3B,EACD,CACE,MAAO,EACP,UAAW,EACX,WAAY,EACZ,YAAa,EACb,aAAc,EACd,gBAAiB,EACjB,eAAgB,EACjB,CACF,CACK,EAAI,EAAQ,OACZ,EAAuB,CAC3B,MAAO,EAAI,MAAQ,EACnB,UAAW,EAAI,UAAY,EAC3B,WAAY,EAAI,WAAa,EAC7B,YAAa,EAAI,YAAc,EAC/B,aAAc,EAAI,aAAe,EACjC,gBAAiB,EAAI,gBAAkB,EACvC,eAAgB,EAAI,eAAiB,EACtC,CAID,OAHI,EAAI,UAAY,IAAA,KAClB,EAAS,QAAU,EAAI,SAElB,EC3HT,SAAgB,EACd,EACA,EACS,CACT,IAAI,EAAS,EACb,IAAK,IAAM,KAAQ,EACb,EAAS,EAAS,QAAU,IAAS,EAAS,IAChD,IAGJ,OAAO,IAAW,EAAS,OAG7B,SAAS,EACP,EACA,EACA,EACc,CACd,IAAM,EAAmB,EAAE,CAC3B,GAAI,EAAO,eAAgB,CACzB,IAAM,EAAW,EAAS,aAAa,CACvC,IAAK,IAAM,KAAU,EAAO,eACrB,EAAS,SAAS,EAAO,aAAa,CAAC,EAC1C,EAAO,KAAK,mBAAmB,EAAO,GAAG,CAS/C,OALI,EAAO,cAAgB,CAAC,EAAuB,EAAW,EAAO,aAAa,EAChF,EAAO,KACL,kBAAkB,EAAO,aAAa,KAAK,KAAK,CAAC,sBAAsB,EAAU,KAAK,KAAK,CAAC,GAC7F,CAEI,EAAO,SAAW,EACrB,CAAE,OAAQ,GAAM,OAAQ,8BAA+B,CACvD,CAAE,OAAQ,GAAO,OAAQ,EAAO,KAAK,KAAK,CAAE,CAGlD,SAAS,EAAgB,EAAoB,EAA8B,CACzE,IAAM,EAAO,EAAU,OAAQ,CAAC,EAAO,CAAE,CACvC,IAAK,EAAM,aACX,SAAU,OACV,QAAS,KACT,IAAK,CAAE,GAAG,QAAQ,IAAK,mBAAoB,EAAM,SAAU,CAC5D,CAAC,CACF,GAAI,EAAK,MACP,MAAO,CAAE,OAAQ,GAAO,OAAQ,wBAAwB,EAAK,MAAM,UAAW,CAEhF,IAAM,EACJ,IAAI,EAAK,QAAU,IAAI,MAAM,GAAG,EAAK,OAAS,MAAM,EAAK,OAAO,MAAM,GAAK,KAAK,MAAM,EAAG,IAAI,CAC/F,MAAO,CAAE,OAAQ,EAAK,SAAW,EAAG,OAAQ,GAAU,QAAQ,EAAK,SAAU,CAU/E,SAAgB,EAAU,EAAkC,CAC1D,GAAM,CAAE,QAAS,EACX,EAAwB,EAAE,CAShC,GAPI,EAAK,QAAU,UACjB,EAAM,KAAK,EAAgB,EAAO,EAAK,EAAK,IAAK,EAAK,cAAgB,YAAY,CAAC,CAAC,CAElF,EAAK,QACP,EAAM,KAAK,EAAa,EAAK,OAAQ,EAAM,SAAU,EAAM,UAAU,CAAC,CAGpE,EAAM,SAAW,EACnB,MAAO,CAAE,OAAQ,GAAM,OAAQ,uCAAwC,CAGzE,IAAM,EAAS,EAAM,OAAQ,GAAM,CAAC,EAAE,OAAO,CAC7C,OAAO,EAAO,SAAW,EACrB,CAAE,OAAQ,GAAM,OAAQ,EAAM,IAAK,GAAM,EAAE,OAAO,CAAC,KAAK,KAAK,CAAE,CAC/D,CAAE,OAAQ,GAAO,OAAQ,EAAO,IAAK,GAAM,EAAE,OAAO,CAAC,KAAK,KAAK,CAAE,CCjFvE,MAAa,EAAgC,EAC3C,QAAQ,IAAI,gCACZ,GACD,CAYD,SAAS,EAAY,EAAyC,CAI5D,OAHK,EAGE,EAAQ,YAAc,EAAQ,aAF5B,EAMX,eAAsB,EAAQ,EAAgB,EAAuC,CACnF,IAAM,EAAM,EAAK,UAAc,KAAK,KAAK,EACnC,EAAY,EAAK,WAAa,EAC9B,EAAoB,EAAE,CACxB,EAAc,EAEd,EAAe,EAYf,EAAiB,EAEf,MAAgC,CACpC,GAAI,EAAK,YAAc,IAAA,GACrB,MAAO,GAET,IAAM,EAAgB,EAAe,EAAI,EAAc,EAAe,EAAK,UAAY,EAAK,KAC5F,OAAO,EAAc,EAAiB,GAAiB,EAAK,WAG9D,IAAK,IAAI,EAAQ,EAAG,EAAQ,EAAK,KAAM,IAAS,CAC9C,GAAI,GAAgB,CAAE,CACpB,EAAK,KAAK,CACR,QAAS,cACT,OAAQ,EACR,OAAQ,qBAAqB,EAAY,kBACvC,EAAiB,EAAI,MAAM,EAAe,oBAAsB,GACjE,YAAY,EAAK,UAAU,0BAC7B,CAAC,CACF,SAGF,IAAM,EAAY,EAAgB,EAAK,CACjC,EAAY,GAAK,CACvB,GAAI,CACF,IAAM,EAAW,MAAM,EAAK,QAAQ,OAAO,CACzC,OAAQ,EAAK,OACb,IAAK,EAAU,IACf,UAAW,EAAK,UACjB,CAAC,CACI,EAAS,GAAK,CAAG,EAEvB,GAAI,EAAS,OAAS,QAAS,CAG7B,IACA,EAAK,KAAK,CAAE,QAAS,cAAe,SAAQ,OAAQ,EAAS,OAAQ,CAAC,CACtE,SAEF,GAAI,EAAS,OAAS,SAAU,CAG9B,IACA,EAAK,KAAK,CAAE,QAAS,OAAQ,SAAQ,OAAQ,EAAS,OAAQ,CAAC,CAC/D,SAGF,IAAM,EAAQ,EAAU,EAAS,OAAO,UAAU,CAC5C,EAAU,EAAU,CACxB,OACA,aAAc,EAAU,IACxB,SAAU,EAAS,OAAO,SAC1B,UAAW,GAAO,WAAa,EAAE,CAClC,CAAC,CACE,GACF,GAAe,EAAY,EAAM,QAAQ,CACzC,KAGA,IAGF,IAAM,EAAoB,CACxB,QAAS,EAAQ,OAAS,OAAS,OACnC,SACA,UAAW,EAAS,OAAO,UAC3B,OAAQ,EAAQ,OACjB,CACG,IACF,EAAO,QAAU,EAAM,SAEzB,EAAK,KAAK,EAAO,QACT,CACR,EAAU,SAAS,CAGrB,IAAM,EAAO,EAAK,GAAG,GAAG,CACpB,GACF,EAAK,gBAAgB,EAAK,GAAI,EAAO,EAAK,CAI9C,OAAO,EAAc,EAAM,EAAK,CAIlC,SAAgB,EAAc,EAAgB,EAAwC,CACpF,IAAM,EAAc,EAAK,OAAQ,GAAM,EAAE,UAAY,cAAc,CAAC,OAC9D,EAAY,EAAK,OAAS,EAC1B,EAAS,EAAK,OAAQ,GAAM,EAAE,UAAY,OAAO,CAAC,OAClD,EAAgB,EAAK,QAAU,WAE/B,EAAqB,CACzB,OAAQ,EAAK,GACb,MAAO,EAAK,MACZ,OACA,SACA,YACA,cACA,gBACD,CAID,OAHI,EAAY,GACd,OAAO,OAAO,EAAQ,CAAE,SAAU,EAAS,EAAW,CAAC,CAElD,EAIT,eAAsB,EACpB,EACA,EACA,EACoB,CACpB,IAAM,EAAM,EAAK,UAAc,KAAK,KAAK,EACnC,EAAY,IAAI,KAAK,GAAK,CAAC,CAAC,aAAa,CACzC,EAAwB,EAAE,CAChC,IAAK,IAAM,KAAQ,EACjB,EAAQ,KAAK,MAAM,EAAQ,EAAM,EAAK,CAAC,CAEzC,OAAO,EAAe,EAAS,EAAc,EAAU,CAIzD,SAAgB,EACd,EACA,EACA,EACW,CACX,IAAM,EAAS,EAAM,OAAQ,GAAM,EAAE,eAAiB,EAAE,WAAa,IAAA,GAAU,CACzE,EAAY,EAAM,QAAQ,EAAK,IAAM,EAAM,EAAE,KAAK,OAAQ,EAAE,CAC5D,EAAa,EAAM,QAAQ,EAAK,IAAM,EAAM,EAAE,YAAa,EAAE,CAC7D,EAAiB,IAAc,EAAI,EAAI,EAAa,EAKpD,EAAY,EAHC,EAAM,QAAS,GAChC,EAAE,KAAK,IAAK,GAAM,EAAE,QAAQ,CAAC,OAAQ,GAAuB,IAAM,IAAA,GAAU,CAC7E,CAC2C,CAEtC,EAAuB,CAC3B,eACA,QAAS,GAAW,SAAW,UAC/B,YACA,QACA,YAAa,EAAM,OAAQ,GAAM,EAAE,cAAc,CAAC,OAClD,cAAe,EAAM,OAAQ,GAAM,CAAC,EAAE,cAAc,CAAC,OACrD,iBACA,QAAS,EAAiB,EAC3B,CACD,GAAI,EAAO,OAAS,EAAG,CACrB,IAAM,EAAM,EAAO,QAAQ,EAAK,IAAM,GAAO,EAAE,UAAY,GAAI,EAAE,CACjE,OAAO,OAAO,EAAW,CAAE,gBAAiB,EAAM,EAAO,OAAQ,CAAC,CAKpE,OAHI,GACF,OAAO,OAAO,EAAW,CAAE,iBAAkB,EAAW,CAAC,CAEpD,ECvMT,MAAa,EAA0B,EAAa,QAAQ,IAAI,wBAA4B,GAAG,CAE/F,SAAgB,EAAc,EAA4B,CAKxD,OAJK,EAAW,EAAK,CAId,CAAE,QADM,KAAK,MAAM,EAAa,EAAM,OAAO,CAAC,CAC5B,SAAW,EAAE,CAAE,CAH/B,CAAE,QAAS,EAAE,CAAE,CAM1B,SAAgB,EAAc,EAAc,EAA0B,CACpE,EAAU,EAAQ,EAAK,CAAE,CAAE,UAAW,GAAM,CAAC,CAC7C,EAAc,EAAM,GAAG,KAAK,UAAU,EAAM,KAAM,EAAE,CAAC,IAAK,OAAO,CAInE,SAAgB,EACd,EACA,EACA,EAC2B,CAC3B,OAAO,EAAK,QAAQ,KAAM,GAAM,EAAE,UAAY,GAAW,EAAE,eAAiB,EAAa,CAI3F,SAAgB,EACd,EACA,EACA,EACc,CACd,IAAM,EAAoD,EAAE,CAC5D,IAAK,IAAM,KAAK,EAAU,MACxB,EAAc,EAAE,QAAU,EAAE,SAG9B,IAAM,EAAuB,CAC3B,QAAS,EAAU,QACnB,aAAc,EAAU,aACxB,WAAY,EAAU,UACtB,SACA,gBACD,CAQD,OAPI,EAAU,kBAAoB,IAAA,IAChC,OAAO,OAAO,EAAO,CAAE,gBAAiB,EAAU,gBAAiB,CAAC,CAM/D,CAAE,QAAS,CAAC,GAHN,EAAK,QAAQ,OACvB,GAAM,EAAE,EAAE,UAAY,EAAM,SAAW,EAAE,eAAiB,EAAM,cAClE,CAC2B,EAAM,CAAE,CAItC,SAAgB,EACd,EACA,EACmB,CACnB,GAAI,EAAU,QACZ,MAAO,CACL,OAAQ,cACR,SAAU,CACR,qBAAqB,EAAU,eAAiB,KAAK,QAAQ,EAAE,CAAC,0CACjE,CACD,aAAc,EAAE,CACjB,CAEH,GAAI,CAAC,EACH,MAAO,CACL,OAAQ,cACR,SAAU,CACR,0BAA0B,EAAU,QAAQ,UAAU,EAAU,aAAa,yCAC9E,CACD,aAAc,EAAE,CACjB,CAGH,IAAM,EAAqB,EAAE,CACvB,EAAoB,EAAE,CAE5B,IAAK,IAAM,KAAK,EAAU,MAAO,CAC/B,GAAI,CAAC,EAAE,cACL,SAEF,IAAM,EAAS,EAAS,cAAc,EAAE,QACxC,GAAI,IAAW,GAAK,EAAE,WAAa,EAAG,CACpC,EAAQ,KAAK,EAAE,OAAO,CACtB,EAAS,KAAK,eAAe,EAAE,OAAO,yBAAyB,CAC/D,SAKE,EAAE,WAAa,IAAA,IAAa,IAAW,IAAA,IACzC,EAAS,KACP,IAAI,EAAE,OAAO,oFACd,CAIL,IAAI,EACA,EAAU,kBAAoB,IAAA,IAAa,EAAS,kBAAoB,IAAA,KAC1E,GAAW,EAAU,gBAAkB,EAAS,iBAAmB,IAC/D,EAAU,CAAC,GACb,EAAS,KACP,2BAA2B,KAAK,IAAI,EAAQ,CAAC,QAAQ,EAAE,CAAC,OACjD,EAAS,gBAAkB,KAAK,QAAQ,EAAE,CAAC,OAAO,EAAU,gBAAkB,KAAK,QAAQ,EAAE,CAAC,IACtG,EAOL,IAAM,EAA6B,CACjC,OAHA,EAAQ,OAAS,GAAM,IAAY,IAAA,IAAa,EAAU,CAAC,EAGvC,YAAc,KAClC,SAAU,EAAS,OAAS,EAAI,EAAW,CAAC,yBAAyB,CACrE,aAAc,EACf,CAID,OAHI,IAAY,IAAA,IACd,OAAO,OAAO,EAAS,CAAE,gBAAiB,EAAS,CAAC,CAE/C,ECtIT,SAAS,EAAI,EAAmC,CAC9C,OAAO,IAAU,IAAA,GAAY,IAAM,IAAI,EAAQ,KAAK,QAAQ,EAAE,CAAC,GAGjE,SAAgBC,EAAa,EAAsB,EAAoC,CACrF,IAAM,EAAkB,EAAE,CAEpB,EACJ,EAAQ,SAAW,YACf,YACA,EAAQ,SAAW,KACjB,KACA,EAAQ,SAAW,cACjB,yBACA,cAEV,EAAM,KAAK,eAAe,IAAW,CACrC,EAAM,KACJ,SAAS,EAAU,QAAQ,UAAU,EAAU,aAAa,OAAO,EAAU,YAC9E,CACD,EAAM,KAAK,GAAG,CAEd,EAAM,KAAK,iEAAiE,CAC5E,IAAK,IAAM,KAAK,EAAU,MAAO,CAC/B,IAAM,EAAK,EAAE,OAAO,OAAO,GAAG,CAAC,MAAM,EAAG,GAAG,CACrC,EAAQ,EAAE,MAAM,OAAO,EAAE,CACzB,GAAQ,EAAE,cAAgB,EAAI,EAAE,SAAS,CAAG,OAAO,SAAS,EAAE,CAC9D,EAAO,GAAG,EAAE,OAAO,GAAG,EAAE,YAAY,SAAS,EAAE,CAC/C,EAAQ,OAAO,EAAE,YAAY,CAAC,SAAS,EAAE,CAC/C,EAAM,KAAK,GAAG,EAAG,GAAG,EAAM,GAAG,EAAK,GAAG,EAAK,GAAG,IAAQ,CAOvD,GALA,EAAM,KAAK,GAAG,CAEd,EAAM,KACJ,oBAAoB,EAAI,EAAU,gBAAgB,CAAC,YAAY,EAAU,YAAY,kBAAkB,EAAU,cAAc,GAChI,CACG,EAAQ,kBAAoB,IAAA,GAAW,CACzC,IAAM,EAAO,EAAQ,iBAAmB,EAAI,IAAM,GAClD,EAAM,KAAK,oBAAoB,IAAO,EAAQ,gBAAgB,QAAQ,EAAE,CAAC,IAAI,CAE/E,EAAM,KAAK,qBAAqB,EAAU,eAAiB,KAAK,QAAQ,EAAE,CAAC,WAAW,CAEtF,IAAM,EAAI,EAAU,iBAChB,GACF,EAAM,KACJ,oBAAoB,EAAE,MAAM,QAAQ,EAAE,CAAC,WAAW,EAAE,UAAU,QAAQ,EAAE,CAAC,UACnE,EAAE,WAAW,QAAQ,EAAE,CAAC,UAAU,KAAK,MAAM,EAAE,YAAc,EAAE,aAAa,CAAC,SACpF,CAGH,EAAM,KAAK,GAAG,CACd,IAAK,IAAM,KAAW,EAAQ,SAC5B,EAAM,KAAK,KAAK,IAAU,CAU5B,OAPI,EAAU,cAAgB,IAC5B,EAAM,KAAK,GAAG,CACd,EAAM,KACJ,SAAS,EAAU,cAAc,2FAClC,EAGI,EAAM,KAAK;EAAK,CClCzB,MAAM,EAAqB,CAAC,OAAQ,WAAY,YAAY,CAG5D,SAAgB,EAAqB,EAAmB,EAAkC,CACxF,IAAI,EACE,EAAyB,EAAE,CAC3B,EAAe,IAAI,IACrB,EAAQ,EAEZ,IAAK,IAAM,KAAQ,EAAQ,MAAM;EAAK,CAAE,CACtC,GAAI,EAAK,MAAM,GAAK,GAClB,SAEF,IAAI,EACJ,GAAI,CACF,EAAQ,KAAK,MAAM,EAAK,MAClB,CACN,SAGF,GAAI,EAAM,OAAS,iBAAmB,IAAW,IAAA,GAAW,CAC1D,IAAM,EAAO,EAAM,SAAU,KACzB,OAAO,GAAS,UAAY,EAAK,MAAM,GAAK,KAC9C,EAAS,GAEX,SAGF,GAAI,EAAM,OAAS,kBAAmB,CACpC,IAAM,EAAO,EAAM,SAAU,KACzB,OAAO,GAAS,UAClB,EAAa,KAAK,EAAK,CAEzB,IAAM,EAAO,EAAM,SAAU,UAC7B,GAAI,OAAO,GAAS,UAAY,EAC9B,IAAK,IAAM,KAAO,EAAoB,CACpC,IAAM,EAAS,EAAiC,GAC5C,OAAO,GAAU,UAAY,IAAU,IACzC,EAAa,IAAI,EAAM,CAI7B,SAGE,EAAM,OAAS,YACjB,IAIJ,MAAO,CACL,YACA,eACA,aAAc,CAAC,GAAG,EAAa,CAAC,MAAM,CACtC,QACA,GAAI,IAAW,IAAA,GAAyB,EAAE,CAAf,CAAE,SAAQ,CACtC,CAYH,SAAS,EAAe,EAAmC,CACzD,IAAM,EACJ,EAAS,SAAW,IAAA,GAKhB;0CAJA,cAAc,EAAS,OACpB,MAAM;EAAK,CACX,IAAK,GAAS,KAAK,IAAO,CAC1B,KAAK;EAAK,GAGb,EACJ,EAAS,aAAa,OAAS,EAC3B,gDAAgD,EAAS,aAAa,KAAK,MAAM,CAAC,IAClF,GAEN,MAAO,kCAAkC,EAAS,UAAU;;EAE5D,EAAY;;EAEZ,EAAS;;;EAMX,SAAS,EAAa,EAAmC,CACvD,IAAM,EACJ,EAAS,aAAa,OAAS,EAC3B,EAAS,aAAa,IAAK,GAAM,OAAO,EAAE,IAAI,CAAC,KAAK;EAAK,CACzD,uBAEN,MAAO,eAAe,EAAS,UAAU;;WAEhC,EAAS,MAAM;SACjB,EAAS,aAAa,OAAO;;;;EAIpC,EAAM;;;;;;;;;;;;;;;;EAkBR,SAAgB,EACd,EACA,EACA,EAA8D,GAC/C,CACf,IAAM,EAAU,EAAiB,EAAU,CAC3C,GAAI,IAAY,IAAA,GACd,MAAU,MACR,8BAA8B,EAAU,cAAc,EAAc,EAAU,CAAC,GAChF,CAGH,IAAM,EAAW,EAAqB,EAAW,EAAQ,CACnD,EAAU,EAAK,EAAW,aAAa,EAAU,MAAM,EAAG,EAAE,GAAG,CACrE,GAAI,EAAW,EAAQ,CACrB,MAAU,MACR,GAAG,EAAQ,uEACZ,CAOH,OAJA,EAAU,EAAK,EAAS,UAAU,CAAE,CAAE,UAAW,GAAM,CAAC,CACxD,EAAc,EAAK,EAAS,YAAY,CAAE,EAAe,EAAS,CAAE,OAAO,CAC3E,EAAc,EAAK,EAAS,gBAAgB,CAAE,EAAa,EAAS,CAAE,OAAO,CAEtE,CAAE,UAAS,WAAU,CAG9B,SAAS,GAAiB,EAAuC,CAC/D,IAAM,EAAO,EAAc,EAAU,CACrC,OAAO,EAAW,EAAK,CAAG,EAAa,EAAM,OAAO,CAAG,IAAA"}