sdkproof 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kalpit Rathore
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,192 @@
+# sdkproof
+
+`npx sdkproof zod` asks a model to write small realistic jobs against a package, then compiles every answer against the real installed package with `tsc`, the TypeScript compiler. Pass = it compiles. No model judges another model.
+
+You need an API key for the model you want to score. The other half of the tool, `npx sdkproof drift <package>`, needs no key at all.
+
+## What a run looks like
+
+```
+$ npx sdkproof @tanstack/react-table --model claude-sonnet-5 --limit 3
+
+Target: @tanstack/react-table@9.1.2
+Drift: v8 -> v9 — 40 functions, hooks and classes gone from the entrypoint with no deprecation first, in a major that is 1 months old
+Generating 3 x 1 ...
+Verifying  ✗✗✗
+
+────────────────────────────────────────────────────────────
+  @tanstack/react-table@9.1.2 — 0 of 3 answers compiled
+────────────────────────────────────────────────────────────
+
+  claude-sonnet-5           0/3   compiled  (0%, 95% CI 0–56.1%)
+
+  WHAT BROKE
+
+  basic-people-table  (core, easy)
+    TS2724: '"@tanstack/react-table"' has no exported member named 'getCoreRowModel'. Did you mean 'createCoreRowModel'?
+    +6 more diagnostics
+
+  sortable-product-headers  (sorting, easy)
+    TS2724: '"@tanstack/react-table"' has no exported member named 'useReactTable'. Did you mean 'ReactTable'?
+    +7 more diagnostics
+
+  global-search-employees  (filtering, medium)
+    TS2724: '"@tanstack/react-table"' has no exported member named 'useReactTable'. Did you mean 'ReactTable'?
+    +7 more diagnostics
+
+  FAILURE PATTERNS
+    deprecated-or-removed    8x
+    wrong-arguments          3x
+    other                    12x
+```
+
+That is a real run on 22 Aug 2026, trimmed at the last line. `TS2724` is the compiler saying "this import doesn't exist, did you mean this other thing".
+
+TanStack Table v9 renamed `useReactTable` to `useTable` & renamed the whole row-model family, so `getCoreRowModel` became `createCoreRowModel`. The model still writes v8 from memory. It looks right and it doesn't compile.
+
+## What it measures, and what it doesn't
+
+It measures **the model**, not the library. A 0 of 3 says the model has not caught up with v9 yet. It says nothing about whether v9 is a good release.
+
+The one real limit: it type-checks, it doesn't run anything. Code that compiles can still be wrong at runtime. That is on purpose — a type error against the real installed package is a fact the compiler produced, and anything softer would be one AI grading another, which is the thing this exists to avoid.
+
+## Use it
+
+```bash
+npx sdkproof <package>            # score it
+npx sdkproof drift <package>      # what its latest major removed, no API key
+```
+
+Keys, in the environment or a `.env` in the current directory:
+
+```bash
+export ANTHROPIC_API_KEY=...      # scores claude-opus-5 by default
+export OPENAI_API_KEY=...         # scores gpt-5 by default
+```
+
+Both set = both get scored on the same tasks, side by side.
+
+```bash
+npx sdkproof zod
+npx sdkproof @apollo/client@3 --model claude-opus-5 --model gpt-5
+npx sdkproof next --task-count 20 --out ./reports
+npx sdkproof drift @tanstack/react-table
+```
+
+## Check whether it's worth scoring first
+
+`drift` reads two published versions straight off npm and diffs the type declarations. No install, no model, no key, a few seconds.
+
+```
+$ npx sdkproof drift @apollo/client
+
+────────────────────────────────────────────────────────────
+  @apollo/client  v3.14.1 -> v4.2.12
+────────────────────────────────────────────────────────────
+
+  v4 landed 12 months ago
+  132 exported symbols -> 133  (entry-only diff)
+
+  WHAT LEFT (24) — functions, hooks and classes gone from the entrypoint with no deprecation first
+  This is what a model trained on v3 will still write.
+
+    ApolloConsumer
+    ApolloProvider
+    createQueryPreloader
+    getApolloContext
+    useApolloClient
+    useLazyQuery
+    useMutation
+    useQuery
+    useSubscription
+    useSuspenseQuery
+    ...
+
+  Deprecated first, then removed (8) — these rarely produce drift:
+    ApolloError, fromError, fromPromise, isApolloError, parser, resetApolloContext, throwServerError, toPromise
+
+  (29 type-only export(s) also left the entrypoint. They are listed in --json; a model writes a hook far more often than it writes a type name.)
+
+  WORTH SCORING — 24 functions, hooks and classes gone from the entrypoint with no deprecation first, in a major that is 12 months old
+```
+
+Apollo Client 4 moved its React hooks to a different import path, which is why `useQuery` reads as gone from the package root. That's the drift a model hits: the import it writes from memory no longer resolves.
+
+Three things narrow that list, all measured across 35 libraries:
+
+- **Values before types.** A model writes a hook far more often than it writes a type name, so the 29 removed type aliases sit behind the 24 callables.
+- **A removal that was deprecated first produces almost no drift.** The deprecation is a signal in the training data; a silent removal isn't. Apollo's eight deprecated-then-removed symbols are separated out for that reason.
+- **The window closes.** How old the major is predicts drift better than how big it was. Apollo v4 scored 0 of 12 once it had been out long enough, & zustand v5 was fully absorbed by about 19 months.
+
+## How a run works
+
+1. Installs the package into a throwaway sandbox under `~/.cache/sdkproof`. Lifecycle scripts are off unless you pass `--allow-scripts`.
+2. Writes 12 small coding tasks from that exact version's own README.
+3. Asks the model to solve them, with no hints about the API surface.
+4. Compiles every answer against the real installed package and reports what broke.
+
+Step 2 has two rules the tool enforces rather than requests. A task skeleton may not import from the package under test, and neither the task nor its skeleton may name an export of it. Hand the model `import { useTable }` and you have answered the hard half of the question before it starts. Tasks that break either rule are thrown away and counted in the output.
+
+Tasks are cached per `package@version`, so a re-run scores the same set. `--no-task-cache` writes a fresh one.
+
+## Options
+
+```
+--model <id>          model under test; repeatable. claude-*, gpt-*, or an
+                      explicit anthropic:<id> / openai:<id>
+--task-model <id>     model that writes the tasks (default: the first --model)
+--tasks <file>        use a task file instead of writing one
+--task-count <n>      how many tasks to write (default 12)
+--limit <n>           run only the first n tasks
+--concurrency <n>     generations in flight (default 5; drop it if the API 429s)
+--max-tokens <n>      per-generation cap (default 16000)
+--out <dir>           write <package>@<version>.md and .json there
+--json                print the raw result JSON instead of the report
+--markdown            print the markdown report instead of the terminal one
+--no-task-cache       re-write the tasks even if this version has a cached set
+--fresh               reinstall the sandbox instead of reusing the cached one
+--allow-scripts       run npm lifecycle scripts during install (off by default)
+--tsc <path>          use a different tsc entrypoint
+--from <v> --to <v>   drift: compare two specific versions
+```
+
+## Things that will trip you up
+
+**A package with no type declarations can't be scored.** There is nothing for the compiler to check against, and the run stops and says so.
+
+**A refusal is not a failure.** If the model won't attempt a task, no code was written, so nothing about the library was measured. Refusals are reported separately and left out of the score.
+
+**A lost task is worse than a slow one.** If generation errors out — an overloaded API, a timeout — the run prints `INCOMPLETE RUN` and exits 3. A partial run that loses the hard tasks scores higher than the real one, so don't publish that number.
+
+**A thin README makes weak tasks.** Plenty of packages ship a badge wall that links out to a docs site. The tasks then lean on what the model already knows about the package. The run says so when it happens.
+
+## In a script
+
+```ts
+import { run, computeDrift, driftVerdict, fetchPackument, anthropicAdapter } from "sdkproof";
+
+// Check the cheap thing first — no model call, no install.
+const packument = await fetchPackument("@apollo/client");
+const verdict = driftVerdict(await computeDrift(packument));
+if (!verdict.worth) {
+  console.log(`skipping: ${verdict.reason}`);
+} else {
+  const out = await run({
+    spec: "@apollo/client",
+    adapters: [anthropicAdapter("claude-opus-5")],
+    taskCount: 12,
+    concurrency: 5,
+    log: console.error,
+  });
+  console.log(out.markdown);
+  if (out.incomplete) process.exit(3);
+}
+```
+
+Exit codes: `0` fine, `1` bad arguments or a broken target, `2` the API key or account is dead, `3` the run lost tasks and is not publishable.
+
+## Requirements
+
+Node 20.12 or newer, `npm` on the PATH, and network access to the npm registry. One dependency: `typescript`.
+
+MIT. Part of [SDKProof](https://sdkproof.dev) — [source](https://github.com/Kalpitrathore/sdkproof/tree/main/packages/cli).

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,201 @@
+#!/usr/bin/env node
+import { readFileSync } from "node:fs";
+import { createRequire } from "node:module";
+import path from "node:path";
+import { FatalApiError } from "./core/model.js";
+import { computeDrift, driftVerdict } from "./drift.js";
+import { adapterFor, defaultAdapters, DEFAULT_ANTHROPIC_MODEL, DEFAULT_OPENAI_MODEL, hasKeyFor, parseModelRef, } from "./models.js";
+import { fetchPackument, resolveVersion } from "./registry.js";
+import { renderDrift } from "./report.js";
+import { parseSpec, run } from "./run.js";
+const require_ = createRequire(import.meta.url);
+function version() {
+    try {
+        return JSON.parse(readFileSync(require_.resolve("../package.json"), "utf8")).version;
+    }
+    catch {
+        return "0.0.0";
+    }
+}
+const HELP = `
+sdkproof — does the code an AI writes for this package actually compile?
+
+USAGE
+  npx sdkproof <package>[@version]      score a package
+  npx sdkproof drift <package>          what its latest major removed (no API key needed)
+
+HOW A RUN WORKS
+  1. installs the package into a throwaway sandbox
+  2. writes small, realistic coding tasks from that version's own README
+  3. asks the model to solve them, with no hints about the API surface
+  4. compiles every answer against the real installed package with tsc
+
+  Pass = it compiles. No model judges another model.
+
+OPTIONS
+  --model <id>          model under test; repeatable. Accepts claude-*, gpt-*,
+                        or an explicit anthropic:<id> / openai:<id>.
+                        Default: every provider you have a key for.
+  --task-model <id>     model that writes the tasks (default: the first --model)
+  --tasks <file>        use a task file instead of writing one
+  --task-count <n>      how many tasks to write (default 12)
+  --limit <n>           run only the first n tasks
+  --concurrency <n>     generations in flight (default 5; drop it if the API 429s)
+  --max-tokens <n>      per-generation cap (default 16000)
+  --out <dir>           write <package>@<version>.md and .json there
+  --json                print the raw result JSON instead of the report
+  --markdown            print the markdown report instead of the terminal one
+  --no-task-cache       re-write the tasks even if this version has a cached set
+  --fresh               reinstall the sandbox instead of reusing the cached one
+  --allow-scripts       run npm lifecycle scripts during install (off by default)
+  --tsc <path>          use a different tsc entrypoint
+  --from <v> --to <v>   drift: compare two specific versions
+  -h, --help            this
+  -v, --version         print the version
+
+KEYS
+  ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN, and/or OPENAI_API_KEY.
+  A .env in the current directory is loaded if present.
+  \`drift\` needs no key at all.
+
+EXAMPLES
+  npx sdkproof zod
+  npx sdkproof @tanstack/react-table@9 --model claude-opus-5 --out ./reports
+  npx sdkproof drift @apollo/client
+`;
+const VALUE_FLAGS = new Set([
+    "--model", "--task-model", "--tasks", "--task-count", "--limit",
+    "--concurrency", "--max-tokens", "--out", "--tsc", "--from", "--to",
+]);
+function parseArgs(argv) {
+    const positional = [];
+    const values = new Map();
+    const bools = new Set();
+    for (let i = 0; i < argv.length; i++) {
+        const a = argv[i];
+        if (!a.startsWith("-")) {
+            positional.push(a);
+            continue;
+        }
+        if (VALUE_FLAGS.has(a)) {
+            const v = argv[++i];
+            if (v === undefined)
+                throw new Error(`${a} needs a value`);
+            values.set(a, [...(values.get(a) ?? []), v]);
+            continue;
+        }
+        bools.add(a);
+    }
+    return { positional, values, bools };
+}
+const num = (f, name, fallback) => {
+    const raw = f.values.get(name)?.[0];
+    if (raw === undefined)
+        return fallback;
+    const n = Number(raw);
+    if (!Number.isFinite(n) || n <= 0)
+        throw new Error(`${name} must be a positive number, got "${raw}"`);
+    return Math.floor(n);
+};
+async function main() {
+    // A .env beside the user's project is where an API key usually already lives.
+    try {
+        process.loadEnvFile?.(path.join(process.cwd(), ".env"));
+    }
+    catch {
+        // no .env — use the ambient environment
+    }
+    const flags = parseArgs(process.argv.slice(2));
+    // Checked before the help fallback: `--version` carries no positional, and
+    // the fallback would otherwise answer it with the whole help text.
+    if (flags.bools.has("-v") || flags.bools.has("--version")) {
+        console.log(version());
+        return 0;
+    }
+    const askedForHelp = flags.bools.has("-h") || flags.bools.has("--help");
+    if (askedForHelp || !flags.positional.length) {
+        console.log(HELP.trim());
+        return askedForHelp ? 0 : 1;
+    }
+    const log = (line) => console.error(line);
+    if (flags.positional[0] === "drift") {
+        const target = flags.positional[1];
+        if (!target)
+            throw new Error("drift needs a package name: npx sdkproof drift <package>");
+        const { name, version: wanted } = parseSpec(target);
+        const packument = await fetchPackument(name);
+        const d = await computeDrift(packument, {
+            from: flags.values.get("--from")?.[0],
+            to: flags.values.get("--to")?.[0] ?? (wanted ? resolveVersion(packument, wanted) : undefined),
+        });
+        if (flags.bools.has("--json")) {
+            console.log(JSON.stringify({ ...d, verdict: driftVerdict(d) }, null, 2));
+        }
+        else {
+            console.log(renderDrift(d, driftVerdict(d)));
+        }
+        return 0;
+    }
+    const modelFlags = flags.values.get("--model") ?? [];
+    const adapters = modelFlags.length
+        ? modelFlags.map((m) => {
+            const ref = parseModelRef(m);
+            if (!hasKeyFor(ref.provider)) {
+                throw new Error(ref.provider === "anthropic"
+                    ? `--model ${m} needs ANTHROPIC_API_KEY (or ANTHROPIC_AUTH_TOKEN) in the environment`
+                    : `--model ${m} needs OPENAI_API_KEY in the environment`);
+            }
+            return adapterFor(ref);
+        })
+        : defaultAdapters();
+    if (!adapters.length) {
+        console.error([
+            "No model API key found.",
+            "",
+            "  export ANTHROPIC_API_KEY=...     scores " + DEFAULT_ANTHROPIC_MODEL,
+            "  export OPENAI_API_KEY=...        scores " + DEFAULT_OPENAI_MODEL,
+            "",
+            "Or run the part that needs no key:",
+            `  npx sdkproof drift ${flags.positional[0]}`,
+        ].join("\n"));
+        return 1;
+    }
+    const taskModelFlag = flags.values.get("--task-model")?.[0];
+    const outcome = await run({
+        spec: flags.positional[0],
+        adapters,
+        taskModel: taskModelFlag ? adapterFor(parseModelRef(taskModelFlag)) : undefined,
+        taskCount: num(flags, "--task-count", 12),
+        tasksFile: flags.values.get("--tasks")?.[0],
+        limit: flags.values.get("--limit") ? num(flags, "--limit", 0) : undefined,
+        concurrency: num(flags, "--concurrency", 5),
+        maxTokens: flags.values.get("--max-tokens") ? num(flags, "--max-tokens", 16000) : undefined,
+        fresh: flags.bools.has("--fresh"),
+        allowScripts: flags.bools.has("--allow-scripts"),
+        noTaskCache: flags.bools.has("--no-task-cache"),
+        tscEntry: flags.values.get("--tsc")?.[0],
+        outDir: flags.values.get("--out")?.[0],
+        log,
+    });
+    if (flags.bools.has("--json"))
+        console.log(outcome.json);
+    else if (flags.bools.has("--markdown"))
+        console.log(outcome.markdown);
+    else
+        console.log(outcome.terminal);
+    // An incomplete run is not a result. Exiting non-zero keeps it out of CI
+    // dashboards that would otherwise publish a score built on a shrunk denominator.
+    return outcome.incomplete ? 3 : 0;
+}
+main()
+    .then((code) => process.exit(code))
+    .catch((e) => {
+    if (e instanceof FatalApiError) {
+        console.error(`\nRUN ABORTED — ${e.message}`);
+        console.error("Nothing further was attempted. Fix the account, then re-run.");
+        process.exit(2);
+    }
+    console.error(`\nsdkproof: ${e.message}`);
+    process.exit(1);
+});
+//# sourceMappingURL=cli.js.map

package/dist/cli.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACvC,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAC5C,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AAChD,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AACxD,OAAO,EACL,UAAU,EACV,eAAe,EACf,uBAAuB,EACvB,oBAAoB,EACpB,SAAS,EACT,aAAa,GACd,MAAM,aAAa,CAAC;AACrB,OAAO,EAAE,cAAc,EAAE,cAAc,EAAE,MAAM,eAAe,CAAC;AAC/D,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAC1C,OAAO,EAAE,SAAS,EAAE,GAAG,EAAE,MAAM,UAAU,CAAC;AAE1C,MAAM,QAAQ,GAAG,aAAa,CAAC,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC;AAEhD,SAAS,OAAO;IACd,IAAI,CAAC;QACH,OAAO,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,QAAQ,CAAC,OAAO,CAAC,iBAAiB,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC;IACvF,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,OAAO,CAAC;IACjB,CAAC;AACH,CAAC;AAED,MAAM,IAAI,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CA6CZ,CAAC;AAQF,MAAM,WAAW,GAAG,IAAI,GAAG,CAAC;IAC1B,SAAS,EAAE,cAAc,EAAE,SAAS,EAAE,cAAc,EAAE,SAAS;IAC/D,eAAe,EAAE,cAAc,EAAE,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM;CACpE,CAAC,CAAC;AAEH,SAAS,SAAS,CAAC,IAAc;IAC/B,MAAM,UAAU,GAAa,EAAE,CAAC;IAChC,MAAM,MAAM,GAAG,IAAI,GAAG,EAAoB,CAAC;IAC3C,MAAM,KAAK,GAAG,IAAI,GAAG,EAAU,CAAC;IAChC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,IAAI,CAAC,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;YACvB,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;YACnB,SAAS;QACX,CAAC;QACD,IAAI,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;YACvB,MAAM,CAAC,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACpB,IAAI,CAAC,KAAK,SAAS;gBAAE,MAAM,IAAI,KAAK,CAAC,GAAG,CAAC,gBAAgB,CAAC,CAAC;YAC3D,MAAM,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;YAC7C,SAAS;QACX,CAAC;QACD,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;IACf,CAAC;IACD,OAAO,EAAE,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC;AACvC,CAAC;AAED,MAAM,GAAG,GAAG,CAAC,CAAQ,EAAE,IAAY,EAAE,QAAgB,EAAU,EAAE;IAC/D,MAAM,GAAG,GAAG,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IACpC,IAAI,GAAG,KAAK,SAAS;QAAE,OAAO,QAAQ,CAAC;IACvC,MAAM,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IACtB,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC;QAAE,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,oCAAoC,GAAG,GAAG,CAAC,CAAC;IACtG,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;AACvB,CAAC,CAAC;AAEF,KAAK,UAAU,IAAI;IACjB,8EAA8E;IAC9E,IAAI,CAAC;QACH,OAAO,CAAC,WAAW,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,EAAE,MAAM,CAAC,CAAC,CAAC;IAC1D,CAAC;IAAC,MAAM,CAAC;QACP,wCAAwC;IAC1C,CAAC;IAED,MAAM,KAAK,GAAG,SAAS,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAC/C,2EAA2E;IAC3E,mEAAmE;IACnE,IAAI,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,WAAW,CAAC,EAAE,CAAC;QAC1D,OAAO,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;QACvB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,MAAM,YAAY,GAAG,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;IACxE,IAAI,YAAY,IAAI,CAAC,KAAK,CAAC,UAAU,CAAC,MAAM,EAAE,CAAC;QAC7C,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC;QACzB,OAAO,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC9B,CAAC;IAED,MAAM,GAAG,GAAG,CAAC,IAAY,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAElD,IAAI,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,KAAK,OAAO,EAAE,CAAC;QACpC,MAAM,MAAM,GAAG,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;QACnC,IAAI,CAAC,MAAM;YAAE,MAAM,IAAI,KAAK,CAAC,0DAA0D,CAAC,CAAC;QACzF,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;QACpD,MAAM,SAAS,GAAG,MAAM,cAAc,CAAC,IAAI,CAAC,CAAC;QAC7C,MAAM,CAAC,GAAG,MAAM,YAAY,CAAC,SAAS,EAAE;YACtC,IAAI,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,CAAC;YACrC,EAAE,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,cAAc,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;SAC9F,CAAC,CAAC;QACH,IAAI,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;YAC9B,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,GAAG,CAAC,EAAE,OAAO,EAAE,YAAY,CAAC,CAAC,CAAC,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;QAC3E,CAAC;aAAM,CAAC;YACN,OAAO,CAAC,GAAG,CAAC,WAAW,CAAC,CAAC,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QAC/C,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,UAAU,GAAG,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC;IACrD,MAAM,QAAQ,GAAG,UAAU,CAAC,MAAM;QAChC,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE;YACnB,MAAM,GAAG,GAAG,aAAa,CAAC,CAAC,CAAC,CAAC;YAC7B,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;gBAC7B,MAAM,IAAI,KAAK,CACb,GAAG,CAAC,QAAQ,KAAK,WAAW;oBAC1B,CAAC,CAAC,WAAW,CAAC,uEAAuE;oBACrF,CAAC,CAAC,WAAW,CAAC,0CAA0C,CAC3D,CAAC;YACJ,CAAC;YACD,OAAO,UAAU,CAAC,GAAG,CAAC,CAAC;QACzB,CAAC,CAAC;QACJ,CAAC,CAAC,eAAe,EAAE,CAAC;IAEtB,IAAI,CAAC,QAAQ,CAAC,MAAM,EAAE,CAAC;QACrB,OAAO,CAAC,KAAK,CACX;YACE,yBAAyB;YACzB,EAAE;YACF,4CAA4C,GAAG,uBAAuB;YACtE,4CAA4C,GAAG,oBAAoB;YACnE,EAAE;YACF,oCAAoC;YACpC,wBAAwB,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE;SAC9C,CAAC,IAAI,CAAC,IAAI,CAAC,CACb,CAAC;QACF,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,aAAa,GAAG,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,cAAc,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IAC5D,MAAM,OAAO,GAAG,MAAM,GAAG,CAAC;QACxB,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC;QACzB,QAAQ;QACR,SAAS,EAAE,aAAa,CAAC,CAAC,CAAC,UAAU,CAAC,aAAa,CAAC,aAAa,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS;QAC/E,SAAS,EAAE,GAAG,CAAC,KAAK,EAAE,cAAc,EAAE,EAAE,CAAC;QACzC,SAAS,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC;QAC3C,KAAK,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,EAAE,SAAS,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS;QACzE,WAAW,EAAE,GAAG,CAAC,KAAK,EAAE,eAAe,EAAE,CAAC,CAAC;QAC3C,SAAS,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,cAAc,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,EAAE,cAAc,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS;QAC3F,KAAK,EAAE,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,SAAS,CAAC;QACjC,YAAY,EAAE,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,iBAAiB,CAAC;QAChD,WAAW,EAAE,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,iBAAiB,CAAC;QAC/C,QAAQ,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC;QACxC,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC;QACtC,GAAG;KACJ,CAAC,CAAC;IAEH,IAAI,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC;QAAE,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;SACpD,IAAI,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,YAAY,CAAC;QAAE,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;;QACjE,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;IAEnC,yEAAyE;IACzE,iFAAiF;IACjF,OAAO,OAAO,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AACpC,CAAC;AAED,IAAI,EAAE;KACH,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;KAClC,KAAK,CAAC,CAAC,CAAC,EAAE,EAAE;IACX,IAAI,CAAC,YAAY,aAAa,EAAE,CAAC;QAC/B,OAAO,CAAC,KAAK,CAAC,mBAAmB,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC;QAC9C,OAAO,CAAC,KAAK,CAAC,8DAA8D,CAAC,CAAC;QAC9E,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,CAAC,KAAK,CAAC,eAAgB,CAAW,CAAC,OAAO,EAAE,CAAC,CAAC;IACrD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}

package/dist/core/classify.d.ts

@@ -0,0 +1,4 @@
+import type { FailureCategory, FailurePattern, Verdict } from "./types.ts";
+export declare function categorize(code: string): FailureCategory;
+/** Bucket every error across all verdicts into ranked failure patterns. */
+export declare function classify(verdicts: Verdict[]): FailurePattern[];

package/dist/core/classify.js

@@ -0,0 +1,72 @@
+// Map tsc diagnostic codes to human failure categories.
+const CODE_CATEGORY = {
+    TS2339: "hallucinated-member", // property does not exist
+    TS2551: "hallucinated-member", // property does not exist — did you mean
+    TS2353: "hallucinated-member", // invented object-literal field
+    TS2561: "hallucinated-member",
+    TS2694: "hallucinated-member",
+    // The three canonical shapes of "you imported the API that used to be here":
+    // the member moved entrypoint (TS2305), the same with a did-you-mean attached
+    // (TS2724), or the default export is gone (TS1192).
+    //
+    // TS2305 and TS1192 were added to API_SHAPE_CODES on 2026-08-13 but never
+    // here, so they fell through to "other" — and "other" is what the scorecard
+    // prints as the top failure pattern. Apollo Client 4, this project's flagship
+    // finding, publishes "other: 4" for four TS2305s that are the entire point of
+    // the measurement. Found 2026-08-18 re-scoring the AI SDK, whose scorecard
+    // led with "other: 13x".
+    //
+    // TS2724 moves with them: it is TS2305 with a suggestion attached, and
+    // splitting one error family across two categories is the same inconsistency
+    // the 08-13 fix was about. `deprecated-or-removed` was declared in
+    // FailureCategory and mapped from nothing; for a harness whose whole subject
+    // is removed and moved APIs, it is the accurate label for all three.
+    TS2305: "deprecated-or-removed",
+    TS2724: "deprecated-or-removed",
+    TS1192: "deprecated-or-removed",
+    TS2554: "wrong-arguments", // wrong number of arguments
+    TS2345: "wrong-arguments", // argument type not assignable
+    // The same family one level up: the call or the generic was handed the wrong
+    // shape. Unmapped until 2026-08-22, when a `zod` run through the published
+    // CLI reported "other: 2x" for two TS2769s that were the whole finding —
+    // the identical failure the 08-18 note describes for TS2305.
+    TS2769: "wrong-arguments", // no overload matches this call
+    TS2314: "wrong-arguments", // generic type requires N type argument(s)
+    TS2558: "wrong-arguments", // expected N type arguments, but got M
+    TS2707: "wrong-arguments", // generic type requires between N and M type arguments
+    TS2347: "wrong-arguments", // untyped function calls may not accept type arguments
+    TS2559: "type-mismatch", // no properties in common
+    TS2322: "type-mismatch", // type not assignable
+    TS2307: "bad-import", // cannot find module
+};
+export function categorize(code) {
+    return CODE_CATEGORY[code] ?? "other";
+}
+/** Bucket every error across all verdicts into ranked failure patterns. */
+export function classify(verdicts) {
+    const byCategory = new Map();
+    for (const v of verdicts) {
+        for (const e of v.errors) {
+            const category = categorize(e.code);
+            const existing = byCategory.get(category);
+            if (existing) {
+                existing.count += 1;
+            }
+            else {
+                byCategory.set(category, {
+                    category,
+                    count: 1,
+                    example: { taskId: v.taskId, model: v.model, message: e.message },
+                });
+            }
+        }
+    }
+    // "other" is the residual bucket and must never lead the scorecard. Ranking
+    // purely by count let it: a single failed type import produces one TS2305
+    // plus one TS7031 per parameter that lost its annotation, so on the AI SDK
+    // the DOWNSTREAM noise (11x "implicitly has an 'any' type") outranked the
+    // four removed types that caused it. The consequence was printed as the
+    // headline and the cause was printed underneath.
+    return [...byCategory.values()].sort((a, b) => Number(a.category === "other") - Number(b.category === "other") || b.count - a.count);
+}
+//# sourceMappingURL=classify.js.map

package/dist/core/classify.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"classify.js","sourceRoot":"","sources":["../../src/core/classify.ts"],"names":[],"mappings":"AAEA,wDAAwD;AACxD,MAAM,aAAa,GAAoC;IACrD,MAAM,EAAE,qBAAqB,EAAE,0BAA0B;IACzD,MAAM,EAAE,qBAAqB,EAAE,yCAAyC;IACxE,MAAM,EAAE,qBAAqB,EAAE,gCAAgC;IAC/D,MAAM,EAAE,qBAAqB;IAC7B,MAAM,EAAE,qBAAqB;IAC7B,6EAA6E;IAC7E,8EAA8E;IAC9E,oDAAoD;IACpD,EAAE;IACF,0EAA0E;IAC1E,4EAA4E;IAC5E,8EAA8E;IAC9E,8EAA8E;IAC9E,2EAA2E;IAC3E,yBAAyB;IACzB,EAAE;IACF,uEAAuE;IACvE,6EAA6E;IAC7E,mEAAmE;IACnE,6EAA6E;IAC7E,qEAAqE;IACrE,MAAM,EAAE,uBAAuB;IAC/B,MAAM,EAAE,uBAAuB;IAC/B,MAAM,EAAE,uBAAuB;IAC/B,MAAM,EAAE,iBAAiB,EAAE,4BAA4B;IACvD,MAAM,EAAE,iBAAiB,EAAE,+BAA+B;IAC1D,6EAA6E;IAC7E,2EAA2E;IAC3E,yEAAyE;IACzE,6DAA6D;IAC7D,MAAM,EAAE,iBAAiB,EAAE,gCAAgC;IAC3D,MAAM,EAAE,iBAAiB,EAAE,2CAA2C;IACtE,MAAM,EAAE,iBAAiB,EAAE,uCAAuC;IAClE,MAAM,EAAE,iBAAiB,EAAE,uDAAuD;IAClF,MAAM,EAAE,iBAAiB,EAAE,uDAAuD;IAClF,MAAM,EAAE,eAAe,EAAE,0BAA0B;IACnD,MAAM,EAAE,eAAe,EAAE,sBAAsB;IAC/C,MAAM,EAAE,YAAY,EAAE,qBAAqB;CAC5C,CAAC;AAEF,MAAM,UAAU,UAAU,CAAC,IAAY;IACrC,OAAO,aAAa,CAAC,IAAI,CAAC,IAAI,OAAO,CAAC;AACxC,CAAC;AAED,2EAA2E;AAC3E,MAAM,UAAU,QAAQ,CAAC,QAAmB;IAC1C,MAAM,UAAU,GAAG,IAAI,GAAG,EAAmC,CAAC;IAC9D,KAAK,MAAM,CAAC,IAAI,QAAQ,EAAE,CAAC;QACzB,KAAK,MAAM,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,CAAC;YACzB,MAAM,QAAQ,GAAG,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;YACpC,MAAM,QAAQ,GAAG,UAAU,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;YAC1C,IAAI,QAAQ,EAAE,CAAC;gBACb,QAAQ,CAAC,KAAK,IAAI,CAAC,CAAC;YACtB,CAAC;iBAAM,CAAC;gBACN,UAAU,CAAC,GAAG,CAAC,QAAQ,EAAE;oBACvB,QAAQ;oBACR,KAAK,EAAE,CAAC;oBACR,OAAO,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,CAAC,KAAK,EAAE,OAAO,EAAE,CAAC,CAAC,OAAO,EAAE;iBAClE,CAAC,CAAC;YACL,CAAC;QACH,CAAC;IACH,CAAC;IACD,4EAA4E;IAC5E,0EAA0E;IAC1E,2EAA2E;IAC3E,0EAA0E;IAC1E,wEAAwE;IACxE,iDAAiD;IACjD,OAAO,CAAC,GAAG,UAAU,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAClC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,QAAQ,KAAK,OAAO,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,QAAQ,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAC/F,CAAC;AACJ,CAAC"}

package/dist/core/model.d.ts

@@ -0,0 +1,30 @@
+export interface GenerateRequest {
+    system: string;
+    user: string;
+    maxTokens?: number;
+}
+/** A model that generates code for a task. The "agent under test." */
+export interface ModelAdapter {
+    /** the model id, used as the scorecard column name */
+    id: string;
+    generate(req: GenerateRequest): Promise<string>;
+}
+/**
+ * Thrown when a model refuses the identical prompt every time it is sampled.
+ * Typed so the CLI can record it as a refusal rather than a generic failure —
+ * a refusal is an unmeasured task, not a failed one.
+ */
+export declare class RefusalError extends Error {
+    readonly attempts: number;
+    constructor(attempts: number);
+}
+/**
+ * An error that makes every remaining request pointless — an exhausted credit
+ * balance, a revoked key. Distinct from a refusal (one task) or a 529 (retry),
+ * because the right response is to stop the run, not to keep paying for
+ * failures. On 2026-08-05 a credit exhaustion burned 30 further attempts
+ * against a dead key before the run ended with nothing.
+ */
+export declare class FatalApiError extends Error {
+    constructor(message: string);
+}

package/dist/core/model.js

@@ -0,0 +1,27 @@
+/**
+ * Thrown when a model refuses the identical prompt every time it is sampled.
+ * Typed so the CLI can record it as a refusal rather than a generic failure —
+ * a refusal is an unmeasured task, not a failed one.
+ */
+export class RefusalError extends Error {
+    attempts;
+    constructor(attempts) {
+        super(`model refused this task ${attempts}x — not library drift, excluded from the score`);
+        this.attempts = attempts;
+        this.name = "RefusalError";
+    }
+}
+/**
+ * An error that makes every remaining request pointless — an exhausted credit
+ * balance, a revoked key. Distinct from a refusal (one task) or a 529 (retry),
+ * because the right response is to stop the run, not to keep paying for
+ * failures. On 2026-08-05 a credit exhaustion burned 30 further attempts
+ * against a dead key before the run ended with nothing.
+ */
+export class FatalApiError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = "FatalApiError";
+    }
+}
+//# sourceMappingURL=model.js.map

package/dist/core/model.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"model.js","sourceRoot":"","sources":["../../src/core/model.ts"],"names":[],"mappings":"AAaA;;;;GAIG;AACH,MAAM,OAAO,YAAa,SAAQ,KAAK;IAChB,QAAQ;IAA7B,YAAqB,QAAgB;QACnC,KAAK,CAAC,2BAA2B,QAAQ,gDAAgD,CAAC,CAAC;wBADxE,QAAQ;QAE3B,IAAI,CAAC,IAAI,GAAG,cAAc,CAAC;IAC7B,CAAC;CACF;AAED;;;;;;GAMG;AACH,MAAM,OAAO,aAAc,SAAQ,KAAK;IACtC,YAAY,OAAe;QACzB,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,eAAe,CAAC;IAC9B,CAAC;CACF"}

package/dist/core/prompt.d.ts

@@ -0,0 +1,4 @@
+import type { PromptTarget, Task } from "./types.ts";
+export declare const GENERATION_SYSTEM: string;
+export declare function buildUserPrompt(task: Task, spec: PromptTarget, context?: string): string;
+export declare function extractCode(text: string): string;

package/dist/core/prompt.js

@@ -0,0 +1,43 @@
+export const GENERATION_SYSTEM = "You are an expert TypeScript developer. Implement the requested function using the specified library. " +
+    "Respond with ONLY the complete TypeScript module inside a single ```ts code block — no explanation before or after. " +
+    "Keep the provided imports and any `declare` lines exactly as given. " +
+    "Fill in the function body so the module type-checks against the real installed package.";
+export function buildUserPrompt(task, spec, context) {
+    const base = [
+        `Library: ${spec.displayName}, imported from "${spec.packageName}".`,
+        spec.docsHint,
+        "",
+        `Task: ${task.prompt}`,
+        "",
+        "Complete this module:",
+        "```ts",
+        task.skeleton,
+        "```",
+    ].join("\n");
+    // No context => byte-identical to every prompt that produced a published
+    // score. That equality is asserted in test/prompt.test.ts, because a silent
+    // change here would move six numbers that are already on the web.
+    if (!context)
+        return base;
+    return [
+        "Project context — the agent files this library ships:",
+        "",
+        context,
+        "",
+        "---",
+        "",
+        base,
+    ].join("\n");
+}
+// Prefer the *longest* fenced ts/js block; fall back to the raw text with any
+// stray fence markers stripped. Taking the longest (not the first) guards
+// against a stray early ``` — in a comment or string — truncating the capture,
+// which otherwise produces a broken fragment and a false compile failure.
+const FENCE = /```(?:ts|typescript|tsx|js|javascript)?[^\S\r\n]*\r?\n([\s\S]*?)```/g;
+export function extractCode(text) {
+    const blocks = [...text.matchAll(FENCE)].map((m) => m[1].trim());
+    if (blocks.length > 0)
+        return blocks.reduce((a, b) => (b.length > a.length ? b : a));
+    return text.replace(/^```[a-z]*[^\S\r\n]*\r?\n?/i, "").replace(/```\s*$/i, "").trim();
+}
+//# sourceMappingURL=prompt.js.map

package/dist/core/prompt.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"prompt.js","sourceRoot":"","sources":["../../src/core/prompt.ts"],"names":[],"mappings":"AAEA,MAAM,CAAC,MAAM,iBAAiB,GAC5B,wGAAwG;IACxG,sHAAsH;IACtH,sEAAsE;IACtE,yFAAyF,CAAC;AAE5F,MAAM,UAAU,eAAe,CAAC,IAAU,EAAE,IAAkB,EAAE,OAAgB;IAC9E,MAAM,IAAI,GAAG;QACX,YAAY,IAAI,CAAC,WAAW,oBAAoB,IAAI,CAAC,WAAW,IAAI;QACpE,IAAI,CAAC,QAAQ;QACb,EAAE;QACF,SAAS,IAAI,CAAC,MAAM,EAAE;QACtB,EAAE;QACF,uBAAuB;QACvB,OAAO;QACP,IAAI,CAAC,QAAQ;QACb,KAAK;KACN,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACb,yEAAyE;IACzE,4EAA4E;IAC5E,kEAAkE;IAClE,IAAI,CAAC,OAAO;QAAE,OAAO,IAAI,CAAC;IAC1B,OAAO;QACL,uDAAuD;QACvD,EAAE;QACF,OAAO;QACP,EAAE;QACF,KAAK;QACL,EAAE;QACF,IAAI;KACL,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,8EAA8E;AAC9E,0EAA0E;AAC1E,+EAA+E;AAC/E,0EAA0E;AAC1E,MAAM,KAAK,GAAG,sEAAsE,CAAC;AAErF,MAAM,UAAU,WAAW,CAAC,IAAY;IACtC,MAAM,MAAM,GAAG,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;IACjE,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACrF,OAAO,IAAI,CAAC,OAAO,CAAC,6BAA6B,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,UAAU,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;AACxF,CAAC"}

package/dist/core/score.d.ts

@@ -0,0 +1,3 @@
+import type { LostTask, ArmScore, Refusal, Result, Verdict } from "./types.ts";
+/** Aggregate verdicts into a per-model + overall score with ranked failures. */
+export declare function score(library: string, libraryVersion: string, generatedAt: string, verdicts: Verdict[], refusals?: Refusal[], contextArms?: ArmScore[], lost?: LostTask[]): Result;