peerbench 0.0.2-alpha.0 → 0.0.2-alpha.2

package/README.md

@@ -1,12 +1,11 @@
 # `peerbench` SDK
 
-This package is the shared “domain core” for _building benchmarks_ in a standardized, portable way. It gives you a consistent set of _persistable entities_ (schemas + types), and a consistent set of _runtime contracts_ (loaders, runners, scorers, providers) so the same benchmark can run in a CLI, a web app, a worker, or anything else.
+This package is the shared “domain core” for _building benchmarks_ in a standardized, portable way. It gives you a consistent set of _persistable entities_ (schemas + types), and a consistent set of _runtime contracts_ (runners, scorers, providers, storages, aggregators) so the same benchmark can run in a CLI, a web app, a worker, or anything else.
 
-> _Runtime_ refers to the codebase (a CLI, a webapp, a background service etc.) that uses the SDK.
+If you’re implementing a new benchmark, the SDK is the part that keeps it portable instead of glued to one runtime. If you’re integrating peerbench SDK into a runtime, the SDK is the part you don’t want to rewrite in every repo.
 
-If you’re implementing a new benchmark, the SDK is the part that keeps it portable instead of glued to one runtime. If you’re integrating Peerbench into a runtime, the SDK is the part you don’t want to rewrite in every repo.
-
-> This package does not support CommonJS
+> - _Runtime_ refers to the codebase that uses peerbench SDK (a CLI, a webapp, a background service etc.)
+> - This package does not support CommonJS
 
 ## What is a benchmark?
 
@@ -18,39 +17,43 @@ If you look at widely-used benchmarks, the pattern is always the same even when
 - In BIG-bench style task suites, you have many different task types and you want a consistent way to run and score them.
 - In HELM-style evaluations, you care about not only “did it answer correctly”, but also how you ran it (prompting setup, constraints, metadata) and how you report results.
 
-Those benchmarks differ in details, but they all boil down to the same building blocks: a dataset of test cases, a way to run a system on each test case, and a way to score the output. The Peerbench SDK is designed so these patterns can be represented with the same portable shape.
+Those benchmarks differ in details, but they all boil down to the same building blocks: a dataset of test cases, a way to run a system on each test case, and a way to score the output. The peerbench SDK is designed so these patterns can be represented with the same portable shape.
 
 ## The mental model
 
-Now that we agree on what a benchmark is, we can talk about how Peerbench represents it.
+Now that we agree on what a benchmark is, we can talk about how peerbench represents it.
 
-Peerbench is deliberately boring here. It doesn’t try to invent a new “benchmark framework”. It gives you a small set of building blocks that you can compose. If you understand these pieces, you can read any benchmark implementation and know where to look.
+peerbench is deliberately boring here. It doesn’t try to invent a new “benchmark framework”. It gives you a small set of building blocks that you can compose. If you understand these pieces, you can read any benchmark implementation and know where to look.
 
 ### Entities (the things you store)
 
-When you run an evaluation, you end up with data that you want to store, query, re-score, and share. Peerbench standardizes that output by modeling it as a small set of entities.
+When you run an evaluation, you end up with data that you want to store, query, re-score, and share. peerbench standardizes that output by modeling it as a small set of entities.
 
-This SDK assumes four core entities:
+This SDK assumes three core entities:
 
-- `BenchmarkSpec`: optional benchmark-level configuration (think: “applies to the whole dataset/run”).
 - `TestCase`: a single input/task.
 - `Response`: the model output for a specific test case (`testCaseId` points to `TestCase.id`).
-- `Score`: an evaluation result for a specific response (`responseId` points to `Response.id`).
+- `Score` (optional): an evaluation result for a specific response (`responseId` points to `Response.id`).
 
 Everything else in the SDK exists to create these entities in a predictable way.
 
-Two fields show up everywhere:
+Three fields show up everywhere:
 
 - `kind` tells you _what type_ of entity something is. It is a stable string you pick (descriptive).
 - `schemaVersion` tells you _which version_ of that entity shape you’re looking at.
+- `namespace` tells you which “owner” defines that kind (e.g peerbench.ai).
+
+This is why peerbench leans on [Zod](https://zod.dev) schemas: it keeps the persisted data contract explicit and runtime-validated.
 
-This is why Peerbench leans on [Zod](https://zod.dev) schemas: it keeps the persisted data contract explicit and runtime-validated.
+### Storage (how entities are persisted)
 
-### Loader (how raw data becomes test cases)
+The SDK does not prescribe how you ingest datasets. Runtimes often load test cases from JSON/JSONL, DB rows, Parquet, or an API. Storages are the abstractions that allow you to standardize the way you load the data from a source to the memory.
 
-In real projects, test cases live in many places: JSON files, JSONL streams, a database, Parquet, an API, etc.
+peerbench SDK provides some pre-defined storage abstractions you can use out of the box:
 
-A loader is the piece that reads that raw data and returns `TestCase[]` (and optionally existing `Response[]` / `Score[]`). The important point is not the file format. The important point is that the loader is where your “raw input → Peerbench entities” mapping lives.
+- file-based storage with custom codecs (`FileStorage`)
+- JSON array files (`JSONFileStorage`)
+- SQLite storage (codec-based)
 
 ### Provider (how you talk to a model)
 
@@ -65,13 +68,13 @@ If you already have your own service in front of the model, you can still model
 
 ### Runner (how you execute one test case)
 
-A runner is the execution part of a benchmark. A runner function takes whatever inputs it needs, calls a provider, and produces a `Response`. It may also produce a `Score` (directly, or via a scorer).
+A runner is the execution part of a benchmark. A runner function takes whatever inputs it needs, calls a provider, and produces a `Response`. It may also produce a `Score` (via a scorer).
 
 Runners are indented to be “per test case” because it keeps the benchmark logic small and easy to compose. Running a whole dataset is orchestration, and orchestration is where runtimes differ (parallelism, retries, persistence, budgets, progress UI).
 
 There is no restriction that a benchmark must have exactly one runner. You can export multiple runner functions (different modes, different prompts, different providers, different scoring strategies). The runtime just needs to pick the runner it wants to use.
 
-One practical convention you will see in the examples is `runConfig`. It’s runner-specific, and it’s usually kept as a simple JSON-serializable object so you can store it alongside your run and reproduce it later. This is a best practice, not a hard restriction: if something doesn’t belong in `runConfig`, you can pass it as a normal parameter next to it.
+One practical convention you will see in the examples is `runConfig`. It’s runner-specific, and it’s recommended to kept as a simple JSON-serializable object so you can store it alongside your run and reproduce it later.
 
 ### Scorer (how you judge a response)
 
@@ -79,7 +82,7 @@ A scorer produces a numeric result. Some scorers are deterministic (same input
 
 A scorer takes what it needs. Sometimes it’s “expected + actual strings”. Sometimes it’s “a list of required fields + a JSON output”. The runner decides what to pass into the scorer, because the runner is the piece that knows how the benchmark is structured.
 
-If your benchmark can be scored in multiple ways, a runner can accept multiple scorer implementations and choose between them based on `scorer.kind`. The examples in `packages/sdk-0.2/src/benchmarks/example/` show what that looks like in code.
+If your benchmark can be scored in multiple ways, a runner can accept multiple scorer implementations and choose between them. The examples in `packages/sdk-0.2/src/benchmarks/examples/` show what that looks like in code.
 
 ## What the SDK does vs what the runtime does
 
@@ -87,14 +90,15 @@ It’s easy to accidentally push “too much responsibility” to the SDK and en
 
 This SDK tries to draw a clean line:
 
-The SDK is responsible for:
+It is responsible for:
 
 - defining and validating entity shapes (Zod schemas are the source of truth)
-- providing base contracts and reusable building blocks (schemas + loaders + runners + scorers)
+- providing base contracts and reusable building blocks (schemas + runners + scorers + storages + aggregators)
 - defining provider/scorer contracts so you can swap backends without rewriting benchmarks
 
 The runtime is responsible for:
 
+- sourcing test cases (JSON/DB/Parquet/API/etc.) and mapping them into `TestCase` entities
 - orchestration across many test cases (parallelism, retries, persistence, resuming, progress UI)
 - deciding how/where entities are stored (DB schema, file layout, caching)
 - secrets and private content (API keys, redacted prompts, access control)
@@ -110,7 +114,6 @@ In practice, the benchmark implementer is responsible for:
 
 - choosing stable `kind` strings (namespaced, descriptive) and bumping `schemaVersion` on breaking changes
 - defining the schemas that are safe to store and share (and keeping secrets out of them)
-- deciding how raw datasets map into `TestCase` entities (loader)
 - deciding how a test case is executed (runner) and how it becomes a `Response`
 - deciding how scoring works (inline in runner, a separate scorer, or multiple scorers)
 
@@ -125,19 +128,22 @@ Benchmarks can implement everything themselves, but they can also reuse the SDK
 A “benchmark” in this SDK is not a magical object. It is a small folder that exports a few well-known pieces. The simplest complete benchmark usually includes:
 
 1. schemas (test case / response / score)
-2. a loader (how test cases are read from disk/DB/etc.)
-3. a runner (how a single test case is executed)
-4. one or more scorers (optional)
+2. a runner (how a single test case is executed)
+3. one or more scorers (if the SDK provided scorers do not work)
+4. (optional) one or more storages (how entities are persisted)
 
 You can see a compact, end-to-end reference in:
 
-- `packages/sdk-0.2/src/benchmarks/example/basic/`
+- `packages/sdk-0.2/src/benchmarks/examples/echo-basic/`
+- `packages/sdk-0.2/src/benchmarks/examples/text-transform/`
+- `packages/sdk-0.2/src/benchmarks/examples/exact-match-scorer/`
+- `packages/sdk-0.2/src/benchmarks/examples/mcq-qa-templated/`
 
 ### 1) Schemas: the source of truth
 
 Schemas are the core of a benchmark. They are the entities that hold the data.
 
-In `packages/sdk-0.2/src/benchmarks/example/basic/test-cases/echo.v1.ts` you can see the pattern:
+In `packages/sdk-0.2/src/benchmarks/examples/echo-basic/schema-sets/echo.v1.ts` you can see the pattern:
 
 - define a test case schema (`kind` + `schemaVersion` + benchmark fields)
 - define a response schema for that test case
@@ -153,7 +159,8 @@ import { BaseTestCaseSchemaV1, defineTestCaseSchema } from "peerbench/schemas";
 
 export const MyTestCaseSchemaV1 = defineTestCaseSchema({
   baseSchema: BaseTestCaseSchemaV1,
-  kind: "mybench.ts.someTask",
+  namespace: "example.peerbench.ai",
+  kind: "llm/my-benchmark",
   schemaVersion: 1,
   fields: {
     prompt: z.string(),
@@ -161,38 +168,19 @@ export const MyTestCaseSchemaV1 = defineTestCaseSchema({
 });
 ```
 
-### 2) Loader: how test cases become entities
-
-A loader reads external data and returns in-memory entities:
-
-```ts
-type LoaderResult<TTestCase> = {
-  testCases: TTestCase[];
-  responses: [];
-  scores: [];
-};
-```
-
-In the basic example (`packages/sdk-0.2/src/benchmarks/example/basic/loader.ts`) the loader reads a JSON array and maps it into `TestCase` entities.
-
-### 3) Provider: how runners talk to models
+### 2) Provider: how runners talk to models
 
 Runners communicate with models through a provider implementation. That’s how the same benchmark can run against different backends without rewriting the benchmark.
 
-There are also example providers meant to be read as reference implementations:
-
-- `packages/sdk-0.2/src/providers/example/echo.ts` (no network calls; returns deterministic content)
-- `packages/sdk-0.2/src/providers/example/restapi.ts` (calls your own REST “agent service”)
-
 If you already have a service in front of your model, the REST API provider example shows the pattern: accept the SDK’s `messages + model` input, translate it to an HTTP request, and translate the HTTP response back into a single string. Nothing else is required.
 
-### 4) Runner: run one test case
+### 3) Runner: run one test case
 
 A runner function typically executes one test case and returns `{ response, score? }`.
 
 This is intentional. Running many test cases is orchestration, and orchestration is where runtimes differ the most (parallelism, retries, persistence, resuming, UI, cost limits). The runner is the small, portable unit.
 
-In the basic example runner (`packages/sdk-0.2/src/benchmarks/example/basic/runner.ts`) you can see the responsibilities:
+In the example runners (e.g. `packages/sdk-0.2/src/benchmarks/examples/echo-basic/runner.ts`) you can see the responsibilities:
 
 - format a test case into provider-friendly input (for chat models, `messages[]`)
 - call `provider.forward(...)`
@@ -215,7 +203,7 @@ const response = ResponseSchemaV1.new({
 });
 ```
 
-### 5) Scorers: optional, but powerful
+### 5) Scorers
 
 Some benchmarks are easy to score deterministically (string match, regex extraction, set coverage). Some benchmarks need semantic judgment. Some benchmarks want both.
 
@@ -223,70 +211,106 @@ That’s why scorers are separate objects and why runners can accept more than o
 
 The examples show:
 
-- a deterministic scorer (`packages/sdk-0.2/src/benchmarks/example/basic/scorer.ts`)
-- a non-deterministic scorer (`packages/sdk-0.2/src/scorers/llm-judge.ts`)
-- a runner that can switch based on `scorer.kind` (`packages/sdk-0.2/src/benchmarks/example/basic/runner.ts`)
+- deterministic scoring inside a runner (`packages/sdk-0.2/src/benchmarks/examples/text-transform/runner.ts`)
+- multi-scorer dispatch (`packages/sdk-0.2/src/benchmarks/examples/exact-match-scorer/runner.ts`)
+- `MCQScorer` and `LLMAsAJudgeScorer` in `peerbench/scorers`
 
-## Usage: run a single test case end-to-end
-
-First, pick a benchmark and a provider:
-
-```ts
-import { example } from "peerbench/benchmarks";
-import { ExampleEchoLLMProvider } from "peerbench/providers";
+`LLMAsAJudgeScorer` returns a normalized `value` in the `0..1` range (inclusive).
 
-const provider = new ExampleEchoLLMProvider();
-```
+## Usage: run a single test case end-to-end
 
-Then build a test case entity and run it:
+First, define schemas and a runner (this is the “portable benchmark code”):
 
 ```ts
-const testCase = example.ExampleEchoTestCaseSchemaV1.new({
-  id: "tc-1",
-  instruction: "Repeat the input exactly",
-  input: "hello",
-  expectedOutput: "hello",
+import { defineRunner, idGeneratorUUIDv7 } from "peerbench";
+import { AbstractLLMProvider } from "peerbench/providers";
+import {
+  BaseResponseSchemaV1,
+  BaseScoreSchemaV1,
+  BaseTestCaseSchemaV1,
+  defineResponseSchema,
+  defineScoreSchema,
+  defineTestCaseSchema,
+} from "peerbench/schemas";
+import { ResponseExtensions } from "peerbench/schemas/extensions";
+import z from "zod";
+
+const Namespace = "example.peerbench.ai" as const;
+const Kind = "llm/echo-basic" as const;
+
+const TestCaseSchemaV1 = defineTestCaseSchema({
+  baseSchema: BaseTestCaseSchemaV1,
+  namespace: Namespace,
+  kind: Kind,
+  schemaVersion: 1,
+  fields: { input: z.string() },
 });
 
-const scorer = new example.ExampleExactMatchScorer();
+const ResponseSchemaV1 = defineResponseSchema({
+  baseSchema: BaseResponseSchemaV1,
+  namespace: Namespace,
+  kind: Kind,
+  schemaVersion: 1,
+  fields: { ...ResponseExtensions.ExtensionLLMResponseFieldsV1 },
+});
 
-const { response, score } = await example.runTestCase({
-  testCase,
-  provider,
-  scorer,
-  runConfig: { model: "example-model" },
+const ScoreSchemaV1 = defineScoreSchema({
+  baseSchema: BaseScoreSchemaV1,
+  namespace: Namespace,
+  kind: Kind,
+  schemaVersion: 1,
+  fields: {},
 });
-```
 
-If you want to load test cases instead of constructing them manually, use the loader:
+export const runner = defineRunner(
+  {
+    schemaSets: [
+      {
+        testCase: TestCaseSchemaV1,
+        response: ResponseSchemaV1,
+        score: ScoreSchemaV1,
+      },
+    ],
+    providers: [AbstractLLMProvider],
+    scorers: [],
+    runConfigSchema: { model: z.string() },
+  },
+  async ({ testCase, provider, runConfig, idGenerators }) => {
+    const providerResponse = await provider.forward({
+      model: runConfig.model,
+      messages: [{ role: "user", content: `Echo:\n${testCase.input}` }],
+    });
 
-```ts
-const loader = new example.ExampleJSONDataLoader();
-const { testCases } = await loader.loadData({
-  content: new TextEncoder().encode(
-    JSON.stringify([
+    const response = await ResponseSchemaV1.newWithId(
       {
-        id: "tc-1",
-        kind: "example.ts.echo",
-        schemaVersion: 1,
-        instruction: "Repeat the input exactly",
-        input: "hello",
-        expectedOutput: "hello",
+        data: providerResponse.data,
+        startedAt: providerResponse.startedAt,
+        completedAt: providerResponse.completedAt,
+        testCaseId: testCase.id,
+        modelSlug: runConfig.model,
+        provider: provider.kind,
+        inputTokensUsed: providerResponse.inputTokensUsed,
+        outputTokensUsed: providerResponse.outputTokensUsed,
+        inputCost: providerResponse.inputCost,
+        outputCost: providerResponse.outputCost,
       },
-    ])
-  ),
-});
+      idGenerators?.response ?? idGeneratorUUIDv7
+    );
+
+    return { response };
+  }
+);
 ```
 
 ## Usage: what the runtime adds (orchestration)
 
-Once you have `runTestCase(...)`, the runtime’s job is mostly about repetition and persistence.
+Once you have a runner, the runtime’s job is mostly about repetition and persistence.
 
 For example, a very small orchestrator might do:
 
 ```ts
 for (const testCase of testCases) {
-  const result = await example.runTestCase({ testCase, provider, runConfig });
+  const result = await runner({ testCase, provider, runConfig });
   // store `result.response` and `result.score` somewhere durable
   // decide how to handle errors, retries, progress, and budgets
 }
@@ -296,14 +320,14 @@ That loop is where your product decisions live. The SDK is intentionally not opi
 
 ## More examples to read
 
-The `example` benchmark is split into folders that each teach one idea:
+The examples under `packages/sdk-0.2/src/benchmarks/examples/` each teach one idea:
 
-- `packages/sdk-0.2/src/benchmarks/example/basic/`: the simplest complete example
-- `packages/sdk-0.2/src/benchmarks/example/multi-kind/`: one runner, multiple test case kinds
-- `packages/sdk-0.2/src/benchmarks/example/multi-scorer/`: one runner, multiple scorer implementations
+- `echo-basic`: minimal schema set + runner + storage examples
+- `text-transform`: one runner supports multiple kinds + deterministic scoring
+- `exact-match-scorer`: scorer dispatch pattern (algo scorer vs LLM judge scorer)
+- `mcq-qa-templated`: template variables + MCQ/QA tasks
 
 ## Design notes
 
 - Schemas are runtime-validated (Zod) so “type-only drift” doesn’t silently corrupt stored data.
 - Runners are per-test-case so they stay small and portable; runtimes keep orchestration control.
-- Kinds are namespaced strings (e.g. `example.ts.echo`) to avoid collisions across benchmarks.

package/dist/aggregators/index.d.ts

@@ -0,0 +1,67 @@
+import { BaseScoreV1, BaseTestCaseV1, BaseResponseV1 } from '../schemas/index.js';
+import { a as InferExtension } from '../index-BAioQhp2.js';
+import z__default from 'zod';
+import '../provider-BDjGp2y-.js';
+import '../abstract-Dec9Sc5O.js';
+
+declare abstract class AbstractAggregator {
+    abstract push(params: {
+        score: BaseScoreV1;
+        testCase?: BaseTestCaseV1;
+        response?: BaseResponseV1;
+    }): Promise<void>;
+    abstract aggregate(config?: unknown): Promise<unknown>;
+}
+
+/**
+ * Provides a set of fields that holds information about the LLM and its response.
+ */
+declare const ExtensionLLMResponseFieldsV1: {
+    data: z__default.ZodString;
+    modelSlug: z__default.ZodString;
+    provider: z__default.ZodString;
+    systemPromptId: z__default.ZodOptional<z__default.ZodString>;
+    inputTokensUsed: z__default.ZodOptional<z__default.ZodNumber>;
+    outputTokensUsed: z__default.ZodOptional<z__default.ZodNumber>;
+    inputCost: z__default.ZodOptional<z__default.ZodString>;
+    outputCost: z__default.ZodOptional<z__default.ZodString>;
+};
+
+/**
+ * Provides a set of fields that holds information about the LLM model
+ * that was used to judge the response.
+ */
+declare const ExtensionLLMAsAJudgeScoreFieldsV1: {
+    scorerAISystemPrompt: z__default.ZodOptional<z__default.ZodString>;
+    scorerAISystemPromptId: z__default.ZodOptional<z__default.ZodString>;
+    scorerAIProvider: z__default.ZodOptional<z__default.ZodString>;
+    scorerAIModelSlug: z__default.ZodOptional<z__default.ZodString>;
+    scorerAIInputTokensUsed: z__default.ZodOptional<z__default.ZodNumber>;
+    scorerAIOutputTokensUsed: z__default.ZodOptional<z__default.ZodNumber>;
+    scorerAIInputCost: z__default.ZodOptional<z__default.ZodString>;
+    scorerAIOutputCost: z__default.ZodOptional<z__default.ZodString>;
+};
+
+declare class AvgAggregator extends AbstractAggregator {
+    private separateBySystemPrompt;
+    private scores;
+    constructor(params: {
+        separateBySystemPrompt?: boolean;
+    });
+    push(params: {
+        score: InferExtension<typeof ExtensionLLMAsAJudgeScoreFieldsV1, BaseScoreV1>;
+        response: InferExtension<typeof ExtensionLLMResponseFieldsV1, BaseResponseV1>;
+        testCase?: BaseTestCaseV1;
+    }): Promise<void>;
+    aggregate(): Promise<{
+        [k: string]: {
+            average: number;
+            model: string;
+            total: number;
+            count: number;
+            systemPromptId?: string;
+        };
+    }>;
+}
+
+export { AbstractAggregator, AvgAggregator };

package/dist/aggregators/index.js

@@ -0,0 +1,46 @@
+import "../chunk-PZ5AY32C.js";
+
+// src/aggregators/abstract.ts
+var AbstractAggregator = class {
+};
+
+// src/aggregators/llm/avg.ts
+var AvgAggregator = class extends AbstractAggregator {
+  separateBySystemPrompt = false;
+  scores = {};
+  constructor(params) {
+    super();
+    this.separateBySystemPrompt = params.separateBySystemPrompt ?? false;
+  }
+  async push(params) {
+    const model = params.response.modelSlug;
+    const compositeKey = model + (params.response.systemPromptId ?? "");
+    const key = this.separateBySystemPrompt ? compositeKey : model;
+    if (!this.scores[key]) {
+      this.scores[key] = {
+        model,
+        systemPromptId: params.response.systemPromptId,
+        count: 0,
+        total: 0
+      };
+    }
+    this.scores[key].total += params.score.value;
+    this.scores[key].count++;
+  }
+  async aggregate() {
+    return Object.fromEntries(
+      Object.entries(this.scores).map(([model, score]) => [
+        model,
+        {
+          ...score,
+          average: score.total / score.count
+        }
+      ])
+    );
+  }
+};
+export {
+  AbstractAggregator,
+  AvgAggregator
+};
+//# sourceMappingURL=index.js.map

package/dist/aggregators/index.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../../src/aggregators/abstract.ts","../../src/aggregators/llm/avg.ts"],"sourcesContent":["import { BaseResponseV1, BaseTestCaseV1 } from \"@/schemas\";\nimport { BaseScoreV1 } from \"@/schemas/score\";\n\nexport abstract class AbstractAggregator {\n  abstract push(params: {\n    score: BaseScoreV1;\n    testCase?: BaseTestCaseV1;\n    response?: BaseResponseV1;\n  }): Promise<void>;\n\n  abstract aggregate(config?: unknown): Promise<unknown>;\n}\n","import { BaseResponseV1, BaseScoreV1, BaseTestCaseV1 } from \"@/schemas\";\nimport { AbstractAggregator } from \"../abstract\";\nimport { InferExtension } from \"@/utilities\";\nimport { ExtensionLLMResponseFieldsV1 } from \"@/schemas/extensions/response/llm\";\nimport { ExtensionLLMAsAJudgeScoreFieldsV1 } from \"@/schemas/extensions/score/llm-as-a-judge-scorer\";\n\nexport class AvgAggregator extends AbstractAggregator {\n  private separateBySystemPrompt: boolean = false;\n  private scores: Record<\n    string,\n    {\n      model: string;\n      total: number;\n      count: number;\n      systemPromptId?: string;\n    }\n  > = {};\n\n  constructor(params: { separateBySystemPrompt?: boolean }) {\n    super();\n    this.separateBySystemPrompt = params.separateBySystemPrompt ?? false;\n  }\n\n  override async push(params: {\n    score: InferExtension<\n      typeof ExtensionLLMAsAJudgeScoreFieldsV1,\n      BaseScoreV1\n    >;\n    response: InferExtension<\n      typeof ExtensionLLMResponseFieldsV1,\n      BaseResponseV1\n    >;\n    testCase?: BaseTestCaseV1;\n  }) {\n    const model = params.response.modelSlug;\n    const compositeKey = model + (params.response.systemPromptId ?? \"\");\n    const key = this.separateBySystemPrompt ? compositeKey : model;\n\n    if (!this.scores[key]) {\n      this.scores[key] = {\n        model,\n        systemPromptId: params.response.systemPromptId,\n        count: 0,\n        total: 0,\n      };\n    }\n\n    this.scores[key].total += params.score.value;\n    this.scores[key].count++;\n  }\n\n  override async aggregate() {\n    return Object.fromEntries(\n      Object.entries(this.scores).map(([model, score]) => [\n        model,\n        {\n          ...score,\n          average: score.total / score.count,\n        },\n      ])\n    );\n  }\n}\n"],"mappings":";;;AAGO,IAAe,qBAAf,MAAkC;AAQzC;;;ACLO,IAAM,gBAAN,cAA4B,mBAAmB;AAAA,EAC5C,yBAAkC;AAAA,EAClC,SAQJ,CAAC;AAAA,EAEL,YAAY,QAA8C;AACxD,UAAM;AACN,SAAK,yBAAyB,OAAO,0BAA0B;AAAA,EACjE;AAAA,EAEA,MAAe,KAAK,QAUjB;AACD,UAAM,QAAQ,OAAO,SAAS;AAC9B,UAAM,eAAe,SAAS,OAAO,SAAS,kBAAkB;AAChE,UAAM,MAAM,KAAK,yBAAyB,eAAe;AAEzD,QAAI,CAAC,KAAK,OAAO,GAAG,GAAG;AACrB,WAAK,OAAO,GAAG,IAAI;AAAA,QACjB;AAAA,QACA,gBAAgB,OAAO,SAAS;AAAA,QAChC,OAAO;AAAA,QACP,OAAO;AAAA,MACT;AAAA,IACF;AAEA,SAAK,OAAO,GAAG,EAAE,SAAS,OAAO,MAAM;AACvC,SAAK,OAAO,GAAG,EAAE;AAAA,EACnB;AAAA,EAEA,MAAe,YAAY;AACzB,WAAO,OAAO;AAAA,MACZ,OAAO,QAAQ,KAAK,MAAM,EAAE,IAAI,CAAC,CAAC,OAAO,KAAK,MAAM;AAAA,QAClD;AAAA,QACA;AAAA,UACE,GAAG;AAAA,UACH,SAAS,MAAM,QAAQ,MAAM;AAAA,QAC/B;AAAA,MACF,CAAC;AAAA,IACH;AAAA,EACF;AACF;","names":[]}