@zukhruf/evals 0.8.7 → 0.8.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -94,7 +94,7 @@ const pageKinds = new ScorerSet(
94
94
 
95
95
  ## Compare two runs
96
96
 
97
- Status: design for review (backlog #3870, [ADR 0004](docs/adr/0004-two-runs-are-compared-by-a-paired-bootstrap-over-their-tests.md)). The released package does not have `Run` or `PairedBootstrap` yet.
97
+ The package has `Run` and `PairedBootstrap` (backlog #3870, [ADR 0004](docs/adr/0004-two-runs-are-compared-by-a-paired-bootstrap-over-their-tests.md)).
98
98
 
99
99
  A caller runs one harness two times on the same tests: once with the baseline model and once with a change, such as another model. Two means side by side do not say if the change is better. `PairedBootstrap` gives one verdict for each name: better, worse or inconclusive, with the difference, its interval and a p-value.
100
100
 
@@ -146,7 +146,7 @@ const brier = paired.verdict(baseline, change, 'brier', {
146
146
  A run holds the scorecards of one model. A second `add` of a test id is one more epoch of that test: startup-credits adds three runs of one model as three epochs of one `Run`. To test many names, call `verdict` once for each name, and correct the p-values, such as with `holm` of `@zukhruf/mteb`. To test one name, use its p-value as it is.
147
147
 
148
148
  ```ts
149
- // @zukhruf/evals: two runs side by side (design for review)
149
+ // @zukhruf/evals: two runs side by side
150
150
  import type {
151
151
  ConfidenceIntervalMethod,
152
152
  DifferenceInterval,
@@ -287,6 +287,23 @@ run.add(new LabelComparer().compare('a', testCase), testCase.weight);
287
287
  - `form` is the arithmetic of the F-measure (ADR 0003). `new CountsForm()` is scikit-learn's and the default. It also gives the bits of TREC's F0.5 for integer counts. `new PrecisionRecallForm()` is van Rijsbergen's and SOB's.
288
288
  - `confusionMatrix()` throws for multilabel counts, as scikit-learn raises.
289
289
 
290
+ ### The class set
291
+
292
+ Status: design for review (backlog #3970, [ADR 0006](docs/adr/0006-classification-counts-only-the-classes-that-the-caller-gives.md)). The released package takes the classes from the answers, as scikit-learn's `labels=None` does.
293
+
294
+ A harness can state any text, such as the answer null of a failed call. If the answers give the classes, each such text adds a class, and the macro averages change. So the caller gives the class set, scikit-learn's `labels`, to the comparer and to the counts. The answers never add a class.
295
+
296
+ ```ts
297
+ // Proposal: no code exists yet.
298
+ import { ClassCounts, LabelComparer } from '@zukhruf/evals/classification';
299
+
300
+ const pageKinds = ['program', 'home', 'list', 'investor', 'other'];
301
+ const comparer = new LabelComparer({ labels: pageKinds });
302
+ const counts = new ClassCounts({ labels: pageKinds, multilabel: false });
303
+ ```
304
+
305
+ A label outside the class set counts in no class, as scikit-learn gives it when the caller gives `labels`. A stated label outside it is a miss of the expected class, and it is not a predicted count of any class. `LabelsMatch` still reads it as a wrong answer. `PerClass` and `AllClassMatches` read the class set of the comparison, so they take no `labels`. `ClassCounts.ofIndicators` takes its classes from the columns, as today. To port a scikit-learn call with `labels=None`, give the sorted labels of both sides.
306
+
290
307
  ### Probabilities
291
308
 
292
309
  `ProbabilityComparer` reads the stated probability of each class of an answer and gives a `ProbabilityComparison`. The answer is a `Map`, a plain object or the JSON text of an object. A key is a number label when the expected label is a number. A probability outside [0, 1] throws a `RangeError`. The case has one expected label.
@@ -344,6 +361,53 @@ export default {
344
361
 
345
362
  The name is the promptfoo `metric`, unless the scorer assertion gives its own `metric`. A name that the module does not hold is a compile error. promptfoo shows the mean of each `metric` over the tests. A `ScorerAssertion` gives one value for one test, so a micro average needs `PromptfooAssertion`, whose ratios keep their wholes.
346
363
 
364
+ ### A failed call
365
+
366
+ Status: design for review (backlog #3970, [ADR 0005](docs/adr/0005-a-failed-call-is-the-answer-null.md)). The released package does not have `FailedCall` yet.
367
+
368
+ A failed call is a model call that failed for its one test, also after its retries. promptfoo does not grade a result whose provider gives `error` or no output. That result is an error result: it has no named scores, so its case adds nothing to the wholes, and the recall goes up. A failed call is an answer of the harness: the answer null, the text `'null'`. promptfoo grades it, and each family reads it as no answer. `FailedCall` is a coinage of this package. A consumer uses one of two forms.
369
+
370
+ When the provider is your own, return `new FailedCall(reason).response(metadata)`. promptfoo grades the result, and `promptfoo retry <evalId>` does not run it again. The reason is in `metadata.failed`.
371
+
372
+ ```ts
373
+ // Proposal: no code exists yet.
374
+ import { FailedCall } from '@zukhruf/evals/promptfoo';
375
+
376
+ const [result] = await rule.compute([row.input], options?.abortSignal);
377
+ if (result !== undefined && 'error' in result)
378
+ return new FailedCall(result.error).response(metadata);
379
+ ```
380
+
381
+ When the provider is a built-in one, such as `openai:chat`, an `afterEach` extension hook grades each error result on `FailedCall.answer`. The result stays an error result, so `promptfoo retry <evalId>` and `--retry-errors` run it again.
382
+
383
+ ```ts
384
+ // Proposal: no code exists yet.
385
+ import {
386
+ type AfterEachExtensionHookContext,
387
+ ResultFailureReason,
388
+ assertions,
389
+ } from 'promptfoo';
390
+
391
+ import { FailedCall } from '@zukhruf/evals/promptfoo';
392
+
393
+ export async function afterEach({
394
+ test,
395
+ result,
396
+ }: AfterEachExtensionHookContext): Promise<void> {
397
+ if (result.failureReason !== ResultFailureReason.ERROR) return;
398
+ const graded = await assertions.runAssertions({
399
+ test,
400
+ providerResponse: { output: FailedCall.answer },
401
+ vars: test.vars ?? {},
402
+ });
403
+ Object.assign(result.namedScores, graded.namedScores);
404
+ }
405
+ ```
406
+
407
+ A hook cannot write `namedScoreWeights`, so promptfoo gives each score of the hook the weight 1. Use the hook only with scorer assertions whose metrics weigh 1, as in SOB. A `PromptfooAssertion` gives each ratio its whole as the weight, so it needs the first form.
408
+
409
+ A failure of the run, such as a 401, is not a failed call: each call would fail. Throw it. The result then stays an error result, and `promptfoo retry <evalId>` runs it again.
410
+
347
411
  ## What the package does not do
348
412
 
349
413
  - It does not run a harness, load a dataset, store results or show them. promptfoo does that.
@@ -726,4 +790,6 @@ The package was built in two items: mutex #3888 (the core, `./promptfoo` and the
726
790
  - [ADR 0001: one package, with a subpath for each shape of answer](docs/adr/0001-one-package-with-a-subpath-for-each-shape-of-answer.md)
727
791
  - [ADR 0002: the core is generic over the answer](docs/adr/0002-the-core-is-generic-over-the-answer.md)
728
792
  - [ADR 0003: classification follows scikit-learn, from the counts of each class](docs/adr/0003-classification-follows-scikit-learn-from-the-counts-of-each-class.md)
729
- - [ADR 0004: two runs are compared by a paired bootstrap over their tests](docs/adr/0004-two-runs-are-compared-by-a-paired-bootstrap-over-their-tests.md) (design for review)
793
+ - [ADR 0004: two runs are compared by a paired bootstrap over their tests](docs/adr/0004-two-runs-are-compared-by-a-paired-bootstrap-over-their-tests.md)
794
+ - [ADR 0005: a failed call is the answer null](docs/adr/0005-a-failed-call-is-the-answer-null.md) (design for review)
795
+ - [ADR 0006: classification counts only the classes that the caller gives](docs/adr/0006-classification-counts-only-the-classes-that-the-caller-gives.md) (design for review)
package/dist/index.d.ts CHANGED
@@ -6,6 +6,8 @@ export type { Check, Scorecard } from './scoring/scorecard.ts';
6
6
  export type { CaseReader, ScorecardScorer, Weighted, } from './scoring/scorecard-scorer.ts';
7
7
  export { Totals } from './scoring/totals.ts';
8
8
  export { Bootstrap, type WeightedScorecard } from './scoring/bootstrap.ts';
9
+ export { Run } from './scoring/run.ts';
10
+ export { PairedBootstrap, type Verdict } from './scoring/paired-bootstrap.ts';
9
11
  export type { Scorer } from './scorers/scorer.ts';
10
12
  export type { Matches } from './scorers/matches.ts';
11
13
  export { Precision } from './scorers/precision.ts';
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,wBAAwB,CAAC;AAC/C,OAAO,EAAE,WAAW,EAAE,MAAM,gCAAgC,CAAC;AAC7D,OAAO,EAAE,MAAM,EAAE,MAAM,yBAAyB,CAAC;AACjD,OAAO,EACL,UAAU,EACV,KAAK,YAAY,EACjB,mBAAmB,GACpB,MAAM,iCAAiC,CAAC;AACzC,YAAY,EAAE,KAAK,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AAC/D,YAAY,EACV,UAAU,EACV,eAAe,EACf,QAAQ,GACT,MAAM,+BAA+B,CAAC;AACvC,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,SAAS,EAAE,KAAK,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAC3E,YAAY,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAClD,YAAY,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AACpD,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,KAAK,EAAE,MAAM,qBAAqB,CAAC;AAC5C,OAAO,EAAE,EAAE,EAAE,MAAM,iBAAiB,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,IAAI,EAAE,MAAM,mBAAmB,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,OAAO,EAAE,MAAM,uBAAuB,CAAC;AAChD,OAAO,EAAE,gBAAgB,EAAE,MAAM,gCAAgC,CAAC;AAClE,OAAO,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAC9D,OAAO,EAAE,MAAM,EAAE,MAAM,sBAAsB,CAAC;AAC9C,YAAY,EAAE,QAAQ,EAAE,MAAM,uBAAuB,CAAC;AACtD,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,SAAS,EAAE,MAAM,yBAAyB,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,wBAAwB,CAAC;AAC/C,OAAO,EAAE,WAAW,EAAE,MAAM,gCAAgC,CAAC;AAC7D,OAAO,EAAE,MAAM,EAAE,MAAM,yBAAyB,CAAC;AACjD,OAAO,EACL,UAAU,EACV,KAAK,YAAY,EACjB,mBAAmB,GACpB,MAAM,iCAAiC,CAAC;AACzC,YAAY,EAAE,KAAK,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AAC/D,YAAY,EACV,UAAU,EACV,eAAe,EACf,QAAQ,GACT,MAAM,+BAA+B,CAAC;AACvC,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,SAAS,EAAE,KAAK,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAC3E,OAAO,EAAE,GAAG,EAAE,MAAM,kBAAkB,CAAC;AACvC,OAAO,EAAE,eAAe,EAAE,KAAK,OAAO,EAAE,MAAM,+BAA+B,CAAC;AAC9E,YAAY,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAClD,YAAY,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AACpD,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,KAAK,EAAE,MAAM,qBAAqB,CAAC;AAC5C,OAAO,EAAE,EAAE,EAAE,MAAM,iBAAiB,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,IAAI,EAAE,MAAM,mBAAmB,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,OAAO,EAAE,MAAM,uBAAuB,CAAC;AAChD,OAAO,EAAE,gBAAgB,EAAE,MAAM,gCAAgC,CAAC;AAClE,OAAO,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAC9D,OAAO,EAAE,MAAM,EAAE,MAAM,sBAAsB,CAAC;AAC9C,YAAY,EAAE,QAAQ,EAAE,MAAM,uBAAuB,CAAC;AACtD,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,SAAS,EAAE,MAAM,yBAAyB,CAAC"}
package/dist/index.js CHANGED
@@ -4,6 +4,8 @@ export { Counts } from "./aggregation/counts.js";
4
4
  export { CountsForm, PrecisionRecallForm, } from "./aggregation/f-measure-form.js";
5
5
  export { Totals } from "./scoring/totals.js";
6
6
  export { Bootstrap } from "./scoring/bootstrap.js";
7
+ export { Run } from "./scoring/run.js";
8
+ export { PairedBootstrap } from "./scoring/paired-bootstrap.js";
7
9
  export { Precision } from "./scorers/precision.js";
8
10
  export { Recall } from "./scorers/recall.js";
9
11
  export { FBeta } from "./scorers/f-beta.js";
package/dist/index.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,wBAAwB,CAAC;AAC/C,OAAO,EAAE,WAAW,EAAE,MAAM,gCAAgC,CAAC;AAC7D,OAAO,EAAE,MAAM,EAAE,MAAM,yBAAyB,CAAC;AACjD,OAAO,EACL,UAAU,EAEV,mBAAmB,GACpB,MAAM,iCAAiC,CAAC;AAOzC,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,SAAS,EAA0B,MAAM,wBAAwB,CAAC;AAG3E,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,KAAK,EAAE,MAAM,qBAAqB,CAAC;AAC5C,OAAO,EAAE,EAAE,EAAE,MAAM,iBAAiB,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,IAAI,EAAE,MAAM,mBAAmB,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,OAAO,EAAE,MAAM,uBAAuB,CAAC;AAChD,OAAO,EAAE,gBAAgB,EAAE,MAAM,gCAAgC,CAAC;AAClE,OAAO,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAC9D,OAAO,EAAE,MAAM,EAAE,MAAM,sBAAsB,CAAC;AAE9C,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,SAAS,EAAE,MAAM,yBAAyB,CAAC"}
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,wBAAwB,CAAC;AAC/C,OAAO,EAAE,WAAW,EAAE,MAAM,gCAAgC,CAAC;AAC7D,OAAO,EAAE,MAAM,EAAE,MAAM,yBAAyB,CAAC;AACjD,OAAO,EACL,UAAU,EAEV,mBAAmB,GACpB,MAAM,iCAAiC,CAAC;AAOzC,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,SAAS,EAA0B,MAAM,wBAAwB,CAAC;AAC3E,OAAO,EAAE,GAAG,EAAE,MAAM,kBAAkB,CAAC;AACvC,OAAO,EAAE,eAAe,EAAgB,MAAM,+BAA+B,CAAC;AAG9E,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,MAAM,EAAE,MAAM,qBAAqB,CAAC;AAC7C,OAAO,EAAE,KAAK,EAAE,MAAM,qBAAqB,CAAC;AAC5C,OAAO,EAAE,EAAE,EAAE,MAAM,iBAAiB,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,IAAI,EAAE,MAAM,mBAAmB,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,EAAE,OAAO,EAAE,MAAM,uBAAuB,CAAC;AAChD,OAAO,EAAE,gBAAgB,EAAE,MAAM,gCAAgC,CAAC;AAClE,OAAO,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAC9D,OAAO,EAAE,MAAM,EAAE,MAAM,sBAAsB,CAAC;AAE9C,OAAO,EAAE,OAAO,EAAE,MAAM,sBAAsB,CAAC;AAC/C,OAAO,EAAE,SAAS,EAAE,MAAM,yBAAyB,CAAC"}
@@ -0,0 +1,24 @@
1
+ import { type ConfidenceIntervalMethod, DifferenceInterval } from '@zukhruf/statistics';
2
+ import { type Run } from './run.ts';
3
+ export type Verdict = {
4
+ readonly name: string;
5
+ readonly result: 'better' | 'worse' | 'inconclusive';
6
+ readonly difference: DifferenceInterval;
7
+ readonly p: number;
8
+ readonly improvements: number;
9
+ readonly regressions: number;
10
+ readonly ties: number;
11
+ readonly tests: number;
12
+ };
13
+ export declare class PairedBootstrap {
14
+ #private;
15
+ constructor({ nResamples, randomIndex, interval, }: {
16
+ readonly nResamples: number;
17
+ readonly randomIndex: (n: number) => number;
18
+ readonly interval: ConfidenceIntervalMethod;
19
+ });
20
+ verdict(baseline: Run, change: Run, name: string, { higherIsBetter }: {
21
+ readonly higherIsBetter: boolean;
22
+ }): Verdict;
23
+ }
24
+ //# sourceMappingURL=paired-bootstrap.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"paired-bootstrap.d.ts","sourceRoot":"","sources":["../../src/scoring/paired-bootstrap.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,wBAAwB,EAC7B,kBAAkB,EAEnB,MAAM,qBAAqB,CAAC;AAI7B,OAAO,EAAE,KAAK,GAAG,EAAY,MAAM,UAAU,CAAC;AAE9C,MAAM,MAAM,OAAO,GAAG;IACpB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,MAAM,EAAE,QAAQ,GAAG,OAAO,GAAG,cAAc,CAAC;IAErD,QAAQ,CAAC,UAAU,EAAE,kBAAkB,CAAC;IAExC,QAAQ,CAAC,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,YAAY,EAAE,MAAM,CAAC;IAC9B,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB,CAAC;AAgBF,qBAAa,eAAe;;gBAKd,EACV,UAAU,EACV,WAAW,EACX,QAAQ,GACT,EAAE;QACD,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;QAC5B,QAAQ,CAAC,WAAW,EAAE,CAAC,CAAC,EAAE,MAAM,KAAK,MAAM,CAAC;QAC5C,QAAQ,CAAC,QAAQ,EAAE,wBAAwB,CAAC;KAC7C;IAMD,OAAO,CACL,QAAQ,EAAE,GAAG,EACb,MAAM,EAAE,GAAG,EACX,IAAI,EAAE,MAAM,EACZ,EAAE,cAAc,EAAE,EAAE;QAAE,QAAQ,CAAC,cAAc,EAAE,OAAO,CAAA;KAAE,GACvD,OAAO;CA8GX"}
@@ -0,0 +1,94 @@
1
+ import { DifferenceInterval, IidBootstrap, } from '@zukhruf/statistics';
2
+ import { RatioOfSums } from "../aggregation/ratio-of-sums.js";
3
+ import { runTests } from "./run.js";
4
+ // The paired bootstrap of two runs on the same tests (Koehn 2004): each
5
+ // resample draws as many tests as there are, with replacement, and gives the
6
+ // ratio of sums of the name over the change minus the same over the baseline,
7
+ // on the same tests. The verdict reads the interval of these differences
8
+ // against 0, in the direction of the name (ADR 0004). The generator keeps its
9
+ // state between calls, so the order of the calls is the order of the draws.
10
+ export class PairedBootstrap {
11
+ #bootstrap;
12
+ #interval;
13
+ #nResamples;
14
+ constructor({ nResamples, randomIndex, interval, }) {
15
+ this.#bootstrap = new IidBootstrap({ nResamples, randomIndex });
16
+ this.#interval = interval;
17
+ this.#nResamples = nResamples;
18
+ }
19
+ verdict(baseline, change, name, { higherIsBetter }) {
20
+ const pairs = PairedBootstrap.#pairs(baseline, change, name);
21
+ const distribution = this.#bootstrap.distribution(pairs, PairedBootstrap.#difference);
22
+ const interval = this.#interval.of(distribution);
23
+ const difference = new DifferenceInterval(interval.low, interval.high, PairedBootstrap.#difference(pairs));
24
+ const direction = higherIsBetter ? 1 : -1;
25
+ const signs = pairs.map((pair) => Math.sign((pair.change.value - pair.baseline.value) * direction));
26
+ return {
27
+ name,
28
+ result: PairedBootstrap.#result(difference, higherIsBetter),
29
+ difference,
30
+ p: this.#p(distribution),
31
+ improvements: signs.filter((sign) => sign > 0).length,
32
+ regressions: signs.filter((sign) => sign < 0).length,
33
+ ties: signs.filter((sign) => sign === 0).length,
34
+ tests: pairs.length,
35
+ };
36
+ }
37
+ // Two times the smaller share of the differences at or below 0 and at or
38
+ // above 0, at most 1, as mteb's pairedBootstrapTest gives it.
39
+ #p(distribution) {
40
+ const share = (side) => distribution.filter(side).length / this.#nResamples;
41
+ return Math.min(1, 2 *
42
+ Math.min(share((value) => value <= 0), share((value) => value >= 0)));
43
+ }
44
+ static #result(difference, higherIsBetter) {
45
+ const [better, worse] = higherIsBetter
46
+ ? [difference.isAbove(0), difference.isBelow(0)]
47
+ : [difference.isBelow(0), difference.isAbove(0)];
48
+ if (better)
49
+ return 'better';
50
+ if (worse)
51
+ return 'worse';
52
+ return 'inconclusive';
53
+ }
54
+ static #difference(pairs) {
55
+ return (RatioOfSums.of(pairs.map((pair) => pair.change.times(pair.weight)))
56
+ .value -
57
+ RatioOfSums.of(pairs.map((pair) => pair.baseline.times(pair.weight)))
58
+ .value);
59
+ }
60
+ // The tests of the two runs, paired by test id in the order of the
61
+ // baseline's first adds. The pairing refuses a test id that one run does
62
+ // not have, as mteb's pairedBootstrapTest refuses (strike 2,
63
+ // docs/copied-code.md).
64
+ static #pairs(baseline, change, name) {
65
+ const baselineTests = runTests(baseline);
66
+ const changeTests = runTests(change);
67
+ const unpaired = [...baselineTests.keys(), ...changeTests.keys()].filter((id) => !(baselineTests.has(id) && changeTests.has(id)));
68
+ if (unpaired.length > 0)
69
+ throw new RangeError(`A paired bootstrap needs the same test ids in the baseline and the change. Not in both: ${unpaired.join(', ')}`);
70
+ const joined = [...baselineTests].flatMap(([id, test]) => {
71
+ const changeTest = changeTests.get(id);
72
+ return changeTest === undefined
73
+ ? []
74
+ : [{ id, baseline: test, change: changeTest }];
75
+ });
76
+ const reweighted = joined.filter((test) => test.baseline.weight !== test.change.weight);
77
+ if (reweighted.length > 0)
78
+ throw new RangeError(`A test has the same weight in the baseline and the change. Other weights: ${reweighted
79
+ .map((test) => `${test.id} (${test.baseline.weight} and ${test.change.weight})`)
80
+ .join(', ')}`);
81
+ for (const [run, tests] of [
82
+ ['baseline', baselineTests],
83
+ ['change', changeTests],
84
+ ])
85
+ if (![...tests.values()].some((test) => test.gives(name)))
86
+ throw new Error(`No scorecard of the ${run} run gave a ratio named "${name}"`);
87
+ return joined.map((test) => ({
88
+ baseline: test.baseline.mean(name),
89
+ change: test.change.mean(name),
90
+ weight: test.baseline.weight,
91
+ }));
92
+ }
93
+ }
94
+ //# sourceMappingURL=paired-bootstrap.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"paired-bootstrap.js","sourceRoot":"","sources":["../../src/scoring/paired-bootstrap.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,kBAAkB,EAClB,YAAY,GACb,MAAM,qBAAqB,CAAC;AAE7B,OAAO,EAAE,WAAW,EAAE,MAAM,iCAAiC,CAAC;AAE9D,OAAO,EAAY,QAAQ,EAAE,MAAM,UAAU,CAAC;AAuB9C,wEAAwE;AACxE,6EAA6E;AAC7E,8EAA8E;AAC9E,yEAAyE;AACzE,8EAA8E;AAC9E,4EAA4E;AAC5E,MAAM,OAAO,eAAe;IACjB,UAAU,CAAe;IACzB,SAAS,CAA2B;IACpC,WAAW,CAAS;IAE7B,YAAY,EACV,UAAU,EACV,WAAW,EACX,QAAQ,GAKT;QACC,IAAI,CAAC,UAAU,GAAG,IAAI,YAAY,CAAC,EAAE,UAAU,EAAE,WAAW,EAAE,CAAC,CAAC;QAChE,IAAI,CAAC,SAAS,GAAG,QAAQ,CAAC;QAC1B,IAAI,CAAC,WAAW,GAAG,UAAU,CAAC;IAChC,CAAC;IAED,OAAO,CACL,QAAa,EACb,MAAW,EACX,IAAY,EACZ,EAAE,cAAc,EAAwC;QAExD,MAAM,KAAK,GAAG,eAAe,CAAC,MAAM,CAAC,QAAQ,EAAE,MAAM,EAAE,IAAI,CAAC,CAAC;QAC7D,MAAM,YAAY,GAAG,IAAI,CAAC,UAAU,CAAC,YAAY,CAC/C,KAAK,EACL,eAAe,CAAC,WAAW,CAC5B,CAAC;QACF,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC,YAAY,CAAC,CAAC;QACjD,MAAM,UAAU,GAAG,IAAI,kBAAkB,CACvC,QAAQ,CAAC,GAAG,EACZ,QAAQ,CAAC,IAAI,EACb,eAAe,CAAC,WAAW,CAAC,KAAK,CAAC,CACnC,CAAC;QACF,MAAM,SAAS,GAAG,cAAc,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QAC1C,MAAM,KAAK,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAC/B,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,GAAG,SAAS,CAAC,CACjE,CAAC;QACF,OAAO;YACL,IAAI;YACJ,MAAM,EAAE,eAAe,CAAC,OAAO,CAAC,UAAU,EAAE,cAAc,CAAC;YAC3D,UAAU;YACV,CAAC,EAAE,IAAI,CAAC,EAAE,CAAC,YAAY,CAAC;YACxB,YAAY,EAAE,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC,CAAC,CAAC,MAAM;YACrD,WAAW,EAAE,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC,CAAC,CAAC,MAAM;YACpD,IAAI,EAAE,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,MAAM;YAC/C,KAAK,EAAE,KAAK,CAAC,MAAM;SACpB,CAAC;IACJ,CAAC;IAED,yEAAyE;IACzE,8DAA8D;IAC9D,EAAE,CAAC,YAA+B;QAChC,MAAM,KAAK,GAAG,CAAC,IAAgC,EAAE,EAAE,CACjD,YAAY,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;QACtD,OAAO,IAAI,CAAC,GAAG,CACb,CAAC,EACD,CAAC;YACC,IAAI,CAAC,GAAG,CACN,KAAK,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,IAAI,CAAC,CAAC,EAC5B,KAAK,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,IAAI,CAAC,CAAC,CAC7B,CACJ,CAAC;IACJ,CAAC;IAED,MAAM,CAAC,OAAO,CACZ,UAA8B,EAC9B,cAAuB;QAEvB,MAAM,CAAC,MAAM,EAAE,KAAK,CAAC,GAAG,cAAc;YACpC,CAAC,CAAC,CAAC,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;YAChD,CAAC,CAAC,CAAC,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC;QACnD,IAAI,MAAM;YAAE,OAAO,QAAQ,CAAC;QAC5B,IAAI,KAAK;YAAE,OAAO,OAAO,CAAC;QAC1B,OAAO,cAAc,CAAC;IACxB,CAAC;IAED,MAAM,CAAC,WAAW,CAAC,KAAsB;QACvC,OAAO,CACL,WAAW,CAAC,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC;aAChE,KAAK;YACR,WAAW,CAAC,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC;iBAClE,KAAK,CACT,CAAC;IACJ,CAAC;IAED,mEAAmE;IACnE,yEAAyE;IACzE,6DAA6D;IAC7D,wBAAwB;IACxB,MAAM,CAAC,MAAM,CAAC,QAAa,EAAE,MAAW,EAAE,IAAY;QACpD,MAAM,aAAa,GAAG,QAAQ,CAAC,QAAQ,CAAC,CAAC;QACzC,MAAM,WAAW,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC;QACrC,MAAM,QAAQ,GAAG,CAAC,GAAG,aAAa,CAAC,IAAI,EAAE,EAAE,GAAG,WAAW,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CACtE,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC,CAAC,aAAa,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,WAAW,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CACxD,CAAC;QACF,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC;YACrB,MAAM,IAAI,UAAU,CAClB,2FAA2F,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACjH,CAAC;QACJ,MAAM,MAAM,GAAG,CAAC,GAAG,aAAa,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,EAAE,IAAI,CAAC,EAAE,EAAE;YACvD,MAAM,UAAU,GAAG,WAAW,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;YACvC,OAAO,UAAU,KAAK,SAAS;gBAC7B,CAAC,CAAC,EAAE;gBACJ,CAAC,CAAC,CAAC,EAAE,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,UAAU,EAAE,CAAC,CAAC;QACnD,CAAC,CAAC,CAAC;QACH,MAAM,UAAU,GAAG,MAAM,CAAC,MAAM,CAC9B,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,MAAM,KAAK,IAAI,CAAC,MAAM,CAAC,MAAM,CACtD,CAAC;QACF,IAAI,UAAU,CAAC,MAAM,GAAG,CAAC;YACvB,MAAM,IAAI,UAAU,CAClB,6EAA6E,UAAU;iBACpF,GAAG,CACF,CAAC,IAAI,EAAE,EAAE,CACP,GAAG,IAAI,CAAC,EAAE,KAAK,IAAI,CAAC,QAAQ,CAAC,MAAM,QAAQ,IAAI,CAAC,MAAM,CAAC,MAAM,GAAG,CACnE;iBACA,IAAI,CAAC,IAAI,CAAC,EAAE,CAChB,CAAC;QACJ,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI;YACzB,CAAC,UAAU,EAAE,aAAa,CAAC;YAC3B,CAAC,QAAQ,EAAE,WAAW,CAAC;SACf;YACR,IAAI,CAAC,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;gBACvD,MAAM,IAAI,KAAK,CACb,uBAAuB,GAAG,4BAA4B,IAAI,GAAG,CAC9D,CAAC;QACN,OAAO,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;YAC3B,QAAQ,EAAE,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC;YAClC,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC;YAC9B,MAAM,EAAE,IAAI,CAAC,QAAQ,CAAC,MAAM;SAC7B,CAAC,CAAC,CAAC;IACN,CAAC;CACF"}
@@ -0,0 +1,7 @@
1
+ import type { Scorecard } from './scorecard.ts';
2
+ import { TestEpochs } from './test-epochs.ts';
3
+ export declare class Run {
4
+ add(testId: string, scorecard: Scorecard, weight: number): void;
5
+ }
6
+ export declare function runTests(run: Run): ReadonlyMap<string, TestEpochs>;
7
+ //# sourceMappingURL=run.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"run.d.ts","sourceRoot":"","sources":["../../src/scoring/run.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAChD,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AAU9C,qBAAa,GAAG;IACd,GAAG,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,EAAE,SAAS,EAAE,MAAM,EAAE,MAAM,GAAG,IAAI;CAWhE;AAED,wBAAgB,QAAQ,CAAC,GAAG,EAAE,GAAG,GAAG,WAAW,CAAC,MAAM,EAAE,UAAU,CAAC,CAElE"}
@@ -0,0 +1,19 @@
1
+ import { TestEpochs } from "./test-epochs.js";
2
+ // The tests of each run, in the order of their first add. PairedBootstrap of
3
+ // this package reads them through runTests; the package does not export it.
4
+ const testsOfRun = new WeakMap();
5
+ const testsOf = (run) => testsOfRun.getOrInsertComputed(run, () => new Map());
6
+ // The scorecards of one model on many tests. A second add of a test id is
7
+ // one more epoch of that test.
8
+ export class Run {
9
+ add(testId, scorecard, weight) {
10
+ const test = testsOf(this).getOrInsertComputed(testId, () => new TestEpochs(weight));
11
+ if (test.weight !== weight)
12
+ throw new RangeError(`Test "${testId}" has the weight ${test.weight}, not ${weight}: the weight belongs to the case`);
13
+ test.add(scorecard);
14
+ }
15
+ }
16
+ export function runTests(run) {
17
+ return testsOf(run);
18
+ }
19
+ //# sourceMappingURL=run.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"run.js","sourceRoot":"","sources":["../../src/scoring/run.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AAE9C,6EAA6E;AAC7E,4EAA4E;AAC5E,MAAM,UAAU,GAAG,IAAI,OAAO,EAAgC,CAAC;AAC/D,MAAM,OAAO,GAAG,CAAC,GAAQ,EAAE,EAAE,CAC3B,UAAU,CAAC,mBAAmB,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,IAAI,GAAG,EAAE,CAAC,CAAC;AAEvD,0EAA0E;AAC1E,+BAA+B;AAC/B,MAAM,OAAO,GAAG;IACd,GAAG,CAAC,MAAc,EAAE,SAAoB,EAAE,MAAc;QACtD,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,mBAAmB,CAC5C,MAAM,EACN,GAAG,EAAE,CAAC,IAAI,UAAU,CAAC,MAAM,CAAC,CAC7B,CAAC;QACF,IAAI,IAAI,CAAC,MAAM,KAAK,MAAM;YACxB,MAAM,IAAI,UAAU,CAClB,SAAS,MAAM,oBAAoB,IAAI,CAAC,MAAM,SAAS,MAAM,kCAAkC,CAChG,CAAC;QACJ,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;IACtB,CAAC;CACF;AAED,MAAM,UAAU,QAAQ,CAAC,GAAQ;IAC/B,OAAO,OAAO,CAAC,GAAG,CAAC,CAAC;AACtB,CAAC"}
@@ -0,0 +1,11 @@
1
+ import { Ratio } from '../aggregation/ratio.ts';
2
+ import type { Scorecard } from './scorecard.ts';
3
+ export declare class TestEpochs {
4
+ #private;
5
+ readonly weight: number;
6
+ constructor(weight: number);
7
+ add(scorecard: Scorecard): void;
8
+ gives(name: string): boolean;
9
+ mean(name: string): Ratio;
10
+ }
11
+ //# sourceMappingURL=test-epochs.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"test-epochs.d.ts","sourceRoot":"","sources":["../../src/scoring/test-epochs.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,KAAK,EAAE,MAAM,yBAAyB,CAAC;AAChD,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAGhD,qBAAa,UAAU;;IACrB,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;gBAGZ,MAAM,EAAE,MAAM;IAI1B,GAAG,CAAC,SAAS,EAAE,SAAS,GAAG,IAAI;IAI/B,KAAK,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO;IAO5B,IAAI,CAAC,IAAI,EAAE,MAAM,GAAG,KAAK;CAS1B"}
@@ -0,0 +1,25 @@
1
+ import { RatioOfSums } from "../aggregation/ratio-of-sums.js";
2
+ import { Ratio } from "../aggregation/ratio.js";
3
+ // The epochs of one test of a run, with the weight of its case.
4
+ export class TestEpochs {
5
+ weight;
6
+ #scorecards = [];
7
+ constructor(weight) {
8
+ this.weight = weight;
9
+ }
10
+ add(scorecard) {
11
+ this.#scorecards.push(scorecard);
12
+ }
13
+ gives(name) {
14
+ return this.#scorecards.some((scorecard) => scorecard.ratios.has(name));
15
+ }
16
+ // The mean of the epochs: Ratio(sum of the parts / k, sum of the wholes / k)
17
+ // for k epochs (Miller 2024, §3.1). A scorecard without the name counts as
18
+ // Ratio(0, 0), as in Totals.
19
+ mean(name) {
20
+ const sum = RatioOfSums.of(this.#scorecards.map((scorecard) => scorecard.ratios.get(name) ?? new Ratio(0, 0))).ratio;
21
+ const k = this.#scorecards.length;
22
+ return new Ratio(sum.part / k, sum.whole / k);
23
+ }
24
+ }
25
+ //# sourceMappingURL=test-epochs.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"test-epochs.js","sourceRoot":"","sources":["../../src/scoring/test-epochs.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,iCAAiC,CAAC;AAC9D,OAAO,EAAE,KAAK,EAAE,MAAM,yBAAyB,CAAC;AAGhD,gEAAgE;AAChE,MAAM,OAAO,UAAU;IACZ,MAAM,CAAS;IACf,WAAW,GAAgB,EAAE,CAAC;IAEvC,YAAY,MAAc;QACxB,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;IACvB,CAAC;IAED,GAAG,CAAC,SAAoB;QACtB,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACnC,CAAC;IAED,KAAK,CAAC,IAAY;QAChB,OAAO,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC;IAC1E,CAAC;IAED,6EAA6E;IAC7E,2EAA2E;IAC3E,6BAA6B;IAC7B,IAAI,CAAC,IAAY;QACf,MAAM,GAAG,GAAG,WAAW,CAAC,EAAE,CACxB,IAAI,CAAC,WAAW,CAAC,GAAG,CAClB,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,IAAI,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAC7D,CACF,CAAC,KAAK,CAAC;QACR,MAAM,CAAC,GAAG,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC;QAClC,OAAO,IAAI,KAAK,CAAC,GAAG,CAAC,IAAI,GAAG,CAAC,EAAE,GAAG,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC;IAChD,CAAC;CACF"}
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@zukhruf/evals",
3
- "version": "0.8.7",
3
+ "version": "0.8.8",
4
4
  "description": "Scores the answers of an eval against their labels, with one import path for each shape of answer: a core of ratios and totals, ./structured for structured values, ./classification for labels as scikit-learn scores them, and ./promptfoo.",
5
5
  "license": "MIT",
6
6
  "repository": {
@@ -63,8 +63,8 @@
63
63
  "access": "public"
64
64
  },
65
65
  "dependencies": {
66
- "@zukhruf/python": "^0.8.7",
67
- "@zukhruf/statistics": "^0.8.7"
66
+ "@zukhruf/python": "^0.8.8",
67
+ "@zukhruf/statistics": "^0.8.8"
68
68
  },
69
69
  "peerDependencies": {
70
70
  "@cfworker/json-schema": "4.1.4",
@@ -80,7 +80,8 @@
80
80
  },
81
81
  "devDependencies": {
82
82
  "@cfworker/json-schema": "4.1.4",
83
- "@zukhruf/testing": "^0.8.7",
83
+ "@zukhruf/random": "*",
84
+ "@zukhruf/testing": "^0.8.8",
84
85
  "munkres": "^2.1.1",
85
86
  "promptfoo": "0.125.1",
86
87
  "python-random": "1.0.0"