textopt 0.1.0 → 0.2.0

package/README.md

@@ -4,6 +4,8 @@ Core interfaces and optimizers for textopt.
 
 This package has no runtime dependencies. For an overview of the algorithms and guidance on choosing one, see the [project README](https://github.com/ctdio/textopt#readme).
 
+The long-form guides ship with the package, under `docs/`, so an installed copy documents the version installed rather than whatever `main` has become. Agent-facing guidance lives there and in the doc comments on the API itself.
+
 ## Entry points
 
 | Import                     | Contains                                                                                                                            |
@@ -95,7 +97,7 @@ evaluate(args: EvaluateArgs<Datum, K>): Promise<EvaluationBatch<Trajectory, Outp
 
 **`transient`** marks scores caused by infrastructure failures such as rate limits, 5xx responses, or network errors. Transient scores are not cached.
 
-**`Optimizer<Stop extends string>`** defines `optimize(task: OptimizerTask) => Promise<OptimizerResult>`. `OptimizerTask` contains the shared run inputs: `seedCandidate`, `trainingSet`, `validationSet`, `testSet`, `adapter`, `maxMetricCalls`, `maxCostUsd`, `maxWallClockMs`, `cacheNamespace`, `retry`, and `signal`. `OptimizerResult` contains `bestCandidate`, `bestScore`, `bestOutputs`, `metricCalls`, `usage`, `testScore`, `testMetricCalls`, `testUsage`, and `stopReason`. Optimizer-specific task and result types extend these interfaces.
+**`Optimizer<Stop extends string>`** defines `optimize(task: OptimizerTask) => Promise<OptimizerResult>`. `OptimizerTask` contains the shared run inputs: `seedCandidate`, `trainingSet`, `validationSet`, `testSet`, `adapter`, `maxMetricCalls`, `maxCostUsd`, `maxWallClockMs`, `cacheNamespace`, `retry`, and `signal`. `OptimizerResult` contains `bestCandidate`, `bestScore`, `bestOutputs`, `metricCalls`, `usage`, `testScore`, `testMetricCalls`, `testUsage`, `warnings`, and `stopReason`. Optimizer-specific task and result types extend these interfaces.
 
 **`maxCostUsd`** and **`maxWallClockMs`** are checked between evaluations, so a run overruns by whatever it had in flight when the ceiling was reached — one evaluation at the default concurrency, and up to `concurrency` of them above it. Neither bounds the held-out sweep, which runs once the search has already stopped: it is reported apart from the search as `testMetricCalls` and `testUsage`, and has to be budgeted for separately. Neither follows from `maxMetricCalls`: reflective search grows the text it optimizes, so late rollouts cost more than early ones, and a run behind a rate limit spends almost nothing while taking as long as the provider makes it take.
 
@@ -107,6 +109,10 @@ evaluate(args: EvaluateArgs<Datum, K>): Promise<EvaluationBatch<Trajectory, Outp
 
 **`testSet`** is excluded from search and evaluated once against the winner. Because candidates are selected on `validationSet`, `bestScore` may be fitted to it. `testScore` measures held-out performance. Test rollouts are reported as `testMetricCalls` costing `testUsage`, outside `maxMetricCalls` and `maxCostUsd` both.
 
+**`validationSet`** defaults to `trainingSet`, and a run that took that default reports a `validationSetReusesTraining` warning: the search selected candidates on the instances reflection read. Pass `validationSet: "reuseTraining"` to accept it by name and silence the warning.
+
+**`warnings`** is a `RunWarning[]` of what a run cannot say about itself from its own numbers — selection that reused the training instances, a seed the metric scored identically on every instance. Never fatal, always present (empty when there is nothing to say), and repeated on the `finish` event so a reporter records them next to the score.
+
 **`TextModel`** is the provider-independent interface `({ prompt, signal }) => Promise<string>`.
 
 ### Values
@@ -121,7 +127,7 @@ For Redis, SQLite, or file-backed caching, implement **`EvaluationCache`** with
 
 **`createEvaluator({ adapter, budget, cache, cacheNamespace, retry, trackOutputs, onEvaluation, signal, cacheHits, usage })`** handles adapter calls, caching, budget accounting, transient scores, and evaluation events. `cacheHits` and `usage` seed the counters from a checkpoint, so a resumed run reports totals rather than deltas. `usage()` covers the charged rollouts a ceiling is checked against; `unchargedUsage()` covers what `charge: false` bought. `evaluate` returns a `ScoredBatch`. `evaluateTraced` returns an `EvaluationBatch`, or `null` when the remaining budget cannot cover the batch. A batch that exceeds the charged budget throws `BudgetExhausted`. All included optimizers use this evaluator.
 
-**`harvestRollouts({ adapter, candidate, data, minScore, maxRollouts, batchSize, maxMetricCalls, maxCostUsd, rng, signal })`** runs a candidate over `data` and returns the `Rollout`s the metric rewarded, alongside `metricCalls` and `attempted`. Omit `minScore` to keep any rollout scoring above zero; omit `maxRollouts` to sweep the whole pool. It carries its own budget and does not use the score cache, because it needs the outputs a cache hit cannot return. `maxCostUsd` bounds its dollars, checked between batches — a caller bounding spend cannot bound this pass from outside, since it runs on its own evaluator. Sweeping a validation set is the mistake to avoid — see [Distilling a run](../../docs/distillation.md).
+**`harvestRollouts({ adapter, candidate, data, minScore, maxRollouts, batchSize, maxMetricCalls, maxCostUsd, rng, signal })`** runs a candidate over `data` and returns the `Rollout`s the metric rewarded, alongside `metricCalls` and `attempted`. Omit `minScore` to keep any rollout scoring above zero; omit `maxRollouts` to sweep the whole pool. It carries its own budget and does not use the score cache, because it needs the outputs a cache hit cannot return. `maxCostUsd` bounds its dollars, checked between batches — a caller bounding spend cannot bound this pass from outside, since it runs on its own evaluator. Sweeping a validation set is the mistake to avoid — see [Distilling a run](./docs/distillation.md).
 
 **`toTrainingJsonl({ rollouts, render })`** serializes harvested rollouts as one chat-messages example per line. `render` turns a rollout into `{ messages }` or returns `null` to skip it, and decides how much of the optimized candidate stays in the training input. Returns the text; writing it is the caller's job.
 
@@ -131,7 +137,11 @@ For Redis, SQLite, or file-backed caching, implement **`EvaluationCache`** with
 
 **`parseProposedText(text)`** extracts a proposal from a reflection response, including responses with fenced blocks or surrounding commentary.
 
-**`createJudge({ model, criteria, scale = 5, renderInput, renderOutput, buildPrompt })`** returns a `Judge<Datum, Output>`: `({ input, output, expected, signal }) => Promise<ScoreResult>`. Each `JudgeCriterion` is graded on a small integer scale and normalized; per-criterion values are returned as `objectiveScores` and the aggregate `score` is their mean. A criterion the judge failed to grade returns a transient score rather than a zero, so the instance is retried instead of recorded as a failure. **`buildJudgePrompt`** is the default template and implements `JudgePromptBuilder`.
+**`createJudge({ model, criteria, scale = 5, renderInput, renderOutput, buildPrompt })`** returns a `Judge<Datum, Output>`: `({ input, output, expected, signal }) => Promise<ScoreResult>`. Each `JudgeCriterion` is graded on a small integer scale and normalized; per-criterion values are returned as `objectiveScores` and the aggregate `score` is their weighted mean. A criterion the judge failed to grade returns a transient score rather than a zero, so the instance is retried instead of recorded as a failure. **`buildJudgePrompt`** is the default template and implements `JudgePromptBuilder`.
+
+A `JudgeCriterion` takes `weight` (default 1; 0 reports the criterion as an objective without letting it move the score) and `gate`. **`gate`** is the grade, on the judge's own scale, that a criterion must reach for the instance to score at all — below it the instance scores 0 whatever the rest said, and the per-criterion objectives still report which one failed it. A mean lets a search trade a hard requirement away: a candidate that tanks one non-negotiable criterion and aces three cosmetic ones outranks the incumbent that kept the rule. Anything you would not ship without is a gate rather than a term in the average.
+
+When `expected` is passed, the default prompt forbids the feedback from restating it. Feedback is rewritten into a reusable instruction, so "the instruction never says to state the thirty-day refund window" is addressed to the instructions as asked and copies the answer key into the prompt.
 
 **`compare({ entrants, seeds, concurrency = 1 })`** runs each entrant over every seed and returns a `Comparison` of `winner`, `summaries`, and `runs`. Entrants are `({ seed }) => Promise<OptimizerResult>`, so the caller builds the optimizer-specific task. Ranking is on `testScore` where a run reports one, because the validation score is the number the search selected against. Each `ComparisonSummary` carries `meanScore`, `sdScore`, `minScore`, `maxScore`, `meanMetricCalls`, `meanCostUsd`, and a paired sign-flip `pValueVsWinner`.
 
@@ -222,11 +232,14 @@ TypeScript infers component names and the datum type from `seedCandidate` and `t
 Extends `Adapter` with what reflection needs:
 
 ```ts
+evaluate(args: EvaluateArgs<Datum, K>): ReflectiveBatch<Trajectory, Output>
 makeReflectiveDataset(args: MakeReflectiveDatasetArgs<Datum, Trajectory, Output, K>): ReflectiveDataset<K>
 proposeNewTexts?(args: ProposeArgs<K>): ComponentPatch<K>   // optional
 ```
 
-Both methods may be synchronous or asynchronous. `ReflectiveDataset` is a partial map from component names to `ReflectiveRecord[]`. Each record contains `inputs`, `generatedOutputs`, `feedback`, `score`, and a typed `evidence` field. Adapters only need to return records for the requested components.
+A `ReflectiveBatch` is an `EvaluationBatch` with `feedback` required. It is optional on the shared type because the searches that never reflect have no use for it; here it is the input to the whole method. An adapter that returns scores and no prose reduces every rollout to a number, and reflection then rewrites the instruction from a prompt whose feedback blocks are empty — a run that spends its whole budget, reports a normal-looking `stopReason`, and has been doing blind search. Nothing in the result distinguishes that from a hard task, so it is a type error instead.
+
+All three methods may be synchronous or asynchronous. `ReflectiveDataset` is a partial map from component names to `ReflectiveRecord[]`. Each record contains `inputs`, `generatedOutputs`, `feedback`, `score`, and a typed `evidence` field. Adapters only need to return records for the requested components.
 
 When `proposeNewTexts` is implemented, the adapter generates proposals without calling `reflect`. The task type still requires `reflect`, so offline runs can pass a stub.
 
@@ -248,7 +261,7 @@ When `proposeNewTexts` is implemented, the adapter generates proposals without c
 
 Each export is a factory. Selector and acceptance interfaces accept custom functions. A `ValEvaluationPolicy` is an object with `selectInstances` and `bestCandidate` methods.
 
-`pairedPermutationAcceptance` and `lowerBoundEvaluationPolicy` exist for metrics whose readings vary between runs of the same text. Both are strictly more conservative than the defaults. In the twenty-seed benchmark the pair scores 0.931 against plain GEPA's 0.920 on the noisy task and 0.945 against 0.947 on the noiseless one, neither gap significant over those seeds — conservative enough to be worth having on a metric that varies, cheap enough not to hurt on one that does not. A sign-flip test also needs a wide enough minibatch to say anything — over three instances the smallest p-value it can produce is 0.125, so at the default `minibatchSize` no proposal clears an `alpha` below that.
+`pairedPermutationAcceptance` and `lowerBoundEvaluationPolicy` exist for metrics whose readings vary between runs of the same text. Both are strictly more conservative than the defaults. In the twenty-seed benchmark the pair scores 0.931 against plain GEPA's 0.920 on the noisy task and 0.945 against 0.947 on the noiseless one, neither gap significant over those seeds — conservative enough to be worth having on a metric that varies, cheap enough not to hurt on one that does not. A sign-flip test also needs a wide enough minibatch to say anything — over three instances the smallest p-value it can produce is 0.125, so at the default `minibatchSize` no proposal clears an `alpha` below that. `GepaOptimizer` refuses that combination at construction rather than letting the run discover it: an acceptance policy reports the smallest batch it could ever accept on as `minimumPairs`, and a `minibatchSize` below it throws.
 
 ### Reflection prompts
 
@@ -496,13 +509,18 @@ The paraphrase prompt receives no score or feedback. Compare random search with
 ```ts
 import { createFileCache } from "textopt/file-cache";
 
-const cache = createFileCache({ path: ".textopt/scores.jsonl" });
+const cache = createFileCache({
+  path: ".textopt/scores.jsonl",
+  namespace: "gpt-4o-mini@t0/scorer-v3",
+});
 ```
 
 An `EvaluationCache` that outlives the process, as an append-only JSONL log. A long run against a real provider is measured in hours and dollars, and an in-memory cache throws all of it away when the run ends.
 
 Append-only rather than rewritten: a score is never invalidated, because the key names the candidate, the instance, and the environment — and a log survives a process killed mid-write, which a file rewritten in place does not. A record that does not parse is dropped rather than fatal, since the last line of an interrupted log is routinely half-written. Later records win, so a re-measured instance replaces its earlier reading.
 
+`namespace` is required and scopes every key to the system the scores measure — model id, decoding settings, scorer version. It is not optional the way `cacheNamespace` is, because this log outlives every part of that system a key does not name: an alias the provider upgraded under you, an edited scorer. Entries written under one namespace are never served to a run under another. Change it whenever anything outside the candidate text changes.
+
 `maxEntries` (default 1,000,000) bounds what is held in memory; the file itself is never trimmed. `entries()` is deliberately absent — it exists so a checkpoint can carry scores that would otherwise be lost, and these are already on disk.
 
 This is the only entry point that imports `node:fs`.

package/dist/bootstrap-search/index.cjs

@@ -1,6 +1,6 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
-const require_reporting = require("../reporting-CNHzbJC-.cjs");
-const require_demos = require("../demos-D2o0qSSo.cjs");
+const require_warnings = require("../warnings-CWRJF-jA.cjs");
+const require_demos = require("../demos-9v5ts7F3.cjs");
 const require_rng = require("../rng-DbA_rPIo.cjs");
 //#region src/bootstrap-search/optimize.ts
 const DEFAULT_CANDIDATES = 16;
@@ -44,17 +44,21 @@ var BootstrapSearchOptimizer = class {
 				task
 			});
 		} finally {
-			await require_reporting.flushReporters(task.reporters ?? []);
+			await require_warnings.flushReporters(task.reporters ?? []);
 		}
 	}
 };
 async function run(args) {
 	const { config, task } = args;
 	const { candidates: shuffledHarvests = DEFAULT_CANDIDATES, maxDemos = DEFAULT_MAX_DEMOS, minDemos = DEFAULT_MIN_DEMOS, maxLabeledDemos = DEFAULT_MAX_LABELED_DEMOS, demoMinScore, stopAtScore, concurrency = 1, seed = 0, trackBestOutputs = false, checkpointCache = true } = config;
-	const { seedCandidate, trainingSet, validationSet = trainingSet, testSet, adapter, demoComponents, renderDemo, goldOutput, maxMetricCalls, cache, cacheNamespace, retry, maxCostUsd, maxWallClockMs, instanceId = require_reporting.defaultInstanceId, reporters = [], onCheckpoint, resumeFrom, signal } = task;
-	const emit = require_reporting.createEmitter(reporters);
-	const deadline = require_reporting.createDeadline({ maxWallClockMs });
-	const components = require_reporting.componentNames(seedCandidate);
+	const { seedCandidate, trainingSet, validationSet: requestedValidationSet, testSet, adapter, demoComponents, renderDemo, goldOutput, maxMetricCalls, cache, cacheNamespace, retry, maxCostUsd, maxWallClockMs, instanceId = require_warnings.defaultInstanceId, reporters = [], onCheckpoint, resumeFrom, signal } = task;
+	const { validationSet, warnings } = require_warnings.resolveValidationSet({
+		validationSet: requestedValidationSet,
+		trainingSet
+	});
+	const emit = require_warnings.createEmitter(reporters);
+	const deadline = require_warnings.createDeadline({ maxWallClockMs });
+	const components = require_warnings.componentNames(seedCandidate);
 	if (trainingSet.length === 0) throw new Error("optimize requires a non-empty trainingSet");
 	if (validationSet.length === 0) throw new Error("optimize requires a non-empty validationSet");
 	if (demoComponents.length === 0) throw new Error("optimize requires at least one demoComponent: this search has nothing to put demonstrations in otherwise");
@@ -67,7 +71,7 @@ async function run(args) {
 		datum,
 		index
 	})) ?? [];
-	const fingerprint = require_reporting.runFingerprint({
+	const fingerprint = require_warnings.runFingerprint({
 		seedCandidate,
 		trainingIds: trainingSet.map((datum, index) => instanceId({
 			datum,
@@ -77,17 +81,17 @@ async function run(args) {
 		seed,
 		...cacheNamespace === void 0 ? {} : { cacheNamespace }
 	});
-	require_reporting.assertResumable({
+	require_warnings.assertResumable({
 		fingerprint,
 		...resumeFrom === void 0 ? {} : { snapshot: resumeFrom }
 	});
 	const rng = require_rng.createSeededRng(seed, resumeFrom?.rngState);
-	const budget = require_reporting.createBudget({
+	const budget = require_warnings.createBudget({
 		maxMetricCalls,
 		spent: resumeFrom?.metricCalls ?? 0
 	});
-	const evaluationCache = cache === false ? void 0 : cache ?? require_reporting.createMemoryCache();
-	const evaluator = require_reporting.createEvaluator({
+	const evaluationCache = cache === false ? void 0 : cache ?? require_warnings.createMemoryCache();
+	const evaluator = require_warnings.createEvaluator({
 		adapter,
 		budget,
 		...retry === void 0 ? {} : { retry },
@@ -124,7 +128,7 @@ async function run(args) {
 		});
 	}
 	const seedEvaluation = resumeFrom === void 0 ? await sweep(seedCandidate, "seed") : void 0;
-	const seedScore = seedEvaluation === void 0 ? resumeFrom.seedScore : require_reporting.requireMeasuredMean({
+	const seedScore = seedEvaluation === void 0 ? resumeFrom.seedScore : require_warnings.requireMeasuredMean({
 		batch: seedEvaluation,
 		phase: "seed"
 	});
@@ -139,7 +143,7 @@ async function run(args) {
 		candidateId: 0,
 		candidate: seedCandidate,
 		aggregateScore: seedScore,
-		instanceScores: require_reporting.instanceRow(seedEvaluation),
+		instanceScores: require_warnings.instanceRow(seedEvaluation),
 		...trackBestOutputs ? { outputs: seedEvaluation.outputs } : {}
 	});
 	function takeSnapshot() {
@@ -175,7 +179,7 @@ async function run(args) {
 			stopReason = "aborted";
 			break;
 		}
-		if (require_reporting.costExhausted({
+		if (require_warnings.costExhausted({
 			usage: evaluator.usage(),
 			maxCostUsd
 		})) {
@@ -197,7 +201,7 @@ async function run(args) {
 			const source = plan[drawn + wave.length];
 			const block = await buildBlock(source);
 			const candidate = withDemos(block);
-			const key = require_reporting.candidateHash(candidate);
+			const key = require_warnings.candidateHash(candidate);
 			const prior = inFlight.get(key);
 			const sweeping = prior === void 0 ? settled(sweep(candidate, "validation")) : prior.then(() => settled(sweep(candidate, "validation")));
 			inFlight.set(key, sweeping);
@@ -216,7 +220,7 @@ async function run(args) {
 			drawn += 1;
 			if (outcome.failed === true) {
 				draining = true;
-				if (outcome.err instanceof require_reporting.BudgetExhausted) {
+				if (outcome.err instanceof require_warnings.BudgetExhausted) {
 					waveStop = "budgetExhausted";
 					continue;
 				}
@@ -228,7 +232,7 @@ async function run(args) {
 				continue;
 			}
 			const evaluation = outcome.evaluation;
-			const score = require_reporting.measuredMean(evaluation);
+			const score = require_warnings.measuredMean(evaluation);
 			if (score === void 0) continue;
 			const accepted = score > bestScore;
 			evaluated.push({
@@ -254,7 +258,7 @@ async function run(args) {
 					candidateId: acceptedCandidates,
 					candidate: entry.candidate,
 					aggregateScore: score,
-					instanceScores: require_reporting.instanceRow(evaluation),
+					instanceScores: require_warnings.instanceRow(evaluation),
 					...trackBestOutputs ? { outputs: evaluation.outputs } : {}
 				});
 				best = entry.candidate;
@@ -284,15 +288,16 @@ async function run(args) {
 		iteration: evaluated.length,
 		charge: false
 	});
-	const testScore = heldOut === void 0 ? void 0 : require_reporting.measuredMean(heldOut);
+	const testScore = heldOut === void 0 ? void 0 : require_warnings.measuredMean(heldOut);
 	emit({
 		type: "finish",
 		reason: stopReason,
+		warnings,
 		bestCandidateId: acceptedCandidates,
 		bestScore,
 		metricCalls: budget.spent(),
 		...testScore === void 0 ? {} : { testScore },
-		...heldOut === void 0 ? {} : { testInstanceScores: require_reporting.instanceRow(heldOut) },
+		...heldOut === void 0 ? {} : { testInstanceScores: require_warnings.instanceRow(heldOut) },
 		...heldOut === void 0 || !trackBestOutputs ? {} : { testOutputs: heldOut.outputs }
 	});
 	return {
@@ -311,6 +316,7 @@ async function run(args) {
 			testMetricCalls: evaluator.unchargedCalls(),
 			testUsage: evaluator.unchargedUsage()
 		},
+		warnings,
 		stopReason
 	};
 	function withDemos(block) {
@@ -384,5 +390,5 @@ function countDemos(block) {
 }
 //#endregion
 exports.BootstrapSearchOptimizer = BootstrapSearchOptimizer;
-exports.isCandidateAccepted = require_reporting.isCandidateAccepted;
-exports.isRunFinished = require_reporting.isRunFinished;
+exports.isCandidateAccepted = require_warnings.isCandidateAccepted;
+exports.isRunFinished = require_warnings.isRunFinished;

package/dist/bootstrap-search/index.d.cts

@@ -1,7 +1,7 @@
-import { a as RunFinished, c as Adapter, i as Reporter, l as Candidate, n as OptimizerEvent, o as isCandidateAccepted, r as ReportableEvent, s as isRunFinished, t as CandidateAccepted, v as UsageTotals } from "../reporting-DQbAohc9.cjs";
+import { S as UsageTotals, a as RunFinished, f as Adapter, i as Reporter, n as OptimizerEvent, o as isCandidateAccepted, p as Candidate, r as ReportableEvent, s as isRunFinished, t as CandidateAccepted } from "../reporting-bq007_2z.cjs";
 import { n as EvaluationCache, t as CachedScore } from "../cache-CuSo0NJ8.cjs";
-import { r as DemoRenderer } from "../demos-CU9dy2oT.cjs";
-import { n as OptimizerResult, r as OptimizerTask, s as EvaluationEvent, t as Optimizer } from "../optimizer-Bh5DPRMH.cjs";
+import { r as DemoRenderer } from "../demos-ByaLZy-Z.cjs";
+import { n as OptimizerResult, r as OptimizerTask, s as EvaluationEvent, t as Optimizer } from "../optimizer-4Zv-Zt2t.cjs";
 //#region src/bootstrap-search/optimize.d.ts
 /**
  * Where a candidate's demo block came from. `zeroShot` holds no demos at all,

package/dist/bootstrap-search/index.d.mts

@@ -1,7 +1,7 @@
-import { a as RunFinished, c as Adapter, i as Reporter, l as Candidate, n as OptimizerEvent, o as isCandidateAccepted, r as ReportableEvent, s as isRunFinished, t as CandidateAccepted, v as UsageTotals } from "../reporting-DQbAohc9.mjs";
+import { S as UsageTotals, a as RunFinished, f as Adapter, i as Reporter, n as OptimizerEvent, o as isCandidateAccepted, p as Candidate, r as ReportableEvent, s as isRunFinished, t as CandidateAccepted } from "../reporting-bq007_2z.mjs";
 import { n as EvaluationCache, t as CachedScore } from "../cache-CuSo0NJ8.mjs";
-import { r as DemoRenderer } from "../demos-1b7JiUue.mjs";
-import { n as OptimizerResult, r as OptimizerTask, s as EvaluationEvent, t as Optimizer } from "../optimizer-Ck6-e_8o.mjs";
+import { r as DemoRenderer } from "../demos-ASsSXYXA.mjs";
+import { n as OptimizerResult, r as OptimizerTask, s as EvaluationEvent, t as Optimizer } from "../optimizer-Ds5mzYjz.mjs";
 //#region src/bootstrap-search/optimize.d.ts
 /**
  * Where a candidate's demo block came from. `zeroShot` holds no demos at all,

package/dist/bootstrap-search/index.mjs

@@ -1,5 +1,5 @@
-import { _ as createMemoryCache, a as isRunFinished, b as createBudget, c as createEvaluator, d as createDeadline, f as assertResumable, g as candidateHash, h as componentNames, i as isCandidateAccepted, l as measuredMean, m as runFingerprint, n as flushReporters, o as BudgetExhausted, r as instanceRow, s as costExhausted, t as createEmitter, u as requireMeasuredMean, v as defaultInstanceId } from "../reporting-DY-DC4HG.mjs";
-import { n as harvestFewShotExamples, t as formatDemos } from "../demos-DE2oxNWX.mjs";
+import { S as createBudget, _ as componentNames, a as instanceRow, b as defaultInstanceId, c as BudgetExhausted, d as measuredMean, f as requireMeasuredMean, g as runFingerprint, i as flushReporters, l as costExhausted, m as assertResumable, o as isCandidateAccepted, p as createDeadline, r as createEmitter, s as isRunFinished, t as resolveValidationSet, u as createEvaluator, v as candidateHash, y as createMemoryCache } from "../warnings-OxvDi9kN.mjs";
+import { n as harvestFewShotExamples, t as formatDemos } from "../demos-Brobjfuc.mjs";
 import { t as createSeededRng } from "../rng-Dtc5eZ_W.mjs";
 //#region src/bootstrap-search/optimize.ts
 const DEFAULT_CANDIDATES = 16;
@@ -50,7 +50,11 @@ var BootstrapSearchOptimizer = class {
 async function run(args) {
 	const { config, task } = args;
 	const { candidates: shuffledHarvests = DEFAULT_CANDIDATES, maxDemos = DEFAULT_MAX_DEMOS, minDemos = DEFAULT_MIN_DEMOS, maxLabeledDemos = DEFAULT_MAX_LABELED_DEMOS, demoMinScore, stopAtScore, concurrency = 1, seed = 0, trackBestOutputs = false, checkpointCache = true } = config;
-	const { seedCandidate, trainingSet, validationSet = trainingSet, testSet, adapter, demoComponents, renderDemo, goldOutput, maxMetricCalls, cache, cacheNamespace, retry, maxCostUsd, maxWallClockMs, instanceId = defaultInstanceId, reporters = [], onCheckpoint, resumeFrom, signal } = task;
+	const { seedCandidate, trainingSet, validationSet: requestedValidationSet, testSet, adapter, demoComponents, renderDemo, goldOutput, maxMetricCalls, cache, cacheNamespace, retry, maxCostUsd, maxWallClockMs, instanceId = defaultInstanceId, reporters = [], onCheckpoint, resumeFrom, signal } = task;
+	const { validationSet, warnings } = resolveValidationSet({
+		validationSet: requestedValidationSet,
+		trainingSet
+	});
 	const emit = createEmitter(reporters);
 	const deadline = createDeadline({ maxWallClockMs });
 	const components = componentNames(seedCandidate);
@@ -287,6 +291,7 @@ async function run(args) {
 	emit({
 		type: "finish",
 		reason: stopReason,
+		warnings,
 		bestCandidateId: acceptedCandidates,
 		bestScore,
 		metricCalls: budget.spent(),
@@ -310,6 +315,7 @@ async function run(args) {
 			testMetricCalls: evaluator.unchargedCalls(),
 			testUsage: evaluator.unchargedUsage()
 		},
+		warnings,
 		stopReason
 	};
 	function withDemos(block) {

package/dist/{demos-D2o0qSSo.cjs → demos-9v5ts7F3.cjs}

@@ -1,4 +1,4 @@
-const require_reporting = require("./reporting-CNHzbJC-.cjs");
+const require_warnings = require("./warnings-CWRJF-jA.cjs");
 //#region src/harvest.ts
 /**
 * Run a candidate over data and keep the rollouts the metric rewarded.
@@ -17,8 +17,8 @@ const require_reporting = require("./reporting-CNHzbJC-.cjs");
 async function harvestRollouts(args) {
 	const { adapter, candidate, data, minScore, maxRollouts = Number.POSITIVE_INFINITY, batchSize = Math.min(maxRollouts, data.length), maxMetricCalls = data.length, maxCostUsd, rng, signal } = args;
 	if (data.length === 0) throw new Error("harvestRollouts requires non-empty data");
-	const budget = require_reporting.createBudget({ maxMetricCalls });
-	const evaluator = require_reporting.createEvaluator({
+	const budget = require_warnings.createBudget({ maxMetricCalls });
+	const evaluator = require_warnings.createEvaluator({
 		adapter,
 		budget,
 		...signal === void 0 ? {} : { signal }
@@ -28,7 +28,7 @@ async function harvestRollouts(args) {
 	let attempted = 0;
 	for (let start = 0; start < order.length; start += batchSize) {
 		if (rollouts.length >= maxRollouts || signal?.aborted) break;
-		if (require_reporting.costExhausted({
+		if (require_warnings.costExhausted({
 			usage: evaluator.usage(),
 			maxCostUsd
 		})) break;

package/dist/{demos-1b7JiUue.d.mts → demos-ASsSXYXA.d.mts}

@@ -1,4 +1,4 @@
-import { c as Adapter, l as Candidate, v as UsageTotals } from "./reporting-DQbAohc9.mjs";
+import { S as UsageTotals, f as Adapter, p as Candidate } from "./reporting-bq007_2z.mjs";
 import { t as Rng } from "./rng-BR5MOedA.mjs";
 //#region src/demos.d.ts
 /**

package/dist/{demos-DE2oxNWX.mjs → demos-Brobjfuc.mjs}

@@ -1,4 +1,4 @@
-import { b as createBudget, c as createEvaluator, s as costExhausted } from "./reporting-DY-DC4HG.mjs";
+import { S as createBudget, l as costExhausted, u as createEvaluator } from "./warnings-OxvDi9kN.mjs";
 //#region src/harvest.ts
 /**
 * Run a candidate over data and keep the rollouts the metric rewarded.

package/dist/{demos-CU9dy2oT.d.cts → demos-ByaLZy-Z.d.cts}

@@ -1,4 +1,4 @@
-import { c as Adapter, l as Candidate, v as UsageTotals } from "./reporting-DQbAohc9.cjs";
+import { S as UsageTotals, f as Adapter, p as Candidate } from "./reporting-bq007_2z.cjs";
 import { t as Rng } from "./rng-BR5MOedA.cjs";
 //#region src/demos.d.ts
 /**

package/dist/file-cache.cjs

@@ -13,22 +13,33 @@ let node_path = require("node:path");
 * Append-only rather than rewritten: a score is never invalidated (the key
 * names the candidate, the instance, and the environment), and a log survives
 * a process killed mid-write, which a file rewritten in place does not.
+*
+* `namespace` is what makes that invariant true. A cached score measures a
+* whole system, not a candidate, and this log outlives every part of that
+* system a run does not pass through the key: the model id behind an alias the
+* provider upgraded, the decoding settings, the scorer's own version. It is
+* required rather than optional because the failure it prevents is silent —
+* scores from one system served to a run of another, with a normal-looking
+* result and no way to read afterwards that it happened.
 */
 function createFileCache(args) {
-	const { path, maxEntries = 1e6 } = args;
+	const { path, namespace, maxEntries = 1e6 } = args;
+	if (namespace.trim() === "") throw new Error("createFileCache requires a non-empty namespace naming the system these scores measure");
 	(0, node_fs.mkdirSync)((0, node_path.dirname)(path), { recursive: true });
 	const log = readLog(path);
 	const entries = log.entries;
+	const scope = (key) => `${namespace}\u0000${key}`;
 	if (log.unterminated) (0, node_fs.appendFileSync)(path, "\n");
 	return {
-		get: (key) => entries.get(key),
+		get: (key) => entries.get(scope(key)),
 		set: (key, cached) => {
-			if (entries.size >= maxEntries && !entries.has(key)) {
+			const scoped = scope(key);
+			if (entries.size >= maxEntries && !entries.has(scoped)) {
 				const oldest = entries.keys().next();
 				if (!oldest.done) entries.delete(oldest.value);
 			}
-			entries.set(key, cached);
-			(0, node_fs.appendFileSync)(path, `${JSON.stringify([key, cached])}\n`);
+			entries.set(scoped, cached);
+			(0, node_fs.appendFileSync)(path, `${JSON.stringify([scoped, cached])}\n`);
 		}
 	};
 }

package/dist/file-cache.d.cts

@@ -11,9 +11,22 @@ import { n as EvaluationCache } from "./cache-CuSo0NJ8.cjs";
  * Append-only rather than rewritten: a score is never invalidated (the key
  * names the candidate, the instance, and the environment), and a log survives
  * a process killed mid-write, which a file rewritten in place does not.
+ *
+ * `namespace` is what makes that invariant true. A cached score measures a
+ * whole system, not a candidate, and this log outlives every part of that
+ * system a run does not pass through the key: the model id behind an alias the
+ * provider upgraded, the decoding settings, the scorer's own version. It is
+ * required rather than optional because the failure it prevents is silent —
+ * scores from one system served to a run of another, with a normal-looking
+ * result and no way to read afterwards that it happened.
  */
 declare function createFileCache(args: {
   path: string;
+  /**
+   * Names the system these scores measure — model id, decoding settings,
+   * scorer version. Change it whenever any of them changes.
+   */
+  namespace: string;
   /** Entries kept in memory. The file itself is never trimmed. */
   maxEntries?: number;
 }): EvaluationCache;

package/dist/file-cache.d.mts

@@ -11,9 +11,22 @@ import { n as EvaluationCache } from "./cache-CuSo0NJ8.mjs";
  * Append-only rather than rewritten: a score is never invalidated (the key
  * names the candidate, the instance, and the environment), and a log survives
  * a process killed mid-write, which a file rewritten in place does not.
+ *
+ * `namespace` is what makes that invariant true. A cached score measures a
+ * whole system, not a candidate, and this log outlives every part of that
+ * system a run does not pass through the key: the model id behind an alias the
+ * provider upgraded, the decoding settings, the scorer's own version. It is
+ * required rather than optional because the failure it prevents is silent —
+ * scores from one system served to a run of another, with a normal-looking
+ * result and no way to read afterwards that it happened.
  */
 declare function createFileCache(args: {
   path: string;
+  /**
+   * Names the system these scores measure — model id, decoding settings,
+   * scorer version. Change it whenever any of them changes.
+   */
+  namespace: string;
   /** Entries kept in memory. The file itself is never trimmed. */
   maxEntries?: number;
 }): EvaluationCache;

package/dist/file-cache.mjs

@@ -12,22 +12,33 @@ import { dirname } from "node:path";
 * Append-only rather than rewritten: a score is never invalidated (the key
 * names the candidate, the instance, and the environment), and a log survives
 * a process killed mid-write, which a file rewritten in place does not.
+*
+* `namespace` is what makes that invariant true. A cached score measures a
+* whole system, not a candidate, and this log outlives every part of that
+* system a run does not pass through the key: the model id behind an alias the
+* provider upgraded, the decoding settings, the scorer's own version. It is
+* required rather than optional because the failure it prevents is silent —
+* scores from one system served to a run of another, with a normal-looking
+* result and no way to read afterwards that it happened.
 */
 function createFileCache(args) {
-	const { path, maxEntries = 1e6 } = args;
+	const { path, namespace, maxEntries = 1e6 } = args;
+	if (namespace.trim() === "") throw new Error("createFileCache requires a non-empty namespace naming the system these scores measure");
 	mkdirSync(dirname(path), { recursive: true });
 	const log = readLog(path);
 	const entries = log.entries;
+	const scope = (key) => `${namespace}\u0000${key}`;
 	if (log.unterminated) appendFileSync(path, "\n");
 	return {
-		get: (key) => entries.get(key),
+		get: (key) => entries.get(scope(key)),
 		set: (key, cached) => {
-			if (entries.size >= maxEntries && !entries.has(key)) {
+			const scoped = scope(key);
+			if (entries.size >= maxEntries && !entries.has(scoped)) {
 				const oldest = entries.keys().next();
 				if (!oldest.done) entries.delete(oldest.value);
 			}
-			entries.set(key, cached);
-			appendFileSync(path, `${JSON.stringify([key, cached])}\n`);
+			entries.set(scoped, cached);
+			appendFileSync(path, `${JSON.stringify([scoped, cached])}\n`);
 		}
 	};
 }