textopt 0.0.0 → 0.2.0

package/docs/adapters.md

@@ -0,0 +1,169 @@
+# Adapters and metrics
+
+An adapter is how an optimizer runs and scores your system. Everything that
+feeds a search — the framework it calls, the model that revises text, a
+model-graded metric, per-module attribution — hangs off it.
+
+## The adapter
+
+The adapter connects an optimizer to the system being evaluated:
+
+```ts
+interface GepaAdapter<Datum, Trajectory, Output, K extends string> {
+  evaluate(
+    args: EvaluateArgs<Datum, K>,
+  ): Promise<EvaluationBatch<Trajectory, Output>>;
+  makeReflectiveDataset(
+    args: MakeReflectiveDatasetArgs<Datum, Trajectory, Output, K>,
+  ): ReflectiveDataset<K>;
+  proposeNewTexts?(args: ProposeArgs<K>): ComponentPatch<K>; // replaces the reflection LLM entirely
+}
+```
+
+Methods may be synchronous or asynchronous; the example shows the asynchronous form.
+
+`evaluate` returns one score per instance and may include textual feedback. GEPA uses that feedback during reflection.
+
+- **`args.run`** identifies the rollout's `iteration`, `phase`, `split`, and `candidateId`. Forward it to your tracing system.
+- **`transient`** marks scores caused by infrastructure failures such as rate limits or 5xx responses. Transient scores are not cached.
+
+### Vercel AI SDK
+
+Beta, and not published to npm: use it from a checkout of this repository until its interface settles.
+
+```ts
+import { createAiSdkAdapter } from "@textopt/ai-sdk";
+import { generateText } from "ai";
+
+const adapter = createAiSdkAdapter<Ticket>({
+  run: ({ candidate, datum, signal }) =>
+    generateText({
+      model: taskModel,
+      system: candidate.system ?? "",
+      prompt: datum.text,
+      abortSignal: signal,
+    }),
+
+  score: ({ datum, output }) =>
+    output === datum.label
+      ? { score: 1, feedback: `Correct: ${datum.label}.` }
+      : {
+          score: 0,
+          feedback: `Predicted "${output}" but the correct queue is "${datum.label}". ${datum.why}`,
+        },
+
+  concurrency: 4,
+});
+```
+
+### LangChain
+
+```ts
+import { createLangChainAdapter } from "@textopt/langchain";
+
+const adapter = createLangChainAdapter<Ticket, string>({
+  buildRunnable: (candidate) => buildChain(candidate.system, candidate.rubric),
+  score: ({ datum, output, trace }) => ({
+    score: grade(datum, output),
+    feedback: explain(trace),
+  }),
+});
+```
+
+The adapter rebuilds the runnable for each candidate. Traces include LLM, tool, and retriever spans; set `includeChainSteps` to include chain spans. Each rollout also includes `textopt_iteration`, `textopt_phase`, `textopt_split`, and `textopt_candidate_id` metadata for filtering in LangSmith.
+
+### Braintrust
+
+Beta, and not published to npm: use it from a checkout of this repository until its interface settles.
+
+```ts
+import {
+  createBraintrustScorer,
+  withBraintrustLogging,
+} from "@textopt/braintrust";
+import { ExactMatch, Levenshtein } from "autoevals";
+
+const score = createBraintrustScorer<string>({
+  scorers: [ExactMatch, Levenshtein],
+  weights: { ExactMatch: 3 },
+});
+
+const adapter = withBraintrustLogging({
+  adapter: baseAdapter,
+  logger: initLogger({ projectName: "ticket-routing" }),
+});
+```
+
+The scorer maps scorer rationales to feedback and individual scores to `objectiveScores`. The logging decorator works with the AI SDK adapter, the LangChain adapter, or a custom adapter.
+
+## The reflection model
+
+`reflect` implements the provider-independent `TextModel` interface: `({ prompt, signal }) => Promise<string>`.
+
+```ts
+import type { TextModel } from "textopt";
+import { generateText } from "ai";
+
+const reflect: TextModel = async ({ prompt, signal }) => {
+  const result = await generateText({ model, prompt, abortSignal: signal });
+  return result.text;
+};
+```
+
+A LangChain chat model, vendor SDK call, local model, or deterministic function can implement `TextModel`. If the adapter implements `proposeNewTexts`, it generates proposals without calling `reflect`; the type still requires `reflect`, so pass a stub as shown in the `pareto` example.
+
+The model under optimization is usually cheaper than the reflection model, which must analyze failures and revise the candidate.
+
+## Judging with a model
+
+When the metric cannot be written as a string match, `createJudge` builds one from a model and returns written feedback alongside the score, which is what reflective search actually runs on:
+
+```ts
+import { createJudge } from "textopt";
+
+const judge = createJudge<Ticket, string>({
+  model: reflect,
+  scale: 5,
+  criteria: [
+    {
+      name: "accuracy",
+      description: "Every claim is supported by the ticket.",
+    },
+    { name: "tone", description: "Direct and free of filler." },
+  ],
+});
+
+const { score, feedback, objectiveScores } = await judge({
+  input: ticket,
+  output: answer,
+});
+```
+
+Each criterion is graded on a small integer scale and normalized afterwards, because models discriminate between 2 and 4 far more reliably than between 0.4 and 0.8. Per-criterion scores are returned as `objectiveScores`, so a Pareto frontier can be taken over them; the aggregate `score` is their mean.
+
+The prompt asks for feedback addressed to the _instructions_ rather than to the graded output. "The instruction never says to state the refund window" is something a rewriting model can act on; "this answer should have mentioned the refund window" is not. A criterion the judge failed to grade comes back as a transient score, so the instance is retried rather than recorded as a zero.
+
+## Multi-module pipelines
+
+When a system runs several modules in sequence and each has its own instruction, reflection is only as good as the evidence it sees — and the evidence a module needs is what _it_ received and produced, not the pipeline's input and final answer. `createPipelineAdapter` builds that attribution:
+
+```ts
+import { createPipelineAdapter } from "textopt/gepa";
+
+const adapter = createPipelineAdapter<Ticket, string, "planner" | "writer">({
+  modules: [
+    {
+      component: "planner",
+      run: ({ instruction, datum }) => plan(instruction, datum),
+    },
+    {
+      component: "writer",
+      run: ({ instruction, input }) => write(instruction, input),
+    },
+  ],
+  score: ({ datum, output, steps }) => judgeAnswer(datum, output, steps),
+  concurrency: 4,
+});
+```
+
+Each module's output is threaded into the next, and the reflective dataset gives each component only its own step. The feedback is end-to-end and every module sees the same string: a metric scores the final output, so nothing in a score alone says which module lost the point. `score` is handed the whole trace for callers who can attribute better.

package/docs/benchmark.md

@@ -0,0 +1,90 @@
+# Benchmark
+
+`pnpm bench` runs every optimizer over four offline tasks and twenty seeds, and writes [`bench/results/latest.json`](../../../bench/results/latest.json).
+
+Each task is a support-ticket policy. A ticket has four features — `tier`, `channel`, `issue` and `region` — and a hidden policy of nine rules says which actions a correct answer must take. The system under optimization reads rules out of the candidate and applies them to the ticket, so a candidate is scored on what it made the system do rather than on the words it contains. No rule keys on `region`: it is there to be mistaken for a reason, and a search that believes it pays for the belief.
+
+All seventy-two feature combinations are dealt into three disjoint splits of twenty-four. The splits are balanced, not merely disjoint — the same mix of every feature value in each — because every search here selects on the validation set and is reported on the test set, and two splits that ask for different things turn a held-out score into a lottery. No pair of features is in lockstep within a split either, so no rule can hide behind another. A candidate that fitted the combinations it was shown scores zero; one that found the rules scores the same on tickets it never saw.
+
+Held-out score, twenty seeds:
+
+| entrant             | `clean`   | `noisy`   | `interacting` | `demonstrated` |
+| ------------------- | --------- | --------- | ------------- | -------------- |
+| `gepa`              | **0.947** | 0.920     | **0.891**     | 0.894          |
+| `gepaVarianceAware` | 0.945     | **0.931** | 0.835         | **0.910**      |
+| `opro`              | 0.285     | 0.283     | 0.170         | 0.693          |
+| `simba`             | 0.249     | 0.282     | 0.185         | 0.657          |
+| `randomSearch`      | 0.266     | 0.268     | 0.111         | 0.646          |
+| `mipro`             | 0.094     | 0.140     | 0.085         | 0.649          |
+| `bootstrapSearch`   | 0.000     | 0.060     | 0.000         | 0.740          |
+
+Read these four rows first. They are what the table has to be judged against:
+
+| reference   | `clean` | `noisy` | `interacting` | `demonstrated` | what it is                                |
+| ----------- | ------- | ------- | ------------- | -------------- | ----------------------------------------- |
+| `policy`    | 1.000   | 0.945   | 1.000         | 1.000          | the rules being searched for; the ceiling |
+| `bestFixed` | 0.400   | 0.409   | 0.400         | 0.583          | the best answer that ignores the ticket   |
+| `shotgun`   | 0.358   | 0.378   | 0.358         | 0.358          | every action on every ticket              |
+| `zeroShot`  | 0.000   | 0.060   | 0.000         | 0.625          | the seed candidate, unoptimised           |
+
+`bestFixed` and `shotgun` are the floors a search has to beat to have found anything. Neither conditions on the ticket at all: `shotgun` sprays all nine actions and eats the bloat penalty for the ones that miss, and `bestFixed` is the best subset of actions to spray, chosen over the training and validation sets and never over the held-out set it is reported on — a floor selected on the number it is published at is an oracle, not a floor. On the three rule-shaped tasks the two GEPA rows are the only entrants that clear either floor. Every other search — `opro`, `simba`, `randomSearch`, `mipro`, `bootstrapSearch` — finishes below a candidate written without searching at all, having spent its whole budget to get there. On `demonstrated` all seven clear the floors, because the seed candidate already does.
+
+## What the diagnosis is worth
+
+Every entrant is run a second time with exactly one thing removed: the metric's per-instance diagnosis is stripped from its prompt, leaving it the blind draw a score-only search already gets. Its settings are the ones it was tuned at, and its proposals are capped at nine — the fewest any entrant makes when left alone — because blind a proposal is a draw from a fixed pool rather than an induction, and an entrant that buys more draws scores better without searching better.
+
+| entrant             | `clean`      | `noisy`      | `interacting` | `demonstrated` |
+| ------------------- | ------------ | ------------ | ------------- | -------------- |
+| `gepa`              | 0.185 −0.762 | 0.179 −0.741 | 0.089 −0.802  | 0.656 −0.238   |
+| `gepaVarianceAware` | 0.059 −0.885 | 0.121 −0.811 | 0.022 −0.813  | 0.631 −0.279   |
+| `opro`              | 0.097 −0.188 | 0.134 −0.149 | 0.065 −0.105  | 0.647 −0.045   |
+| `simba`             | 0.137 −0.112 | 0.169 −0.113 | 0.087 −0.097  | 0.659 +0.002   |
+| `mipro`             | 0.094 0.000  | 0.140 0.000  | 0.085 0.000   | 0.649 0.000    |
+| `randomSearch`      | 0.266 0.000  | 0.268 0.000  | 0.111 0.000   | 0.646 0.000    |
+| `bootstrapSearch`   | 0.000 0.000  | 0.060 0.000  | 0.000 0.000   | 0.740 0.000    |
+
+The blind score first, then what redaction cost against the table above.
+
+This is the largest effect in the benchmark, and it is not a fact about search. Blind at nine proposals apiece the entrants land in a band — 0.022 to 0.266 on the three rule-shaped tasks — and the winner of that band is `randomSearch`, which does not search reflectively at all. `gepa` blind scores 0.185 on `clean` against its own 0.947. Whatever separates these algorithms when they can read the diagnosis, almost none of it survives when they cannot.
+
+The bottom three rows lose exactly nothing, which is the check that the redaction is measuring what it claims to. `bootstrapSearch` calls no proposal model, and neither `mipro` nor `randomSearch` was ever shown a per-instance diagnosis to begin with — so redacting one changes nothing about what they were given, and their scores are identical to the digit. Those are also the only rows whose proposals the cap does not reach: `mipro` proposes its menu up front, seventeen entries on `interacting`, and `randomSearch` takes sixteen to twenty. The cap binds on every entrant whose score moves at all.
+
+What the middle rows show is that reading the diagnosis is necessary but not sufficient. `opro` and `simba` both get it and both lose real ground without it — 0.188 and 0.112 on `clean` — but that is a fifth of what `gepa` loses, because they were converting much less of it into rules in the first place. So the column separates three things a single table cannot: what an entrant is given, what it does with it, and what it would score on neither.
+
+Anyone choosing an optimizer from the table above should read this one alongside it. Most of what the top rows are buying is a proposal step that reads what the metric said about individual tickets. That is a real difference between these algorithms — what each one asks its model for is part of the algorithm — but it is a difference in the question they ask, not in how well they search the answers.
+
+## The tasks
+
+`clean` is the reference case: a noiseless metric with a clean gradient.
+
+`noisy` adds per-instance jitter to the same metric, which is why its ceiling is 0.945 rather than 1.000 — noise costs even a perfect candidate something. It is the one task where `gepaVarianceAware` wins, which is what its acceptance test is for.
+
+`interacting` splits the policy across two components in a pipeline: `triage` holds the rules about the issue, `response` those about the tier and channel, and the system applies `response` only to tickets `triage` already handled correctly. Every improvement to the second component scores nothing until the first is complete, so the task measures where a search spends. It has the widest spread between seeds of any task here (`gepa` sd 0.109), which is what a search that can be sent down the wrong component looks like, and it is the one task where the two GEPA rows genuinely separate.
+
+`demonstrated` changes the system rather than the metric — it answers a share of tickets correctly with no help from its prompt, which is the only condition under which harvesting rollouts has anything to harvest. It is also the task where the unoptimised seed already scores 0.625, and where the entrants that search instructions barely move off it: `mipro` 0.649, `randomSearch` 0.646. `bootstrapSearch` reaches 0.740 while calling no proposal model at all. When a system's failures are inconsistency rather than instruction, searching the instruction is the wrong tool, and this row is what that costs.
+
+It is the one task where `simba` harvests. SIMBA has two mutations — append a rewarded rollout as a demonstration, or ask a model what the better run did and append that as a rule — and which of the two it draws from is tuned like any other setting. Demonstrations are selected here and nowhere else, worth 0.657 against 0.628 for rules alone, and the three tasks above select rules, where a demonstration spends a draw on a rollout that carries nothing its prompt did not.
+
+## Two rows that are not what they look like
+
+`bootstrapSearch` scores 0.000 on `clean` and `interacting`, the only entrant with no seed spread at all on them. There is nothing to harvest there — the answer on those tasks is a pure function of the candidate, so a rollout tells the search nothing its prompt did not, and a demonstration search can only hand a candidate its own words back. It is run on all four tasks rather than only where it wins precisely so that row is visible.
+
+`mipro` scores 0.094 on `clean` and 0.085 on `interacting`. Its menu is proposed once, up front, from a dataset summary and a sample of task inputs — nine reflection calls on `clean`, not one of which carries a per-instance diagnosis. That is MIPROv2's grounded proposer working as designed, and it goes deeper than the stand-in: this benchmark's training data carries no gold labels at all, since what a correct answer requires lives in the metric rather than in the dataset. A real model in MIPRO's up-front slot would have nothing linking features to actions either. On this substrate `mipro` measures propose-before-evidence, and the blind column above puts a number on that: redacting the diagnosis costs it exactly nothing on all four tasks, because it was never shown one.
+
+## How the numbers are produced
+
+Every entrant gets the same `maxMetricCalls` on a task, the same data and the same proposal model. The held-out set is evaluated once, on the candidate the search already chose, outside the budget — so no optimizer can spend rollouts on the number it is reported at.
+
+Settings are tuned per entrant per task over comparable grids, on seeds 100–109, choosing on the validation score alone. The reported score comes from seeds 0–19. Nothing about how an entrant was run is fitted to the seeds or the split it is scored on; `tuning` in the JSON records what each one was given. An earlier version of this benchmark swept one entrant's hyperparameters against the published metric and left the others at a default, which is enough on its own to decide a table.
+
+`p` is a paired sign-flip test against the winner, Holm-adjusted across the entrants in the same comparison. It is withheld where every seed produced the same margin: a test over twenty identical differences reports a precision that twenty runs of one realization never earned. `distinctScores` says how many distinct outcomes an entrant actually had — `randomSearch` is 1 everywhere, since it takes no seed and is deterministic against a deterministic model.
+
+The two GEPA rows separate on one task out of four. Holm-adjusted, the winner's margin over the other runs p = 0.027 on `interacting`, and 0.334, 0.088 and 0.125 on `clean`, `noisy` and `demonstrated` — so only `interacting` clears 0.05, where plain GEPA wins. Every other margin in the table clears it at p < 0.0001. Read the top two rows as one result with two spellings everywhere except `interacting`.
+
+## What this does and does not tell you
+
+The proposal model is a deterministic stand-in. It induces a rule only when a feature value and a missing action co-occur across at least two tickets in its prompt, and otherwise draws from the hundred and eight rules the language can express, of which nine are correct.
+
+That stand-in has one property no real model has: it is exact counting, so it sharpens with the size of its reflection batch and stops being wrong at all past roughly a dozen observations. At that point tuning stops selecting a search and starts selecting whichever entrant asks for the biggest prompt. The grid is therefore bounded at a batch of nine, where the proposer is right about four times in five, and this bound is load-bearing — every minibatch-shaped entrant tunes straight to it. That is a modelling decision to keep the comparison about search, and it is the first thing to be suspicious of if these numbers ever look too clean.
+
+So these numbers are about search behaviour under a fixed, weak proposer. They are not about what any of these optimizers will do with a real model that can read a dataset and write a sensible instruction unprompted. Read the table as evidence about the searches, read the blind column, `bestFixed` and `shotgun` as the scale it should be read at, and then run [`compare()`](./evaluation.md#comparing-optimizers) on your own task and metric — which is the only measurement that answers the question you actually have.

package/docs/data-prep.md

@@ -0,0 +1,113 @@
+# Preparing the data a search runs on
+
+The data decides what a search can find. A metric can be perfect and a budget
+generous, and a run will still report a number that means nothing if the rows
+are arranged wrong. Do this before the metric pre-flight, because two of the
+checks there need splits that already exist.
+
+## The three sets do different jobs
+
+```ts
+await optimizer.optimize({
+  trainingSet, // reflection reads these: outputs, feedback, what went wrong
+  validationSet, // the search selects candidates against these
+  testSet, // scored once, at the end, on the winner only
+  // …
+});
+```
+
+`trainingSet` is where evidence comes from. Reflective search mines these rows
+for what a candidate got wrong, so a training row earns its place by being
+_diagnostic_ — a row every candidate passes teaches the rewriter nothing.
+
+`validationSet` is what selection pressure is applied to, for the whole run.
+`bestScore` is a mean over it, which is why `bestScore` is partly fitted and
+the result says so.
+
+`testSet` is the only set nothing was ever selected against, and `testScore` is
+therefore the only number in a result you can report to someone else. Omit it
+and there is no such number — `testScore` is simply absent.
+
+Passing no `validationSet` at all silently reuses `trainingSet`, and the run
+carries `validationSetReusesTraining` to say so. That is the right default for
+a first look and the wrong thing to report from, because reflection mined the
+same rows that then judged the result.
+
+## Split by group, never by row
+
+This is the failure that survives every other check.
+
+Real datasets contain families: the same ticket filed twice, a question and its
+paraphrase, five rows generated from one template, transcripts from one long
+session. Shuffle and slice, and a family lands on both sides of the boundary.
+Reflection then reads one member and the search is selected on its twin, so the
+prompt memorises a fact that scores on validation and generalises to nothing.
+The gap does not show up as a warning; it shows up as a `testScore` well below
+`bestScore`, or worse, not at all if you never held out a test set.
+
+Read how the corpus was built before you infer anything about it. Structure
+that is expensive to detect from the rows is often free to read about: a
+benchmark that says it ships contrast pairs, a generator with a template count,
+a README naming the sampling frame. Deliberate structure is the kind that leaks
+worst and the kind most likely to be documented.
+
+Then find the grouping key — a source id, a customer, a template, a document, a
+session. If there is no explicit key, near-duplicate text is a usable stand-in: normalise whitespace and case, and cluster on a shared
+distinctive phrase or a similarity threshold. Then assign whole groups to
+splits. Never assign rows.
+
+Check it afterwards rather than trusting it: no group id appears in two splits,
+and the three splits are disjoint and cover the rows you meant to use.
+
+## Every class in every split
+
+A class absent from `validationSet` is a class the search is free to break,
+because nothing measures it. A class absent from `testSet` means the final
+number says nothing about it.
+
+Rare classes are where this bites, and they are usually the ones that matter —
+the refund case, the abuse report, the one legal asked about. Stratify the
+split by label so each set holds some of each, and when a class is too small to
+stratify, say so out loud rather than letting the split decide silently.
+
+Imbalance also quietly sets the metric's ceiling. If 80% of rows are one easy
+class, a candidate that handles only that class scores 0.8, and the 0.8 looks
+like progress.
+
+## Sizes
+
+There is no defensible universal number, but the constraints are real:
+
+| Set             | What sets the floor                                                                                    |
+| --------------- | ------------------------------------------------------------------------------------------------------ |
+| `trainingSet`   | Enough diagnostic rows that a minibatch shows a candidate failing in more than one way                 |
+| `validationSet` | Enough that the mean is not moved by one instance; this is also what a full sweep costs per acceptance |
+| `testSet`       | Enough that a difference you would act on is larger than its own noise                                 |
+
+A validation set of 10 makes every instance worth 10% of the score. A search
+will find the one instance it can flip.
+
+Cost scales with `validationSet`, not with the other two — see `tuning.md` for
+what a sweep costs per optimizer, and note that SIMBA reserves
+`min(candidates + 1, maxSteps + 1) × |val|` rollouts before the first step.
+
+## Order of operations
+
+1. Dedup exact repeats. They inflate whichever split they land in.
+2. Find the grouping key. Cluster near-duplicates if there is none.
+3. Split by group, stratified by label.
+4. Verify: groups disjoint across splits, every class present in each.
+5. Only now run the metric pre-flight in `metric-preflight.md`, using the
+   splits you just made.
+
+## What to do when there is not enough data
+
+Small sets are the normal case, not a failure. What matters is that the
+smallness is stated rather than hidden:
+
+- Prefer a real `testSet` over a larger `validationSet`. A fitted number
+  measured on more rows is still fitted.
+- With too few rows to make three sets, make two and report that `bestScore` is
+  all you have — do not pass `testSet` and then report `bestScore` anyway.
+- Do not manufacture rows by paraphrasing existing ones into another split.
+  That is the group-leakage failure, performed deliberately.

package/docs/distillation.md

@@ -0,0 +1,128 @@
+# Distilling a run
+
+Prompt optimization makes a prompt better partly by making it longer, and you
+pay for that length on every inference forever. Distillation is what makes the
+length free: run the optimized candidate on the strong model, keep the rollouts
+the metric rewarded, and train a smaller model on them. The text moves into
+weights.
+
+textopt collects and serializes the training data. It never trains anything —
+that is a provider's job, and the providers change.
+
+## The order matters
+
+Distill before optimizing and you freeze whatever your first draft happened to
+do. The search is what finds the behaviour worth freezing.
+
+["Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better
+Together"](https://arxiv.org/abs/2407.10930) goes further: alternating the two
+beats weight optimization alone by up to 60% and prompt optimization alone by
+up to 6%. The second prompt pass is an ordinary `optimize()` call with an
+adapter pointed at the fine-tuned model, so nothing here is a one-way door —
+provided the render step below leaves a prompt to optimize.
+
+## Harvesting
+
+`harvestRollouts` runs a candidate over data and keeps what the metric
+rewarded. It is the same primitive `harvestFewShotExamples` uses, without the
+four-example ceiling:
+
+```ts
+import { harvestRollouts } from "textopt";
+
+const harvest = await harvestRollouts({
+  adapter,
+  candidate: result.bestCandidate, // the run's winner
+  data: unlabeledPool,
+  minScore: 0.9,
+  maxMetricCalls: 5000,
+});
+
+harvest.rollouts; // { input, output, score }[]
+harvest.attempted; // instances run, including the ones that failed the bar
+```
+
+Filtering on `minScore` is what makes this rejection sampling rather than
+imitation: you are copying the strong model on the occasions your metric says
+it was right. A boolean metric makes `minScore: 1` the only sensible threshold;
+a graded metric asked for a perfect score throws away every rollout that was
+most of the way there, which on a hard task is all of them.
+
+**Sweep the right data.** Not the validation set — that is the set that
+selected the winning candidate, so its rollouts are enriched for the
+candidate's fit to those particular instances rather than to the task, and
+there are only ever tens to low hundreds of them. Use the training set, or
+better, a pool held out of the run entirely. Volume comes from the sweep, not
+from the run: a search bounded at 150 metric calls is a seed, and 5,000
+unlabeled instances is a dataset.
+
+The harvest carries its own budget. It is a separate spend from the run that
+produced the candidate, and `maxMetricCalls` bounds it.
+
+## Rendering
+
+`toTrainingJsonl` serializes rollouts as one chat-messages example per line —
+the shape Axolotl, Together, Fireworks and the Hugging Face trainers all
+ingest. Your `render` callback decides what each example looks like, because
+only you can read a `Datum`:
+
+```ts
+import { toTrainingJsonl } from "textopt";
+
+const jsonl = toTrainingJsonl({
+  rollouts: harvest.rollouts,
+  render: ({ rollout }) => ({
+    messages: [
+      { role: "user", content: rollout.input.question },
+      { role: "assistant", content: String(rollout.output) },
+    ],
+  }),
+});
+```
+
+Return `null` from `render` to drop a rollout. Write the string yourself —
+nothing here touches the filesystem.
+
+### How much of the prompt to leave in
+
+This is the decision that matters, and there is no default worth picking for
+you.
+
+**Drop it entirely.** The input is the bare instance; the whole optimized
+candidate moves into weights. Maximum compression, and it works when the task
+is inferable from the input distribution. It fails when the task specification
+lives in the prompt — a custom label taxonomy, an output schema, a routing
+rubric — because the student then has to reverse-engineer that from examples.
+It also ends the alternating loop above: there is no prompt left to optimize.
+
+**Keep a short task statement.** Train on a brief instruction plus the
+instance, serve with the same brief instruction. What you distill away is the
+optimized delta — the tips and examples the search accreted — rather than the
+statement of the job. This is the right default for most systems.
+
+**Keep the full optimized candidate.** No compression at all, which is the
+point: the student learns under the prompt it will be served with, and the next
+optimization pass tunes that prompt against the new weights. This is the
+BetterTogether shape.
+
+## Checking it worked
+
+The number that matters is not the training loss. Point an adapter at the
+fine-tuned model and run both against the same held-out `testSet` — data that
+neither the prompt search nor the fine-tune ever saw:
+
+```ts
+import { compare } from "textopt";
+
+const comparison = await compare({
+  seeds: [0, 1, 2, 3, 4],
+  entrants: {
+    teacher: () => evaluatePrompted(teacherAdapter, result.bestCandidate),
+    student: () => evaluateDistilled(studentAdapter),
+  },
+});
+```
+
+`compare()` ranks on `testScore` and reports a paired sign-flip p-value against
+the winner, so "the distilled model matches the prompted one" is a measurement
+rather than an impression. See [Measuring a result](evaluation.md).

package/docs/evaluation.md

@@ -0,0 +1,87 @@
+# Measuring a result
+
+Two questions a finished run cannot answer about itself: how much of its score
+is fitted to the set that picked the winner, and whether it beat another
+optimizer by more than noise.
+
+## What a run says about itself
+
+`result.warnings` is what a run could see about its own measurement and its
+numbers could not say. It is never fatal and never empty of meaning: an entry is
+a reason to read `bestScore` as less than it appears.
+
+| Code                          | What it means                                                                                                                |
+| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| `validationSetReusesTraining` | No `validationSet` was given, so selection ran on the instances reflection read. Pass one, or `"reuseTraining"` to accept it |
+| `seedScoreSaturated`          | The seed already scores perfectly on every validation instance, so every proposal ties and acceptance resolves noise         |
+| `seedScoreFloored`            | The seed scores 0 on every validation instance — a seed with everything to gain, or a metric that scores nothing             |
+
+The last two are read once, off the seed's own validation row, before the search
+spends anything. Both describe a metric that does not separate the instances it
+is being asked to rank candidates by, which no amount of search budget fixes.
+
+The same list rides on the `finish` event, so a reporter writing a score
+somewhere permanent writes the caveat beside it rather than leaving it in a
+console nobody kept.
+
+## Held-out evaluation
+
+The optimizer selects candidates against `validationSet`, so `bestScore` is fitted to that set and may overstate performance on unseen data.
+
+Pass a `testSet` and the winner is scored on it once, after the search is over:
+
+```ts
+const result = await optimizer.optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  testSet, // never seen by the search
+  adapter,
+  reflect,
+  maxMetricCalls: 300,
+});
+
+result.bestScore; // on the validation set — the search selected for this
+result.testScore; // on instances no candidate was ever selected against
+result.testMetricCalls; // charged separately, not against maxMetricCalls
+result.testUsage; // and costed separately, outside maxCostUsd
+```
+
+The gap between `bestScore` and `testScore` estimates validation overfitting. Test rollouts are reported separately and are outside every ceiling: they do not count against `maxMetricCalls`, and their tokens are in `testUsage` rather than `usage`, because the sweep runs after the search has already stopped. Budget for it the way you would budget for one full validation sweep. The resume fingerprint ignores `testSet`, so it can be added when resuming a run.
+
+All optimizers expose these fields through `OptimizerTask` and `OptimizerResult`.
+
+## Reading the winning text
+
+`result.bestCandidate` is a prompt. That it is readable is the entire advantage over tuning weights, and it is the check no metric performs. Look for:
+
+- **Absorbed facts.** Specific names, numbers, or dates from validation instances written into the instruction. That is memorisation, and it will not transfer, whatever `testScore` said.
+- **Accreted rules.** A long tail of narrow "if the input mentions X, do Y" clauses is a search patching instances rather than learning the task.
+- **Instructions that contradict.** Reflective search appends; nothing prunes.
+
+If the prompt looks wrong and the number looks good, believe the prompt.
+
+## Comparing optimizers
+
+A difference in means over a handful of seeds is usually noise. `compare()` runs each entrant over the same seeds, ranks them on `testScore` where a run reports one, and reports a paired sign-flip p-value against the winner:
+
+```ts
+import { compare } from "textopt";
+
+const comparison = await compare({
+  seeds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
+  concurrency: 4,
+  entrants: {
+    gepa: ({ seed }) => new GepaOptimizer({ seed }).optimize(task()),
+    opro: ({ seed }) => new OproOptimizer({ seed }).optimize(task()),
+  },
+});
+
+comparison.winner; // highest mean score
+comparison.summaries; // mean, sd, min, max, rollouts, cost, pValueVsWinner
+comparison.runs; // every individual run
+```
+
+Entrants are functions of a seed, not optimizer instances: the seed is constructor config, and every optimizer here is deterministic given one, so comparing two entrants at a single seed compares two anecdotes. Build a fresh task inside each entrant — a shared reflection model with internal state would make each result depend on the runs before it.
+
+Ranking on `testScore` matters. The validation score is the number the search selected against for its whole run, so an entrant that overfits looks strongest on exactly the number it fitted.