textopt 0.0.0 → 0.2.0

package/docs/metric-preflight.md

@@ -0,0 +1,132 @@
+# Validating a metric before a search runs
+
+Three things have to be true before any number a run reports means anything:
+
+- the splits must not leak — see [Preparing the data](./data-prep.md)
+- the metric must have range: it must separate candidates, and move over an
+  interval wide enough to see
+- its noise must be smaller than the effect you are looking for
+
+They are preconditions, not a checklist. Everything downstream — which
+optimizer, which minibatch size, which acceptance test — is reasoning stacked
+on an instrument, and reasoning about the instrument is cheaper than every
+argument it supports. All three are checkable in an hour, without an optimizer.
+
+The four checks below establish them, in the order to run them. Do not start a
+real run until they pass.
+
+## 1. Discrimination on known controls
+
+The highest-value check and the one almost nobody runs.
+
+Write three to five candidates whose ranking you already know:
+
+- the seed
+- one deliberately sabotaged — drop the constraint you care most about
+- one hand-tuned, as good as you can write by hand
+- optionally: an empty instruction, and one that is verbose but wrong
+
+Score each on a slice of the validation set and compare the ranking to your
+own. If the metric does not reproduce it, no search will find anything. It
+will wander, accept noise, and report a number.
+
+This is also the fastest way to discover reward hacking before the search
+does. A verbose-but-wrong candidate that outranks a terse correct one tells you
+the judge is paying for length.
+
+## 2. Test-retest variance
+
+Score the _same_ candidate over the _same_ instances three times, with caching
+off. The per-instance spread is the metric's own noise.
+
+That number decides two things the defaults cannot decide for you:
+
+- whether `pairedPermutationAcceptance` and `lowerBoundEvaluationPolicy` are
+  worth their cost. Turn them on when you have measured large variance, not on
+  principle — on the benchmark's pipeline task the pair loses to plain GEPA at
+  p = 0.027, because a significance bar on every acceptance is expensive when
+  improvements to one component only pay off after another is finished
+- how wide a minibatch has to be to say anything. A sign-flip test over n
+  instances bottoms out at a p-value of 2^-n
+
+Deterministic metric, zero variance: keep the defaults.
+
+## 3. Spread, not ceiling or floor
+
+Look at the distribution of the seed's per-instance scores, not its mean.
+
+- **Everything at the ceiling.** Nothing to improve; every proposal ties and
+  acceptance resolves ties by whatever the noise did. A run reports
+  `seedScoreSaturated`.
+- **Everything at zero.** Ambiguous. It is what a seed with everything to gain
+  looks like _and_ what a broken metric looks like, and the score cannot tell
+  them apart — check 1 can. Reported as `seedScoreFloored`.
+- **Everything identical in the middle.** The metric is measuring something the
+  candidate does not control.
+
+What you want is real per-instance variation. GEPA's frontier is taken over
+validation instances, so a seed row with no spread makes the frontier
+degenerate before the search starts.
+
+Then ask the same question of the scale itself. A judge's aggregate is bounded
+by 0 and 1, but the interval it actually moves in on realistic candidates is
+usually narrower, and it is the realized span — not the theoretical one — that
+every later number is denominated in. Score your controls from check 1, take
+the highest and the lowest, and write the difference down. If the best
+candidate you can write by hand scores 1.0 and the sabotaged one scores 0.6,
+your instrument has a span of 0.4, and an improvement that reads as "+0.02" is
+5% of the available range rather than 2% of it.
+
+A narrow span is not automatically wrong, but it is always worth an
+explanation. The common cause is a requirement enforced twice: see the note on
+`gate` below.
+
+## 4. Agreement with your own labels
+
+Hand-label 30–50 outputs — good, bad, and the ambiguous middle — then have the
+judge grade the same ones and compare.
+
+If the judge cannot reproduce your labels, the search is optimizing the judge's
+idiosyncrasies rather than the task. That is still optimization, and the number
+will still climb.
+
+Use the disagreements to rewrite the criteria. A criterion the judge and you
+read differently is a criterion description problem, not a model problem.
+
+## What to fix, in order
+
+| Finding                                 | Fix                                                                     |
+| --------------------------------------- | ----------------------------------------------------------------------- |
+| Ranking of controls is wrong            | The criteria. Nothing downstream is worth doing until this passes       |
+| Judge rewards length or confident tone  | Add an explicit brevity criterion; make correctness the higher `weight` |
+| A hard requirement can be averaged away | Give it a `gate`, and keep its `weight` low — see below                 |
+| Realized span is much narrower than 0–1 | Look for a requirement enforced twice, by a `gate` and a heavy `weight` |
+| Variance is large                       | Widen the minibatch first, then consider the significance guards        |
+| Seed saturated                          | A harder validation set, or a metric that separates these instances     |
+| Seed floored and controls do not rank   | The metric is broken, not the seed                                      |
+| Judge disagrees with your labels        | Rewrite the criterion descriptions; re-check against the same labels    |
+
+## Enforce a requirement once
+
+`gate` and `weight` are different instruments and it is tempting to reach for
+both. A gate makes a requirement non-negotiable: below the threshold the
+instance scores 0 whatever else it did. Once a gate is doing that, a heavy
+weight on the same criterion buys nothing — every candidate that clears the
+gate satisfies the requirement, so the weight is only redistributing score
+among candidates that already agree.
+
+What it costs is range. A heavy weight on a criterion nearly every surviving
+candidate maxes out pins that share of the aggregate near its ceiling, and the
+span left for the search to move in is whatever the other criteria carry.
+Gate the requirement, then weight it low, and let the weights go to the
+criteria that actually vary.
+
+The general form: when a second mechanism starts enforcing a constraint, check
+whether the first still needs to.
+
+## A note on what this is
+
+This is a manual procedure, not a library function. `createJudge` and the
+optimizers give you the pieces; the judgement is yours. If you run these checks
+often against the same task, the natural thing is to script them — a few
+`adapter.evaluate` calls over a fixed batch and a printed table.

package/docs/optimizers.md

@@ -0,0 +1,293 @@
+# Optimizers
+
+What each search reads, and what its knobs do. See [Tuning a run](./tuning.md)
+for what a run costs and [Benchmark](./benchmark.md) for how the six compare on
+offline tasks.
+
+## Choosing an optimizer
+
+All six use the same `Optimizer` interface, adapter, and budget accounting.
+
+| Optimizer                  | Required signal                   | Use it for                                                        |
+| -------------------------- | --------------------------------- | ----------------------------------------------------------------- |
+| `GepaOptimizer`            | per-instance **textual feedback** | revising text from written explanations of failures               |
+| `SimbaOptimizer`           | per-instance **textual feedback** | noisy metrics, where the same instance scores differently per run |
+| `OproOptimizer`            | a **scalar** score                | proposing text from a history of scored attempts                  |
+| `MiproOptimizer`           | a **scalar** score                | searching combinations of interacting component options           |
+| `BootstrapSearchOptimizer` | a **scalar** score                | few-shot demonstrations, with no proposal model at all            |
+| `RandomSearchOptimizer`    | a **scalar** score                | establishing a score-independent paraphrasing baseline            |
+
+Use GEPA when the metric can explain failures in text. A scalar such as `0.0` gives its reflection step little useful information. OPRO only needs scalar scores: its prompt lists previous attempts by score and asks the model to improve on them.
+
+SIMBA also needs textual feedback, but reads a different signal. Instead of reflecting on a failure, it runs several programs over the same instance and reflects on the _contrast_ between the best and worst run of it — a comparison with the input held fixed. That contrast has to exist to be worth paying for, so SIMBA earns its extra rollouts on metrics that vary between runs and wastes them on ones that do not.
+
+GEPA and OPRO update components separately. MIPRO instead searches combinations of per-component options, which lets it find options that work well only together. It screens configurations on minibatches and evaluates promising ones against the full validation set.
+
+MIPRO's default multivariate TPE models complete configurations. Set `multivariate: false` to model each component independently; that usually needs fewer observations but cannot model interactions between components.
+
+`BootstrapSearchOptimizer` writes no text at all. It harvests demonstrations from rollouts the metric already rewarded and searches over which set of them to keep, so it needs no proposal model and no textual feedback. Try it first when the instruction is roughly right and the output format is not.
+
+`RandomSearchOptimizer` paraphrases components without using their scores. Run it as a baseline to check whether reflection improves enough to justify its model calls.
+
+```ts
+import { SimbaOptimizer } from "textopt/simba";
+import { OproOptimizer } from "textopt/opro";
+import { MiproOptimizer } from "textopt/mipro";
+import { BootstrapSearchOptimizer } from "textopt/bootstrap-search";
+import { RandomSearchOptimizer } from "textopt/random-search";
+```
+
+[Benchmark](./benchmark.md) reports what each search does on four offline tasks; read it as evidence about the search rather than about your task, and run [`compare()`](./evaluation.md#comparing-optimizers) on your own metric before choosing.
+
+## How GEPA works
+
+GEPA maintains a pool of candidates and a **Pareto frontier over validation instances**, not objectives. A candidate remains on the frontier while it has the best score for at least one instance. Parent sampling is weighted by the number of instances each candidate wins, preserving candidates with useful strengths even when their mean score is lower.
+
+Each iteration:
+
+1. **Select** a parent from the frontier, and one or more of its components to update.
+2. **Evaluate** the parent on a fresh minibatch, capturing traces.
+3. **Reflect** on per-component evidence from the scored batch: inputs, outputs, feedback, and scores. Recent rejected proposals are included to reduce repetition.
+4. **Screen** the child on the same minibatch. Only improvements proceed.
+5. **Sweep** accepted children over the validation set and update the frontier.
+6. **Merge** lineages that improved different components. Merge is enabled by default for multi-component candidates.
+
+`maxMetricCalls` limits scored rollouts; cache hits do not count. Reflection calls have a separate limit.
+
+Based on _GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning_.
+
+## Configuring GEPA
+
+Pass search settings to the `GepaOptimizer` constructor:
+
+```ts
+new GepaOptimizer({
+  minibatchSize: 3,
+  maxIterations: 50,
+  seed: 11,
+  proposals: { perIteration: 3, concurrency: 3, selection: "best" },
+  reflection: { maxCalls: 40, maxRecords: 5, maxCharacters: 20_000 },
+  // ...or `strategies: diverseReflectionStrategies()` in place of one prompt
+  merge: { enabled: true, maxInvocations: 5 },
+  candidateSelector: paretoSelector(),
+  acceptance: improvementAcceptance(),
+  skipPerfectScore: true,
+  rejectedProposalMemory: 3,
+  trackBestOutputs: true,
+});
+```
+
+`optimize` takes a `GepaTask`. Required fields are `seedCandidate`, `trainingSet`, `adapter`, `reflect`, and `maxMetricCalls`. `validationSet` defaults to `trainingSet`; `testSet` is optional and held out. Other options are `componentSelector`, `batchSampler`, `valEvaluationPolicy`, `instanceId`, `cache`, `reporters`, `onCheckpoint`, `resumeFrom`, and `signal`.
+
+Component names are inferred from `seedCandidate`. Other positions use `NoInfer`, so misspelled component names fail type checking.
+
+`proposals.perIteration` is the setting that moves the bill: each proposal is priced at two minibatch evaluations, its parent's and its own, so raising it from one to three triples what an iteration costs before any child is swept ([Sizing a run](./tuning.md#sizing-a-run)). What that buys is width — every slot draws its own parent and its own minibatch, so the slots diagnose different failures and `concurrency` can run them at once, and `selection: "best"` then promotes only the strongest improving child of the batch. Pair it with `reflection.strategies`, which is what keeps two slots that landed on the same parent from writing the same revision.
+
+`textopt/gepa` exports the following strategies: `paretoSelector`, `currentBestSelector`, `epsilonGreedySelector`, `topKParetoSelector`, `roundRobinComponentSelector`, `allComponentsSelector`, `improvementAcceptance`, `pairedPermutationAcceptance`, `fullEvaluationPolicy`, `subsampledEvaluationPolicy`, and `lowerBoundEvaluationPolicy`.
+
+### Proposal strategies
+
+By default, every proposal uses the same reflection prompt. Multiple calls against one parent can therefore produce similar revisions.
+
+`reflection.strategies` rotates over several framings instead, one per proposal slot:
+
+```ts
+import { diverseReflectionStrategies } from "textopt/gepa";
+
+new GepaOptimizer({
+  proposals: { perIteration: 4, concurrency: 4 },
+  reflection: { strategies: diverseReflectionStrategies() },
+});
+```
+
+The included rotation alternates the standard prompt with prompts that simplify, generalize, or rewrite the candidate. The builders are exported as `buildReflectionPrompt`, `buildSimplifyPrompt`, `buildGeneralizePrompt`, and `buildRewritePrompt`; custom `ReflectionPromptBuilder` functions use the same interface.
+
+This behavior is opt-in. The default is `buildReflectionPrompt`, adapted from the GEPA paper's reflection prompt rather than copied from it: it carries the same evidence and asks for the same thing, but tags and wording differ from the reference implementation's template, so proposals drawn from identical evidence will not match it.
+
+## Few-shot demos
+
+Few-shot examples can be stored in a candidate component and optimized with the other text. textopt can populate that component from successful training rollouts.
+
+Before optimization, `harvestFewShotExamples` evaluates the seed candidate on `trainingSet` and keeps high-scoring rollouts:
+
+```ts
+import { harvestFewShotExamples } from "textopt";
+
+const { block, demos, metricCalls } = await harvestFewShotExamples({
+  adapter,
+  candidate: seedCandidate,
+  trainingSet,
+  minScore: 0.9,
+  maxDemos: 4,
+});
+
+const seed = { instruction: "Route the ticket.", demos: block };
+```
+
+During optimization, `createDemoProposer` reads demos from the existing reflective dataset without additional rollouts or reflection calls:
+
+```ts
+import { createDemoProposer } from "textopt/gepa";
+
+const adapter = {
+  ...baseAdapter,
+  proposeNewTexts: createDemoProposer({
+    components: ["demos"],
+    minScore: 0.9,
+    maxDemos: 4,
+    fallback: baseProposer, // writes the other components normally
+  }),
+};
+```
+
+Each proposal appends demos to its parent's block. A demo remains in the lineage only when the candidate containing it is accepted.
+
+## The other optimizers
+
+Apart from SIMBA, these use the base `Adapter`: they need no reflective dataset. Their results include both `seedScore` and `bestScore`.
+
+### SIMBA
+
+```ts
+import { SimbaOptimizer } from "textopt/simba";
+
+const result = await new SimbaOptimizer({
+  minibatchSize: 16,
+  candidates: 4,
+  maxSteps: 8,
+  seed: 11,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter, // the base Adapter — no makeReflectiveDataset needed
+  reflect,
+  demoComponents: ["demos"], // optional; enables the appendDemo mutation
+  maxMetricCalls: 900, // 250 reserved for finalists, 80 a step, over 50 validation instances
+});
+
+result.finalists; // the step winners, scored on the full validation set, best first
+```
+
+Each step samples several programs from the pool over one minibatch, ranks the instances by how much those programs disagreed, and mutates toward whichever run won. Two mutations are drawn at random per instance:
+
+- **`appendDemo`** keeps the winning rollout as a few-shot example. Costs no model call. Requires `demoComponents`.
+- **`appendRule`** shows the better and worse run of the same instance to `reflect` and appends the advice it returns to each instruction component.
+
+Neither replaces text, so candidates accumulate; demonstrations are dropped at a Poisson rate so a growing block cannot crowd out the instruction. `strategies` pins the mutation to one of the two.
+
+Ported from DSPy's SIMBA with two deliberate changes. A trajectory sample runs one program across the whole minibatch rather than resampling a program per instance, because the adapter owns decoding here and there is no temperature knob to vary — the variability comes from the program pool instead. And the percentile guards are strict rather than inclusive, so a step on which every rollout ties still produces a mutation instead of doing nothing at all.
+
+Only the finalists are scored on the full validation set: the step winners are sampled evenly across the run, so early winners stay in the running. Those rollouts are reserved before the search starts, which is why a small `maxMetricCalls` buys fewer steps than the arithmetic suggests — see [Sizing a run](./tuning.md#sizing-a-run) for what the reserve costs.
+
+Its batch defaults are wide on purpose: a step reads the disagreement between programs over a batch, and a narrow batch leaves little to rank. When a run has to get cheaper, lower `candidates` first — it shrinks both the step and the reserve, where `minibatchSize` shrinks the step alone and narrows the batch the ranking reads.
+
+`concurrency` overlaps the two places a step's work is independent: scoring the candidates it built, and the finalist sweeps at the end. The trajectory samples and the mutations that read them stay in sequence — each reads what the one before it wrote — so the ceiling on what it buys is the reserve and the scoring, not the whole step. See [Concurrency](./tuning.md#concurrency).
+
+### Bootstrapped few-shot search
+
+```ts
+import { BootstrapSearchOptimizer } from "textopt/bootstrap-search";
+
+const result = await new BootstrapSearchOptimizer({
+  candidates: 16,
+  maxDemos: 4,
+  seed: 11,
+}).optimize({
+  seedCandidate: { instruction, demos: "" },
+  trainingSet,
+  validationSet,
+  adapter,
+  demoComponents: ["demos"],
+  goldOutput: (datum) => datum.answer, // optional; enables the labels-only candidate
+  maxMetricCalls: 1500, // 19 candidates swept over 50 validation instances, plus their harvests
+});
+
+result.candidates; // every set tried, with its source and how many demos it held
+```
+
+DSPy's `BootstrapFewShotWithRandomSearch`. It calls no model to write text: every candidate is assembled from outputs the system itself produced, so the search costs rollouts and nothing else. The fixed candidates come first — zero-shot, then labels-only when `goldOutput` is given, then one unshuffled full-size harvest — followed by shuffled harvests of random size.
+
+Zero-shot stays in the running throughout. Demonstrations can hurt, and a search that cannot return "no demos" has no baseline to report against.
+
+`concurrency` overlaps a candidate's sweep with the harvest of the candidates behind it, which is most of the wall clock on a run whose harvests are as expensive as its sweeps. Harvesting itself stays in plan order, because every harvest draws from the same random stream. It is ignored when `stopAtScore` is set, and it moves checkpointing from once per candidate to once per wave — see [Concurrency](./tuning.md#concurrency).
+
+`candidates` is the whole search: each one is a fresh harvest and a full sweep, so it sets both the breadth and the bill. `demoMinScore` is the knob that surprises — a strict threshold does not cost less, it costs more, because a harvest keeps rolling out training instances until it has collected `maxDemos` of them or run out of set. If harvests come back with no demos, the seed cannot yet produce work its own metric rewards, and few-shot search is the wrong tool until it can.
+
+### OPRO
+
+```ts
+const result = await new OproOptimizer({
+  proposalsPerRound: 4,
+  historySize: 10,
+  maxReflectionCalls: 40,
+  seed: 11,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter,
+  reflect,
+  maxMetricCalls: 2000, // 4 proposals a round, each swept over 50 validation instances
+});
+
+result.trajectory; // every candidate scored, in the order it was tried
+```
+
+By default, every proposal is scored on the full `validationSet`. With `scoringSetSize`, proposals are screened on a fixed subset of `trainingSet`, and the incumbent receives a full sweep every `fullEvalInterval` rounds. In a 30-instance validation set, screening on 12 instances halved rollout count without reducing the measured best score.
+
+The meta-prompt lists the strongest attempts in ascending score order, placing the best attempt nearest the request. `scoreScale` converts scores to integers (100 by default), because models distinguish values such as 41 and 68 more reliably than 0.41 and 0.68.
+
+Because every proposal in a round sees the same history and is screened against the same incumbent, `concurrency` runs a whole round — proposals and screens together — at once, and the round still records its attempts in the order it drew them.
+
+Rounds are where OPRO gets its signal: every proposal in a round sees the same history, so `proposalsPerRound` widens a round rather than deepening the search, and the history a later prompt reads only grows between rounds. A run also moves one component per round, in turn, so a two-component candidate needs twice the rounds to revise each as often. Budget for rounds first, then set `proposalsPerRound` to what a round can afford — `maxReflectionCalls` caps the two together, at `maxReflectionCalls / proposalsPerRound` rounds.
+
+### MIPRO
+
+```ts
+const result = await new MiproOptimizer({
+  instructionsPerComponent: 5,
+  maxTrials: 30,
+  seed: 11,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter,
+  reflect,
+  demoComponents: ["demos"], // menu bootstrapped from the training set
+  maxMetricCalls: 1600, // 30 trials of 35, six sweeps of 50, and the demo harvests
+});
+
+result.menu; // the space that was searched, per component
+result.observations; // every configuration tried, and which earned a full sweep
+```
+
+MIPRO first builds a menu for each component from the seed text and variants generated by `reflect`. A TPE surrogate then proposes configurations from those menus. Trials run on minibatches; selected configurations receive a full validation sweep.
+
+`componentOptions` adds menu entries without reflection calls. `demoComponents` builds menus of few-shot blocks from successful training rollouts, allowing instructions and demonstrations to be searched together.
+
+Every `fullEvalInterval` trials, MIPRO fully evaluates the unswept configuration with the highest average minibatch score. Averaging repeated observations reduces the effect of a lucky minibatch.
+
+The space is the product of the menus, so it grows multiplicatively with components while `maxTrials` grows by hand — three components of six options each is 216 configurations, and the default thirty trials sees a seventh of them. The surrogate also spends its first ten trials sampling at random before it models anything, so a short run is mostly random search with extra steps. Give it trials in proportion to the menu, or trim the menu with `instructionsPerComponent`.
+
+### Random search
+
+```ts
+const result = await new RandomSearchOptimizer({
+  variants: 4,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter,
+  reflect,
+  maxMetricCalls: 1000, // 4 variants a round, each swept over 50 validation instances
+});
+```
+
+Random search paraphrases one component per round and keeps the highest-scoring candidate. Its prompt receives no performance data. Compare it with a reflective optimizer under the same metric budget to measure the benefit of reflection.
+
+`concurrency` runs a round's variants at once, proposal and sweep together, and is the one setting here that changes nothing about the search — the round still accepts a variant only if it beat every variant drawn before it.
+
+Being a baseline is the whole configuration: give it the `maxMetricCalls` and `validationSet` of the run it stands against, and change nothing else. A baseline on a smaller budget answers a different question than the one being asked of it. [`compare()`](./evaluation.md#comparing-optimizers) runs both over the same seeds and ranks them on the held-out score, but each entrant builds its own task, so keeping the budgets equal is still yours to do.

package/docs/tuning.md

@@ -0,0 +1,130 @@
+# Tuning a run
+
+What a search spends and how to bound it: the arithmetic for pricing a run
+before starting it, what to read when one disappoints, the guards for a metric
+that does not return the same number twice, and the ceilings on money, time,
+and repeated work.
+
+## Sizing a run
+
+`maxMetricCalls` is the only hard bound on a search, and a run that cannot afford its next unit of work stops rather than throws: the result carries `stopReason: "budgetExhausted"` and whatever had been found by then. An underfunded run looks exactly like a finished one, so price it before starting it.
+
+GEPA, MIPRO, OPRO, and random search sweep the seed candidate over the validation set before anything else, and bootstrap search's first candidate is the zero-shot one, which is that same sweep. SIMBA scores its seed alongside its finalists instead. After that, each spends in units of its own — `|val|` below is the size of the validation set, `|train|` the training set:
+
+| Optimizer                  | One unit of search                                                                   | Charged besides                                                                                         |
+| -------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
+| `GepaOptimizer`            | an iteration: `perIteration × minibatchSize × 2 + \|val\|`                           | —                                                                                                       |
+| `SimbaOptimizer`           | a step: `(candidates + 1) × minibatchSize`                                           | `min(candidates + 1, maxSteps + 1) × \|val\|`, reserved before the first step                           |
+| `MiproOptimizer`           | a trial: `minibatchSize`                                                             | `\|val\|` every `fullEvalInterval` trials, and up to `demoSets × \|train\|` to bootstrap each demo menu |
+| `OproOptimizer`            | a round: up to `proposalsPerRound × \|val\|`, or `× scoringSetSize` when that is set | `\|val\|` every `fullEvalInterval` rounds, once `scoringSetSize` is set                                 |
+| `BootstrapSearchOptimizer` | a candidate: `\|val\|`, plus up to `\|train\|` to harvest its demos                  | —                                                                                                       |
+| `RandomSearchOptimizer`    | a round: `variants × \|val\|`                                                        | —                                                                                                       |
+
+GEPA's doubling is the parent: each proposal scores its parent and its child on the same minibatch, because acceptance is a paired comparison rather than a threshold. The trailing `|val|` is the sweep a child earns by improving, reserved before the iteration starts rather than discovered missing once there is something to promote. The others refuse work for the same reason — MIPRO stops as soon as it can no longer afford a sweep, because a reading nothing can act on buys nothing.
+
+SIMBA is the one worth doing the arithmetic for, because its reserve comes off the top. Under its defaults — `candidates: 6`, `minibatchSize: 32`, `maxSteps: 8` — against a 50-instance validation set, it holds back 350 rollouts for the finalists and spends 224 per step, so eight steps need about 2,150. The same run under `maxMetricCalls: 600` takes one step and stops.
+
+### Sizing the sets
+
+`validationSet` defaults to `trainingSet`. That is the right default for a first run and the wrong number to report: the search selected against those instances, so `bestScore` is fitted to them. Under reflective search it is worse than ordinary overfitting — the reflection prompt asks the model to mine domain facts out of the traces it is shown, so those facts come out of the very instances that then select the candidate carrying them.
+
+A run that took the default says so: `result.warnings` carries a `validationSetReusesTraining` entry, and so does the `finish` event. Pass `validationSet: "reuseTraining"` to accept the reuse by name and silence it. [Held-out evaluation](./evaluation.md#held-out-evaluation) is how to find out what it cost.
+
+Validation size multiplies almost every row of the table, so it decides what a run costs. Shrinking it is the wrong lever — it makes the number that picks the winner noisier. Screen on something smaller and sweep rarely instead: that is what OPRO's `scoringSetSize`, MIPRO's minibatch trials, and GEPA's minibatch screening are for.
+
+Minibatch defaults differ by an order of magnitude between optimizers — GEPA 3, SIMBA 32, MIPRO 35 — and do not transfer. GEPA compares a child against its own parent on the same instances, so three of them already say something. MIPRO hands the batch mean to its surrogate as an absolute reading of a configuration. SIMBA ranks the instances in a batch by how much its programs disagreed on them. Carry one optimizer's number to another and the search reads noise.
+
+### When a run disappoints
+
+| What happened                                                             | Where to look                                                                                                                  |
+| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| stopped short of `maxIterations`, `maxSteps`, `maxTrials`, or `maxRounds` | `stopReason`. `budgetExhausted` is the arithmetic above, not a failure                                                         |
+| `stopReason` is `reflectionBudgetExhausted`                               | GEPA's `reflection.maxCalls` or OPRO's `maxReflectionCalls`. Both are separate from the rollout budget and default unbounded   |
+| `bestScore` improved but `testScore` did not                              | the validation set is too small or too easy to separate candidates. The gap is the measurement working                         |
+| accepted candidates do not hold up when re-evaluated                      | metric noise — [Noisy metrics](#noisy-metrics), including the cost of turning both guards on                                   |
+| proposals repeat themselves                                               | `reflection.strategies` for the framing, `rejectedProposalMemory` for what the prompt is told has already failed               |
+| MIPRO settles on the seed                                                 | the menus were the search space: read `result.menu`, add `componentOptions`, and set `multivariate: false` when trials are few |
+| the run cost more money or took longer than expected                      | `maxCostUsd` and `maxWallClockMs` — [Budgets, cost, and time](#budgets-cost-and-time)                                          |
+
+### What to try first
+
+`BootstrapSearchOptimizer` answers the cheapest question worth asking first — whether the instruction is already fine and consistency is what is failing — and it calls no proposal model to do it, so a run costs rollouts and nothing else. Its two answers are both worth having: on the benchmark's `demonstrated` task it beats every entrant that searches instructions except the two GEPA rows, and on `clean` it scores zero, which is what "your system is not inconsistent, it is mis-instructed" looks like. Reach for a reflective search once that has been ruled out, and pick between them with [`compare()`](./evaluation.md#comparing-optimizers) under one budget rather than from the [benchmark table](./benchmark.md).
+
+## Minibatch sizes do not transfer
+
+The defaults differ by an order of magnitude — GEPA 3, SIMBA 32, MIPRO 35 — because they mean different things. GEPA compares a child against its own parent on the same instances, so three already say something. MIPRO hands the batch mean to a surrogate as an absolute reading. SIMBA ranks instances within a batch by how much its programs disagreed. Carrying one number to another optimizer makes it read noise.
+
+## Noisy metrics
+
+The default acceptance rule takes any minibatch improvement, and the default winner is the highest validation mean. Both are the right reading when a rollout of the same text on the same instance always scores the same. When it does not — a sampled model, a judge, a flaky tool — a run accumulates changes that only ever won a coin flip.
+
+```ts
+import {
+  lowerBoundEvaluationPolicy,
+  pairedPermutationAcceptance,
+} from "textopt/gepa";
+
+new GepaOptimizer({
+  minibatchSize: 8,
+  acceptance: pairedPermutationAcceptance({ alpha: 0.2 }),
+}).optimize({
+  ...task,
+  valEvaluationPolicy: lowerBoundEvaluationPolicy({ z: 1 }),
+});
+```
+
+`pairedPermutationAcceptance` accepts only when the paired per-instance improvement survives a sign-flip test at `alpha`. `lowerBoundEvaluationPolicy` returns the candidate with the best mean minus `z` standard errors, rather than the best mean.
+
+Both are strictly more conservative. On the benchmark that buys a little where it should and costs nothing where it should not: on the `noisy` task the pair scores 0.931 against plain GEPA's 0.920, and on the noiseless `clean` task 0.945 against 0.947 — a difference no larger than the seed spread. Neither gap clears significance over twenty seeds (Holm-adjusted p = 0.088 and 0.334), so read this as "not expensive" rather than as a demonstration that it works. Where it does lose clearly is the pipeline task, 0.835 against 0.891 at p = 0.027: a significance bar on every acceptance is expensive when improvements to one component only pay off after another is finished. Turn them on when you have measured the metric's own variance and found it large, not on principle.
+
+A minibatch also has to be wide enough for the test to say anything: a sign-flip test over three instances cannot produce a p-value below 0.125, so at the default `minibatchSize` of 3 no proposal can ever be accepted at `alpha` below that. `GepaOptimizer` throws on that combination at construction. Both halves are reasonable on their own and only their product is wrong, so the constructor is the one place that can see it — and the run it prevents is one that spends its entire budget, returns the seed, and reports a `stopReason` that looks like any other.
+
+## Budgets, cost, and time
+
+`maxMetricCalls` bounds rollouts. It does not bound money or time, and on a long run neither of those follows from it:
+
+```ts
+await optimizer.optimize({
+  ...task,
+  maxMetricCalls: 2000,
+  maxCostUsd: 25, // stops at the first decision point past the ceiling
+  maxWallClockMs: 30 * 60_000, // overruns by at most one evaluation
+  retry: { attempts: 2, delayMs: 500 },
+});
+
+result.usage; // { inputTokens, outputTokens, totalTokens, costUsd, rollouts }
+result.testUsage; // the same, for the held-out sweep, which no ceiling bounds
+```
+
+- **`maxCostUsd`** exists because reflective search grows the text it optimizes, so late rollouts cost more than early ones. It is checked between evaluations and reads whatever usage the adapter reported; an adapter that reports none can never trigger it. `priceUsage` fills in `costUsd` on a rollout's usage from a `TokenPricing` table; adapters call it so there is something to read.
+- **`maxWallClockMs`** exists because a run behind a rate limit spends almost nothing and takes as long as the provider makes it take. This is what makes an optimizer safe to put behind a request timeout or a nightly job. `stopReason` is `"deadlineReached"`. Both ceilings are checked between evaluations, so a run overruns them by whatever it had in flight when they were reached — one evaluation at the default concurrency, and up to `concurrency` of them above it.
+- **Neither ceiling bounds the held-out sweep.** A `testSet` is measured once, after the search has already stopped, and charging it would let the size of a held-out set decide which candidate wins. So it is reported apart from the search: `testMetricCalls` rollouts costing `testUsage`, neither of them inside `metricCalls` or `usage`. Budget for it separately — a `testSet` the size of the validation set costs one full sweep on top of whatever `maxCostUsd` allowed.
+- **`retry`** re-runs instances the adapter marked `transient`. A rate limit or a 5xx otherwise costs the instance either an unexplained zero or a hole in the candidate's coverage. Retries are charged like any other rollout but never overdraw the budget. Defaults to two attempts, 500 ms apart, doubling.
+
+## Concurrency
+
+Every optimizer runs one evaluation at a time by default. `concurrency` raises that, and each optimizer applies it to the part of its loop where the work is genuinely independent:
+
+| Optimizer             | What overlaps                                                                                                                   |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| GEPA                  | `proposals.concurrency` — the proposals of an iteration, and then the validation sweeps of the children that survived screening |
+| SIMBA                 | the candidates a step built, scored on its minibatch, and the finalist sweeps that pick the winner                              |
+| OPRO                  | the proposals of a round, and the screen each of them is scored on                                                              |
+| Random search         | the variants of a round, and the sweep each of them is scored on                                                                |
+| Bootstrapped few-shot | a candidate's sweep with the harvest of the candidates behind it                                                                |
+| MIPRO                 | the instruction proposals for one component                                                                                     |
+
+Raising it does not change what a run finds. Every one of these fans out only after the random stream has been drawn and the whole schedule has been priced against the allowance, and commits results in the order the search proposed them rather than the order they returned — so a seeded run reaches the same candidates, the same winner and the same rollout count at any concurrency. The parts that would not survive that are left in sequence: MIPRO's trials condition on the observations before them, and SIMBA's trajectory samples and mutations read state the sample before them wrote.
+
+What does change is what the provider sees. The limits multiply — `compare` concurrency, then the optimizer's, then the adapter's — so GEPA at `proposals: { concurrency: 4 }` over an adapter at `concurrency: 4` is sixteen calls in flight. Past what a provider tolerates this stops being a timing question: a throttled rollout comes back marked `transient`, which is left out of the candidate's mean rather than scored as a zero, retried at the run's expense, and never cached. A run that spends its allowance on retries and measures its candidates on fewer instances than it asked for looks, from the outside, like a seed that went badly.
+
+Two smaller costs come with raising it. The cost and deadline ceilings overrun by whatever was in flight when they were hit. And bootstrapped few-shot search checkpoints once per wave rather than once per candidate, so a killed run loses up to `concurrency` candidates instead of one; it also ignores `concurrency` when `stopAtScore` is set, since a wave cannot know it has already passed the target.
+
+## Caching, checkpoints, resume
+
+- **Caching.** Cache keys include the split, complete candidate, and instance ID. Cache hits do not count against the metric budget. Instance IDs default to a content hash, falling back to the row position for a datum whose whole content the hash cannot read — one that will not serialize, and one that JSON does not reach at all, such as a bare Map, a Set, or a class instance holding its state privately. The fallback reads the whole datum, not its fields: two rows that differ only inside a nested Map, Set, or function-valued property serialize alike, hash alike, and share cached scores. Pass `instanceId` for data like that, and for non-serializable data or readable trace IDs. Set `cache: false` to disable caching.
+- **`cacheNamespace`.** A cached score is a measurement of a whole system, not of a candidate. Set `cacheNamespace` to name the model id, decoding settings, and scorer version, and change it whenever anything outside the candidate text changes — otherwise a run silently reuses scores measured under a system it is no longer running.
+- **Durable caching.** `createFileCache({ path, namespace })` from `textopt/file-cache` is an append-only log that outlives the process, so a crashed run, a re-run with a changed budget, and a second experiment over the same validation set do not pay for identical rollouts again. `namespace` is required there rather than optional as `cacheNamespace` is: a log on disk outlives every part of the measured system a key does not name, and entries written under one namespace are never served to a run under another. It needs `node:fs`; for Redis or SQLite, implement `EvaluationCache` yourself.
+- **Checkpoints.** Every optimizer takes `onCheckpoint` and `resumeFrom`, and returns its final `snapshot`. Each restores exactly the state that is expensive or unrepeatable: GEPA's candidate pool, rejections and merge state; MIPRO's option menus and surrogate observations; OPRO's screening slice and score histories; SIMBA's program pool and step winners; the search's budget, RNG, sampler position and cached scores throughout. A fingerprint refuses a checkpoint from a different seed candidate, instance set, or random seed. Snapshots are plain JSON and are never mutated by the run that resumes from them.
+- **Events.** Every optimizer takes `reporters`, an array of `Reporter<Event>` with an optional `onEvent` and an optional `flush`. Each search emits its own discriminated union, and all of them emit `start`, `evaluation`, `candidateAccepted` and `finish`. The last two carry a shared payload — the candidate text, its per-instance row over the validation set, and the winner's held-out row — so a reporter written against `ReportableEvent` drops into any optimizer without knowing which one it is.
+- **Result.** Every result carries `bestCandidate`, `bestScore`, `metricCalls`, `usage`, `stopReason`, and `snapshot`, plus whatever its own search can report.

package/package.json

@@ -1,10 +1,11 @@
 {
   "name": "textopt",
-  "version": "0.0.0",
-  "description": "Prompt optimization for TypeScript, with GEPA, OPRO, MIPRO, and random search behind a shared interface",
+  "version": "0.2.0",
+  "description": "Prompt optimization for TypeScript, with GEPA, SIMBA, OPRO, MIPRO, bootstrapped few-shot search, and random search behind a shared interface",
   "keywords": [
     "prompt-optimization",
     "gepa",
+    "simba",
     "mipro",
     "opro",
     "llm",
@@ -25,7 +26,7 @@
   "type": "module",
   "sideEffects": false,
   "engines": {
-    "node": ">=20"
+    "node": ">=22"
   },
   "publishConfig": {
     "access": "public"
@@ -126,7 +127,8 @@
   "module": "./dist/index.mjs",
   "types": "./dist/index.d.cts",
   "files": [
-    "dist"
+    "dist",
+    "docs"
   ],
   "scripts": {
     "build": "tsdown",