textopt 0.1.0 → 0.2.0

package/docs/evaluation.md

@@ -0,0 +1,87 @@
+# Measuring a result
+
+Two questions a finished run cannot answer about itself: how much of its score
+is fitted to the set that picked the winner, and whether it beat another
+optimizer by more than noise.
+
+## What a run says about itself
+
+`result.warnings` is what a run could see about its own measurement and its
+numbers could not say. It is never fatal and never empty of meaning: an entry is
+a reason to read `bestScore` as less than it appears.
+
+| Code                          | What it means                                                                                                                |
+| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| `validationSetReusesTraining` | No `validationSet` was given, so selection ran on the instances reflection read. Pass one, or `"reuseTraining"` to accept it |
+| `seedScoreSaturated`          | The seed already scores perfectly on every validation instance, so every proposal ties and acceptance resolves noise         |
+| `seedScoreFloored`            | The seed scores 0 on every validation instance — a seed with everything to gain, or a metric that scores nothing             |
+
+The last two are read once, off the seed's own validation row, before the search
+spends anything. Both describe a metric that does not separate the instances it
+is being asked to rank candidates by, which no amount of search budget fixes.
+
+The same list rides on the `finish` event, so a reporter writing a score
+somewhere permanent writes the caveat beside it rather than leaving it in a
+console nobody kept.
+
+## Held-out evaluation
+
+The optimizer selects candidates against `validationSet`, so `bestScore` is fitted to that set and may overstate performance on unseen data.
+
+Pass a `testSet` and the winner is scored on it once, after the search is over:
+
+```ts
+const result = await optimizer.optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  testSet, // never seen by the search
+  adapter,
+  reflect,
+  maxMetricCalls: 300,
+});
+
+result.bestScore; // on the validation set — the search selected for this
+result.testScore; // on instances no candidate was ever selected against
+result.testMetricCalls; // charged separately, not against maxMetricCalls
+result.testUsage; // and costed separately, outside maxCostUsd
+```
+
+The gap between `bestScore` and `testScore` estimates validation overfitting. Test rollouts are reported separately and are outside every ceiling: they do not count against `maxMetricCalls`, and their tokens are in `testUsage` rather than `usage`, because the sweep runs after the search has already stopped. Budget for it the way you would budget for one full validation sweep. The resume fingerprint ignores `testSet`, so it can be added when resuming a run.
+
+All optimizers expose these fields through `OptimizerTask` and `OptimizerResult`.
+
+## Reading the winning text
+
+`result.bestCandidate` is a prompt. That it is readable is the entire advantage over tuning weights, and it is the check no metric performs. Look for:
+
+- **Absorbed facts.** Specific names, numbers, or dates from validation instances written into the instruction. That is memorisation, and it will not transfer, whatever `testScore` said.
+- **Accreted rules.** A long tail of narrow "if the input mentions X, do Y" clauses is a search patching instances rather than learning the task.
+- **Instructions that contradict.** Reflective search appends; nothing prunes.
+
+If the prompt looks wrong and the number looks good, believe the prompt.
+
+## Comparing optimizers
+
+A difference in means over a handful of seeds is usually noise. `compare()` runs each entrant over the same seeds, ranks them on `testScore` where a run reports one, and reports a paired sign-flip p-value against the winner:
+
+```ts
+import { compare } from "textopt";
+
+const comparison = await compare({
+  seeds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
+  concurrency: 4,
+  entrants: {
+    gepa: ({ seed }) => new GepaOptimizer({ seed }).optimize(task()),
+    opro: ({ seed }) => new OproOptimizer({ seed }).optimize(task()),
+  },
+});
+
+comparison.winner; // highest mean score
+comparison.summaries; // mean, sd, min, max, rollouts, cost, pValueVsWinner
+comparison.runs; // every individual run
+```
+
+Entrants are functions of a seed, not optimizer instances: the seed is constructor config, and every optimizer here is deterministic given one, so comparing two entrants at a single seed compares two anecdotes. Build a fresh task inside each entrant — a shared reflection model with internal state would make each result depend on the runs before it.
+
+Ranking on `testScore` matters. The validation score is the number the search selected against for its whole run, so an entrant that overfits looks strongest on exactly the number it fitted.

package/docs/metric-preflight.md

@@ -0,0 +1,132 @@
+# Validating a metric before a search runs
+
+Three things have to be true before any number a run reports means anything:
+
+- the splits must not leak — see [Preparing the data](./data-prep.md)
+- the metric must have range: it must separate candidates, and move over an
+  interval wide enough to see
+- its noise must be smaller than the effect you are looking for
+
+They are preconditions, not a checklist. Everything downstream — which
+optimizer, which minibatch size, which acceptance test — is reasoning stacked
+on an instrument, and reasoning about the instrument is cheaper than every
+argument it supports. All three are checkable in an hour, without an optimizer.
+
+The four checks below establish them, in the order to run them. Do not start a
+real run until they pass.
+
+## 1. Discrimination on known controls
+
+The highest-value check and the one almost nobody runs.
+
+Write three to five candidates whose ranking you already know:
+
+- the seed
+- one deliberately sabotaged — drop the constraint you care most about
+- one hand-tuned, as good as you can write by hand
+- optionally: an empty instruction, and one that is verbose but wrong
+
+Score each on a slice of the validation set and compare the ranking to your
+own. If the metric does not reproduce it, no search will find anything. It
+will wander, accept noise, and report a number.
+
+This is also the fastest way to discover reward hacking before the search
+does. A verbose-but-wrong candidate that outranks a terse correct one tells you
+the judge is paying for length.
+
+## 2. Test-retest variance
+
+Score the _same_ candidate over the _same_ instances three times, with caching
+off. The per-instance spread is the metric's own noise.
+
+That number decides two things the defaults cannot decide for you:
+
+- whether `pairedPermutationAcceptance` and `lowerBoundEvaluationPolicy` are
+  worth their cost. Turn them on when you have measured large variance, not on
+  principle — on the benchmark's pipeline task the pair loses to plain GEPA at
+  p = 0.027, because a significance bar on every acceptance is expensive when
+  improvements to one component only pay off after another is finished
+- how wide a minibatch has to be to say anything. A sign-flip test over n
+  instances bottoms out at a p-value of 2^-n
+
+Deterministic metric, zero variance: keep the defaults.
+
+## 3. Spread, not ceiling or floor
+
+Look at the distribution of the seed's per-instance scores, not its mean.
+
+- **Everything at the ceiling.** Nothing to improve; every proposal ties and
+  acceptance resolves ties by whatever the noise did. A run reports
+  `seedScoreSaturated`.
+- **Everything at zero.** Ambiguous. It is what a seed with everything to gain
+  looks like _and_ what a broken metric looks like, and the score cannot tell
+  them apart — check 1 can. Reported as `seedScoreFloored`.
+- **Everything identical in the middle.** The metric is measuring something the
+  candidate does not control.
+
+What you want is real per-instance variation. GEPA's frontier is taken over
+validation instances, so a seed row with no spread makes the frontier
+degenerate before the search starts.
+
+Then ask the same question of the scale itself. A judge's aggregate is bounded
+by 0 and 1, but the interval it actually moves in on realistic candidates is
+usually narrower, and it is the realized span — not the theoretical one — that
+every later number is denominated in. Score your controls from check 1, take
+the highest and the lowest, and write the difference down. If the best
+candidate you can write by hand scores 1.0 and the sabotaged one scores 0.6,
+your instrument has a span of 0.4, and an improvement that reads as "+0.02" is
+5% of the available range rather than 2% of it.
+
+A narrow span is not automatically wrong, but it is always worth an
+explanation. The common cause is a requirement enforced twice: see the note on
+`gate` below.
+
+## 4. Agreement with your own labels
+
+Hand-label 30–50 outputs — good, bad, and the ambiguous middle — then have the
+judge grade the same ones and compare.
+
+If the judge cannot reproduce your labels, the search is optimizing the judge's
+idiosyncrasies rather than the task. That is still optimization, and the number
+will still climb.
+
+Use the disagreements to rewrite the criteria. A criterion the judge and you
+read differently is a criterion description problem, not a model problem.
+
+## What to fix, in order
+
+| Finding                                 | Fix                                                                     |
+| --------------------------------------- | ----------------------------------------------------------------------- |
+| Ranking of controls is wrong            | The criteria. Nothing downstream is worth doing until this passes       |
+| Judge rewards length or confident tone  | Add an explicit brevity criterion; make correctness the higher `weight` |
+| A hard requirement can be averaged away | Give it a `gate`, and keep its `weight` low — see below                 |
+| Realized span is much narrower than 0–1 | Look for a requirement enforced twice, by a `gate` and a heavy `weight` |
+| Variance is large                       | Widen the minibatch first, then consider the significance guards        |
+| Seed saturated                          | A harder validation set, or a metric that separates these instances     |
+| Seed floored and controls do not rank   | The metric is broken, not the seed                                      |
+| Judge disagrees with your labels        | Rewrite the criterion descriptions; re-check against the same labels    |
+
+## Enforce a requirement once
+
+`gate` and `weight` are different instruments and it is tempting to reach for
+both. A gate makes a requirement non-negotiable: below the threshold the
+instance scores 0 whatever else it did. Once a gate is doing that, a heavy
+weight on the same criterion buys nothing — every candidate that clears the
+gate satisfies the requirement, so the weight is only redistributing score
+among candidates that already agree.
+
+What it costs is range. A heavy weight on a criterion nearly every surviving
+candidate maxes out pins that share of the aggregate near its ceiling, and the
+span left for the search to move in is whatever the other criteria carry.
+Gate the requirement, then weight it low, and let the weights go to the
+criteria that actually vary.
+
+The general form: when a second mechanism starts enforcing a constraint, check
+whether the first still needs to.
+
+## A note on what this is
+
+This is a manual procedure, not a library function. `createJudge` and the
+optimizers give you the pieces; the judgement is yours. If you run these checks
+often against the same task, the natural thing is to script them — a few
+`adapter.evaluate` calls over a fixed batch and a printed table.

package/docs/optimizers.md

@@ -0,0 +1,293 @@
+# Optimizers
+
+What each search reads, and what its knobs do. See [Tuning a run](./tuning.md)
+for what a run costs and [Benchmark](./benchmark.md) for how the six compare on
+offline tasks.
+
+## Choosing an optimizer
+
+All six use the same `Optimizer` interface, adapter, and budget accounting.
+
+| Optimizer                  | Required signal                   | Use it for                                                        |
+| -------------------------- | --------------------------------- | ----------------------------------------------------------------- |
+| `GepaOptimizer`            | per-instance **textual feedback** | revising text from written explanations of failures               |
+| `SimbaOptimizer`           | per-instance **textual feedback** | noisy metrics, where the same instance scores differently per run |
+| `OproOptimizer`            | a **scalar** score                | proposing text from a history of scored attempts                  |
+| `MiproOptimizer`           | a **scalar** score                | searching combinations of interacting component options           |
+| `BootstrapSearchOptimizer` | a **scalar** score                | few-shot demonstrations, with no proposal model at all            |
+| `RandomSearchOptimizer`    | a **scalar** score                | establishing a score-independent paraphrasing baseline            |
+
+Use GEPA when the metric can explain failures in text. A scalar such as `0.0` gives its reflection step little useful information. OPRO only needs scalar scores: its prompt lists previous attempts by score and asks the model to improve on them.
+
+SIMBA also needs textual feedback, but reads a different signal. Instead of reflecting on a failure, it runs several programs over the same instance and reflects on the _contrast_ between the best and worst run of it — a comparison with the input held fixed. That contrast has to exist to be worth paying for, so SIMBA earns its extra rollouts on metrics that vary between runs and wastes them on ones that do not.
+
+GEPA and OPRO update components separately. MIPRO instead searches combinations of per-component options, which lets it find options that work well only together. It screens configurations on minibatches and evaluates promising ones against the full validation set.
+
+MIPRO's default multivariate TPE models complete configurations. Set `multivariate: false` to model each component independently; that usually needs fewer observations but cannot model interactions between components.
+
+`BootstrapSearchOptimizer` writes no text at all. It harvests demonstrations from rollouts the metric already rewarded and searches over which set of them to keep, so it needs no proposal model and no textual feedback. Try it first when the instruction is roughly right and the output format is not.
+
+`RandomSearchOptimizer` paraphrases components without using their scores. Run it as a baseline to check whether reflection improves enough to justify its model calls.
+
+```ts
+import { SimbaOptimizer } from "textopt/simba";
+import { OproOptimizer } from "textopt/opro";
+import { MiproOptimizer } from "textopt/mipro";
+import { BootstrapSearchOptimizer } from "textopt/bootstrap-search";
+import { RandomSearchOptimizer } from "textopt/random-search";
+```
+
+[Benchmark](./benchmark.md) reports what each search does on four offline tasks; read it as evidence about the search rather than about your task, and run [`compare()`](./evaluation.md#comparing-optimizers) on your own metric before choosing.
+
+## How GEPA works
+
+GEPA maintains a pool of candidates and a **Pareto frontier over validation instances**, not objectives. A candidate remains on the frontier while it has the best score for at least one instance. Parent sampling is weighted by the number of instances each candidate wins, preserving candidates with useful strengths even when their mean score is lower.
+
+Each iteration:
+
+1. **Select** a parent from the frontier, and one or more of its components to update.
+2. **Evaluate** the parent on a fresh minibatch, capturing traces.
+3. **Reflect** on per-component evidence from the scored batch: inputs, outputs, feedback, and scores. Recent rejected proposals are included to reduce repetition.
+4. **Screen** the child on the same minibatch. Only improvements proceed.
+5. **Sweep** accepted children over the validation set and update the frontier.
+6. **Merge** lineages that improved different components. Merge is enabled by default for multi-component candidates.
+
+`maxMetricCalls` limits scored rollouts; cache hits do not count. Reflection calls have a separate limit.
+
+Based on _GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning_.
+
+## Configuring GEPA
+
+Pass search settings to the `GepaOptimizer` constructor:
+
+```ts
+new GepaOptimizer({
+  minibatchSize: 3,
+  maxIterations: 50,
+  seed: 11,
+  proposals: { perIteration: 3, concurrency: 3, selection: "best" },
+  reflection: { maxCalls: 40, maxRecords: 5, maxCharacters: 20_000 },
+  // ...or `strategies: diverseReflectionStrategies()` in place of one prompt
+  merge: { enabled: true, maxInvocations: 5 },
+  candidateSelector: paretoSelector(),
+  acceptance: improvementAcceptance(),
+  skipPerfectScore: true,
+  rejectedProposalMemory: 3,
+  trackBestOutputs: true,
+});
+```
+
+`optimize` takes a `GepaTask`. Required fields are `seedCandidate`, `trainingSet`, `adapter`, `reflect`, and `maxMetricCalls`. `validationSet` defaults to `trainingSet`; `testSet` is optional and held out. Other options are `componentSelector`, `batchSampler`, `valEvaluationPolicy`, `instanceId`, `cache`, `reporters`, `onCheckpoint`, `resumeFrom`, and `signal`.
+
+Component names are inferred from `seedCandidate`. Other positions use `NoInfer`, so misspelled component names fail type checking.
+
+`proposals.perIteration` is the setting that moves the bill: each proposal is priced at two minibatch evaluations, its parent's and its own, so raising it from one to three triples what an iteration costs before any child is swept ([Sizing a run](./tuning.md#sizing-a-run)). What that buys is width — every slot draws its own parent and its own minibatch, so the slots diagnose different failures and `concurrency` can run them at once, and `selection: "best"` then promotes only the strongest improving child of the batch. Pair it with `reflection.strategies`, which is what keeps two slots that landed on the same parent from writing the same revision.
+
+`textopt/gepa` exports the following strategies: `paretoSelector`, `currentBestSelector`, `epsilonGreedySelector`, `topKParetoSelector`, `roundRobinComponentSelector`, `allComponentsSelector`, `improvementAcceptance`, `pairedPermutationAcceptance`, `fullEvaluationPolicy`, `subsampledEvaluationPolicy`, and `lowerBoundEvaluationPolicy`.
+
+### Proposal strategies
+
+By default, every proposal uses the same reflection prompt. Multiple calls against one parent can therefore produce similar revisions.
+
+`reflection.strategies` rotates over several framings instead, one per proposal slot:
+
+```ts
+import { diverseReflectionStrategies } from "textopt/gepa";
+
+new GepaOptimizer({
+  proposals: { perIteration: 4, concurrency: 4 },
+  reflection: { strategies: diverseReflectionStrategies() },
+});
+```
+
+The included rotation alternates the standard prompt with prompts that simplify, generalize, or rewrite the candidate. The builders are exported as `buildReflectionPrompt`, `buildSimplifyPrompt`, `buildGeneralizePrompt`, and `buildRewritePrompt`; custom `ReflectionPromptBuilder` functions use the same interface.
+
+This behavior is opt-in. The default is `buildReflectionPrompt`, adapted from the GEPA paper's reflection prompt rather than copied from it: it carries the same evidence and asks for the same thing, but tags and wording differ from the reference implementation's template, so proposals drawn from identical evidence will not match it.
+
+## Few-shot demos
+
+Few-shot examples can be stored in a candidate component and optimized with the other text. textopt can populate that component from successful training rollouts.
+
+Before optimization, `harvestFewShotExamples` evaluates the seed candidate on `trainingSet` and keeps high-scoring rollouts:
+
+```ts
+import { harvestFewShotExamples } from "textopt";
+
+const { block, demos, metricCalls } = await harvestFewShotExamples({
+  adapter,
+  candidate: seedCandidate,
+  trainingSet,
+  minScore: 0.9,
+  maxDemos: 4,
+});
+
+const seed = { instruction: "Route the ticket.", demos: block };
+```
+
+During optimization, `createDemoProposer` reads demos from the existing reflective dataset without additional rollouts or reflection calls:
+
+```ts
+import { createDemoProposer } from "textopt/gepa";
+
+const adapter = {
+  ...baseAdapter,
+  proposeNewTexts: createDemoProposer({
+    components: ["demos"],
+    minScore: 0.9,
+    maxDemos: 4,
+    fallback: baseProposer, // writes the other components normally
+  }),
+};
+```
+
+Each proposal appends demos to its parent's block. A demo remains in the lineage only when the candidate containing it is accepted.
+
+## The other optimizers
+
+Apart from SIMBA, these use the base `Adapter`: they need no reflective dataset. Their results include both `seedScore` and `bestScore`.
+
+### SIMBA
+
+```ts
+import { SimbaOptimizer } from "textopt/simba";
+
+const result = await new SimbaOptimizer({
+  minibatchSize: 16,
+  candidates: 4,
+  maxSteps: 8,
+  seed: 11,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter, // the base Adapter — no makeReflectiveDataset needed
+  reflect,
+  demoComponents: ["demos"], // optional; enables the appendDemo mutation
+  maxMetricCalls: 900, // 250 reserved for finalists, 80 a step, over 50 validation instances
+});
+
+result.finalists; // the step winners, scored on the full validation set, best first
+```
+
+Each step samples several programs from the pool over one minibatch, ranks the instances by how much those programs disagreed, and mutates toward whichever run won. Two mutations are drawn at random per instance:
+
+- **`appendDemo`** keeps the winning rollout as a few-shot example. Costs no model call. Requires `demoComponents`.
+- **`appendRule`** shows the better and worse run of the same instance to `reflect` and appends the advice it returns to each instruction component.
+
+Neither replaces text, so candidates accumulate; demonstrations are dropped at a Poisson rate so a growing block cannot crowd out the instruction. `strategies` pins the mutation to one of the two.
+
+Ported from DSPy's SIMBA with two deliberate changes. A trajectory sample runs one program across the whole minibatch rather than resampling a program per instance, because the adapter owns decoding here and there is no temperature knob to vary — the variability comes from the program pool instead. And the percentile guards are strict rather than inclusive, so a step on which every rollout ties still produces a mutation instead of doing nothing at all.
+
+Only the finalists are scored on the full validation set: the step winners are sampled evenly across the run, so early winners stay in the running. Those rollouts are reserved before the search starts, which is why a small `maxMetricCalls` buys fewer steps than the arithmetic suggests — see [Sizing a run](./tuning.md#sizing-a-run) for what the reserve costs.
+
+Its batch defaults are wide on purpose: a step reads the disagreement between programs over a batch, and a narrow batch leaves little to rank. When a run has to get cheaper, lower `candidates` first — it shrinks both the step and the reserve, where `minibatchSize` shrinks the step alone and narrows the batch the ranking reads.
+
+`concurrency` overlaps the two places a step's work is independent: scoring the candidates it built, and the finalist sweeps at the end. The trajectory samples and the mutations that read them stay in sequence — each reads what the one before it wrote — so the ceiling on what it buys is the reserve and the scoring, not the whole step. See [Concurrency](./tuning.md#concurrency).
+
+### Bootstrapped few-shot search
+
+```ts
+import { BootstrapSearchOptimizer } from "textopt/bootstrap-search";
+
+const result = await new BootstrapSearchOptimizer({
+  candidates: 16,
+  maxDemos: 4,
+  seed: 11,
+}).optimize({
+  seedCandidate: { instruction, demos: "" },
+  trainingSet,
+  validationSet,
+  adapter,
+  demoComponents: ["demos"],
+  goldOutput: (datum) => datum.answer, // optional; enables the labels-only candidate
+  maxMetricCalls: 1500, // 19 candidates swept over 50 validation instances, plus their harvests
+});
+
+result.candidates; // every set tried, with its source and how many demos it held
+```
+
+DSPy's `BootstrapFewShotWithRandomSearch`. It calls no model to write text: every candidate is assembled from outputs the system itself produced, so the search costs rollouts and nothing else. The fixed candidates come first — zero-shot, then labels-only when `goldOutput` is given, then one unshuffled full-size harvest — followed by shuffled harvests of random size.
+
+Zero-shot stays in the running throughout. Demonstrations can hurt, and a search that cannot return "no demos" has no baseline to report against.
+
+`concurrency` overlaps a candidate's sweep with the harvest of the candidates behind it, which is most of the wall clock on a run whose harvests are as expensive as its sweeps. Harvesting itself stays in plan order, because every harvest draws from the same random stream. It is ignored when `stopAtScore` is set, and it moves checkpointing from once per candidate to once per wave — see [Concurrency](./tuning.md#concurrency).
+
+`candidates` is the whole search: each one is a fresh harvest and a full sweep, so it sets both the breadth and the bill. `demoMinScore` is the knob that surprises — a strict threshold does not cost less, it costs more, because a harvest keeps rolling out training instances until it has collected `maxDemos` of them or run out of set. If harvests come back with no demos, the seed cannot yet produce work its own metric rewards, and few-shot search is the wrong tool until it can.
+
+### OPRO
+
+```ts
+const result = await new OproOptimizer({
+  proposalsPerRound: 4,
+  historySize: 10,
+  maxReflectionCalls: 40,
+  seed: 11,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter,
+  reflect,
+  maxMetricCalls: 2000, // 4 proposals a round, each swept over 50 validation instances
+});
+
+result.trajectory; // every candidate scored, in the order it was tried
+```
+
+By default, every proposal is scored on the full `validationSet`. With `scoringSetSize`, proposals are screened on a fixed subset of `trainingSet`, and the incumbent receives a full sweep every `fullEvalInterval` rounds. In a 30-instance validation set, screening on 12 instances halved rollout count without reducing the measured best score.
+
+The meta-prompt lists the strongest attempts in ascending score order, placing the best attempt nearest the request. `scoreScale` converts scores to integers (100 by default), because models distinguish values such as 41 and 68 more reliably than 0.41 and 0.68.
+
+Because every proposal in a round sees the same history and is screened against the same incumbent, `concurrency` runs a whole round — proposals and screens together — at once, and the round still records its attempts in the order it drew them.
+
+Rounds are where OPRO gets its signal: every proposal in a round sees the same history, so `proposalsPerRound` widens a round rather than deepening the search, and the history a later prompt reads only grows between rounds. A run also moves one component per round, in turn, so a two-component candidate needs twice the rounds to revise each as often. Budget for rounds first, then set `proposalsPerRound` to what a round can afford — `maxReflectionCalls` caps the two together, at `maxReflectionCalls / proposalsPerRound` rounds.
+
+### MIPRO
+
+```ts
+const result = await new MiproOptimizer({
+  instructionsPerComponent: 5,
+  maxTrials: 30,
+  seed: 11,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter,
+  reflect,
+  demoComponents: ["demos"], // menu bootstrapped from the training set
+  maxMetricCalls: 1600, // 30 trials of 35, six sweeps of 50, and the demo harvests
+});
+
+result.menu; // the space that was searched, per component
+result.observations; // every configuration tried, and which earned a full sweep
+```
+
+MIPRO first builds a menu for each component from the seed text and variants generated by `reflect`. A TPE surrogate then proposes configurations from those menus. Trials run on minibatches; selected configurations receive a full validation sweep.
+
+`componentOptions` adds menu entries without reflection calls. `demoComponents` builds menus of few-shot blocks from successful training rollouts, allowing instructions and demonstrations to be searched together.
+
+Every `fullEvalInterval` trials, MIPRO fully evaluates the unswept configuration with the highest average minibatch score. Averaging repeated observations reduces the effect of a lucky minibatch.
+
+The space is the product of the menus, so it grows multiplicatively with components while `maxTrials` grows by hand — three components of six options each is 216 configurations, and the default thirty trials sees a seventh of them. The surrogate also spends its first ten trials sampling at random before it models anything, so a short run is mostly random search with extra steps. Give it trials in proportion to the menu, or trim the menu with `instructionsPerComponent`.
+
+### Random search
+
+```ts
+const result = await new RandomSearchOptimizer({
+  variants: 4,
+}).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  adapter,
+  reflect,
+  maxMetricCalls: 1000, // 4 variants a round, each swept over 50 validation instances
+});
+```
+
+Random search paraphrases one component per round and keeps the highest-scoring candidate. Its prompt receives no performance data. Compare it with a reflective optimizer under the same metric budget to measure the benefit of reflection.
+
+`concurrency` runs a round's variants at once, proposal and sweep together, and is the one setting here that changes nothing about the search — the round still accepts a variant only if it beat every variant drawn before it.
+
+Being a baseline is the whole configuration: give it the `maxMetricCalls` and `validationSet` of the run it stands against, and change nothing else. A baseline on a smaller budget answers a different question than the one being asked of it. [`compare()`](./evaluation.md#comparing-optimizers) runs both over the same seeds and ranks them on the held-out score, but each entrant builds its own task, so keeping the budgets equal is still yours to do.