orizu 0.5.19 → 0.5.21

package/vendor/skills/orizu-cli/references/migrate-existing-gepa-setup.md

@@ -0,0 +1,303 @@
+# Migrating an existing GEPA setup into Orizu
+
+For a customer already running official GEPA from a plain script — inline
+`trainset`/`valset`, a seed prompt in a dict, their own `metric(row, output)`.
+
+The ordering rule: **nothing is optimized until parity is proven.** The last
+step before `run-gepa` is always `orizu scorers verify-parity`, which runs the
+migrated scorer runner under the exact payload `run-gepa` sends it and the
+customer's original metric over the same rows, and exits 0 only if they agree.
+
+Commands are shown with `--local`; drop it against a hosted server.
+
+## 0. Read their script
+
+Note the row shape and which field is the stable id, `trainset`/`valset`
+membership, the seed candidate text, the metric's `module:function`, and what
+it returns (a float, a dict with `score`, or GEPA's `EvaluationResult` — all
+three are accepted; `objective_scores` is ignored). If rows have no stable id,
+add one now: every step below pairs on it.
+
+## 1. Set up, then export the rows
+
+```bash
+orizu setup --team <team-slug>
+```
+
+One dataset holds every row; the split set decides train vs validation. Dump
+rows verbatim from their script, and mirror their lists exactly — a resampled
+split makes every later number incomparable with the numbers they have.
+
+Build the dataset from the **id-deduplicated union**: a row reused in both
+`TRAINSET` and `VALSET` would otherwise be written twice, and the upload route
+rejects duplicate canonical row ids (`app/api/datasets/route.ts:206-211`,
+"csvData must contain unique canonical row ids"). The partition lists keep the
+original ids independently, so a shared row can still appear in both.
+
+```python
+import json
+seen, rows = set(), []
+for row in TRAINSET + VALSET:
+    if row["id"] in seen:
+        continue
+    seen.add(row["id"])
+    rows.append(row)
+with open("dataset.jsonl", "w") as handle:
+    for row in rows:
+        handle.write(json.dumps(row) + "\n")
+with open("split.json", "w") as handle:
+    json.dump({"name": "default", "strategy": "predefined", "seed": 42, "partitions": [
+        {"name": "train", "row_ids": [row["id"] for row in TRAINSET]},
+        {"name": "validation", "row_ids": [row["id"] for row in VALSET]},
+        {"name": "test", "row_ids": []},
+    ]}, handle)
+```
+
+## 2. Upload, version, split
+
+```bash
+orizu --local datasets upload --file ./dataset.jsonl \
+  --project <team>/<project> --name "GEPA migration rows" --json
+
+orizu --local datasets versions create <dataset-id-or-name> \
+  --project <team>/<project> --label v1 --json      # -> dataset_version_id
+
+orizu --local datasets splits create <dataset-version-id> \
+  --from-file ./split.json --json                   # -> split_set_id
+```
+
+`--from-file` preserves their partitions; the ratio flags
+(`--train/--validation/--test`) resample and must not be used here.
+
+## 3. Candidate runner and seed prompt
+
+The candidate runner does what `task_lm` did: candidate prompt + row -> output.
+
+```bash
+orizu --local runners push ./candidate-runner \
+  --project <team>/<project> --name gepa-candidate-runner --label default --json
+
+orizu --local prompts push ./prompt \
+  --project <team>/<project> --runner-version <candidate-runner-version-id> --json
+```
+
+`./prompt` holds `orizu.prompt.json` plus the body file it names. Only `name`
+is enforced (`app/api/cli/prompts/route.ts:220-221`); `body_file` defaults to
+`prompt.md`, `body_kind` to `text` (`packages/cli/src/index.ts:3966-3985`) and
+`provider_settings` to `{}`. Write all four anyway — a migration whose model
+settings are implicit is a migration nobody can reproduce:
+
+```json
+{
+  "name": "gepa-seed-prompt",
+  "body_file": "prompt.md",
+  "body_kind": "text",
+  "provider_settings": { "model": "claude-sonnet-4-6", "temperature": 0 }
+}
+```
+
+`prompt.md` carries their seed candidate text verbatim. Returns the seed
+`prompt_version_id`.
+
+## 4. Wrap their metric in a scorer runner
+
+Do not rewrite the metric — wrap it, or parity is unachievable by construction.
+`scorer-runner/manifest.json` declares
+`"command": ["python3", "runner.py"]` and `"scorer_input_contract": "flat_row"`.
+Under `flat_row`, `input["row"]` is the dataset row with the candidate output
+injected at `model_output` (or the `--scorer-candidate-field` field) and
+`input["model_output"]` is that same output, so `runner.py` is two lines:
+`result = metric(payload["row"], payload["model_output"])`, then write the score
+to `ORIZU_RUNNER_OUTPUT_PATH`. Extract it the same three ways `verify-parity`
+extracts the original metric's result, or the two sides disagree on a shape step
+0 says is supported:
+
+```python
+score = result["score"] if isinstance(result, dict) else getattr(result, "score", result)
+json.dump({"score": float(score)}, open(os.environ["ORIZU_RUNNER_OUTPUT_PATH"], "w"))
+```
+
+A bare `float(getattr(result, "score", result))` raises `TypeError` on the dict
+form, failing every row here and again under `run-gepa`.
+
+**If the metric lives inside their script, extract it first.** The one-file
+shape in step 0 usually defines `metric` inside `optimize.py` next to
+`import gepa`; vendoring that file into the runner drags the `gepa` import into
+the scrubbed runner environment and every row raises `ModuleNotFoundError`. Move
+the function to a `metric.py` with no optimizer imports and have `optimize.py`
+do `from metric import metric` — their script keeps working unchanged, the
+runner copy carries no dependencies, and `--original metric:metric` is valid for
+both. (`docs/requirements/official-gepa/stranger/` ships this two-file shape.)
+
+**Copy the metric module into `./scorer-runner/` before pushing.** The runner
+executes with its working directory set to the *materialized runner version* and
+a scrubbed environment, so the `metric.py` sitting next to their `optimize.py`
+is not importable — a runner that does `import metric` raises
+`ModuleNotFoundError` on every row, here and again under `run-gepa`. Vendor the
+file (or the whole package) into the runner directory so the registered runner
+bytes contain everything the metric needs:
+
+```bash
+cp ./metric.py ./scorer-runner/metric.py     # or: cp -R ./their_package ./scorer-runner/
+```
+
+`verify-parity` still imports the customer's ORIGINAL metric from the current
+directory, so the two copies must be the same file — re-copy whenever it changes,
+or the parity you prove is against a stale runner.
+
+A DSPy metric is `metric(example, pred)`; the 3-line adapter is:
+
+```python
+def metric(row, model_output):                      # what verify-parity calls
+    import dspy
+    return dspy_metric(dspy.Example(**row), dspy.Prediction(answer=model_output))
+```
+
+```bash
+orizu --local runners push ./scorer-runner \
+  --project <team>/<project> --name gepa-scorer-runner --label default --json
+
+orizu --local prompts push ./scorer-prompt \
+  --project <team>/<project> --runner-version <scorer-runner-version-id> --json
+
+orizu --local scorers register --project <team>/<project> \
+  --name gepa-migrated-metric --manifest ./scorer.manifest.json \
+  --prompt-version <scorer-prompt-version-id> \
+  --runner-version <scorer-runner-version-id> \
+  --label production --json                        # -> scorer_version_id
+```
+
+`scorer.manifest.json` needs all four fields below;
+`app/api/cli/scorers/route.ts` rejects a manifest missing `implementation_kind`
+or `metric_key`, and `mode` must be `row` or `set`:
+
+```json
+{
+  "name": "gepa-migrated-metric",
+  "mode": "row",
+  "implementation_kind": "prompt_runner",
+  "metric_key": "score"
+}
+```
+
+`prompt_runner` is the only kind `runners exec --scorer-version` and
+`verify-parity` can execute, and it requires a backing prompt version. For a
+pure-code metric with no judge text, `./scorer-prompt` is a short prompt
+version documenting the rubric; the runner never reads its body.
+
+## 5. Produce `outputs.jsonl` by running the seed once
+
+A scorer scores *(row, output)*, so parity needs one candidate output per row.
+
+```bash
+orizu --local runners exec \
+  --prompt-version <seed-prompt-version-id> \
+  --runner-version <candidate-runner-version-id> \
+  --dataset-version <dataset-version-id> \
+  --split-set <split-set-id> --split validation \
+  --out ./results.jsonl
+
+jq -c '{row_id: .row_id, model_output: .model_response}' ./results.jsonl > ./outputs.jsonl
+```
+
+`outputs.jsonl` must cover exactly the rows being checked: a missing or extra
+`row_id`, a duplicate, or a non-string `model_output` is refused before
+anything runs.
+
+## 6. Prove parity
+
+```bash
+orizu --local scorers verify-parity \
+  --scorer-version <scorer-version-id> \
+  --dataset-version <dataset-version-id> \
+  --split-set <split-set-id> --split validation \
+  --outputs ./outputs.jsonl \
+  --original metric:metric \
+  --json
+```
+
+- `--original <module>:<function>` is imported from the **current directory**,
+  so run this where their script lives.
+- The input contract and candidate field resolve exactly as `run-gepa` resolves
+  them: flag > runner manifest (`scorer_input_contract`,
+  `candidate_output_field`) > defaults (`gepa`, `model_output`). Because a
+  `gepa`-contract runner is fed a different payload shape entirely,
+  verify-parity refuses anything that does not resolve to `flat_row` — declare
+  `"scorer_input_contract": "flat_row"` in the runner manifest (step 4) or pass
+  `--scorer-input-contract flat_row`, and pass the SAME value to `run-gepa`.
+- `--scorer-candidate-field <field>` when the judge reads the candidate from a
+  row field other than `model_output` — the same value you pass to `run-gepa`.
+  It may not be blank.
+- `--tolerance <float>` defaults to `0` (exact). Raise it only for a genuinely
+  non-deterministic metric, and say so in the report.
+- `--python <cmd>` picks the interpreter (else `PYTHON`, else `python3`). The
+  metric runs in the customer's own unscrubbed environment, so an LLM-judge
+  metric still sees its API keys.
+- `--runner-dir <dir>` runs local scorer-runner bytes, but only if they hash to
+  the registered runner version; drifted bytes exit 2.
+- `--limit <n>` checks the first N rows of the partition — a smoke check, not
+  the proof: it always reports `parity: false` with `scope: {compared, total}`,
+  so a limited run can never be mistaken for a full one. The same full
+  `outputs.jsonl` works: outputs must cover the limited rows and belong to the
+  split, and need not be trimmed to N.
+- A repeated option takes its LAST value, matching run-gepa's scalar options.
+- If a sampled row already has the candidate field as a key, the command exits 2
+  rather than overwriting a value the original metric reads.
+
+Exit `0` parity proven (>= 1 row compared, no mismatch, no row error); `1` at
+least one mismatch or row error — read `mismatches`/`errors` in the `--json`
+report, fix the runner, repeat; `2` the check could not run (bad arguments, no
+rows, unverified runner dir, original metric not importable).
+
+Do not continue on exit 1 or 2. A scorer that disagrees with their metric
+optimizes the wrong thing, and every number afterwards is wrong.
+
+**Prove parity on BOTH partitions before declaring the scorer migrated.**
+`run-gepa` scores `train` and `validation`, and training rows drive GEPA's
+proposals: a scorer that agrees on validation but diverges on a training row
+that exercises a different schema branch changes the experiment's result while
+passing this gate. Repeat steps 5 and 6 with `--split train` (its own
+`outputs.jsonl` from the same seed run) and require exit 0 on both.
+
+A `parity: true` report means every row of that partition was compared and
+agreed. A limited run reports `parity: false` with
+`scope: {compared, total}` and the line "Smoke check passed on N of M rows"
+— it exits 0, but it is not the proof.
+
+## 7. Optimizer and run
+
+```bash
+orizu --local optimizers push ./optimizer \
+  --project <team>/<project> --name gepa-optimizer --label gepa-v1 --json
+
+orizu --local optimizations run-gepa \
+  --project <team>/<project> \
+  --optimizer-version-id <optimizer-version-id> \
+  --candidate-version-id <seed-prompt-version-id> \
+  --runner-version-id <candidate-runner-version-id> \
+  --candidate-runner-dir ./candidate-runner \
+  --scorer-version-id <scorer-version-id> \
+  --scorer-runner-version-id <scorer-runner-version-id> \
+  --scorer-runner-dir ./scorer-runner \
+  --dataset-version-id <dataset-version-id> \
+  --split-set-id <split-set-id> \
+  --train-split train --val-split validation \
+  --scorer-input-contract flat_row \
+  --reflection-max-tokens 8192 \
+  --engine official
+```
+
+`--reflection-max-tokens` is not optional here: with `--engine official` the
+reflection model defaults to `anthropic/claude-opus-4-7`, and every non-`openai/`
+reflection model is refused at launch without it
+(`packages/cli/src/gepa-engine-dispatch.ts:301-304`, again at
+`packages/orizu-gepa/src/orizu_gepa_connector/runtime.py:193-196`). Drop it only
+if you also pass an `openai/...` `--reflection-model`.
+
+`./optimizer` needs `manifest.json` with `"optimizer_family": "gepa"`. See
+`references/optimization-with-gepa.md` for reflection-model and budget flags.
+Pass `--scorer-candidate-field <field>` here too if you passed it to
+`verify-parity`: the payload the runner sees must be the one parity was proven
+under. Afterwards, `orizu optimizations promote <run-id> --candidate <id>
+--label production` promotes the winner — and with parity proven, its score is
+comparable to the seed score their own script reports.

package/vendor/skills/orizu-cli/references/optimization-reports.md

@@ -130,7 +130,7 @@ Set an explicit expectation before the run: for example, "if best kappa does not
 
 ### Budget Versus Iteration Cap
 
-Default budget presets are DSPy-style metric-call budgets. `light`, `medium`, and `heavy` map to DSPy's `auto="light"`, `auto="medium"`, and `auto="heavy"` scales; `auto` is treated as the balanced `medium` preset and is the default when no budget control is provided. The final metric-call limit scales with validation-set size, rather than using fixed tiny constants. `--budget`, `--max-metric-calls`, `--max-full-evals`, and `--max-iterations` are mutually exclusive CLI budget controls; choose only one.
+Default budget presets are DSPy-style metric-call budgets. `light`, `medium`, and `heavy` map to DSPy's `auto="light"`, `auto="medium"`, and `auto="heavy"` scales; `auto` is treated as the balanced `medium` preset and is the default when no budget control is provided. The final metric-call limit scales with validation-set size, rather than using fixed tiny constants. `--budget`, `--max-metric-calls`, `--max-full-evals`, `--max-iterations`, and `--max-candidate-proposals` are mutually exclusive CLI budget controls; choose only one. The proposal cap is available only on the official engine.
 
 Metric-call and full-eval budgets are checked only between iterations. Once an iteration starts, `run-gepa` completes the parent minibatch, child minibatch, and full validation eval for an accepted child even if that overshoots the nominal budget. If a run ends as `budget_exhausted`, it pauses and does not auto-promote. If auto-promotion matters, choose a single budget control large enough to cover the intended search.
 

package/vendor/skills/orizu-cli/references/optimization-with-gepa.md

@@ -2,6 +2,8 @@
 
 How to optimize text candidates against validated judges/scorers. For Orizu-tracked runs, prefer `prompt-control-plane.md` and the bundled `orizu optimizations run-gepa` command when optimizing one text candidate. Use this reference for GEPA mechanics, custom optimizer implementations, and optional DSPy context for customers already using DSPy.
 
+For an instruction set, select the profile explicitly with `--instruction-set <name> --model-config <identity>` rather than `--prompt` or `--prompt-version` (those selectors are mutually exclusive). The connector resolves that model config's production profile, or the set default, and optimizes its complete component map. `--component-selector round-robin` is the default and updates one component per round; `--component-selector all` updates every component per round. A multi-component candidate is sent as a component map, while a set of one keeps the existing prompt-body contract. Git-pinned components and malformed component maps are refused before a run starts; multi-component auto-promotion is deliberately refused until component-map promotion is available.
+
 ## Inputs
 
 You should arrive here with:
@@ -50,7 +52,7 @@ Use the prompt control plane when you want runs, candidates, score charts, Paret
 1. Push the candidate runner and prompt/judge prompt.
 2. Register a row scorer for reflection. GEPA reflection requires row-level feedback.
 3. Snapshot a dataset version and create a train/validation split set.
-4. Use `orizu optimizations run-gepa` for the common text-candidate case, or `orizu optimizations start` plus event logging for a custom optimizer.
+4. Use `orizu optimizations run-gepa` for the common text-candidate case. It uses official GEPA by default; `--engine legacy` is a frozen safety hatch while migration parity is proven. Use `orizu optimizations start` plus event logging for a custom optimizer.
 5. Use set scorers for selection/tracked reporting when the meaningful metric is batch-level; execute builtin set scorers with `orizu scorers exec` or submit precomputed aggregates with `orizu scores submit --aggregate`.
 6. Promote only candidates that passed validation.
 7. Write and attach an optimization report from the local logs or export artifact; see `optimization-reports.md`.
@@ -66,6 +68,29 @@ Reflection output contract:
 
 Full command syntax and event contracts: `prompt-control-plane.md`.
 
+## Migrating from the legacy engine (release cli-v0.5.20+)
+
+Since release `cli-v0.5.20` (`orizu@0.5.20` on npm), `run-gepa` defaults to the official GEPA engine. The flag surface and validation rules match legacy almost everywhere — a valid legacy command runs unchanged, with two exceptions: the reflection-cap requirement below, and stricter `=`-form parsing (`--flag=` with an empty value, or an `=`-attached value that itself starts with `--`, is refused pre-launch; legacy's argparse accepted both — a scripted `--scorer-candidate-field=$FIELD` with `FIELD` unset now fails with the flag named). What actually differs:
+
+- **Validation happens before launch.** Legacy validated flags inside the python process (an argparse exit); the official dispatcher enforces the flag-shape rules below before anything spawns and names the offending flag in the error (the budget-conflict error lists the exclusive set rather than the flags you passed; value errors — e.g. a non-integer or non-positive budget number — are still reported by the engine and name the underlying setting). The rules carried over from legacy: `--budget` / `--candidate-selection-strategy` / `--scorer-input-contract` validate their choices, boolean flags take no value, abbreviated flag spellings are rejected, and budget controls are mutually exclusive — at most one of `--budget`, `--max-metric-calls`, `--max-full-evals`, `--max-iterations`. The official-only `--max-candidate-proposals` joins that exclusion set; passing no budget control defaults to `--budget auto`.
+- **A reflection cap is required for non-OpenAI reflection models.** `--reflection-max-tokens` is required at launch whenever the reflection model does not start with `openai/` — including the default (`anthropic/claude-opus-4-7`) and bare names like `gpt-4o` — so the run refuses immediately instead of failing every reflection call mid-run.
+- **The skilled proposer is an opt-in proposal path.** Reach for `--candidate-proposer skilled-proposer` when you want to evaluate the upstream proposer's alternative reflection process and can accept its first-use network, CPython, and managed-dependency requirements. Evaluate the shipped empty-skills proposer through the controlled A/B below, and ground quality or generalization claims in that evidence. Environment setup and recovery live in `prompt-control-plane.md` ("Skilled proposer").
+- **Seed preflight: same gate, better evidence.** Both engines refuse a degenerate seed (every valid row pinned to the same bound — the worst score, or uniformly `0.0` under a lower-is-better scorer, where the perfect bound is indistinguishable from a scorer silently zeroing on a mismatched input shape) or an all-errored seed at launch, and both accept `--allow-degenerate-seed` to bypass. The bypass covers *both* refusals — including the broken-scorer one — so a bypassed run can spend budget on a scorer that never works; use it only when a weak seed is expected. New on the official engine: the refusal message carries per-row evidence (row ids, scores, scorer output and errors); `<log-dir>/preflight-refused-<uuid>/preflight.json` (not written under `--no-local-log`) instead stores redacted `{row_id, score, error_class}` entries unless `--log-row-snapshots` is passed — a sibling of the run directories, since no run record exists when a launch is refused.
+- **Metric accounting differs.** Official counts include every evaluated row; legacy excluded some cached work. Do not compare raw metric-call totals across engines (details in ADR-019).
+- **Artifacts.** Both engines write `events.jsonl`, `evaluations.jsonl`, `reflections.jsonl`, and `result.json` locally; `lm_stats.json` (reflection usage) is official-only. Dashboard rendering is identical, and run metadata records `engine: official` or `engine: legacy`.
+
+Terminal semantics are unchanged between engines: exhausting any budget control other than `--max-iterations` ends the run `paused` (`pause_reason: budget_exhausted`) with no auto-promotion, while completing the configured `--max-iterations` is a normal `succeeded` finish that may auto-promote. A paused run's validated candidates can still be promoted manually — only automatic promotion is gated on how the run ended. Falling back to `--engine legacy` after a budget-exhausted pause reproduces the same paused outcome; it will not turn the run into a `succeeded` one.
+
+**Fallback:** rerun with `--engine legacy` for the previous engine's exact behavior, dropping `--max-candidate-proposals` first if you used it (it is official-only; the CLI refuses it under legacy before launch). Legacy is frozen (no new features) and will be removed at the M3 milestone (ADR-019) — if you fall back because the official engine misbehaved, capture both run ids and report the pair.
+
+### Compare the skilled proposer with the default
+
+Run two arms with the same candidate prompt/version, dataset version and split set, candidate and scorer runners, seed, reflection model/settings, thread count, and one shared overall budget such as `--max-candidate-proposals 1`. Keep the default arm unselected; add `--candidate-proposer skilled-proposer` plus bounded `--proposal-max-calls` and `--proposal-max-tokens` only to the selected arm. The comparison is ready when both terminal states, retained candidates, scores, proposal evidence, and total run usage are captured. A fixed GEPA seed aligns optimizer sampling but does not make provider or scorer calls deterministic, so treat the pair as process and outcome evidence rather than an intrinsic quality claim.
+
+For the selected arm, each provider-bearing proposal call appends `proposal-observability/events.jsonl`. Success records contain `source`, `status`, `attempt`, `correlation_id`, `provider`, `model`, nullable `request_id`, `latency_ms`, `cache_state`, and `usage.{input_tokens,output_tokens,total_tokens}`. Failure records contain `source`, `status`, `attempt`, `correlation_id`, `provider`, `cache_state`, `failure_code`, and optional `usage`; the latest durable failure is also written to `proposal-failures/latest.json` with `source`, `code`, `detail`, and optional `correlation_id`.
+
+With local logging, those paths sit under `<log-dir>/<run-id>/`, and terminal `proposal-observability/lm_stats.json` is a flat object with `total_tokens_in`, `total_tokens_out`, and `total_tokens`. Under `--no-local-log`, per-call and failure records move under `.orizu/proposal-observability/<run-id>/`, and no terminal `lm_stats.json` is written. Finish the comparison after transport-bearing calls that reached provider completion or failure have records, `proposal_observability_event_failed` is absent, and, when local logging is enabled, terminal usage totals reconcile with those records.
+
 ## Step 1: Wrap your application as an Orizu runner
 
 For Orizu-tracked optimization, the candidate runner receives one dataset row and one candidate text body through the file contract. The scorer runner, by default, receives a GEPA-shaped `row` — `{source_row, candidate_id, candidate_output, candidate_raw_response, candidate_error}` — and returns a score and feedback. See `prompt-control-plane.md` ("Scorer-Runner Input Contracts") for the exact runner I/O shapes.

package/vendor/skills/orizu-cli/references/prompt-control-plane.md

@@ -22,16 +22,80 @@ Use this reference for the Phase 0 prompt, judge, scorer, runner, score, run, an
 7. For common text-candidate optimization, prefer `orizu optimizations run-gepa`; it starts the run and logs events for you.
 8. After `run-gepa`, inspect `logs/<optimization_run_id>` first; it is the complete local trace for coding-agent analysis.
 9. Use `orizu optimizations export <run-id> --out <run-id>.optimization.json` when the local log is missing or the run happened elsewhere.
-10. Write and attach a markdown report for finished, failed, or cancelled runs; use `optimization-reports.md` for structure and diagnostic guidance.
-11. For custom optimizers, start an optimization run before local execution, then stream events into that run.
-12. Use bare HTTP for optimization events; use `orizu log` only as a shell fallback.
-13. Promote only accepted candidates; rejected candidates stay in optimization events.
+10. Promote an accepted candidate with `orizu optimizations promote <run-id> --candidate <id> [--label production]`; tuple output reports its profile version and each component as changed or carried.
+11. Write and attach a markdown report for finished, failed, or cancelled runs; use `optimization-reports.md` for structure and diagnostic guidance.
+12. For custom optimizers, start an optimization run before local execution, then stream events into that run.
+13. Use bare HTTP for optimization events; use `orizu log` only as a shell fallback.
+14. Promote only accepted candidates; rejected candidates stay in optimization events.
 
 Customer model-provider secrets stay local. Do not upload Anthropic/OpenAI/etc. API keys to Orizu.
 
+## Instruction sets
+
+An instruction set is a set of named components (key → text); order carries no meaning.
+It has one profile per model config. Existing prompts appear as sets of one component. Inspect or
+write them with:
+
+```bash
+orizu instruction-sets list --project core/evals --status active --json
+orizu instruction-sets show planner --project core/evals --status active --json
+orizu instruction-sets create ./orizu.instruction-set.json --project core/evals --model-config anthropic/claude-haiku
+orizu instruction-sets push ./orizu.instruction-set.json --project core/evals
+orizu instruction-sets profiles new planner --project core/evals --model-config anthropic/claude-haiku
+orizu instruction-sets profiles promote planner --project core/evals --model-config anthropic/claude-haiku --version 2
+orizu instruction-sets profiles rollback planner --project core/evals --model-config anthropic/claude-haiku --to 1
+orizu instruction-sets default show planner --project core/evals
+orizu instruction-sets default move planner --project core/evals --model-config anthropic/claude-haiku --version 2
+orizu instruction-sets shape add planner --project core/evals --key safety --from ./orizu.instruction-set.json
+orizu instruction-sets shape remove planner --project core/evals --key safety
+```
+
+The manifest contains `name`, `shape` (the set of component keys; order carries no meaning), and `components`; every shape key
+needs a set-wide component (`key` plus `text` or a manifest-relative `path`). A
+component may additionally specify `modelConfig` to override that key in a named
+profile. Create materializes each named profile with its complete base-plus-
+override component map. The default starts on the seed profile version. A profile with a
+production version resolves to that component map; a profile without one resolves to the
+default.
+
+Shape changes create unpromoted profile heads but leave the default and
+production pointers in place. The instruction set does not resolve for affected
+model configs until those pointers move to their new shape-change versions; use
+the follow-up commands printed by the text CLI.
+
+Before a human moves a default, `default show` lists the model configs whose
+resolution would change. A move targets a version by model-config identity and
+version number, and the database rejects an unsealed or non-commit-anchored
+component map. Shape changes create a new `shape_change` version for every profile;
+they do not repoint the default or any production label.
+
+Sync an offline runner directory with `orizu instruction-sets sync planner
+--out ./instructions --project core/evals`. The layout is `manifest.json`,
+`default/<key>.md`, and `profiles/<identity-slug>/<key>.md`; loaders are
+`loadInstructionSet(dir, name, modelConfigIdentity)` in TypeScript and
+`load_instruction_set(dir, name, model_config_identity)` in Python. A
+Git-pinned component is recorded in the manifest but is intentionally not
+downloaded; either loader raises `instruction_set_component_unavailable`. A
+missing or unreadable local component file is distinct:
+`instruction_set_component_unreadable`.
+
+Use `profiles new` to seed an additional model-config profile from the default;
+hosted agents may do this because it does not move production. Use `profiles
+promote` to move only that profile's production label; use `profiles rollback
+--to <n>` to create a new rollback version from the target version's component
+component map **and settings version** before moving it. Promotion and rollback are
+human-only because they move production labels.
+
+For a single-component instruction set that wraps an existing prompt, only a
+promotion or rollback of the set's default profile also moves that prompt's
+`production` label in the same transaction. The coupling is intentionally
+one-way: a later direct prompt-label move does not change the instruction-set
+profile label. Human CLI output names the prompt as `default profile → prompt
+label` when a profile command performed this mirror.
+
 Execution/privacy defaults:
 
-- Runner subprocesses receive only the file-contract paths plus a small allowlist of provider/runtime environment variables. Orizu API tokens are not passed into runner processes.
+- Runner subprocesses receive the input/output file-contract paths plus a small allowlist of provider/runtime environment variables. Orizu API tokens are not passed into runner processes. When an execution carries an instruction set, `ORIZU_INSTRUCTION_SET_DIR` is set explicitly to a fresh per-row synced layout; it is absent for legacy executions and is never inherited from the shell.
 - `orizu optimizations run-gepa` redacts dataset row payloads and reflection text in logged events by default. Use `--log-row-snapshots` only when the customer explicitly wants raw row and prompt text in the optimization event stream.
 - `run-gepa` still writes complete local traces under `logs/<optimization_run_id>` by default. Treat those logs as sensitive: they include row inputs, model outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, and candidate text.
 - Runner artifacts, runner output, score result uploads, and optimization event payloads are size-capped. If a run needs larger observability payloads, store the large artifact separately and log a pointer.
@@ -52,7 +116,7 @@ Required file: `manifest.json`.
 }
 ```
 
-For `runners exec`, the command must read input JSON from `ORIZU_RUNNER_INPUT_PATH` and write output JSON to `ORIZU_RUNNER_OUTPUT_PATH`.
+For `runners exec`, the command must read input JSON from `ORIZU_RUNNER_INPUT_PATH` and write output JSON to `ORIZU_RUNNER_OUTPUT_PATH`. When present, `ORIZU_INSTRUCTION_SET_DIR` contains the same layout written by `instruction-sets sync`; use `loadInstructionSet`/`load_instruction_set` to read materialized component text. Pinned components are represented in the manifest and are intentionally not materialized as `.md` files.
 
 Input shape:
 
@@ -68,12 +132,24 @@ Input shape:
       "max_tokens": 4096
     }
   },
+  "instruction_set": {
+    "name": "planner",
+    "model_config": { "identity": "openai/gpt-5.4" },
+    "shape": ["system", "tools"],
+    "prompt_component_key": "system",
+    "components": { "system": "inline component bytes" },
+    "pinned_components": {
+      "tools": { "repoPath": "skills/tools.md", "contentSha": "…", "commitSha": "…" }
+    }
+  },
   "prompt_version_id": "uuid",
   "runner_version_id": "uuid",
   "run_id": null
 }
 ```
 
+`instruction_set` is additive. `prompt_component_key` names the component that contains the requested legacy prompt version; candidate evaluation replaces that component's seed bytes with the candidate. `components` contains only inline text; Git-only components are absent there and appear in `pinned_components` with camelCase Git pins. Runners mirror whether the exec-context response contains `prompt.body`; the explicit `instructionSetProfileVersionId` parameter is a server surface that ALI-1536 sends. The route keeps byte-identical `prompt.body` for every implicit prompt-version caller and omits it only for an explicit multi-component map; explicit set-of-one responses retain it. Top-level JSON key order is not part of the contract.
+
 Output shape:
 
 ```json
@@ -424,8 +500,9 @@ Failed or deliberately skipped JSON enrichment carries a named
 sealed version behind each measurement even when the canonical body could not
 be measured. Length enrichment is best-effort: a supporting query failure
 leaves the list available with null stats and `enrichment_failed`. At most the
-first 500 sorted summaries are enriched per request; later summaries carry
-null stats and `measurement_cap_exceeded`.
+first 500 sorted summaries are enriched per request, and canonical-body
+resolution has a 15-second server budget. A summary skipped by either bound
+carries null stats and `measurement_cap_exceeded`.
 
 Archive or restore a prompt:
 
@@ -705,6 +782,7 @@ orizu --local optimizations run-gepa \
   --split-set-id <split-set-id> \
   --train-split train \
   --val-split validation \
+  --engine official \
   --log-dir ./logs \
   --minibatch-size 3 \
   --num-threads auto
@@ -712,12 +790,14 @@ orizu --local optimizations run-gepa \
 
 Useful GEPA flags:
 
+- `--engine official|legacy` defaults to `official`, which runs the bundled official GEPA connector. Use `legacy` only as the frozen recovery hatch; it keeps the historical Python loop and does not support `--max-candidate-proposals` or the skilled proposer.
 - `--scorer-input-contract gepa|flat_row` selects the scorer-runner input shape. Use `flat_row` to reuse a judge runner written for `runners exec --scorer-version` without a hand-written adapter; add `--scorer-candidate-field <row-field>` when that judge reads the candidate output from a specific row field (e.g. `draft`). Passing a candidate field under the `gepa` contract is refused at launch, not silently ignored. See "Scorer-Runner Input Contracts" above.
 - `--allow-degenerate-seed` opts out of the launch-time refusal when the seed scores the worst possible value on every validation row. Leave it off by default — a uniformly-worst seed is almost always a scorer contract mismatch.
-- Budget controls are mutually exclusive: choose at most one of `--budget auto|light|medium|heavy`, `--max-metric-calls <n>`, `--max-full-evals <n>`, or `--max-iterations <n>`. With none provided, `run-gepa` defaults to `--budget auto`, the balanced medium preset.
+- Budget controls are mutually exclusive: choose at most one of `--budget auto|light|medium|heavy`, `--max-metric-calls <n>`, `--max-full-evals <n>`, `--max-iterations <n>`, or `--max-candidate-proposals <n>`. With none provided, `run-gepa` defaults to `--budget auto`, the balanced medium preset. `--max-candidate-proposals` is available only with `--engine official`.
 - `--minibatch-size <n>` defaults to 3.
 - `--num-threads auto|N` defaults to `auto`; auto caps row-evaluation concurrency from mini-batch size, validation-set size, 2x CPU count, memory estimate, file-descriptor limit, and a 64-thread default ceiling. Set `ORIZU_GEPA_AUTO_THREADS_MAX` or use `--num-threads <n>` only when the runner/provider capacity is known.
-- `--candidate-selection-strategy pareto|current_best|epsilon_greedy`; default is `pareto`.
+- `--candidate-selection-strategy pareto|current_best|epsilon_greedy`; default is `pareto`. With `epsilon_greedy`, `--epsilon <n>` controls the random-selection probability (default `0.1`, clamped to `0`–`1`).
+- `--objective <text>` replaces the reflection objective when the optimization goal differs from the default instruction to maximize evaluator score while preserving intended behavior.
 - `--reflection-model <provider/model>`, `--reflection-temperature <n>`, `--reflection-prompt-template <text|@file>`.
 - `--reflection-max-tokens <n>` is explicit provider config, not a global default. It maps to Anthropic `max_tokens` and OpenAI `max_output_tokens`; Anthropic native Messages reflection requires it, while OpenAI can omit it unless the user wants a cap.
 - `--reflection-retry-attempts` and `--reflection-http-timeout-seconds` tune transient reflection-provider retries. Exhausted retryable failures log `reflection_failed`, count against candidate-proposal budget, and continue with the next iteration.
@@ -728,6 +808,24 @@ Useful GEPA flags:
 - `--log-dir <dir>` controls the local log root; default is `logs`.
 - `--no-local-log` disables local trace files. Use this only when the environment must not persist raw rows or reflection context.
 
+### Skilled proposer
+
+`--candidate-proposer skilled-proposer` selects the skilled proposer; omission keeps the default official proposer. The flag is official-engine-only and accepts that one value. A missing value is refused before launch as `--candidate-proposer requires a value that does not start with --`; another value is refused as `--candidate-proposer must be skilled-proposer`; selecting it with `--engine legacy` is refused with `--candidate-proposer is supported by the official GEPA engine only`.
+
+`--proposal-max-calls <positive-integer>` and `--proposal-max-tokens <positive-integer>` default to unset and are valid only with the skilled proposer. A missing operand names that flag; using either ceiling without the selection produces `--proposal-max-calls and --proposal-max-tokens require --candidate-proposer skilled-proposer`; the legacy-engine error names the individual flag. Integer and positive-value validation happens in the connector after launch and names `ORIZU_PROPOSAL_MAX_CALLS` or `ORIZU_PROPOSAL_MAX_TOKENS`.
+
+The call ceiling counts each transport-bearing proposer bridge call after its success or failure is recorded; provider retries inside one bridge call remain one counted call. The token ceiling uses accumulated provider-derived `total_tokens`, including recorded usage from unsuccessful completions. Both ceilings stop new proposal work at the next safe boundary, so an in-flight call can meet or pass a ceiling. They govern only the skilled proposer and remain independent of overall GEPA controls such as `--max-metric-calls`; set at least one proposal ceiling and one overall GEPA ceiling, then start the run only when both limits match the intended experiment.
+
+#### Managed environment lifecycle and recovery
+
+The selected `run-gepa` invocation prepares the environment automatically. It preflights CPython 3.10–3.14, installs the checked-in exact-version and SHA-256 lock with binary-only selection on first use, smoke-tests the staged environment, and publishes it atomically. Use `--python <command>` to select that interpreter; otherwise a nonempty `PYTHON` is used, falling back to `python3`. Supported wheel targets are macOS arm64 on macOS 15 or later and glibc 2.28 or later on Linux x86_64. Run the selected command; managed-environment readiness is proven when optimizer execution begins without an `ALI_1505_*` environment diagnostic.
+
+Published environments live under the working-directory-relative `.orizu/cache/skilled-proposer/venvs/<publish-key>`. The key covers the manager schema, lock digest, and interpreter/platform identity, so changing working directory, OS, kernel, interpreter, or lock can build another environment. Matching environments are validated before reuse, and concurrent first uses wait for the same atomic publication. Reclaim old environments by deleting the entire regenerable `.orizu/cache/skilled-proposer` directory, then rerun from the intended working directory until preflight completes.
+
+A nonempty `SSL_CERT_FILE` is passed unchanged to both pip and provider traffic. Empty or absent values trigger best-effort system-bundle discovery. `ALI_1505_SSL_CERT_FILE_UNRESOLVED` means pip fell back to its bundled trust store and provider traffic may need an explicit PEM bundle; `ALI_1505_SSL_CONTEXT_FAILED` means the explicit path or PEM contents must be corrected. Set the working bundle and retry until the selected run passes TLS setup.
+
+An interrupted first build leaves no published partial environment, so retry the same selected command. For `ALI_1505_UNSUPPORTED_CPYTHON`, choose an admitted interpreter with `--python` or `PYTHON`; for lock, wheel, install, validation, or publication diagnostics, preserve the named code and retry until the optimization starts. The CLI owns the lock and installation path; lock refresh remains a reviewed release operation.
+
 Local `run-gepa` logs:
 
 - The optimizer prints `[orizu-gepa] local log: <path>` after the run starts.
@@ -756,6 +854,11 @@ orizu --local optimizations export <optimization-run-id> \
 
 Use export when the local log is unavailable, the run happened on another machine, or a coding agent needs a portable single JSON artifact. The export fetches all optimization events, derives seed vs best, Pareto frontier, score-over-time, candidates, iterations, minibatch rows, and validation rows, and rehydrates row inputs from the dataset version artifact when possible. Server events may not contain row snapshots or reflection prompts unless the run used `--log-row-snapshots`; reflection responses are included for bundled `run-gepa` runs.
 
+Both diff-comment read routes (`/api/optimization-runs/<run-id>/diff-comments` and
+`/api/prompts/<prompt-id>/diff-comments`) return `componentKey` for tuple
+comments. Preserve it on writes and use it in grouping keys: equal line ranges
+from different instruction-set components are distinct comments.
+
 The v1 export keeps the run row's `best_candidate_id` in
 `summary.bestCandidateId` when event derivation rejects that identifier as an
 unknown candidate. Candidate detail can therefore be absent for the preserved