orizu 0.5.21 → 0.5.23

package/vendor/skills/orizu-cli/references/optimization-with-gepa.md

@@ -1,21 +1,21 @@
 # Optimization With GEPA
 
-How to optimize text candidates against validated judges/scorers. For Orizu-tracked runs, prefer `prompt-control-plane.md` and the bundled `orizu optimizations run-gepa` command when optimizing one text candidate. Use this reference for GEPA mechanics, custom optimizer implementations, and optional DSPy context for customers already using DSPy.
+How to optimize an instruction-set profile against validated judges and scorers. Prepare the complete instruction-set manifest, follow the [Authority map](../SKILL.md#authority-map) for its mutation, and then run the bundled `orizu optimizations run-gepa` flow. Use this reference for GEPA mechanics, custom optimizer implementations, and optional DSPy context for customers already using DSPy.
 
-For an instruction set, select the profile explicitly with `--instruction-set <name> --model-config <identity>` rather than `--prompt` or `--prompt-version` (those selectors are mutually exclusive). The connector resolves that model config's production profile, or the set default, and optimizes its complete component map. `--component-selector round-robin` is the default and updates one component per round; `--component-selector all` updates every component per round. A multi-component candidate is sent as a component map, while a set of one keeps the existing prompt-body contract. Git-pinned components and malformed component maps are refused before a run starts; multi-component auto-promotion is deliberately refused until component-map promotion is available.
+Select the profile explicitly with `--instruction-set <slug-or-exact-name> --model-config <identity>`. Those two selectors replace the legacy `--candidate-version-id` path and cannot be combined with it. The connector resolves that model config's production profile, or the set default when the profile has no production version, and optimizes its complete component map. `--component-selector round-robin` is the default and updates one component per round; `--component-selector all` updates every component per round. The runner receives a multi-component candidate as a component map, while a one-component set keeps the existing single-body runner contract. Git-pinned components and malformed component maps are refused before a run starts. Automatic promotion is refused for multi-component sets; write the report, obtain the human promotion decision, and follow the Authority map. Manifest create/push is the pre-run seed path, not a second accepted-candidate materialization; after a run, use the human one-shot or equivalent idempotent two-stage path in step 9.
 
 ## Inputs
 
 You should arrive here with:
-- One or more **validated judges** (TPR > 90%, TNR > 90% on a held-out test set).
+- One or more **validated judges** that clear their agreed judge trust bar (see `building-judges.md`).
 - A **dataset** of inputs to optimize against — usually the same exported labels, plus any harder cases you've added since.
-- A **starting prompt or program** for the LLM application you want to improve.
+- An **instruction set** whose selected profile contains the starting component values for the LLM application you want to improve.
 
 If you don't have a validated judge, stop. Optimizing against an unvalidated judge means you'll hill-climb on a noisy or biased signal — Goodhart's law in action.
 
 ## Why GEPA-Style Optimization
 
-- **GEPA** is a gradient-free prompt optimizer that uses an LLM to propose prompt edits, scores candidates against your metric, and keeps the best. It's well-suited to prompt-level optimization where you can't backprop through the model.
+- **GEPA** is a gradient-free text optimizer that uses an LLM to propose component edits, scores candidates against your metric, and keeps the best. It is well-suited to instruction optimization where you cannot backpropagate through the model.
 - In Orizu, runners execute candidates, scorers produce metrics/feedback, and optimization events make the loop inspectable and promotable.
 - DSPy is not part of Orizu's bundled optimizer. Treat DSPy examples here as an external integration pattern only.
 
@@ -40,52 +40,72 @@ If you don't have a validated judge, stop. Optimizing against an unvalidated jud
               └──────────┬──────────────┘
                          ▼
               ┌─────────────────────────┐
-              │ Optimized prompt        │
+              │ Optimized profile       │
               │ Compare before/after    │
               └─────────────────────────┘
 ```
 
 ## Orizu-tracked optimization
 
-Use the prompt control plane when you want runs, candidates, score charts, Pareto/frontier views, and promotions in Orizu:
+Use the control plane described in `prompt-control-plane.md` when you want runs, candidates, score charts, Pareto/frontier views, and promotions in Orizu:
 
-1. Push the candidate runner and prompt/judge prompt.
-2. Register a row scorer for reflection. GEPA reflection requires row-level feedback.
-3. Snapshot a dataset version and create a train/validation split set.
-4. Use `orizu optimizations run-gepa` for the common text-candidate case. It uses official GEPA by default; `--engine legacy` is a frozen safety hatch while migration parity is proven. Use `orizu optimizations start` plus event logging for a custom optimizer.
-5. Use set scorers for selection/tracked reporting when the meaningful metric is batch-level; execute builtin set scorers with `orizu scorers exec` or submit precomputed aggregates with `orizu scores submit --aggregate`.
-6. Promote only candidates that passed validation.
-7. Write and attach an optimization report from the local logs or export artifact; see `optimization-reports.md`.
+1. Create the instruction set from its prepared manifest with `orizu instructions create`, or push a new profile version with `orizu instructions push`, using the executor selected by the Authority map. Keep the set's shape fixed during a run.
+2. Push the candidate runner and, for an LLM evaluator, the judge rubric with `orizu judges push`.
+3. Register a row scorer for reflection. GEPA reflection requires row-level feedback.
+4. Snapshot a dataset version. Create one split set with train, validation, and a reserved final-held-out partition; pass only the train and validation partition names to `run-gepa`. Never pass the final-held-out partition as `--train-split` or `--val-split`; the CLI accepts arbitrary partition names and does not enforce this boundary.
+5. Push the optimizer implementation with `orizu optimizers push ./optimizer --project <team/project> --name <optimizer-name> --json` and retain its returned optimizer version ID.
+6. Use `orizu optimizations run-gepa --optimizer-version-id <optimizer-version-id>` with the set slug and model-config identity. It uses official GEPA by default; `--engine legacy` is a frozen safety hatch while migration parity is proven. Use `orizu optimizations start` plus event logging for a custom optimizer. Candidates are ranked on the partition passed as `--val-split`; use that partition for validation, not Final-held-out.
+7. Use set scorers for selection/tracked reporting when the meaningful metric is batch-level; execute builtin set scorers with `orizu scorers exec` or submit precomputed aggregates with `orizu scores submit --aggregate`.
+8. Write and attach the optimization report by following “How To Build The Report” step 5 in `references/optimization-reports.md`: “Determine whether the current execution surfaces can reproduce the selected system before claiming a Final-held-out comparison.” Its “Flat-row scorer contract,” “Default GEPA scorer contract,” and “Instruction-set execution limits” subsections own the executable boundaries and current gaps.
+9. After the human accepts the report, follow the [Authority map](../SKILL.md#authority-map). Simpler one-shot path: a human curator runs `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`, materializing and labeling once. In the two-stage path for multi-component or deliberately unlabeled materialization, an agent may materialize with `orizu optimizations promote <run-id> --candidate <id> --project <team/project>`. A human curator re-runs that same promotion as `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`. The idempotent finalizer reuses the existing materialized profile version, moves production to it, and creates no duplicate profile version or candidate-promoted event.
 
-The bundled Orizu GEPA-style optimizer supports configurable budget, minibatch size (default 3), candidate selection strategy, reflection model/template, reflection provider settings, evaluation caching, and optional auto-promotion. It redacts row snapshots and reflection text by default; only pass `--log-row-snapshots` when raw data in event logs is intentional.
+The bundled Orizu GEPA-style optimizer supports configurable budget, minibatch size (default 3), candidate selection strategy, reflection model/template, reflection provider settings, evaluation caching, and optional auto-promotion. Use the report-first path: run without auto-promotion, write the report, obtain the human promotion decision, then route the manual promotion command through the Authority map. Only pass `--log-row-snapshots` when raw data in event logs is intentional.
+
+Budget behavior:
+- The `auto`, `light`, `medium`, and `heavy` presets are DSPy-style metric-call budgets whose limits scale with the validation-set size.
+- Budgets are checked after in-run seed validation and between completed iterations, never mid-iteration; once an iteration starts, its parent minibatch, child minibatch, and, for an accepted child, full validation complete even if that work overshoots the selected limit.
 
 Reflection output contract:
-- The reflective LM's final text is used verbatim as the next candidate prompt.
-- The default reflection prompt asks for only the complete updated prompt body. Do not ask the model to wrap the candidate in markdown fences or tags; real prompts often contain those characters.
-- Put provider-native reasoning controls in `--reflection-provider-settings <json|@file>`, not in the prompt body. For OpenAI reasoning models, use a shape such as `{"reasoning":{"effort":"medium","summary":"auto"}}`. For Anthropic Claude models with thinking controls, use a shape such as `{"thinking":{"type":"adaptive","display":"omitted"},"output_config":{"effort":"medium"}}`.
+- The official engine uses the decoded reflective LM response string as the next value of the selected component without trimming; the legacy engine uses `response.strip()`, removing leading and trailing whitespace. With `round-robin`, other component values are read-only context; with `all`, each component is reflected independently and the results form one complete candidate profile.
+- The default reflection template asks for only the complete updated component value. Do not ask the model to wrap the value in markdown fences or tags; instruction components often contain those characters.
+- Put provider-native reasoning controls in `--reflection-provider-settings <json|@file>`, not in an instruction component. For OpenAI reasoning models, use a shape such as `{"reasoning":{"effort":"medium","summary":"auto"}}`. For Anthropic Claude models with thinking controls, use a shape such as `{"thinking":{"type":"adaptive"},"output_config":{"effort":"medium"}}`.
+- Treat scorer feedback as directional signal: with `higher_is_better: true`, describe lower scores as failures or opportunities, not as a numeric loss; omit fields labeled `informational; not scored` from feedback sent to reflection.
 - Reflection max-token limits are explicit. `--reflection-max-tokens <n>` maps to Anthropic `max_tokens` and OpenAI `max_output_tokens`; Anthropic native Messages reflection requires it, while OpenAI may omit it when no cap is desired.
-- Reflection HTTP calls retry transient failures by default (`--reflection-retry-attempts 3`, `--reflection-http-timeout-seconds 180`). Exhausted retryable failures emit `reflection_failed`, consume one candidate-proposal budget slot, and move to the next iteration instead of killing the run.
+- Reflection HTTP calls retry transient failures by default (`--reflection-retry-attempts 3`, `--reflection-http-timeout-seconds 180`). Legacy GEPA logs an exhausted retryable reflection failure, charges proposal and iteration budgets, and continues. The official engine increments proposal usage only after successful `on_proposal_end`; skilled-proposer failures re-raise and stop.
 
 Full command syntax and event contracts: `prompt-control-plane.md`.
 
+For staff-enabled hosted optimization, a human/PAT caller adds `--hosted` and
+uses a named `--budget auto|light|medium|heavy`. Hosted launch does not require
+the local candidate/scorer runner directories or a local log directory, and it
+never sends provider credentials or runner bytes from the customer's machine.
+Numeric budget controls are refused because they cannot be compared honestly
+to the named team ceiling. For response-loss recovery, reuse an explicit
+`--launch-intent-id <uuid>`; a changed project or job specification requires a
+new intent. A hosted agent prepares the command and hands it to a human.
+
 ## Migrating from the legacy engine (release cli-v0.5.20+)
 
 Since release `cli-v0.5.20` (`orizu@0.5.20` on npm), `run-gepa` defaults to the official GEPA engine. The flag surface and validation rules match legacy almost everywhere — a valid legacy command runs unchanged, with two exceptions: the reflection-cap requirement below, and stricter `=`-form parsing (`--flag=` with an empty value, or an `=`-attached value that itself starts with `--`, is refused pre-launch; legacy's argparse accepted both — a scripted `--scorer-candidate-field=$FIELD` with `FIELD` unset now fails with the flag named). What actually differs:
 
-- **Validation happens before launch.** Legacy validated flags inside the python process (an argparse exit); the official dispatcher enforces the flag-shape rules below before anything spawns and names the offending flag in the error (the budget-conflict error lists the exclusive set rather than the flags you passed; value errors — e.g. a non-integer or non-positive budget number — are still reported by the engine and name the underlying setting). The rules carried over from legacy: `--budget` / `--candidate-selection-strategy` / `--scorer-input-contract` validate their choices, boolean flags take no value, abbreviated flag spellings are rejected, and budget controls are mutually exclusive — at most one of `--budget`, `--max-metric-calls`, `--max-full-evals`, `--max-iterations`. The official-only `--max-candidate-proposals` joins that exclusion set; passing no budget control defaults to `--budget auto`.
+- **Validation happens before launch.** Legacy validated flags inside the python process (an argparse exit); the official dispatcher enforces the flag-shape rules below before anything spawns and names the offending flag in the error (the budget-conflict error lists the exclusive set rather than the flags you passed; value errors — e.g. a non-integer or non-positive budget number — are still reported by the engine and name the underlying setting). The rules carried over from legacy: `--budget` / `--candidate-selection-strategy` / `--scorer-input-contract` validate their choices, boolean flags take no value, abbreviated flag spellings are rejected, and budget controls are mutually exclusive — at most one of `--budget`, `--max-metric-calls`, `--max-full-evals`, `--max-iterations`. The official-only `--max-candidate-proposals` joins that exclusion set. With no budget control, `--budget auto` selects the balanced `medium` preset.
+- **Context and runtime.** Supply `--optimizer-version-id`, `--runner-version-id`, `--candidate-runner-dir`, `--scorer-version-id`, `--scorer-runner-version-id`, `--scorer-runner-dir`, `--dataset-version-id`, `--split-set-id`, `--train-split`, and `--val-split`; use `--python` to select the interpreter and `--json` for structured output.
+- **Search and reproducibility.** Tune `--minibatch-size`, `--candidate-selection-strategy`, `--epsilon`, `--objective`, and `--num-threads`; use `--seed`, `--disable-evaluation-cache`, and optional `--metadata <json>` when reproducibility or run attribution requires them. Automatic row-evaluation concurrency is bounded by the larger of the minibatch and validation workloads, twice the detected CPU count, available memory after its reserve, available file descriptors after headroom, and a configurable ceiling whose default is 64 (`ORIZU_GEPA_AUTO_THREADS_MAX`). An explicit positive thread count bypasses that auto-scaling calculation.
+- **Reflection controls.** Select `--reflection-model` and `--reflection-temperature`; a perfect selected minibatch skips reflection and child creation by default under `--skip-perfect-parent-reflection`, while `--no-skip-perfect-parent-reflection` overrides that behavior.
+- **Promotion flags.** Do not use `--auto-promote` or `--promotion-label` in the report-first workflow: write the report, obtain the human promotion decision, and follow the Authority map.
 - **A reflection cap is required for non-OpenAI reflection models.** `--reflection-max-tokens` is required at launch whenever the reflection model does not start with `openai/` — including the default (`anthropic/claude-opus-4-7`) and bare names like `gpt-4o` — so the run refuses immediately instead of failing every reflection call mid-run.
-- **The skilled proposer is an opt-in proposal path.** Reach for `--candidate-proposer skilled-proposer` when you want to evaluate the upstream proposer's alternative reflection process and can accept its first-use network, CPython, and managed-dependency requirements. Evaluate the shipped empty-skills proposer through the controlled A/B below, and ground quality or generalization claims in that evidence. Environment setup and recovery live in `prompt-control-plane.md` ("Skilled proposer").
+- **The skilled proposer is an opt-in proposal path.** Reach for `--candidate-proposer skilled-proposer` when you want to evaluate the upstream proposer's alternative reflection process and can accept its first-use network, CPython, and managed-dependency requirements. It prepares or reuses a managed Python environment. Its aggregate `--proposal-max-calls` and `--proposal-max-tokens` budgets are independent of metric-call limits and per-response reflection limits. Add `--candidate-proposer-config @file` for explicit skills/guidance; do not combine the selection with `--reflection-prompt-template`. Evaluate the shipped empty-skills proposer or a pinned config through the controlled A/B below, and ground quality or generalization claims in that evidence. Config schema, environment setup, and recovery live in `prompt-control-plane.md` ("Skilled proposer").
 - **Seed preflight: same gate, better evidence.** Both engines refuse a degenerate seed (every valid row pinned to the same bound — the worst score, or uniformly `0.0` under a lower-is-better scorer, where the perfect bound is indistinguishable from a scorer silently zeroing on a mismatched input shape) or an all-errored seed at launch, and both accept `--allow-degenerate-seed` to bypass. The bypass covers *both* refusals — including the broken-scorer one — so a bypassed run can spend budget on a scorer that never works; use it only when a weak seed is expected. New on the official engine: the refusal message carries per-row evidence (row ids, scores, scorer output and errors); `<log-dir>/preflight-refused-<uuid>/preflight.json` (not written under `--no-local-log`) instead stores redacted `{row_id, score, error_class}` entries unless `--log-row-snapshots` is passed — a sibling of the run directories, since no run record exists when a launch is refused.
 - **Metric accounting differs.** Official counts include every evaluated row; legacy excluded some cached work. Do not compare raw metric-call totals across engines (details in ADR-019).
 - **Artifacts.** Both engines write `events.jsonl`, `evaluations.jsonl`, `reflections.jsonl`, and `result.json` locally; `lm_stats.json` (reflection usage) is official-only. Dashboard rendering is identical, and run metadata records `engine: official` or `engine: legacy`.
 
-Terminal semantics are unchanged between engines: exhausting any budget control other than `--max-iterations` ends the run `paused` (`pause_reason: budget_exhausted`) with no auto-promotion, while completing the configured `--max-iterations` is a normal `succeeded` finish that may auto-promote. A paused run's validated candidates can still be promoted manually — only automatic promotion is gated on how the run ended. Falling back to `--engine legacy` after a budget-exhausted pause reproduces the same paused outcome; it will not turn the run into a `succeeded` one.
+Terminal semantics are unchanged between engines: exhausting any budget control other than `--max-iterations` ends the run `paused` (`pause_reason: budget_exhausted`), while completing the configured `--max-iterations` is a normal `succeeded` finish. In both cases, keep the report-first path: no auto-promotion; write the report, obtain the human promotion decision, and follow the Authority map for manual promotion. Falling back to `--engine legacy` after a budget-exhausted pause reproduces the same paused outcome; it will not turn the run into a `succeeded` one.
 
 **Fallback:** rerun with `--engine legacy` for the previous engine's exact behavior, dropping `--max-candidate-proposals` first if you used it (it is official-only; the CLI refuses it under legacy before launch). Legacy is frozen (no new features) and will be removed at the M3 milestone (ADR-019) — if you fall back because the official engine misbehaved, capture both run ids and report the pair.
 
 ### Compare the skilled proposer with the default
 
-Run two arms with the same candidate prompt/version, dataset version and split set, candidate and scorer runners, seed, reflection model/settings, thread count, and one shared overall budget such as `--max-candidate-proposals 1`. Keep the default arm unselected; add `--candidate-proposer skilled-proposer` plus bounded `--proposal-max-calls` and `--proposal-max-tokens` only to the selected arm. The comparison is ready when both terminal states, retained candidates, scores, proposal evidence, and total run usage are captured. A fixed GEPA seed aligns optimizer sampling but does not make provider or scorer calls deterministic, so treat the pair as process and outcome evidence rather than an intrinsic quality claim.
+Run two arms with the same instruction-set profile version, dataset version and split set, candidate and scorer runners, seed, reflection model/settings, thread count, and one shared overall budget such as `--max-candidate-proposals 1`. Keep the default arm unselected; add `--candidate-proposer skilled-proposer`, the same pinned `--candidate-proposer-config @file` when testing configuration, and bounded `--proposal-max-calls` / `--proposal-max-tokens` only to the selected arm. The comparison is ready when both terminal states, retained candidates, scores, proposal evidence, and total run usage are captured. A fixed GEPA seed aligns optimizer sampling but does not make provider or scorer calls deterministic, so treat the pair as process and outcome evidence rather than an intrinsic quality claim.
 
 For the selected arm, each provider-bearing proposal call appends `proposal-observability/events.jsonl`. Success records contain `source`, `status`, `attempt`, `correlation_id`, `provider`, `model`, nullable `request_id`, `latency_ms`, `cache_state`, and `usage.{input_tokens,output_tokens,total_tokens}`. Failure records contain `source`, `status`, `attempt`, `correlation_id`, `provider`, `cache_state`, `failure_code`, and optional `usage`; the latest durable failure is also written to `proposal-failures/latest.json` with `source`, `code`, `detail`, and optional `correlation_id`.
 
@@ -93,7 +113,7 @@ With local logging, those paths sit under `<log-dir>/<run-id>/`, and terminal `p
 
 ## Step 1: Wrap your application as an Orizu runner
 
-For Orizu-tracked optimization, the candidate runner receives one dataset row and one candidate text body through the file contract. The scorer runner, by default, receives a GEPA-shaped `row` — `{source_row, candidate_id, candidate_output, candidate_raw_response, candidate_error}` — and returns a score and feedback. See `prompt-control-plane.md` ("Scorer-Runner Input Contracts") for the exact runner I/O shapes.
+For Orizu-tracked optimization, the candidate runner receives one dataset row and the candidate profile through the file contract. A one-component profile is passed as a single body for runner compatibility; a multi-component profile is passed as a complete component map. The scorer runner, by default, receives a GEPA-shaped `row` — `{source_row, candidate_id, candidate_output, candidate_raw_response, candidate_error}` — and returns a score and feedback. See `prompt-control-plane.md` ("Scorer-Runner Input Contracts") for the exact runner I/O shapes.
 
 **Contract warning:** the GEPA scorer contract differs from the flat-row score-run contract used by `orizu runners exec --scorer-version`. A judge runner written for flat-row score runs will find no output to judge in the GEPA shape and silently score every candidate 0. Do not hand-write a wrapper runner: pass `--scorer-input-contract flat_row` (plus `--scorer-candidate-field <row-field>` if the judge reads the candidate output from a named row field such as `draft`) and `run-gepa` adapts the payload for you while keeping the registered runner bytes unchanged. `run-gepa` also validates the contract on the seed at launch and refuses a uniformly-worst-scoring seed with a diagnosis instead of burning budget.
 
@@ -132,7 +152,7 @@ class SupportAgent(dspy.Module):
         )
 ```
 
-If your real application is multi-step (retrieval + generation + tool use), build a multi-Module program. GEPA can optimize each step's prompt independently.
+If your real application is multi-step (retrieval + generation + tool use), build a multi-Module program. GEPA can optimize each module's instructions independently.
 
 ## Step 2: Register scorers
 
@@ -208,7 +228,7 @@ logs/<optimization_run_id>/
   result.json
 ```
 
-Use this directory as the primary artifact for coding-agent analysis. It contains the full row inputs, model outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, candidate text, and final result. Override the root with `--log-dir <dir>`; disable persistence with `--no-local-log` only when raw rows/reflection context must not be written to disk.
+Use this directory as the preferred artifact for coding-agent analysis. It contains the full row inputs, model outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, candidate text, and final result. Override the root with `--log-dir <dir>`; disable persistence with `--no-local-log` only when raw rows/reflection context must not be written to disk.
 
 If the local log is missing or the run happened elsewhere, export the server-side archive:
 
@@ -218,7 +238,7 @@ orizu optimizations export <optimization-run-id> --out ./optimization.json
 
 Export returns one JSON object with raw events plus derived seed-vs-best, Pareto frontier, candidates, score-over-time, iterations, minibatch rows, validation rows, scorer context, prompt versions, and dataset split information. It fetches all optimization events and rehydrates row inputs from dataset artifacts when possible. Server events redact row snapshots and reflection prompts by default, but bundled `run-gepa` includes reflection responses in the event stream.
 
-After the run ends, write a markdown report before the context is lost. Use `optimization-reports.md` for the report structure: headline score and confusion matrix, what changed, per-row fixed/regressed/persistent failures, optimizer health, recommendations, and what not to do next.
+After the run ends, write a markdown report before the context is lost. Use the authoritative ordered sections in `optimization-reports.md`: Promotion Decision; Run And Evidence; Candidate Comparison (Validation Data); Optimizer Health; Scenario Classes; What Changed In The Selected Version; Final-held-out Result: Seed vs Selected Candidate; Recommendation And Named Next Moves; Reproducibility; and Report Completeness Checklist.
 
 DSPy GEPA example for customers already on DSPy:
 
@@ -247,13 +267,13 @@ optimized_program.save("./optimized_support_agent.json")
 
 GEPA will:
 1. Run the program on `trainset`, score with `metric`.
-2. Use `reflection_lm` to propose prompt edits based on failures.
-3. Score candidate prompts on the trainset, keep the best.
+2. Use `reflection_lm` to propose instruction edits based on failures.
+3. Score candidate programs on the trainset, keep the best.
 4. Validate on `valset` to avoid overfitting.
 
 Orizu's bundled optimizer is intentionally narrower than DSPy GEPA today: text candidates only, local runner/scorer directories, and Orizu event logging built in.
 
-## Step 5: Compare before / after on a held-out set
+## Step 5: Compare before / after on Final-held-out
 
 This is the step teams skip and regret.
 
@@ -274,7 +294,7 @@ def evaluate(program, examples) -> dict:
 
 
 # Hold out a fresh set the optimizer never saw.
-held_out = load_examples("./held-out.jsonl")
+held_out = load_examples("./final-held-out.jsonl")
 
 before = evaluate(SupportAgent(), held_out)
 after = evaluate(optimized_program, held_out)
@@ -287,8 +307,8 @@ print(f"After:  {after}")
 
 ## Step 6: Ship and feed the loop
 
-If the optimized prompt holds up on held-out:
-- Replace the production prompt.
+If the optimized profile holds up on Final-held-out:
+- Promote the validated profile version to production; never promote one component by itself.
 - Sample new production traces over the next week.
 - Upload them as a new dataset (`primer.md` Step 1).
 - Annotate failures the optimized system *now* exhibits — they'll be different from the ones the previous version had.
@@ -300,18 +320,18 @@ Each pass through the loop reveals the next layer.
 ## Common pitfalls
 
 - **Optimizing against an unvalidated judge.** You'll improve the metric and degrade the system. Always validate first.
-- **No held-out comparison.** "It's better, look at the metric" without a held-out set is meaningless — GEPA will overfit if you let it.
+- **No Final-held-out comparison.** "It's better, look at the metric" without Final-held-out is meaningless — GEPA will overfit if you let it.
 - **Hiding regressions in the average.** Track per-failure-mode metrics, not just combined.
 - **Over-budgeting GEPA.** Heavy budgets give diminishing returns and burn LM spend. Start with `auto="light"`, scale up only if needed.
 - **Ignoring temperature.** Run optimization with the same LM config (model, temperature) you use in production. Optimizing against gpt-4o at temp=0 doesn't transfer to gpt-4o-mini at temp=0.7.
-- **Recreating Orizu logging by hand for text prompts.** Use `orizu optimizations run-gepa` unless the optimizer is genuinely custom.
+- **Recreating Orizu logging by hand for instruction sets.** Use `orizu optimizations run-gepa` unless the optimizer is genuinely custom.
 
 ## Checklist
 
 Before declaring an optimization run successful:
 
-- [ ] Each metric is backed by a validated judge (TPR/TNR > 90%)
-- [ ] Train / val / held-out splits, and the held-out set is genuinely untouched during optimization
+- [ ] Each metric is backed by a validated judge that clears its agreed judge trust bar (see `building-judges.md`)
+- [ ] Train and validation partitions plus Final-held-out, which stays genuinely untouched during optimization
 - [ ] Per-metric numbers reviewed (not just combined)
 - [ ] Same LM/temperature in eval as in production
 - [ ] Optimization report written from logs/export and attached with `--report-file` when the run is finished, failed, or cancelled

package/vendor/skills/orizu-cli/references/primer.md

@@ -2,19 +2,44 @@
 
 > **Core idea:** You can't improve what you can't measure — but measuring the wrong thing is worse than not measuring at all. This doc walks from raw failure logs to a continuously-running evaluation loop, and shows how Orizu operationalizes each step.
 
-The loop has four steps:
+The operational loop starts at **J3 — Build the dataset** and branches on the
+approved ground-truth source:
 
 ```
-Upload → Annotate → Judge → Optimize
-   ▲                                │
-   └────────── new traces ──────────┘
+approved golden ground truth covers the scenario classes: J3 → J5
+missing ground truth needs human labels:                    J3 → J4 → J5
+                                                                 │
+                                                                 ▼
+                                                   J6 → new traces → J3
 ```
 
-The CLI covers the full operational loop. Human annotation still happens through apps/tasks, while judge/scorer logic and optimization execution run locally and report back through the prompt control plane. Each step depends on the output of the previous; don't skip ahead.
+In the diagram, **J4 — Ground truth via annotation (conditional)** is taken only
+for missing labels, **J5 — Build & validate judges** consumes the approved
+ground truth, and **J6 — Optimize & promote** feeds new traces into the next J3
+dataset version. Approved golden ground truth therefore goes directly from J3
+to J5 instead of passing through annotation.
+
+The CLI covers the full operational loop. Start instruction sets with `orizu instructions`: one instruction set per agent or LLM experience, one profile per model config, and components owned by that profile. Human annotation still happens through apps/tasks when J4 is needed, while judge/scorer logic and optimization execution run locally and report back through the instruction control plane. Follow the applicable branch and require its prior journey step's exit criterion before continuing.
+
+An instruction-set manifest carries its `name`, optional `description`, fixed, ordered `shape`, and component values. The set's slug is its stable CLI reference even when its display name changes. Profiles have their own component values; components are never shared between profiles or sets. The set-wide default serves model configs without a production profile version.
+
+Start by inspecting the set, then follow the [Authority map](../SKILL.md#authority-map)
+for mutation custody before syncing through the same surface:
+
+```bash
+orizu instructions list --project <team>/<project>
+orizu instructions show <slug-or-exact-name> --project <team>/<project>
+orizu instructions create ./orizu.instruction-set.json --project <team>/<project> --model-config <identity>
+orizu instructions push ./orizu.instruction-set.json --project <team>/<project> --set <slug-or-exact-name>
+orizu instructions profiles new <slug-or-exact-name> --project <team>/<project> --model-config <identity>
+orizu instructions sync <slug-or-exact-name> --out ./instructions --project <team>/<project>
+```
+
+Do not create or mutate standalone prompts. If a legacy identifier leads to a prompt read view, use its owner information to find the instruction set and continue with `orizu instructions`.
 
 ---
 
-## Step 0: Error Analysis (do this before writing any eval)
+## Error analysis (before writing any eval)
 
 The most common mistake teams make is jumping straight to evals before they understand what's actually failing. Error analysis is the prerequisite.
 
@@ -30,11 +55,11 @@ The most common mistake teams make is jumping straight to evals before they unde
 **Decision: should you even write a formal eval for this failure?**
 - If you haven't observed it, no — do error analysis first.
 - If the fix is one-shot and obvious, no — just fix it.
-- If fixing it requires repeated iteration (prompt tweaks, retries, structural changes), yes — that's where evals shine.
+- If fixing it requires repeated iteration (component edits, retries, structural changes), yes — that's where evals shine.
 
 ---
 
-## Step 1: Upload — gather diverse traces
+## J3 detail: Upload — gather diverse traces
 
 ### Why
 
@@ -80,7 +105,7 @@ Full surface: `cli-reference.md`.
 
 ---
 
-## Step 2: Annotate — binary labels per failure mode
+## Conditional J4 detail: Annotate — binary labels per failure mode
 
 ### Why
 
@@ -133,14 +158,14 @@ Authoring the labeler app — contract, design principles, and common patterns:
 
 ---
 
-## Step 3: Judge — turn labels into automated evaluators
+## J5 detail: Judge — turn labels into automated evaluators
 
 ### Why
 
 - **Code assertions first.** If a failure is a rule (keyword present, tool called, format valid), write a code check. Fast, free, deterministic.
 - **LLM-as-a-judge for nuanced criteria only.** Powerful but slow and expensive — reserve for failures code can't catch.
 - **Validate every judge against human labels.** A judge that doesn't agree with humans is a misaligned metric.
-- Track **TPR (True Positive Rate)** and **TNR (True Negative Rate)** separately, not accuracy. Accuracy is misleading on imbalanced data. Target both > 90%.
+- Track **TPR (True Positive Rate)** and **TNR (True Negative Rate)** separately, not accuracy. Accuracy is misleading on imbalanced data. Each judge must clear its agreed judge trust bar (see `building-judges.md`).
 - **A 100% pass rate is a smell** — your evals are saturated. Add harder cases.
 - **Three common eval mistakes:**
   1. Skipping the data — using off-the-shelf metrics ("Helpfulness," "Faithfulness") that don't measure your specific failures.
@@ -154,19 +179,19 @@ There are three **types** of automated evals:
 
 ### In Orizu
 
-This work is still authored by the agent in code, but Orizu now stores the versioned evaluator artifacts and score results:
+This work is still authored by the agent in code, but Orizu stores the versioned instruction, evaluator, and score artifacts:
 
 1. Export labeled data: `orizu tasks export --task <id> --format jsonl --out labels.jsonl`
 2. For each failure mode, choose code assertion or LLM-judge.
 3. Build the judge.
 4. Validate against the labels (train/dev/test split, TPR/TNR).
-5. Push the judge prompt and runner when it is an LLM judge, register a scorer, then submit score runs for prompt versions or optimization candidates.
+5. Keep the subject application's instruction material in its instruction set. For an LLM judge, push the judge artifact and runner, register a scorer, then submit score runs for the set's profile versions or optimization candidates.
 
-Control-plane commands: `prompt-control-plane.md`. Detailed authoring walkthrough — code assertion patterns, LLM-judge prompt scaffold, train/dev/test split, TPR/TNR computation, saturation checks: `building-judges.md`.
+Control-plane commands: `prompt-control-plane.md`. Detailed alignment-first walkthrough — deterministic versus LLM judge choice, binary output contract, judge trust bar agreement, judge-test evidence, and judge optimization: `building-judges.md`.
 
 ---
 
-## Step 4: Optimize — hill-climb against validated judges
+## J6 detail: Optimize — hill-climb against validated judges
 
 ### Why
 
@@ -181,8 +206,8 @@ Done locally, reported to Orizu:
 1. Package the candidate execution as a runner, or use the customer's existing local app wrapper.
 2. Register validated scorers.
 3. Run the bundled Orizu GEPA-style text optimizer, or a custom optimizer against the scorer set.
-4. Stream optimization events to Orizu, promote accepted candidates, and submit comparable scores.
-5. Diff before/after on the eval suite. Ship if it holds; new traces feed back to step 1.
+4. Stream optimization events to Orizu, promote accepted candidates into the target instruction-set profile, and submit comparable scores.
+5. Diff before/after on the eval suite. Ship if it holds; new traces feed back to J3.
 
 Control-plane commands: `prompt-control-plane.md`. Detailed walkthrough — GEPA mechanics, optional DSPy context for customers already using it, and before/after comparison: `optimization-with-gepa.md`.
 
@@ -231,7 +256,7 @@ Production Traffic
       ↓                                         │
   Build Judge (code-first, LLM-as-judge if needed)
       ↓                                         │
-  Validate Judge (TPR > 90%, TNR > 90%)         │
+  Validate Judge (agreed trust bar)             │
       ↓                                         │
   Optimize (GEPA-style search against scorers) │
       ↓                                         │