orizu 0.5.30 → 0.5.32

package/vendor/skills/orizu/references/building-judges.md

@@ -18,13 +18,17 @@ The downstream judge-building steps are identical for all three forms. Also
 arrive with:
 
 - one or more specific binary **failure modes**, such as
-  `correctly_identified_issue` or `escalated_when_required`;
+  `correctly_identified_issue` or `escalated_when_required`, or one pairwise or
+  multi-class decision (see “Multi-class and pairwise judges”);
 - the original dataset row for every label; and
 - multiply labeled rows for measuring human–human agreement.
 
 If the labels are Likert scores or an “overall quality” bundle, return to the
 First win's conditional ground-truth annotation stage and rewrite the rubric as one binary question
 per failure mode before building a judge.
+Pairwise labels (`a`, `b`, or `tie`) and other fixed-category labels are
+already decision-shaped: keep them and follow “Multi-class and pairwise
+judges” below.
 
 - A **judge** is the evaluator, often an LLM instruction set plus a runner.
 - A **scorer** is the metric contract stored in Orizu. It names the score,
@@ -42,6 +46,8 @@ positive rate)** is the fraction of human-labeled failures the judge catches.
 correctly leaves unflagged.
 
 Measure kappa, TPR, and TNR separately for every failure mode on judge-test rows.
+For a pairwise or multi-class judge, TPR and TNR are undefined; use kappa plus
+each label's recall and precision (see “Multi-class and pairwise judges”).
 Always show the confusion matrix, sample count, class balance, and scenario-class
 breakdown beside them. A judge that flags everything can have perfect TPR and
 useless TNR; a judge that passes everything has the inverse failure. If a
@@ -128,7 +134,8 @@ def has_case_reference(input: dict, output: dict) -> bool:
 ```
 
 For a nuanced failure mode, make the LLM judge return one binary decision and a
-short reason. Its instructions should contain:
+short reason (for pairwise or multi-class labels, one label from the fixed set;
+see “Multi-class and pairwise judges”). Its instructions should contain:
 
 1. the binary question;
 2. specific pass and fail criteria;
@@ -154,6 +161,114 @@ Frame every scorer output so higher means better; for the binary row score,
 declaring `higher_is_better: true`, because that gives optimization and
 reflection mixed directional signals.
 
+### Multi-class and pairwise judges
+
+When reviewers compared two outputs and picked `a`, `b`, or `tie`, or chose one
+of a fixed set of categories, the judge predicts that same label. Its strict
+JSON result is `{"label": "a", "reason": "…"}` with `label` exactly one of the
+allowed values. Orizu does not check judge labels: the runner passes
+`model_response` through unchanged and `cohens_kappa` accepts any label as a new
+class. Your judge runner must validate the label and turn anything else into a
+row error.
+
+For pairwise judges, the same runner controls position bias; GEPA changes the
+instructions, not the runner, so no other component can. The default is to
+pick a random-looking but fixed order for each row, map the verdict back to the
+dataset's own `a` and `b` before writing `model_response`, and record the order
+shown, so the scorer compares labels directly. Derive the order from a hash of
+the row's own content, not from `random.random()`: GEPA calls the runner
+separately for the seed and for every candidate, so a fresh draw each call
+would show the same row in different orders to different candidates, and
+position bias would then decide part of which candidate wins. The candidate
+runner receives the dataset row's fields, not Orizu's row id, and Python's
+built-in `hash()` changes between processes, so use a stable digest. Showing each pair in both orders instead doubles
+judge calls against the run budget; if you do it, fold the two verdicts into
+the single per-row output: the same verdict in both orders keeps that verdict,
+and different verdicts become `tie`. The core of the default runner, which reads the candidate-runner contract
+(`payload["row"]` is the flat dataset row):
+
+```python
+import hashlib, json, os
+
+ALLOWED = {"a", "b", "tie"}
+payload = json.load(open(os.environ["ORIZU_RUNNER_INPUT_PATH"]))
+row = payload["row"]  # flat dataset row: {"reply_a": ..., "reply_b": ..., ...}
+digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).digest()
+swapped = digest[0] % 2 == 1  # same order for this row in every call
+first, second = (row["reply_b"], row["reply_a"]) if swapped else (row["reply_a"], row["reply_b"])
+verdict = call_judge(instructions, first, second)  # parsed JSON, any value
+if not isinstance(verdict, dict) or verdict.get("label") not in ALLOWED:
+    output = {"model_response": None,
+              "error": f"Judge label must be one of {sorted(ALLOWED)}; got {verdict!r}"}
+else:
+    if swapped and verdict["label"] != "tie":
+        verdict["label"] = "b" if verdict["label"] == "a" else "a"
+    verdict["shown_order"] = "b_first" if swapped else "a_first"
+    output = {"model_response": verdict, "error": None}
+json.dump(output, open(os.environ["ORIZU_RUNNER_OUTPUT_PATH"], "w"))
+```
+
+The GEPA alignment row scorer compares the candidate judge's label with the
+human label and explains every miss, because reflection learns from that
+feedback. A match scores 1 and anything else scores 0. Under the default GEPA
+scorer contract, the scorer's `payload["row"]` is not the flat dataset row but
+a wrapper (`prompt-control-plane.md`, “GEPA scorer-runner contract”): the
+dataset row is under `source_row`, and `candidate_output` is exactly what the
+judge runner wrote as `model_response`, passed through unchanged. With the
+runner above that is the validated verdict object, or `None` when the runner
+reported an error:
+
+```python
+import json, os
+
+ALLOWED = {"a", "b", "tie"}
+payload = json.load(open(os.environ["ORIZU_RUNNER_INPUT_PATH"]))
+wrapper = payload["row"]  # GEPA wrapper, not the flat dataset row
+row = wrapper["source_row"]  # the same flat dataset row the judge runner saw
+human = row["preferred"]  # the dataset's human-label field
+verdict = wrapper.get("candidate_output")
+judge = verdict.get("label") if isinstance(verdict, dict) else None
+error = None
+if human not in ALLOWED:
+    score = 0.0
+    feedback = "Row not scored: the human label is outside the allowed set (a data problem, not a judge problem)."
+    error = f"Human label {human!r} is not one of {sorted(ALLOWED)}; fix the label field or ALLOWED"
+elif wrapper.get("candidate_error"):
+    score = 0.0
+    feedback = "Row not scored: the judge runner reported an error (provider failure or invalid label), so there is no verdict to compare."
+    error = f"Judge produced no usable output: {wrapper['candidate_error']}"
+elif judge not in ALLOWED:
+    score, feedback = 0.0, f"Judge output must be JSON with label one of {sorted(ALLOWED)}; got {verdict!r}"
+elif judge == human:
+    score, feedback = 1.0, f"Matched reviewers ({human})."
+else:
+    score, feedback = 0.0, f"Reviewers chose {human}; judge chose {judge}. Judge reason: {verdict.get('reason', '')}"
+json.dump({"score": score, "feedback": feedback, "error": error},
+          open(os.environ["ORIZU_RUNNER_OUTPUT_PATH"], "w"))
+```
+
+Every row still needs a numeric `score`. A row the scorer cannot judge (a
+human label outside the allowed set, or a judge row the runner reported as an error) carries its
+reason in `error`, and the run marks the row as errored. Its `feedback` must
+still be a short, non-empty, factual note: reflection refuses a row with empty
+feedback and stops the run. Keep that note about the missing verdict, not about
+the judge's instructions, so reflection does not try to fix a provider failure
+by rewriting the judge. A seed on which every row errors is refused
+at launch.
+
+Register it with `higher_is_better: true` and run GEPA exactly as in
+“Optimize the judge instructions with GEPA” below. Compared with the binary
+case, the label field and the allowed labels differ, and so does the kappa set
+scorer: reuse the `cohens_kappa` builtin from `prompt-control-plane.md` with
+`predicted_label` mapped to the judge's `label`, drop `positive_class`, and
+replace the binary `diagnostics_schema` keys (`confusion_matrix`,
+`flag_recall`, `ok_recall`, and the rest) with `labels` and `per_class`.
+Without a positive class the scorer computes multi-class kappa and accuracy
+over every label it sees and reports per-label precision and recall under
+`per_class` only; it emits no `confusion_matrix` or `<label>_precision` keys.
+Agree the trust bar on that kappa plus the per-label recall in `per_class`,
+especially for `tie`, which is usually the rarest label.
+
 ## Split, iterate, and validate
 
 Build the labeled judge pool only from the application's train and validation
@@ -167,6 +282,9 @@ judge-test split is yours to define:
 | Judge dev / validation (`validation`) | ~40% | Iterate and compare candidates |
 | Judge test (`judge-test`) | ~40% | One alignment validation after the judge candidate is frozen |
 
+A small judge-train partition is normal for GEPA: tens of rows are common. Do
+not shrink validation or judge-test to enlarge it.
+
 Author `judge-split.json` with explicit row membership for all three judge
 partitions. Every `row_ids` entry must come from the application's train or
 validation partition; never include an application final-held-out row. Require
@@ -319,7 +437,9 @@ agreed judge trust bar, the scorer versions are registered, `runners exec`, `sco
 and the measured `scorers exec` run have all succeeded, and the human curator has
 accepted that score run. Gate decisions only on accepted evidence. A locally
 computed alignment report with no accepted Orizu score run does not clear the
-trust bar.
+trust bar. Judge GEPA (“Optimize the judge instructions with GEPA” below) runs
+inside this stage, before the bar is cleared; the accepted judge-test evidence
+for the selected judge is what completes it.
 
 Re-run this evidence path after labels, judge instructions, runners, or scorer
 definitions change. Builtin set scorers, their dependency mapping, and aggregate
@@ -329,14 +449,21 @@ submission behavior are specified in `prompt-control-plane.md`.
 
 Judge building is itself an optimization problem. Optimize the LLM judge's own
 instructions against human labels; do not optimize it against its own previous
-decisions.
+decisions. This run happens before the judge clears its trust bar, so the
+“validated judges only” rule for application optimization does not gate it:
+here the gate is the human labels plus the alignment scorer that compares the
+candidate judge with them. Judge-test still decides, once, after the run,
+whether the selected judge clears its bar. For an LLM judge built from human labels, this
+run is the default way to reach the trust bar, not an optional extra; the data
+guidance in “Run it” in `optimization-with-gepa.md` applies here too.
 
 Manage a composite judge as one `orizu instructions` instruction set. Components
 can separate the rubric, boundary examples, output contract, and failure-mode
 rules, while the complete component map remains the candidate. The candidate
 runner executes those candidate judge instructions on human-labeled train and
-validation rows. The row-mode alignment scorer compares the binary candidate
-decision with the human label and returns per-row feedback for reflection.
+validation rows. The row-mode alignment scorer compares the candidate's
+decision (binary, pairwise, or multi-class) with the human label and returns
+per-row feedback for reflection.
 
 Materialize those components in `orizu.instruction-set.json` before GEPA. Create
 the stable instruction set once, or push a revised seed for a later optimization
@@ -476,11 +603,14 @@ in `prompt-control-plane.md` before choosing either contract.
 
 ## Exit criterion checklist
 
-- [ ] Each failure mode is binary and has adequate positive and negative rows.
+- [ ] Each judge decision is either binary with adequate positive and negative
+      rows, or pairwise or multi-class with adequate rows for every label.
 - [ ] Human–human kappa is computed from multiply labeled rows; low agreement
       routes back to rubric and label repair.
 - [ ] Kappa, TPR, TNR, confusion matrix, sample size, class balance, and
-      scenario-class results are reported per failure mode.
+      scenario-class results are reported per failure mode; for a pairwise or
+      multi-class judge, kappa, accuracy, and `per_class` precision and recall
+      stand in for TPR, TNR, and the confusion matrix.
 - [ ] The agent taught the decision-class tradeoffs, the user chose each
       judge trust bar, and the agreed value is recorded in the output.
 - [ ] Development rows were used for iteration and judge-test was used only

package/vendor/skills/orizu/references/cli-reference.md

@@ -114,7 +114,7 @@ Behavior:
 - `skills path --json` returns where this skill's source lives (`root`, `skillMd`, `source`, `cliVersion`, `skillHash`) so you can read it without installing, and verify an installed copy matches the CLI.
 - `skills status` reports each install target as missing/current/stale/broken-link; `skills update` refreshes stale copies and repairs broken symlinks.
 - `install-skill` selects targets by agent name; `--scope global` (default) installs user-level (`~/.agents/skills`, `~/.claude/skills`), `--scope project` installs into the repo. Project-scope installs always copy; global installs symlink when the CLI path is stable. Advanced target IDs remain available via `--target`.
-- `orizu setup` is the guided onboarding flow (login → choose/create team → choose/create project → workspace contract → coding-agent skill → optional launch). Non-interactive setup requires authentication plus explicit team and project choose/create flags; run `orizu login --headless` first when needed. After interactive skill-installation consent, a destination picker starts with `Universal (.agents — Codex and others) — ~/.agents/skills/orizu` and shows each detected native target with its `~/.../orizu` destination on the same line before any write. Config-root detection requires a directory rather than a same-named regular file. Every visible destination starts selected, but every destination can be deselected. Universal and OpenCode are independent choices, so either can be installed without the other. Confirming an empty selection skips skill installation successfully. Non-interactive `--agent` semantics are unchanged: repeat `--agent` to opt into installs, each selected agent always includes Universal plus its native target only where needed, duplicate destinations are removed, and no agent flags writes no skills. Non-interactive setup never launches. Interactive setup can hand off to installed Claude, Codex, Pi, OpenCode, and Cursor commands. Available commands appear in that order, with the first available command selected by default. Other is always last and is the only option when no supported command is installed. Enter launches the highlighted agent; Esc skips agent launch and completes setup. A selected agent receives one positional prompt naming the selected `team/project` and `https://orizu.ai/llms.txt`, without shell interpolation. Selecting Other prints the complete prompt for copying without launching anything. `--handoff` explicitly prints the prompt; `--launch claude|codex|pi|opencode|cursor` explicitly selects an installed agent, still requires an interactive terminal, and asks for confirmation unless `--yes` is supplied. OpenCode launches as `opencode run <prompt>`. Cursor uses the `agent` executable, not the Cursor editor `cursor` command. The official [Cursor CLI overview](https://cursor.com/docs/cli/overview) and [parameter reference](https://cursor.com/docs/cli/reference/parameters), retrieved 2026-08-30, document direct `agent "prompt"` and `agent -p "prompt"` initial prompts; setup uses the direct positional form without a shell. Authenticated workspace setup still materializes every project in the team. The contract uses root `AGENTS.md`, `CLAUDE.md`, `Memory.md`, `orizu.team.json`, project manifests, primitive directories, and gitignore policy. `.orizu/` and `.logs/` are ignored cache/generated state, not the durable contract.
+- `orizu setup` is the guided onboarding flow (login → choose/create team → choose/create project → workspace contract → coding-agent skill → optional launch). Non-interactive setup requires authentication plus explicit team and project choose/create flags; run `orizu login --headless` first when needed. After interactive skill-installation consent, a destination picker starts with `Universal (.agents — Codex and others) — ~/.agents/skills/orizu` and shows each detected native target with its `~/.../orizu` destination on the same line before any write. Config-root detection requires a directory rather than a same-named regular file. Every visible destination starts selected, but every destination can be deselected. Universal and OpenCode are independent choices, so either can be installed without the other. Confirming an empty selection skips skill installation successfully. Non-interactive `--agent` semantics are unchanged: repeat `--agent` to opt into installs, each selected agent always includes Universal plus its native target only where needed, duplicate destinations are removed, and no agent flags writes no skills. Non-interactive setup never launches. Interactive setup can hand off to installed Claude, Codex, Pi, OpenCode, and Cursor commands. Available commands appear in that order, with the first available command selected by default. Other is always last and is the only option when no supported command is installed. Enter launches the highlighted agent; Esc skips agent launch and completes setup. A selected agent receives one positional prompt naming the selected `team/project` and `https://orizu.ai/llms.txt`, without shell interpolation. Selecting Other prints the complete prompt for copying without launching anything. `--handoff` explicitly prints the prompt; `--launch claude|codex|pi|opencode|cursor` explicitly selects an installed agent, still requires an interactive terminal, and asks for confirmation unless `--yes` is supplied. OpenCode launches as `opencode run <prompt>`. Cursor uses the `agent` executable, not the Cursor editor `cursor` command. The official [Cursor CLI overview](https://cursor.com/docs/cli/overview) and [parameter reference](https://cursor.com/docs/cli/reference/parameters), retrieved 2026-08-30, document direct `agent "prompt"` and `agent -p "prompt"` initial prompts; setup uses the direct positional form without a shell. Authenticated workspace setup materializes only the project named by `--project` or created by `--create-project`, with no checklist, interactive or not. When the project is picked interactively instead, a checklist of the team's projects follows, all checked (Esc keeps them all), and setup materializes the checked ones plus the active project. `--fix` restores every team project. The contract uses root `AGENTS.md`, `CLAUDE.md`, `Memory.md`, `orizu.team.json`, project manifests, primitive directories, and gitignore policy. `.orizu/` and `.logs/` are ignored cache/generated state, not the durable contract.
 
 ### Teams
 
@@ -523,7 +523,7 @@ exist.
 Optimization trace commands:
 
 ```bash
-orizu optimizations run-gepa ... [--scorer-input-contract gepa|flat_row] [--scorer-candidate-field <row-field>] [--allow-degenerate-seed] [--candidate-selection-strategy pareto|current_best|epsilon_greedy] [--epsilon N] [--objective <text>] [--candidate-proposer skilled-proposer] [--candidate-proposer-config @file] [--proposal-max-calls N] [--proposal-max-tokens N] [--python <command>] [--num-threads auto|N] [--reflection-retry-attempts 3] [--reflection-http-timeout-seconds 180] [--log-dir logs] [--no-local-log]
+orizu optimizations run-gepa ... (--candidate-version-id <id> | --instruction-set <set[/profile[@vN]]> [--model-config <identity>] [--profile-version <N>]) [--scorer-input-contract gepa|flat_row] [--scorer-candidate-field <row-field>] [--allow-degenerate-seed] [--candidate-selection-strategy pareto|current_best|epsilon_greedy] [--epsilon N] [--objective <text>] [--candidate-proposer skilled-proposer] [--candidate-proposer-config @file] [--proposal-max-calls N] [--proposal-max-tokens N] [--python <command>] [--num-threads auto|N] [--reflection-retry-attempts 3] [--reflection-http-timeout-seconds 180] [--log-dir logs] [--no-local-log]
 orizu optimizations run-gepa --hosted ... --budget auto|light|medium|heavy [--launch-intent-id <uuid>]
 orizu optimizations export <optimization-run-id> --out ./optimization.json
 orizu optimizations export <optimization-run-id> --json

package/vendor/skills/orizu/references/migrate-existing-gepa-setup.md

@@ -353,8 +353,7 @@ reflection model is refused at launch without it
 `packages/orizu-gepa/src/orizu_gepa_connector/runtime.py:193-196`). Drop it only
 if you also pass an `openai/...` `--reflection-model`.
 
-`--instruction-set` and `--model-config` select exactly one Profile. The CLI
-uses only that Profile's Production Version. A missing Profile refuses with `instruction_set_profile_not_found`, and an unpromoted Profile refuses with `instruction_set_profile_not_promoted`, instead of using Default; the CLI also refuses the
+`--instruction-set` and `--model-config` (or one `set/profile` specifier) select exactly one Profile. A plain `set/profile` starts the run from that Profile's Production Version. To start from any other version, promoted or not, name it with `set/profile@vN` or `--profile-version <N>`; its components must match the set's current shape exactly, or launch refuses and names the newest matching version. Production must match the shape too. A missing Profile refuses with `instruction_set_profile_not_found`, and an unpromoted Profile with no version named refuses with `instruction_set_profile_not_promoted`, instead of using Default; the CLI also refuses the
 legacy `--candidate-version-id` selector when the instruction-set selectors are present. `round-robin` updates one component per
 round; use `--component-selector all` only when every component should be
 reflected on each round.

package/vendor/skills/orizu/references/optimization-reports.md

@@ -68,7 +68,7 @@ Keep the three data roles separate:
 - Validation data ranks every candidate that received a full validation evaluation. A candidate comparison must use this data, because the optimizer is allowed to inspect validation results while choosing.
 - Use Final-held-out only for the untouched final check after selecting one candidate, and compare only the seed with that selected candidate. Do not rank candidates on Final-held-out or use its results to choose a different candidate.
 
-For a multi-component instruction-set run, the **seed** is the customer-authored profile version the run started from, and each candidate is a complete proposed profile version. A plain prompt and a set-of-one instruction set are both recorded with a `prompt_version_id` and no `instruction_set_profile_version_id`; that storage fact makes legacy promotion available but does not make it a valid shipping path for the set-of-one run. Report each scenario class from the ratified improvement plan separately, such as “refund requests with missing order details”; never replace scenario-class results with pass/fail buckets.
+For a multi-component instruction-set run, the **seed** is the profile version the run started from (production unless the run named another version, promoted or not), and each candidate is a complete proposed profile version. A plain prompt and a set-of-one instruction set are both recorded with a `prompt_version_id` and no `instruction_set_profile_version_id`; that storage fact makes legacy promotion available but does not make it a valid shipping path for the set-of-one run. Report each scenario class from the ratified improvement plan separately, such as “refund requests with missing order details”; never replace scenario-class results with pass/fail buckets.
 
 Prefer the local log directory printed by `run-gepa`. Core files (conditional files are listed below):
 

package/vendor/skills/orizu/references/optimization-with-gepa.md

@@ -2,9 +2,26 @@
 
 How to optimize an instruction-set profile against validated judges and scorers. Prepare the complete instruction-set manifest, follow the [Authority map](authority-map.md) for its mutation, and then run the bundled `orizu optimizations run-gepa` flow. Use this reference for GEPA mechanics, custom optimizer implementations, and optional DSPy context for customers already using DSPy.
 
+## Run it
+
+Once a validated judge or scorer exists, launch `orizu optimizations run-gepa`. That run is the default end of First win, not an optional extra.
+
+How much data is enough: tens to a few hundred labelled rows is normal; under 100 is common. GEPA learns from per-row scorer feedback, not from gradients, so data size alone is not a reason to skip the run. Neither is a seed that already looks close to the human ceiling, nor a hand-written "GEPA-style" rewrite. Run it and let the validation split choose. When a non-seed candidate is selected, the untouched Final-held-out comparison in Promote decides whether it helped. When GEPA keeps the seed, record the seed-selected outcome; when search produces no valid candidate, record the no-valid-candidate outcome. Both are valid results, not wasted runs, and Promote reports each through its own branch. If you still believe a run is not worth it, record the reason and get the human's approval before skipping, as `flows/first-win.md` describes.
+
+Shortest path to a first run (each item matches "Orizu-tracked optimization" step 1–6 below):
+
+1. Create or push the instruction set from its manifest, and keep the profile version it reports. A new set's first version (v1) has no Production pointer, so name it at launch with `@v1` or `--profile-version 1`; never promote it just to make the run start, because moving Production is a human decision.
+2. Push the candidate runner (and, for an LLM evaluator, the judge rubric).
+3. Register a row scorer whose feedback explains each score.
+4. Snapshot a dataset version, then create one split set with train, validation, and a reserved Final-held-out partition. Pass only train and validation to `run-gepa`, and never pass Final-held-out as `--train-split` or `--val-split`: the CLI does not enforce that boundary.
+5. Push the optimizer with `orizu optimizers push`.
+6. Launch `orizu optimizations run-gepa` on the train and validation partitions with `--budget light`. Pass `--reflection-max-tokens <n>`: the default reflection model, `anthropic/claude-opus-4-7`, uses the Anthropic Messages protocol, and launch refuses without a cap. Scale the budget up only if the run is still finding better candidates.
+
 When First win loads this reference, stop after the optimization run records a selected candidate or no-valid-candidate outcome. Do not write the decision report or run a promotion command until `flows/promote.md` is active. Steps 8–9 below describe Promote-owned continuation mechanics, not First win work.
 
-Select the Profile explicitly with `--instruction-set <slug-or-exact-name> --model-config <identity>`. Those two selectors replace the legacy `--candidate-version-id` path and cannot be combined with it. The connector optimizes that Profile's complete Production component map. If the Profile is missing, launch refuses with `instruction_set_profile_not_found`; if it is unpromoted, launch refuses with `instruction_set_profile_not_promoted`. It never substitutes the Default Profile. `--component-selector round-robin` is the default and updates one component per round; `--component-selector all` updates every component per round. The runner receives a multi-component candidate as a component map, while a one-component set keeps the existing single-body runner contract. Git-pinned components and malformed component maps are refused before a run starts. Automatic promotion is refused for multi-component sets; write the report, obtain the human promotion decision, and follow the Authority map. Manifest create/push is the pre-run seed path, not a second accepted-candidate materialization; after a run, use the human one-shot or equivalent idempotent two-stage path in step 9.
+## Selecting the profile and seed
+
+Select the Profile explicitly with `--instruction-set <slug-or-exact-name> --model-config <identity>`, or with one specifier such as `--instruction-set planner/openai/gpt-5.4`. These selectors replace the legacy `--candidate-version-id` path and cannot be combined with it. The run starts from its *seed*: that Profile's Production component map unless you name another version with `planner/openai/gpt-5.4@v3` or `--profile-version 3` (a version number only; `v3` also works). Any version of the Profile can be the seed, promoted or not, and the seed brings its own components and model settings. When the specifier and the flags both name the profile or the version, they must agree or launch refuses with `instruction_set_seed_conflict`. A named version that does not exist refuses with `instruction_set_profile_version_not_found` and names the newest version. The seed's components must match the set's current shape exactly, with none missing and none the set has since removed. Launch refuses a named version that does not with `instruction_set_profile_version_incomplete` and names the newest matching version, and refuses a Production that does not (locally with `instruction_set_profile_production_incomplete`; a hosted launch names the production version). If the Profile is missing or its model config is archived, launch refuses with `instruction_set_profile_not_found`; if no version is named and it is unpromoted, launch refuses with `instruction_set_profile_not_promoted`. It never substitutes the Default Profile. When you promote a candidate whose seed is older than the Profile's current Production, `optimizations promote` warns that the changes made since the seed are not in the candidate; tell the human before they decide. `--component-selector round-robin` is the default and updates one component per round; `--component-selector all` updates every component per round. The runner receives a multi-component candidate as a component map, while a one-component set keeps the existing single-body runner contract. Git-pinned components and malformed component maps are refused before a run starts. Automatic promotion is refused for multi-component sets; write the report, obtain the human promotion decision, and follow the Authority map. Manifest create/push is how a profile's first version is created before a run, not a second accepted-candidate materialization; after a run, use the human one-shot or equivalent idempotent two-stage path in step 9.
 
 ## Inputs
 
@@ -13,7 +30,7 @@ You should arrive here with:
 - A **dataset** of inputs to optimize against: usually the same exported labels, plus any harder cases you've added since.
 - An **instruction set** whose selected profile contains the starting component values for the LLM application you want to improve.
 
-If you don't have a validated judge, stop. Optimizing against an unvalidated judge means you'll hill-climb on a noisy or biased signal: Goodhart's law in action.
+Validate the judge first (see `building-judges.md`): optimizing against an unvalidated judge hill-climbs a noisy or biased signal. Once it clears its trust bar, run GEPA. The validated judge, not the size of the dataset, is the gate. When the thing being optimized is the judge itself, there is no earlier judge to validate: the gate is the human labels plus the alignment scorer that compares the judge with them (see “Optimize the judge instructions with GEPA” in `building-judges.md`).
 
 ## Why GEPA-style optimization
 
@@ -342,9 +359,10 @@ Each pass through the loop reveals the next layer.
 ## Common pitfalls
 
 - Optimizing against an unvalidated judge. You'll improve the metric and degrade the system. Always validate first.
-- No Final-held-out comparison. "It's better, look at the metric" without Final-held-out is meaningless: GEPA will overfit if you let it.
+- Skipping the Final-held-out comparison. "It's better, look at the metric" without Final-held-out is meaningless. The validation split selects the candidate and Final-held-out decides whether it helped; with both in place, a small dataset is not an overfitting reason to skip the run.
+- Skipping the run, or replacing it with a hand-written "GEPA-style" rewrite. Neither produces a recorded run, candidates, or the evidence Promote needs.
 - Hiding regressions in the average. Track per-failure-mode metrics alongside the combined score.
-- Over-budgeting GEPA. Heavy budgets give diminishing returns and burn LM spend. Start with `auto="light"`, scale up only if needed.
+- Over-budgeting GEPA. Heavy budgets give diminishing returns and burn LM spend. Start with `auto="light"` (`--budget light` for `run-gepa`) and scale up only if needed; a light budget on a small dataset is a normal first run.
 - Ignoring temperature. Run optimization with the same LM config (model, temperature) you use in production. Optimizing against gpt-4o at temp=0 doesn't transfer to gpt-4o-mini at temp=0.7.
 - Recreating Orizu logging by hand for instruction sets. Use `orizu optimizations run-gepa` unless the optimizer is genuinely custom.
 

package/vendor/skills/orizu/references/primer.md

@@ -13,7 +13,7 @@ Onboard → First win ───────────────→ Promote
                                                    └─ after Promote: Recurse, Triage, or Expand by user interest
 ```
 
-First win builds the evidence and ends at a completed optimization run. Promote owns reporting, validation, and the human promotion decision. Recurse turns newly deployed traces into a later same-surface cycle; Triage starts from an incident and closes its eval gap before a fix; Expand chooses and plans another surface. Onboard owns pre-data inventory and plan ratification.
+First win builds the evidence and ends at a completed optimization run, or at a skip the human approved and recorded. Promote owns reporting, validation, and the human promotion decision. Recurse turns newly deployed traces into a later same-surface cycle; Triage starts from an incident and closes its eval gap before a fix; Expand chooses and plans another surface. Onboard owns pre-data inventory and plan ratification.
 
 The CLI covers this operational loop. Start instruction sets with `orizu instructions`: one instruction set per agent or LLM experience, one profile per model config, and components owned by that profile. Human annotation happens through apps/tasks only for required ground truth, while judge/scorer logic and optimization execution run locally and report back through the instruction control plane. The active flow file owns ordering and its exit criterion; this primer explains why the stages exist rather than replacing that procedure.
 
@@ -205,7 +205,7 @@ Done locally, reported to Orizu:
 2. Register validated scorers.
 3. Run the bundled Orizu GEPA-style text optimizer, or a custom optimizer against the scorer set.
 4. Stream optimization events to Orizu and complete the run with a selected candidate or a recorded no-valid-candidate outcome.
-5. Stop First win at that completed run. Route to Promote for the report, supported Final-held-out comparison, validation evidence, and human ship, gather-more-evidence, or do-not-promote decision. After Promote, use Recurse for a human-agreed cadence that turns traces from the currently deployed version into a new immutable dataset version.
+5. Stop First win at that completed run, or at a skip the human approved and recorded. Route to Promote for the report, supported Final-held-out comparison, validation evidence, and human ship, gather-more-evidence, or do-not-promote decision. After Promote, use Recurse for a human-agreed cadence that turns traces from the currently deployed version into a new immutable dataset version.
 
 Control-plane commands: `prompt-control-plane.md`. Detailed walkthrough: GEPA mechanics, optional DSPy context for customers already using it, and before/after comparison: `optimization-with-gepa.md`.
 

package/vendor/skills/orizu/references/prompt-control-plane.md

@@ -811,7 +811,12 @@ orizu --local optimizations run-gepa \
 Useful GEPA flags:
 
 - `--instruction-set <slug-or-exact-name> --model-config <identity>` selects a
-  resolving instruction-set profile. `--component-selector round-robin|all`
+  resolving instruction-set profile. A specifier `set/profile` or
+  `set/profile@vN` names the profile without `--model-config`;
+  `@vN` or `--profile-version <N>` starts the run from that version (the
+  *seed*) instead of production. The seed must match the set's current shape
+  exactly (no missing or removed components); launch refuses one that does not
+  and names the newest version that matches. `--component-selector round-robin|all`
   controls which components GEPA proposes changing; `all` lets it optimize the
   complete component map. `--candidate-version-id` remains a compatibility
   input for a single legacy prompt version and is mutually exclusive with the

package/vendor/skills/orizu/scripts/test-app.mjs

@@ -9,6 +9,7 @@
  *   1. App.tsx has a single named default export (function or class).
  *   2. Default export's first parameter destructures inputData, onComplete,
  *      initialValues (and not the deprecated data, onSubmit).
+ *      The default export must not be async.
  *   3. Imports resolve against the platform-provided component registry.
  *   4. input.json and output.json use only the supported JSON Schema subset:
  *      type, required, properties, items, enum.
@@ -121,6 +122,48 @@ function warn(message) {
   issues.push({ level: "warn", message });
 }
 
+/**
+ * Removes line and block comments, walking left to right so a `//` or `/*`
+ * inside a quoted string or template literal is left alone. A block comment
+ * keeps its newlines, so line numbers in later messages stay true.
+ * Regex literals are not tracked; a quote inside one can hide a comment.
+ */
+function stripComments(code) {
+  let out = "";
+  let quote = "";
+  let i = 0;
+  while (i < code.length) {
+    const char = code[i];
+    const next = code[i + 1];
+    if (quote) {
+      out += char;
+      if (char === "\\" && next !== undefined) {
+        out += next;
+        i += 2;
+        continue;
+      }
+      // Single and double quoted strings end at the line's end; templates may span lines.
+      if (char === quote || (char === "\n" && quote !== "`")) quote = "";
+      i++;
+    } else if (char === '"' || char === "'" || char === "`") {
+      quote = char;
+      out += char;
+      i++;
+    } else if (char === "/" && next === "/" && code[i - 1] !== ":") {
+      while (i < code.length && code[i] !== "\n") i++;
+    } else if (char === "/" && next === "*") {
+      const close = code.indexOf("*/", i + 2);
+      const end = close === -1 ? code.length : close + 2;
+      out += code.slice(i, end).replace(/[^\n]/g, " ");
+      i = end;
+    } else {
+      out += char;
+      i++;
+    }
+  }
+  return out;
+}
+
 function checkAppFile(path) {
   if (!existsSync(path)) {
     err(`App file not found: ${path}`);
@@ -129,13 +172,19 @@ function checkAppFile(path) {
   const src = readFileSync(path, "utf8");
 
   // Strip line comments and block comments to avoid false matches.
-  const stripped = src
-    .replace(/\/\*[\s\S]*?\*\//g, "")
-    .replace(/(^|[^:])\/\/[^\n]*/g, "$1");
+  const stripped = stripComments(src);
+
+  // App code renders as a client component; React 19 cannot render an async one.
+  if (/export\s+default\s+async\s+function\b/.test(stripped)) {
+    err(
+      "App components cannot be async; return the element synchronously and load data in an effect."
+    );
+    return;
+  }
 
   // 1. Find a named default export.
   const fnMatch = stripped.match(
-    /export\s+default\s+(?:async\s+)?function\s+([A-Za-z_$][\w$]*)\s*\(([^)]*)\)/
+    /export\s+default\s+function\s+([A-Za-z_$][\w$]*)\s*\(([^)]*)\)/
   );
   const classMatch = stripped.match(
     /export\s+default\s+class\s+([A-Za-z_$][\w$]*)/