orizu 0.5.21 → 0.5.23

package/vendor/skills/orizu-cli/references/migrate-existing-gepa-setup.md

@@ -1,22 +1,28 @@
 # Migrating an existing GEPA setup into Orizu
 
 For a customer already running official GEPA from a plain script — inline
-`trainset`/`valset`, a seed prompt in a dict, their own `metric(row, output)`.
+`trainset`/`valset`, seed component values in a dict, and their own
+`metric(row, output)`. The dict becomes one instruction-set profile: its keys
+become the set's fixed shape, and its values become that profile's components.
+Use `orizu instructions` as the only creation and update path for those
+customer-owned instructions. Prepare the complete manifest, then follow the
+[Authority map](../SKILL.md#authority-map) for its mutations.
 
 The ordering rule: **nothing is optimized until parity is proven.** The last
 step before `run-gepa` is always `orizu scorers verify-parity`, which runs the
 migrated scorer runner under the exact payload `run-gepa` sends it and the
 customer's original metric over the same rows, and exits 0 only if they agree.
 
-Commands are shown with `--local`; drop it against a hosted server.
+Commands are shown with `--local`; drop it against a hosted server and follow
+the Authority map for surface-specific execution and hand-offs.
 
 ## 0. Read their script
 
 Note the row shape and which field is the stable id, `trainset`/`valset`
-membership, the seed candidate text, the metric's `module:function`, and what
-it returns (a float, a dict with `score`, or GEPA's `EvaluationResult` — all
-three are accepted; `objective_scores` is ignored). If rows have no stable id,
-add one now: every step below pairs on it.
+membership, the seed component map, the metric's `module:function`, and what it
+returns (a float, a dict with `score`, or GEPA's `EvaluationResult` — all three
+are accepted; `objective_scores` is ignored). If rows have no stable id, add
+one now: every step below pairs on it.
 
 ## 1. Set up, then export the rows
 
@@ -24,32 +30,44 @@ add one now: every step below pairs on it.
 orizu setup --team <team-slug>
 ```
 
-One dataset holds every row; the split set decides train vs validation. Dump
-rows verbatim from their script, and mirror their lists exactly — a resampled
-split makes every later number incomparable with the numbers they have.
-
-Build the dataset from the **id-deduplicated union**: a row reused in both
-`TRAINSET` and `VALSET` would otherwise be written twice, and the upload route
-rejects duplicate canonical row ids (`app/api/datasets/route.ts:206-211`,
-"csvData must contain unique canonical row ids"). The partition lists keep the
-original ids independently, so a shared row can still appear in both.
+One dataset holds every row; require four disjoint, nonempty partitions:
+`TRAINSET`, `VALSET`, `PARITYSET`, and `FINAL_HELD_OUT`. Mirror existing
+train/validation membership exactly. Here, `parity` is a migration-only
+auxiliary partition, not an application partition; the canonical J3
+application doctrine remains train, validation, and final-held-out. Source and
+approve representative `PARITYSET` rows before reserving final-held-out, then
+reserve `FINAL_HELD_OUT` only from the remaining rows. Apply J3's coverage
+discipline:
+every named scenario class must have nonzero train, validation, and
+final-held-out counts. Stop when that coverage, ordering, or four-way
+disjointness cannot be proven; a resampled or overlapping split makes later
+comparisons invalid.
 
 ```python
 import json
-seen, rows = set(), []
-for row in TRAINSET + VALSET:
-    if row["id"] in seen:
-        continue
-    seen.add(row["id"])
-    rows.append(row)
+partitions = {
+    "train": TRAINSET,
+    "validation": VALSET,
+    "parity": PARITYSET,
+    "final-held-out": FINAL_HELD_OUT,
+}
+if any(not partition for partition in partitions.values()):
+    raise ValueError("train, validation, parity, and final-held-out must be nonempty")
+partition_ids = {
+    name: [row["id"] for row in partition]
+    for name, partition in partitions.items()
+}
+all_ids = [row_id for ids in partition_ids.values() for row_id in ids]
+if len(all_ids) != len(set(all_ids)):
+    raise ValueError("train, validation, parity, and final-held-out must be disjoint")
+rows = [row for partition in partitions.values() for row in partition]
 with open("dataset.jsonl", "w") as handle:
     for row in rows:
         handle.write(json.dumps(row) + "\n")
 with open("split.json", "w") as handle:
     json.dump({"name": "default", "strategy": "predefined", "seed": 42, "partitions": [
-        {"name": "train", "row_ids": [row["id"] for row in TRAINSET]},
-        {"name": "validation", "row_ids": [row["id"] for row in VALSET]},
-        {"name": "test", "row_ids": []},
+        {"name": name, "row_ids": row_ids}
+        for name, row_ids in partition_ids.items()
     ]}, handle)
 ```
 
@@ -59,45 +77,56 @@ with open("split.json", "w") as handle:
 orizu --local datasets upload --file ./dataset.jsonl \
   --project <team>/<project> --name "GEPA migration rows" --json
 
-orizu --local datasets versions create <dataset-id-or-name> \
-  --project <team>/<project> --label v1 --json      # -> dataset_version_id
-
-orizu --local datasets splits create <dataset-version-id> \
+orizu --local datasets splits create <upload-returned-dataset-version-id> \
   --from-file ./split.json --json                   # -> split_set_id
 ```
 
+Upload creates the initial `v1` snapshot. Capture its returned `dataset_version_id` and use that ID directly for split creation; reserve `datasets versions create` for later snapshots with a different label.
+
 `--from-file` preserves their partitions; the ratio flags
 (`--train/--validation/--test`) resample and must not be used here.
 
-## 3. Candidate runner and seed prompt
+## 3. Candidate runner and seed instruction set
 
-The candidate runner does what `task_lm` did: candidate prompt + row -> output.
+The candidate runner does what `task_lm` did: candidate profile + row ->
+output. A one-component profile keeps the single-body runner contract. For a
+multi-component profile, read the complete map from
+`input["instruction_set"]["components"]`; components belong only to this
+profile and are never shared across profiles or sets.
 
 ```bash
 orizu --local runners push ./candidate-runner \
-  --project <team>/<project> --name gepa-candidate-runner --label default --json
+  --project <team>/<project> --name gepa-candidate-runner --json
 
-orizu --local prompts push ./prompt \
-  --project <team>/<project> --runner-version <candidate-runner-version-id> --json
+orizu --local instructions create ./orizu.instruction-set.json \
+  --project <team>/<project> \
+  --runner-version <candidate-runner-version-id> \
+  --model-config <provider/model> --json
 ```
 
-`./prompt` holds `orizu.prompt.json` plus the body file it names. Only `name`
-is enforced (`app/api/cli/prompts/route.ts:220-221`); `body_file` defaults to
-`prompt.md`, `body_kind` to `text` (`packages/cli/src/index.ts:3966-3985`) and
-`provider_settings` to `{}`. Write all four anyway — a migration whose model
-settings are implicit is a migration nobody can reproduce:
+The agent-created runner version is unlabeled. No standalone runner-label command exists. Following the [Authority map](../SKILL.md#authority-map), if a human accepts its exact ID as default, the human curator runs `orizu --local runners pull gepa-candidate-runner --project <team>/<project> --version <candidate-runner-version-id> --out ./candidate-runner-default-handoff`, leaves that fresh exact-version directory unchanged, then runs `orizu --local runners push ./candidate-runner-default-handoff --project <team>/<project> --name gepa-candidate-runner --label default --json`. The manifest
+carries the set's `name`, human-readable `description`, fixed `shape`, and component values. Put each original dict
+value in a file without changing its bytes. This one-component example maps
+the old `system` key to `system.md`:
 
 ```json
 {
-  "name": "gepa-seed-prompt",
-  "body_file": "prompt.md",
-  "body_kind": "text",
-  "provider_settings": { "model": "claude-sonnet-4-6", "temperature": 0 }
+  "name": "gepa-agent-instructions",
+  "description": "Instructions for the migrated GEPA application",
+  "shape": ["system"],
+  "components": [
+    { "key": "system", "path": "./system.md" }
+  ]
 }
 ```
 
-`prompt.md` carries their seed candidate text verbatim. Returns the seed
-`prompt_version_id`.
+For a multi-component seed, list every original key once in `shape`, in stable
+order, and add one component entry per key. `--model-config` selects the
+profile whose seed is created; it must name an existing model config with the
+same settings used by the original script. The JSON result includes the set's
+stable `slug` and its default profile version. Record the slug — later reads,
+optimization, archiving, and restoration should use it even if the display
+name changes.
 
 ## 4. Wrap their metric in a scorer runner
 
@@ -155,19 +184,18 @@ def metric(row, model_output):                      # what verify-parity calls
 
 ```bash
 orizu --local runners push ./scorer-runner \
-  --project <team>/<project> --name gepa-scorer-runner --label default --json
+  --project <team>/<project> --name gepa-scorer-runner --json
 
-orizu --local prompts push ./scorer-prompt \
+orizu --local judges push ./judge \
   --project <team>/<project> --runner-version <scorer-runner-version-id> --json
 
 orizu --local scorers register --project <team>/<project> \
   --name gepa-migrated-metric --manifest ./scorer.manifest.json \
-  --prompt-version <scorer-prompt-version-id> \
-  --runner-version <scorer-runner-version-id> \
-  --label production --json                        # -> scorer_version_id
+  --prompt-version <judge-version-id> \
+  --runner-version <scorer-runner-version-id> --json # -> scorer_version_id
 ```
 
-`scorer.manifest.json` needs all four fields below;
+Both agent-created versions are unlabeled. For accepted versions, a human curator materializes the exact runner with `orizu --local runners pull gepa-scorer-runner --project <team>/<project> --version <scorer-runner-version-id> --out ./scorer-runner-default-handoff`, leaves that fresh directory unchanged, runs `orizu --local runners push ./scorer-runner-default-handoff --project <team>/<project> --name gepa-scorer-runner --label default --json`, then runs `orizu --local scorers labels set gepa-migrated-metric production --version <scorer-version-id> --project <team>/<project> --json`. `scorer.manifest.json` needs all four fields below;
 `app/api/cli/scorers/route.ts` rejects a manifest missing `implementation_kind`
 or `metric_key`, and `mode` must be `row` or `set`:
 
@@ -180,29 +208,41 @@ or `metric_key`, and `mode` must be `row` or `set`:
 }
 ```
 
-`prompt_runner` is the only kind `runners exec --scorer-version` and
-`verify-parity` can execute, and it requires a backing prompt version. For a
-pure-code metric with no judge text, `./scorer-prompt` is a short prompt
-version documenting the rubric; the runner never reads its body.
+`prompt_runner` is the only implementation kind that `runners exec
+--scorer-version` and `verify-parity` can execute, and the scorer contract
+currently names its backing judge artifact with `--prompt-version`. That flag
+receives the `prompt_version_id` returned by `orizu judges push`; do not replace
+the judge command with a standalone prompt mutation.
 
-## 5. Produce `outputs.jsonl` by running the seed once
+For an LLM metric, `./judge` contains the evaluator rubric the scorer runner
+executes. For a pure-code metric, keep a short judge rubric that documents the
+deterministic rule and expected output, even though the runner does not read
+its body. The judge is the evaluator artifact; the scorer is the metric
+contract that gives its numeric result a name, mode, and direction.
 
-A scorer scores *(row, output)*, so parity needs one candidate output per row.
+## 5. Freeze the original setup's outputs
 
-```bash
-orizu --local runners exec \
-  --prompt-version <seed-prompt-version-id> \
-  --runner-version <candidate-runner-version-id> \
-  --dataset-version <dataset-version-id> \
-  --split-set <split-set-id> --split validation \
-  --out ./results.jsonl
+A scorer scores *(row, output)*, so parity needs one candidate output per row
+in its development corpus. Use the customer's existing script to run its
+original `task_lm` once with the unchanged seed component map on train,
+validation, and the mandatory `parity` partition, and record the exact output
+passed to `metric(row, output)`. Write `outputs-train.jsonl`,
+`outputs-validation.jsonl`, and `outputs-parity.jsonl`. Each JSONL row is:
 
-jq -c '{row_id: .row_id, model_output: .model_response}' ./results.jsonl > ./outputs.jsonl
+```json
+{"row_id":"<the row's stable id>","model_output":"<the exact output>"}
 ```
 
-`outputs.jsonl` must cover exactly the rows being checked: a missing or extra
-`row_id`, a duplicate, or a non-string `model_output` is refused before
-anything runs.
+Do not produce this evidence by running the migrated scorer or by starting an
+optimization. The output is a fixed subject shared by the original metric and
+the migrated scorer, so a mismatch isolates the scorer migration rather than
+mixing in candidate-runner or provider drift. Keep final-held-out outside this
+evidence. After scorer parity is proven, characterize the candidate runner
+separately against the same parity rows before starting GEPA.
+
+Each parity output file must cover exactly the rows being checked: a
+missing or extra `row_id`, a duplicate, or a non-string `model_output` is
+refused before anything runs.
 
 ## 6. Prove parity
 
@@ -211,7 +251,15 @@ orizu --local scorers verify-parity \
   --scorer-version <scorer-version-id> \
   --dataset-version <dataset-version-id> \
   --split-set <split-set-id> --split validation \
-  --outputs ./outputs.jsonl \
+  --outputs ./outputs-validation.jsonl \
+  --original metric:metric \
+  --json
+
+orizu --local scorers verify-parity \
+  --scorer-version <scorer-version-id> \
+  --dataset-version <dataset-version-id> \
+  --split-set <split-set-id> --split parity \
+  --outputs ./outputs-parity.jsonl \
   --original metric:metric \
   --json
 ```
@@ -237,9 +285,9 @@ orizu --local scorers verify-parity \
   the registered runner version; drifted bytes exit 2.
 - `--limit <n>` checks the first N rows of the partition — a smoke check, not
   the proof: it always reports `parity: false` with `scope: {compared, total}`,
-  so a limited run can never be mistaken for a full one. The same full
-  `outputs.jsonl` works: outputs must cover the limited rows and belong to the
-  split, and need not be trimmed to N.
+  so a limited run can never be mistaken for a full one. The full partition's
+  output file works: outputs must cover the limited rows, belong to the split,
+  and need not be trimmed to N.
 - A repeated option takes its LAST value, matching run-gepa's scalar options.
 - If a sampled row already has the candidate field as a key, the command exits 2
   rather than overwriting a value the original metric reads.
@@ -252,12 +300,17 @@ rows, unverified runner dir, original metric not importable).
 Do not continue on exit 1 or 2. A scorer that disagrees with their metric
 optimizes the wrong thing, and every number afterwards is wrong.
 
-**Prove parity on BOTH partitions before declaring the scorer migrated.**
-`run-gepa` scores `train` and `validation`, and training rows drive GEPA's
-proposals: a scorer that agrees on validation but diverges on a training row
-that exercises a different schema branch changes the experiment's result while
-passing this gate. Repeat steps 5 and 6 with `--split train` (its own
-`outputs.jsonl` from the same seed run) and require exit 0 on both.
+**Prove parity on train, validation, and parity before declaring the scorer
+migrated.** `run-gepa` scores train and validation, and training rows drive
+GEPA's proposals: a scorer that agrees on validation but diverges on a
+different schema branch changes the experiment's result while passing this
+gate. Repeat the command with `--split train` and `outputs-train.jsonl`; the
+explicit `parity` command above covers the independently selected corpus.
+Require exit 0 on all three parity runs.
+
+Keep final-held-out untouched until J6. It is exercised exactly once, at the
+final seed-vs-selected comparison, by the already-parity-proven scorer and
+supplies no scorer-development feedback.
 
 A `parity: true` report means every row of that partition was compared and
 agreed. A limited run reports `parity: false` with
@@ -268,12 +321,14 @@ agreed. A limited run reports `parity: false` with
 
 ```bash
 orizu --local optimizers push ./optimizer \
-  --project <team>/<project> --name gepa-optimizer --label gepa-v1 --json
+  --project <team>/<project> --name gepa-optimizer --json
 
 orizu --local optimizations run-gepa \
   --project <team>/<project> \
   --optimizer-version-id <optimizer-version-id> \
-  --candidate-version-id <seed-prompt-version-id> \
+  --instruction-set <instruction-set-slug> \
+  --model-config <provider/model> \
+  --component-selector round-robin \
   --runner-version-id <candidate-runner-version-id> \
   --candidate-runner-dir ./candidate-runner \
   --scorer-version-id <scorer-version-id> \
@@ -294,10 +349,23 @@ reflection model is refused at launch without it
 `packages/orizu-gepa/src/orizu_gepa_connector/runtime.py:193-196`). Drop it only
 if you also pass an `openai/...` `--reflection-model`.
 
+`--instruction-set` and `--model-config` select exactly one profile. The CLI
+uses that profile's production version when it has one and otherwise uses the
+set default; it refuses the legacy `--candidate-version-id` selector when the
+instruction-set selectors are present. `round-robin` updates one component per
+round; use `--component-selector all` only when every component should be
+reflected on each round.
+
 `./optimizer` needs `manifest.json` with `"optimizer_family": "gepa"`. See
 `references/optimization-with-gepa.md` for reflection-model and budget flags.
 Pass `--scorer-candidate-field <field>` here too if you passed it to
 `verify-parity`: the payload the runner sees must be the one parity was proven
-under. Afterwards, `orizu optimizations promote <run-id> --candidate <id>
---label production` promotes the winner — and with parity proven, its score is
-comparable to the seed score their own script reports.
+under. Select one candidate on validation evidence, then reserve one final-held-out seed-versus-selected-candidate comparison for the promotion decision.
+Write and attach the optimization report, then obtain the human decision. Following the [Authority map](../SKILL.md#authority-map), in the simpler one-shot path a human curator runs `orizu --local optimizations promote <run-id> --candidate <candidate-id> --label production --project <team>/<project> --json` after acceptance. In the equivalent two-stage path for separate materialization, the coding agent materializes once with the unlabeled command below.
+
+```bash
+orizu --local optimizations promote <run-id> --candidate <candidate-id> \
+  --project <team>/<project> --json
+```
+
+A human curator re-runs the same promotion as `orizu --local optimizations promote <run-id> --candidate <candidate-id> --label production --project <team>/<project> --json`. The idempotent finalizer finds the existing profile version by run and candidate provenance, moves production to it, and creates no duplicate profile version or candidate-promoted event.