orizu 0.5.21 → 0.5.22

package/vendor/skills/orizu-cli/references/optimization-with-gepa.md

@@ -1,21 +1,21 @@
 # Optimization With GEPA
 
-How to optimize text candidates against validated judges/scorers. For Orizu-tracked runs, prefer `prompt-control-plane.md` and the bundled `orizu optimizations run-gepa` command when optimizing one text candidate. Use this reference for GEPA mechanics, custom optimizer implementations, and optional DSPy context for customers already using DSPy.
+How to optimize an instruction-set profile against validated judges and scorers. For Orizu-tracked runs, a coding agent prepares the complete instruction-set manifest, a human curator creates or updates the set from the local CLI, and then the agent runs the bundled `orizu optimizations run-gepa` flow. Use this reference for GEPA mechanics, custom optimizer implementations, and optional DSPy context for customers already using DSPy.
 
-For an instruction set, select the profile explicitly with `--instruction-set <name> --model-config <identity>` rather than `--prompt` or `--prompt-version` (those selectors are mutually exclusive). The connector resolves that model config's production profile, or the set default, and optimizes its complete component map. `--component-selector round-robin` is the default and updates one component per round; `--component-selector all` updates every component per round. A multi-component candidate is sent as a component map, while a set of one keeps the existing prompt-body contract. Git-pinned components and malformed component maps are refused before a run starts; multi-component auto-promotion is deliberately refused until component-map promotion is available.
+Select the profile explicitly with `--instruction-set <slug-or-exact-name> --model-config <identity>`. Those two selectors replace the legacy `--candidate-version-id` path and cannot be combined with it. The connector resolves that model config's production profile, or the set default when the profile has no production version, and optimizes its complete component map. `--component-selector round-robin` is the default and updates one component per round; `--component-selector all` updates every component per round. The runner receives a multi-component candidate as a component map, while a one-component set keeps the existing single-body runner contract. Git-pinned components and malformed component maps are refused before a run starts. Automatic promotion is refused for multi-component sets. Human/local CLI only: after validation, a coding agent writes the complete component map back to the manifest and prepares exact handoff commands; a human curator runs `orizu instructions push` and `orizu instructions profiles promote`.
 
 ## Inputs
 
 You should arrive here with:
 - One or more **validated judges** (TPR > 90%, TNR > 90% on a held-out test set).
 - A **dataset** of inputs to optimize against — usually the same exported labels, plus any harder cases you've added since.
-- A **starting prompt or program** for the LLM application you want to improve.
+- An **instruction set** whose selected profile contains the starting component values for the LLM application you want to improve.
 
 If you don't have a validated judge, stop. Optimizing against an unvalidated judge means you'll hill-climb on a noisy or biased signal — Goodhart's law in action.
 
 ## Why GEPA-Style Optimization
 
-- **GEPA** is a gradient-free prompt optimizer that uses an LLM to propose prompt edits, scores candidates against your metric, and keeps the best. It's well-suited to prompt-level optimization where you can't backprop through the model.
+- **GEPA** is a gradient-free text optimizer that uses an LLM to propose component edits, scores candidates against your metric, and keeps the best. It is well-suited to instruction optimization where you cannot backpropagate through the model.
 - In Orizu, runners execute candidates, scorers produce metrics/feedback, and optimization events make the loop inspectable and promotable.
 - DSPy is not part of Orizu's bundled optimizer. Treat DSPy examples here as an external integration pattern only.
 
@@ -40,29 +40,30 @@ If you don't have a validated judge, stop. Optimizing against an unvalidated jud
               └──────────┬──────────────┘
                          ▼
               ┌─────────────────────────┐
-              │ Optimized prompt        │
+              │ Optimized profile       │
               │ Compare before/after    │
               └─────────────────────────┘
 ```
 
 ## Orizu-tracked optimization
 
-Use the prompt control plane when you want runs, candidates, score charts, Pareto/frontier views, and promotions in Orizu:
+Use the control plane described in `prompt-control-plane.md` when you want runs, candidates, score charts, Pareto/frontier views, and promotions in Orizu:
 
-1. Push the candidate runner and prompt/judge prompt.
-2. Register a row scorer for reflection. GEPA reflection requires row-level feedback.
-3. Snapshot a dataset version and create a train/validation split set.
-4. Use `orizu optimizations run-gepa` for the common text-candidate case. It uses official GEPA by default; `--engine legacy` is a frozen safety hatch while migration parity is proven. Use `orizu optimizations start` plus event logging for a custom optimizer.
-5. Use set scorers for selection/tracked reporting when the meaningful metric is batch-level; execute builtin set scorers with `orizu scorers exec` or submit precomputed aggregates with `orizu scores submit --aggregate`.
-6. Promote only candidates that passed validation.
-7. Write and attach an optimization report from the local logs or export artifact; see `optimization-reports.md`.
+1. Human/local CLI only: a human curator creates the instruction set from its prepared manifest with `orizu instructions create`, or pushes a new profile version with `orizu instructions push`. The coding agent prepares the manifest and handoff; keep the set's shape fixed during a run.
+2. Push the candidate runner and, for an LLM evaluator, the judge rubric with `orizu judges push`.
+3. Register a row scorer for reflection. GEPA reflection requires row-level feedback.
+4. Snapshot a dataset version and create a train/validation split set.
+5. Use `orizu optimizations run-gepa` with the set slug and model-config identity. It uses official GEPA by default; `--engine legacy` is a frozen safety hatch while migration parity is proven. Use `orizu optimizations start` plus event logging for a custom optimizer.
+6. Use set scorers for selection/tracked reporting when the meaningful metric is batch-level; execute builtin set scorers with `orizu scorers exec` or submit precomputed aggregates with `orizu scores submit --aggregate`.
+7. Promote only candidates that passed validation, as complete profile versions rather than individual components.
+8. Write and attach an optimization report from the local logs or export artifact; see `optimization-reports.md`.
 
-The bundled Orizu GEPA-style optimizer supports configurable budget, minibatch size (default 3), candidate selection strategy, reflection model/template, reflection provider settings, evaluation caching, and optional auto-promotion. It redacts row snapshots and reflection text by default; only pass `--log-row-snapshots` when raw data in event logs is intentional.
+The bundled Orizu GEPA-style optimizer supports configurable budget, minibatch size (default 3), candidate selection strategy, reflection model/template, reflection provider settings, evaluation caching, and optional auto-promotion, which is human-only; coding agents must not enable it. It redacts row snapshots and reflection text by default; only pass `--log-row-snapshots` when raw data in event logs is intentional.
 
 Reflection output contract:
-- The reflective LM's final text is used verbatim as the next candidate prompt.
-- The default reflection prompt asks for only the complete updated prompt body. Do not ask the model to wrap the candidate in markdown fences or tags; real prompts often contain those characters.
-- Put provider-native reasoning controls in `--reflection-provider-settings <json|@file>`, not in the prompt body. For OpenAI reasoning models, use a shape such as `{"reasoning":{"effort":"medium","summary":"auto"}}`. For Anthropic Claude models with thinking controls, use a shape such as `{"thinking":{"type":"adaptive","display":"omitted"},"output_config":{"effort":"medium"}}`.
+- The reflective LM's final text is used verbatim as the next value of the selected component. With `round-robin`, other component values are read-only context; with `all`, each component is reflected independently and the results form one complete candidate profile.
+- The default reflection template asks for only the complete updated component value. Do not ask the model to wrap the value in markdown fences or tags; instruction components often contain those characters.
+- Put provider-native reasoning controls in `--reflection-provider-settings <json|@file>`, not in an instruction component. For OpenAI reasoning models, use a shape such as `{"reasoning":{"effort":"medium","summary":"auto"}}`. For Anthropic Claude models with thinking controls, use a shape such as `{"thinking":{"type":"adaptive","display":"omitted"},"output_config":{"effort":"medium"}}`.
 - Reflection max-token limits are explicit. `--reflection-max-tokens <n>` maps to Anthropic `max_tokens` and OpenAI `max_output_tokens`; Anthropic native Messages reflection requires it, while OpenAI may omit it when no cap is desired.
 - Reflection HTTP calls retry transient failures by default (`--reflection-retry-attempts 3`, `--reflection-http-timeout-seconds 180`). Exhausted retryable failures emit `reflection_failed`, consume one candidate-proposal budget slot, and move to the next iteration instead of killing the run.
 
@@ -74,18 +75,18 @@ Since release `cli-v0.5.20` (`orizu@0.5.20` on npm), `run-gepa` defaults to the
 
 - **Validation happens before launch.** Legacy validated flags inside the python process (an argparse exit); the official dispatcher enforces the flag-shape rules below before anything spawns and names the offending flag in the error (the budget-conflict error lists the exclusive set rather than the flags you passed; value errors — e.g. a non-integer or non-positive budget number — are still reported by the engine and name the underlying setting). The rules carried over from legacy: `--budget` / `--candidate-selection-strategy` / `--scorer-input-contract` validate their choices, boolean flags take no value, abbreviated flag spellings are rejected, and budget controls are mutually exclusive — at most one of `--budget`, `--max-metric-calls`, `--max-full-evals`, `--max-iterations`. The official-only `--max-candidate-proposals` joins that exclusion set; passing no budget control defaults to `--budget auto`.
 - **A reflection cap is required for non-OpenAI reflection models.** `--reflection-max-tokens` is required at launch whenever the reflection model does not start with `openai/` — including the default (`anthropic/claude-opus-4-7`) and bare names like `gpt-4o` — so the run refuses immediately instead of failing every reflection call mid-run.
-- **The skilled proposer is an opt-in proposal path.** Reach for `--candidate-proposer skilled-proposer` when you want to evaluate the upstream proposer's alternative reflection process and can accept its first-use network, CPython, and managed-dependency requirements. Evaluate the shipped empty-skills proposer through the controlled A/B below, and ground quality or generalization claims in that evidence. Environment setup and recovery live in `prompt-control-plane.md` ("Skilled proposer").
+- **The skilled proposer is an opt-in proposal path.** Reach for `--candidate-proposer skilled-proposer` when you want to evaluate the upstream proposer's alternative reflection process and can accept its first-use network, CPython, and managed-dependency requirements. Add `--candidate-proposer-config @file` for explicit skills/guidance; do not combine the selection with `--reflection-prompt-template`. Evaluate the shipped empty-skills proposer or a pinned config through the controlled A/B below, and ground quality or generalization claims in that evidence. Config schema, environment setup, and recovery live in `prompt-control-plane.md` ("Skilled proposer").
 - **Seed preflight: same gate, better evidence.** Both engines refuse a degenerate seed (every valid row pinned to the same bound — the worst score, or uniformly `0.0` under a lower-is-better scorer, where the perfect bound is indistinguishable from a scorer silently zeroing on a mismatched input shape) or an all-errored seed at launch, and both accept `--allow-degenerate-seed` to bypass. The bypass covers *both* refusals — including the broken-scorer one — so a bypassed run can spend budget on a scorer that never works; use it only when a weak seed is expected. New on the official engine: the refusal message carries per-row evidence (row ids, scores, scorer output and errors); `<log-dir>/preflight-refused-<uuid>/preflight.json` (not written under `--no-local-log`) instead stores redacted `{row_id, score, error_class}` entries unless `--log-row-snapshots` is passed — a sibling of the run directories, since no run record exists when a launch is refused.
 - **Metric accounting differs.** Official counts include every evaluated row; legacy excluded some cached work. Do not compare raw metric-call totals across engines (details in ADR-019).
 - **Artifacts.** Both engines write `events.jsonl`, `evaluations.jsonl`, `reflections.jsonl`, and `result.json` locally; `lm_stats.json` (reflection usage) is official-only. Dashboard rendering is identical, and run metadata records `engine: official` or `engine: legacy`.
 
-Terminal semantics are unchanged between engines: exhausting any budget control other than `--max-iterations` ends the run `paused` (`pause_reason: budget_exhausted`) with no auto-promotion, while completing the configured `--max-iterations` is a normal `succeeded` finish that may auto-promote. A paused run's validated candidates can still be promoted manually — only automatic promotion is gated on how the run ended. Falling back to `--engine legacy` after a budget-exhausted pause reproduces the same paused outcome; it will not turn the run into a `succeeded` one.
+Terminal semantics are unchanged between engines: exhausting any budget control other than `--max-iterations` ends the run `paused` (`pause_reason: budget_exhausted`) with no auto-promotion, while completing the configured `--max-iterations` is a normal `succeeded` finish that may auto-promote only when the human-only option was deliberately enabled; coding agents must not enable it. A paused run's validated candidates can still be promoted manually by a human curator — only automatic promotion is gated on how the run ended. Falling back to `--engine legacy` after a budget-exhausted pause reproduces the same paused outcome; it will not turn the run into a `succeeded` one.
 
 **Fallback:** rerun with `--engine legacy` for the previous engine's exact behavior, dropping `--max-candidate-proposals` first if you used it (it is official-only; the CLI refuses it under legacy before launch). Legacy is frozen (no new features) and will be removed at the M3 milestone (ADR-019) — if you fall back because the official engine misbehaved, capture both run ids and report the pair.
 
 ### Compare the skilled proposer with the default
 
-Run two arms with the same candidate prompt/version, dataset version and split set, candidate and scorer runners, seed, reflection model/settings, thread count, and one shared overall budget such as `--max-candidate-proposals 1`. Keep the default arm unselected; add `--candidate-proposer skilled-proposer` plus bounded `--proposal-max-calls` and `--proposal-max-tokens` only to the selected arm. The comparison is ready when both terminal states, retained candidates, scores, proposal evidence, and total run usage are captured. A fixed GEPA seed aligns optimizer sampling but does not make provider or scorer calls deterministic, so treat the pair as process and outcome evidence rather than an intrinsic quality claim.
+Run two arms with the same instruction-set profile version, dataset version and split set, candidate and scorer runners, seed, reflection model/settings, thread count, and one shared overall budget such as `--max-candidate-proposals 1`. Keep the default arm unselected; add `--candidate-proposer skilled-proposer`, the same pinned `--candidate-proposer-config @file` when testing configuration, and bounded `--proposal-max-calls` / `--proposal-max-tokens` only to the selected arm. The comparison is ready when both terminal states, retained candidates, scores, proposal evidence, and total run usage are captured. A fixed GEPA seed aligns optimizer sampling but does not make provider or scorer calls deterministic, so treat the pair as process and outcome evidence rather than an intrinsic quality claim.
 
 For the selected arm, each provider-bearing proposal call appends `proposal-observability/events.jsonl`. Success records contain `source`, `status`, `attempt`, `correlation_id`, `provider`, `model`, nullable `request_id`, `latency_ms`, `cache_state`, and `usage.{input_tokens,output_tokens,total_tokens}`. Failure records contain `source`, `status`, `attempt`, `correlation_id`, `provider`, `cache_state`, `failure_code`, and optional `usage`; the latest durable failure is also written to `proposal-failures/latest.json` with `source`, `code`, `detail`, and optional `correlation_id`.
 
@@ -93,7 +94,7 @@ With local logging, those paths sit under `<log-dir>/<run-id>/`, and terminal `p
 
 ## Step 1: Wrap your application as an Orizu runner
 
-For Orizu-tracked optimization, the candidate runner receives one dataset row and one candidate text body through the file contract. The scorer runner, by default, receives a GEPA-shaped `row` — `{source_row, candidate_id, candidate_output, candidate_raw_response, candidate_error}` — and returns a score and feedback. See `prompt-control-plane.md` ("Scorer-Runner Input Contracts") for the exact runner I/O shapes.
+For Orizu-tracked optimization, the candidate runner receives one dataset row and the candidate profile through the file contract. A one-component profile is passed as a single body for runner compatibility; a multi-component profile is passed as a complete component map. The scorer runner, by default, receives a GEPA-shaped `row` — `{source_row, candidate_id, candidate_output, candidate_raw_response, candidate_error}` — and returns a score and feedback. See `prompt-control-plane.md` ("Scorer-Runner Input Contracts") for the exact runner I/O shapes.
 
 **Contract warning:** the GEPA scorer contract differs from the flat-row score-run contract used by `orizu runners exec --scorer-version`. A judge runner written for flat-row score runs will find no output to judge in the GEPA shape and silently score every candidate 0. Do not hand-write a wrapper runner: pass `--scorer-input-contract flat_row` (plus `--scorer-candidate-field <row-field>` if the judge reads the candidate output from a named row field such as `draft`) and `run-gepa` adapts the payload for you while keeping the registered runner bytes unchanged. `run-gepa` also validates the contract on the seed at launch and refuses a uniformly-worst-scoring seed with a diagnosis instead of burning budget.
 
@@ -132,7 +133,7 @@ class SupportAgent(dspy.Module):
         )
 ```
 
-If your real application is multi-step (retrieval + generation + tool use), build a multi-Module program. GEPA can optimize each step's prompt independently.
+If your real application is multi-step (retrieval + generation + tool use), build a multi-Module program. GEPA can optimize each module's instructions independently.
 
 ## Step 2: Register scorers
 
@@ -208,7 +209,7 @@ logs/<optimization_run_id>/
   result.json
 ```
 
-Use this directory as the primary artifact for coding-agent analysis. It contains the full row inputs, model outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, candidate text, and final result. Override the root with `--log-dir <dir>`; disable persistence with `--no-local-log` only when raw rows/reflection context must not be written to disk.
+Use this directory as the preferred artifact for coding-agent analysis. It contains the full row inputs, model outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, candidate text, and final result. Override the root with `--log-dir <dir>`; disable persistence with `--no-local-log` only when raw rows/reflection context must not be written to disk.
 
 If the local log is missing or the run happened elsewhere, export the server-side archive:
 
@@ -247,8 +248,8 @@ optimized_program.save("./optimized_support_agent.json")
 
 GEPA will:
 1. Run the program on `trainset`, score with `metric`.
-2. Use `reflection_lm` to propose prompt edits based on failures.
-3. Score candidate prompts on the trainset, keep the best.
+2. Use `reflection_lm` to propose instruction edits based on failures.
+3. Score candidate programs on the trainset, keep the best.
 4. Validate on `valset` to avoid overfitting.
 
 Orizu's bundled optimizer is intentionally narrower than DSPy GEPA today: text candidates only, local runner/scorer directories, and Orizu event logging built in.
@@ -287,8 +288,8 @@ print(f"After:  {after}")
 
 ## Step 6: Ship and feed the loop
 
-If the optimized prompt holds up on held-out:
-- Replace the production prompt.
+If the optimized profile holds up on held-out:
+- Promote the validated profile version to production; never promote one component by itself.
 - Sample new production traces over the next week.
 - Upload them as a new dataset (`primer.md` Step 1).
 - Annotate failures the optimized system *now* exhibits — they'll be different from the ones the previous version had.
@@ -304,7 +305,7 @@ Each pass through the loop reveals the next layer.
 - **Hiding regressions in the average.** Track per-failure-mode metrics, not just combined.
 - **Over-budgeting GEPA.** Heavy budgets give diminishing returns and burn LM spend. Start with `auto="light"`, scale up only if needed.
 - **Ignoring temperature.** Run optimization with the same LM config (model, temperature) you use in production. Optimizing against gpt-4o at temp=0 doesn't transfer to gpt-4o-mini at temp=0.7.
-- **Recreating Orizu logging by hand for text prompts.** Use `orizu optimizations run-gepa` unless the optimizer is genuinely custom.
+- **Recreating Orizu logging by hand for instruction sets.** Use `orizu optimizations run-gepa` unless the optimizer is genuinely custom.
 
 ## Checklist
 

package/vendor/skills/orizu-cli/references/primer.md

@@ -10,7 +10,23 @@ Upload → Annotate → Judge → Optimize
    └────────── new traces ──────────┘
 ```
 
-The CLI covers the full operational loop. Human annotation still happens through apps/tasks, while judge/scorer logic and optimization execution run locally and report back through the prompt control plane. Each step depends on the output of the previous; don't skip ahead.
+The CLI covers the full operational loop. Start instruction artifacts with `orizu instructions`: one instruction set per agent or LLM experience, one profile per model config, and components owned by that profile. Human annotation still happens through apps/tasks, while judge/scorer logic and optimization execution run locally and report back through the instruction control plane. Each step depends on the output of the previous; don't skip ahead.
+
+An instruction-set manifest carries its `name`, optional `description`, fixed, ordered `shape`, and component values. The set's slug is its stable CLI reference even when its display name changes. Profiles have their own component values; components are never shared between profiles or sets. The set-wide default serves model configs without a production profile version.
+
+Start by inspecting or creating the set, then push and sync through the same surface:
+
+```bash
+orizu instructions list --project <team>/<project>
+orizu instructions show <slug-or-exact-name> --project <team>/<project>
+# Human/local CLI only: a human runs create/push; hosted agents may run profiles new and sync.
+orizu instructions create ./orizu.instruction-set.json --project <team>/<project> --model-config <identity>
+orizu instructions push ./orizu.instruction-set.json --project <team>/<project> --set <slug-or-exact-name>
+orizu instructions profiles new <slug-or-exact-name> --project <team>/<project> --model-config <identity>
+orizu instructions sync <slug-or-exact-name> --out ./instructions --project <team>/<project>
+```
+
+Do not create or mutate standalone prompts. If a legacy identifier leads to a prompt read view, use its owner information to find the instruction set and continue with `orizu instructions`.
 
 ---
 
@@ -30,7 +46,7 @@ The most common mistake teams make is jumping straight to evals before they unde
 **Decision: should you even write a formal eval for this failure?**
 - If you haven't observed it, no — do error analysis first.
 - If the fix is one-shot and obvious, no — just fix it.
-- If fixing it requires repeated iteration (prompt tweaks, retries, structural changes), yes — that's where evals shine.
+- If fixing it requires repeated iteration (component edits, retries, structural changes), yes — that's where evals shine.
 
 ---
 
@@ -154,13 +170,13 @@ There are three **types** of automated evals:
 
 ### In Orizu
 
-This work is still authored by the agent in code, but Orizu now stores the versioned evaluator artifacts and score results:
+This work is still authored by the agent in code, but Orizu stores the versioned instruction, evaluator, and score artifacts:
 
 1. Export labeled data: `orizu tasks export --task <id> --format jsonl --out labels.jsonl`
 2. For each failure mode, choose code assertion or LLM-judge.
 3. Build the judge.
 4. Validate against the labels (train/dev/test split, TPR/TNR).
-5. Push the judge prompt and runner when it is an LLM judge, register a scorer, then submit score runs for prompt versions or optimization candidates.
+5. Keep the subject application's instruction material in its instruction set. For an LLM judge, push the judge artifact and runner, register a scorer, then submit score runs for the set's profile versions or optimization candidates.
 
 Control-plane commands: `prompt-control-plane.md`. Detailed authoring walkthrough — code assertion patterns, LLM-judge prompt scaffold, train/dev/test split, TPR/TNR computation, saturation checks: `building-judges.md`.
 
@@ -181,7 +197,7 @@ Done locally, reported to Orizu:
 1. Package the candidate execution as a runner, or use the customer's existing local app wrapper.
 2. Register validated scorers.
 3. Run the bundled Orizu GEPA-style text optimizer, or a custom optimizer against the scorer set.
-4. Stream optimization events to Orizu, promote accepted candidates, and submit comparable scores.
+4. Stream optimization events to Orizu, promote accepted candidates into the target instruction-set profile, and submit comparable scores.
 5. Diff before/after on the eval suite. Ship if it holds; new traces feed back to step 1.
 
 Control-plane commands: `prompt-control-plane.md`. Detailed walkthrough — GEPA mechanics, optional DSPy context for customers already using it, and before/after comparison: `optimization-with-gepa.md`.