orizu 0.5.21 → 0.5.23

package/vendor/skills/orizu-cli/references/cli-reference.md

@@ -33,7 +33,7 @@ Behavior:
 - `orizu logout` revokes the current PAT remotely when possible and then clears local credentials.
 - PATs can also be revoked from the Personal Tokens page in Orizu.
 - PAT authorization follows the owning user's current team/project roles, so demotion or removal takes effect without rotating the token.
-- Hosted sandboxes are pre-authenticated via `ORIZU_TOKEN_FILE` (no `orizu login` needed); an explicit `ORIZU_TOKEN` env var overrides both the token file and stored credentials when set.
+- A hosted session is pre-authenticated via `ORIZU_TOKEN_FILE` (no `orizu login` needed); an explicit `ORIZU_TOKEN` env var overrides both the token file and stored credentials when set.
 
 ### Agent Setup
 
@@ -67,7 +67,8 @@ orizu teams members role --team my-team --email person@example.com --role admin
 
 Allowed roles:
 - `admin`
-- `member`
+- `curator`
+- `judge`
 
 ### Projects
 
@@ -79,56 +80,77 @@ orizu projects create --name "Quality Eval" --team my-team
 
 ### Instruction sets
 
+Follow the [Authority map](../SKILL.md#authority-map) for the executor on the current surface.
+
 ```bash
-orizu instruction-sets list --project my-team/quality-eval [--status active|archived|all] [--json]
-orizu instruction-sets show <set> --project my-team/quality-eval [--status active|archived|all] [--json]
-orizu instruction-sets create ./orizu.instruction-set.json --project my-team/quality-eval [--runner-version <id>] [--model-config <identity>] [--json]
-orizu instruction-sets push ./orizu.instruction-set.json --project my-team/quality-eval [--runner-version <id>] [--json]
-orizu instruction-sets sync <set> --out ./instructions --project my-team/quality-eval [--model-config <identity>] [--json]
-orizu instruction-sets profiles new <set> --project my-team/quality-eval --model-config <identity> [--json]
-orizu instruction-sets profiles promote <set> --project my-team/quality-eval --model-config <identity> --version <n> [--json]
-orizu instruction-sets profiles rollback <set> --project my-team/quality-eval --model-config <identity> --to <n> [--json]
-orizu instruction-sets default show <set> --project my-team/quality-eval [--json]
-orizu instruction-sets default move <set> --project my-team/quality-eval --model-config <identity> --version <n> [--json]
-orizu instruction-sets shape add <set> --project my-team/quality-eval --key <key> --from <manifest> [--json]
-orizu instruction-sets shape remove <set> --project my-team/quality-eval --key <key> [--json]
+orizu instructions list --project my-team/quality-eval [--status active|archived|all] [--json]
+orizu instructions show <set> --project my-team/quality-eval [--status active|archived|all] [--json]
+orizu instructions create ./orizu.instruction-set.json --project my-team/quality-eval [--runner-version <id>] [--model-config <identity>] [--json]
+orizu instructions push ./orizu.instruction-set.json --project my-team/quality-eval [--set <slug-or-exact-name>] [--runner-version <id>] [--json]
+orizu instructions sync <set> --out ./instructions --project my-team/quality-eval [--model-config <identity>] [--json]
+orizu instructions profiles new <set> --project my-team/quality-eval --model-config <identity> [--json]
+orizu instructions profiles promote <set> --project my-team/quality-eval --model-config <identity> --version <n> [--json]
+orizu instructions profiles rollback <set> --project my-team/quality-eval --model-config <identity> --to <n> [--json]
+orizu instructions default show <set> --project my-team/quality-eval [--json]
+orizu instructions default move <set> --project my-team/quality-eval --model-config <identity> --version <n> [--json]
+orizu instructions shape add <set> --project my-team/quality-eval --key <key> --from <manifest> [--json]
+orizu instructions shape remove <set> --project my-team/quality-eval --key <key> [--json]
+orizu instructions archive <slug-or-exact-name> --project my-team/quality-eval [--json]
+orizu instructions restore <slug-or-exact-name> --project my-team/quality-eval [--json]
 ```
 
-`list` returns each instruction set and its ordered shape. `show` returns the
-default and every model-config profile's production state. `create` and `push`
-read a local manifest; JSON output is one document per line.
+`instructions` is the customer-facing command namespace. The legacy
+`instruction-sets` spelling remains a compatibility alias, but new workflows
+should not teach it. A set can be addressed by its stable slug or exact name;
+the slug does not change when the display name changes. `list` returns each
+instruction set and its ordered shape. `show` returns the default and every
+model-config profile's production state. Use `--status archived` or `--status
+all` to include archived sets. The [Authority map](../SKILL.md#authority-map) selects the
+executor for each surface. Archive and restore change visibility only:
+archived sets continue to resolve and sync. `create` and `push` read a local
+manifest; JSON output is one document per line. `push --set` updates the named
+or slug-addressed set independently of the manifest's display name.
 
 `profiles new` copies the default component map into a model-config profile without
-changing its production resolution; hosted agents may create this seed because
-it does not move production. `profiles promote` moves that profile to a
+changing its production resolution. `profiles promote` moves that profile to a
 version, and `profiles rollback` creates a new version copied from `--to`,
 including that target version's settings version, before moving production.
-These label-moving commands are human-only. For a single-component set that
-wraps a prompt, only default-profile promote and rollback also move the wrapped
+`SKILL.md` supplies the promotion-decision and execution hand-off. For a
+single-component set that wraps a prompt, only default-profile promote and rollback also move the wrapped
 prompt's production label in the same transaction and print `default profile →
 prompt label`. This coupling is one-way: moving the prompt label directly does
 not repoint the profile.
 
 `default show` reports the default and the model configs that currently resolve
-to it. `default move` is human-only and first reads that impact before moving
+to it. `default move` first reads that impact before moving
 the pointer to a sealed, commit-anchored profile version. `shape add` and
 `shape remove` create a new complete version for every profile; they never move
 the default or a production label. The set does not resolve for affected model
 configs until those pointers move to the new shape-change profile versions; the
 text CLI prints the required follow-up commands.
 
-The manifest is a JSON object with `name`, ordered `shape`, and `components`.
-Every component supplies `{ key, text }` or `{ key, path }`; paths are relative
-to the manifest. Set-wide components must cover the complete shape. A component
-may add `modelConfig` for a profile override: `create` materializes that named
-profile with the set-wide component map plus its overrides, while `push` updates named
+The manifest is a JSON object with `name`, optional unversioned `description`,
+ordered `shape`, and `components`. The description belongs to the instruction
+set rather than any profile and round-trips through sync. Every component
+supplies `{ key, text }` or `{ key, path }`; paths are relative to the manifest.
+Set-wide components must cover the complete shape. A component may add
+`modelConfig` for a profile override: `create` materializes that named profile
+with the set-wide component map plus its overrides, while `push` updates named
 profiles already present on the set. `push` refuses a manifest whose shape
 differs from the stored set.
 
-`sync` writes `<out>/<set>/manifest.json`, `default/<key>.md`, and
+`sync` writes `<out>/<stable-set-slug>/manifest.json`, `default/<key>.md`, and
 `profiles/<identity-slug>/<key>.md` for offline execution. TypeScript runners
-use `loadInstructionSet(dir, name, modelConfigIdentity)` from `orizu`; Python
-runners use `orizu_gepa_connector.instruction_set_loader.load_instruction_set`.
+use `loadInstructionSet(dir, setReference, modelConfigIdentity)` from `orizu`;
+Python runners use
+`orizu_gepa_connector.instruction_set_loader.load_instruction_set`. Prefer the
+stable slug as `setReference`; both loaders also resolve the manifest's exact
+name for compatibility with directories synced before slugs shipped. The
+manifest's immutable `projectId` and `instructionSetId` prevent a shared
+`--out` root from replacing another project's same-slug set.
+Identity-less pre-slug trees cannot prove project ownership and are preserved;
+sync returns `instruction_set_sync_legacy_identity_required` until the operator
+moves the old tree aside or selects a clean output root.
 Both select that model config's production component map or the default without a
 network call. Git-pinned components have no local bytes: loading one raises
 `instruction_set_component_unavailable` until the runner supplies those bytes.
@@ -347,9 +369,12 @@ Defaults:
 - format defaults to `jsonl`
 - output file defaults to `<taskId>.<format>`
 
-### Prompt Control Plane
+### Instruction Control Plane
 
-For prompts, judges, runners, run submission, optimizer artifacts, live event logging, and accepted-candidate promotion, read `prompt-control-plane.md`. For markdown reports attached to optimization runs, read `optimization-reports.md`.
+For instruction sets, judges, runners, run submission, optimizer artifacts,
+live event logging, and accepted-candidate promotion, read
+`prompt-control-plane.md`. For markdown reports attached to optimization runs,
+read `optimization-reports.md`.
 
 Diff-comment export:
 
@@ -374,23 +399,28 @@ Unrecoverable context is retained with a named degradation reason rather than
 dropping a comment. Optimization export includes the same data at
 `diffComments` using hunk detail. It also includes
 `diffCommentsSuppressedReason`: normally `null`, or `agent_project_scope` when
-a hosted agent exports a run outside its assigned project. In that case
+a hosted session exports a run outside its assigned project. In that case
 `diffComments` is empty, and the marker reveals nothing about whether comments
 exist.
 
 Optimization trace commands:
 
 ```bash
-orizu optimizations run-gepa ... [--scorer-input-contract gepa|flat_row] [--scorer-candidate-field <row-field>] [--allow-degenerate-seed] [--candidate-selection-strategy pareto|current_best|epsilon_greedy] [--epsilon N] [--objective <text>] [--candidate-proposer skilled-proposer] [--proposal-max-calls N] [--proposal-max-tokens N] [--python <command>] [--num-threads auto|N] [--reflection-retry-attempts 3] [--reflection-http-timeout-seconds 180] [--log-dir logs] [--no-local-log]
+orizu optimizations run-gepa ... [--scorer-input-contract gepa|flat_row] [--scorer-candidate-field <row-field>] [--allow-degenerate-seed] [--candidate-selection-strategy pareto|current_best|epsilon_greedy] [--epsilon N] [--objective <text>] [--candidate-proposer skilled-proposer] [--candidate-proposer-config @file] [--proposal-max-calls N] [--proposal-max-tokens N] [--python <command>] [--num-threads auto|N] [--reflection-retry-attempts 3] [--reflection-http-timeout-seconds 180] [--log-dir logs] [--no-local-log]
+orizu optimizations run-gepa --hosted ... --budget auto|light|medium|heavy [--launch-intent-id <uuid>]
 orizu optimizations export <optimization-run-id> --out ./optimization.json
 orizu optimizations export <optimization-run-id> --json
 ```
 
 Behavior:
+- Hosted optimization is a human/PAT-only handoff for enabled teams. It makes
+  one server launch request, does not require local runner/log directories,
+  refuses numeric budget controls, and queues the run for the hosted
+  coordinator. Reuse `--launch-intent-id` after response loss.
 - `run-gepa` sends the scorer runner a GEPA-shaped row (`{source_row, candidate_output, …}`) by default. Judge runners written for flat-row score runs (`runners exec --scorer-version`) need `--scorer-input-contract flat_row` (plus `--scorer-candidate-field <row-field>` when the judge reads the candidate output from a named row field) or they silently score everything 0. `run-gepa` validates the contract on the seed at launch and refuses a uniformly-worst seed unless `--allow-degenerate-seed` is set. See `prompt-control-plane.md` ("Scorer-Runner Input Contracts").
 - `run-gepa` defaults `--num-threads` to `auto`, resolving a row-evaluation parallelism cap from mini-batch size, validation-set size, 2x CPU count, memory estimate, file-descriptor limit, and a 64-thread default ceiling. Set `ORIZU_GEPA_AUTO_THREADS_MAX` or use `--num-threads <n>` only when the runner/provider capacity is known.
 - `run-gepa` retries transient reflection-provider failures by default. If retries are exhausted, it logs `reflection_failed`, counts that reflection proposal against budget, and continues with the next iteration.
-- The skilled proposer is official-engine-only and prepares its managed Python environment on the selected run. Flag validation, proposal budgets, and recovery: `prompt-control-plane.md` ("Skilled proposer").
+- The skilled proposer is official-engine-only and prepares its managed Python environment on the selected run. `--candidate-proposer-config @file` adds explicit skills/guidance and requires that selection; the selection itself is incompatible with `--reflection-prompt-template`. Schema, proposal budgets, and recovery: `prompt-control-plane.md` ("Skilled proposer").
 - `run-gepa` writes a complete local trace under `logs/<optimization_run_id>` by default.
 - The local trace is the best artifact for coding-agent analysis because it includes full rows, outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, candidate text, and `result.json`.
 - `optimizations export` writes a portable JSON artifact from server data when the local log is unavailable or the run happened elsewhere.
@@ -455,8 +485,8 @@ Use these shortcuts only in TTY environments where prompts can run.
 
 - `tasks create` creates a draft by default, does not require `--assignees`, and pins the app's current version when the task is created.
 - `tasks create --publish --assignees <...>` intentionally creates and ships immediately.
-- `tasks create|publish|assign --assignment-file <path>` is mutually exclusive with `--assignees` and accepts emails or user IDs in the JSONL manifest.
-- `tasks assign --assignees` and `tasks publish --assignees` expect user IDs, not emails.
+- `--assignees` and assignment-file rows accept member emails or user IDs. Email selectors resolve to canonical member IDs within the task's project team; ambiguous or unknown emails are rejected.
+- `tasks create|publish|assign --assignment-file <path>` is mutually exclusive with `--assignees`.
 - Assignment queue reads are assignee-self-only; use task status/export as the operator summary path.
 - Assignment completion payloads are validated against the pinned app-version `output_json_schema`.
 - `datasets delete-rows` requires `--row-ids`.
@@ -465,7 +495,7 @@ Use these shortcuts only in TTY environments where prompts can run.
 - `--row-ids` is the canonical row selection for delete operations.
 - Locked datasets reject append/edit/delete row mutations.
 - Row deletes are rejected when targeted rows are assignment-referenced.
-- Login currently requires callback availability on `127.0.0.1:43123`.
+- Login callback binding defaults to `127.0.0.1:43123`. If that port is unavailable, set `ORIZU_AUTH_PORT` to an available port from 1024–65535.
 - New CLI logins use personal access tokens rather than short-lived Supabase session credentials.
 - In non-interactive contexts, pass explicit selection flags.
 
@@ -481,7 +511,7 @@ Worker assignment reads are self-only:
 - Use `tasks status` and `tasks export` for operator-side reporting.
 ### Offline instruction-set sync
 
-`orizu instruction-sets sync <set> --project <team/project> --out <dir>` writes
+`orizu instructions sync <set> --project <team/project> --out <dir>` writes
 an atomic local manifest. A resolver failure for either the default or any
 selected profile refuses the entire sync as `instruction_set_unresolvable`; the
 route never silently downgrades a failed production component map to default bytes. A

package/vendor/skills/orizu-cli/references/dataset-design.md

@@ -0,0 +1,100 @@
+# Dataset design
+
+Use this reference for **J3 — Build the dataset** after the improvement plan is ratified and before annotation or judge work. Treat the plan's approved source choice and named scenario classes as input constraints rather than reopening them while moving data.
+
+## Carry the approved source choice into the dataset
+
+Inventory each approved source and record its owner, row shape, time range, scenario-class coverage, ground-truth field, and whether the human spot-check approved it. When existing golden data is available from an observability provider, eval store, or user behavior, import those rows and preserve their approved ground truth. Do not propose a new annotation round for ground truth the customer already has; record why annotation is skipped or reduced.
+
+If there is no approved golden data, continue with representative production traces, a random sample, and structured synthetic cases for scenario classes that real traces do not cover. Keep provenance on every row so later versions can be audited, and have the human ratify the source choice and spot-check examples before the dataset is used.
+
+## Build a diverse sample and make coverage visible
+
+Create a coverage table from the ratified plan before sampling. Give every scenario class a planned row count, source, ground-truth status, and actual row count. Preserve the production distribution with a random sample, then deliberately add rare, costly, new, and previously failing cases. Keep successful cases as controls. Deduplicate repeated traces and keep stable row IDs; do not let one conversation or near-duplicate appear on both sides of the split.
+
+Before upload, fail the coverage check if any scenario class from the plan has zero rows. Report both the natural distribution and any deliberate over-sampling so later results are interpreted per scenario class rather than as an unexplained aggregate.
+
+## Design train, validation, final-held-out, and the mini-batch together
+
+Build one predefined split with exactly three application partitions for the primary dataset at dataset-build time: train, validation, and final-held-out. Auxiliary partitions, such as a migration-only parity corpus, may coexist in specialized workflows without changing this three-partition application doctrine. Validation is the valset that ranks candidates during optimization. Apply the Vocabulary contract to final-held-out: exclude its rows from candidate ranking, selection, reflection, and judge development.
+
+**Judge-development data (J5)** comes from the labeled export, and `references/building-judges.md` owns its split mechanics; exclude final-held-out rows from all judge-development material (judge-dev and judge-test/alignment) because judge selection on final-held-out labels biases the final comparison through the judge itself.
+
+Final-held-out rows need no human labels; exclude their row IDs from annotation tasks.
+
+Put representative, approved rows from every scenario class in both validation and final-held-out, isolate both from training and from each other, and keep their intended membership unchanged. Record all three partitions' row IDs in the predefined split file so the decisions can be reproduced. Every scenario class in validation or final-held-out must also have training rows; otherwise the optimizer is being assessed on a class it had no opportunity to learn.
+
+Size the mini-batch from the least common training class. For `K` scenario classes and a desired 95% chance that all classes appear, use the conservative miss bound `0.05 / K`. For each scenario class `i`, let `p_i` be that class's share of the training-split rows; choose the smallest positive integer mini-batch size `b` where `(1 - p_i)^b <= 0.05 / K` for every `i`. Pass the computed value to `orizu optimizations run-gepa --minibatch-size <b>`; `--minibatch-size` defaults to `3`, so do not omit it when the computed value differs.
+
+Budget this together with the valset. With the skilled/custom proposer, the engine can silently clamp the reflection mini-batch to `max(1, max_metric_calls - len(valset))`. Do not infer safety from the user-facing budget flag: explicit `--max-metric-calls`, `--max-full-evals`, named `--budget` presets, and the default `--budget auto` all resolve to a metric-call limit that can trigger this clamp. With the default proposer, this clamp does not apply.
+
+No budget dry-run exists, so calculate the resolved metric-call limit before launching a skilled/custom proposer. Explicit `--max-metric-calls n` resolves to `n`; `--max-full-evals f` resolves to `f * (len(trainset) + len(valset))`; and named/default `--budget` presets resolve through the formula in `packages/orizu-gepa-python/src/orizu_gepa/optimizer.py:361-405` using the chosen preset, component count, and valset size. Require `resolved_metric_calls - len(valset) >= b`; otherwise increase or change the budget, or record that the coverage guarantee does not hold. If the available rows or optimization budget cannot support the computed size, add rows, combine only genuinely equivalent scenario classes with human approval, or record that the plan is not ready for optimization.
+
+The canonical split-file schema and version/split command contract live in `references/prompt-control-plane.md` under “Dataset Versions And Splits.” J3 adds the coverage requirement: use a predefined three-way split when scenario-class coverage matters, and build its partitions from the independently reviewed coverage table rather than a random split that could omit a class. For example:
+
+```json
+{
+  "name": "v1",
+  "strategy": "predefined",
+  "seed": 42,
+  "partitions": [
+    { "name": "train", "row_ids": ["policy-1", "escalation-1", "safety-1"] },
+    { "name": "validation", "row_ids": ["policy-2", "escalation-2", "safety-2"] },
+    { "name": "final-held-out", "row_ids": ["policy-3", "escalation-3", "safety-3"] }
+  ]
+}
+```
+
+For the Final-held-out seed-vs-selected procedure and its supported-surface stop conditions, follow step 5 under “How To Build The Report” in `references/optimization-reports.md`.
+
+## Upload, version, and create the split set
+
+Normalize approved rows to CSV, JSON, or JSONL, with stable row identity and provenance fields, then upload them and request the machine-readable result:
+
+```bash
+orizu datasets upload --file ./dataset.jsonl --project <team/project> --name "<dataset-name>" --readme-file ./dataset-readme.md --json
+```
+
+Upload already creates the first immutable snapshot (labelled `v1` by default). Retain the returned `dataset_id` and `dataset_version_id`; use that `dataset_version_id` directly to create the split set, and retain its returned `split_set_id` in the improvement plan:
+
+```bash
+orizu datasets splits create <upload-returned-dataset-version-id> --from-file ./split.json --json
+```
+
+Reserve `orizu datasets versions create` for a subsequent snapshot, and give it a new label such as `v2`; labels are unique within a dataset. The dataset README should record source approval, source locators, sampling decisions, deduplication rules, the coverage table, ground-truth status, and the split rationale.
+
+The dataset version freezes its rows. Server-side split membership is mutable, but the CLI exposes no split-set read operation, so membership cannot be re-read through the CLI. Treat the independently reviewed local split file plus the returned split-set ID as the canonical record available to this workflow. Do not mutate that server-side split set; pass the recorded `--dataset-version-id` and `--split-set-id` to `orizu optimizations run-gepa`, and record the CLI verification gap explicitly rather than claiming the server membership was re-verified.
+
+## Guard live rows before annotation and version later data
+
+Use the dataset mutation contract in `SKILL.md` under “Execution facts” rather than inferring lock or row-edit semantics here.
+
+After source approval and the split audit, the dataset must be locked before annotation starts because tasks pin live dataset rows rather than the immutable version:
+
+```bash
+orizu datasets lock --dataset <dataset-id> --project <team/project> --reason "Approved dataset v1"
+```
+
+The lock is a hard gate. If the user refuses it, annotation cannot proceed safely: labelers and exports dereference live row data, while responses store no response-time row snapshot. A mutate-label-restore sequence could therefore attach a label to different content without leaving evidence in the final rows.
+
+When new data arrives, keep the prior dataset version, reviewed split file, and recorded split-set ID unchanged. Clone the locked dataset, add the approved rows to the unlocked clone, repeat the scenario-class coverage and split audit, and create a subsequent snapshot with a non-default label plus a new split set:
+
+```bash
+orizu datasets clone --dataset <dataset-id> --project <team/project> --name "<dataset-name>-v2"
+orizu datasets append --dataset <cloned-dataset-id> --project <team/project> --file ./new-rows.jsonl
+orizu datasets versions create <cloned-dataset-id> --project <team/project> --label v2 --readme-file ./dataset-readme-v2.md --json
+orizu datasets splits create <new-dataset-version-id> --from-file ./split-v2.json --json
+```
+
+Never rewrite a prior version to make later labels or corrections appear historical.
+
+## Exit criterion
+
+J3 exits only when the evidence records all of the following:
+
+- the human-ratified source choice and spot-check;
+- the plan's complete scenario-class list, with nonzero total, training, validation, and final-held-out counts for every scenario class;
+- an immutable dataset version ID plus the reviewed local split file and recorded split-set ID whose predefined train, validation, and final-held-out rows match that coverage table;
+- final-held-out row IDs excluded from all judge-development material (judge-dev and judge-test/alignment) and reserved for the final J6 comparison;
+- for the annotation path, proof that the dataset was locked before any task was created; and
+- either approved golden ground truth with annotation skipped or reduced, or rows ready for the eval-strategy and annotation path.

package/vendor/skills/orizu-cli/references/eval-strategy.md

@@ -0,0 +1,64 @@
+# Eval strategy and labeler validation
+
+Use this reference only after **J3 — Build the dataset** has versioned the PRIMARY dataset and created its three-way split set. If approved golden data already supplies all required ground truth, record the source, approval, and scenario-class coverage, then skip annotation. If only unresolved rows need labels, clone or filter those rows into a separate dataset because automatic assignments cover every current dataset row and explicit assignment files require uniform whole-dataset coverage.
+
+Final-held-out rows need no human labels; exclude their row IDs from annotation tasks and from the unresolved-rows clone/filter.
+
+A derived unresolved-rows dataset is a labeling vehicle, not an optimization dataset. Preserve each source row's canonical `id` field from `datasets download` in the derived row's `row_data`; this is the stable identifier used by split files. Before creating any task, create and record the derived dataset's immutable version ID and lock the live dataset; it does not need its own per-class three-way split. Task creation waits for both records.
+
+## Guide the eval-strategy conversation
+
+Start from the ratified improvement plan and J3 coverage table; do not ask the human to restate facts already recorded. Walk through the decisions below with concrete rows, propose a draft answer where the evidence supports one, and ask the human to correct or ratify it.
+
+1. **Experience and decision.** Ask which exact user experience the eval represents and which change or release decision it must inform. Confirm the unit being judged: one response, conversation, agent step, or complete run.
+2. **Good and bad by scenario class.** For every scenario class, show representative rows and draft one binary question per observed failure mode. Ask what visible evidence makes each answer pass, fail, or unlabelable. Do not bundle independent judgments or replace them with a Likert question.
+3. **Eval-set composition.** Confirm which versioned rows need human labels, the intended counts per scenario class, and any rare cases that must be present even when production frequency is low. Keep J3 validation for candidate ranking and keep its final-held-out partition under the separate partition doctrine in `references/dataset-design.md`.
+4. **Ground truth.** Ask who is qualified to label each question, what context they need, how ambiguity or disagreement will be adjudicated, and whether user behavior or another approved golden-data field already answers it. Import approved golden data instead of relabeling it.
+5. **Downstream validation.** Explain that these labels will later validate automated judges. Ask which decision class each judge will power and record it for **J5 — Build & validate judges**, where the human sets the numerical judge trust bar from measured agreement, TPR, and TNR rather than accepting a default here.
+
+Persist the ratified eval strategy in project context. It must name the experience and unit, dataset version and split set, scenario classes and row counts, binary questions with pass/fail/unlabelable rules, ground-truth source, labeler qualifications, disagreement path, golden-data decision, and the future decision attached to each judge trust bar. Do not author the labeler until the human ratifies this record.
+
+`references/building-judges.md` owns the J5 input envelope and is being aligned there. This reference records the evidence handed to that contract. For a golden-data-only path, validate the field mapping against the binary questions, spot-check every scenario class with the human, and record the PRIMARY versioned dataset ID plus the approved ground-truth field per failure mode in the plan and report; then skip the remaining J4 annotation path.
+
+For a mixed path, after annotation export, group derived records by the canonical `id` at `dataset_row.row_data.id`, the same identifier emitted by `datasets download` and used by split files, and require each key to match one primary row. Preserve every per-rater record in each row group, including the rater identity from the export, then aggregate according to the eval strategy's agreement rules. Record J5's labeled input as the union of the golden field mapping for golden rows and the grouped task export for annotated rows, keyed by that canonical primary-row `id`. Continue below only for questions or rows whose ground truth still requires human labels.
+
+## Validate the labeler before tasks or labels
+
+After the eval strategy is ratified, author the task-specific app with `references/building-apps.md`. That reference is the single home for the runtime contract, component choices, schemas, smoke test, preview mechanics, and app accessibility rules.
+
+Run the smoke test and preview flow from that reference with representative rows from every scenario class. Inspect the resulting screenshot yourself, then ask the human for pointed feedback: does the app show the evidence needed for each judgment, ask the intended binary question in the plan's language, distinguish unanswered from false, produce useful ground truth, and make the primary action clear? Ask what is confusing or missing and which representative row is hardest to label. Revise and preview again until the human explicitly confirms the app works.
+
+Record the rows previewed, the feedback, the revisions, and the approval. Do not create or publish a task, and do not start labeling, until this labeler-feedback gate is confirmed.
+
+Use the draft-first ordering, URL-approval gate, and assignee-selector contract in `SKILL.md` under “Execution facts”; use `references/cli-reference.md` for the command surface. This reference adds one versioning safeguard: the task pins the app version selected at creation.
+
+If draft testing reveals a problem, revise through `orizu apps update`, repeat the preview and pointed-feedback gate, capture the returned version number `n`, and explicitly repin the draft before testing its URL again:
+
+```bash
+orizu tasks update --task <taskId> --app <appId> --version <n>
+```
+
+Then return to the approval and publish sequence in “Execution facts.” If the strategy requires an explicit row-to-labeler map, it must still cover every row in this task's separate annotation dataset uniformly.
+
+After labeling, inspect the machine-readable status before task completion or export:
+
+```bash
+orizu tasks status --task <task-id> --json
+```
+
+Verify that `task.counts.completed` equals `task.totalRequiredAssignments` and that in-progress, pending, paused, and skipped counts are zero. Status proves assignment states, not response custody. After those counts pass, export the response-bearing JSONL before completing the task:
+
+```bash
+orizu tasks export --task <task-id> --format jsonl --out ./labels.jsonl
+```
+
+Parse every non-empty JSONL line. Require the export row count to equal `task.totalRequiredAssignments`, require each `dataset_row_id` to appear exactly `task.requiredAssignmentsPerRow` times, and require every record to carry a non-null `response_id` and `response_data`. Any mismatch means a planned row lacks a response and blocks completion. Only after this response-level proof may the task be completed and its report published:
+
+```bash
+orizu tasks complete --task <task-id>
+orizu tasks report set --task <task-id> --report-file ./task-report.md --json
+```
+
+## Exit criterion
+
+The annotation path exits only when the ratified eval strategy, labeler feedback, app approval, complete assignment-status evidence, response-complete exported labels, and published task report are recorded together. The golden-data path exits with the PRIMARY versioned dataset ID and approved per-failure-mode field mapping recorded. The mixed path additionally records the canonical-row-`id`-keyed union of golden mappings and per-rater annotated exports. In every path, report ground-truth and disagreement counts per scenario class so J5 receives auditable evidence rather than only an aggregate.

package/vendor/skills/orizu-cli/references/instructions-after-prompts.md

@@ -0,0 +1,115 @@
+# Instructions After Prompts
+
+Use instruction sets for customer-owned model instructions. The older prompt
+read surface remains available for locating legacy artifacts, but standalone
+prompt content and production mutations deliberately redirect to `orizu
+instructions`.
+
+## Find Existing Material
+
+Legacy prompts appear as one-component sets in the instruction inventory:
+
+```bash
+orizu instructions list --project <team/project> --status all --json
+orizu instructions show <slug-or-exact-name> --project <team/project> --status all --json
+```
+
+Use prompt reads only when you need to correlate an old prompt identifier or
+download its files:
+
+```bash
+orizu prompts list --project <team/project> --status all
+orizu prompts pull <prompt-id-or-name> --project <team/project> --out ./legacy-prompt
+```
+
+The text output ends with the owning instruction set's stable slug and
+component key. List and pull include the prompt display name and identifier:
+
+```text
+Owner: Generator (prompt-1) -> planner-agent / system
+```
+
+Use the slug after `->` for instruction commands and the final segment for the
+component key. Address scorer bindings directly with that instruction-set slug
+and component key. An ownerless judge or draft prompt may have no owner line.
+
+## Understand Deliberate Refusals
+
+Standalone prompt content, production-pointer mutations, and prompt-addressed
+scorer bindings fail before making a request. The CLI prints these exact
+replacements:
+
+- `Use: orizu instructions push <manifest> --project <team/project>`
+- `Use: orizu instructions archive <slug-or-exact-name> --project <team/project>`
+- `Use: orizu instructions restore <slug-or-exact-name> --project <team/project>`
+- `Use: orizu instructions profiles promote <set> --project <team/project> --model-config <identity> --version <n>`
+- `Use: orizu instructions scorers set-headline <set> --key <component-key> --scorer-version <id> --project <team/project>`
+- `Use: orizu instructions scorers add <set> --key <component-key> --scorer-version <id> --project <team/project>`
+
+Older clients and direct API callers can reach two additional server-side
+prompt-write conflicts. Both fail closed with an exact JSON error:
+
+| Write path | Conflict | Measured response | Next action |
+| --- | --- | --- | --- |
+| Sessionless prompt registration from an older client or direct API call | The name has set-owned storage and no ownerless lineage. | 409 — `Use: orizu instructions push; prompt name belongs to an instruction set` | Use the owning set's manifest through `orizu instructions`; the current CLI redirects before HTTP. |
+| Session-scoped prompt draft (`prompts push ... --session ...`) | The name belongs to components in more than one instruction set and has no ownerless lineage. | 409 — `session draft prompt name is ambiguous across instruction sets` | Do not guess by name. Inventory the sets and component keys, then prepare the intended instruction-set manifest. |
+
+These client redirects and server conflicts do not apply to prompt reads, judge
+artifacts, prompt reports, instruction-addressed scorer bindings, unambiguous
+session-scoped drafts, or non-production prompt labels.
+
+## Agent preparation
+
+From a hosted session, inventory every active and archived set, choose a stable
+target name, component keys, model-config profiles, and complete component
+text, then prepare a manifest such as:
+
+```json
+{
+  "name": "planner-agent",
+  "description": "Instructions used by the planner agent.",
+  "shape": ["system", "tools"],
+  "components": [
+    { "key": "system", "path": "./system.md" },
+    { "key": "tools", "path": "./tools.md" }
+  ]
+}
+```
+
+Commit the complete manifest and its component files, then follow the
+[Authority map](../SKILL.md#authority-map) for each mutation.
+
+## Consolidation
+
+Consolidation changes ownership and visibility. Create the consolidated set
+through the surface selected by the [Authority map](../SKILL.md#authority-map) and inspect its complete
+profiles before changing any pointer:
+
+```bash
+orizu instructions create ./orizu.instruction-set.json --project <team/project> --model-config <identity> --json
+orizu instructions show <slug-or-exact-name> --project <team/project> --status all --json
+```
+
+After verifying the new set, archive each redundant one-component
+set. Archive changes visibility only; the archived set still resolves and
+syncs until the pointer move is complete, so it remains a recovery
+source.
+
+```bash
+orizu instructions archive <slug-or-exact-name> --project <team/project> --json
+```
+
+After the explicit promotion decision, move the consolidated set's default or
+promote a named model-config profile through the Authority map:
+
+```bash
+orizu instructions default move <set> --project <team/project> --model-config <identity> --version <n> --json
+orizu instructions profiles promote <set> --project <team/project> --model-config <identity> --version <n> --json
+```
+
+Confirm the resolved default and every production profile after the move. If a
+redundant set must become visible again, restore it through the Authority map:
+
+```bash
+orizu instructions restore <slug-or-exact-name> --project <team/project> --json
+```