orizu 0.5.21 → 0.5.23

package/vendor/skills/orizu-cli/SKILL.md

@@ -1,208 +1,134 @@
 ---
 name: orizu-cli
-description: Use when the user wants to improve an LLM application's performance in a measurable way, or mentions Orizu by name. Triggers include improving model or agent performance, collecting human feedback on model outputs, converting feedback into evals, building or crafting evals, running prompt optimization (including one-off prompt tweaks the user is willing to validate with evals), working with Orizu datasets, tasks, apps, prompts, judges, scorers, runners, score runs, or optimization runs, hill-climbing on metrics, or building "continually learning" agents. Orizu is a platform for building evals first, then improving LLM applications by optimizing against them. The CLI handles human feedback collection, eval creation, versioned prompt and scorer artifacts, local runner execution, score submission, and prompt optimization. Do NOT use for prompt advice when the user has explicitly said they don't want to set up evals.
+description: Use when the user mentions Orizu; wants measurable improvement for a specific LLM application or agent; collects human feedback on model outputs or turns failures into evals; validates judges; optimizes instruction sets; migrates an existing GEPA setup; or establishes continuous improvement. Step aside and handle directly generic instruction-writing theory or one-off edits the user explicitly will not validate with evals.
 ---
 
 # Orizu
 
-Orizu improves LLM applications by building evals first, then optimizing against them. The end-to-end loop has four steps: **Upload → Annotate → Judge → Optimize**. The CLI covers human-eval collection plus the Phase 0 prompt control plane: prompts, judges, scorers, runners, local runner execution, score submission, packaged GEPA-style text optimization, and live optimization logging.
+Orizu improves LLM applications with an evals-first workflow, then optimizes against the resulting evidence. Use `orizu instructions` as the primary path for customer instruction sets. Follow the applicable J0–J6 path in order and satisfy each applicable exit criterion; J4 is conditional on missing approved ground truth. Read `references/primer.md` for the methodology and `references/cli-reference.md` for the current command surface instead of relying on remembered commands.
 
-For end-to-end methodology and rationale, read `references/primer.md`. The reference docs below cover specific stages in depth.
+The journey uses **datasets** for versioned rows, **apps** and **tasks** for labels, **judges**, **runners**, and **scorers** for automated evaluation, **instruction sets** for customer instruction work, and **optimization runs** for searching profile candidates.
 
-## When to step aside
+## Critical: scorer-runner input contract
 
-Skip this skill when:
-- The user wants a one-off prompt tweak they explicitly won't validate with evals.
-- The user has already said they don't want to set up evals.
-- The request is generic prompt-engineering theory unrelated to a specific app.
+A judge runner built for flat rows will silently score every candidate 0 unless `flat_row` is selected by `--scorer-input-contract` or the runner manifest and, when needed, the candidate field is selected by flag or manifest.
 
-# Orizu CLI basics
+# Vocabulary
 
-## Prerequisites
+- **Evals-first** — understand observed failures, build and validate evaluations, then optimize the LLM application against that evidence.
+- **Journey step** — one ordered stage in the workflow: J0 — Discover & evaluate, J1 — Install & set up, J2 — Assess & plan, J3 — Build the dataset, J4 — Ground truth via annotation (conditional), J5 — Build & validate judges, or J6 — Optimize & promote.
+- **Exit criterion** — the checkable evidence required before leaving one journey step for the next.
+- **Human-only** — an action the agent prepares with exact evidence and an authorized human executes.
+- **Local surface** — a CLI action executed under the user's token; the authority map, not the surface, determines whether an agent or human executes it.
+- **Hosted session** — a platform-managed coding-agent session authenticated with session credentials and bound to its signed team and project scope.
+- **Instruction set** — the customer-facing identity for one agent or LLM experience, with a fixed ordered component shape and model-config profiles that own component values.
+- **Judge trust bar** — the user-agreed metric thresholds a judge must clear for a named downstream decision and failure mode.
+- **Decision class** — the judge's downstream use: gatekeeper, optimization signal, or triage / monitoring.
+- **Scenario class** — a named, meaningful family of cases from the ratified improvement plan whose coverage and results are tracked separately.
+- **Final-held-out** — the application partition reserved for J6's single seed-versus-selected-candidate comparison.
 
-- Node.js 20+ installed.
-- Install the CLI: `npm i -g orizu`.
-- Run `orizu setup` to sign in, initialize the workbench, and install the global Codex/Claude Code skill symlinks.
-- Or install this skill directly from the CLI package with `orizu install-skill --agent codex --yes` (or `--agent claude`; advanced target IDs remain available via `--target`).
-- Or read the bundled skill in place without installing: `orizu skills path --json` returns `root`, `skillMd`, `source`, `cliVersion`, and `skillHash`.
-- Before doing Orizu work, verify the runtime with `orizu --version` and `orizu capabilities --json`. If the CLI is missing or stale, pause and ask the user to update it.
-- After CLI upgrades, check installed copies with `orizu skills status` and refresh them with `orizu skills update`.
-- Rediscover the command surface any time with `orizu --help`, `orizu <group> --help`, or `orizu capabilities --json`.
-- Every command supports `--json` (prefix `orizu --json <cmd>` or trailing flag) and emits a single JSON document; prefer it over parsing human-formatted text. Long-running commands emit the JSON summary as the final stdout line.
-- Login callback requires `127.0.0.1:43123`.
-- Credentials are stored at `~/.config/orizu/credentials.json`.
-- Current `orizu login` creates a user-owned personal access token for the CLI and stores it as a v3 API-key credential. The raw token is not recoverable after creation.
+# Journey
 
-## Login
+## J0 — Discover & evaluate
 
-- Verify auth: `orizu whoami --json` (returns `{user, server}`; non-zero exit means sign in is needed).
-- Start the login flow: `orizu login` (opens a browser tab).
-- Approving login in the browser creates a personal access token for that Orizu user. The token inherits the user's current team/project access and loses access if the user's role or membership changes.
-- Clear local credentials and revoke the current CLI token: `orizu logout`.
-- Revoke other CLI tokens from the Personal Tokens page in Orizu.
-- Auth failure loop: `orizu login` → `orizu whoami` → rerun the original command.
-- Hosted sandboxes are pre-authenticated via `ORIZU_TOKEN_FILE` (no `orizu login` needed); an explicit `ORIZU_TOKEN` env var overrides both the token file and stored credentials when set.
+- Before installation, explain what Orizu does, inspect only the context the user has supplied, and give an honest fit assessment for the user's LLM application.
+- Recommend adoption when repeated iteration would benefit from measurable evaluations. Step aside for generic instruction-writing theory or a one-off edit the user explicitly will not validate with evals.
 
-## Teams and projects
+**Exit criterion:** The human has the fit assessment and makes the adoption decision.
 
-Projects live inside teams. Resolve or create both before any workflow command.
+## J1 — Install & set up
 
-- `orizu teams list` — list teams the user belongs to.
-- `orizu teams create --name "<team name>"` — create a team.
-- `orizu projects list [--team <teamSlug>]` — list projects (optionally scoped to a team).
-- `orizu projects create --name "<project name>" --team <teamSlug>` — create a project.
+- For local use, install Node.js 20+, run `npm i -g orizu`, then run `orizu setup` so browser login, the workbench, and skill links are initialized. The callback defaults to `127.0.0.1:43123`; after a collision, set `ORIZU_AUTH_PORT` to an available port from 1024–65535.
+- Verify auth and runtime with `orizu whoami --json` and `orizu capabilities --json`. Verify skill installation and sync separately with `orizu skills status --json`; use `orizu skills update` if stale. Follow the Authority map for surface-specific setup.
+- Establish the team and project with the user's naming choices as directed by the Authority map. Projects live inside teams.
+- Browser login creates a user-owned personal access token. The server/UI cannot redisplay the raw token after issuance, but the local v3 credential retains it for CLI auth. The token inherits current access and loses access when the user's role or membership changes.
 
-For team membership, app, dataset, and task command surfaces, see `references/cli-reference.md`.
+**Exit criterion:** The team and project exist; the skill is verified in sync by `orizu skills status` locally, or shipped and present in a hosted session; and both `orizu whoami` and `orizu capabilities` succeed.
 
-# Evals best practices
+## J2 — Assess & plan
 
-Walk through the four steps in order — each depends on the output of the previous. Methodology and rationale: `references/primer.md`.
+- Read `references/assess-and-plan.md` and follow its survey, decision, conversation, and persistence workflow before moving any data.
 
-## 1. Upload — gather diverse traces
+**Exit criterion:** A human-ratified `improvement-plan.md` is committed at the plain-repo root or at `projects/<directorySlug>/improvement-plan.md` in a workbench; there, it and its `memory.md` pointer are committed together.
 
-Principles:
-- Mix production traces with a **random sample**. Thumbs-down feedback alone is biased toward extreme failures and misses subtle ones.
-- Aim for **~100+ diverse traces**. Stop adding when you stop discovering new failure modes (theoretical saturation).
-- Synthetic data is fine when real traces are scarce — vary structured inputs (feature × persona × scenario), seed from real logs, and filter for difficulty. Don't ask "generate 50 test cases" cold.
+## J3 — Build the dataset
 
-CLI:
-```bash
-orizu datasets upload --project <teamSlug>/<projectSlug> --file <traces.jsonl> --name "<batch>"
-orizu datasets append --dataset <datasetId> --file <more.jsonl>
-orizu datasets lock --dataset <datasetId> --reason "Freeze for labeling"
-```
+- Follow `references/dataset-design.md` for the ratified source choice, scenario-class coverage, three-way split and mini-batch design, the reserved final-held-out comparison split, versioning, and the pre-annotation mutation guard.
 
-Deeper: `references/primer.md` (Step 1, Step 0 error analysis); `references/cli-reference.md` (datasets surface).
+**Exit criterion:** The primary dataset's immutable version and split set cover every scenario class; golden ground truth shortcuts J4 only, while any annotation dataset is versioned, locked, and ready for labeling.
 
-## 2. Annotate — binary labels per failure mode
+## J4 — Ground truth via annotation (conditional)
 
-Principles:
-- **Binary, not Likert.** Pass/fail forces a ship/no-ship decision; 3/5 doesn't.
-- **One question per failure mode.** Don't bundle correctness, tone, and helpfulness.
-- **Annotate failures you've actually observed.** Hypothetical-failure labels are low-value.
-- **Custom UI per task** — generic annotation interfaces collapse signal. The labeler should feel like an app annotators want to use, not a form they tolerate.
+- Follow `references/eval-strategy.md` for the annotation decision, guided eval strategy, labeler-validation gate, task-version pin, and completeness check.
 
-CLI:
-```bash
-orizu apps create --project <teamSlug>/<projectSlug> --name "<labeler>" --dataset <datasetId> \
-  --file <App.tsx> --input-schema <input.json> --output-schema <output.json>
+**Exit criterion:** Golden ground truth is recorded as the reason annotation was skipped, or labels are exported and the approved task report is published.
 
-orizu tasks create --project <teamSlug>/<projectSlug> --dataset <datasetId> --app <appId> \
-  --title "<round>" --labels-per-item 2
+## J5 — Build & validate judges
 
-# Open the returned task URL and test the draft manually before assigning.
-orizu tasks publish --task <taskId> --assignees <userId1,userId2>
-# Or publish exact row assignments:
-orizu tasks publish --task <taskId> --assignment-file ./assignments.jsonl
+- Build automated evaluators from the ground truth: use code assertions for deterministic rules and LLM judges only for nuanced criteria. A 100% pass rate is a saturation warning, not proof that the evaluator is useful.
+- Validate against judge-test human labels. Track TPR and TNR per failure mode, and use Cohen's kappa alongside raw agreement when establishing the judge trust bar with the user for the decision it powers.
+- Store the versioned judge and runner, then register the row or set scorer that defines how its output is displayed, compared, and used. GEPA reflection needs a row-mode scorer because it consumes per-row feedback.
+- Read `references/building-judges.md` for judge trust bar agreement, submitted alignment evidence, and judge optimization; its agreed judge trust bars govern over older fixed defaults. Read `references/prompt-control-plane.md` for artifact contracts.
 
-orizu tasks status --task <taskId>
-orizu tasks report set --task <taskId> --report-file ./task-report.md
-orizu tasks export --task <taskId> --format jsonl --out ./labels.jsonl
-```
+**Exit criterion:** Every gating judge clears its agreed judge trust bar; its scorers are registered; `runners exec` evidence, `scores submit`, and a measured `scorers exec` run are submitted; and the agent hands `orizu scores accept <score-run-id>` to a human curator, whose acceptance is required before gating decisions.
 
-Authoring the labeler: **`references/building-apps.md`** covers the component contract, design principles, common patterns (trace exploration, side-by-side, text annotation, etc.), the offline smoke test at `scripts/test-app.mjs` (runs on plain `node`), and `orizu apps preview`. When you generate or modify an app, run the preview command with a representative sample row, inspect the screenshot, and revise the app until the rendered workflow appears to match the user's intended review task before `orizu apps create`.
+## J6 — Optimize & promote
 
-Deeper: `references/primer.md` (Step 2); `references/cli-reference.md` (apps + tasks surface).
+- Optimize only against validated judges. Package candidate execution as a runner and run the optimizer against registered scorers. After selection, run one final comparison of the selected candidate against the seed on the final-held-out partition, with per-failure-mode results.
+- Use `orizu instructions` for the instruction set. Each run targets one model-config profile and treats its complete component map as the candidate; promotion never moves one component independently.
+- Inspect the local run logs when available, otherwise export the run. Write and attach a markdown report using `references/optimization-reports.md`; the report should explain candidate tradeoffs and regressions clearly enough for the human to make the promotion decision.
 
-## 3. Judge — turn labels into automated evaluators
+Bundled `run-gepa` flag behavior and execution semantics live in `references/optimization-with-gepa.md`.
+Read that reference before configuring a run; keep this journey focused on workflow and exit criteria.
 
-Use the Phase 0 prompt control plane when the user wants judges stored in Orizu as versioned prompt artifacts, runner zips, scorer definitions, and submitted score results. Read **`references/prompt-control-plane.md`** before using those commands. When the user mentions prompt feedback, prompt comments, or asks what to focus on for the next prompt version, inspect comment threads with `orizu comments list --prompt <prompt-id-or-name> --project <team>/<project>` before editing or pushing.
+**Exit criterion:** The human has made a promotion decision from the optimization report; if shipping, the new profile version is live.
 
-Judge construction details are still useful when authoring the evaluator itself.
+# Improve continuously
 
-A judge is the evaluator prompt artifact. A scorer is the metric contract that names what score is being produced and how it should be displayed, compared, and used in optimizations. For UI-visible prompt performance, register a scorer and submit score runs; do not rely only on raw `runs submit`.
+- Feed improved-system traces into a new dataset version and repeat from dataset building. Each report should recommend the next scenario classes or failure modes to investigate.
+- Propose a cadence for fresh-trace collection, evaluation, and optimization; set it up only after the user agrees, then complete the first cycle.
 
-Set scorers are aggregate metric contracts. Use `orizu scorers exec` for builtin set metrics such as Cohen's kappa, accuracy, precision, recall, and F1; it consumes subject results or dependency row-scorer results, computes one aggregate, stores row evidence, and submits a canonical score run by default. If you already computed an aggregate locally, use `orizu scores submit --aggregate` with explicit `scoreValue`, `diagnostics`, `feedbackSummary`, and optional `rowEvidence`. `runners exec --scorer-version` remains a low-level row-runner compatibility command. Use a row-mode scorer, not a set scorer, for GEPA reflection because reflection needs per-row feedback.
-
-Principles:
-- **Code assertions first.** If a failure is a rule (keyword present, tool called, format valid), write a code check. Fast, free, deterministic.
-- **LLM-as-a-judge for nuanced criteria only.** Always validate against your human labels.
-- **TPR > 90% and TNR > 90%** before trusting a judge. Track each separately — accuracy is misleading on imbalanced data.
-- **A 100% pass rate is a smell.** Your evals are saturated; add harder cases.
-
-Authoring workflow:
-1. Export labels: `orizu tasks export --task <id> --format jsonl --out labels.jsonl`.
-2. For each failure mode, choose code assertion or LLM-judge.
-3. Build the judge.
-4. Validate: split labels train/dev/test (20/40/40), measure TPR + TNR on test, target both > 90%.
-5. Run the validated judge over future outputs.
-
-Detailed walkthrough — code assertion patterns, LLM-judge prompt scaffold, train/dev/test split, TPR/TNR computation, saturation: **`references/building-judges.md`**.
-
-## 4. Optimize — hill-climb against validated judges
-
-Use the Phase 0 prompt control plane when the user wants an optimizer run to appear in Orizu, stream live events, or promote accepted candidates into the prompt timeline. For text-candidate GEPA-style optimization, prefer the bundled `orizu optimizations run-gepa` flow. Read **`references/prompt-control-plane.md`** before wiring those endpoints.
-
-GEPA details below are still useful for local optimizer implementation. DSPy is only relevant when the customer already uses DSPy or asks for an external DSPy GEPA implementation.
-
-Principles:
-- Only optimize against judges you've validated. Otherwise you Goodhart your way to a worse system.
-- Compare before/after on the **same held-out eval suite**. Don't trust vibes.
-- Read **per-failure-mode metrics**, not just combined — averages hide regressions.
-- Improved-system traces feed back into step 1; the loop continues.
-
-Local execution workflow:
-1. Package the candidate execution as an Orizu runner.
-2. Register validated row/set scorers.
-3. Run the bundled GEPA-style optimizer or a custom optimizer against the scorer set.
-4. Inspect `logs/<optimization_run_id>` for full local optimization traces, especially `evaluations.jsonl`, `reflections.jsonl`, `events.jsonl`, and `result.json`.
-5. If the local log is unavailable or the run happened remotely, use `orizu optimizations export <run-id> --out <run-id>.optimization.json`.
-6. Write a markdown optimization report from the logs/export and attach it with `--report-file <path>` when finishing, failing, or cancelling the run.
-7. Diff before/after on a held-out set; ship if it holds.
-
-Bundled `run-gepa` reflection behavior:
-- **Scorer contract:** `run-gepa` sends the scorer runner a GEPA-shaped row (`{source_row, candidate_id, candidate_output, …}`), NOT the flat-row shape used by `runners exec --scorer-version`. A judge runner built for flat-row score runs will silently score every candidate 0 unless you pass `--scorer-input-contract flat_row` (add `--scorer-candidate-field <row-field>`, e.g. `draft`, when the judge reads the candidate output from a named row field). Do not hand-write an adapter runner — the flag applies the official adapter without changing the registered runner bytes, so it composes with `--scorer-runner-dir` content-sha verification. `run-gepa` validates the contract on the seed at launch and refuses a uniformly-worst-scoring seed with a diagnosis; `--allow-degenerate-seed` overrides only when the seed genuinely deserves the worst score everywhere. Both contracts: `references/prompt-control-plane.md` ("Scorer-Runner Input Contracts").
-- Budget controls are mutually exclusive. Use at most one of `--budget auto|light|medium|heavy`, `--max-metric-calls <n>`, `--max-full-evals <n>`, `--max-iterations <n>`, or `--max-candidate-proposals <n>`. The proposal cap is official-engine-only. If none is provided, `run-gepa` uses `--budget auto`, the balanced medium preset.
-- **Skilled proposer:** opt in with `--candidate-proposer skilled-proposer` on the official engine; the selected run prepares or reuses its managed Python environment automatically. Bound proposal work with `--proposal-max-tokens` and `--proposal-max-calls`, independently of `--max-metric-calls`. Read `references/prompt-control-plane.md` ("Skilled proposer") for validation and recovery; readiness is proven when optimizer execution begins without an `ALI_1505_*` environment diagnostic.
-- The reflective LM's final text is used verbatim as the next candidate prompt. It should return only the complete updated prompt body, not analysis, labels, XML tags, or markdown fences.
-- Keep provider-native reasoning controls separate from the prompt text with `--reflection-provider-settings <json|@file>`.
-- Reflection max-token limits are explicit config, not implicit defaults. Use `--reflection-max-tokens <n>` when a provider requires an output cap; this maps to Anthropic `max_tokens` and OpenAI `max_output_tokens`.
-- Anthropic native Messages reflection requires `max_tokens`, so `run-gepa` with an `anthropic/...` reflection model must receive `--reflection-max-tokens <n>` and should fail fast if it is omitted. OpenAI reflection may omit `max_output_tokens` unless the user explicitly wants a cap.
-- Reflection HTTP calls retry transient provider failures by default (`--reflection-retry-attempts 3`, `--reflection-http-timeout-seconds 180`). Retryable failures include timeouts, connection reset/abort, HTTP 429, and common server/capacity statuses. If reflection still fails after retries, `run-gepa` logs `reflection_failed`, counts the proposal against budget, and continues to the next iteration.
-- By default, `run-gepa` skips reflection and child creation when every row in the selected parent mini-batch is already perfect (`1.0` for higher-is-better scorers, `0.0` for lower-is-better scorers). This saves reflection tokens because the child cannot beat a saturated mini-batch. Use `--no-skip-perfect-parent-reflection` to force reflection anyway, or `--skip-perfect-parent-reflection` to make the default explicit in scripted runs.
-- By default, `run-gepa` sets `--num-threads auto` for row evaluations. Auto uses mini-batch size, validation-set size, 2x CPU count, memory estimate, file-descriptor limit, and a 64-thread default ceiling to avoid launching too many runner subprocesses at once while still scaling large runs. Set `ORIZU_GEPA_AUTO_THREADS_MAX` or use `--num-threads <n>` only when the runner/provider capacity is known.
-- `run-gepa` writes complete local logs by default under `logs/<optimization_run_id>`; override with `--log-dir <dir>` or disable with `--no-local-log`.
-- Server optimization events redact row snapshots and reflection prompts by default; the local log keeps full row inputs, outputs, scores, feedback, reflection prompts, and reflection responses for later agent analysis.
-- OpenAI example: `--reflection-model openai/gpt-5 --reflection-provider-settings '{"reasoning":{"effort":"medium","summary":"auto"}}'`.
-- Anthropic example: `--reflection-model anthropic/claude-opus-4-7 --reflection-max-tokens 4096 --reflection-provider-settings '{"thinking":{"type":"adaptive","display":"omitted"},"output_config":{"effort":"medium"}}'`.
-
-Detailed walkthrough — GEPA mechanics, Orizu-tracked optimization, optional DSPy context for customers already using it, and before/after comparison: **`references/optimization-with-gepa.md`**. Report-writing structure and interpretation guidance: **`references/optimization-reports.md`**.
+The loop is established when a cadence is agreed and its first cycle is complete, with new data feeding the next J3 dataset version.
 
 # Reference index
 
-- `references/primer.md` — methodology end-to-end (read first when in doubt about *why*).
-- `references/cli-reference.md` — full CLI command surface.
-- `references/prompt-control-plane.md` — Phase 0 prompts, judges, scorers, runners, score submission, optimizer event logging, bundled GEPA, and promotion endpoints.
-- `references/building-apps.md` — labeler app contract, design principles, common patterns, offline smoke test.
-- `references/building-judges.md` — judge/scorer authoring + TPR/TNR validation.
-- `references/optimization-with-gepa.md` — GEPA optimization loop, with DSPy only as external context.
-- `references/migrate-existing-gepa-setup.md` — migrating a customer's existing official-GEPA script into Orizu, ending in `orizu scorers verify-parity` before any optimization.
-- `references/optimization-reports.md` — report template, GEPA diagnostics, kappa/confusion-matrix guidance, and next-step recommendations.
-- `scripts/test-app.mjs` — smoke test for `App.tsx` + schemas before `orizu apps create` (runs on plain `node`).
-
-# Execution rules
-
-- Prefer explicit flags in non-TTY contexts; reserve interactive fallback for TTY.
-- Canonical selectors: `--team`, `--project`, `--app`, `--task`, `--dataset`, `--assignees`, `--assignment-file`.
-- `tasks create` creates drafts by default. Test the returned task URL manually, then use `tasks publish --task <taskId> --assignees <userId1,userId2>` or `tasks publish --task <taskId> --assignment-file <path>` after approval.
-- `tasks assign --assignees` and `tasks publish --assignees` take user IDs, not emails. `--assignment-file` accepts emails or user IDs in JSONL rows shaped as `{ "rowId": "...", "assignee": "..." }` or `{ "rowId": "...", "assignees": ["..."] }`.
-- `datasets edit-rows` requires a non-empty string `id` on each row in `--file`.
-- `datasets delete-rows` uses `--row-ids` as the canonical selector.
-- `datasets delete` requires interactive terminal confirmation; there is no non-interactive confirmation option.
-- Locked datasets reject append/edit/delete row mutations.
-- `--output-schema` JSON Schema validation surface is restricted to `type`, `required`, `properties`, `items`, `enum`.
-- Export defaults: `--format jsonl`, output `<taskId>.<format>`.
-- Task reports can be uploaded only after the task is `paused` or `completed`; rerun `tasks report set` or `tasks report upload` to replace the markdown report.
-- Prompt control-plane commands should use ids for dataset versions, split sets, prompt versions, scorer versions, runner versions, optimizer versions, and optimization runs.
-- Optimization exports default to `<run-id>.optimization.json`; prefer the existing `logs/<run-id>` directory from `run-gepa` when it is available because it contains the full local trace without needing server rehydration.
-- When ending an optimization run, attach a markdown report with `--report-file` when possible. Use `references/optimization-reports.md` for what to include.
-- Use `orizu comments list --prompt <prompt-id-or-name> --project <team>/<project> [--label <label> | --version <id>] [--json]` to list prompt-level discussion threads with open/resolved status, selected text/line context, and replies.
-- Use `orizu comments diff --run <optimization-run-id> [--from <candidate-id> --to <candidate-id>] [--detail hunk|diff|full] [--json]` or `orizu comments diff --prompt <prompt-id> [--from <version> --to <version>] [--detail hunk|diff|full] [--json]` to export comments grouped by candidate/version pair with anchored diff context. Exactly one target is required; pair selectors must be supplied together, and prompt selectors are integer version numbers.
-
-# Where things live
-
-- **Orizu is the source of truth for run metadata and pointers**: production/default prompt/scorer/runner pointers, version lineage, score runs, optimization runs, and report history all live in Orizu — query them with `orizu ...` commands (e.g. `orizu prompts list`, `orizu prompts versions`, `orizu runs ...`), never by reading repo files.
-- **The repo is the working set (files only)**: source, configs, `App.tsx`, schemas, and local traces under `logs/<run-id>`. It is the scratch space you edit and commit; it does NOT hold the canonical pointer/run state — that is in Orizu.
-- **Hosted sessions ship a pre-authenticated CLI**: inside a hosted agent sandbox the `orizu` CLI is already authenticated via `ORIZU_TOKEN_FILE` (with `ORIZU_BASE_URL` pointing at the API). You do NOT run `orizu login` — just call `orizu ...`. The bearer carries **full team-curator authority while your session is active** (ADR-008): anything a curator-role human can read or write in your team — create/read/update projects, datasets (+ rows/artifact bytes), prompts, scorers, runners/optimizers, apps, tasks, runs, score runs, reports, and promotion-manifest proposals — you can too, across the whole team. (The full per-role access matrix lives in the Orizu repo at `docs/authorization-matrix.md`.)
-- **Git-canonical artifacts: always write with `--session`, and "committed" means bytes-in-branch**: prompts/judges, scorers, runners, and optimizers are git-canonical (ADR-007) — their bytes must live as commits on your session branch, not only as Orizu version rows. From a hosted session, always pass `--session <session-id>` on artifact writes (`prompts push`, `judges push`, `scorers register`, `runners push`, `optimizers push`); the legacy sessionless create routes reject agent bearers with a 403 pointing you back at `--session`. The definition of done for any artifact you claim to have committed is that the bytes are actually in the branch: run `git ls-files <path>` (and check the commit) before reporting an artifact as committed — an Orizu version id alone is not proof the bytes exist anywhere.
-- **Human-only actions will 403**: promotion-manifest **approval / apply / reject**, production and default pointer flips, session/credential lifecycle (starting or ending hosted sessions, minting tokens, deploy keys), integration/connector credential reads (secrets reach your sandbox only as environment variables resolved at boot — never via API/DB), and team/member/installation/billing management. Do not attempt them from a hosted session — they return 403 by design. Prepare a promotion manifest and hand it to a human curator to approve and apply.
+- `references/primer.md` — end-to-end evals-first methodology.
+- `references/assess-and-plan.md` — codebase survey, user-owned decisions, plan conversation, and durable J2 artifact.
+- `references/cli-reference.md` — current CLI command surface.
+- `references/dataset-design.md` — golden-data import, scenario-class coverage, splits, and versioning.
+- `references/eval-strategy.md` — annotation decision, eval strategy, and labeler validation before task publish.
+- `references/building-apps.md` — labeler contract, design, patterns, preview, and smoke test.
+- `references/building-judges.md` — judge/scorer authoring and alignment validation.
+- `references/prompt-control-plane.md` — artifact contracts, score submission, optimizer behavior, and promotion endpoints.
+- `references/optimization-with-gepa.md` — GEPA workflow and optional DSPy context.
+- `references/migrate-existing-gepa-setup.md` — migration ending in scorer parity before optimization.
+- `references/optimization-reports.md` — report structure and interpretation guidance.
+- `references/instructions-after-prompts.md` — prompts-era compatibility, including the remaining session-scoped read surface; read before changing legacy material.
+- `scripts/test-app.mjs` — plain-Node app contract smoke test.
+
+# Execution facts
+
+- `--json` may appear before or after a command. Non-streaming commands emit one JSON document, with long-running summaries on the final stdout line; streaming `orizu run tail --json` instead emits JSONL, one event object per line.
+- Task creation is draft-first; publish only after the returned task URL has been tested and approved. Both `--assignees` and assignment-file rows accept user IDs or member emails.
+- Dataset locks reject append, edit, and delete-row mutations, but whole-dataset deletion does not check the lock. `datasets edit-rows --file` requires every row to carry a non-empty string `id`.
+
+# Authority map
+
+This is the single source of truth for who runs each action. Replace placeholders with resolved values before giving a hand-off. The full per-role matrix is `docs/authorization-matrix.md`.
+
+| Surface | Work | Agent runs | Human role or exact hand-off |
+| --- | --- | --- | --- |
+| All surfaces | Resolve state | Query Orizu for instruction-set defaults, profile production pointers, scorer/runner pointers, version lineage, score runs, optimization runs, and report history; for example, `orizu instructions list --project <team/project>`, `orizu instructions show <set> --project <team/project>`, and `orizu runs ...`. | Orizu is the source of truth for this state. |
+| All surfaces | Edit files | Edit and commit source, configs, `App.tsx`, schemas, and local traces under `logs/<run-id>` in the repo working set. | Repo files are not canonical pointer or run state. |
+| Local agent under the user's token | Workflow | Survey the customer codebase, resolve state, prepare the plan, files, manifests, diffs, and evidence, then ask the human to ratify the workflow design. | The human moments are plan ratification and execution of all production/default pointer moves. |
+| Local agent under the user's token | Catalog mutations and pointer preparation | After plan ratification, pull instruction material from the customer codebase and run `orizu instructions create <manifest> --project <team/project>`, `orizu instructions push <manifest> --project <team/project> --set <slug-or-exact-name>`, shape add/remove, archive/restore itself under the user's token. For pointer changes, gather only read-only resolution and binding evidence, prepare the exact commands, then hand them off. | Per the Micro-round 3 custody ruling, a human admin or curator using a user token executes every runner default-label move, optimizer-label move, scorer production-label move, profile promote/rollback, default move, and labeled optimization promotion on every surface. |
+| Hosted session | Start and scope | Use the pre-authenticated CLI from `ORIZU_TOKEN_FILE` and `ORIZU_BASE_URL`; resolve only the session's signed team with `orizu teams list`, then list or create projects only within it. Run `orizu projects create --name "<project name>" --team <signed-team>` when needed. | Team creation is platform-enforced human-only. Hand off: `orizu teams create --name "<team name>"`, then ask the human to start a new hosted session scoped to that team; the current session cannot switch teams. |
+| Hosted session | Team-wide curator work | While the hosted session is active and its kill switch is clear, perform setup plus ordinary curator reads and writes within the signed team. Run `orizu instructions profiles new <set> --project <team/project> --model-config <identity>` and `orizu instructions sync <set> --out <dir> --project <team/project>`. For git-canonical judges, scorers, runners, and optimizers, write with `--session <session-id>`, keep bytes as commits on the session branch, then verify with `git ls-files <path>` and inspect the commit. | The agent executes this work; an Orizu version id alone is not proof of committed bytes. Signed-project exceptions are in the next row. |
+| Hosted session | Signed-project work | Run optimization mutations only for the JWT's project, keep git-canonical draft writes on that session and project, and read diff comments only for that project. | Sibling-project optimization mutations, draft writes, and diff-comment reads are out of scope even where ordinary curator access is team-wide. |
+| Hosted session | Materialize an instruction set | When the survey finds only a candidate seed, turn it into the complete manifest. | ADR-008 requires a human curator using a local CLI user token. Create/push are hosted CLI route-enforced; direct RPC remains agent-writable and is a fix-forward gap. Hand off `orizu instructions create <manifest> --project <team/project>` for a fresh set, or `orizu instructions push <manifest> --project <team/project> --set <slug-or-exact-name>` for an existing set. Their measured 403 is `Sessionless registration is not available to the hosted agent: this artifact class is git-canonical (ADR-007) and its bytes must be committed to the session branch. Re-run with --session <session-id>.` None of these six catalog mutation commands accepts `--session`. |
+| Hosted session | Profiles and component shape | Prepare any shape-change manifest. | A human admin or curator with `can_manage_project` runs `orizu instructions shape add <set> --project <team/project> --key <key> --from <manifest>` or `orizu instructions shape remove <set> --project <team/project> --key <key>`. Route and database enforcement return `Agents cannot change instruction-set shapes`. |
+| Hosted session | Instruction-set visibility | Prepare the requested visibility change. | A human admin or curator with `can_manage_project` runs `orizu instructions archive <slug-or-exact-name> --project <team/project>` or `orizu instructions restore <slug-or-exact-name> --project <team/project>`. Hosted CLI route enforcement returns `Agent sessions cannot perform this action`; direct RPC remains agent-writable. |
+| Hosted session | Instruction scorers | Run `orizu instructions scorers set-headline <set> --key <component-key> --scorer-version <id> --project <team/project>` and `orizu instructions scorers add <set> --key <component-key> --scorer-version <id> --project <team/project>`; prepare scorer-pointer evidence. | A human admin or curator with `can_manage_project` runs `orizu scorers labels set <scorer-name> production --version <scorer-version-id> --project <team/project>`. The measured 403 is `Agents cannot move scorer labels; labels are human-only pointers. Propose the label change as a promotion manifest for a human to apply.` |
+| Hosted session | Promotion | After report acceptance, the two-stage path materializes once with unlabeled `orizu optimizations promote <run-id> --candidate <id> --project <team/project>`; hand the exact run id, candidate id, project, and report evidence to the human. | For the staged path, a human admin or curator with `can_manage_project` re-runs the same promotion as `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`. The idempotent finalizer finds the existing version by run and candidate provenance, moves production to that exact materialized version, and creates no duplicate profile version or candidate-promoted event. Equivalent simpler one-shot path: without prior materialization, the human runs the same labeled command once. A human admin or curator also runs `orizu manifests approve <id>`, `orizu manifests reject <id>`, and, after approval, `orizu manifests apply <id>`. |
+| Hosted session | Review instructions | Read diff comments with `orizu comments diff --run <run-id> --from <candidate-id> --to <candidate-id> --detail hunk` or `orizu comments diff --prompt <prompt-id> --from <version> --to <version> --detail hunk`; keep report-comment CLI targets inside the signed project and prepare proposed review text. | A human admin or curator adds or deletes a diff comment. To change one, delete it and re-add it; the mutation APIs expose POST and DELETE, not edit. Hand off the target, exact anchor, exact text, and named add/delete action. |
+| Hosted session | Session, credentials, and administration | Use connector secrets only through environment variables resolved at boot; prepare requested values without reading credential material. | A team ADMIN runs `orizu teams members add --email <email> --team <team>`, `orizu teams members remove --email <email> --team <team>`, or `orizu teams members role --team <team> --email <email> --role <admin\|curator\|judge>`. A session owner or team admin/curator may run `orizu session end --session <id>` for manual teardown; the coordinator owns automatic lifetime and provisions session and service-internal canonical-write credentials. A human runs `orizu session start --hosted --task "<prompt>" --project <team/project>`. For provider installation, kill switch, billing, deploy keys, or credential reads, ask the human to complete the named UI action and report completion without sharing secrets. |

package/vendor/skills/orizu-cli/references/assess-and-plan.md

@@ -0,0 +1,107 @@
+# Assess & plan
+
+This journey turns evidence from the customer's codebase into an evals-first improvement plan. It ends before dataset construction: the durable exit artifact is a human-ratified `improvement-plan.md` written to the canonical path in section 7 and committed before data access.
+
+## 1. Hold the data boundary
+
+Survey code, configuration, documentation, and the names and wiring of integrations. Inventory candidate customer data, but do not open or read, export, sample, transform, or upload it. Do not call a provider, run a production query, copy traces, or create an Orizu dataset during this journey. Inspect secret names and configuration shape only; never read credential values.
+
+Only after the user ratifies the plan and the ratified plan is committed at its canonical path, together with the required workbench memory pointer, may J3 — Build the dataset access the approved sources. If the user asks to move data while decisions remain open, show the missing decisions and ask for ratification first.
+
+## 2. Survey the codebase
+
+Locate every LLM surface, not only obvious instruction files. Search for model SDK and API calls, agent entry points, RAG or retrieval pipelines, tool definitions, structured-output parsers, instruction and template files, Model Config identity and settings, and fallbacks. For each surface, record:
+
+- the user-visible job, owner, entry point, and execution flow;
+- its instruction set or candidate seed, using the instructions CLI command surface when it is already registered;
+- Model Config identity and settings, tools, retrieval dependencies, output contracts, and safety boundaries;
+- observed failure reports and other evidence, clearly separated from hypotheses;
+- existing evals, tests, judges, golden labels, and deployment or monitoring gates;
+- where Orizu could add measurement leverage and where it would add no value.
+
+Do not collapse multiple surfaces into one optimization target. Call out repeated infrastructure and dependencies, but keep each surface's failures and evidence attributable.
+
+## 3. Inventory candidate data sources
+
+Build a source inventory from code and integration metadata without accessing the underlying records. Include unavailable sources and explain the gap; absence is a planning fact, not permission to substitute synthetic data silently.
+
+| Candidate source | What to establish without opening records | Decision to bring to the user |
+|---|---|---|
+| Production logs | Owning system, retention window, schema, volume, sampling hooks, redaction boundary | Whether access is allowed and which representative sampling policy is acceptable |
+| Observability providers | Wired provider and project, trace shape, feedback or score fields, export boundary | Whether this provider is authoritative and which fields may leave it |
+| User-behavior signals | Feedback controls, retries, abandonments, escalations, edits, or downstream outcomes | Which signals mean success or failure and how selection bias will be handled |
+| Existing golden datasets | Location, provenance, scenario coverage, label meaning, freshness, and split history | Whether the labels remain trusted ground truth or need human spot-checking |
+
+Also note synthetic generation as a possible gap-filler, never as an automatic replacement for missing real evidence. Record privacy, security, consent, residency, retention, and identifier-handling constraints for every candidate source.
+
+## 4. Surface the judgment calls
+
+Convert survey gaps into an explicit decision ledger. The human owns these calls; evidence may support a recommendation, but the agent does not decide by omission. At minimum, surface:
+
+- the user outcome, baseline, success measure, acceptable regressions, and priority order;
+- the observed failure modes and scenario class boundaries worth measuring first;
+- the data-source decision, including access, sampling, privacy, retention, and whether synthetic cases may fill a named gap;
+- whether existing labels are trusted ground truth, whether humans must annotate, and who can make that judgment;
+- the judge type for each failure mode and the judge trust bar appropriate to the decision it will power;
+- the customer-facing instruction set, model-config profile, components, and other code that are in or out of the optimization target;
+- Final-held-out validation, per-scenario regression limits, cost and latency limits, rollback evidence, and who owns the promotion decision.
+
+Mark each item **decided**, **proposed**, or **open**, with its evidence and owner. Never convert a default, convention, inferred preference, or agent recommendation into a user decision.
+
+## 5. Run the guided conversation
+
+Use a pointed, grilling-style conversation to resolve the ledger, not a generic intake questionnaire:
+
+1. Present the codebase assessment first: observed facts, hypotheses, gaps, Orizu opportunities, and the decisions they create.
+2. Ask the highest-leverage open question in context. Explain why it matters, give concrete options and tradeoffs when useful, and state a recommendation separately from the decision.
+3. Probe vague goals. Ask what observable outcome would disprove success, which scenario class matters first, and what regression would block shipping.
+4. Challenge contradictions and hidden assumptions respectfully. For example, a source cannot be both unavailable and the required ground truth; a judge cannot gate promotion before its trust bar is agreed and validated.
+5. Reflect the answer into the ledger and confirm the consequence. Then take the next open decision.
+
+Never answer your own questions, infer approval from silence, or bury a choice in implementation detail. If the repository or supplied context already records an answer, use it and cite it; do not re-ask the user unless the records conflict or the decision is explicitly stale.
+
+After the ledger is resolved, present the complete plan and ask the user to revise, reject, or explicitly ratify it. “Looks plausible,” lack of objection, and approval of one source are not plan ratification. Change the plan from **Proposed** to **Ratified** only when the user explicitly ratifies the complete plan.
+
+## 6. Write `improvement-plan.md`
+
+Write a specific plan that another agent can execute without reconstructing the conversation. Use this outline, adapting detail rather than deleting unresolved decisions:
+
+1. **Status and ratification** — Proposed or Ratified; Orizu `team/project`; ratifier, date, and the user statement or recorded decision that constitutes approval.
+2. **Desired outcome and success measures** — user-visible goal, current evidence, target measures, constraints, and non-goals.
+3. **Codebase assessment** — every LLM surface, its entry point and flow, instruction set or seed, Model Config, dependencies, owner, existing evals, and Orizu opportunity.
+4. **Failure modes and scenario classes** — observed evidence, priority, coverage boundary, and explicit hypotheses that still need data.
+5. **Source inventory and data-source decision** — all candidate sources, the chosen sources and rejected alternatives, access owner, sampling, privacy, retention, redaction, and spot-check plan.
+6. **Ground truth and annotation** — trusted golden labels or human labeling path, decision owner, and how label quality will be checked.
+7. **Judge plan and trust bars** — code assertion or LLM judge per failure mode, validation evidence required, and the user-owned judge trust bar for each downstream decision.
+8. **Instruction set and optimization target** — target profile and complete component map, other code in scope, exclusions, and the held-constant environment.
+9. **Final-held-out validation and promotion decision** — split strategy, scenario-class measures, regression limits, cost and latency bounds, ship/rollback evidence, and the human promotion owner.
+10. **Phased work and handoffs** — ordered dataset, annotation, judge, optimization, and report work with human gates before irreversible or production-impacting actions.
+11. **Open decisions** — owner, evidence needed, and why each item blocks ratification or a later phase.
+
+Do not disguise open decisions as assumptions. If any ratification-blocking item remains, keep the status Proposed and continue the conversation. Once the user ratifies, update the status and ratification record, remove resolved items from Open decisions, and write `improvement-plan.md` to the canonical path below. Complete its required memory update and commit before accessing any approved data source.
+
+## 7. Persist only what Orizu supports today
+
+The plan artifact is always named `improvement-plan.md`; its canonical path depends on the workspace:
+
+- **Orizu workbench:** resolve the selected project's `directorySlug` through the same per-project manifest that locates `projects/<directorySlug>/memory.md`: find the `projects/<directorySlug>/orizu.project.json` whose project identity matches the resolved Orizu `team/project`, then use that containing directory. Write `projects/<directorySlug>/improvement-plan.md`. Add a concise decision summary and relative plan link to the sibling `memory.md`; do not duplicate the whole plan there.
+- **Plain repository:** write `improvement-plan.md` at the repository root and record the resolved Orizu `team/project` inside it.
+
+If the matching `projects/<directorySlug>/orizu.project.json` is absent, or a resolved project directory lacks its sibling `memory.md` or any required primitive directory, do not guess the directory slug or create missing entries by hand.
+With existing authenticated CLI credentials, run `orizu setup --team <teamSlug> --non-interactive --verbose` from the workbench root; the current directory is the default workspace.
+Authenticated setup reconciles each required project path independently: it creates missing files and directories while preserving existing ones.
+Empty project directories are not committed and must be recreated after a fresh clone.
+This per-path seeding does not require `--fix`. Use a separate `orizu setup --team <teamSlug> --fix --non-interactive --verbose` run only when independently diagnosed ADR-004 case or pointer damage also needs repair.
+Verbose output shows the actions.
+When setup completes, re-resolve `orizu.project.json` and continue only from the directory whose manifest identity matches the resolved Orizu project.
+If the manifest is still absent, another required entry remains missing, or setup reports an authentication or repair error, stop and ask the user to repair the workbench instead of writing the plan to a guessed path.
+Setup repairs the local checkout only: it does not commit or push.
+Track newly materialized project-stub files and commit them together with the plan and memory pointer.
+
+The repo is truth for the plan contents. In a workbench, both `projects/<directorySlug>/improvement-plan.md` and its sibling `memory.md` pointer must be tracked and committed together. In a plain repository, the root plan must be tracked and committed. These commits make the ratified plan durable and keep later work in the approved project.
+
+There is no dedicated CLI command for uploading an improvement plan or arbitrary project context. The generic `orizu workspace apply <path>` command can promote a registered repo-owned resource such as project `memory.md`; it does not make the plan a DB-native artifact or introduce a plan-specific storage format. Richer structured persistence is arriving through the **J2 — Assess & plan** project; until it ships in the installed CLI, use only the Git-backed files above.
+
+## Exit criterion
+
+J2 is complete only when the user has explicitly ratified the complete plan and `improvement-plan.md` records that ratification and the resolved Orizu project. The canonical plan must be tracked and committed; in a workbench, its sibling `memory.md` pointer must be tracked and committed together with it. Otherwise remain in assess-and-plan; do not start dataset work.