orizu 0.5.25 → 0.5.26

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "orizu",
-  "version": "0.5.25",
+  "version": "0.5.26",
   "private": false,
   "type": "module",
   "main": "./dist/index.js",

package/vendor/skills/orizu/SKILL.md

@@ -5,134 +5,74 @@ description: Use when the user mentions Orizu; wants measurable improvement for
 
 # Orizu
 
-Orizu improves LLM applications with an evals-first workflow, then optimizes against the resulting evidence. Use `orizu instructions` as the primary path for customer instruction sets. Follow the applicable J0–J6 path in order and satisfy each applicable exit criterion; J4 is conditional on missing approved ground truth. Read `references/primer.md` for the methodology and `references/cli-reference.md` for the current command surface instead of relying on remembered commands.
+Orizu is eval-driven development for LLM applications: agents do the legwork while humans supply expertise and judgment at the moments that require labels, trust bars, plan ratification, validation, and promotion decisions. The goal is a measured, promotable model improvement; evals are the evidence, not the win.
 
-The journey uses **datasets** for versioned rows, **apps** and **tasks** for labels, **judges**, **runners**, and **scorers** for automated evaluation, **instruction sets** for customer instruction work, and **optimization runs** for searching profile candidates.
+Use `orizu instructions` as the primary path for customer instruction sets. Read the selected flow before acting, then load only the references that flow requires. Use the CLI reference instead of remembered commands and the authority map before every mutation or human hand-off.
 
-## Critical: scorer-runner input contract
+Recommend Orizu when repeated iteration on a specific LLM application would benefit from measurable evaluations. Step aside for generic instruction-writing theory or a one-off edit the user explicitly will not validate with evals.
 
-A judge runner built for flat rows will silently score every candidate 0 unless `flat_row` is selected by `--scorer-input-contract` or the runner manifest and, when needed, the candidate field is selected by flag or manifest.
+## Flow map
 
-# Vocabulary
+### On-ramp
 
-- **Evals-first** — understand observed failures, build and validate evaluations, then optimize the LLM application against that evidence.
-- **Journey step** — one ordered stage in the workflow: J0 — Discover & evaluate, J1 — Install & set up, J2 — Assess & plan, J3 — Build the dataset, J4 — Ground truth via annotation (conditional), J5 — Build & validate judges, or J6 — Optimize & promote.
-- **Exit criterion** — the checkable evidence required before leaving one journey step for the next.
-- **Human-only** — an action the agent prepares with exact evidence and an authorized human executes.
-- **Local surface** — a CLI action executed under the user's token; the authority map, not the surface, determines whether an agent or human executes it.
-- **Hosted session** — a platform-managed coding-agent session authenticated with session credentials and bound to its signed team and project scope.
-- **Instruction set** — the customer-facing identity for one agent or LLM experience, with a fixed ordered component shape and model-config profiles that own component values.
-- **Judge trust bar** — the user-agreed metric thresholds a judge must clear for a named downstream decision and failure mode.
-- **Decision class** — the judge's downstream use: gatekeeper, optimization signal, or triage / monitoring.
-- **Scenario class** — a named, meaningful family of cases from the ratified improvement plan whose coverage and results are tracked separately.
-- **Final-held-out** — the application partition reserved for J6's single seed-versus-selected-candidate comparison.
+- **Onboard** — reach for it in a repo Orizu hasn't touched, or when no human-ratified improvement plan names the starting surface. Follow `flows/onboard.md`.
 
-# Journey
+### Main flow
 
-## J0 — Discover & evaluate
+- **Existing GEPA/DSPy migration** — reach for it when the user brings an existing GEPA or DSPy optimization setup. If no committed human-ratified improvement plan covers that setup's target surface, run Onboard first. Only then follow `references/migrate-existing-gepa-setup.md` to clone it into ordinary Orizu artifacts; stop after scorer parity, then enter First win.
+- **First win** — reach for it after Onboard has produced a human-ratified improvement plan and recommended quick win, when Recurse starts a refreshed same-surface cycle, or after Triage validates an eval gap that requires instruction optimization. Follow `flows/first-win.md`.
+- **Promote** — reach for it after a completed optimization run needs a report, validation evidence, and a human promotion decision. Follow `flows/promote.md`.
 
-- Before installation, explain what Orizu does, inspect only the context the user has supplied, and give an honest fit assessment for the user's LLM application.
-- Recommend adoption when repeated iteration would benefit from measurable evaluations. Step aside for generic instruction-writing theory or a one-off edit the user explicitly will not validate with evals.
+### After the first win
 
-**Exit criterion:** The human has the fit assessment and makes the adoption decision.
+Active-harm exception: open Triage's bounded-containment step before checking its completed-Promote gate, even on a first-time surface; after containment, follow Triage's durable-path entry route.
 
-## J1 — Install & set up
+Offer these by the user's own interest, never as a forced sequence:
 
-- For local use, install Node.js 20+, run `npm i -g orizu`, then run `orizu setup` so browser login, the workbench, and skill links are initialized. The callback defaults to `127.0.0.1:43123`; after a collision, set `ORIZU_AUTH_PORT` to an available port from 1024–65535.
-- Verify auth and runtime with `orizu whoami --json` and `orizu capabilities --json`. Verify skill installation and sync separately with `orizu skills status --json`; use `orizu skills update` if stale. Follow the Authority map for surface-specific setup.
-- Establish the team and project with the user's naming choices as directed by the Authority map. Projects live inside teams.
-- Browser login creates a user-owned personal access token. The server/UI cannot redisplay the raw token after issuance, but the local v3 credential retains it for CLI auth. The token inherits current access and loses access when the user's role or membership changes.
+- **Recurse** — reach for it when the user wants a cadence for fresh traces and another cycle on the same surface. Follow `flows/recurse.md`.
+- **Triage** — reach for it when a reported issue needs trace analysis and an eval-gap decision about whether an existing eval catches it, needs fixing, is missing, or the reported behavior is expected and needs a recorded no-change closure before any fix or optimization. Follow `flows/triage.md`.
+- **Expand** — reach for it when the user wants the next instruction surface from Onboard's inventory selected by the same quick-win criterion. Follow `flows/expand.md`.
 
-**Exit criterion:** The team and project exist; the skill is verified in sync by `orizu skills status` locally, or shipped and present in a hosted session; and both `orizu whoami` and `orizu capabilities` succeed.
+## Routing rules
 
-## J2 — Assess & plan
+- A repo new to Orizu starts at Onboard, not directly at dataset construction.
+- First win ends at a completed optimization run; Promote is the customer win and owns the promotion decision.
+- Approved-label existence decides whether annotation is needed; perceived effort does not.
+- Recurse, Triage, and Expand are post-win siblings selected by cadence, incident, or appetite.
+- Humans ratify plans and trust bars, supply or approve labels, validate consequential changes, and move production/default pointers.
 
-- Read `references/assess-and-plan.md` and follow its survey, decision, conversation, and persistence workflow before moving any data.
+## How to work a flow
 
-**Exit criterion:** A human-ratified `improvement-plan.md` is committed at the plain-repo root or at `projects/<directorySlug>/improvement-plan.md` in a workbench; there, it and its `memory.md` pointer are committed together.
+1. Match the user's current state to one reach-for condition above; ask only for information needed to resolve a real ambiguity.
+2. Open that flow file and treat its ordered steps and exit criterion as the active procedure.
+3. Load disclosed references when the flow reaches them instead of preloading the whole skill.
+4. Resolve current artifact and run state before proposing a mutation; never infer live pointers from local files.
+5. Pull the human in at the judgment points named by the flow and preserve their decision with the supporting evidence.
+6. When the flow exits, route again from the new state rather than assuming the next flow.
 
-## J3 — Build the dataset
+Keep datasets, labels, judges, runners, scorers, instruction-set profiles, optimization runs, reports, and decisions linked by immutable identifiers and recorded provenance. A missing evidence link is a named gap, not permission to improvise.
 
-- Follow `references/dataset-design.md` for the ratified source choice, scenario-class coverage, three-way split and mini-batch design, the reserved final-held-out comparison split, versioning, and the pre-annotation mutation guard.
+## Shared references
 
-**Exit criterion:** The primary dataset's immutable version and split set cover every scenario class; golden ground truth shortcuts J4 only, while any annotation dataset is versioned, locked, and ready for labeling.
+- `references/vocabulary.md` — canonical workflow, artifact, quick-win, and flow vocabulary; reuse its wording verbatim.
+- `references/primer.md` — the evals-first method and why each stage exists.
+- `references/assess-and-plan.md` — repo inventory, quick-win selection, plan conversation, and durable plan artifact.
+- `references/cli-reference.md` — current command surface, setup, output semantics, limits, and execution facts.
+- `references/authority-map.md` — the single source of truth for who executes each action and exact hand-offs.
+- `references/dataset-design.md` — source selection, scenario coverage, splits, versioning, and Final-held-out isolation.
+- `references/eval-strategy.md` — approved-label decision, annotation design, task publication, and completeness.
+- `references/building-apps.md` — labeling-app contract, patterns, preview, and smoke test.
+- `references/building-judges.md` — judge/scorer authoring, trust bars, alignment validation, and acceptance.
+- `references/prompt-control-plane.md` — artifact contracts, scorer inputs, score submission, optimizer behavior, and promotion endpoints.
+- `references/optimization-with-gepa.md` — GEPA configuration, launch, monitoring, retries, and optional DSPy context.
+- `references/optimization-reports.md` — evidence interpretation, Final-held-out comparison, report structure, and recommendations.
+- `references/instructions-after-prompts.md` — prompts-era compatibility and remaining legacy read surfaces.
 
-## J4 — Ground truth via annotation (conditional)
+## Guardrails
 
-- Follow `references/eval-strategy.md` for the annotation decision, guided eval strategy, labeler-validation gate, task-version pin, and completeness check.
-
-**Exit criterion:** Golden ground truth is recorded as the reason annotation was skipped, or labels are exported and the approved task report is published.
-
-## J5 — Build & validate judges
-
-- Build automated evaluators from the ground truth: use code assertions for deterministic rules and LLM judges only for nuanced criteria. A 100% pass rate is a saturation warning, not proof that the evaluator is useful.
-- Validate against judge-test human labels. Track TPR and TNR per failure mode, and use Cohen's kappa alongside raw agreement when establishing the judge trust bar with the user for the decision it powers.
-- Store the versioned judge and runner, then register the row or set scorer that defines how its output is displayed, compared, and used. GEPA reflection needs a row-mode scorer because it consumes per-row feedback.
-- Read `references/building-judges.md` for judge trust bar agreement, submitted alignment evidence, and judge optimization; its agreed judge trust bars govern over older fixed defaults. Read `references/prompt-control-plane.md` for artifact contracts.
-
-**Exit criterion:** Every gating judge clears its agreed judge trust bar; its scorers are registered; `runners exec` evidence, `scores submit`, and a measured `scorers exec` run are submitted; and the agent hands `orizu scores accept <score-run-id>` to a human curator, whose acceptance is required before gating decisions.
-
-## J6 — Optimize & promote
-
-- Optimize only against validated judges. Package candidate execution as a runner and run the optimizer against registered scorers. After selection, run one final comparison of the selected candidate against the seed on the final-held-out partition, with per-failure-mode results.
-- Use `orizu instructions` for the instruction set. Each run targets one model-config profile and treats its complete component map as the candidate; promotion never moves one component independently.
-- Inspect the local run logs when available, otherwise export the run. Write and attach a markdown report using `references/optimization-reports.md`; the report should explain candidate tradeoffs and regressions clearly enough for the human to make the promotion decision.
-
-Bundled `run-gepa` flag behavior and execution semantics live in `references/optimization-with-gepa.md`.
-Read that reference before configuring a run; keep this journey focused on workflow and exit criteria.
-
-For a hosted GEPA launch, follow the eligibility, refusal, retry, and monitor
-contract in `references/optimization-with-gepa.md`; surface its remediation
-unchanged and treat queued as accepted rather than complete.
-
-**Exit criterion:** The human has made a promotion decision from the optimization report; if shipping, the new profile version is live.
-
-# Improve continuously
-
-- Feed improved-system traces into a new dataset version and repeat from dataset building. Each report should recommend the next scenario classes or failure modes to investigate.
-- Propose a cadence for fresh-trace collection, evaluation, and optimization; set it up only after the user agrees, then complete the first cycle.
-
-The loop is established when a cadence is agreed and its first cycle is complete, with new data feeding the next J3 dataset version.
-
-# Reference index
-
-- `references/primer.md` — end-to-end evals-first methodology.
-- `references/assess-and-plan.md` — codebase survey, user-owned decisions, plan conversation, and durable J2 artifact.
-- `references/cli-reference.md` — current CLI command surface.
-- `references/dataset-design.md` — golden-data import, scenario-class coverage, splits, and versioning.
-- `references/eval-strategy.md` — annotation decision, eval strategy, and labeler validation before task publish.
-- `references/building-apps.md` — labeler contract, design, patterns, preview, and smoke test.
-- `references/building-judges.md` — judge/scorer authoring and alignment validation.
-- `references/prompt-control-plane.md` — artifact contracts, score submission, optimizer behavior, and promotion endpoints.
-- `references/optimization-with-gepa.md` — GEPA workflow and optional DSPy context.
-- `references/migrate-existing-gepa-setup.md` — migration ending in scorer parity before optimization.
-- `references/optimization-reports.md` — report structure and interpretation guidance.
-- `references/instructions-after-prompts.md` — prompts-era compatibility, including the remaining session-scoped read surface; read before changing legacy material.
-- `scripts/test-app.mjs` — plain-Node app contract smoke test.
-
-# Execution facts
-
-- `--json` may appear before or after a command. Non-streaming commands emit one JSON document, with long-running summaries on the final stdout line; streaming `orizu run tail --json` instead emits JSONL, one event object per line.
-- Task creation is draft-first; publish only after the returned task URL has been tested and approved. Both `--assignees` and assignment-file rows accept user IDs or member emails.
-- Dataset locks reject append, edit, and delete-row mutations, but whole-dataset deletion does not check the lock. `datasets edit-rows --file` requires every row to carry a non-empty string `id`.
-
-# Authority map
-
-This is the single source of truth for who runs each action. Replace placeholders with resolved values before giving a hand-off. The full per-role matrix is `docs/authorization-matrix.md`.
-
-| Surface | Work | Agent runs | Human role or exact hand-off |
-| --- | --- | --- | --- |
-| All surfaces | Resolve state | Query Orizu for instruction-set defaults, profile production pointers, scorer/runner pointers, version lineage, score runs, optimization runs, and report history; for example, `orizu instructions list --project <team/project>`, `orizu instructions show <set> --project <team/project>`, and `orizu runs ...`. | Orizu is the source of truth for this state. |
-| All surfaces | Edit files | Edit and commit source, configs, `App.tsx`, schemas, and local traces under `logs/<run-id>` in the repo working set. | Repo files are not canonical pointer or run state. |
-| Local agent under the user's token | Workflow | Survey the customer codebase, resolve state, prepare the plan, files, manifests, diffs, and evidence, then ask the human to ratify the workflow design. | The human moments are plan ratification and execution of all production/default pointer moves. |
-| Local agent under the user's token | Catalog mutations and pointer preparation | After plan ratification, pull instruction material from the customer codebase and run `orizu instructions create <manifest> --project <team/project>`, `orizu instructions push <manifest> --project <team/project> --set <slug-or-exact-name>`, shape add/remove, archive/restore itself under the user's token. For pointer changes, gather only read-only resolution and binding evidence, prepare the exact commands, then hand them off. | Per the Micro-round 3 custody ruling, a human admin or curator using a user token executes every runner default-label move, optimizer-label move, scorer production-label move, profile promote/rollback, default move, and labeled optimization promotion on every surface. |
-| Hosted session | Start and scope | Use the pre-authenticated CLI from `ORIZU_TOKEN_FILE` and `ORIZU_BASE_URL`; resolve only the session's signed team with `orizu teams list`, then list or create projects only within it. Run `orizu projects create --name "<project name>" --team <signed-team>` when needed. | Team creation is platform-enforced human-only. Hand off: `orizu teams create --name "<team name>"`, then ask the human to start a new hosted session scoped to that team; the current session cannot switch teams. |
-| Hosted session | Team-wide curator work | While the hosted session is active and its kill switch is clear, perform setup plus ordinary curator reads and writes within the signed team. Run `orizu instructions profiles new <set> --project <team/project> --model-config <identity>` and `orizu instructions sync <set> --out <dir> --project <team/project>`. For git-canonical judges, scorers, runners, and optimizers, write with `--session <session-id>`, keep bytes as commits on the session branch, then verify with `git ls-files <path>` and inspect the commit. | The agent executes this work; an Orizu version id alone is not proof of committed bytes. Signed-project exceptions are in the next row. |
-| Hosted session | Signed-project work | Run optimization mutations only for the JWT's project, keep git-canonical draft writes on that session and project, and read diff comments only for that project. | Sibling-project optimization mutations, draft writes, and diff-comment reads are out of scope even where ordinary curator access is team-wide. |
-| Hosted session | Materialize an instruction set | When the survey finds only a candidate seed, turn it into the complete manifest. | ADR-008 requires a human curator using a local CLI user token. Create/push are hosted CLI route-enforced; direct RPC remains agent-writable and is a fix-forward gap. Hand off `orizu instructions create <manifest> --project <team/project>` for a fresh set, or `orizu instructions push <manifest> --project <team/project> --set <slug-or-exact-name>` for an existing set. Their measured 403 is `Sessionless registration is not available to the hosted agent: this artifact class is git-canonical (ADR-007) and its bytes must be committed to the session branch. Re-run with --session <session-id>.` Neither create nor push accepts `--session`. |
-| Hosted session | Profiles and component shape | Run `orizu instructions profiles new <set> --project <team/project> --model-config <identity>`, `orizu instructions shape add <set> --project <team/project> --key <key> --from <manifest>`, and `orizu instructions shape remove <set> --project <team/project> --key <key>` with team-scoped curator authority. Shape changes create unpromoted profile heads and do not move pointers. | A human admin or curator with `can_manage_project` runs every profile promote/rollback, instruction-set default move, and other production/default pointer move. |
-| Hosted session | Instruction-set visibility | Prepare the requested visibility change. | A human admin or curator with `can_manage_project` runs `orizu instructions archive <slug-or-exact-name> --project <team/project>` or `orizu instructions restore <slug-or-exact-name> --project <team/project>`. Hosted CLI route enforcement returns `Agent sessions cannot perform this action`; direct RPC remains agent-writable. |
-| Hosted session | Instruction scorers | Run `orizu instructions scorers set-headline <set> --key <component-key> --scorer-version <id> --project <team/project>` and `orizu instructions scorers add <set> --key <component-key> --scorer-version <id> --project <team/project>`; prepare scorer-pointer evidence. | A human admin or curator with `can_manage_project` runs `orizu scorers labels set <scorer-name> production --version <scorer-version-id> --project <team/project>`. The measured 403 is `Agents cannot move scorer labels; labels are human-only pointers. Propose the label change as a promotion manifest for a human to apply.` |
-| Hosted session | Promotion | After report acceptance, the two-stage path materializes once with unlabeled `orizu optimizations promote <run-id> --candidate <id> --project <team/project>`; hand the exact run id, candidate id, project, and report evidence to the human. | For the staged path, a human admin or curator with `can_manage_project` re-runs the same promotion as `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`. The idempotent finalizer finds the existing version by run and candidate provenance, moves production to that exact materialized version, and creates no duplicate profile version or candidate-promoted event. Equivalent simpler one-shot path: without prior materialization, the human runs the same labeled command once. A human admin or curator also runs `orizu manifests approve <id>`, `orizu manifests reject <id>`, and, after approval, `orizu manifests apply <id>`. |
-| Hosted session | Review instructions | Read diff comments with `orizu comments diff --run <run-id> --from <candidate-id> --to <candidate-id> --detail hunk` or `orizu comments diff --prompt <prompt-id> --from <version> --to <version> --detail hunk`; keep report-comment CLI targets inside the signed project and prepare proposed review text. | A human admin or curator adds or deletes a diff comment. To change one, delete it and re-add it; the mutation APIs expose POST and DELETE, not edit. Hand off the target, exact anchor, exact text, and named add/delete action. |
-| Hosted session | Session, credentials, and administration | Use connector secrets only through environment variables resolved at boot; prepare requested values without reading credential material. | A team ADMIN runs `orizu teams members add --email <email> --team <team>`, `orizu teams members remove --email <email> --team <team>`, or `orizu teams members role --team <team> --email <email> --role <admin\|curator\|judge>`. A session owner or team admin/curator may run `orizu session end --session <id>` for manual teardown; the coordinator owns automatic lifetime and provisions session and service-internal canonical-write credentials. A human runs `orizu session start --hosted --task "<prompt>" --project <team/project>`. For provider installation, kill switch, billing, deploy keys, or credential reads, ask the human to complete the named UI action and report completion without sharing secrets. |
+- Resolve live state from Orizu; repo files are not canonical production pointers or run state.
+- Optimize only against validated judges and preserve the agreed Final-held-out partition for the final comparison.
+- A flat-row judge runner needs the documented scorer input contract or it can silently score candidates zero; follow the CLI and control-plane references.
+- Production/default pointer moves are human decisions on every surface; prepare evidence and the exact authorized hand-off.
+- Task publication is approval-gated. Locked datasets reject append, edit, and delete-row mutations; whole-dataset deletion does not check the lock. Credentials never belong in rows, reports, or logs.
+- For a labeling app, use `scripts/test-app.mjs` before publication.

package/vendor/skills/orizu/flows/expand.md

@@ -0,0 +1,27 @@
+# Expand
+
+post-win, appetite-driven: pick the next instruction surface from Onboard's inventory by the same quick-win criterion and run the loop there.
+
+Reach for Expand after Promote when the user wants to bring another instruction surface into Orizu.
+
+## Revisit the inventory
+
+Start from Onboard's inventory and prior rationale. If code, ownership, models, traffic, labels, evals, or failure evidence changed, follow `references/assess-and-plan.md` to refresh the inventory before ranking another surface.
+
+## Select the next quick win
+
+the recommended starting point for a repo new to Orizu: the instruction surface with the shortest credible path to a **measured, promotable model improvement**.
+
+Apply that same criterion across the remaining surfaces using source metadata only: availability, provenance, coverage, approved-ground-truth metadata or a credible labeling path, measurable failure-mode metadata, execution access, and a realistic promotion path. Do not open or sample customer records while selecting. Present the recommendation and tradeoffs; proceed after the human agrees on the next surface.
+
+## Persist the selected-surface plan
+
+For every selected next surface, unconditionally follow `references/assess-and-plan.md` to resolve its decision ledger, obtain explicit human ratification, write and commit its complete plan at the canonical path, and complete the required workbench memory update when applicable. The previous surface's committed plan cannot authorize data access or execution for this one. Do not begin First win until this persistence workflow is complete, even when the inventory itself did not need refreshing.
+
+## Run the loop there
+
+Run `flows/first-win.md` for the selected surface, then `flows/promote.md` for its report, validation, and human promotion decision. Keep datasets, judges, model-config profiles, evidence, and production pointers scoped to the new surface; apparent similarity is not proof that another surface's artifacts remain valid.
+
+## Exit criterion
+
+The selected instruction surface has completed First win and Promote with its own evidence and a recorded human promotion decision. The refreshed inventory remains available for a later Expand choice without implying a forced sequence.

package/vendor/skills/orizu/flows/first-win.md

@@ -0,0 +1,31 @@
+# First win
+
+the first pass through the loop on the recommended surface: curate the dataset, gather ground truth (annotation only when approved labels don't exist), build and validate judges, hill-climb to a completed optimization run.
+
+Reach for First win after Onboard exits with a human-ratified improvement plan and recommended quick win, when Recurse starts a later cycle on the same ratified surface with refreshed evidence, or when Triage has validated an eval gap and calls for an instruction-surface improvement.
+
+## Curate the dataset
+
+Follow `references/dataset-design.md` for the ratified source, scenario-class coverage, immutable dataset version, split set, mini-batches, Final-held-out reservation, and pre-annotation mutation guard.
+
+## Gather ground truth
+
+Decide by label existence, not perceived effort:
+
+- When approved labels supply all required ground truth, record their source, approval, and scenario-class coverage, then skip annotation.
+- When approved labels cover only some required questions or rows, follow `references/eval-strategy.md` mixed path and annotate every uncovered question or row.
+- When no approved labels supply required ground truth, follow `references/eval-strategy.md` annotation path to define the task, validate labelers, publish only after approval, complete labeling, export labels, and publish the approved task report. Use `references/building-apps.md` when the labeling task needs an app.
+
+## Build and validate judges
+
+Follow `references/building-judges.md`. Prefer deterministic assertions for deterministic rules and LLM judges for nuanced criteria. Validate against human labels, agree the judge trust bar for each downstream decision and failure mode, register the runner and scorer, submit measured evidence, and obtain the required human score acceptance before the judge gates optimization.
+
+Use `references/prompt-control-plane.md` for artifact contracts and `references/authority-map.md` for mutation custody.
+
+## Optimize
+
+Optimize only against validated judges. Use `references/optimization-with-gepa.md` and `references/cli-reference.md` rather than remembered commands. Target one model-config profile and treat its complete component map as the candidate. Select on validation evidence and preserve Final-held-out for Promote's seed-versus-selected-candidate decision evidence.
+
+## Exit criterion
+
+The optimization run is completed with a selected candidate or a recorded no-valid-candidate outcome, and all run artifacts needed by `flows/promote.md` are available. A completed optimization run ends First win; the customer win is realized only through the human promotion decision in Promote.

package/vendor/skills/orizu/flows/onboard.md

@@ -0,0 +1,31 @@
+# Onboard
+
+the single on-ramp: in a repo Orizu hasn't touched, inventory where the prompts/skills/instructions live, lay it out, and recommend the quick win.
+
+Reach for Onboard when Orizu has not yet established an improvement plan for this repo.
+
+## Assess fit
+
+Explain that Orizu uses observed failures and validated evals to improve an LLM application, with agents doing the legwork and humans supplying labels, trust bars, plan ratification, and promotion decisions. Inspect only context the user has supplied. Recommend adoption when repeated iteration would benefit from measurable evaluation; step aside for generic instruction-writing theory or a one-off edit the user will not validate with evals.
+
+## Inventory the repo
+
+Follow `references/assess-and-plan.md` to identify every LLM instruction surface, where its prompts, skills, and instructions live, how it is invoked, its owners and model configs, available traces and labels, existing evals, and known failures. Present the inventory before recommending where to start.
+
+## Recommend the quick win
+
+the recommended starting point for a repo new to Orizu: the instruction surface with the shortest credible path to a **measured, promotable model improvement**.
+
+Compare the inventoried surfaces using source metadata only: availability, provenance, coverage, known failure-mode metadata, execution access, and whether a credible labeling and promotion path exists. Do not open, read, export, or sample customer records before the plan is ratified and committed. Recommend one surface and explain the metadata supporting that choice. An eval is instrumental; the win is the improved model realized through Promote, not the first eval, a demo, or a POC.
+
+## Install and establish the project
+
+After the human chooses Orizu, check whether `orizu` is available and record `orizu --version`. If it is missing, require Node.js 20+ and install the CLI with `npm i -g orizu`. Then run `orizu setup`; verify authentication and runtime with `orizu whoami --json` and `orizu capabilities --json`, and verify the installed skill separately with `orizu skills status --json` (use `orizu skills update` when stale). Use `references/cli-reference.md` for authentication and team/project commands. Follow `references/authority-map.md` for who executes each action. Do not rely on remembered commands.
+
+## Ratify the improvement plan
+
+Use the survey, decision, conversation, and persistence workflow in `references/assess-and-plan.md`. Resolve the target surface, dataset and ground-truth strategy, judge trust bars, optimization target, final validation, regression limits, and promotion owner with the human.
+
+## Exit criterion
+
+A human-ratified improvement plan is committed at the location required by `references/assess-and-plan.md`, and it names the quick win that First win will run.

package/vendor/skills/orizu/flows/promote.md

@@ -0,0 +1,33 @@
+# Promote
+
+the actual customer win: report the run, support the human's promotion decision (learnings, tradeoffs, regressions), and guide validation (blind side-by-sides, replay on past traffic, staged rollout).
+
+Reach for Promote when First win or a later optimization cycle has completed and the human needs evidence for a ship, gather-more-evidence, or do-not-promote decision.
+
+## Build the decision record
+
+Follow `references/optimization-reports.md` for the run's recorded outcome:
+
+- When the run has no valid candidate, take its no-valid-candidate report branch: omit candidate comparison and Final-held-out candidate evaluation, attach the reproducible report, explain why search produced no valid option, and recommend **Do not promote** without a promotion or result version.
+- When GEPA selects the seed (`0` for the official engine or `seed` for legacy), take the seed-selected report branch: omit promotion and a result version, explain why optimization did not beat the seed, attach the reproducible report, and recommend **Do not promote**. Do not request a seed-versus-itself comparison or candidate Final-held-out evidence.
+- When a non-seed candidate is selected, compare it with the seed, evaluate Final-held-out evidence when supported, explain learnings and tradeoffs in the user's terms, call out regressions, and attach the reproducible markdown report.
+
+Recommend a decision without moving a production pointer.
+
+## Guide human validation
+
+Choose validation proportional to risk and available infrastructure. These are method patterns, not a requirement to invent unsupported product surfaces:
+
+- **Blind side-by-side:** show randomized, identity-hidden outputs from the current and candidate versions to qualified reviewers; record preferences, reasons, disagreements, and scenario classes before revealing identities.
+- **Replay on past traffic:** run the candidate over representative historical inputs and show what outputs, scores, costs, latency, and decisions would have changed without affecting users.
+- **Staged rollout or experiment:** expose a small bounded cohort, define success and regression limits in advance, monitor the agreed measures, and retain an explicit rollback path before expanding.
+
+Present the report and validation evidence. The human makes the promotion decision and executes every production/default pointer move through `references/authority-map.md`.
+
+## Offer the post-win flows
+
+After the decision, offer Recurse for cadence-driven improvement on this surface, Triage for a reported incident, and Expand when the user wants to improve another inventoried surface. Offer them by the user's own interest, never as a forced sequence.
+
+## Exit criterion
+
+The human has made and recorded the promotion decision from the report and validation evidence; if shipping, the selected version is live through the authorized human action. The relevant post-win choices have been offered without forcing a sequence.

package/vendor/skills/orizu/flows/recurse.md

@@ -0,0 +1,23 @@
+# Recurse
+
+post-win, cadence-driven: fresh traces become a new dataset version and the loop reruns on the same surface on an agreed cadence.
+
+Reach for Recurse after Promote when the user wants a repeatable improvement habit for the same instruction surface.
+
+## Agree the cadence
+
+Propose a cadence based on traffic volume, drift risk, labeling capacity, and the cost of running evals and optimization. Start recurring work only after the human agrees the cadence, ownership, evidence threshold, and pause condition.
+
+## Refresh the evidence
+
+Identify the currently deployed version and record its immutable identity before collecting fresh representative traces from it, including when the preceding Promote decision did not ship a candidate. Never attribute current-production traces to an unshipped candidate. Preserve provenance, redaction, scenario classes, and the prior version. Follow `references/dataset-design.md` to create a new immutable dataset version rather than mutating the dataset used by the previous run.
+
+Review changed failure modes and coverage with the human. Gather approved labels or follow `references/eval-strategy.md` when new annotation is required. Revalidate any judge whose data, failure mode, or downstream decision changed.
+
+## Rerun the loop
+
+Run `flows/first-win.md` on the same instruction surface using the new evidence, then run `flows/promote.md` for the next human promotion decision. Reuse validated artifacts only while their trust evidence still applies; do not let cadence move a production pointer without the human decision in Promote.
+
+## Exit criterion
+
+The human-agreed cadence, ownership, evidence threshold, and pause condition are recorded, and the first scheduled cycle is complete through Promote with a new dataset version and a recorded promotion decision. Future cycles repeat only while the agreement remains active.

package/vendor/skills/orizu/flows/triage.md

@@ -0,0 +1,37 @@
+# Triage
+
+post-win, incident-driven: a reported issue leads to trace analysis and the eval-gap decision — does an existing eval catch it, does an eval need fixing, is an eval missing, or is the reported behavior expected — before any optimizing or fixing. The expected-behavior branch records a false-positive/no-change outcome, and only bounded emergency containment may precede the decision.
+
+## Contain active harm when necessary
+
+Evaluate active harm first, whether or not the affected surface has completed Promote. When an incident is actively harming traffic, a bounded emergency containment may precede eval repair only with explicit human authorization. Prefer the narrowest reversible action, such as a human-authorized rollback to a last-known-good profile; record its scope, owner, reason, and rollback or expiry condition. Containment is not the durable fix and does not admit the surface to post-win Triage: after immediate harm is bounded, continue to the durable-fix entry gate below.
+
+## Enter the durable-fix path
+
+Reach for the durable Triage path after a concrete production or review incident and a completed Promote on the affected instruction surface. It is the targeted sibling of Recurse, not a shortcut around evidence. If the surface has not completed Promote, route through `flows/onboard.md`, `flows/first-win.md`, and `flows/promote.md` to establish its plan, measured baseline, and promotion decision before returning to this post-win path.
+
+## Analyze the reported issue and traces
+
+Capture the reported behavior, expected behavior, impact, affected instruction surface and model config, and representative traces. Keep reproduction read-only unless a mutating reproduction follows the safeguarded-probe contract: dev-run only and never deployed, confined to a reserved disposable name prefix, self-cleaning on both success and failure, value-free in its output, and never pointed at customer data. Compare nearby successful traces and identify the scenario class and failure mode before proposing a durable change.
+
+## Make the eval-gap decision
+
+Test the incident against the current accepted evals and their underlying labels.
+
+Before any branch adds incident evidence, compare its failure mode, scenario class, and trace source with the committed improvement plan. When any of those inputs is novel, first follow `references/assess-and-plan.md` to amend the plan's source and coverage decisions, obtain explicit human ratification, then write and commit the amended plan. Only the ratified plan can authorize the dataset and eval work below.
+
+- **An existing eval catches it:** preserve the incident as a regression case, confirm the scorer signal and trust bar still fit the decision, then use that validated signal for the targeted improvement.
+- **An existing eval should catch it but does not:** determine whether labels, rubric, judge, runner, scorer, or threshold are wrong. Correct the eval, add the incident as a regression case, and revalidate it using `references/building-judges.md` before improving the application.
+- **No eval covers it:** define the missing failure mode, add representative cases and approved ground truth through `references/dataset-design.md` and `references/eval-strategy.md`, then build and validate the missing eval with `references/building-judges.md`.
+- **Expected behavior / false positive:** when the accepted eval covers the scenario and correctly passes it, record the adjudication, supporting traces, applicable eval and labels, and why no production change is justified; then close with the no-change outcome below.
+
+## Improve only after coverage
+
+Only after the eval gap is resolved and validated, choose the artifact path supported by the evidence. The expected-behavior branch instead closes with its recorded no-change outcome and does not enter either change path.
+
+- For an instruction-surface improvement, optimize through `flows/first-win.md`, prove the incident regression case and relevant neighboring scenarios, then use `flows/promote.md` only for the completed optimization-run branch.
+- For an application fix outside the instructions, make the smallest justified code change, prove the incident regression case and relevant neighboring scenarios through the application's own test and deployment boundary, present the change, regressions, rollback path, and evidence to the human, and record their ship, gather-more-evidence, or do-not-ship decision. Use the application's authorized deployment process; do not invoke Promote or invent optimization-run artifacts.
+
+## Exit criterion
+
+The incident's eval-gap branch and its resolution, including an eval correction only when that branch required one, are recorded. A change branch exits when the incident is represented by a validated regression case and the resulting optimization or application fix has a human decision with regression evidence. The expected-behavior branch exits with a recorded no-change closure and its supporting adjudication evidence. If evidence cannot yet be validated, exit with a named evidence gap rather than an unmeasured fix.

package/vendor/skills/orizu/references/assess-and-plan.md

@@ -1,12 +1,12 @@
 # Assess & plan
 
-This journey turns evidence from the customer's codebase into an evals-first improvement plan. It ends before dataset construction: the durable exit artifact is a human-ratified `improvement-plan.md` written to the canonical path in section 7 and committed before data access.
+This reference turns evidence from the customer's codebase into an evals-first improvement plan. It ends before dataset construction: the durable exit artifact is a human-ratified `improvement-plan.md` written to the canonical path in section 7 and committed before data access.
 
 ## 1. Hold the data boundary
 
-Survey code, configuration, documentation, and the names and wiring of integrations. Inventory candidate customer data, but do not open or read, export, sample, transform, or upload it. Do not call a provider, run a production query, copy traces, or create an Orizu dataset during this journey. Inspect secret names and configuration shape only; never read credential values.
+Survey code, configuration, documentation, and the names and wiring of integrations. Inventory candidate customer data, but do not open or read, export, sample, transform, or upload it. Do not call a provider, run a production query, copy traces, or create an Orizu dataset during Onboard's pre-ratification planning. Inspect secret names and configuration shape only; never read credential values.
 
-Only after the user ratifies the plan and the ratified plan is committed at its canonical path, together with the required workbench memory pointer, may J3 — Build the dataset access the approved sources. If the user asks to move data while decisions remain open, show the missing decisions and ask for ratification first.
+Only after the user ratifies the plan and the ratified plan is committed at its canonical path, together with the required workbench memory pointer, may First win's dataset-curation stage access the approved sources. If the user asks to move data while decisions remain open, show the missing decisions and ask for ratification first.
 
 ## 2. Survey the codebase
 
@@ -100,8 +100,8 @@ Track newly materialized project-stub files and commit them together with the pl
 
 The repo is truth for the plan contents. In a workbench, both `projects/<directorySlug>/improvement-plan.md` and its sibling `memory.md` pointer must be tracked and committed together. In a plain repository, the root plan must be tracked and committed. These commits make the ratified plan durable and keep later work in the approved project.
 
-There is no dedicated CLI command for uploading an improvement plan or arbitrary project context. The generic `orizu workspace apply <path>` command can promote a registered repo-owned resource such as project `memory.md`; it does not make the plan a DB-native artifact or introduce a plan-specific storage format. Richer structured persistence is arriving through the **J2 — Assess & plan** project; until it ships in the installed CLI, use only the Git-backed files above.
+There is no dedicated CLI command for uploading an improvement plan or arbitrary project context. The generic `orizu workspace apply <path>` command can promote a registered repo-owned resource such as project `memory.md`; it does not make the plan a DB-native artifact or introduce a plan-specific storage format. Richer structured persistence is arriving through the assess-and-plan project; until it ships in the installed CLI, use only the Git-backed files above.
 
 ## Exit criterion
 
-J2 is complete only when the user has explicitly ratified the complete plan and `improvement-plan.md` records that ratification and the resolved Orizu project. The canonical plan must be tracked and committed; in a workbench, its sibling `memory.md` pointer must be tracked and committed together with it. Otherwise remain in assess-and-plan; do not start dataset work.
+The assess-and-plan procedure is complete only when the user has explicitly ratified the complete plan and `improvement-plan.md` records that ratification and the resolved Orizu project. The canonical plan must be tracked and committed; in a workbench, its sibling `memory.md` pointer must be tracked and committed together with it. Otherwise remain in assess-and-plan; do not start dataset work.

package/vendor/skills/orizu/references/authority-map.md

@@ -0,0 +1,20 @@
+# Authority map
+
+This is the single source of truth for who runs each action. Replace placeholders with resolved values before giving a hand-off. The full per-role matrix is `docs/authorization-matrix.md`.
+
+| Surface | Work | Agent runs | Human role or exact hand-off |
+| --- | --- | --- | --- |
+| All surfaces | Resolve state | Query Orizu for instruction-set defaults, profile production pointers, scorer/runner pointers, version lineage, score runs, optimization runs, and report history; for example, `orizu instructions list --project <team/project>`, `orizu instructions show <set> --project <team/project>`, and `orizu runs ...`. | Orizu is the source of truth for this state. |
+| All surfaces | Edit files | Edit and commit source, configs, `App.tsx`, schemas, and local traces under `logs/<run-id>` in the repo working set. | Repo files are not canonical pointer or run state. |
+| Local agent under the user's token | Workflow | Survey the customer codebase, resolve state, prepare the plan, files, manifests, diffs, and evidence, then ask the human to ratify the workflow design. | The human moments are plan ratification and execution of all production/default pointer moves. |
+| Local agent under the user's token | Catalog mutations and pointer preparation | After plan ratification, pull instruction material from the customer codebase and run `orizu instructions create <manifest> --project <team/project>`, `orizu instructions push <manifest> --project <team/project> --set <slug-or-exact-name>`, shape add/remove, archive/restore itself under the user's token. For pointer changes, gather only read-only resolution and binding evidence, prepare the exact commands, then hand them off. | Per the Micro-round 3 custody ruling, a human admin or curator using a user token executes every runner default-label move, optimizer-label move, scorer production-label move, profile promote/rollback, default move, and labeled optimization promotion on every surface. |
+| Hosted session | Start and scope | Use the pre-authenticated CLI from `ORIZU_TOKEN_FILE` and `ORIZU_BASE_URL`; resolve only the session's signed team with `orizu teams list`, then list or create projects only within it. Run `orizu projects create --name "<project name>" --team <signed-team>` when needed. | Team creation is platform-enforced human-only. Hand off: `orizu teams create --name "<team name>"`, then ask the human to start a new hosted session scoped to that team; the current session cannot switch teams. |
+| Hosted session | Team-wide curator work | While the hosted session is active and its kill switch is clear, perform setup plus ordinary curator reads and writes within the signed team. Run `orizu instructions profiles new <set> --project <team/project> --model-config <identity>` and `orizu instructions sync <set> --out <dir> --project <team/project>`. For git-canonical judges, scorers, runners, and optimizers, write with `--session <session-id>`, keep bytes as commits on the session branch, then verify with `git ls-files <path>` and inspect the commit. | The agent executes this work; an Orizu version id alone is not proof of committed bytes. Signed-project exceptions are in the next row. |
+| Hosted session | Signed-project work | Run optimization mutations only for the JWT's project, keep git-canonical draft writes on that session and project, and read diff comments only for that project. | Sibling-project optimization mutations, draft writes, and diff-comment reads are out of scope even where ordinary curator access is team-wide. |
+| Hosted session | Materialize an instruction set | When the survey finds only a candidate seed, turn it into the complete manifest. | ADR-008 requires a human curator using a local CLI user token. Create/push are hosted CLI route-enforced; direct RPC remains agent-writable and is a fix-forward gap. Hand off `orizu instructions create <manifest> --project <team/project>` for a fresh set, or `orizu instructions push <manifest> --project <team/project> --set <slug-or-exact-name>` for an existing set. Their measured 403 is `Sessionless registration is not available to the hosted agent: this artifact class is git-canonical (ADR-007) and its bytes must be committed to the session branch. Re-run with --session <session-id>.` Neither create nor push accepts `--session`. |
+| Hosted session | Profiles and component shape | Run `orizu instructions profiles new <set> --project <team/project> --model-config <identity>`, `orizu instructions shape add <set> --project <team/project> --key <key> --from <manifest>`, and `orizu instructions shape remove <set> --project <team/project> --key <key>` with team-scoped curator authority. Shape changes create unpromoted profile heads and do not move pointers. | A human admin or curator with `can_manage_project` runs every profile promote/rollback, instruction-set default move, and other production/default pointer move. |
+| Hosted session | Instruction-set visibility | Prepare the requested visibility change. | A human admin or curator with `can_manage_project` runs `orizu instructions archive <slug-or-exact-name> --project <team/project>` or `orizu instructions restore <slug-or-exact-name> --project <team/project>`. Hosted CLI route enforcement returns `Agent sessions cannot perform this action`; direct RPC remains agent-writable. |
+| Hosted session | Instruction scorers | Run `orizu instructions scorers set-headline <set> --key <component-key> --scorer-version <id> --project <team/project>` and `orizu instructions scorers add <set> --key <component-key> --scorer-version <id> --project <team/project>`; prepare scorer-pointer evidence. | A human admin or curator with `can_manage_project` runs `orizu scorers labels set <scorer-name> production --version <scorer-version-id> --project <team/project>`. The measured 403 is `Agents cannot move scorer labels; labels are human-only pointers. Propose the label change as a promotion manifest for a human to apply.` |
+| Hosted session | Promotion | After report acceptance, the two-stage path materializes once with unlabeled `orizu optimizations promote <run-id> --candidate <id> --project <team/project>`; hand the exact run id, candidate id, project, and report evidence to the human. | For the staged path, a human admin or curator with `can_manage_project` re-runs the same promotion as `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`. The idempotent finalizer finds the existing version by run and candidate provenance, moves production to that exact materialized version, and creates no duplicate profile version or candidate-promoted event. Equivalent simpler one-shot path: without prior materialization, the human runs the same labeled command once. A human admin or curator also runs `orizu manifests approve <id>`, `orizu manifests reject <id>`, and, after approval, `orizu manifests apply <id>`. |
+| Hosted session | Review instructions | Read diff comments with `orizu comments diff --run <run-id> --from <candidate-id> --to <candidate-id> --detail hunk` or `orizu comments diff --prompt <prompt-id> --from <version> --to <version> --detail hunk`; keep report-comment CLI targets inside the signed project and prepare proposed review text. | A human admin or curator adds or deletes a diff comment. To change one, delete it and re-add it; the mutation APIs expose POST and DELETE, not edit. Hand off the target, exact anchor, exact text, and named add/delete action. |
+| Hosted session | Session, credentials, and administration | Use connector secrets only through environment variables resolved at boot; prepare requested values without reading credential material. | A team ADMIN runs `orizu teams members add --email <email> --team <team>`, `orizu teams members remove --email <email> --team <team>`, or `orizu teams members role --team <team> --email <email> --role <admin\|curator\|judge>`. A session owner or team admin/curator may run `orizu session end --session <id>` for manual teardown; the coordinator owns automatic lifetime and provisions session and service-internal canonical-write credentials. A human runs `orizu session start --hosted --task "<prompt>" --project <team/project>`. For provider installation, kill switch, billing, deploy keys, or credential reads, ask the human to complete the named UI action and report completion without sharing secrets. |

package/vendor/skills/orizu/references/building-apps.md

@@ -2,7 +2,13 @@
 
 You are reading the canonical reference for coding agents producing Orizu apps. Orizu apps are React/TSX components that render inside Orizu's task and preview screens to collect feedback from human reviewers: ratings, comments, comparisons, annotations, rankings, corrections, and agent-transcript review. This document covers everything needed to author a working app — runtime contract, conventions, design principles, the import registry, the per-component reference, recipes, and common pitfalls. Read top-to-bottom on first encounter; jump to specific sections by anchor afterwards.
 
-If you are an external coding agent that found this via `orizu.ai/docs/llms.txt`, treat this document as the source of truth for both the runtime contract and the available primitives.
+If you are an external coding agent that found this via `orizu.ai/docs/references/building-apps`, treat this document as the source of truth for both the runtime contract and the available primitives.
+
+## Where this reference fits
+
+This deep reference owns only the app: the one-row React interface that shows evidence and returns a structured response. When the goal is human ground truth for an eval, start with the [eval strategy](eval-strategy.md) to ratify the questions, rows, labeler qualifications, and disagreement path before authoring the app. After that strategy is ratified, author, smoke-test, preview, and obtain approval for the app here.
+
+A task is the separate labeling round that binds a pinned app version to dataset rows and assignments. The [CLI reference](cli-reference.md) owns task creation, draft-URL approval, publication, assignment, status, export, and completion commands. Return to the [eval strategy](eval-strategy.md) for response-completeness proof and the annotation exit criterion.
 
 ---