orizu 0.5.25 → 0.5.26

package/vendor/skills/orizu/references/primer.md

@@ -2,28 +2,24 @@
 
 > **Core idea:** You can't improve what you can't measure — but measuring the wrong thing is worse than not measuring at all. This doc walks from raw failure logs to a continuously-running evaluation loop, and shows how Orizu operationalizes each step.
 
-The operational loop starts at **J3 — Build the dataset** and branches on the
-approved ground-truth source:
-
-```
-approved golden ground truth covers the scenario classes: J3 → J5
-missing ground truth needs human labels:                    J3 → J4 → J5
-                                                                 │
-                                                                 ▼
-                                                   J6 → new traces → J3
+The operational method is organized by the installed skill's flows:
+
+```text
+Onboard → First win ───────────────→ Promote
+            ├─ complete approved ground truth → judges → optimization run
+            ├─ partial approved coverage → annotate uncovered ground truth → judges → optimization run
+            └─ no approved coverage → annotation → judges → optimization run
+                                                   │
+                                                   └─ after Promote: Recurse, Triage, or Expand by user interest
 ```
 
-In the diagram, **J4 — Ground truth via annotation (conditional)** is taken only
-for missing labels, **J5 — Build & validate judges** consumes the approved
-ground truth, and **J6 — Optimize & promote** feeds new traces into the next J3
-dataset version. Approved golden ground truth therefore goes directly from J3
-to J5 instead of passing through annotation.
+First win builds the evidence and ends at a completed optimization run. Promote owns reporting, validation, and the human promotion decision. Recurse turns newly deployed traces into a later same-surface cycle; Triage starts from an incident and closes its eval gap before a fix; Expand chooses and plans another surface. Onboard owns pre-data inventory and plan ratification.
 
-The CLI covers the full operational loop. Start instruction sets with `orizu instructions`: one instruction set per agent or LLM experience, one profile per model config, and components owned by that profile. Human annotation still happens through apps/tasks when J4 is needed, while judge/scorer logic and optimization execution run locally and report back through the instruction control plane. Follow the applicable branch and require its prior journey step's exit criterion before continuing.
+The CLI covers this operational loop. Start instruction sets with `orizu instructions`: one instruction set per agent or LLM experience, one profile per model config, and components owned by that profile. Human annotation happens through apps/tasks only for required ground truth, while judge/scorer logic and optimization execution run locally and report back through the instruction control plane. The active flow file owns ordering and its exit criterion; this primer explains why the stages exist rather than replacing that procedure.
 
 An instruction-set manifest carries its `name`, optional `description`, fixed, ordered `shape`, and component values. The set's slug is its stable CLI reference even when its display name changes. Profiles have their own component values; components are never shared between profiles or sets. The set-wide default serves model configs without a production profile version.
 
-Start by inspecting the set, then follow the [Authority map](../SKILL.md#authority-map)
+Start by inspecting the set, then follow the [Authority map](authority-map.md)
 for mutation custody before syncing through the same surface:
 
 ```bash
@@ -59,7 +55,7 @@ The most common mistake teams make is jumping straight to evals before they unde
 
 ---
 
-## J3 detail: Upload — gather diverse traces
+## First win detail: Upload — gather diverse traces
 
 ### Why
 
@@ -105,7 +101,7 @@ Full surface: `cli-reference.md`.
 
 ---
 
-## Conditional J4 detail: Annotate — binary labels per failure mode
+## Conditional First win detail: Annotate — binary labels per failure mode
 
 ### Why
 
@@ -158,7 +154,7 @@ Authoring the labeler app — contract, design principles, and common patterns:
 
 ---
 
-## J5 detail: Judge — turn labels into automated evaluators
+## First win detail: Judge — turn labels into automated evaluators
 
 ### Why
 
@@ -191,7 +187,7 @@ Control-plane commands: `prompt-control-plane.md`. Detailed alignment-first walk
 
 ---
 
-## J6 detail: Optimize — hill-climb against validated judges
+## First win detail: Optimize — hill-climb against validated judges
 
 ### Why
 
@@ -206,8 +202,8 @@ Done locally, reported to Orizu:
 1. Package the candidate execution as a runner, or use the customer's existing local app wrapper.
 2. Register validated scorers.
 3. Run the bundled Orizu GEPA-style text optimizer, or a custom optimizer against the scorer set.
-4. Stream optimization events to Orizu, promote accepted candidates into the target instruction-set profile, and submit comparable scores.
-5. Diff before/after on the eval suite. Ship if it holds; new traces feed back to J3.
+4. Stream optimization events to Orizu and complete the run with a selected candidate or a recorded no-valid-candidate outcome.
+5. Stop First win at that completed run. Route to Promote for the report, supported Final-held-out comparison, validation evidence, and human ship, gather-more-evidence, or do-not-promote decision. After Promote, use Recurse for a human-agreed cadence that turns traces from the currently deployed version into a new immutable dataset version.
 
 Control-plane commands: `prompt-control-plane.md`. Detailed walkthrough — GEPA mechanics, optional DSPy context for customers already using it, and before/after comparison: `optimization-with-gepa.md`.
 

package/vendor/skills/orizu/references/prompt-control-plane.md

@@ -1,6 +1,6 @@
 # Instruction Control Plane
 
-Use this reference for the Phase 0 instruction, judge, scorer, runner, score, run, and optimization event control plane. Follow the [Authority map](../SKILL.md#authority-map) before executing its command sequences.
+Use this reference for the Phase 0 instruction, judge, scorer, runner, score, run, and optimization event control plane. Follow the [Authority map](authority-map.md) before executing its command sequences.
 
 ## Contents
 
@@ -23,7 +23,7 @@ Use this reference for the Phase 0 instruction, judge, scorer, runner, score, ru
 8. After `run-gepa`, inspect `logs/<optimization_run_id>` first; it is the complete local trace for coding-agent analysis.
 9. Use `orizu optimizations export <run-id> --out <run-id>.optimization.json` when the local log is missing or the run happened elsewhere.
 10. Write and attach a markdown report for finished, failed, or cancelled runs; use `optimization-reports.md` for structure and diagnostic guidance.
-11. Follow the [Authority map](../SKILL.md#authority-map). Simpler one-shot path: after the human accepts the report, a human curator runs `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`, materializing and labeling once. Equivalent two-stage path: an agent runs `orizu optimizations promote <run-id> --candidate <id> --project <team/project>` after acceptance. A human curator re-runs the same promotion as `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`. The idempotent finalizer finds the exact existing materialized version by run and candidate provenance, moves production, and creates no duplicate profile version or candidate-promoted event.
+11. Follow the [Authority map](authority-map.md). Simpler one-shot path: after the human accepts the report, a human curator runs `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`, materializing and labeling once. Equivalent two-stage path: an agent runs `orizu optimizations promote <run-id> --candidate <id> --project <team/project>` after acceptance. A human curator re-runs the same promotion as `orizu optimizations promote <run-id> --candidate <id> --label production --project <team/project>`. The idempotent finalizer finds the exact existing materialized version by run and candidate provenance, moves production, and creates no duplicate profile version or candidate-promoted event.
 12. For custom optimizers, start an optimization run before local execution, then stream events into that run.
 13. Use bare HTTP for optimization events; use `orizu log` only as a shell fallback.
 14. Promote only accepted candidates; rejected candidates stay in optimization events.
@@ -35,7 +35,7 @@ Customer model-provider secrets stay local. Do not upload Anthropic/OpenAI/etc.
 Work within the fixed, ordered component shape and model-config profiles defined
 for instruction sets in Vocabulary. Existing prompts appear as one-component
 sets. Inspect or write them with the customer-facing `orizu instructions`
-namespace. Follow the [Authority map](../SKILL.md#authority-map) for mutation
+namespace. Follow the [Authority map](authority-map.md) for mutation
 custody before copying a command:
 
 ```bash
@@ -560,7 +560,7 @@ orizu --local scorers list --project <team>/<project>
 orizu --local scorers detail <scorer-id-or-name> --project <team>/<project> --json
 ```
 
-Production scorer labels are pointer moves. The agent prepares the scorer name, version id, and evidence. After human acceptance, follow the [Authority map](../SKILL.md#authority-map): a human curator runs `orizu --local scorers labels set hip-note-judge-score production --version <scorer-version-id> --project <team>/<project> --json`.
+Production scorer labels are pointer moves. The agent prepares the scorer name, version id, and evidence. After human acceptance, follow the [Authority map](authority-map.md): a human curator runs `orizu --local scorers labels set hip-note-judge-score production --version <scorer-version-id> --project <team>/<project> --json`.
 
 Use scorer versions directly with the runner contract:
 
@@ -1054,7 +1054,7 @@ JUDGE_VERSION_ID="$(orizu --local judges push ./judge --project hip/judge-optimi
 SCORER_VERSION_ID="$(orizu --local scorers register --project hip/judge-optimization --name hip-note-judge-score --manifest ./scorer.manifest.json --prompt-version "$JUDGE_VERSION_ID" --runner-version "$RUNNER_VERSION_ID" --json | jq -r .scorer_version_id)"
 ```
 
-Human curator hand-off through the [Authority map](../SKILL.md#authority-map) (do not run as the agent): no standalone runner-label command exists, so the human first runs `orizu --local runners pull hip-note-judge-runner --project hip/judge-optimization --version "$RUNNER_VERSION_ID" --out ./runner-default-handoff`, leaves that fresh exact-version directory unchanged, then runs `orizu --local runners push ./runner-default-handoff --project hip/judge-optimization --name hip-note-judge-runner --label default --json`. The human curator then runs `orizu --local scorers labels set hip-note-judge-score production --version "$SCORER_VERSION_ID" --project hip/judge-optimization --json`.
+Human curator hand-off through the [Authority map](authority-map.md) (do not run as the agent): no standalone runner-label command exists, so the human first runs `orizu --local runners pull hip-note-judge-runner --project hip/judge-optimization --version "$RUNNER_VERSION_ID" --out ./runner-default-handoff`, leaves that fresh exact-version directory unchanged, then runs `orizu --local runners push ./runner-default-handoff --project hip/judge-optimization --name hip-note-judge-runner --label default --json`. The human curator then runs `orizu --local scorers labels set hip-note-judge-score production --version "$SCORER_VERSION_ID" --project hip/judge-optimization --json`.
 The agent continues with an unlabeled optimizer version:
 
 ```bash

package/vendor/skills/orizu/references/vocabulary.md

@@ -0,0 +1,27 @@
+# Vocabulary
+
+Use these terms exactly across Orizu's method, skill, references, and reports.
+
+## Workflow vocabulary
+
+- **Evals-first** — understand observed failures, build and validate evaluations, then optimize the LLM application against that evidence.
+- **Exit criterion** — the checkable evidence required before leaving one flow or ordered stage for the next.
+- **Human-only** — an action the agent prepares with exact evidence and an authorized human executes.
+- **Local surface** — a CLI action executed under the user's token; the authority map, not the surface, determines whether an agent or human executes it.
+- **Hosted session** — a platform-managed coding-agent session authenticated with session credentials and bound to its signed team and project scope.
+- **Instruction set** — the customer-facing identity for one agent or LLM experience, with a fixed ordered component shape and model-config profiles that own component values.
+- **Judge trust bar** — the user-agreed metric thresholds a judge must clear for a named downstream decision and failure mode.
+- **Decision class** — the judge's downstream use: gatekeeper, optimization signal, or triage / monitoring.
+- **Scenario class** — a named, meaningful family of cases from the ratified improvement plan whose coverage and results are tracked separately.
+- **Final-held-out** — the application partition reserved for Promote's single seed-versus-selected-candidate comparison.
+
+## Method vocabulary
+
+- **Flow** — one named, top-level path through the Orizu method that an agent (or human) works end to end: a defined entry condition ("reach for X when Y"), ordered steps, and an exit criterion.
+- **Quick win** — the recommended starting point for a repo new to Orizu: the instruction surface with the shortest credible path to a **measured, promotable model improvement**.
+- **Onboard (flow)** — the single on-ramp: in a repo Orizu hasn't touched, inventory where the prompts/skills/instructions live, lay it out, and recommend the quick win.
+- **First win (flow)** — the first pass through the loop on the recommended surface: curate the dataset, gather ground truth (annotation only when approved labels don't exist), build and validate judges, hill-climb to a completed optimization run.
+- **Promote (flow)** — the actual customer win: report the run, support the human's promotion decision (learnings, tradeoffs, regressions), and guide validation (blind side-by-sides, replay on past traffic, staged rollout).
+- **Recurse (flow)** — post-win, cadence-driven: fresh traces become a new dataset version and the loop reruns on the same surface on an agreed cadence.
+- **Triage (flow)** — post-win, incident-driven: a reported issue leads to trace analysis and the eval-gap decision — does an existing eval catch it, does an eval need fixing, is an eval missing, or is the reported behavior expected — before any optimizing or fixing.
+- **Expand (flow)** — post-win, appetite-driven: pick the next instruction surface from Onboard's inventory by the same quick-win criterion and run the loop there.