orizu 0.3.4 → 0.3.6

package/vendor/skills/orizu-cli/references/building-judges.md

@@ -0,0 +1,218 @@
+# Building Judges And Scorers
+
+How to turn human-labeled annotations into automated evaluators. Use this reference for authoring and validating the evaluator logic; use `prompt-control-plane.md` to push versioned prompt artifacts, runner artifacts, scorer definitions, and score runs into Orizu.
+
+## Inputs
+
+You should arrive here with:
+- A labeled task export: `orizu tasks export --task <taskId> --format jsonl --out labels.jsonl`
+- One or more **specific binary failure modes** the labels measure (e.g. `correctly_identified_issue`, `escalated_when_required`).
+- The original dataset row each label refers to (the export includes `dataset_row_id` and the row payload).
+
+If you have Likert or "overall quality" labels, stop and re-annotate. Judges built on Likert data are unreliable. See `primer.md` Step 2.
+
+Terminology:
+- A **judge** is the evaluator itself, often an LLM prompt plus runner.
+- A **scorer** is the metric contract stored in Orizu. It names the score, directionality, mode (`row` or `set`), display format, and backing implementation.
+- Row scorers return per-row scores and feedback. Set scorers compute aggregate metrics such as Cohen's kappa, accuracy, precision, recall, F1, or a custom reducer over row-level outputs.
+- Use `orizu scorers exec` for scorer-level evaluation. Builtin set scorers consume subject results or dependency row-scorer results and submit one aggregate score run by default. `runners exec --scorer-version` is a low-level row-runner compatibility command.
+
+## Choosing the judge type
+
+| Failure mode looks like | Use                                       |
+|-------------------------|-------------------------------------------|
+| Rule-shaped (keyword present, tool called, format valid, length within bounds) | **Code assertion** |
+| Subjective / nuanced (offered the right resolution, tone matches policy, escalation was warranted) | **LLM-as-a-judge** |
+| Path-of-life (block before user sees the response — toxicity, PII leak) | **Guardrail** (small classifier or code, runs synchronously; not covered here) |
+
+Default to code. LLM judges cost money, time, and trust budget — earn them.
+
+## Code assertions
+
+A code assertion is a pure function:
+
+```python
+def passes(input: dict, output: dict) -> bool:
+    # Returns True if the output is acceptable for this failure mode.
+    ...
+```
+
+**Patterns that work well:**
+- Keyword/regex match on output text
+- Structural check on JSON output
+- Tool-call verification (did the agent call the expected tool with valid args?)
+- Length/format bounds (response is within N tokens, ends with a question mark, etc.)
+- Numeric tolerance for known answers
+
+**Example — case reference number present in support response:**
+
+```python
+import re
+
+CASE_REF_RE = re.compile(r"\bCASE-\d{6}\b")
+
+def has_case_reference(input: dict, output: dict) -> bool:
+    return bool(CASE_REF_RE.search(output["text"]))
+```
+
+Run it over the labeled export and confirm it agrees with the human labels for this failure mode. If it does, you're done — code assertions don't need TPR/TNR validation when they're tautologically correct (the rule *is* the failure definition).
+
+If your code assertion disagrees with humans on >5% of cases, your rule isn't capturing the failure mode and you probably need an LLM judge instead.
+
+## LLM-as-a-judge
+
+For nuanced failures, build a judge that takes `(input, output)` and returns `pass | fail`.
+
+### Step 1: Split the labels
+
+Split your labeled export 20 / 40 / 40:
+
+| Set    | Size | Purpose                                            |
+|--------|------|----------------------------------------------------|
+| Train  | ~20% | Few-shot examples in the judge prompt              |
+| Dev    | ~40% | Iterate on the judge prompt; measure TPR / TNR     |
+| Test   | ~40% | Final validation; touch only when you think you're done |
+
+Random split is fine for stationary data. If labels skew (e.g. 80% pass, 20% fail), stratify so each split preserves the ratio.
+
+### Step 2: Write the judge prompt
+
+Structure:
+1. **Task framing.** What's being judged. State it as a binary question.
+2. **Failure-mode definition.** Be specific. Include positive and negative criteria.
+3. **Few-shot examples.** Draw 4–8 from the train set. Half pass, half fail. Include short rationales.
+4. **Output format.** Force structured output: `{"pass": true|false, "reason": "..."}`. Reason is for debugging, not scoring.
+
+**Example skeleton:**
+
+```python
+JUDGE_PROMPT = """You are evaluating a customer support agent response.
+
+The failure mode under evaluation: **escalated_when_required**.
+- PASS: When the customer's situation requires human escalation (refund > $500, account compromise, legal threat, distressed user), the agent escalated.
+- FAIL: The situation required escalation and the agent did not, OR the agent escalated unnecessarily.
+
+Respond with JSON only: {"pass": <true|false>, "reason": "<one sentence>"}.
+
+Examples:
+<input>{{example_1_input}}</input>
+<output>{{example_1_output}}</output>
+<judgment>{"pass": true, "reason": "Distressed user, agent handed off."}</judgment>
+
+<input>{{example_2_input}}</input>
+<output>{{example_2_output}}</output>
+<judgment>{"pass": false, "reason": "Refund $1200, agent processed without escalation."}</judgment>
+
+... 4–8 examples total ...
+
+Now evaluate:
+<input>{input}</input>
+<output>{output}</output>
+<judgment>"""
+```
+
+### Step 3: Validate on dev set
+
+Run the judge over the dev set and compare against human labels. Compute:
+
+```
+TPR = (judge says fail AND human says fail)  /  (human says fail)
+TNR = (judge says pass AND human says pass)  /  (human says pass)
+```
+
+**Targets: TPR > 90% AND TNR > 90%.**
+
+Why both: a judge that flags everything has TPR=100% and a useless TNR. A judge that passes everything is the inverse. Both matter; track them separately.
+
+If TPR is low → judge is missing failures. Tighten the failure-mode definition; add few-shots that demonstrate the failures it's missing.
+
+If TNR is low → judge is over-flagging. Loosen overly broad criteria; add few-shots of edge-case passes.
+
+### Step 4: Final test
+
+Run the judge once on the held-out test set. If TPR/TNR drops more than a few points vs. dev, you overfit to dev — go back, regenerate dev/test split, retry. **Don't iterate on the test set.**
+
+### Step 5: Use the judge
+
+Run the validated judge over future outputs to score them. Wire it into:
+- CI/CD on a curated test set (block merges that drop pass rate)
+- Online monitoring on sampled traffic (alert on regressions)
+- Optimization loop as a metric (`optimization-with-gepa.md`)
+- Orizu scorers and score runs (`prompt-control-plane.md`) so prompt detail, changelog, scorer detail, and optimization surfaces show comparable performance.
+
+## Set scorers for judge-vs-gold agreement
+
+Use a set scorer when the meaningful metric is only defined over a batch, not one row. Cohen's kappa is the recommended headline for binary judge-vs-human agreement because it corrects for label imbalance and has a chance-agreement zero point.
+
+For a binary `ok` / `flag` judge:
+- Register the headline scorer as `mode: "set"`, `metric_key: "cohens_kappa"`, `score_format: "number"`, and `higher_is_better: true`.
+- Set `implementation_kind: "builtin_metric"`, `builtin_metric: "cohens_kappa"`, an explicit `input_mapping`, and `builtin_metric_config.positive_class: "flag"`.
+- Execute with `orizu scorers exec` when row-scorer results are available as a stored score run or local JSONL.
+- Include `accuracy`, `sample_size`, a confusion matrix, and per-class precision/recall in diagnostics.
+- Treat `flag` as the positive class by default, so `tp` means human=`flag` and judge=`flag`, `fn` means human=`flag` and judge=`ok`, `fp` means human=`ok` and judge=`flag`, and `tn` means human=`ok` and judge=`ok`.
+- Include `flag_recall` prominently because it answers whether the judge catches the failures humans care about.
+- On small validation sets, call out uncertainty in `feedbackSummary` rather than over-interpreting a single kappa point estimate.
+
+Recommended diagnostics shape:
+
+```json
+{
+  "sample_size": 15,
+  "accuracy": 0.73,
+  "confusion_matrix": {
+    "tp": 5,
+    "fn": 1,
+    "fp": 3,
+    "tn": 6,
+    "positive_class": "flag"
+  },
+  "flag_recall": 0.83,
+  "flag_precision": 0.63,
+  "ok_recall": 0.67,
+  "ok_precision": 0.86,
+  "class_balance": {
+    "human_flag": 6,
+    "human_ok": 9
+  }
+}
+```
+
+## Saturation check
+
+If your judge eventually reports 100% pass on a test set, the eval is **saturated** — it's no longer finding failures. That's not victory; it means you need harder cases. Sample fresh production traces, look for ones the current system handles ambiguously, and add them.
+
+## Storing judges
+
+Treat each judge as code under version control, and push the runnable Orizu representation when using the control plane:
+
+```
+evals/
+  <project>/
+    judges/
+      escalated_when_required.py     # the judge function
+      escalated_when_required.test.py# TPR/TNR test against the held-out export
+      labels.jsonl                   # the export this was validated on
+      README.md                      # what failure mode, what the threshold story is
+```
+
+For Orizu-managed workflows:
+
+1. Push the runner with `orizu runners push`.
+2. Push the LLM judge prompt with `orizu judges push` when applicable.
+3. Register the scorer with `orizu scorers register`.
+4. For row scorers, execute the scorer runner with `orizu runners exec --scorer-version ...` and submit row results with `orizu scores submit`.
+5. For builtin set scorers, execute with `orizu scorers exec`; for locally precomputed set scores, submit the aggregate object with `orizu scores submit --aggregate`.
+
+When you re-export labels (e.g. after annotating more data), re-run validation. Judges and scorers drift as the system and data drift; periodic revalidation catches it.
+
+## Checklist
+
+Before declaring a judge production-ready:
+
+- [ ] Failure mode is specific and binary (not "overall quality")
+- [ ] Code assertion attempted first; LLM judge only if rule-shaping wasn't enough
+- [ ] If LLM judge: prompt has clear pass/fail criteria + 4–8 stratified few-shots
+- [ ] Train / dev / test split (20 / 40 / 40)
+- [ ] TPR > 90% AND TNR > 90% on test
+- [ ] Stored in version control alongside the labels it was validated on
+- [ ] If using Orizu control plane, scorer registered with readable metric name, direction, mode, and dataset requirements

package/vendor/skills/orizu-cli/references/cli-reference.md

@@ -0,0 +1,324 @@
+# Orizu CLI Reference
+
+## Contents
+
+- [Default Command Strategy](#default-command-strategy)
+- [Command Matrix](#command-matrix)
+- [End-to-End Flows](#end-to-end-flows)
+- [Notes and Limits](#notes-and-limits)
+
+## Default Command Strategy
+
+1. Verify auth first: `orizu whoami`.
+2. Prefer explicit flags over prompts.
+3. Use interactive fallback only in TTY sessions.
+4. Validate command output before proceeding.
+5. On failure, fix flags/identifiers and rerun.
+
+## Command Matrix
+
+### Authentication
+
+```bash
+orizu login
+orizu whoami
+orizu logout
+```
+
+Behavior:
+- `orizu login` opens browser approval and creates a user-owned personal access token for the CLI.
+- New logins store v3 API-key credentials in `~/.config/orizu/credentials.json`.
+- The CLI still reads older session credentials during rollout, but new logins replace them with PAT credentials.
+- `orizu logout` revokes the current PAT remotely when possible and then clears local credentials.
+- PATs can also be revoked from the Personal Tokens page in Orizu.
+- PAT authorization follows the owning user's current team/project roles, so demotion or removal takes effect without rotating the token.
+
+### Teams
+
+```bash
+orizu teams list
+orizu teams create --name "My Team"
+orizu teams members list --team my-team
+orizu teams members add --email person@example.com --team my-team
+orizu teams members remove --email person@example.com --team my-team
+orizu teams members role --team my-team --email person@example.com --role admin
+```
+
+Allowed roles:
+- `admin`
+- `member`
+
+### Projects
+
+```bash
+orizu projects list
+orizu projects list --team my-team
+orizu projects create --name "Quality Eval" --team my-team
+```
+
+### Apps
+
+```bash
+orizu apps list --project my-team/quality-eval
+```
+
+Create from file:
+
+```bash
+orizu apps create \
+  --project my-team/quality-eval \
+  --name "Labeling App" \
+  --dataset <datasetId> \
+  --file ./apps/LabelingApp.tsx \
+  --input-schema ./schemas/input.json \
+  --output-schema ./schemas/output.json
+```
+
+Preview locally before upload:
+
+```bash
+orizu apps preview \
+  --file ./apps/LabelingApp.tsx \
+  --input-schema ./schemas/input.json \
+  --output-schema ./schemas/output.json \
+  --sample-row ./fixtures/sample-row.json \
+  --screenshot ./preview.png
+```
+
+Use `--headed` for visible Chromium review and `--keep-open` to leave the browser running.
+
+Update from file:
+
+```bash
+orizu apps update \
+  --app <appId> \
+  --file ./apps/LabelingApp.v2.tsx \
+  --input-schema ./schemas/input.json \
+  --output-schema ./schemas/output.json
+```
+
+Link dataset:
+
+```bash
+orizu apps link-dataset --app <appId> --dataset <datasetId>
+```
+
+Export app source:
+
+```bash
+orizu apps export --app <appId> --project my-team/quality-eval --out ./apps/LabelingApp.tsx
+orizu apps export --app <appId> --version 2
+```
+
+### Datasets
+
+Row identity contract:
+- `row.id` is the canonical row identifier; the rest of the JSON object is the payload.
+- `row_index` is not part of canonical runtime selection. CLI flags like `--row-ids` always take canonical `id` values.
+
+```bash
+orizu datasets upload --file ./data.csv --project my-team/quality-eval --name "Batch 1"
+orizu datasets download --dataset <datasetId|datasetUrl> --format jsonl --out ./dataset.jsonl
+orizu datasets append --dataset <datasetId|datasetUrl> --file ./new-rows.jsonl
+orizu datasets edit-rows --dataset <datasetId|datasetUrl> --file ./edited-rows.jsonl
+orizu datasets delete-rows --dataset <datasetId|datasetUrl> --row-ids row-1,row-2
+orizu datasets delete --dataset <datasetId|datasetUrl>
+orizu datasets lock --dataset <datasetId|datasetUrl> --reason "Finalize for labeling"
+orizu datasets clone --dataset <datasetId|datasetUrl> --name "Batch 1 Copy"
+```
+
+Dataset version and split-set commands used by prompt runs are covered in `prompt-control-plane.md`.
+
+Supported file types:
+- `.csv`
+- `.json` (array of objects)
+- `.jsonl` (one object per line)
+
+Delete rows selectors:
+- `--row-ids <id1,id2>` (canonical selector)
+
+Delete dataset:
+- `datasets delete` permanently deletes the dataset and requires an interactive terminal confirmation.
+- There is no non-interactive confirmation flag.
+
+Edit rows requirements:
+- `--file` rows must include canonical `id` for each row being updated.
+
+### Tasks
+
+```bash
+orizu tasks list
+orizu tasks list --project my-team/quality-eval
+```
+
+Create:
+
+```bash
+orizu tasks create \
+  --project my-team/quality-eval \
+  --dataset <datasetId> \
+  --app <appId> \
+  --title "Round 1 labeling" \
+  --instructions "Follow rubric v1" \
+  --labels-per-item 2
+```
+
+Behavior:
+- task creation creates a draft by default and returns a task URL to test manually before assigning
+- use `--publish --assignees <userIdOrEmail1,userIdOrEmail2>` only when you intentionally want to create and ship immediately
+- after manually approving a draft, run `orizu tasks publish --task <taskId> --assignees <userId1,userId2>`
+- task creation resolves and stores the app's pinned current `version_id` at create time
+- downstream consumers (exports, judges, optimization) should trust the task's pinned `version_id`, not the app's current pointer
+- dataset compatibility is validated against that pinned app version before the task is inserted
+- malformed JSON and mixed-type assignee arrays fail with deterministic `400` responses
+- assignment fanout enforces unique `(assignee, row)` pairs; `--labels-per-item` cannot exceed the number of unique assignees, and the backend shortfalls instead of duplicating
+
+Publish:
+
+```bash
+orizu tasks publish --task <taskId> --assignees <userId1,userId2>
+```
+
+Assign:
+
+```bash
+orizu tasks assign --task <taskId> --assignees <userId1,userId2>
+```
+
+Status:
+
+```bash
+orizu tasks status --task <taskId>
+orizu tasks status --task <taskId> --json
+```
+
+Includes:
+- task metadata
+- progress counts
+- per-assignee breakdown
+- paused assignments as a distinct count, not folded into pending
+
+Report:
+
+```bash
+orizu tasks report set --task <taskId> --report-file ./task-report.md
+orizu tasks report upload --task <taskId> --report @./task-report.md
+```
+
+Behavior:
+- replaces the current task report if one already exists
+- accepted only when the task is `paused` or `completed`
+- accepts the same `--report`, `--report @file`, and `--report-file` Markdown inputs as optimization reports
+
+Export:
+
+```bash
+orizu tasks export --task <taskId> --format jsonl --out ./labels.jsonl
+```
+
+Formats:
+- `csv`
+- `json`
+- `jsonl`
+
+Defaults:
+- format defaults to `jsonl`
+- output file defaults to `<taskId>.<format>`
+
+### Prompt Control Plane
+
+For prompts, judges, runners, run submission, optimizer artifacts, live event logging, and accepted-candidate promotion, read `prompt-control-plane.md`. For markdown reports attached to optimization runs, read `optimization-reports.md`.
+
+Optimization trace commands:
+
+```bash
+orizu optimizations run-gepa ... [--log-dir logs] [--no-local-log]
+orizu optimizations export <optimization-run-id> --out ./optimization.json
+orizu optimizations export <optimization-run-id> --json
+```
+
+Behavior:
+- `run-gepa` writes a complete local trace under `logs/<optimization_run_id>` by default.
+- The local trace is the best artifact for coding-agent analysis because it includes full rows, outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, candidate text, and `result.json`.
+- `optimizations export` writes a portable JSON artifact from server data when the local log is unavailable or the run happened elsewhere.
+- Server optimization events redact row snapshots and reflection prompts by default; export rehydrates row inputs from dataset artifacts when possible and includes bundled `run-gepa` reflection responses.
+
+## End-to-End Flows
+
+### New Team to Export
+
+```bash
+orizu login
+orizu teams create --name "Ops Eval"
+orizu projects create --name "Support QA" --team ops-eval
+
+orizu datasets upload --project ops-eval/support-qa --file ./datasets/support.jsonl --name "Support Batch 1"
+orizu datasets append --dataset <datasetId> --file ./datasets/support-extra.jsonl
+orizu datasets edit-rows --dataset <datasetId> --file ./datasets/support-edits.jsonl
+orizu datasets delete-rows --dataset <datasetId> --row-ids row-10,row-11
+orizu datasets delete --dataset <datasetId>
+orizu datasets lock --dataset <datasetId> --reason "Freeze for labeling"
+orizu datasets clone --dataset <datasetId> --name "Support Batch 1 Copy"
+
+orizu apps create \
+  --project ops-eval/support-qa \
+  --name "Support Labeler" \
+  --dataset <datasetId> \
+  --file ./apps/SupportLabeler.tsx \
+  --input-schema ./schemas/support-input.json \
+  --output-schema ./schemas/support-output.json
+
+orizu apps link-dataset --app <appId> --dataset <datasetId>
+
+orizu tasks create \
+  --project ops-eval/support-qa \
+  --dataset <datasetId> \
+  --app <appId> \
+  --title "Support QA Round 1" \
+  --labels-per-item 2
+
+# Open the returned task URL and test the draft manually before assigning.
+orizu tasks publish --task <taskId> --assignees <userId1,userId2>
+
+orizu tasks status --task <taskId>
+orizu tasks export --task <taskId> --format csv --out ./support-round1.csv
+```
+
+### Interactive-First Shortcuts
+
+```bash
+orizu apps list
+orizu teams members add --email new-person@example.com
+orizu datasets upload --file ./data.csv
+orizu tasks export
+```
+
+Use these shortcuts only in TTY environments where prompts can run.
+
+## Notes and Limits
+
+- `tasks create` creates a draft by default, does not require `--assignees`, and pins the app's current version when the task is created.
+- `tasks create --publish --assignees <...>` intentionally creates and ships immediately.
+- `tasks assign` and `tasks publish` expect user IDs, not emails.
+- Assignment queue reads are assignee-self-only; use task status/export as the operator summary path.
+- Assignment completion payloads are validated against the pinned app-version `output_json_schema`.
+- `datasets delete-rows` requires `--row-ids`.
+- `datasets delete` requires interactive terminal confirmation and has no non-interactive confirmation flag.
+- `datasets edit-rows` requires row objects in `--file` to include canonical `id`.
+- `--row-ids` is the canonical row selection for delete operations.
+- Locked datasets reject append/edit/delete row mutations.
+- Row deletes are rejected when targeted rows are assignment-referenced.
+- Login currently requires callback availability on `127.0.0.1:43123`.
+- New CLI logins use personal access tokens rather than short-lived Supabase session credentials.
+- In non-interactive contexts, pass explicit selection flags.
+
+Output-schema validation surface:
+- `--output-schema` JSON is validated against a subset of JSON Schema only: `type`, `required`, `properties`, `items`, `enum`. Other keywords (`pattern`, `format`, `oneOf`, etc.) are ignored. See `building-apps.md` for the contract.
+
+Hugging Face / external dataset auth:
+- Never persist auth tokens in row payloads, dataset metadata, exports, or logs.
+- Store only non-secret credential references (e.g. `huggingface.token_ref`) in source metadata.
+
+Worker assignment reads are self-only:
+- Regular members cannot see other assignees' queues or response payloads.
+- Use `tasks status` and `tasks export` for operator-side reporting.