orizu 0.3.7 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +55 -6
- package/dist/banner.js +20 -0
- package/dist/csv.js +117 -9
- package/dist/help.js +114 -15
- package/dist/index.js +828 -67
- package/dist/setup-prompt.js +22 -0
- package/dist/skill-installer.js +243 -17
- package/dist/task-assignment-manifest.js +59 -0
- package/dist/workspace.js +76 -0
- package/package.json +5 -1
- package/src/banner.ts +21 -0
- package/src/csv.ts +140 -10
- package/src/help.ts +114 -15
- package/src/index.ts +933 -70
- package/src/setup-prompt.ts +27 -0
- package/src/skill-installer.ts +343 -17
- package/src/task-assignment-manifest.ts +96 -0
- package/src/workspace.ts +112 -0
- package/vendor/orizu-gepa-python/src/orizu_gepa/__init__.py +2 -0
- package/vendor/orizu-gepa-python/src/orizu_gepa/cli.py +57 -3
- package/vendor/orizu-gepa-python/src/orizu_gepa/client.py +37 -1
- package/vendor/orizu-gepa-python/src/orizu_gepa/optimizer.py +233 -17
- package/vendor/orizu-gepa-python/src/orizu_gepa/reflection.py +98 -17
- package/vendor/skills/orizu-cli/SKILL.md +13 -6
- package/vendor/skills/orizu-cli/references/cli-reference.md +34 -5
- package/vendor/skills/orizu-cli/references/optimization-reports.md +2 -2
- package/vendor/skills/orizu-cli/references/optimization-with-gepa.md +1 -0
- package/vendor/skills/orizu-cli/references/prompt-control-plane.md +17 -5
|
@@ -12,8 +12,9 @@
|
|
|
12
12
|
1. Verify auth first: `orizu whoami`.
|
|
13
13
|
2. Prefer explicit flags over prompts.
|
|
14
14
|
3. Use interactive fallback only in TTY sessions.
|
|
15
|
-
4.
|
|
16
|
-
5.
|
|
15
|
+
4. Prefer `--json` for machine-readable output — every command supports it (prefix `orizu --json <cmd>` or trailing flag). Long-running commands emit the JSON summary as the final stdout line.
|
|
16
|
+
5. Validate command output before proceeding.
|
|
17
|
+
6. On failure, fix flags/identifiers and rerun.
|
|
17
18
|
|
|
18
19
|
## Command Matrix
|
|
19
20
|
|
|
@@ -33,6 +34,25 @@ Behavior:
|
|
|
33
34
|
- PATs can also be revoked from the Personal Tokens page in Orizu.
|
|
34
35
|
- PAT authorization follows the owning user's current team/project roles, so demotion or removal takes effect without rotating the token.
|
|
35
36
|
|
|
37
|
+
### Agent Setup
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
orizu capabilities --json
|
|
41
|
+
orizu skills path [--skill-md] [--json]
|
|
42
|
+
orizu skills status [--json]
|
|
43
|
+
orizu skills update [--dry-run] [--json]
|
|
44
|
+
orizu install-skill [--agent <claude|codex>]... [--scope global|project] [--mode auto|link|copy] [--yes] [--dry-run] [--json]
|
|
45
|
+
orizu setup [--agent <claude|codex>]... [--skip-login] [--no-install] [--workspace|--no-workspace] [--no-handoff] [--yes] [--dry-run] [--no-input] [--json]
|
|
46
|
+
orizu setup prompt [--json]
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Behavior:
|
|
50
|
+
- `capabilities --json` returns the full machine-readable command manifest, including `globalOptions`.
|
|
51
|
+
- `skills path --json` returns where this skill's source lives (`root`, `skillMd`, `source`, `cliVersion`, `skillHash`) so you can read it without installing, and verify an installed copy matches the CLI.
|
|
52
|
+
- `skills status` reports each install target as missing/current/stale/broken-link; `skills update` refreshes stale copies and repairs broken symlinks.
|
|
53
|
+
- `install-skill` selects targets by agent name; `--scope global` (default) installs user-level, `--scope project` into the repo. Project-scope installs always copy; global installs symlink when the CLI path is stable. Advanced target IDs remain available via `--target`.
|
|
54
|
+
- `orizu setup` is the guided onboarding flow (login → agent integration → `.orizu/` workspace → handoff prompt). `orizu setup prompt` prints the repo-adoption prompt for handing to a coding agent.
|
|
55
|
+
|
|
36
56
|
### Teams
|
|
37
57
|
|
|
38
58
|
```bash
|
|
@@ -166,23 +186,28 @@ orizu tasks create \
|
|
|
166
186
|
Behavior:
|
|
167
187
|
- task creation creates a draft by default and returns a task URL to test manually before assigning
|
|
168
188
|
- use `--publish --assignees <userIdOrEmail1,userIdOrEmail2>` only when you intentionally want to create and ship immediately
|
|
189
|
+
- use `--assignment-file <path>` instead of `--assignees` when specific rows should go to specific labellers
|
|
169
190
|
- after manually approving a draft, run `orizu tasks publish --task <taskId> --assignees <userId1,userId2>`
|
|
191
|
+
- or run `orizu tasks publish --task <taskId> --assignment-file <path>` to publish the exact row map
|
|
170
192
|
- task creation resolves and stores the app's pinned current `version_id` at create time
|
|
171
193
|
- downstream consumers (exports, judges, optimization) should trust the task's pinned `version_id`, not the app's current pointer
|
|
172
194
|
- dataset compatibility is validated against that pinned app version before the task is inserted
|
|
173
195
|
- malformed JSON and mixed-type assignee arrays fail with deterministic `400` responses
|
|
174
196
|
- assignment fanout enforces unique `(assignee, row)` pairs; `--labels-per-item` cannot exceed the number of unique assignees, and the backend shortfalls instead of duplicating
|
|
197
|
+
- explicit assignment manifests are JSONL with `rowId` plus `assignee` or `assignees`; row IDs are canonical dataset row `id` values, assignees may be emails or user IDs, and V1 publish requires whole-dataset uniform coverage
|
|
175
198
|
|
|
176
199
|
Publish:
|
|
177
200
|
|
|
178
201
|
```bash
|
|
179
202
|
orizu tasks publish --task <taskId> --assignees <userId1,userId2>
|
|
203
|
+
orizu tasks publish --task <taskId> --assignment-file ./assignments.jsonl
|
|
180
204
|
```
|
|
181
205
|
|
|
182
206
|
Assign:
|
|
183
207
|
|
|
184
208
|
```bash
|
|
185
209
|
orizu tasks assign --task <taskId> --assignees <userId1,userId2>
|
|
210
|
+
orizu tasks assign --task <taskId> --assignment-file ./assignments.jsonl --replace-existing
|
|
186
211
|
```
|
|
187
212
|
|
|
188
213
|
Status:
|
|
@@ -232,13 +257,14 @@ For prompts, judges, runners, run submission, optimizer artifacts, live event lo
|
|
|
232
257
|
Optimization trace commands:
|
|
233
258
|
|
|
234
259
|
```bash
|
|
235
|
-
orizu optimizations run-gepa ... [--num-threads auto|N] [--log-dir logs] [--no-local-log]
|
|
260
|
+
orizu optimizations run-gepa ... [--num-threads auto|N] [--reflection-retry-attempts 3] [--reflection-http-timeout-seconds 180] [--log-dir logs] [--no-local-log]
|
|
236
261
|
orizu optimizations export <optimization-run-id> --out ./optimization.json
|
|
237
262
|
orizu optimizations export <optimization-run-id> --json
|
|
238
263
|
```
|
|
239
264
|
|
|
240
265
|
Behavior:
|
|
241
|
-
- `run-gepa` defaults `--num-threads` to `auto`, resolving a
|
|
266
|
+
- `run-gepa` defaults `--num-threads` to `auto`, resolving a row-evaluation parallelism cap from mini-batch size, validation-set size, 2x CPU count, memory estimate, file-descriptor limit, and a 64-thread default ceiling. Set `ORIZU_GEPA_AUTO_THREADS_MAX` or use `--num-threads <n>` only when the runner/provider capacity is known.
|
|
267
|
+
- `run-gepa` retries transient reflection-provider failures by default. If retries are exhausted, it logs `reflection_failed`, counts that reflection proposal against budget, and continues with the next iteration.
|
|
242
268
|
- `run-gepa` writes a complete local trace under `logs/<optimization_run_id>` by default.
|
|
243
269
|
- The local trace is the best artifact for coding-agent analysis because it includes full rows, outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, candidate text, and `result.json`.
|
|
244
270
|
- `optimizations export` writes a portable JSON artifact from server data when the local log is unavailable or the run happened elsewhere.
|
|
@@ -280,6 +306,8 @@ orizu tasks create \
|
|
|
280
306
|
|
|
281
307
|
# Open the returned task URL and test the draft manually before assigning.
|
|
282
308
|
orizu tasks publish --task <taskId> --assignees <userId1,userId2>
|
|
309
|
+
# Or publish a deterministic row map:
|
|
310
|
+
orizu tasks publish --task <taskId> --assignment-file ./assignments.jsonl
|
|
283
311
|
|
|
284
312
|
orizu tasks status --task <taskId>
|
|
285
313
|
orizu tasks export --task <taskId> --format csv --out ./support-round1.csv
|
|
@@ -300,7 +328,8 @@ Use these shortcuts only in TTY environments where prompts can run.
|
|
|
300
328
|
|
|
301
329
|
- `tasks create` creates a draft by default, does not require `--assignees`, and pins the app's current version when the task is created.
|
|
302
330
|
- `tasks create --publish --assignees <...>` intentionally creates and ships immediately.
|
|
303
|
-
- `tasks assign` and
|
|
331
|
+
- `tasks create|publish|assign --assignment-file <path>` is mutually exclusive with `--assignees` and accepts emails or user IDs in the JSONL manifest.
|
|
332
|
+
- `tasks assign --assignees` and `tasks publish --assignees` expect user IDs, not emails.
|
|
304
333
|
- Assignment queue reads are assignee-self-only; use task status/export as the operator summary path.
|
|
305
334
|
- Assignment completion payloads are validated against the pinned app-version `output_json_schema`.
|
|
306
335
|
- `datasets delete-rows` requires `--row-ids`.
|
|
@@ -130,9 +130,9 @@ Set an explicit expectation before the run: for example, "if best kappa does not
|
|
|
130
130
|
|
|
131
131
|
### Budget Versus Iteration Cap
|
|
132
132
|
|
|
133
|
-
Default budget presets are DSPy-style metric-call budgets. `light`, `medium`, and `heavy` map to DSPy's `auto="light"`, `auto="medium"`, and `auto="heavy"` scales; `auto` is treated as the balanced `medium` preset. The final metric-call limit scales with validation-set size, rather than using fixed tiny constants. `--max-metric-calls
|
|
133
|
+
Default budget presets are DSPy-style metric-call budgets. `light`, `medium`, and `heavy` map to DSPy's `auto="light"`, `auto="medium"`, and `auto="heavy"` scales; `auto` is treated as the balanced `medium` preset and is the default when no budget control is provided. The final metric-call limit scales with validation-set size, rather than using fixed tiny constants. `--budget`, `--max-metric-calls`, `--max-full-evals`, and `--max-iterations` are mutually exclusive CLI budget controls; choose only one.
|
|
134
134
|
|
|
135
|
-
|
|
135
|
+
Metric-call and full-eval budgets are checked only between iterations. Once an iteration starts, `run-gepa` completes the parent minibatch, child minibatch, and full validation eval for an accepted child even if that overshoots the nominal budget. If a run ends as `budget_exhausted`, it pauses and does not auto-promote. If auto-promotion matters, choose a single budget control large enough to cover the intended search.
|
|
136
136
|
|
|
137
137
|
### Reflection Signal Quality
|
|
138
138
|
|
|
@@ -62,6 +62,7 @@ Reflection output contract:
|
|
|
62
62
|
- The default reflection prompt asks for only the complete updated prompt body. Do not ask the model to wrap the candidate in markdown fences or tags; real prompts often contain those characters.
|
|
63
63
|
- Put provider-native reasoning controls in `--reflection-provider-settings <json|@file>`, not in the prompt body. For OpenAI reasoning models, use a shape such as `{"reasoning":{"effort":"medium","summary":"auto"}}`. For Anthropic Claude models with thinking controls, use a shape such as `{"thinking":{"type":"adaptive","display":"omitted"},"output_config":{"effort":"medium"}}`.
|
|
64
64
|
- Reflection max-token limits are explicit. `--reflection-max-tokens <n>` maps to Anthropic `max_tokens` and OpenAI `max_output_tokens`; Anthropic native Messages reflection requires it, while OpenAI may omit it when no cap is desired.
|
|
65
|
+
- Reflection HTTP calls retry transient failures by default (`--reflection-retry-attempts 3`, `--reflection-http-timeout-seconds 180`). Exhausted retryable failures emit `reflection_failed`, consume one candidate-proposal budget slot, and move to the next iteration instead of killing the run.
|
|
65
66
|
|
|
66
67
|
Full command syntax and event contracts: `prompt-control-plane.md`.
|
|
67
68
|
|
|
@@ -332,9 +332,21 @@ List:
|
|
|
332
332
|
|
|
333
333
|
```bash
|
|
334
334
|
orizu --local prompts list --project <team>/<project>
|
|
335
|
+
orizu --local prompts list --project <team>/<project> --status archived
|
|
336
|
+
orizu --local prompts list --project <team>/<project> --status all
|
|
335
337
|
orizu --local judges list --project <team>/<project>
|
|
336
338
|
```
|
|
337
339
|
|
|
340
|
+
Prompt and judge lists show active artifacts by default. Use `--status archived`
|
|
341
|
+
or `--status all` when you need archived artifacts.
|
|
342
|
+
|
|
343
|
+
Archive or restore a prompt:
|
|
344
|
+
|
|
345
|
+
```bash
|
|
346
|
+
orizu --local prompts archive <prompt-id-or-name> --project <team>/<project>
|
|
347
|
+
orizu --local prompts restore <prompt-id-or-name> --project <team>/<project>
|
|
348
|
+
```
|
|
349
|
+
|
|
338
350
|
List prompt comment threads for the latest version, or a specific label/version:
|
|
339
351
|
|
|
340
352
|
```bash
|
|
@@ -607,20 +619,19 @@ orizu --local optimizations run-gepa \
|
|
|
607
619
|
--train-split train \
|
|
608
620
|
--val-split validation \
|
|
609
621
|
--log-dir ./logs \
|
|
610
|
-
--budget light \
|
|
611
|
-
--max-iterations 3 \
|
|
612
622
|
--minibatch-size 3 \
|
|
613
623
|
--num-threads auto
|
|
614
624
|
```
|
|
615
625
|
|
|
616
626
|
Useful GEPA flags:
|
|
617
627
|
|
|
618
|
-
- `--budget auto|light|medium|heavy`, `--max-metric-calls <n>`, `--max-full-evals <n>`.
|
|
628
|
+
- Budget controls are mutually exclusive: choose at most one of `--budget auto|light|medium|heavy`, `--max-metric-calls <n>`, `--max-full-evals <n>`, or `--max-iterations <n>`. With none provided, `run-gepa` defaults to `--budget auto`, the balanced medium preset.
|
|
619
629
|
- `--minibatch-size <n>` defaults to 3.
|
|
620
|
-
- `--num-threads auto|N` defaults to `auto`; auto caps row-evaluation concurrency from mini-batch size, validation-set size, CPU count, memory estimate, file-descriptor limit, and
|
|
630
|
+
- `--num-threads auto|N` defaults to `auto`; auto caps row-evaluation concurrency from mini-batch size, validation-set size, 2x CPU count, memory estimate, file-descriptor limit, and a 64-thread default ceiling. Set `ORIZU_GEPA_AUTO_THREADS_MAX` or use `--num-threads <n>` only when the runner/provider capacity is known.
|
|
621
631
|
- `--candidate-selection-strategy pareto|current_best|epsilon_greedy`; default is `pareto`.
|
|
622
632
|
- `--reflection-model <provider/model>`, `--reflection-temperature <n>`, `--reflection-prompt-template <text|@file>`.
|
|
623
633
|
- `--reflection-max-tokens <n>` is explicit provider config, not a global default. It maps to Anthropic `max_tokens` and OpenAI `max_output_tokens`; Anthropic native Messages reflection requires it, while OpenAI can omit it unless the user wants a cap.
|
|
634
|
+
- `--reflection-retry-attempts` and `--reflection-http-timeout-seconds` tune transient reflection-provider retries. Exhausted retryable failures log `reflection_failed`, count against candidate-proposal budget, and continue with the next iteration.
|
|
624
635
|
- `--reflection-provider-settings <json|@file>` passes provider-native reflection settings separately from the prompt text. Anthropic example: `{"thinking":{"type":"adaptive","display":"omitted"},"output_config":{"effort":"medium"}}`. OpenAI example: `{"reasoning":{"effort":"medium","summary":"auto"}}`.
|
|
625
636
|
- `--disable-evaluation-cache` turns off candidate/row/scorer cache reuse.
|
|
626
637
|
- `--auto-promote --promotion-label <label>` promotes the best candidate at the end.
|
|
@@ -718,7 +729,8 @@ Rules:
|
|
|
718
729
|
- `eventId` must be stable and unique within one optimization run.
|
|
719
730
|
- `eventLayer` is `core`, `extension`, or `system`.
|
|
720
731
|
- Generic core events: `run_started`, `iteration_started`, `candidate_proposed`, `candidate_scored`, `candidate_recommended`, `iteration_completed`, `run_completed`, `run_failed`.
|
|
721
|
-
- GEPA extension events: `seed_val_set_started`, `seed_val_set_completed`, `parent_minibatch_started`, `parent_minibatch_completed`, `reflection_started`, `reflection_completed`, `child_candidate_created`, `child_minibatch_started`, `child_minibatch_completed`, `acceptance_decision_made`, `merge_started`, `merge_completed`.
|
|
732
|
+
- GEPA extension events: `seed_val_set_started`, `seed_val_set_completed`, `parent_minibatch_started`, `parent_minibatch_completed`, `reflection_started`, `reflection_completed`, `child_candidate_created`, `child_minibatch_started`, `child_minibatch_completed`, `acceptance_decision_made`, `optimization_progress`, `merge_started`, `merge_completed`.
|
|
733
|
+
- Emit `optimization_progress` after each completed iteration with `percent`, `metric_calls_used`, `metric_call_budget`, and `metric_calls_remaining`. Cap displayed `percent` at `100` and floor `metric_calls_remaining` at `0`; a started iteration may spend past the nominal metric-call budget before the optimizer pauses.
|
|
722
734
|
- Do not send per-LM-call telemetry to this endpoint in Phase 0. Put aggregate call/token/cost stats in iteration or run payloads.
|
|
723
735
|
|
|
724
736
|
Shell fallback:
|