orizu 0.5.21 → 0.5.22
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -6
- package/dist/gepa-engine-dispatch.d.ts.map +1 -1
- package/dist/gepa-engine-dispatch.js +20 -1
- package/dist/gepa-python-paths.d.ts.map +1 -1
- package/dist/gepa-python-paths.js +25 -6
- package/dist/help.d.ts.map +1 -1
- package/dist/help.js +36 -14
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +4 -39
- package/dist/instruction-set-loader.d.ts.map +1 -1
- package/dist/instruction-set-loader.js +57 -5
- package/dist/instruction-sets-cli.d.ts +5 -2
- package/dist/instruction-sets-cli.d.ts.map +1 -1
- package/dist/instruction-sets-cli.js +134 -24
- package/dist/skilled-proposer-config.d.ts +2 -0
- package/dist/skilled-proposer-config.d.ts.map +1 -0
- package/dist/skilled-proposer-config.js +212 -0
- package/dist/skilled-proposer-launch.d.ts +4 -0
- package/dist/skilled-proposer-launch.d.ts.map +1 -1
- package/dist/skilled-proposer-launch.js +65 -1
- package/package.json +1 -1
- package/src/gepa-engine-dispatch.ts +21 -1
- package/src/gepa-python-paths.ts +24 -6
- package/src/help.ts +36 -14
- package/src/index.ts +6 -53
- package/src/instruction-set-loader.ts +55 -5
- package/src/instruction-sets-cli.ts +173 -19
- package/src/skilled-proposer-config.ts +191 -0
- package/src/skilled-proposer-launch.ts +59 -1
- package/vendor/orizu-gepa/src/orizu_gepa_connector/engine.py +36 -3
- package/vendor/orizu-gepa/src/orizu_gepa_connector/instruction_set_loader.py +31 -3
- package/vendor/orizu-gepa/src/orizu_gepa_connector/skilled_proposer_bridge.py +354 -5
- package/vendor/skills/orizu-cli/SKILL.md +50 -16
- package/vendor/skills/orizu-cli/references/building-judges.md +2 -2
- package/vendor/skills/orizu-cli/references/cli-reference.md +57 -28
- package/vendor/skills/orizu-cli/references/instructions-after-prompts.md +120 -0
- package/vendor/skills/orizu-cli/references/migrate-existing-gepa-setup.md +109 -49
- package/vendor/skills/orizu-cli/references/optimization-reports.md +16 -11
- package/vendor/skills/orizu-cli/references/optimization-with-gepa.md +29 -28
- package/vendor/skills/orizu-cli/references/primer.md +21 -5
- package/vendor/skills/orizu-cli/references/prompt-control-plane.md +132 -85
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Instruction Control Plane
|
|
2
2
|
|
|
3
|
-
Use this reference for the Phase 0
|
|
3
|
+
Use this reference for the Phase 0 instruction, judge, scorer, runner, score, run, and optimization event control plane. These commands are for coding agents that need to push local artifacts into Orizu, run them locally, submit scores, and stream optimization progress back to the platform, except for instruction-set mutations and pointer moves explicitly marked for a human using the local CLI.
|
|
4
4
|
|
|
5
5
|
## Contents
|
|
6
6
|
|
|
@@ -16,13 +16,13 @@ Use this reference for the Phase 0 prompt, judge, scorer, runner, score, run, an
|
|
|
16
16
|
1. Verify auth: `orizu --local whoami`.
|
|
17
17
|
2. Export API context for scripts: `eval "$(orizu --local env --project <team>/<project>)"`.
|
|
18
18
|
3. Create immutable dataset versions before creating splits.
|
|
19
|
-
4. Push runners before
|
|
20
|
-
5. Register scorers after their backing prompt/runner exists, then bind headline/tracked scorers to
|
|
19
|
+
4. Push runners before preparing instruction-set handoffs, judges, or prompt-runner scorers; instruction components pin runner versions. A human curator runs the instruction-set mutation after the runner version exists.
|
|
20
|
+
5. Register scorers after their backing judge prompt/runner exists, then bind headline/tracked scorers to instruction components when the UI should show those metrics.
|
|
21
21
|
6. Use `runners exec` to prove the runner contract locally before submitting runs or score runs.
|
|
22
22
|
7. For common text-candidate optimization, prefer `orizu optimizations run-gepa`; it starts the run and logs events for you.
|
|
23
23
|
8. After `run-gepa`, inspect `logs/<optimization_run_id>` first; it is the complete local trace for coding-agent analysis.
|
|
24
24
|
9. Use `orizu optimizations export <run-id> --out <run-id>.optimization.json` when the local log is missing or the run happened elsewhere.
|
|
25
|
-
10.
|
|
25
|
+
10. Create a profile version from an accepted candidate with `orizu optimizations promote <run-id> --candidate <id>`; output reports its profile version and each component as changed or carried. Give that version to a human to inspect and move the production pointer.
|
|
26
26
|
11. Write and attach a markdown report for finished, failed, or cancelled runs; use `optimization-reports.md` for structure and diagnostic guidance.
|
|
27
27
|
12. For custom optimizers, start an optimization run before local execution, then stream events into that run.
|
|
28
28
|
13. Use bare HTTP for optimization events; use `orizu log` only as a shell fallback.
|
|
@@ -32,31 +32,43 @@ Customer model-provider secrets stay local. Do not upload Anthropic/OpenAI/etc.
|
|
|
32
32
|
|
|
33
33
|
## Instruction sets
|
|
34
34
|
|
|
35
|
-
An instruction set
|
|
36
|
-
It has one profile per model config. Existing prompts appear as
|
|
37
|
-
write them with
|
|
35
|
+
An instruction set has a fixed, ordered shape of named components (key → text).
|
|
36
|
+
It has one profile per model config. Existing prompts appear as one-component
|
|
37
|
+
sets. Inspect or write them with the customer-facing `orizu instructions`
|
|
38
|
+
namespace:
|
|
38
39
|
|
|
39
40
|
```bash
|
|
40
|
-
orizu
|
|
41
|
-
orizu
|
|
42
|
-
|
|
43
|
-
orizu
|
|
44
|
-
orizu instruction-
|
|
45
|
-
orizu
|
|
46
|
-
|
|
47
|
-
orizu
|
|
48
|
-
orizu
|
|
49
|
-
orizu
|
|
50
|
-
|
|
41
|
+
orizu instructions list --project core/evals --status active --json
|
|
42
|
+
orizu instructions show planner --project core/evals --status active --json
|
|
43
|
+
# Human/local CLI only: a human runs create/push; hosted agents may run profiles new.
|
|
44
|
+
orizu instructions create ./orizu.instruction-set.json --project core/evals --model-config anthropic/claude-haiku
|
|
45
|
+
orizu instructions push ./orizu.instruction-set.json --project core/evals --set planner
|
|
46
|
+
orizu instructions profiles new planner --project core/evals --model-config anthropic/claude-haiku
|
|
47
|
+
# Human/local CLI only: profile production-pointer moves.
|
|
48
|
+
orizu instructions profiles promote planner --project core/evals --model-config anthropic/claude-haiku --version 2
|
|
49
|
+
orizu instructions profiles rollback planner --project core/evals --model-config anthropic/claude-haiku --to 1
|
|
50
|
+
orizu instructions default show planner --project core/evals
|
|
51
|
+
# Human/local CLI only: the set-wide default-pointer move.
|
|
52
|
+
orizu instructions default move planner --project core/evals --model-config anthropic/claude-haiku --version 2
|
|
53
|
+
# Human/local CLI only: shape mutations.
|
|
54
|
+
orizu instructions shape add planner --project core/evals --key safety --from ./orizu.instruction-set.json
|
|
55
|
+
orizu instructions shape remove planner --project core/evals --key safety
|
|
56
|
+
# Human/local CLI only: visibility mutations.
|
|
57
|
+
orizu instructions archive planner --project core/evals --json
|
|
58
|
+
orizu instructions restore planner --project core/evals --json
|
|
51
59
|
```
|
|
52
60
|
|
|
53
|
-
The manifest contains `name`,
|
|
54
|
-
|
|
55
|
-
component
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
61
|
+
The manifest contains `name`, optional unversioned `description`, the ordered
|
|
62
|
+
component-key `shape`, and `components`; every shape key needs a set-wide
|
|
63
|
+
component (`key` plus `text` or a manifest-relative `path`). A component may
|
|
64
|
+
additionally specify `modelConfig` to override that key in a named profile.
|
|
65
|
+
Create materializes each named profile with its complete base-plus-override
|
|
66
|
+
component map. The default starts on the seed profile version. A profile with a
|
|
67
|
+
production version resolves to that component map; a profile without one
|
|
68
|
+
resolves to the default. Sets accept either their stable slug or exact display
|
|
69
|
+
name. Archiving changes list visibility only, so archived sets still resolve
|
|
70
|
+
and sync; use `--status archived` or `--status all` to find them before
|
|
71
|
+
restoring.
|
|
60
72
|
|
|
61
73
|
Shape changes create unpromoted profile heads but leave the default and
|
|
62
74
|
production pointers in place. The instruction set does not resolve for affected
|
|
@@ -69,21 +81,29 @@ version number, and the database rejects an unsealed or non-commit-anchored
|
|
|
69
81
|
component map. Shape changes create a new `shape_change` version for every profile;
|
|
70
82
|
they do not repoint the default or any production label.
|
|
71
83
|
|
|
72
|
-
Sync an offline runner directory with `orizu
|
|
73
|
-
--out ./instructions --project core/evals`. The layout is
|
|
84
|
+
Sync an offline runner directory with `orizu instructions sync planner
|
|
85
|
+
--out ./instructions --project core/evals`. The layout root is the stable set
|
|
86
|
+
slug (`./instructions/planner/`), containing `manifest.json`,
|
|
74
87
|
`default/<key>.md`, and `profiles/<identity-slug>/<key>.md`; loaders are
|
|
75
|
-
`loadInstructionSet(dir,
|
|
76
|
-
`load_instruction_set(dir,
|
|
88
|
+
`loadInstructionSet(dir, setReference, modelConfigIdentity)` in TypeScript and
|
|
89
|
+
`load_instruction_set(dir, set_reference, model_config_identity)` in Python.
|
|
90
|
+
Prefer the stable slug; exact manifest names remain accepted for compatibility. A
|
|
77
91
|
Git-pinned component is recorded in the manifest but is intentionally not
|
|
78
92
|
downloaded; either loader raises `instruction_set_component_unavailable`. A
|
|
79
93
|
missing or unreadable local component file is distinct:
|
|
80
94
|
`instruction_set_component_unreadable`.
|
|
81
95
|
|
|
96
|
+
The sync manifest carries immutable `projectId` and `instructionSetId`; modern
|
|
97
|
+
sync replacement requires both identities as well as the slug to match.
|
|
98
|
+
Identity-less pre-slug trees are never auto-migrated: sync preserves them and
|
|
99
|
+
returns `instruction_set_sync_legacy_identity_required`, requiring a verified
|
|
100
|
+
clean output root or an explicit operator move.
|
|
101
|
+
|
|
82
102
|
Use `profiles new` to seed an additional model-config profile from the default;
|
|
83
103
|
hosted agents may do this because it does not move production. Use `profiles
|
|
84
104
|
promote` to move only that profile's production label; use `profiles rollback
|
|
85
105
|
--to <n>` to create a new rollback version from the target version's component
|
|
86
|
-
|
|
106
|
+
map **and settings version** before moving it. Promotion and rollback are
|
|
87
107
|
human-only because they move production labels.
|
|
88
108
|
|
|
89
109
|
For a single-component instruction set that wraps an existing prompt, only a
|
|
@@ -96,7 +116,7 @@ label` when a profile command performed this mirror.
|
|
|
96
116
|
Execution/privacy defaults:
|
|
97
117
|
|
|
98
118
|
- Runner subprocesses receive the input/output file-contract paths plus a small allowlist of provider/runtime environment variables. Orizu API tokens are not passed into runner processes. When an execution carries an instruction set, `ORIZU_INSTRUCTION_SET_DIR` is set explicitly to a fresh per-row synced layout; it is absent for legacy executions and is never inherited from the shell.
|
|
99
|
-
- `orizu optimizations run-gepa` redacts dataset row payloads and reflection text in logged events by default. Use `--log-row-snapshots` only when the customer explicitly wants raw row and
|
|
119
|
+
- `orizu optimizations run-gepa` redacts dataset row payloads and reflection text in logged events by default. Use `--log-row-snapshots` only when the customer explicitly wants raw row and component text in the optimization event stream.
|
|
100
120
|
- `run-gepa` still writes complete local traces under `logs/<optimization_run_id>` by default. Treat those logs as sensitive: they include row inputs, model outputs, scores, feedback, scorer responses, reflection prompts, reflection responses, and candidate text.
|
|
101
121
|
- Runner artifacts, runner output, score result uploads, and optimization event payloads are size-capped. If a run needs larger observability payloads, store the large artifact separately and log a pointer.
|
|
102
122
|
|
|
@@ -116,7 +136,7 @@ Required file: `manifest.json`.
|
|
|
116
136
|
}
|
|
117
137
|
```
|
|
118
138
|
|
|
119
|
-
For `runners exec`, the command must read input JSON from `ORIZU_RUNNER_INPUT_PATH` and write output JSON to `ORIZU_RUNNER_OUTPUT_PATH`. When present, `ORIZU_INSTRUCTION_SET_DIR` contains the same layout written by `
|
|
139
|
+
For `runners exec`, the command must read input JSON from `ORIZU_RUNNER_INPUT_PATH` and write output JSON to `ORIZU_RUNNER_OUTPUT_PATH`. When present, `ORIZU_INSTRUCTION_SET_DIR` contains the same layout written by `instructions sync`; use `loadInstructionSet`/`load_instruction_set` to read materialized component text. Pinned components are represented in the manifest and are intentionally not materialized as `.md` files.
|
|
120
140
|
|
|
121
141
|
Input shape:
|
|
122
142
|
|
|
@@ -237,36 +257,15 @@ seed before iterating. If the seed scores the worst possible value on every
|
|
|
237
257
|
validation row (0.0 for higher-is-better, 1.0 for lower-is-better), errors on
|
|
238
258
|
every row, or returns nothing parseable, the run fails loudly with a
|
|
239
259
|
contract-mismatch diagnosis instead of silently burning budget on zeroed
|
|
240
|
-
candidates. A uniformly-worst seed is almost always a harness bug, not
|
|
241
|
-
|
|
260
|
+
candidates. A uniformly-worst seed is almost always a harness bug, not bad
|
|
261
|
+
component values; override with `--allow-degenerate-seed` only when the seed genuinely
|
|
242
262
|
deserves the worst score everywhere.
|
|
243
263
|
|
|
244
|
-
###
|
|
264
|
+
### Judge Directory
|
|
245
265
|
|
|
246
266
|
Required files: `orizu.prompt.json` plus the body file referenced by `body_file`.
|
|
247
267
|
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
```json
|
|
251
|
-
{
|
|
252
|
-
"name": "hip-note-generator",
|
|
253
|
-
"role": "production_inference",
|
|
254
|
-
"description": "Generates a HIP note label.",
|
|
255
|
-
"body_file": "prompt.md",
|
|
256
|
-
"body_kind": "text",
|
|
257
|
-
"version_label": "v1",
|
|
258
|
-
"provider_settings": {
|
|
259
|
-
"model": "claude-sonnet-4-6",
|
|
260
|
-
"temperature": 0,
|
|
261
|
-
"max_tokens": 4096
|
|
262
|
-
},
|
|
263
|
-
"provenance": {
|
|
264
|
-
"kind": "coding-agent-edit"
|
|
265
|
-
}
|
|
266
|
-
}
|
|
267
|
-
```
|
|
268
|
-
|
|
269
|
-
Judge prompt:
|
|
268
|
+
Judge artifact:
|
|
270
269
|
|
|
271
270
|
```json
|
|
272
271
|
{
|
|
@@ -461,11 +460,16 @@ orizu --local runners exec \
|
|
|
461
460
|
|
|
462
461
|
Omit `--runner-dir` to download and materialize the pinned runner version from Orizu. `--out` may end in `.jsonl` or `.jsonl.gz`.
|
|
463
462
|
|
|
464
|
-
###
|
|
463
|
+
### Instructions And Judges
|
|
464
|
+
|
|
465
|
+
Prepare the complete instruction-set manifest and hand it to a human curator;
|
|
466
|
+
the judge remains an agent-writable git-canonical artifact.
|
|
465
467
|
|
|
466
468
|
```bash
|
|
467
|
-
|
|
469
|
+
# Human/local CLI only: a human curator with a user token runs this mutation.
|
|
470
|
+
orizu --local instructions push ./orizu.instruction-set.json \
|
|
468
471
|
--project <team>/<project> \
|
|
472
|
+
--set <slug-or-exact-name> \
|
|
469
473
|
--runner-version <runner-version-id> \
|
|
470
474
|
--json
|
|
471
475
|
|
|
@@ -475,22 +479,27 @@ orizu --local judges push ./judge \
|
|
|
475
479
|
--json
|
|
476
480
|
```
|
|
477
481
|
|
|
478
|
-
|
|
482
|
+
Instruction push returns the updated set and profile versions. Judge push
|
|
483
|
+
returns `prompt_version_id` for the judge artifact used by scorer bindings.
|
|
479
484
|
|
|
480
485
|
List:
|
|
481
486
|
|
|
482
487
|
```bash
|
|
483
|
-
orizu --local
|
|
484
|
-
orizu --local
|
|
485
|
-
orizu --local
|
|
488
|
+
orizu --local instructions list --project <team>/<project>
|
|
489
|
+
orizu --local instructions list --project <team>/<project> --status archived
|
|
490
|
+
orizu --local instructions list --project <team>/<project> --status all
|
|
486
491
|
orizu --local judges list --project <team>/<project>
|
|
487
492
|
```
|
|
488
493
|
|
|
489
|
-
|
|
490
|
-
or `--status all` when you need archived artifacts.
|
|
494
|
+
Instruction and judge lists show active artifacts by default. Use `--status
|
|
495
|
+
archived` or `--status all` when you need archived artifacts. Existing prompts
|
|
496
|
+
appear in the instruction list as one-component sets; see
|
|
497
|
+
`instructions-after-prompts.md` for ownership output, deliberate standalone
|
|
498
|
+
mutation refusals, and manual consolidation.
|
|
491
499
|
|
|
492
|
-
|
|
493
|
-
`LINES`, `CHARS`, and `WORDS`, measured from the latest
|
|
500
|
+
Compatibility prompt and judge list tables include `ID`, `NAME`, `ROLE`,
|
|
501
|
+
`STATUS`, `TOKENS`, `LINES`, `CHARS`, and `WORDS`, measured from the latest
|
|
502
|
+
sealed version. `—`
|
|
494
503
|
means the canonical body could not be measured; zero remains `0`. The `~`
|
|
495
504
|
prefix on token counts means approximate: Orizu uses one fixed, model-agnostic
|
|
496
505
|
`gpt-tokenizer` encoding rather than claiming an exact count for every model.
|
|
@@ -504,13 +513,18 @@ first 500 sorted summaries are enriched per request, and canonical-body
|
|
|
504
513
|
resolution has a 15-second server budget. A summary skipped by either bound
|
|
505
514
|
carries null stats and `measurement_cap_exceeded`.
|
|
506
515
|
|
|
507
|
-
Archive or restore
|
|
516
|
+
Archive or restore an instruction set by stable slug or exact name. A coding
|
|
517
|
+
agent prepares the exact set reference and hands the mutation to a human:
|
|
508
518
|
|
|
509
519
|
```bash
|
|
510
|
-
|
|
511
|
-
orizu --local
|
|
520
|
+
# Human/local CLI only: a human curator with a user token runs these mutations.
|
|
521
|
+
orizu --local instructions archive <slug-or-exact-name> --project <team>/<project>
|
|
522
|
+
orizu --local instructions restore <slug-or-exact-name> --project <team>/<project>
|
|
512
523
|
```
|
|
513
524
|
|
|
525
|
+
Archive and restore affect visibility only. Archived sets still resolve and
|
|
526
|
+
sync, and can be inspected with `instructions show --status archived|all`.
|
|
527
|
+
|
|
514
528
|
List prompt comment threads for the latest version, or a specific label/version:
|
|
515
529
|
|
|
516
530
|
```bash
|
|
@@ -520,14 +534,19 @@ orizu --local comments list --prompt <prompt-id-or-name> \
|
|
|
520
534
|
[--json]
|
|
521
535
|
```
|
|
522
536
|
|
|
523
|
-
Human output shows the thread count, open/resolved counts, selected
|
|
537
|
+
Human output shows the thread count, open/resolved counts, selected component
|
|
538
|
+
text or source line, each top-level comment body, and replies. Use `--json` when
|
|
539
|
+
an agent or script needs structured `summary` and `comments` data. Check
|
|
540
|
+
unresolved comments before drafting or pushing the next instruction version.
|
|
524
541
|
|
|
525
|
-
Move
|
|
542
|
+
Move an instruction profile's production pointer (human-only):
|
|
526
543
|
|
|
527
544
|
```bash
|
|
528
|
-
|
|
545
|
+
# Human/local CLI only: the human curator moves the production pointer.
|
|
546
|
+
orizu --local instructions profiles promote hip-note-judge \
|
|
529
547
|
--project <team>/<project> \
|
|
530
|
-
--
|
|
548
|
+
--model-config <identity> \
|
|
549
|
+
--version <n> \
|
|
531
550
|
--json
|
|
532
551
|
```
|
|
533
552
|
|
|
@@ -682,18 +701,27 @@ orizu --local scores submit ./candidate-scores.jsonl \
|
|
|
682
701
|
--json
|
|
683
702
|
```
|
|
684
703
|
|
|
685
|
-
Bind scorers to
|
|
704
|
+
Bind scorers to the instruction component that owns the evaluated text.
|
|
705
|
+
Scorer binding is ordinary curator work, not one of the human-only catalog
|
|
706
|
+
mutations, so hosted agents may set the headline scorer:
|
|
686
707
|
|
|
687
708
|
```bash
|
|
688
|
-
orizu --local
|
|
709
|
+
orizu --local instructions scorers set-headline <set> \
|
|
710
|
+
--key <component-key> \
|
|
689
711
|
--project <team>/<project> \
|
|
690
712
|
--scorer-version <scorer-version-id> \
|
|
691
713
|
--dataset-version <dataset-version-id> \
|
|
692
714
|
--split-set <split-set-id> \
|
|
693
715
|
--split validation \
|
|
694
716
|
--json
|
|
717
|
+
```
|
|
695
718
|
|
|
696
|
-
|
|
719
|
+
For the same reason, hosted agents may add tracked scorers to the
|
|
720
|
+
set-and-component address:
|
|
721
|
+
|
|
722
|
+
```bash
|
|
723
|
+
orizu --local instructions scorers add <set> \
|
|
724
|
+
--key <component-key> \
|
|
697
725
|
--project <team>/<project> \
|
|
698
726
|
--scorer-version <scorer-version-id> \
|
|
699
727
|
--dataset-version <dataset-version-id> \
|
|
@@ -702,6 +730,12 @@ orizu --local prompts scorers add <prompt-id> \
|
|
|
702
730
|
--json
|
|
703
731
|
```
|
|
704
732
|
|
|
733
|
+
The legacy `prompts scorers set-headline` and `prompts scorers add` spellings
|
|
734
|
+
refuse before making a request. Their measured replacements are:
|
|
735
|
+
|
|
736
|
+
- `Use: orizu instructions scorers set-headline <set> --key <component-key> --scorer-version <id> --project <team/project>`
|
|
737
|
+
- `Use: orizu instructions scorers add <set> --key <component-key> --scorer-version <id> --project <team/project>`
|
|
738
|
+
|
|
705
739
|
### Runs
|
|
706
740
|
|
|
707
741
|
Submit local runner output:
|
|
@@ -772,7 +806,9 @@ For bundled text-candidate GEPA-style optimization, let the CLI manage start/eve
|
|
|
772
806
|
orizu --local optimizations run-gepa \
|
|
773
807
|
--project <team>/<project> \
|
|
774
808
|
--optimizer-version-id <optimizer-version-id> \
|
|
775
|
-
--
|
|
809
|
+
--instruction-set <slug-or-exact-name> \
|
|
810
|
+
--model-config <identity> \
|
|
811
|
+
--component-selector all \
|
|
776
812
|
--runner-version-id <runner-version-id> \
|
|
777
813
|
--candidate-runner-dir ./candidate-runner \
|
|
778
814
|
--scorer-version-id <row-scorer-version-id> \
|
|
@@ -790,6 +826,12 @@ orizu --local optimizations run-gepa \
|
|
|
790
826
|
|
|
791
827
|
Useful GEPA flags:
|
|
792
828
|
|
|
829
|
+
- `--instruction-set <slug-or-exact-name> --model-config <identity>` selects a
|
|
830
|
+
resolving instruction-set profile. `--component-selector round-robin|all`
|
|
831
|
+
controls which components GEPA proposes changing; `all` lets it optimize the
|
|
832
|
+
complete component map. `--candidate-version-id` remains a compatibility
|
|
833
|
+
input for a single legacy prompt version and is mutually exclusive with the
|
|
834
|
+
instruction-set selector.
|
|
793
835
|
- `--engine official|legacy` defaults to `official`, which runs the bundled official GEPA connector. Use `legacy` only as the frozen recovery hatch; it keeps the historical Python loop and does not support `--max-candidate-proposals` or the skilled proposer.
|
|
794
836
|
- `--scorer-input-contract gepa|flat_row` selects the scorer-runner input shape. Use `flat_row` to reuse a judge runner written for `runners exec --scorer-version` without a hand-written adapter; add `--scorer-candidate-field <row-field>` when that judge reads the candidate output from a specific row field (e.g. `draft`). Passing a candidate field under the `gepa` contract is refused at launch, not silently ignored. See "Scorer-Runner Input Contracts" above.
|
|
795
837
|
- `--allow-degenerate-seed` opts out of the launch-time refusal when the seed scores the worst possible value on every validation row. Leave it off by default — a uniformly-worst seed is almost always a scorer contract mismatch.
|
|
@@ -801,9 +843,9 @@ Useful GEPA flags:
|
|
|
801
843
|
- `--reflection-model <provider/model>`, `--reflection-temperature <n>`, `--reflection-prompt-template <text|@file>`.
|
|
802
844
|
- `--reflection-max-tokens <n>` is explicit provider config, not a global default. It maps to Anthropic `max_tokens` and OpenAI `max_output_tokens`; Anthropic native Messages reflection requires it, while OpenAI can omit it unless the user wants a cap.
|
|
803
845
|
- `--reflection-retry-attempts` and `--reflection-http-timeout-seconds` tune transient reflection-provider retries. Exhausted retryable failures log `reflection_failed`, count against candidate-proposal budget, and continue with the next iteration.
|
|
804
|
-
- `--reflection-provider-settings <json|@file>` passes provider-native reflection settings separately from the
|
|
846
|
+
- `--reflection-provider-settings <json|@file>` passes provider-native reflection settings separately from the component text. Anthropic example: `{"thinking":{"type":"adaptive","display":"omitted"},"output_config":{"effort":"medium"}}`. OpenAI example: `{"reasoning":{"effort":"medium","summary":"auto"}}`.
|
|
805
847
|
- `--disable-evaluation-cache` turns off candidate/row/scorer cache reuse.
|
|
806
|
-
- `--auto-promote --promotion-label <label>`
|
|
848
|
+
- `--auto-promote --promotion-label <label>` can move a label to the best candidate at the end. Coding agents must not pass either flag; hand the accepted profile version to a human for any production-pointer move.
|
|
807
849
|
- `--log-row-snapshots` includes raw row and reflection text in events; leave off by default.
|
|
808
850
|
- `--log-dir <dir>` controls the local log root; default is `logs`.
|
|
809
851
|
- `--no-local-log` disables local trace files. Use this only when the environment must not persist raw rows or reflection context.
|
|
@@ -812,6 +854,10 @@ Useful GEPA flags:
|
|
|
812
854
|
|
|
813
855
|
`--candidate-proposer skilled-proposer` selects the skilled proposer; omission keeps the default official proposer. The flag is official-engine-only and accepts that one value. A missing value is refused before launch as `--candidate-proposer requires a value that does not start with --`; another value is refused as `--candidate-proposer must be skilled-proposer`; selecting it with `--engine legacy` is refused with `--candidate-proposer is supported by the official GEPA engine only`.
|
|
814
856
|
|
|
857
|
+
`--candidate-proposer-config <@file|path>` is optional and requires that selection on the official engine. Its strict schema-v1 object requires `schemaVersion: 1` and `skills: []`; each ordered skill is exactly either `{ "path": "..." }` (with optional `name`/`description` overrides) or `{ "name": "...", "description": "...", "inline": "..." }`. Optional fields are `additionalInstructionsFile`, `baseInstructionsFile`, positive-integer-or-null `maxWords`, `maxTokens`, and `maxExamples`, plus `onError: "keep"|"raise"`. Paths are config-file-relative; directories read only `SKILL.md`. Missing, unreadable, empty, malformed, ambiguous, duplicate-name, and unknown-field inputs fail before launch. `baseInstructionsFile` is advanced use: it replaces upstream's built-in anti-overfitting meta-prompt.
|
|
858
|
+
|
|
859
|
+
`--reflection-prompt-template` is incompatible whenever skilled-proposer is selected, with or without a proposer config; it configures the ordinary GEPA reflection path and would otherwise be silently ignored. The three token controls have different scopes: `--reflection-max-tokens` caps one provider response, config `maxTokens` caps a proposed instruction, and `--proposal-max-tokens` caps aggregate proposal-provider usage.
|
|
860
|
+
|
|
815
861
|
`--proposal-max-calls <positive-integer>` and `--proposal-max-tokens <positive-integer>` default to unset and are valid only with the skilled proposer. A missing operand names that flag; using either ceiling without the selection produces `--proposal-max-calls and --proposal-max-tokens require --candidate-proposer skilled-proposer`; the legacy-engine error names the individual flag. Integer and positive-value validation happens in the connector after launch and names `ORIZU_PROPOSAL_MAX_CALLS` or `ORIZU_PROPOSAL_MAX_TOKENS`.
|
|
816
862
|
|
|
817
863
|
The call ceiling counts each transport-bearing proposer bridge call after its success or failure is recorded; provider retries inside one bridge call remain one counted call. The token ceiling uses accumulated provider-derived `total_tokens`, including recorded usage from unsuccessful completions. Both ceilings stop new proposal work at the next safe boundary, so an in-flight call can meet or pass a ceiling. They govern only the skilled proposer and remain independent of overall GEPA controls such as `--max-metric-calls`; set at least one proposal ceiling and one overall GEPA ceiling, then start the run only when both limits match the intended experiment.
|
|
@@ -854,10 +900,11 @@ orizu --local optimizations export <optimization-run-id> \
|
|
|
854
900
|
|
|
855
901
|
Use export when the local log is unavailable, the run happened on another machine, or a coding agent needs a portable single JSON artifact. The export fetches all optimization events, derives seed vs best, Pareto frontier, score-over-time, candidates, iterations, minibatch rows, and validation rows, and rehydrates row inputs from the dataset version artifact when possible. Server events may not contain row snapshots or reflection prompts unless the run used `--log-row-snapshots`; reflection responses are included for bundled `run-gepa` runs.
|
|
856
902
|
|
|
857
|
-
Both diff-comment read routes (`/api/optimization-runs/<run-id>/diff-comments`
|
|
858
|
-
`/api/prompts/<prompt-id>/diff-comments`) return `componentKey` for
|
|
859
|
-
comments. Preserve it on writes and use it in grouping keys:
|
|
860
|
-
from different instruction-set components are distinct
|
|
903
|
+
Both diff-comment read routes (`/api/optimization-runs/<run-id>/diff-comments`
|
|
904
|
+
and `/api/prompts/<prompt-id>/diff-comments`) return `componentKey` for
|
|
905
|
+
multi-component comments. Preserve it on writes and use it in grouping keys:
|
|
906
|
+
equal line ranges from different instruction-set components are distinct
|
|
907
|
+
comments.
|
|
861
908
|
|
|
862
909
|
The v1 export keeps the run row's `best_candidate_id` in
|
|
863
910
|
`summary.bestCandidateId` when event derivation rejects that identifier as an
|