@evolvingmachines/evolve 0.0.67 → 0.0.68
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/chunk-FHL4GIAZ.js +1 -0
- package/dist/chunk-ORD7JDZB.js +1 -0
- package/dist/chunk-SQDR2TAW.js +427 -0
- package/dist/cli/index.cjs +38 -38
- package/dist/cli/index.d.cts +1 -1
- package/dist/cli/index.d.ts +1 -1
- package/dist/cli/index.js +30 -30
- package/dist/index.cjs +65 -60
- package/dist/index.d.cts +332 -29
- package/dist/index.d.ts +332 -29
- package/dist/index.js +48 -43
- package/dist/managed-modal-IXZBT4D5.js +4 -0
- package/dist/resumable-XNW5SKRW.js +1 -0
- package/dist/{types-84M4J_Vm.d.cts → types-DGx_qo9M.d.cts} +118 -77
- package/dist/{types-84M4J_Vm.d.ts → types-DGx_qo9M.d.ts} +118 -77
- package/harness-capabilities.json +325 -77
- package/hosted-error-codes.json +1 -0
- package/package.json +9 -7
- package/skills/create-adapter/SKILL.md +1 -1
- package/skills/create-task/SKILL.md +1 -1
- package/skills/evolve/SKILL.md +1 -1
- package/skills/evolve-evals/references/cli-reference/agent.md +1 -1
- package/skills/evolve-evals/references/cli-reference/analyze.md +6 -3
- package/skills/evolve-evals/references/cli-reference/check.md +4 -3
- package/skills/evolve-evals/references/cli-reference/index.md +1 -1
- package/skills/evolve-evals/references/cli-reference/run.md +4 -3
- package/skills/evolve-evals/references/core-concepts/agents.md +7 -3
- package/skills/evolve-evals/references/core-concepts/analyze.md +2 -2
- package/skills/evolve-evals/references/core-concepts/check.md +2 -2
- package/skills/evolve-evals/references/core-concepts/datasets.md +1 -1
- package/skills/evolve-evals/references/core-concepts/index.md +1 -1
- package/skills/evolve-evals/references/core-concepts/jobs.md +3 -3
- package/skills/evolve-evals/references/core-concepts/models.md +8 -9
- package/skills/evolve-evals/references/core-concepts/sandboxes.md +1 -1
- package/skills/evolve-evals/references/core-concepts/secrets.md +1 -1
- package/skills/evolve-evals/references/core-concepts/skills.md +1 -1
- package/skills/evolve-evals/references/core-concepts/task-verifiers.md +1 -1
- package/skills/evolve-evals/references/core-concepts/teams.md +1 -1
- package/skills/evolve-evals/references/core-concepts/upload.md +1 -1
- package/skills/evolve-evals/references/getting-started/first-task.md +1 -1
- package/skills/evolve-evals/references/getting-started/quick-start.md +1 -1
- package/skills/evolve-evals/references/index.md +1 -1
- package/skills/evolve-evals/references/sdk/http.md +1 -1
- package/skills/evolve-evals/references/sdk/python.md +1 -1
- package/skills/evolve-evals/references/sdk/typescript.md +1 -1
- package/skills/evolve-evals/references/sdk-reference/analyses.md +11 -4
- package/skills/evolve-evals/references/sdk-reference/checks.md +11 -4
- package/skills/evolve-evals/references/sdk-reference/error-codes.md +1 -0
- package/skills/evolve-evals/references/sdk-reference/jobs.md +2 -2
- package/skills/evolve-evals/references/sdk-reference/meta.md +2 -1
- package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +6 -4
- package/skills/evolve-evals/references/sdk-reference/methods/checks.md +8 -4
- package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +5 -3
- package/skills/evolve-evals/references/sdk-reference/methods/meta.md +15 -4
- package/skills/evolve-evals/references/sdk-reference/skills.md +2 -2
- package/skills/evolve-evals/references/sdk-reference/types.md +11 -5
- package/spec/openapi.yaml +290 -137
- package/dist/chunk-HQJI3HHU.js +0 -1
- package/dist/chunk-RFLMRPGK.js +0 -427
- package/dist/managed-modal-IJW62SSK.js +0 -4
- package/dist/resumable-HFSICSH3.js +0 -1
package/spec/openapi.yaml
CHANGED
|
@@ -101,9 +101,9 @@ info:
|
|
|
101
101
|
directory and runs it as a new local Harbor job
|
|
102
102
|
(their src/harbor/analyze/analyzer.py `run_analyze`). Here analysis is
|
|
103
103
|
a NATIVE capability of the platform — the worker runs the same
|
|
104
|
-
analyzer (claude-code over the trial's
|
|
105
|
-
task, rubric-driven, `analysis.json`
|
|
106
|
-
criterion) server-side, and the result is stored and served on the
|
|
104
|
+
analyzer (the chosen agent, claude-code by default, over the trial's
|
|
105
|
+
Harbor-shape tree plus its task, rubric-driven, `analysis.json`
|
|
106
|
+
validated criterion-for-criterion) server-side, and the result is stored and served on the
|
|
107
107
|
trial body (`Trial.analysis`) instead of written into a local
|
|
108
108
|
directory. Two named deviations from Harbor's shape: (1) server-side
|
|
109
109
|
execution — no wrapper job appears anywhere on this surface; the
|
|
@@ -111,8 +111,8 @@ info:
|
|
|
111
111
|
`JobCreate.analyze` runs the analyzer per trial as each trial settles,
|
|
112
112
|
which Harbor has no equivalent of (their analyze is always a manual
|
|
113
113
|
follow-up command; `POST /api/jobs/{jobId}/analyze` is that manual
|
|
114
|
-
verb here). Vocabulary is Harbor's: `model_name`, rubric criteria
|
|
115
|
-
`{name, description, guidance}` (their cli/analyze.py `--model` /
|
|
114
|
+
verb here). Vocabulary is Harbor's: `agent`, `model_name`, rubric criteria
|
|
115
|
+
`{name, description, guidance}` (their cli/analyze.py `--agent` / `--model` /
|
|
116
116
|
`--rubric`, cli/quality_checker/models.py RubricCriterion), result
|
|
117
117
|
`{summary, checks{outcome, explanation}, estimated_cost_usd}` (their
|
|
118
118
|
analyze/models.py AnalyzeResult). Selection and width are Harbor's
|
|
@@ -132,7 +132,7 @@ info:
|
|
|
132
132
|
own record: the caller uploads the task directory (or a directory of
|
|
133
133
|
task directories) as one archive (`POST /api/checks`, multipart —
|
|
134
134
|
`archive` + `config`), the worker runs the SAME checker per task
|
|
135
|
-
(claude-code over the task's byte-sacred tree, the platform's default
|
|
135
|
+
(the chosen agent, claude-code by default, over the task's byte-sacred tree, the platform's default
|
|
136
136
|
check prompt and rubric rendered through Harbor's own tokens,
|
|
137
137
|
`check-result.json` validated criterion for criterion) server-side,
|
|
138
138
|
and the report is served as the `Check`
|
|
@@ -140,12 +140,14 @@ info:
|
|
|
140
140
|
deviations: server-side execution (no wrapper job on this surface);
|
|
141
141
|
a stored record instead of a local report file; a typed `failure
|
|
142
142
|
{phase, message}` per task where Harbor's QualityCheckResult carries
|
|
143
|
-
`error: str` (the TrialAnalysis dialect);
|
|
144
|
-
|
|
145
|
-
`
|
|
143
|
+
`error: str` (the TrialAnalysis dialect); an omitted model resolving
|
|
144
|
+
as the analyzer's — the platform's pick,
|
|
145
|
+
`openrouter/deepseek/deepseek-v4.1-flash`, on every agent whose roster
|
|
146
|
+
carries it, else the agent's own default — where Harbor's check
|
|
147
|
+
defaults to `claude-sonnet-4-6` (one rule for both rubric agents);
|
|
146
148
|
and the platform's `reasoning_effort` and `sandbox_provider` beside
|
|
147
149
|
Harbor's knobs, exactly as on `AnalyzeConfigInput`. Vocabulary is
|
|
148
|
-
Harbor's: `model_name`, `rubric`, `prompt` (their cli/analyze.py:88-
|
|
150
|
+
Harbor's: `agent`, `model_name`, `rubric`, `prompt` (their cli/analyze.py:88-
|
|
149
151
|
103), `n_concurrent` (:117-119), `include_task_names` /
|
|
150
152
|
`exclude_task_names` / `n_tasks` (:121-131; fnmatch globs and the cap
|
|
151
153
|
of checker.py:132-138), result `{task_name, checks{outcome,
|
|
@@ -1459,9 +1461,11 @@ paths:
|
|
|
1459
1461
|
│ │ at Harbor's own tee name for the
|
|
1460
1462
|
│ │ harness (claude-code.txt, codex.txt,
|
|
1461
1463
|
│ │ gemini-cli.txt, qwen-code.txt,
|
|
1462
|
-
│ │ kimi-code.txt, opencode.txt
|
|
1463
|
-
│ │ droid.txt,
|
|
1464
|
-
│ │
|
|
1464
|
+
│ │ kimi-code.txt, opencode.txt, pi.txt,
|
|
1465
|
+
│ │ antigravity-stream.jsonl; droid.txt,
|
|
1466
|
+
│ │ prime-agent.txt, dsh.txt, zcode.txt, and
|
|
1467
|
+
│ │ stdout.log for a harness Harbor has
|
|
1468
|
+
│ │ no name for)
|
|
1465
1469
|
│ ├── stderr.log the harness process's stderr stream,
|
|
1466
1470
|
│ │ when stored
|
|
1467
1471
|
│ ├── trace-parsed.jsonl the parsed event trace, one
|
|
@@ -1569,7 +1573,8 @@ paths:
|
|
|
1569
1573
|
`CancelledError`) and any still-QUEUED trials of a cancelled source.
|
|
1570
1574
|
The source job must be terminal. An UPLOADED job is refused
|
|
1571
1575
|
(`job_uploaded`, 409): it is a record of a run that happened
|
|
1572
|
-
elsewhere, not something the platform can re-run.
|
|
1576
|
+
elsewhere, not something the platform can re-run. A selection whose
|
|
1577
|
+
arm is a retired agent is refused (`agent_retired`, 409).
|
|
1573
1578
|
|
|
1574
1579
|
Supports `Idempotency-Key` with a fingerprint namespaced to this
|
|
1575
1580
|
route, so a create key can never replay a resume.
|
|
@@ -1643,7 +1648,8 @@ paths:
|
|
|
1643
1648
|
caller-SELECTED trials of the source, linked back via `source_jobs`
|
|
1644
1649
|
(`action: "retry"`). The source job is never mutated. An UPLOADED
|
|
1645
1650
|
source is refused (`job_uploaded`, 409): a record of a run that
|
|
1646
|
-
happened elsewhere cannot be re-run here.
|
|
1651
|
+
happened elsewhere cannot be re-run here. A selection whose arm is a
|
|
1652
|
+
retired agent is refused (`agent_retired`, 409).
|
|
1647
1653
|
|
|
1648
1654
|
The body selects the trials — `trial_ids` XOR `failed_only`:
|
|
1649
1655
|
- Omitted body (or `{}`): every trial of the job. The source must be
|
|
@@ -1801,9 +1807,9 @@ paths:
|
|
|
1801
1807
|
summary: Analyze a job's trial traces (rubric-driven)
|
|
1802
1808
|
description: |
|
|
1803
1809
|
Runs Harbor-style trace analysis over a terminal job's trials,
|
|
1804
|
-
server-side: for each trial the analyzer agent (claude-code
|
|
1805
|
-
own sandbox, under the network Harbor's analyze
|
|
1806
|
-
it — open by default) reads the trial's Harbor-shape tree plus its
|
|
1810
|
+
server-side: for each trial the analyzer agent (`agent`, claude-code
|
|
1811
|
+
by default, in its own sandbox, under the network Harbor's analyze
|
|
1812
|
+
wrapper task gives it — open by default) reads the trial's Harbor-shape tree plus its
|
|
1807
1813
|
original task and rules every rubric criterion, storing
|
|
1808
1814
|
`{summary, checks, estimated_cost_usd}` on the trial
|
|
1809
1815
|
(`Trial.analysis`) and the aggregate on the job
|
|
@@ -1823,7 +1829,7 @@ paths:
|
|
|
1823
1829
|
`max_concurrent_analyses`).
|
|
1824
1830
|
|
|
1825
1831
|
An analyzer run that completes without a valid `analysis.json` is
|
|
1826
|
-
automatically re-run at most once (fresh sandbox, same model and
|
|
1832
|
+
automatically re-run at most once (fresh sandbox, same agent, model and
|
|
1827
1833
|
rubric); the analysis object's `attempts` records how far the
|
|
1828
1834
|
ladder went. A run cut by its budget is not that class: it settles
|
|
1829
1835
|
`failed` with phase `timeout` at once and is never re-run — a
|
|
@@ -3274,7 +3280,8 @@ paths:
|
|
|
3274
3280
|
Harbor's hosted `trial retry <id>`) address the trial, not the job.
|
|
3275
3281
|
The source JOB may still be running: a settled trial's facts are
|
|
3276
3282
|
final, so "run it again" is well-defined the moment it settles. A
|
|
3277
|
-
trial of an UPLOADED job is refused (`job_uploaded`, 409)
|
|
3283
|
+
trial of an UPLOADED job is refused (`job_uploaded`, 409), and a
|
|
3284
|
+
trial whose arm is a retired agent by `agent_retired` (409).
|
|
3278
3285
|
|
|
3279
3286
|
Supports `Idempotency-Key` — the identical fingerprint the job-level
|
|
3280
3287
|
door computes for this one-trial selection, because it IS the
|
|
@@ -4999,17 +5006,30 @@ paths:
|
|
|
4999
5006
|
body, as the TEMPLATE a caller's own `prompt` replaces — with its
|
|
5000
5007
|
`{trial_path}`, `{task_section}` and `{criteria_guidance}` slots
|
|
5001
5008
|
unrendered; the output section Harbor appends is not part of it),
|
|
5002
|
-
`reasoning_effort` (the
|
|
5009
|
+
`reasoning_effort` (the resolved model's per-model effort) and
|
|
5003
5010
|
`sandbox_provider` (the fleet default). Read them to see what a
|
|
5004
5011
|
bare `evolve analyze` runs under, or to start an edited rubric or
|
|
5005
|
-
prompt from the platform's own. `
|
|
5006
|
-
|
|
5007
|
-
and
|
|
5008
|
-
|
|
5009
|
-
|
|
5012
|
+
prompt from the platform's own. With `agent`, the same five knobs
|
|
5013
|
+
as `analyze: {agent}` resolves them — that agent's default model
|
|
5014
|
+
and the effort it takes (`evolve analyze --show-defaults -a`).
|
|
5015
|
+
`model_name` and `rubric` sent back with the same `agent` resolve
|
|
5016
|
+
as if nothing was sent; `prompt`, `reasoning_effort` and
|
|
5017
|
+
`sandbox_provider` sent back are stored as named, so the analysis
|
|
5018
|
+
echoes the prompt text and keeps that effort and provider even
|
|
5019
|
+
when the fleet default later changes. Harbor has no such verb:
|
|
5010
5020
|
its defaults are files on disk (analyze/prompts/analyze.txt and its
|
|
5011
5021
|
default analyze rubric), which a hosted caller cannot open — the
|
|
5012
5022
|
hosted reading of the same files.
|
|
5023
|
+
parameters:
|
|
5024
|
+
- name: agent
|
|
5025
|
+
in: query
|
|
5026
|
+
description: >
|
|
5027
|
+
Resolve the defaults for this agent instead of the default one
|
|
5028
|
+
(`AnalyzeConfigInput.agent`'s vocabulary, `GET /api/meta`
|
|
5029
|
+
`analyze.agents[].name`); an unknown name is refused
|
|
5030
|
+
`invalid_input` naming `agent`.
|
|
5031
|
+
schema:
|
|
5032
|
+
type: string
|
|
5013
5033
|
responses:
|
|
5014
5034
|
'200':
|
|
5015
5035
|
description: The resolved defaults.
|
|
@@ -5017,6 +5037,8 @@ paths:
|
|
|
5017
5037
|
application/json:
|
|
5018
5038
|
schema:
|
|
5019
5039
|
$ref: '#/components/schemas/AnalyzeDefaults'
|
|
5040
|
+
'400':
|
|
5041
|
+
$ref: '#/components/responses/BadRequest'
|
|
5020
5042
|
'401':
|
|
5021
5043
|
$ref: '#/components/responses/Unauthorized'
|
|
5022
5044
|
'429':
|
|
@@ -5692,8 +5714,8 @@ paths:
|
|
|
5692
5714
|
|
|
5693
5715
|
analyze-<trial>__<7>/
|
|
5694
5716
|
├── config.json TrialConfig, partial: the wrapper
|
|
5695
|
-
│ task's name, the agent (
|
|
5696
|
-
│ the model), and under `x_evolve`
|
|
5717
|
+
│ task's name, the agent (its Harbor
|
|
5718
|
+
│ name, the model), and under `x_evolve`
|
|
5697
5719
|
│ the frozen policy — rubric, prompt,
|
|
5698
5720
|
│ reasoning_effort — plus the analyzed
|
|
5699
5721
|
│ trial/job/task and the provider
|
|
@@ -5721,13 +5743,15 @@ paths:
|
|
|
5721
5743
|
│ validator (an infrastructure phase)
|
|
5722
5744
|
├── agent/
|
|
5723
5745
|
│ ├── claude-code.txt the analyzer's stdout stream at
|
|
5724
|
-
│ │ Harbor's own tee name for
|
|
5746
|
+
│ │ Harbor's own tee name for its agent
|
|
5747
|
+
│ │ (claude-code.txt, codex.txt, …)
|
|
5725
5748
|
│ ├── stderr.log its stderr, when stored
|
|
5726
5749
|
│ ├── trace-parsed.jsonl the run's parsed event trace
|
|
5727
5750
|
│ ├── .claude/ … the captured home at its real names,
|
|
5728
5751
|
│ ├── agent-home.json the capture record beside it
|
|
5729
|
-
│ └── sessions/ … Harbor's own
|
|
5730
|
-
│
|
|
5752
|
+
│ └── sessions/ … Harbor's own copies of the home, as
|
|
5753
|
+
│ on a trial (claude: the .claude/
|
|
5754
|
+
│ subtree, claude_code.py:482)
|
|
5731
5755
|
├── verifier/ ONLY when the validator ruled —
|
|
5732
5756
|
│ │ completed, or failed `invalid_result`
|
|
5733
5757
|
│ ├── test-stdout.txt the validator's printed reasons
|
|
@@ -5804,8 +5828,8 @@ paths:
|
|
|
5804
5828
|
gzipped tarball (`archive`), or a published dataset is named
|
|
5805
5829
|
instead (`dataset`, the hosted form; `CheckRequest.dataset` states
|
|
5806
5830
|
the deviation and its reason), and for each task directory it holds
|
|
5807
|
-
the checker agent (claude-code
|
|
5808
|
-
sandbox, under the network Harbor's check wrapper task gives it —
|
|
5831
|
+
the checker agent (`agent`, claude-code by default — Harbor's own
|
|
5832
|
+
check agent — in its own sandbox, under the network Harbor's check wrapper task gives it —
|
|
5809
5833
|
open by default) reads EVERY file of the task and rules every rubric
|
|
5810
5834
|
criterion `pass`, `fail`, `not_applicable` or `unknown` with a
|
|
5811
5835
|
rationale and the evidence behind it, writing `check-result.json`
|
|
@@ -5978,18 +6002,31 @@ paths:
|
|
|
5978
6002
|
body, as the TEMPLATE a caller's own `prompt` replaces — with its
|
|
5979
6003
|
`{task_path}`, `{file_tree}` and `{criteria_guidance}` slots
|
|
5980
6004
|
unrendered; the output section Harbor appends is not part of it),
|
|
5981
|
-
`reasoning_effort` (the
|
|
6005
|
+
`reasoning_effort` (the resolved model's per-model effort) and
|
|
5982
6006
|
`sandbox_provider` (the fleet default). Read them to see what a
|
|
5983
6007
|
bare `evolve check` runs under, or to start an edited rubric or
|
|
5984
|
-
prompt from the platform's own. `
|
|
5985
|
-
|
|
5986
|
-
|
|
6008
|
+
prompt from the platform's own. With `agent`, the same five knobs
|
|
6009
|
+
as a check with that `agent` resolves them — its default model and
|
|
6010
|
+
the effort it takes (`evolve check --show-defaults -a`).
|
|
6011
|
+
`model_name` and `rubric` sent back with the same `agent` resolve
|
|
6012
|
+
as if nothing was sent; `prompt`, `reasoning_effort` and
|
|
6013
|
+
`sandbox_provider` sent back are stored as named, so the check
|
|
5987
6014
|
echoes the prompt text and keeps that effort and provider even
|
|
5988
6015
|
when the fleet default later changes. Harbor
|
|
5989
6016
|
has no such verb: its defaults are files on disk
|
|
5990
6017
|
(cli/quality_checker/default-rubric.toml, analyze/prompts/
|
|
5991
6018
|
check.txt), which a hosted caller cannot open — the hosted reading
|
|
5992
6019
|
of the same files.
|
|
6020
|
+
parameters:
|
|
6021
|
+
- name: agent
|
|
6022
|
+
in: query
|
|
6023
|
+
description: >
|
|
6024
|
+
Resolve the defaults for this agent instead of the default one
|
|
6025
|
+
(`CheckConfigInput.agent`'s vocabulary, `GET /api/meta`
|
|
6026
|
+
`analyze.agents[].name`); an unknown name is refused
|
|
6027
|
+
`invalid_input` naming `agent`.
|
|
6028
|
+
schema:
|
|
6029
|
+
type: string
|
|
5993
6030
|
responses:
|
|
5994
6031
|
'200':
|
|
5995
6032
|
description: The resolved defaults.
|
|
@@ -5997,6 +6034,8 @@ paths:
|
|
|
5997
6034
|
application/json:
|
|
5998
6035
|
schema:
|
|
5999
6036
|
$ref: '#/components/schemas/CheckDefaults'
|
|
6037
|
+
'400':
|
|
6038
|
+
$ref: '#/components/responses/BadRequest'
|
|
6000
6039
|
'401':
|
|
6001
6040
|
$ref: '#/components/responses/Unauthorized'
|
|
6002
6041
|
'429':
|
|
@@ -10856,6 +10895,10 @@ components:
|
|
|
10856
10895
|
- agent_config_unsupported
|
|
10857
10896
|
- agent_config_key_refused
|
|
10858
10897
|
- agent_preset_unsupported
|
|
10898
|
+
# A new run naming a retired built-in agent: job create, analyze or
|
|
10899
|
+
# check (400), or resume/retry of a job whose arm it is (409). Details
|
|
10900
|
+
# name each one and its replacement (`retired_agents` in GET /api/meta).
|
|
10901
|
+
- agent_retired
|
|
10859
10902
|
- provider_unsupported
|
|
10860
10903
|
- job_not_found
|
|
10861
10904
|
- job_not_terminal
|
|
@@ -11663,7 +11706,10 @@ components:
|
|
|
11663
11706
|
properties:
|
|
11664
11707
|
name:
|
|
11665
11708
|
type: string
|
|
11666
|
-
description:
|
|
11709
|
+
description: >
|
|
11710
|
+
Agent name — a built-in or one registered under /api/agents. A
|
|
11711
|
+
built-in listed under `GET /api/meta` `retired_agents` is refused
|
|
11712
|
+
(`agent_retired`, details naming its replacement).
|
|
11667
11713
|
model_name:
|
|
11668
11714
|
type: string
|
|
11669
11715
|
description: >
|
|
@@ -12359,7 +12405,7 @@ components:
|
|
|
12359
12405
|
type: object
|
|
12360
12406
|
description: |
|
|
12361
12407
|
Trace-analysis configuration — Harbor's `harbor analyze` vocabulary
|
|
12362
|
-
(their cli/analyze.py: `--model`, `--rubric`, `--prompt`). PRESENCE of this
|
|
12408
|
+
(their cli/analyze.py: `--agent`, `--model`, `--rubric`, `--prompt`). PRESENCE of this
|
|
12363
12409
|
object is the switch: on `JobCreate.analyze` it arms the embedded
|
|
12364
12410
|
trigger (each trial is analyzed server-side right after it settles;
|
|
12365
12411
|
CANCELLED trials are skipped — withdrawn work is not billed an
|
|
@@ -12371,13 +12417,13 @@ components:
|
|
|
12371
12417
|
`GET /api/analyses/defaults`; Harbor's own analyze-rubric.toml
|
|
12372
12418
|
still runs when passed as `rubric`).
|
|
12373
12419
|
|
|
12374
|
-
The analyzer
|
|
12375
|
-
|
|
12376
|
-
analyze wrapper task gives it (open by default;
|
|
12377
|
-
unchanged) — on the provider `sandbox_provider`
|
|
12378
|
-
platform's analysis default when it names none; its
|
|
12379
|
-
per analysis and metered as its own line, never
|
|
12380
|
-
trial's own bill.
|
|
12420
|
+
The analyzer runs the agent `agent` names (claude, Harbor's own
|
|
12421
|
+
default claude-code, when it names none) in its own sandbox, under
|
|
12422
|
+
the network Harbor's analyze wrapper task gives it (open by default;
|
|
12423
|
+
the model door is unchanged) — on the provider `sandbox_provider`
|
|
12424
|
+
names, or the platform's analysis default when it names none; its
|
|
12425
|
+
spend is capped per analysis and metered as its own line, never
|
|
12426
|
+
blended into the trial's own bill.
|
|
12381
12427
|
|
|
12382
12428
|
Which trials, and how wide, are Harbor's own analyze options with
|
|
12383
12429
|
their exact names: `n_concurrent` (`-n/--n-concurrent`), `passing`
|
|
@@ -12393,9 +12439,7 @@ components:
|
|
|
12393
12439
|
takes it; on `JobCreate.analyze` the trials do not exist yet.
|
|
12394
12440
|
|
|
12395
12441
|
Harbor's other analyze options, and why they are not here:
|
|
12396
|
-
`-
|
|
12397
|
-
instruction templates, the roster check and the sandbox policy are
|
|
12398
|
-
written against it); `--job-name` and `-o/--jobs-dir` — a hosted
|
|
12442
|
+
`--job-name` and `-o/--jobs-dir` — a hosted
|
|
12399
12443
|
analysis attaches to its source job and is served on that job's
|
|
12400
12444
|
trials, so there is no wrapper job to name and no directory to
|
|
12401
12445
|
write; `-k/--n-attempts` (k analyses per trial) — deferred: each
|
|
@@ -12410,30 +12454,52 @@ components:
|
|
|
12410
12454
|
(task quality checks, their cli/analyze.py:126-137) and are not
|
|
12411
12455
|
adopted here.
|
|
12412
12456
|
properties:
|
|
12457
|
+
agent:
|
|
12458
|
+
type: string
|
|
12459
|
+
default: claude
|
|
12460
|
+
description: >
|
|
12461
|
+
The agent the analyzer runs on — Harbor's `-a/--agent` (their
|
|
12462
|
+
cli/analyze.py:258, default claude-code), named in the
|
|
12463
|
+
`agents[].name` vocabulary (claude, not claude-code) so one set
|
|
12464
|
+
of names covers arms and reviewers: any built-in agent
|
|
12465
|
+
`analyze.agents` lists on `GET /api/meta`. `model_name` must be
|
|
12466
|
+
on this agent's roster and `reasoning_effort` must be one it
|
|
12467
|
+
accepts; omitted, `model_name` is this agent's
|
|
12468
|
+
`analyze.agents[].default_model`. A registered (bring-your-own) agent
|
|
12469
|
+
is not accepted. A built-in listed under `GET /api/meta`
|
|
12470
|
+
`retired_agents` is refused `agent_retired` naming
|
|
12471
|
+
`analyze.agent`, details naming its replacement; analyzing the
|
|
12472
|
+
trials a retired agent ran, with a live agent, stays allowed.
|
|
12473
|
+
Anything else is refused `invalid_input` naming
|
|
12474
|
+
`analyze.agent`. Stored as given and frozen into each analysis
|
|
12475
|
+
(`TrialAnalysis.agent`).
|
|
12413
12476
|
model_name:
|
|
12414
12477
|
type: string
|
|
12415
|
-
|
|
12416
|
-
|
|
12417
|
-
|
|
12418
|
-
|
|
12419
|
-
|
|
12420
|
-
|
|
12421
|
-
|
|
12422
|
-
|
|
12423
|
-
|
|
12424
|
-
|
|
12425
|
-
|
|
12426
|
-
|
|
12427
|
-
|
|
12428
|
-
|
|
12429
|
-
|
|
12430
|
-
|
|
12431
|
-
|
|
12432
|
-
|
|
12433
|
-
|
|
12434
|
-
|
|
12435
|
-
|
|
12436
|
-
|
|
12478
|
+
description: >
|
|
12479
|
+
Model the analyzer agent runs — Harbor's `--model`. Omitted, the
|
|
12480
|
+
agent's default: the platform's pick,
|
|
12481
|
+
openrouter/deepseek/deepseek-v4.1-flash (`GET /api/meta`
|
|
12482
|
+
`analyze.default_model`), on every agent whose roster carries it,
|
|
12483
|
+
else that agent's own default (`GET /api/meta`
|
|
12484
|
+
`agents[].default_model`, the model the SDK runs that agent on
|
|
12485
|
+
when none is named). The platform's pick is DeepSeek V4.1 Flash
|
|
12486
|
+
served through OpenRouter, at its default effort high (the
|
|
12487
|
+
owner's ruling 2026-09-10: far more parallel capacity through
|
|
12488
|
+
OpenRouter's provider pool than one pinned Fireworks host) — a
|
|
12489
|
+
recorded deviation from Harbor's default analyze model (their
|
|
12490
|
+
cli/analyze.py `claude-haiku-4-5`): analysis is input-dominated,
|
|
12491
|
+
and this is the roster's intelligence-per-input-dollar pick;
|
|
12492
|
+
`glm-5.3-flash` (at max, the effort its published scores use)
|
|
12493
|
+
and `haiku` stay on the claude roster as alternatives, `glm-5.3`
|
|
12494
|
+
to escalate, and the same model on its Fireworks route,
|
|
12495
|
+
`fireworks/deepseek-v4.1-flash`, is a further option (the
|
|
12496
|
+
OpenRouter id stays the pick). The value speaks the same
|
|
12497
|
+
vocabulary as `agents[].model_name`: either advertised spelling
|
|
12498
|
+
is accepted and stored AS GIVEN (the default is the roster
|
|
12499
|
+
alias), the wire id is resolved only when the analyzer runs, and
|
|
12500
|
+
every stored analysis serves the spelling it was created under.
|
|
12501
|
+
A named model must be on the agent's roster (`GET /api/meta`,
|
|
12502
|
+
`analyze.agents[].models`); anything else is refused at accept
|
|
12437
12503
|
(`invalid_input`, roster in the message).
|
|
12438
12504
|
rubric:
|
|
12439
12505
|
$ref: '#/components/schemas/Rubric'
|
|
@@ -12472,20 +12538,22 @@ components:
|
|
|
12472
12538
|
type: string
|
|
12473
12539
|
description: >
|
|
12474
12540
|
Reasoning effort the analyzer runs at — the platform's
|
|
12475
|
-
`agents[].reasoning_effort` vocabulary applied to the analyzer
|
|
12476
|
-
|
|
12477
|
-
`analyze.reasoning_efforts` on
|
|
12478
|
-
|
|
12479
|
-
|
|
12480
|
-
|
|
12541
|
+
`agents[].reasoning_effort` vocabulary applied to the analyzer's
|
|
12542
|
+
agent, held to that agent's own rule exactly as an arm on it:
|
|
12543
|
+
the accepted values are `analyze.agents[].reasoning_efforts` on
|
|
12544
|
+
`GET /api/meta` (claude's `effort_support` is `level`, so the
|
|
12545
|
+
whole list; an agent that takes none, gemini, accepts none), and
|
|
12546
|
+
a refused value answers `invalid_input` with the same sentence
|
|
12547
|
+
an arm's does (`analyze.reasoning_effort must be one of: ...`).
|
|
12481
12548
|
Omitted, the PER-MODEL default applies —
|
|
12482
|
-
`analyze.models[].default_reasoning_effort`: `high` on
|
|
12483
|
-
openrouter/deepseek/deepseek-v4.1-flash (the
|
|
12549
|
+
`analyze.agents[].models[].default_reasoning_effort`: `high` on
|
|
12550
|
+
openrouter/deepseek/deepseek-v4.1-flash (the platform's pick;
|
|
12484
12551
|
DeepSeek's own documented default, the owner's ruling
|
|
12485
12552
|
2026-09-10), `max` on glm-5.3-flash (the platform's ruling
|
|
12486
|
-
2026-09-08, the effort its published scores use),
|
|
12487
|
-
|
|
12488
|
-
|
|
12553
|
+
2026-09-08, the effort its published scores use), on any agent
|
|
12554
|
+
that can honor them; the agent's own default
|
|
12555
|
+
(`agents[].default_effort`) on every other roster model; none
|
|
12556
|
+
on an agent that takes none. Whatever resolves is passed to the analyzer
|
|
12489
12557
|
EXPLICITLY on every call — never left to a provider's default —
|
|
12490
12558
|
and recorded as `TrialAnalysis.reasoning_effort`. A hosted
|
|
12491
12559
|
extension, recorded as such: Harbor's `harbor analyze` has no
|
|
@@ -12602,11 +12670,14 @@ components:
|
|
|
12602
12670
|
always states the policy it executes (same law as RetryConfig).
|
|
12603
12671
|
Echoed on the job body when the job was created with `analyze`;
|
|
12604
12672
|
each analysis additionally carries the exact policy IT ran under
|
|
12605
|
-
(`Trial.analysis.model_name` / `.rubric` / `.prompt` /
|
|
12673
|
+
(`Trial.analysis.agent` / `.model_name` / `.rubric` / `.prompt` /
|
|
12606
12674
|
`.reasoning_effort`), which a later manual re-analysis may have
|
|
12607
12675
|
changed.
|
|
12608
|
-
required: [model_name, rubric, prompt, reasoning_effort, sandbox_provider, n_concurrent, passing, failing, n_trials, trial_ids]
|
|
12676
|
+
required: [agent, model_name, rubric, prompt, reasoning_effort, sandbox_provider, n_concurrent, passing, failing, n_trials, trial_ids]
|
|
12609
12677
|
properties:
|
|
12678
|
+
agent:
|
|
12679
|
+
type: string
|
|
12680
|
+
description: The agent this policy's analyses run on — as named, or the default (claude).
|
|
12610
12681
|
model_name:
|
|
12611
12682
|
type: string
|
|
12612
12683
|
rubric:
|
|
@@ -12618,14 +12689,15 @@ components:
|
|
|
12618
12689
|
null = the built-in prompt (the platform's default analyze body).
|
|
12619
12690
|
reasoning_effort:
|
|
12620
12691
|
x-wave: 3
|
|
12621
|
-
type: string
|
|
12692
|
+
type: [string, 'null']
|
|
12622
12693
|
description: >
|
|
12623
12694
|
The effort this policy's analyses run at. Named at create it is
|
|
12624
12695
|
served as stored; when the create named none, this echoes the
|
|
12625
|
-
per-model default of the day for `model_name`
|
|
12626
|
-
next enqueue under this policy stamps (the same nuance
|
|
12627
|
-
`sandbox_provider` below, for the same reason: the default is
|
|
12628
|
-
platform table, resolved where an analysis is enqueued).
|
|
12696
|
+
per-model default of the day for `model_name` on `agent` — the
|
|
12697
|
+
value the next enqueue under this policy stamps (the same nuance
|
|
12698
|
+
as `sandbox_provider` below, for the same reason: the default is
|
|
12699
|
+
a platform table, resolved where an analysis is enqueued). Null
|
|
12700
|
+
when the agent takes no effort.
|
|
12629
12701
|
sandbox_provider:
|
|
12630
12702
|
allOf:
|
|
12631
12703
|
- $ref: '#/components/schemas/SandboxProvider'
|
|
@@ -13964,6 +14036,7 @@ components:
|
|
|
13964
14036
|
- name
|
|
13965
14037
|
- status
|
|
13966
14038
|
- source
|
|
14039
|
+
- agent
|
|
13967
14040
|
- model_name
|
|
13968
14041
|
- reasoning_effort
|
|
13969
14042
|
- sandbox_provider
|
|
@@ -13992,11 +14065,14 @@ components:
|
|
|
13992
14065
|
source:
|
|
13993
14066
|
$ref: '#/components/schemas/CheckSource'
|
|
13994
14067
|
description: What was checked; the row's DATASETS cell (the dataset form's `name@version`; an archive shows as an archive with its sha256).
|
|
14068
|
+
agent:
|
|
14069
|
+
type: string
|
|
14070
|
+
description: The checker's agent, as Check.agent.
|
|
13995
14071
|
model_name:
|
|
13996
14072
|
type: string
|
|
13997
14073
|
description: The checker's model; the row's AGENTS cell, shown like an arm with its effort under it.
|
|
13998
14074
|
reasoning_effort:
|
|
13999
|
-
type: string
|
|
14075
|
+
type: [string, 'null']
|
|
14000
14076
|
description: The effort every task's checker ran at, as Check.reasoning_effort.
|
|
14001
14077
|
sandbox_provider:
|
|
14002
14078
|
allOf:
|
|
@@ -14769,7 +14845,11 @@ components:
|
|
|
14769
14845
|
that failed validation, its reasons in the message — never
|
|
14770
14846
|
re-run: a timeout is deterministic, a second attempt cannot
|
|
14771
14847
|
finish in less time than the first; the message names the
|
|
14772
|
-
budget, the seconds used and the exit code),
|
|
14848
|
+
budget, the seconds used and the exit code), `agent_retired`
|
|
14849
|
+
(an embedded analysis whose job's stored `analyze.agent` was
|
|
14850
|
+
retired after the job was created: recorded when the trial
|
|
14851
|
+
settles, nothing booted or spent, never re-run; the message
|
|
14852
|
+
names the replacement), or an infrastructure
|
|
14773
14853
|
stage of the analyzer run (`mint_key`, `boot`,
|
|
14774
14854
|
`harness_install`, `agent`, `artifact_read`, `lease_expired`,
|
|
14775
14855
|
...) — infrastructure stages never auto re-run.
|
|
@@ -14783,7 +14863,7 @@ components:
|
|
|
14783
14863
|
AnalyzeResult (their analyze/models.py: `summary`, `checks` keyed
|
|
14784
14864
|
by criterion, `estimated_cost_usd`; `trial_name` is the enclosing
|
|
14785
14865
|
trial) with each check carrying the platform's evidence list and
|
|
14786
|
-
fourth outcome (AnalysisCheck); the rest is provenance — which model, rubric and prompt THIS analysis ran
|
|
14866
|
+
fourth outcome (AnalysisCheck); the rest is provenance — which agent, model, rubric and prompt THIS analysis ran
|
|
14787
14867
|
under, its lifecycle status, and its typed failure when it failed.
|
|
14788
14868
|
|
|
14789
14869
|
`estimated_cost_usd` is the analyzer agent's OWN metered spend (its
|
|
@@ -14797,6 +14877,7 @@ components:
|
|
|
14797
14877
|
- job_id
|
|
14798
14878
|
- task_name
|
|
14799
14879
|
- status
|
|
14880
|
+
- agent
|
|
14800
14881
|
- model_name
|
|
14801
14882
|
- reasoning_effort
|
|
14802
14883
|
- rubric
|
|
@@ -14841,6 +14922,12 @@ components:
|
|
|
14841
14922
|
`not_applicable` are their exact words) and the ladder follows
|
|
14842
14923
|
its object's dialect; the UI capitalizes on render, and nothing
|
|
14843
14924
|
ever compares this ladder's words against another ladder's.
|
|
14925
|
+
agent:
|
|
14926
|
+
type: string
|
|
14927
|
+
description: >
|
|
14928
|
+
The agent THIS analysis ran on (`AnalyzeConfigInput.agent`, or
|
|
14929
|
+
the default claude); every analysis from before the choice
|
|
14930
|
+
existed ran on claude and says so.
|
|
14844
14931
|
model_name:
|
|
14845
14932
|
type: string
|
|
14846
14933
|
reasoning_effort:
|
|
@@ -14850,8 +14937,8 @@ components:
|
|
|
14850
14937
|
The reasoning effort THIS analysis ran at — passed to the
|
|
14851
14938
|
analyzer explicitly, so it is what the model was asked for
|
|
14852
14939
|
(`AnalyzeConfigInput.reasoning_effort`, or the per-model
|
|
14853
|
-
default). Null
|
|
14854
|
-
stamped; never a guessed value.
|
|
14940
|
+
default). Null when its agent takes no effort, or on analyses
|
|
14941
|
+
recorded before the effort was stamped; never a guessed value.
|
|
14855
14942
|
rubric:
|
|
14856
14943
|
$ref: '#/components/schemas/Rubric'
|
|
14857
14944
|
prompt:
|
|
@@ -15006,13 +15093,14 @@ components:
|
|
|
15006
15093
|
type: object
|
|
15007
15094
|
description: |
|
|
15008
15095
|
Task-check configuration — Harbor's `harbor check` vocabulary
|
|
15009
|
-
(their cli/analyze.py:84-148 check_command). The
|
|
15096
|
+
(their cli/analyze.py:84-148 check_command). The four rubric-agent
|
|
15010
15097
|
knobs are the analyze door's, with the SAME rules and refusals
|
|
15011
15098
|
(`AnalyzeConfigInput` states them; the param prefix here is
|
|
15012
|
-
`check.`): `
|
|
15013
|
-
|
|
15014
|
-
`
|
|
15015
|
-
|
|
15099
|
+
`check.`): `agent` (`-a/--agent`, :100; default claude, Harbor's
|
|
15100
|
+
claude-code), `model_name` (`-m/--model`; Harbor's check default is
|
|
15101
|
+
`claude-sonnet-4-6`, :101-103 — this platform's is the analyzer's,
|
|
15102
|
+
the agent's default per `AnalyzeConfigInput.model_name`, one rule for
|
|
15103
|
+
both rubric agents, a recorded deviation), `rubric` (`-r/--rubric`; the default
|
|
15016
15104
|
is the platform's check rubric, served by `GET /api/checks/defaults`;
|
|
15017
15105
|
Harbor's own default-rubric.toml still runs when passed),
|
|
15018
15106
|
`prompt` (`-p/--prompt`, the file's TEXT: it replaces the built-in
|
|
@@ -15035,10 +15123,9 @@ components:
|
|
|
15035
15123
|
matched against the directory NAME.
|
|
15036
15124
|
|
|
15037
15125
|
Harbor's other check options, and why they are not here: the same
|
|
15038
|
-
list as the analyze door's — `-
|
|
15039
|
-
|
|
15040
|
-
|
|
15041
|
-
to write), `-k/--n-attempts` (the one hosted auto re-run replaces
|
|
15126
|
+
list as the analyze door's — `-o/--jobs-dir` (a hosted check is its
|
|
15127
|
+
own record, no directory to write; `--job-name` is `name`),
|
|
15128
|
+
`-k/--n-attempts` (the one hosted auto re-run replaces
|
|
15042
15129
|
it; see `TaskCheck.attempts`), `-c/--config`, `--ak`, `--ae`,
|
|
15043
15130
|
`--ek` (local-runner knobs with no hosted meaning), `-q` (a client
|
|
15044
15131
|
rendering option, not a wire field).
|
|
@@ -15051,9 +15138,18 @@ components:
|
|
|
15051
15138
|
The caller's name for the check — Harbor's `--job-name` on
|
|
15052
15139
|
`harbor check` (cli/analyze.py:138). Omitted, the accept
|
|
15053
15140
|
timestamp in Harbor's own shape, `YYYY-MM-DD__HH-MM-SS` (UTC).
|
|
15141
|
+
agent:
|
|
15142
|
+
type: string
|
|
15143
|
+
default: claude
|
|
15144
|
+
description: >
|
|
15145
|
+
The agent the checker runs on — the analyze door's `agent`, same
|
|
15146
|
+
rule: a retired built-in is refused `agent_retired` naming
|
|
15147
|
+
`check.agent`, anything else off the list `invalid_input`.
|
|
15054
15148
|
model_name:
|
|
15055
15149
|
type: string
|
|
15056
|
-
|
|
15150
|
+
description: >
|
|
15151
|
+
Model the checker agent runs — the analyze door's `model_name`,
|
|
15152
|
+
same rule: omitted, the agent's default; named, on the agent's roster.
|
|
15057
15153
|
rubric:
|
|
15058
15154
|
$ref: '#/components/schemas/Rubric'
|
|
15059
15155
|
prompt:
|
|
@@ -15213,6 +15309,7 @@ components:
|
|
|
15213
15309
|
- name
|
|
15214
15310
|
- status
|
|
15215
15311
|
- source
|
|
15312
|
+
- agent
|
|
15216
15313
|
- model_name
|
|
15217
15314
|
- reasoning_effort
|
|
15218
15315
|
- rubric
|
|
@@ -15239,11 +15336,14 @@ components:
|
|
|
15239
15336
|
enum: [queued, running, completed]
|
|
15240
15337
|
source:
|
|
15241
15338
|
$ref: '#/components/schemas/CheckSource'
|
|
15339
|
+
agent:
|
|
15340
|
+
type: string
|
|
15341
|
+
description: The agent every task's checker ran on — named at create, or the default (claude); every check from before the choice existed ran on claude.
|
|
15242
15342
|
model_name:
|
|
15243
15343
|
type: string
|
|
15244
15344
|
reasoning_effort:
|
|
15245
|
-
type: string
|
|
15246
|
-
description: The effort every task's checker ran at — named at create, or the model's default, resolved at accept and stamped.
|
|
15345
|
+
type: [string, 'null']
|
|
15346
|
+
description: The effort every task's checker ran at — named at create, or the model's default on the agent, resolved at accept and stamped; null when the agent takes none.
|
|
15247
15347
|
rubric:
|
|
15248
15348
|
$ref: '#/components/schemas/Rubric'
|
|
15249
15349
|
prompt:
|
|
@@ -15293,14 +15393,18 @@ components:
|
|
|
15293
15393
|
type: object
|
|
15294
15394
|
description: >
|
|
15295
15395
|
The policy an empty `AnalyzeConfigInput` resolves to (GET
|
|
15296
|
-
/api/analyses/defaults)
|
|
15297
|
-
|
|
15396
|
+
/api/analyses/defaults), or one naming only `agent` when the door's
|
|
15397
|
+
`?agent=` names it — each key the same value `AnalyzeConfig`
|
|
15398
|
+
echoes for a job created with that config, except `prompt`, which
|
|
15298
15399
|
`AnalyzeConfig` serves as null (the built-in body) and this serves as
|
|
15299
15400
|
that body's template text.
|
|
15300
|
-
required: [model_name, rubric, prompt, reasoning_effort, sandbox_provider]
|
|
15401
|
+
required: [agent, model_name, rubric, prompt, reasoning_effort, sandbox_provider]
|
|
15301
15402
|
properties:
|
|
15403
|
+
agent:
|
|
15404
|
+
type: string
|
|
15302
15405
|
model_name:
|
|
15303
15406
|
type: string
|
|
15407
|
+
description: The model an omitted `model_name` takes on `agent` (`GET /api/meta` `analyze.agents[].default_model`).
|
|
15304
15408
|
rubric:
|
|
15305
15409
|
$ref: '#/components/schemas/Rubric'
|
|
15306
15410
|
prompt:
|
|
@@ -15310,8 +15414,8 @@ components:
|
|
|
15310
15414
|
`AnalyzeConfigInput.prompt` to run the default body explicitly,
|
|
15311
15415
|
or edit it from here.
|
|
15312
15416
|
reasoning_effort:
|
|
15313
|
-
type: string
|
|
15314
|
-
description: The effort
|
|
15417
|
+
type: [string, 'null']
|
|
15418
|
+
description: The effort `model_name` runs at on `agent` when the config names none; null if that agent takes none.
|
|
15315
15419
|
sandbox_provider:
|
|
15316
15420
|
allOf:
|
|
15317
15421
|
- $ref: '#/components/schemas/SandboxProvider'
|
|
@@ -15320,14 +15424,18 @@ components:
|
|
|
15320
15424
|
type: object
|
|
15321
15425
|
description: >
|
|
15322
15426
|
The policy an empty `CheckConfigInput` resolves to (GET
|
|
15323
|
-
/api/checks/defaults)
|
|
15324
|
-
|
|
15427
|
+
/api/checks/defaults), or one naming only `agent` when the door's
|
|
15428
|
+
`?agent=` names it — each key the same value `Check` echoes for
|
|
15429
|
+
a check created with that config, except `prompt`, which `Check`
|
|
15325
15430
|
serves as null (the built-in body) and this serves as that body's
|
|
15326
15431
|
template text.
|
|
15327
|
-
required: [model_name, rubric, prompt, reasoning_effort, sandbox_provider]
|
|
15432
|
+
required: [agent, model_name, rubric, prompt, reasoning_effort, sandbox_provider]
|
|
15328
15433
|
properties:
|
|
15434
|
+
agent:
|
|
15435
|
+
type: string
|
|
15329
15436
|
model_name:
|
|
15330
15437
|
type: string
|
|
15438
|
+
description: The model an omitted `model_name` takes on `agent` (`GET /api/meta` `analyze.agents[].default_model`).
|
|
15331
15439
|
rubric:
|
|
15332
15440
|
$ref: '#/components/schemas/Rubric'
|
|
15333
15441
|
prompt:
|
|
@@ -15337,8 +15445,8 @@ components:
|
|
|
15337
15445
|
`CheckConfigInput.prompt` to run the default body explicitly,
|
|
15338
15446
|
or edit it from here.
|
|
15339
15447
|
reasoning_effort:
|
|
15340
|
-
type: string
|
|
15341
|
-
description: The effort
|
|
15448
|
+
type: [string, 'null']
|
|
15449
|
+
description: The effort `model_name` runs at on `agent` when the config names none; null if that agent takes none.
|
|
15342
15450
|
sandbox_provider:
|
|
15343
15451
|
allOf:
|
|
15344
15452
|
- $ref: '#/components/schemas/SandboxProvider'
|
|
@@ -16141,7 +16249,10 @@ components:
|
|
|
16141
16249
|
script exited non-zero, or installed no executable): the agent's
|
|
16142
16250
|
fault, never retried automatically; the trial's stored stderr
|
|
16143
16251
|
stream holds the install's output. Remove the registration, add
|
|
16144
|
-
it again, and resume. `
|
|
16252
|
+
it again, and resume. `agent_retired` — the arm's built-in agent
|
|
16253
|
+
is retired (`GET /api/meta` `retired_agents`): refused before any
|
|
16254
|
+
key or sandbox, never retried automatically; `failureDetail`
|
|
16255
|
+
names the replacement. `budget_exhausted` — the
|
|
16145
16256
|
one phase whose status is `BUDGET`, not `INFRASTRUCTURE_ERROR`
|
|
16146
16257
|
— is a ceiling ABOVE the run's own per-trial cap refusing it:
|
|
16147
16258
|
`failureDetail` carries the subject right after the
|
|
@@ -18203,6 +18314,18 @@ components:
|
|
|
18203
18314
|
latest_version:
|
|
18204
18315
|
type: [string, 'null']
|
|
18205
18316
|
|
|
18317
|
+
RetiredAgent:
|
|
18318
|
+
type: object
|
|
18319
|
+
description: >
|
|
18320
|
+
One retired built-in agent and the agent to use instead — the same
|
|
18321
|
+
row the `agent_retired` refusal carries in `details.retired_agents`.
|
|
18322
|
+
required: [agent, replaced_by]
|
|
18323
|
+
properties:
|
|
18324
|
+
agent:
|
|
18325
|
+
type: string
|
|
18326
|
+
replaced_by:
|
|
18327
|
+
type: string
|
|
18328
|
+
|
|
18206
18329
|
AgentModelOption:
|
|
18207
18330
|
type: object
|
|
18208
18331
|
description: One model alias an agent offers, for a picker's option list.
|
|
@@ -18332,41 +18455,71 @@ components:
|
|
|
18332
18455
|
type: integer
|
|
18333
18456
|
agents:
|
|
18334
18457
|
type: array
|
|
18335
|
-
description:
|
|
18458
|
+
description: >
|
|
18459
|
+
Built-in agents a new job may name, and their declared
|
|
18460
|
+
capabilities. A retired agent is not listed here; it appears in
|
|
18461
|
+
`retired_agents`.
|
|
18336
18462
|
items:
|
|
18337
18463
|
$ref: '#/components/schemas/AgentCapability'
|
|
18464
|
+
retired_agents:
|
|
18465
|
+
type: array
|
|
18466
|
+
description: >
|
|
18467
|
+
Built-in agents that are retired: a new job, resume, retry,
|
|
18468
|
+
analysis or check naming one is refused (`agent_retired`), while
|
|
18469
|
+
the jobs, trials and traces it produced stay readable under its
|
|
18470
|
+
name.
|
|
18471
|
+
items:
|
|
18472
|
+
$ref: '#/components/schemas/RetiredAgent'
|
|
18338
18473
|
analyze:
|
|
18339
18474
|
x-wave: 3
|
|
18340
18475
|
type: object
|
|
18341
18476
|
description: >
|
|
18342
|
-
The
|
|
18343
|
-
`AnalyzeConfigInput.
|
|
18344
|
-
`
|
|
18345
|
-
|
|
18346
|
-
|
|
18347
|
-
|
|
18348
|
-
|
|
18349
|
-
|
|
18350
|
-
|
|
18477
|
+
The rubric agents' roster and defaults (the analyzer's and the
|
|
18478
|
+
checker's alike): the agents `AnalyzeConfigInput.agent` and
|
|
18479
|
+
`CheckConfigInput.agent` accept and the one an omitted agent
|
|
18480
|
+
takes (`default_agent`); the platform's model pick
|
|
18481
|
+
(`default_model`); and per agent the model an omitted
|
|
18482
|
+
`model_name` takes (`agents[].default_model`: the platform's
|
|
18483
|
+
pick on every agent whose roster carries it, else the agent's
|
|
18484
|
+
own), the efforts its `reasoning_effort` accepts and every
|
|
18485
|
+
roster model with the effort an omitted `reasoning_effort`
|
|
18486
|
+
takes for it. Published because the agent and the effort change
|
|
18487
|
+
what an analysis costs and how it reads, and a client comparing
|
|
18488
|
+
two waves must know what "omitted" meant in each.
|
|
18489
|
+
required: [default_agent, default_model, agents]
|
|
18351
18490
|
properties:
|
|
18491
|
+
default_agent:
|
|
18492
|
+
type: string
|
|
18352
18493
|
default_model:
|
|
18353
18494
|
type: string
|
|
18354
|
-
|
|
18355
|
-
type: array
|
|
18356
|
-
items:
|
|
18357
|
-
type: string
|
|
18358
|
-
models:
|
|
18495
|
+
agents:
|
|
18359
18496
|
type: array
|
|
18360
18497
|
items:
|
|
18361
18498
|
type: object
|
|
18362
|
-
required: [
|
|
18499
|
+
required: [name, default_model, reasoning_efforts, models]
|
|
18363
18500
|
properties:
|
|
18364
|
-
|
|
18501
|
+
name:
|
|
18365
18502
|
type: string
|
|
18366
|
-
|
|
18367
|
-
type: string
|
|
18368
|
-
default_reasoning_effort:
|
|
18503
|
+
default_model:
|
|
18369
18504
|
type: string
|
|
18505
|
+
description: The model an omitted `model_name` takes on this agent; one of its `models[].alias`.
|
|
18506
|
+
reasoning_efforts:
|
|
18507
|
+
type: array
|
|
18508
|
+
description: Empty for an agent that takes no effort.
|
|
18509
|
+
items:
|
|
18510
|
+
type: string
|
|
18511
|
+
models:
|
|
18512
|
+
type: array
|
|
18513
|
+
items:
|
|
18514
|
+
type: object
|
|
18515
|
+
required: [alias, model_id, default_reasoning_effort]
|
|
18516
|
+
properties:
|
|
18517
|
+
alias:
|
|
18518
|
+
type: string
|
|
18519
|
+
model_id:
|
|
18520
|
+
type: string
|
|
18521
|
+
default_reasoning_effort:
|
|
18522
|
+
type: [string, 'null']
|
|
18370
18523
|
agent_registration:
|
|
18371
18524
|
type: object
|
|
18372
18525
|
description: Rules a bring-your-own agent registration must satisfy.
|