@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (153) hide show
  1. package/dist/{chunk-DK2MGUPB.js → chunk-DI6LPW2B.js} +2 -2
  2. package/dist/cli/index.cjs +29 -29
  3. package/dist/cli/index.d.cts +3 -3
  4. package/dist/cli/index.d.ts +3 -3
  5. package/dist/cli/index.js +18 -18
  6. package/dist/index.cjs +64 -60
  7. package/dist/index.d.cts +297 -18
  8. package/dist/index.d.ts +297 -18
  9. package/dist/index.js +48 -44
  10. package/dist/{types-B_qmzlMk.d.cts → types-Dj0iMl5V.d.cts} +70 -90
  11. package/dist/{types-B_qmzlMk.d.ts → types-Dj0iMl5V.d.ts} +70 -90
  12. package/harness-capabilities.json +285 -0
  13. package/package.json +9 -6
  14. package/skills/create-adapter/SKILL.md +71 -26
  15. package/skills/create-task/SKILL.md +124 -66
  16. package/skills/evolve/SKILL.md +37 -20
  17. package/skills/evolve-evals/SKILL.md +142 -69
  18. package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
  19. package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
  20. package/skills/evolve-evals/references/cli-reference/analyze.md +97 -0
  21. package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
  22. package/skills/evolve-evals/references/cli-reference/check.md +157 -0
  23. package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
  24. package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
  25. package/skills/evolve-evals/references/cli-reference/index.md +149 -0
  26. package/skills/evolve-evals/references/cli-reference/job.md +198 -0
  27. package/skills/evolve-evals/references/cli-reference/run.md +188 -0
  28. package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
  29. package/skills/evolve-evals/references/cli-reference/session.md +56 -0
  30. package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
  31. package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
  32. package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
  33. package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
  34. package/skills/evolve-evals/references/core-concepts/agents.md +101 -0
  35. package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
  36. package/skills/evolve-evals/references/core-concepts/check.md +116 -0
  37. package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
  38. package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
  39. package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
  40. package/skills/evolve-evals/references/core-concepts/index.md +52 -0
  41. package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
  42. package/skills/evolve-evals/references/core-concepts/models.md +73 -0
  43. package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
  44. package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
  45. package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
  46. package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
  47. package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
  48. package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
  49. package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
  50. package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
  51. package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
  52. package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
  53. package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
  54. package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
  55. package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
  56. package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
  57. package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
  58. package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
  59. package/skills/evolve-evals/references/dashboard/checks.md +54 -0
  60. package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
  61. package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
  62. package/skills/evolve-evals/references/dashboard/traces.md +54 -0
  63. package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
  64. package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
  65. package/skills/evolve-evals/references/getting-started/installation.md +79 -0
  66. package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
  67. package/skills/evolve-evals/references/index.md +58 -0
  68. package/skills/evolve-evals/references/sdk/http.md +110 -0
  69. package/skills/evolve-evals/references/sdk/python.md +108 -0
  70. package/skills/evolve-evals/references/sdk/typescript.md +104 -0
  71. package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
  72. package/skills/evolve-evals/references/sdk-reference/analyses.md +125 -0
  73. package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
  74. package/skills/evolve-evals/references/sdk-reference/checks.md +146 -0
  75. package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
  76. package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
  77. package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
  78. package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
  79. package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
  80. package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
  81. package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
  82. package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
  83. package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
  84. package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
  85. package/skills/evolve-evals/references/sdk-reference/methods/checks.md +462 -0
  86. package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
  87. package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
  88. package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1023 -0
  89. package/skills/evolve-evals/references/sdk-reference/methods/meta.md +213 -0
  90. package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
  91. package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
  92. package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
  93. package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
  94. package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
  95. package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
  96. package/skills/evolve-evals/references/sdk-reference/types.md +1255 -0
  97. package/skills/publish/SKILL.md +103 -79
  98. package/skills/rewardkit/SKILL.md +78 -23
  99. package/spec/openapi.yaml +164 -160
  100. package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
  101. package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
  102. package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
  103. package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
  104. package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
  105. package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
  106. package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
  107. package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
  108. package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
  109. package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
  110. package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
  111. package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
  112. package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
  113. package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
  114. package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
  115. package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
  116. package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
  117. package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
  118. package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
  119. package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
  120. package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
  121. package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
  122. package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
  123. package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
  124. package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
  125. package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
  126. package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
  127. package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
  128. package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
  129. package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
  130. package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
  131. package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
  132. package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
  133. package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
  134. package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
  135. package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
  136. package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
  137. package/skills/evolve-evals/references/index.mdx +0 -48
  138. package/skills/evolve-evals/references/sdk/python.mdx +0 -161
  139. package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
  140. package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
  141. package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
  142. package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
  143. package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
  144. package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
  145. package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
  146. package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
  147. package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
  148. package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
  149. package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
  150. package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
  151. package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
  152. package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
  153. package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
@@ -0,0 +1,93 @@
1
+ ---
2
+ title: "Multi-step tasks"
3
+ description: "Give the agent a sequence of instructions while preserving its files."
4
+ ---
5
+
6
+ A multi-step task runs ordered instructions in one shared environment. Files persist between steps; each step starts a fresh agent conversation and receives its own verification result.
7
+
8
+ ```mermaid
9
+ %%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 20, "padding": 10, "subGraphTitleMargin": {"top": 8, "bottom": 24}}, "themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}}}%%
10
+ flowchart TB
11
+ subgraph shared["One shared environment"]
12
+ A["1. Start environment"] --> B["2. Step 1<br/>New conversation"]
13
+ B --> C["3. Verify step 1"]
14
+ C -->|"Threshold met"| D["4. Step 2<br/>New conversation · same files"]
15
+ D --> E["5. Verify step 2"]
16
+ end
17
+ E --> F["6. Select trial reward<br/>mean or final"]
18
+ C -->|"Threshold missed<br/>Skip step 2"| F
19
+ ```
20
+
21
+ This two-step example uses the same environment for both agent steps and their verifiers. Reward thresholds and failure rules decide whether later steps run.
22
+
23
+ ## Directory layout
24
+
25
+ - migrate-then-test/
26
+ - task.toml
27
+ - environment/
28
+ - Dockerfile
29
+ - tests/
30
+ - test.sh
31
+ - steps/
32
+ - 01-migrate/
33
+ - instruction.md
34
+ - tests/
35
+ - test.sh
36
+ - workdir/
37
+ - setup.sh
38
+ - 02-test/
39
+ - instruction.md
40
+ - tests/
41
+ - test.sh
42
+
43
+ The root `tests/` folder can hold shared verifier files. Each step can include optional `workdir/` files and a `setup.sh` inside that folder.
44
+
45
+ Each step needs an instruction and a test script, either its own or the shared root `tests/test.sh`. Step test files override shared files with the same path.
46
+
47
+ ## Declare the order
48
+
49
+ ```toml task.toml
50
+ multi_step_reward_strategy = "mean"
51
+
52
+ [[steps]]
53
+ name = "01-migrate"
54
+ min_reward = 0.5
55
+
56
+ [[steps]]
57
+ name = "02-test"
58
+
59
+ [steps.agent]
60
+ timeout_sec = 600
61
+ ```
62
+
63
+ Every declared step must have a matching directory, and every step directory must be declared. There is no root instruction for this layout.
64
+
65
+ Before a step starts, its optional `workdir/` files are copied into the working directory and `setup.sh` runs if present.
66
+
67
+ ## Rewards and stopping
68
+
69
+ | Setting | Meaning |
70
+ | --- | --- |
71
+ | `mean` | Average each metric across steps with a verifier result; a missing metric counts as `0`. This is the default |
72
+ | `final` | Use the last executed step's result, including a step that stopped the trial early |
73
+ | `min_reward = 0.5` | Require the `reward` key to be at least `0.5`; a missing key also stops the trial |
74
+ | `min_reward = { accuracy = 0.8 }` | Require each named metric to meet its threshold; a missing metric also stops the trial |
75
+
76
+ A setup or healthcheck failure stops the remaining steps. A step error with no verifier result also stops the trial. An agent error alone can still continue when verification returns a result and the reward thresholds pass. Infrastructure failures stop the trial.
77
+
78
+ The trial exposes individual outcomes in `step_results`. A retry starts again at step 1.
79
+
80
+ ## Current limits
81
+
82
+ | Supported per step | Set once for the whole task |
83
+ | --- | --- |
84
+ | Instruction and tests | Environment image and agent user |
85
+ | Agent and verifier timeouts | Network policies |
86
+ | Verifier environment variables | Shared verifier mode |
87
+ | Healthcheck | Sandbox provider |
88
+
89
+ Multi-step tasks require shared verification. Compose, separate verification, per-step agent users, per-step network policies, per-step verifier environments, and per-step collection hooks are rejected. Verifiers run as root. These trials cannot be regraded.
90
+
91
+ `steps.artifacts` does not create a separate snapshot after each step. Save intermediate outputs under distinct paths when you need to inspect them later.
92
+
93
+ To seed the first conversation on Claude Code or Codex, put an ATIF `trajectory.json` inside the first step’s directory, beside its `instruction.md`. A root trajectory or a trajectory in a later step is rejected. Later steps start fresh conversations.
@@ -0,0 +1,341 @@
1
+ ---
2
+ title: "Reward Kit"
3
+ description: "Combine reusable checks, custom Python, and model judges into reward scores."
4
+ ---
5
+
6
+ Reward Kit is the `harbor-rewardkit` Python package. It discovers checks in `tests/`, runs them against the agent's workspace, and writes a reward plus detailed findings.
7
+
8
+ This guide uses the published **0.2.1** package.
9
+
10
+ | Use | For |
11
+ | --- | --- |
12
+ | Built-in checks | Files, commands, JSON, CSV, databases, HTTP, and images |
13
+ | Custom Python | Rules specific to your task |
14
+ | Model judges | Criteria that need a written assessment |
15
+
16
+ ## Build a deterministic verifier
17
+
18
+ Start with [your first task](/getting-started/first-task). Keep its instruction, `task.toml`, and solution. Replace the environment Dockerfile and test files with these four files.
19
+
20
+ This example uses shared verification and requires the exact answer, including its final newline.
21
+
22
+ - hello-world/
23
+ - environment/
24
+ - Dockerfile
25
+ - tests/
26
+ - test.sh
27
+ - checks.py
28
+ - reward.toml
29
+
30
+ ### environment/Dockerfile
31
+
32
+ Install Reward Kit when the image builds. Python 3.12 or newer is required; no package download is needed during verification.
33
+
34
+ ```dockerfile
35
+ FROM python:3.12-slim
36
+ RUN pip install --no-cache-dir \
37
+ 'harbor-rewardkit==0.2.1'
38
+ WORKDIR /app
39
+ ```
40
+
41
+ ### tests/checks.py
42
+
43
+ Combine a built-in file check with one custom criterion.
44
+
45
+ ```python
46
+ from pathlib import Path
47
+ import rewardkit as rk
48
+
49
+ rk.file_exists("answer.txt")
50
+
51
+ @rk.criterion
52
+ def exact_answer(
53
+ workspace: Path,
54
+ ) -> bool:
55
+ path = workspace / "answer.txt"
56
+ if not path.is_file():
57
+ return False
58
+ expected = "Hello, world!\n"
59
+ actual = path.read_text()
60
+ return actual == expected
61
+ ```
62
+
63
+ ### tests/reward.toml
64
+
65
+ Require both checks in `checks.py` to pass.
66
+
67
+ ```toml
68
+ [scoring.checks]
69
+ aggregation = "all-pass"
70
+ ```
71
+
72
+ ### tests/test.sh
73
+
74
+ Run the installed package against `/app` and write the standard reward file.
75
+
76
+ ```bash
77
+ #!/bin/bash
78
+ set -euo pipefail
79
+ python3 -m rewardkit /tests \
80
+ --workspace /app \
81
+ --output \
82
+ /logs/verifier/reward.json
83
+ ```
84
+
85
+ The exact answer scores `1`. A missing file, different text, or missing final newline scores `0`.
86
+
87
+ Reward Kit also writes `reward-details.json` beside `reward.json`, with the individual checks and their results. Use the [first-task workflow](/getting-started/first-task#2-validate-the-metadata) to validate, publish, check, and evaluate this version.
88
+
89
+ ## Write your own checks
90
+
91
+ A criterion receives `workspace: Path` and returns a boolean or number. The example's function has no other arguments, so `@rk.criterion` registers it automatically.
92
+
93
+ For reusable criteria with additional arguments, register each check through `rk`:
94
+
95
+ ```python tests/lines.py
96
+ from pathlib import Path
97
+ import rewardkit as rk
98
+
99
+ @rk.criterion
100
+ def has_lines(
101
+ workspace: Path,
102
+ count: int,
103
+ ) -> bool:
104
+ path = workspace / "answer.txt"
105
+ if not path.is_file():
106
+ return False
107
+ lines = path.read_text().splitlines()
108
+ return len(lines) >= count
109
+
110
+ rk.has_lines(5, weight=2.0)
111
+ rk.has_lines(10, weight=1.0)
112
+ ```
113
+
114
+ Handle expected failures by returning `False` or `0`. An unhandled exception aborts the verifier; it does not become a failed criterion automatically.
115
+
116
+ **Note:**
117
+
118
+ `file_matches()` ignores leading and trailing whitespace. Use a custom check, as above, when exact whitespace matters.
119
+
120
+ ## Built-in criteria
121
+
122
+ Call these through `rk`. File, data, and image paths resolve from `--workspace`; absolute paths stay absolute. All checks also accept `weight=1.0`, a custom `name`, and `isolated=False`.
123
+
124
+ ### Files and text
125
+
126
+ - `file_exists(path)` and `file_not_exists(path)`: test whether a path exists. Directories count too.
127
+
128
+ - `file_contains(path, text)`: find a literal substring.
129
+
130
+ - `file_contains_regex(path, pattern)`: find a Python regular-expression match.
131
+
132
+ - `file_matches(path, expected)`: compare text after trimming both ends.
133
+
134
+ - `files_equal(path1, path2)`: compare two text files after trimming both ends.
135
+
136
+ - `diff_ratio(path, expected)`: return a text-similarity ratio from `0` to `1`, after trimming both ends.
137
+
138
+ ### Commands
139
+
140
+ All four take `cmd`, optional `cwd`, and `timeout=30` seconds. The default working directory is the workspace.
141
+
142
+ - `command_succeeds(cmd)`: require exit code `0`.
143
+
144
+ - `command_output_contains(cmd, text)`: find a substring in stdout.
145
+
146
+ - `command_output_matches(cmd, expected)`: compare stdout after trimming both ends.
147
+
148
+ - `command_output_matches_regex(cmd, pattern)`: search stdout with a Python regular expression.
149
+
150
+ Output checks inspect stdout regardless of the exit code. A timeout fails the check.
151
+
152
+ ### JSON, tables, and databases
153
+
154
+ - `json_key_equals(path, key, expected)`: compare a top-level JSON object key.
155
+
156
+ - `json_path_equals(path, json_path, expected)`: follow dot-separated keys and list indices, such as `items.0.name`. This is not JSONPath syntax.
157
+
158
+ - `csv_cell_equals(path, row, col, expected)`: compare a string cell. Indices start at `0`. A named column treats the first row as a header; an integer column counts that header as row `0`.
159
+
160
+ - `xlsx_cell_equals(path, cell, expected, sheet=None)`: compare a cell such as `A1`. Defaults to the active sheet and reads cached formula values. Requires the `documents` extra.
161
+
162
+ - `sqlite_query_equals(db_path, query, expected)`: compare the first column of the first result row.
163
+
164
+ ### HTTP and images
165
+
166
+ - `http_status_equals(url, status=200)`: send a GET request and compare its status, including HTTP error statuses.
167
+
168
+ - `http_response_contains(url, text)`: find text in a GET response body. Request errors fail the check. Both HTTP criteria accept `timeout=10` seconds and require network access to the URL.
169
+
170
+ - `image_size_equals(path, width, height)`: compare pixel dimensions.
171
+
172
+ - `image_similarity(path1, path2)`: return the fraction of exactly matching RGBA pixels; different dimensions score `0`. Both image criteria require the `image` extra.
173
+
174
+ ### Agent trajectories
175
+
176
+ Supply an ATIF JSON file in the verifier sandbox and pass its `path`. Do not assume an exported trial trace is already present there.
177
+
178
+ - `trajectory_tool_used(tool_name, min_count=1)`: require at least this many calls with the matching function name.
179
+
180
+ - `trajectory_tool_not_used(tool_name)`: require no matching calls.
181
+
182
+ - `trajectory_turn_count(max_turns)`: score `1` within the agent-turn limit, then decrease linearly to `0` at twice the limit. Use a positive limit.
183
+
184
+ All three accept `path`; the package default is `/logs/agent/trajectory.json`. A missing trajectory fails the check, including `trajectory_tool_not_used`.
185
+
186
+ `isolated=True` needs working overlayfs support in the verifier environment. Installing Reward Kit alone does not provide that capability.
187
+
188
+ ## Combine scores
189
+
190
+ Each Python file that registers checks and each judge TOML produces a score. Criteria use a weighted mean by default, with weight `1` each.
191
+
192
+ | Configuration | Controls |
193
+ | --- | --- |
194
+ | `[scoring.checks]` in `reward.toml` | Criteria in `checks.py` |
195
+ | `[scoring]` in a judge TOML | Criteria in that judge file |
196
+ | Named `[[reward]]` in root `reward.toml` | Combined output score |
197
+
198
+ | Aggregation | Result |
199
+ | --- | --- |
200
+ | `weighted-mean` | Weighted average |
201
+ | `weighted-sum` | Sum of score × weight, without normalization |
202
+ | `all-pass` | `1` when every score is greater than `0` |
203
+ | `any-pass` | `1` when any score is greater than `0` |
204
+ | `threshold` | `1` when the weighted mean reaches `threshold`, default `0.5` |
205
+ | `required-pass` | `1` when every non-optional criterion scores above `0`; requires at least one |
206
+
207
+ With only files directly under `tests/`, Reward Kit combines their scores into `reward`. A judge file can set `[judge].weight`; otherwise each file has equal weight.
208
+
209
+ ### Keep separate dimensions and a primary reward
210
+
211
+ Subfolders expose separate score keys. For example, use this layout to keep correctness and quality visible:
212
+
213
+ - tests/
214
+ - test.sh
215
+ - reward.toml
216
+ - correctness/
217
+ - checks.py
218
+ - quality/
219
+ - judge.toml
220
+
221
+ In the root `tests/reward.toml`, add a named aggregation:
222
+
223
+ ```toml
224
+ [[reward]]
225
+ name = "reward"
226
+ aggregation = "weighted-mean"
227
+
228
+ [reward.weights]
229
+ correctness = 3
230
+ quality = 1
231
+ ```
232
+
233
+ This keeps the two dimensions and adds their weighted mean under `reward`.
234
+
235
+ Evolve reads `reward` as the primary score, or the only value in a one-key result. Multiple dimensions without `reward` remain available as metrics but have no primary score.
236
+
237
+ Keep the primary score within `[0, 1]`. Reward Kit does not clamp custom numbers or `weighted-sum` results; an out-of-range primary score becomes `SCORING_ERROR` on Evolve.
238
+
239
+ ## Add a model judge
240
+
241
+ For a task that produces `/app/report.md`, a judge can assess the explanation. Install Reward Kit in the image as above, then add this file beside the programmatic checks:
242
+
243
+ ```toml tests/judge.toml
244
+ [judge]
245
+ judge = "anthropic/claude-sonnet-4-6"
246
+ files = ["/app/report.md"]
247
+ timeout = 120
248
+
249
+ [[criterion]]
250
+ name = "clear-conclusion"
251
+ description = """
252
+ Does the report state a clear conclusion?
253
+ """
254
+ type = "binary"
255
+
256
+ [[criterion]]
257
+ name = "supports-conclusion"
258
+ description = """
259
+ How well does the report support
260
+ its conclusion with evidence?
261
+ """
262
+ type = "likert"
263
+ points = 5
264
+ weight = 2.0
265
+ ```
266
+
267
+ Give the verifier enough time and request the judge credential in `task.toml`. Update the existing `[verifier]` table rather than adding a second one:
268
+
269
+ ```toml task.toml
270
+ [verifier]
271
+ environment_mode = "shared"
272
+ timeout_sec = 180
273
+
274
+ [verifier.env]
275
+ ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
276
+ ```
277
+
278
+ Write the template literally. Evolve supplies a scoped gateway credential and base URL at verification time; you do not place a provider key in the task. Judge calls incur usage and can reach the gateway under `no-network`.
279
+
280
+ Automatic credential detection reads judge TOMLs at the root of a flat `tests/` folder, or one level below it when subfolders exist. Reward Kit itself discovers deeper layouts; keep the explicit credential request above when using them.
281
+
282
+ | Judge value | Managed support |
283
+ | --- | --- |
284
+ | `anthropic/<model>` | Anthropic-family LLM judge |
285
+ | `openai/<model>` | OpenAI-family LLM judge; request `OPENAI_API_KEY` instead |
286
+ | `claude-code` or `codex` | Agent judge; also set `model` and provide its CLI in the verifier image |
287
+
288
+ Use [separate verification](/core-concepts/task-verifiers#separate-verification) when the verifier needs its own image. Declare the report in `artifacts` so it reaches that sandbox. The managed judge path does not provide verifier MCP tools or arbitrary provider credentials.
289
+
290
+ ### Judge options
291
+
292
+ | Option | Behavior |
293
+ | --- | --- |
294
+ | `type = "binary"` | Yes/no becomes `1`/`0` |
295
+ | `type = "likert"`, `points = 5` | A 1–5 rating becomes a score in `[0, 1]` |
296
+ | `type = "numeric"`, `min`, `max` | Normalize the chosen numeric range to `[0, 1]` |
297
+ | `[judge].mode = "individual"` | Grade each criterion separately; allows per-criterion `files` for LLM judges |
298
+
299
+ The default mode is `batched`. A criterion without its own files uses `[judge].files`.
300
+
301
+ ### Additional judge and criterion options
302
+
303
+ In `[judge]`:
304
+
305
+ | Option | Meaning |
306
+ | --- | --- |
307
+ | `timeout` | Response limit in seconds; default `300` |
308
+ | `reasoning_effort` | LLM reasoning setting; default `medium` |
309
+ | `reference` | File or directory supplied as reference material to an LLM judge |
310
+ | `atif-trajectory` | Path to an ATIF file available inside the verifier |
311
+ | `prompt_template` | `.txt` or `.md` path relative to the judge TOML; must contain `{criteria}` |
312
+ | `weight` | This judge file's weight when combined with other files; default `1` |
313
+ | `cwd` | Agent judge's working directory; defaults to the workspace |
314
+ | `version` | Agent CLI version to use |
315
+ | `isolated` | Agent workspace isolation; default `false`, requires overlayfs support |
316
+
317
+ In each `[[criterion]]`, `name` identifies the result and `description` defines the check. `weight` defaults to `1`. `negate = true` changes the score to `1 - score`. `optional = true` excludes that criterion from `required-pass`; other aggregations still include it.
318
+
319
+ Keep the task's verifier timeout long enough for the judge. Agent CLIs must be available in the verifier environment; a blocked runtime installer cannot supply them under `no-network`.
320
+
321
+ For a per-job override, use `--ve REWARDKIT_JUDGE=<judge>` with `evolve run`. `--ve REWARDKIT_MODEL=<model>` selects the model for an agent judge. These are the only two job-level verifier environment overrides; request the matching provider credential family in the task.
322
+
323
+ ## Dependencies and network access
324
+
325
+ | Need | Install in the verifier's image |
326
+ | --- | --- |
327
+ | Core checks and text judges | `harbor-rewardkit==0.2.1` |
328
+ | PDF, DOCX, PPTX, or XLSX support | `harbor-rewardkit[documents]==0.2.1` |
329
+ | Image criteria | `harbor-rewardkit[image]==0.2.1` |
330
+
331
+ Evolve stages an offline Reward Kit bundle for judge-enabled runs, not for every deterministic verifier. It contains the base package and dependencies, not optional extras. The bundled `uvx --from harbor-rewardkit...` path cannot fetch a Reward Kit version or extra absent from that bundle.
332
+
333
+ Installing the package in the image and using `python3 -m rewardkit` makes the dependency choice explicit. HTTP checks and other external calls still need [network access](/core-concepts/task-config#network-access) to their destinations.
334
+
335
+ **[Verifier environments](/core-concepts/task-verifiers)**
336
+
337
+ Choose shared or separate verification and transfer the required files.
338
+
339
+ **[Reward Kit skill](/cli-reference/skills)**
340
+
341
+ Load `evolve skills get rewardkit` for an agent authoring a verifier.
@@ -0,0 +1,73 @@
1
+ ---
2
+ title: "Sandboxes"
3
+ description: "Choose a cloud provider and understand how task requirements affect placement."
4
+ ---
5
+
6
+ Each trial runs in its own cloud sandbox. Use `-e` to request E2B, Daytona, or Modal; the default is Daytona.
7
+
8
+ ```bash
9
+ evolve run -d harbor-examples@1.0 -i hello-world \
10
+ -a codex -m gpt-5.6-luna -e modal \
11
+ --max-trial-spend 1 --max-retries 0 --watch
12
+ ```
13
+
14
+ ## Task requirements come first
15
+
16
+ Evolve checks the requested provider against the task's image, resources, network policy, Compose services, and GPUs.
17
+
18
+ ```mermaid
19
+ %%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 20, "padding": 10}, "themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}}}%%
20
+ flowchart TB
21
+ A["1. Check requested<br/>provider"] -->|"Fits task"| B["2. Use it"]
22
+ A -->|"Does not fit"| C["2. Check alternatives<br/>E2B → Modal<br/>→ Daytona"]
23
+ C -->|"Match"| D["3. Use fallback"]
24
+ C -->|"No match"| E["3. Reject"]
25
+ ```
26
+
27
+ A fallback is recorded as `sandbox_provider_degrade` on the trial. The requested provider on the job and the actual provider on a trial can differ.
28
+
29
+ If no provider can satisfy the task, Evolve rejects the configuration.
30
+
31
+ ```bash
32
+ evolve dataset show harbor-examples@1.0
33
+ ```
34
+
35
+ Inspect the per-task provider verdicts before starting a larger run. A compatible declaration does not guarantee immediate provider capacity.
36
+
37
+ ## Supported configurations
38
+
39
+ | Capability | E2B | Daytona | Modal |
40
+ | --- | --- | --- | --- |
41
+ | Single-container tasks | Yes | Yes | Yes |
42
+ | Docker Compose | Yes | Yes | No |
43
+ | GPUs | No | Subject to live quota and GPU type | Yes |
44
+ | Compose with GPUs | No | No | No |
45
+ | Compose with `no-network` | No | No | No |
46
+
47
+ Network allowlist kinds and resource ceilings also differ. The live [`meta()` document](/sdk-reference/meta) publishes these constraints. Daytona disk and GPU limits depend on the platform's live provider quota.
48
+
49
+ **Note:**
50
+
51
+ A temporary capacity shortage can put a trial back in the queue even with `--max-retries 0`. Capacity waits are bounded and separate from ordinary retry attempts.
52
+
53
+ ## Images and startup
54
+
55
+ Dataset publication prepares task images. A provider may also need to prepare its own cached image or snapshot before the first trial. That first start can take longer than later trials using the cache.
56
+
57
+ Resources come from the task's `task.toml`. See [task configuration](/core-concepts/task-config) for defaults and how to request CPUs, memory, disk, and GPUs.
58
+
59
+ ## GPU costs
60
+
61
+ GPU compute is shown as an estimate, separate from model spend:
62
+
63
+ | Location | Field |
64
+ | --- | --- |
65
+ | Trial | `gpu_cost` |
66
+ | Job totals | `stats.gpu_cost_usd` |
67
+ | CLI trial view | `gpu compute (est.)` |
68
+
69
+ An unavailable estimate includes `unpriced_reason`. Do not add this estimate to the metered model cost and present it as the same charge.
70
+
71
+ ## Analysis and checks
72
+
73
+ These operations also use sandboxes. Choose their provider with `-e` on `evolve analyze` or `evolve check`, or `--analyze-provider` for analysis attached to a run. Their default is Daytona unless the platform operator configures another provider; the [SDK defaults](/sdk-reference/analyses#read-current-defaults) expose the resolved setting.
@@ -0,0 +1,86 @@
1
+ ---
2
+ title: "Secrets"
3
+ description: "Store a credential once and attach it to evaluation jobs by name."
4
+ ---
5
+
6
+ Secret values are write-only. Reads return metadata, and jobs store references to the selected secret rows.
7
+
8
+ ```text
9
+ 1. Store the secret
10
+ GITHUB_TOKEN
11
+ ↓
12
+ 2. Attach its reference
13
+ --secret GITHUB_TOKEN
14
+ ↓
15
+ 3. Run the trial
16
+ Value enters agent environment
17
+ ```
18
+
19
+ ## Store a value
20
+
21
+ ```bash
22
+ printf %s "$GITHUB_TOKEN" | evolve secrets set GITHUB_TOKEN --delivery direct
23
+ ```
24
+
25
+ Piping keeps the value out of the command text. Use `--label staging` to store a labeled variant; the default label is `default`.
26
+
27
+ Eval jobs support **direct** delivery: the value enters the agent sandbox. Brokered secrets are supported by other Evolve surfaces but are rejected for managed evals.
28
+
29
+ ## Attach it to a job
30
+
31
+ ```bash
32
+ evolve run -d my-dataset@1.0 -a codex -m gpt-5.6-luna \
33
+ --secret GITHUB_TOKEN@staging \
34
+ --max-trial-spend 1 --max-retries 0 --watch
35
+ ```
36
+
37
+ | Attachment | Meaning |
38
+ | --- | --- |
39
+ | `GITHUB_TOKEN` | Use the default label, or the only available label |
40
+ | `GITHUB_TOKEN@staging` | Select the `staging` value |
41
+ | `GITHUB_TOKEN@staging=GH_TOKEN` | Expose it as `GH_TOKEN` inside the sandbox |
42
+
43
+ If several labels exist and none is `default`, select one explicitly. Every arm receives the attachment. Reserved routing names and the `EVOLVE_` prefix cannot be overwritten.
44
+
45
+ ## Request it from a task
46
+
47
+ ```toml task.toml
48
+ [environment.env]
49
+ GITHUB_TOKEN = "${GITHUB_TOKEN}"
50
+ ```
51
+
52
+ The template matches the attached environment name. A missing required attachment rejects the job. For a non-secret fallback, use `"${NAME:-fallback}"`.
53
+
54
+ Task literals belong in `[environment.env]`. Managed eval jobs do not accept arbitrary `agent_env` values through `--ae`.
55
+
56
+ ## List, replace, or remove
57
+
58
+ ```bash
59
+ evolve secrets list
60
+ evolve secrets delete GITHUB_TOKEN --label staging
61
+ ```
62
+
63
+ A different value under an existing name and label is rejected with `secret_exists`. Use a new label, or delete the old row before storing its replacement. Removing or disabling a secret needed by an unfinished job can cause its later trials to fail.
64
+
65
+ ### Save and attach in one request
66
+
67
+ The SDK accepts both references and inline values in a job's `secrets` list:
68
+
69
+ ```json
70
+ {
71
+ "secrets": [
72
+ {"name": "GITHUB_TOKEN", "label": "staging"},
73
+ {"name": "SERVICE_TOKEN", "value": "VALUE", "delivery": "direct"}
74
+ ]
75
+ }
76
+ ```
77
+
78
+ An inline value is saved before the job stores its reference. Repeating the same value and delivery resolves to the existing row; a conflicting value is rejected. The CLI equivalent is `--secret-inline NAME[@LABEL]:DELIVERY=VALUE`.
79
+
80
+ **[Secret commands](/cli-reference/secrets)**
81
+
82
+ Storage, labels, and delivery options.
83
+
84
+ **[Secret SDK](/sdk-reference/secrets)**
85
+
86
+ Manage secrets from Python or TypeScript.
@@ -0,0 +1,81 @@
1
+ ---
2
+ title: "Sharing"
3
+ description: "Give someone read access to a job, check, or managed-agent session."
4
+ ---
5
+
6
+ Share a run you created by email or by an unlisted link. Both can be enabled at the same time.
7
+
8
+ | Method | Who can open it? | Sign-in |
9
+ | --- | --- | --- |
10
+ | Email | The account using that address. | Required. |
11
+ | Link | Anyone holding the link. | Not required. |
12
+
13
+ ### By email
14
+
15
+ ```bash
16
+ evolve job share "$JOB_ID" --email teammate@example.org
17
+ ```
18
+
19
+ This sends an email with the run's address. An eligible new recipient also receives a signup link. The run appears under their shared work after sign-in.
20
+
21
+ ### By link
22
+
23
+ ```bash
24
+ evolve job share "$JOB_ID" --link
25
+ ```
26
+
27
+ The command prints an unlisted URL. Keep using that URL until you revoke it.
28
+
29
+ Only the creator can share or revoke a share. Read access does not grant cancellation, retry, analysis, or deletion permission. A recipient who is already a team member retains their separate membership permissions.
30
+
31
+ ## Inspect or revoke access
32
+
33
+ ```bash
34
+ evolve job shares "$JOB_ID"
35
+ evolve job unshare "$JOB_ID" --email teammate@example.org
36
+ evolve job unshare "$JOB_ID" --link
37
+ ```
38
+
39
+ Revoking a link disables it immediately. Enabling link sharing later produces a new link.
40
+
41
+ The same commands apply to other run types:
42
+
43
+ | Run | Commands |
44
+ | --- | --- |
45
+ | Job | `evolve job share`, `unshare`, `shares` |
46
+ | Check | `evolve check share`, `unshare`, `shares` |
47
+ | Managed-agent session | `evolve session share`, `unshare`, `shares` |
48
+
49
+ ## What a link shows
50
+
51
+ | Run | Readable content |
52
+ | --- | --- |
53
+ | Job | Summary, trials, traces, and available permitted logs and files. |
54
+ | Check | Report, task-check results, and downloadable check archive. |
55
+ | Session | Transcript and trace download. Browser replay and session filesystem are not included. |
56
+
57
+ Sharing a run does not make its dataset public. Task-file access still follows the underlying dataset rules.
58
+
59
+ ## In the dashboard
60
+
61
+ ### 1. Open a run you created
62
+
63
+ Open the job, check, or session, then choose **Share**.
64
+
65
+ ### 2. Choose the audience
66
+
67
+ Enter an email address and invite, or enable **Anyone with the link**.
68
+
69
+ ### 3. Review access later
70
+
71
+ Return to the same dialog to inspect recipients or disable the link.
72
+
73
+ Email-shared runs appear in **Shared with me** under your personal workspace's Traces page.
74
+
75
+ **[Share commands](/cli-reference/job)**
76
+
77
+ Exact flags and share-state output.
78
+
79
+ **[Teams](/core-concepts/teams)**
80
+
81
+ Access through workspace membership.