@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{chunk-DK2MGUPB.js → chunk-DI6LPW2B.js} +2 -2
- package/dist/cli/index.cjs +29 -29
- package/dist/cli/index.d.cts +3 -3
- package/dist/cli/index.d.ts +3 -3
- package/dist/cli/index.js +18 -18
- package/dist/index.cjs +64 -60
- package/dist/index.d.cts +297 -18
- package/dist/index.d.ts +297 -18
- package/dist/index.js +48 -44
- package/dist/{types-B_qmzlMk.d.cts → types-Dj0iMl5V.d.cts} +70 -90
- package/dist/{types-B_qmzlMk.d.ts → types-Dj0iMl5V.d.ts} +70 -90
- package/harness-capabilities.json +285 -0
- package/package.json +9 -6
- package/skills/create-adapter/SKILL.md +71 -26
- package/skills/create-task/SKILL.md +124 -66
- package/skills/evolve/SKILL.md +37 -20
- package/skills/evolve-evals/SKILL.md +142 -69
- package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
- package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
- package/skills/evolve-evals/references/cli-reference/analyze.md +97 -0
- package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
- package/skills/evolve-evals/references/cli-reference/check.md +157 -0
- package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
- package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
- package/skills/evolve-evals/references/cli-reference/index.md +149 -0
- package/skills/evolve-evals/references/cli-reference/job.md +198 -0
- package/skills/evolve-evals/references/cli-reference/run.md +188 -0
- package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
- package/skills/evolve-evals/references/cli-reference/session.md +56 -0
- package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
- package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
- package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
- package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
- package/skills/evolve-evals/references/core-concepts/agents.md +101 -0
- package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
- package/skills/evolve-evals/references/core-concepts/check.md +116 -0
- package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
- package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
- package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
- package/skills/evolve-evals/references/core-concepts/index.md +52 -0
- package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
- package/skills/evolve-evals/references/core-concepts/models.md +73 -0
- package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
- package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
- package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
- package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
- package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
- package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
- package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
- package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
- package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
- package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
- package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
- package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
- package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
- package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
- package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
- package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
- package/skills/evolve-evals/references/dashboard/checks.md +54 -0
- package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
- package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
- package/skills/evolve-evals/references/dashboard/traces.md +54 -0
- package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
- package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
- package/skills/evolve-evals/references/getting-started/installation.md +79 -0
- package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
- package/skills/evolve-evals/references/index.md +58 -0
- package/skills/evolve-evals/references/sdk/http.md +110 -0
- package/skills/evolve-evals/references/sdk/python.md +108 -0
- package/skills/evolve-evals/references/sdk/typescript.md +104 -0
- package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
- package/skills/evolve-evals/references/sdk-reference/analyses.md +125 -0
- package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
- package/skills/evolve-evals/references/sdk-reference/checks.md +146 -0
- package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
- package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
- package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
- package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
- package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
- package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
- package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
- package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
- package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
- package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
- package/skills/evolve-evals/references/sdk-reference/methods/checks.md +462 -0
- package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
- package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
- package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1023 -0
- package/skills/evolve-evals/references/sdk-reference/methods/meta.md +213 -0
- package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
- package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
- package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
- package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
- package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
- package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
- package/skills/evolve-evals/references/sdk-reference/types.md +1255 -0
- package/skills/publish/SKILL.md +103 -79
- package/skills/rewardkit/SKILL.md +78 -23
- package/spec/openapi.yaml +164 -160
- package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
- package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
- package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
- package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
- package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
- package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
- package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
- package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
- package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
- package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
- package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
- package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
- package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
- package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
- package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
- package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
- package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
- package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
- package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
- package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
- package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
- package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
- package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
- package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
- package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
- package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
- package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
- package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
- package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
- package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
- package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
- package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
- package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
- package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
- package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
- package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
- package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
- package/skills/evolve-evals/references/index.mdx +0 -48
- package/skills/evolve-evals/references/sdk/python.mdx +0 -161
- package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
- package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
- package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
- package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
- package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
- package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
- package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
- package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
- package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
- package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
- package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
- package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
- package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
- package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
- package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Trials"
|
|
3
|
+
description: "Inspect one agent's attempt: what it did, what it earned, and why it stopped."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
A **trial** is one attempt at one task by one agent arm. Its ID works on its own; you do not need to supply the parent job ID.
|
|
7
|
+
|
|
8
|
+
Use an ID from `evolve job trials "$JOB_ID"` as `$TRIAL_ID`:
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
evolve trial show "$TRIAL_ID"
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## Follow the attempt
|
|
15
|
+
|
|
16
|
+
```text
|
|
17
|
+
1. QUEUED
|
|
18
|
+
Wait for dispatch
|
|
19
|
+
↓
|
|
20
|
+
2. RUNNING
|
|
21
|
+
Prepare → build → boot
|
|
22
|
+
→ install → run the agent
|
|
23
|
+
↓
|
|
24
|
+
3. SCORING
|
|
25
|
+
Run the verifier
|
|
26
|
+
↓
|
|
27
|
+
4. Settle
|
|
28
|
+
Record reward, exception,
|
|
29
|
+
spend, and files
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
This is the normal execution path. A trial can stop earlier with an error, budget refusal, or cancellation. `attempt_phase` distinguishes slow preparation from agent work or verification.
|
|
33
|
+
|
|
34
|
+
## Statuses
|
|
35
|
+
|
|
36
|
+
| Status | What it tells you |
|
|
37
|
+
| --- | --- |
|
|
38
|
+
| `QUEUED` | Waiting to run. |
|
|
39
|
+
| `RUNNING` | Execution is in progress. Check `attempt_phase`. |
|
|
40
|
+
| `SCORING` | The verifier is evaluating the result. |
|
|
41
|
+
| `SCORED` | A valid reward map was recorded. Zero is a real score. |
|
|
42
|
+
| `SCORING_ERROR` | The verifier output was invalid or verification failed. |
|
|
43
|
+
| `INFRASTRUCTURE_ERROR` | Infrastructure prevented a reliable result. Eligible failures may retry. |
|
|
44
|
+
| `BUDGET` | An account, team, or platform budget refused execution. |
|
|
45
|
+
| `INDETERMINATE` | No reliable scoring outcome was recorded. |
|
|
46
|
+
| `CANCELLED` | The trial was stopped. |
|
|
47
|
+
|
|
48
|
+
**Note:**
|
|
49
|
+
|
|
50
|
+
A missing reward is not zero. A multi-key reward map without a primary reward can be `SCORED` while the scalar `reward` is null; inspect `verifier_result.rewards`.
|
|
51
|
+
|
|
52
|
+
### Find the cause
|
|
53
|
+
|
|
54
|
+
| Read | Use it for |
|
|
55
|
+
| --- | --- |
|
|
56
|
+
| `exception_info` | Failure type and explanation. |
|
|
57
|
+
| `attempt_phase` | The current stage of execution. |
|
|
58
|
+
| `n_retries` and `retries` | Automatic retries and earlier outcomes. |
|
|
59
|
+
| `sandbox_provider_degrade` | Why the actual provider differs from the requested one. |
|
|
60
|
+
| `step_results` | Per-step outcomes for a multi-step task. |
|
|
61
|
+
|
|
62
|
+
For `BUDGET`, restore the relevant credits or budget before [resuming the job](/core-concepts/jobs#derive-a-new-job).
|
|
63
|
+
|
|
64
|
+
## The trace
|
|
65
|
+
|
|
66
|
+
The trace shows the prompt, messages, tool calls, and results. Search one trial or the whole job:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
evolve trial trace "$TRIAL_ID"
|
|
70
|
+
evolve trial trace "$TRIAL_ID" --grep 'permission denied' --tail 50
|
|
71
|
+
evolve job grep "$JOB_ID" 'permission denied'
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
| Filter | Meaning |
|
|
75
|
+
| --- | --- |
|
|
76
|
+
| `--type` | Exact event type. |
|
|
77
|
+
| `--grep` | Case-insensitive regular expression. |
|
|
78
|
+
| `--tail` | Last N matching events. |
|
|
79
|
+
| `--cursor` | Continue after an event position. |
|
|
80
|
+
| `--json` | Full events, one JSON object per line. |
|
|
81
|
+
|
|
82
|
+
`trial trace` reads the stored events available now. It does not stay connected for future events. Use the [dashboard viewer](/dashboard/trial-viewer) for an updating trace.
|
|
83
|
+
|
|
84
|
+
## Spend
|
|
85
|
+
|
|
86
|
+
| Field | Reading |
|
|
87
|
+
| --- | --- |
|
|
88
|
+
| `live_spent_usd` | Spend so far while running; a lower bound. |
|
|
89
|
+
| `agent_result.cost_usd` | Recorded agent spend after settlement. |
|
|
90
|
+
| `spend_source` | Whether the reading is final or provisional. |
|
|
91
|
+
| `judge_result` | Verifier judge spend and usage. |
|
|
92
|
+
| `usage` | Gateway token and cost accounting. |
|
|
93
|
+
|
|
94
|
+
`measured` is final. `measured_provisional` can increase. `assumed_cap` means spend was not measured; its stored zero is a placeholder, not proof of a free run.
|
|
95
|
+
|
|
96
|
+
## Artifacts
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
evolve trial download "$TRIAL_ID" --stream verifier
|
|
100
|
+
evolve trial download "$TRIAL_ID" --stream trace-atif
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
The verifier log explains grading. The ATIF trajectory provides a portable conversation record. Raw stdout, stderr, and captured agent-home files are also available.
|
|
104
|
+
|
|
105
|
+
## Download the trial tree
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
evolve trial download "$TRIAL_ID" -o trials/
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
The CLI saves `trials/<trial-id>/`. See [Trial outputs](/core-concepts/trial-outputs#the-trial-directory) for the tree and the difference from a full job archive.
|
|
112
|
+
|
|
113
|
+
## Act on one trial
|
|
114
|
+
|
|
115
|
+
| Command | Result |
|
|
116
|
+
| --- | --- |
|
|
117
|
+
| `evolve trial retry "$TRIAL_ID"` | New job that reruns this settled trial. |
|
|
118
|
+
| `evolve trial regrade "$TRIAL_ID"` | New job that reruns only an eligible verifier. |
|
|
119
|
+
| `evolve trial stop "$TRIAL_ID"` | Stop this live trial; leave its job running. |
|
|
120
|
+
|
|
121
|
+
Retry and regrade create new records. They do not replace the source result. Read [eligibility and selection rules](/core-concepts/jobs#derive-a-new-job) before using them.
|
|
122
|
+
|
|
123
|
+
**[Trial outputs](/core-concepts/trial-outputs)**
|
|
124
|
+
|
|
125
|
+
Files, traces, and the captured filesystem.
|
|
126
|
+
|
|
127
|
+
**[Trial reference](/cli-reference/trial)**
|
|
128
|
+
|
|
129
|
+
Every inspection and control command.
|
|
@@ -0,0 +1,240 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Upload an SDK run"
|
|
3
|
+
description: "Turn a recorded agent run into a job you can inspect and analyze."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
An SDK session records the conversation. An evaluation job holds trials and their results. Package the recorded run as a trial to use job inspection and analysis.
|
|
7
|
+
|
|
8
|
+
Already have a Harbor job folder? [Upload it directly](/core-concepts/upload). To share an SDK session without creating a job, use [session sharing](/cli-reference/session#share-a-session).
|
|
9
|
+
|
|
10
|
+
### 1. Find the local session log
|
|
11
|
+
|
|
12
|
+
Both SDKs write local JSONL logs on the machine running the SDK:
|
|
13
|
+
|
|
14
|
+
```text
|
|
15
|
+
~/.evolve-sdk/
|
|
16
|
+
└── observability/sessions/
|
|
17
|
+
└── <session>.jsonl
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
ls -lt ~/.evolve-sdk/observability/sessions/*.jsonl
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
Choose the file for your run. Its first `_meta` record contains the agent, model, session tag, and timestamp. `_prompt.text` records the input.
|
|
25
|
+
|
|
26
|
+
**Note:**
|
|
27
|
+
|
|
28
|
+
Use a fresh Codex or Claude instance with one foreground `run()`, no resumed context, and no shell commands. After it returns `exitCode: 0`, call `kill()` to close and flush the log.
|
|
29
|
+
|
|
30
|
+
A dashboard trace download contains parsed events, not the local harness stream. Do not pass it to this example.
|
|
31
|
+
|
|
32
|
+
### 2. Package one recorded run
|
|
33
|
+
|
|
34
|
+
Save the script below as `pack-sdk-run.py`. It uses Python's standard library and works with local logs from either SDK.
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
python3 pack-sdk-run.py \
|
|
38
|
+
--session "$HOME/.evolve-sdk/observability/sessions/<session>.jsonl" \
|
|
39
|
+
--task my-task \
|
|
40
|
+
--out ./sdk-job
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Replace the session filename and task name. Use the actual task key if you plan to link a dataset.
|
|
44
|
+
|
|
45
|
+
```text
|
|
46
|
+
sdk-job/
|
|
47
|
+
├── config.json Job name
|
|
48
|
+
├── result.json Stable source ID
|
|
49
|
+
└── my-task__sdk/
|
|
50
|
+
├── result.json Trial metadata and scores
|
|
51
|
+
└── agent/
|
|
52
|
+
├── trajectory.json Recorded opening prompt
|
|
53
|
+
└── .codex/sessions/… Original Codex log
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
For Claude, the original log goes under `agent/.claude/projects/sdk-upload/` instead.
|
|
57
|
+
|
|
58
|
+
| Source | Destination |
|
|
59
|
+
| --- | --- |
|
|
60
|
+
| `_meta.agent` and `_meta.model` | `agent_info` |
|
|
61
|
+
| `_prompt.text` | First user step in `trajectory.json` |
|
|
62
|
+
| Original session file | Recognized harness transcript path |
|
|
63
|
+
| Optional verifier scores | Recorded trial rewards |
|
|
64
|
+
|
|
65
|
+
### Copy the packaging script
|
|
66
|
+
|
|
67
|
+
```python
|
|
68
|
+
import argparse
|
|
69
|
+
import hashlib
|
|
70
|
+
import json
|
|
71
|
+
import math
|
|
72
|
+
import re
|
|
73
|
+
import uuid
|
|
74
|
+
from datetime import datetime
|
|
75
|
+
from pathlib import Path
|
|
76
|
+
|
|
77
|
+
parser = argparse.ArgumentParser()
|
|
78
|
+
parser.add_argument("--session", type=Path, required=True)
|
|
79
|
+
parser.add_argument("--task", required=True)
|
|
80
|
+
parser.add_argument("--out", type=Path, required=True)
|
|
81
|
+
parser.add_argument("--rewards", type=Path)
|
|
82
|
+
args = parser.parse_args()
|
|
83
|
+
|
|
84
|
+
def stop(message):
|
|
85
|
+
parser.error(message)
|
|
86
|
+
|
|
87
|
+
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}", args.task):
|
|
88
|
+
stop("Use the task directory name: letters, digits, dots, _ or -.")
|
|
89
|
+
raw = args.session.expanduser().read_bytes()
|
|
90
|
+
records = []
|
|
91
|
+
for line in raw.decode("utf-8").splitlines():
|
|
92
|
+
try:
|
|
93
|
+
record = json.loads(line)
|
|
94
|
+
except json.JSONDecodeError:
|
|
95
|
+
continue # Keep non-JSON output in the original file.
|
|
96
|
+
if isinstance(record, dict):
|
|
97
|
+
records.append(record)
|
|
98
|
+
|
|
99
|
+
metadata = [r["_meta"] for r in records if "_meta" in r]
|
|
100
|
+
prompts = [r["_prompt"] for r in records if "_prompt" in r]
|
|
101
|
+
if len(metadata) != 1 or len(prompts) != 1:
|
|
102
|
+
stop("Use a local log with one SDK run; do not combine runs or commands.")
|
|
103
|
+
if not any("_sessionEnd" in r for r in records):
|
|
104
|
+
stop("Finish the run and call kill() before packaging its log.")
|
|
105
|
+
meta, prompt = metadata[0], prompts[0]
|
|
106
|
+
if not isinstance(meta, dict) or not isinstance(prompt, dict):
|
|
107
|
+
stop("Expected local SDK metadata and prompt records.")
|
|
108
|
+
harness = meta.get("agent")
|
|
109
|
+
if harness not in ("codex", "claude"):
|
|
110
|
+
stop("This example supports local Codex and Claude SDK logs.")
|
|
111
|
+
if not isinstance(prompt.get("text"), str) or not prompt["text"]:
|
|
112
|
+
stop("The log has no recorded prompt text.")
|
|
113
|
+
if harness == "codex":
|
|
114
|
+
completed = sum(r.get("type") == "turn.completed" for r in records)
|
|
115
|
+
failed = any(r.get("type") == "turn.failed" for r in records)
|
|
116
|
+
if completed != 1 or failed:
|
|
117
|
+
stop("Expected one successfully completed Codex turn.")
|
|
118
|
+
else:
|
|
119
|
+
outcomes = [r for r in records if r.get("type") == "result"]
|
|
120
|
+
if len(outcomes) != 1 or outcomes[0].get("is_error") is not False:
|
|
121
|
+
stop("Expected one successful Claude result.")
|
|
122
|
+
|
|
123
|
+
rewards = None
|
|
124
|
+
if args.rewards:
|
|
125
|
+
rewards = json.loads(args.rewards.read_text())
|
|
126
|
+
if not isinstance(rewards, dict) or not rewards or not all(
|
|
127
|
+
isinstance(v, (int, float)) and not isinstance(v, bool)
|
|
128
|
+
and math.isfinite(v) for v in rewards.values()
|
|
129
|
+
):
|
|
130
|
+
stop("Rewards must be a nonempty JSON object of recorded numeric scores.")
|
|
131
|
+
|
|
132
|
+
source_id = str(uuid.uuid5(
|
|
133
|
+
uuid.NAMESPACE_URL, hashlib.sha256(raw).hexdigest() + ":" + args.task
|
|
134
|
+
))
|
|
135
|
+
trial_name = args.task + "__sdk"
|
|
136
|
+
agent_info = {"name": harness, "version": ""}
|
|
137
|
+
if meta.get("model"):
|
|
138
|
+
agent_info["model_info"] = {"name": meta["model"]}
|
|
139
|
+
result = {
|
|
140
|
+
"task_name": args.task, "trial_name": trial_name, "trial_uri": "",
|
|
141
|
+
"task_id": {}, "task_checksum": "", "config": {},
|
|
142
|
+
"agent_info": agent_info,
|
|
143
|
+
}
|
|
144
|
+
if rewards is not None:
|
|
145
|
+
result["verifier_result"] = {"rewards": rewards}
|
|
146
|
+
trajectory = {
|
|
147
|
+
"schema_version": "ATIF-v1.7", "session_id": meta["tag"],
|
|
148
|
+
"agent": {"name": harness, "version": ""},
|
|
149
|
+
"steps": [{"step_id": 1, "source": "user", "message": prompt["text"]}],
|
|
150
|
+
"notes": "Opening SDK prompt; agent activity is in the native transcript.",
|
|
151
|
+
}
|
|
152
|
+
if harness == "codex":
|
|
153
|
+
date = datetime.fromisoformat(meta["timestamp"].replace("Z", "+00:00"))
|
|
154
|
+
native = Path(".codex/sessions") / date.strftime("%Y/%m/%d") / f"rollout-{source_id}.jsonl"
|
|
155
|
+
else:
|
|
156
|
+
native = Path(".claude/projects/sdk-upload") / f"{source_id}.jsonl"
|
|
157
|
+
|
|
158
|
+
args.out.mkdir(parents=True, exist_ok=False)
|
|
159
|
+
trial = args.out / trial_name
|
|
160
|
+
destination = trial / "agent" / native
|
|
161
|
+
destination.parent.mkdir(parents=True)
|
|
162
|
+
destination.write_bytes(raw)
|
|
163
|
+
for path, value in (
|
|
164
|
+
(args.out / "config.json", {"job_name": args.out.name}),
|
|
165
|
+
(args.out / "result.json", {"id": source_id}),
|
|
166
|
+
(trial / "result.json", result),
|
|
167
|
+
(trial / "agent/trajectory.json", trajectory),
|
|
168
|
+
):
|
|
169
|
+
path.write_text(json.dumps(value, indent=2) + "\n")
|
|
170
|
+
print(f"Ready to upload: {args.out}")
|
|
171
|
+
print("Recorded scores included." if rewards is not None else "No score supplied.")
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Have verifier results? Add `--rewards ./recorded-rewards.json`, containing the verifier's numeric score map. Without it, the import has **no score**. Finishing an agent run does not prove that the task passed.
|
|
175
|
+
|
|
176
|
+
The script preserves the transcript. It does not collect sandbox files or infer costs, errors, or rewards. Add any recorded deliverables under `artifacts/`, verifier logs under `verifier/`, and known result fields using the [job layout](/core-concepts/upload#what-result-json-must-hold).
|
|
177
|
+
|
|
178
|
+
### 3. Upload the job folder
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
evolve upload ./sdk-job
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
If the run used a task from a published dataset, link it by task name:
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
evolve upload ./sdk-job -d my-dataset@1.0
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
Choose one command. The CLI waits for the import and prints its job ID. Uploading the same packaged run twice is refused.
|
|
191
|
+
|
|
192
|
+
### 4. Inspect the imported run
|
|
193
|
+
|
|
194
|
+
Set `$JOB_ID` to the returned ID:
|
|
195
|
+
|
|
196
|
+
```bash
|
|
197
|
+
evolve job show "$JOB_ID"
|
|
198
|
+
evolve job trials "$JOB_ID"
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Set `$TRIAL_ID` to a trial ID from that list, then read its trace:
|
|
202
|
+
|
|
203
|
+
```bash
|
|
204
|
+
evolve trial trace \
|
|
205
|
+
"$TRIAL_ID"
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
Check that the trial has the expected task, agent, model, prompt, and agent activity. Read the import report for skipped trials or missing task links. Then analyze it:
|
|
209
|
+
|
|
210
|
+
```bash
|
|
211
|
+
evolve analyze \
|
|
212
|
+
"$JOB_ID" \
|
|
213
|
+
--watch
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
Uploaded jobs support analysis. They cannot be resumed, retried, or regraded.
|
|
217
|
+
|
|
218
|
+
## Other session formats
|
|
219
|
+
|
|
220
|
+
### A failed or interrupted run
|
|
221
|
+
|
|
222
|
+
Failed runs can also be uploaded. Use the same [job layout](/core-concepts/upload#what-result-json-must-hold), preserve the transcript, and include the recorded `exception_info` when available. The example script stops when a successful harness completion is missing; closing a session alone does not prove completion.
|
|
223
|
+
|
|
224
|
+
Do not invent a reward or an exception from missing data. Without a recorded score or recognized failure, the imported trial is `INDETERMINATE`.
|
|
225
|
+
|
|
226
|
+
### A session with several runs or commands
|
|
227
|
+
|
|
228
|
+
A session is not automatically one trial. Choose the completed agent run and its task before packaging. SDK `_prompt` records can mark runs or shell commands; they do not identify which kind occurred.
|
|
229
|
+
|
|
230
|
+
Separate each evaluation attempt using the original call history and its transcript. Preserve any earlier conversation the attempt depended on. The example refuses combined logs because splitting on `_prompt` alone can misrepresent the run.
|
|
231
|
+
|
|
232
|
+
### A system prompt, another harness, or a dashboard download
|
|
233
|
+
|
|
234
|
+
Use a complete **ATIF trajectory** at `agent/trajectory.json` when the source is not a recognized native transcript. ATIF is the [shared trajectory format](/core-concepts/trial-outputs#the-atif-trajectory).
|
|
235
|
+
|
|
236
|
+
The SDK logger does not add the configured `systemPrompt` to its header. If you retained it, include its actual text as a `system` step in the complete trajectory. Do not reconstruct an unknown prompt.
|
|
237
|
+
|
|
238
|
+
When a recognized native transcript produces parsed events, Evolve uses those events and the opening user prompt; it does not merge in the rest of ATIF. To use your complete ATIF instead, keep original evidence under `agent/source/`, outside the recognized transcript paths.
|
|
239
|
+
|
|
240
|
+
A dashboard session download contains parsed session events. Convert those records to ATIF rather than renaming the file as a Codex or Claude native transcript.
|
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Upload a job"
|
|
3
|
+
description: "Bring existing Harbor-format results into Evolve for inspection, analysis, and sharing."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Upload results you already ran elsewhere. Evolve imports the records; it does not rerun the evaluation.
|
|
7
|
+
|
|
8
|
+
Have an SDK session log? Follow [Upload an SDK run](/core-concepts/upload-sdk-session) to find it, package one run, and inspect the imported trace.
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
evolve upload ./my-job -d harbor-examples@1.0
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
| Source | Command |
|
|
15
|
+
| --- | --- |
|
|
16
|
+
| Job directory | `evolve upload ./my-job` |
|
|
17
|
+
| Compressed archive | `evolve upload ./my-job.tar.gz` |
|
|
18
|
+
| Public HTTPS archive | `evolve upload --from https://example.org/my-job.tar.gz` |
|
|
19
|
+
|
|
20
|
+
The CLI waits for import by default. Use `--no-wait` to return the import ID immediately, then follow it:
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
evolve job import "$IMPORT_ID" --watch
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## What a job folder holds
|
|
27
|
+
|
|
28
|
+
A Harbor job directory can be uploaded directly. It requires `config.json` and `result.json` at the root, plus `result.json` in each trial folder.
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
my-job/
|
|
32
|
+
├── config.json
|
|
33
|
+
├── result.json
|
|
34
|
+
└── <trial-name>/
|
|
35
|
+
├── result.json
|
|
36
|
+
├── lock.json
|
|
37
|
+
├── agent/
|
|
38
|
+
│ ├── trajectory.json
|
|
39
|
+
│ ├── stdout.log
|
|
40
|
+
│ └── ...
|
|
41
|
+
├── verifier/
|
|
42
|
+
├── artifacts/
|
|
43
|
+
└── steps/
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
| Optional trial files | Content |
|
|
47
|
+
| --- | --- |
|
|
48
|
+
| `lock.json` | Task-content identity. |
|
|
49
|
+
| `agent/trajectory.json` | ATIF trace. |
|
|
50
|
+
| `agent/stdout.log` | Process output. |
|
|
51
|
+
| Other `agent/` files | Native sessions and other files. |
|
|
52
|
+
| `verifier/` | Verifier logs and rewards. |
|
|
53
|
+
| `artifacts/` | Collected outputs. |
|
|
54
|
+
| `steps/` | Multi-step results. |
|
|
55
|
+
|
|
56
|
+
Missing optional data stays missing. An imported trial without a trace does not acquire a synthetic trace.
|
|
57
|
+
|
|
58
|
+
## What `result.json` must hold
|
|
59
|
+
|
|
60
|
+
This is a minimal illustrative trial record with a reward. Replace the task, agent, model, and result with the facts of your run.
|
|
61
|
+
|
|
62
|
+
```json
|
|
63
|
+
{
|
|
64
|
+
"task_name": "hello-world",
|
|
65
|
+
"trial_name": "hello-world__attempt-1",
|
|
66
|
+
"trial_uri": "",
|
|
67
|
+
"task_id": {},
|
|
68
|
+
"task_checksum": "",
|
|
69
|
+
"config": {},
|
|
70
|
+
"agent_info": {
|
|
71
|
+
"name": "codex",
|
|
72
|
+
"version": "",
|
|
73
|
+
"model_info": { "name": "gpt-5.6-luna" }
|
|
74
|
+
},
|
|
75
|
+
"verifier_result": { "rewards": { "reward": 1.0 } }
|
|
76
|
+
}
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
The first seven top-level keys are required. `verifier_result` is optional. Include `agent_result` for recorded token and cost data, and `exception_info` for a failure.
|
|
80
|
+
|
|
81
|
+
## What lands
|
|
82
|
+
|
|
83
|
+
| Supplied data | Imported result |
|
|
84
|
+
| --- | --- |
|
|
85
|
+
| Nonempty numeric reward map | `SCORED`, with the recorded rewards. |
|
|
86
|
+
| Verifier result without rewards | `INDETERMINATE`. |
|
|
87
|
+
| No verifier result | Status follows a recognized failure type, otherwise `INDETERMINATE`. |
|
|
88
|
+
| Trace, home, verifier logs, artifacts, step files | Retained for inspection when present. |
|
|
89
|
+
| Existing `analysis.json` | Not imported as an Evolve analysis; analyze the uploaded job separately. |
|
|
90
|
+
|
|
91
|
+
Reported costs and totals remain identified as uploaded data. They are not charges for a new Evolve execution.
|
|
92
|
+
|
|
93
|
+
**Note:**
|
|
94
|
+
|
|
95
|
+
Uploaded jobs support inspection, download, analysis, and sharing. Resume, retry, and regrade are refused with `job_uploaded`.
|
|
96
|
+
|
|
97
|
+
## Task linkage
|
|
98
|
+
|
|
99
|
+
Analysis is more useful when it can also read the task files.
|
|
100
|
+
|
|
101
|
+
### 1. Upload trials
|
|
102
|
+
|
|
103
|
+
Supply the recorded job and trial files.
|
|
104
|
+
|
|
105
|
+
### 2. Resolve task identity
|
|
106
|
+
|
|
107
|
+
Use the first applicable case:
|
|
108
|
+
|
|
109
|
+
- **Case A: `-d name@version` supplied.** Match the task name in that version.
|
|
110
|
+
|
|
111
|
+
- **Case B: the archive names accessible dataset versions.** Match the task-content hash within those versions.
|
|
112
|
+
|
|
113
|
+
- **Case C: neither applies.** Match the task-content hash in the accessible catalog.
|
|
114
|
+
|
|
115
|
+
### 3. Analyze
|
|
116
|
+
|
|
117
|
+
- **Linked task:** Read the trial record and task files.
|
|
118
|
+
|
|
119
|
+
- **Unlinked task:** Read the trial record only. The missing link is reported.
|
|
120
|
+
|
|
121
|
+
A task name alone does not establish an automatic content match. `upload.task_links` records the matching method or why a link was not found.
|
|
122
|
+
|
|
123
|
+
## Runs made with the Evolve SDK
|
|
124
|
+
|
|
125
|
+
An individual managed-agent run is not a job archive. [Upload an SDK run](/core-concepts/upload-sdk-session) provides the local log path, field mapping, packaging script, and upload commands.
|
|
126
|
+
|
|
127
|
+
Use `agent/trajectory.json` for ATIF. Native transcripts must use the harness's recognized session path; arbitrary files under `agent/` are stored but are not automatically parsed as a trace.
|
|
128
|
+
|
|
129
|
+
### Recognized native session examples
|
|
130
|
+
|
|
131
|
+
```text
|
|
132
|
+
Codex
|
|
133
|
+
agent/.codex/sessions/
|
|
134
|
+
└── <year>/
|
|
135
|
+
└── <month>/
|
|
136
|
+
└── <day>/
|
|
137
|
+
└── rollout-<id>.jsonl
|
|
138
|
+
|
|
139
|
+
Claude
|
|
140
|
+
agent/.claude/projects/
|
|
141
|
+
└── <project>/
|
|
142
|
+
└── <session>.jsonl
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
A native transcript that produces parsed events takes precedence over ATIF. Otherwise, the parser falls back to ATIF. ATIF can include a `system` step when the system prompt is available in your source records.
|
|
146
|
+
|
|
147
|
+
## Bounds and duplicates
|
|
148
|
+
|
|
149
|
+
Inspect the import result, including skipped trials and their reasons. A completed import can have omissions.
|
|
150
|
+
|
|
151
|
+
| Refusal | Meaning |
|
|
152
|
+
| --- | --- |
|
|
153
|
+
| `job_already_uploaded` | This job was imported before. Delete the existing import's job before replacing it. |
|
|
154
|
+
| `invalid_trial` | A required trial result could not be parsed. |
|
|
155
|
+
| `trial_too_large` | A trial exceeded a physical file or parsing limit and was skipped. |
|
|
156
|
+
| `upload_too_large` | The archive or required job-level data exceeded an accepted limit. |
|
|
157
|
+
|
|
158
|
+
The capability document reports `limits.uploads.job_archive_bytes`. Local dataset and job archives larger than 256 MiB use resumable upload automatically.
|
|
159
|
+
|
|
160
|
+
**[Upload reference](/cli-reference/upload)**
|
|
161
|
+
|
|
162
|
+
Import options and progress commands.
|
|
163
|
+
|
|
164
|
+
**[Analyze imported results](/core-concepts/analyze)**
|
|
165
|
+
|
|
166
|
+
Review behavior using the retained evidence.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Checks"
|
|
3
|
+
description: "Read task-quality findings and open the evidence behind them."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Find a check in **Jobs** under kind **Check**, or open a job's **Check** tab to see checks of its tasks.
|
|
7
|
+
|
|
8
|
+

|
|
9
|
+
|
|
10
|
+
*Read the overall finding, then inspect the criterion evidence.*
|
|
11
|
+
|
|
12
|
+
```text
|
|
13
|
+
Check report
|
|
14
|
+
├── Source tasks + model + effort + cost
|
|
15
|
+
└── Task results
|
|
16
|
+
├── Summary chip (computed from the outcomes)
|
|
17
|
+
├── Criterion outcomes and evidence
|
|
18
|
+
└── Checker trace
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Start a check
|
|
22
|
+
|
|
23
|
+
Checks start from the CLI or SDK. A job with no task checks shows the corresponding dataset command:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
evolve check -d harbor-examples@1.0 -i hello-world --watch
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This reviews the task material. It does not run another evaluation job against your chosen evaluation agent.
|
|
30
|
+
|
|
31
|
+
## The check page
|
|
32
|
+
|
|
33
|
+
The header identifies the check, source, model, effort, status, and cost. Each task row opens that checker's [trace viewer](/dashboard/trial-viewer).
|
|
34
|
+
|
|
35
|
+
| Action | Availability |
|
|
36
|
+
| --- | --- |
|
|
37
|
+
| **Share** | Creator; manage link and email access. |
|
|
38
|
+
| **Stop** | Creator, while task checks are still live. |
|
|
39
|
+
| **Download** | Once the check is completed. |
|
|
40
|
+
|
|
41
|
+
## Read the finding
|
|
42
|
+
|
|
43
|
+
| Finding | Next step |
|
|
44
|
+
| --- | --- |
|
|
45
|
+
| **Has a problem** | Read the failed criterion and its evidence. |
|
|
46
|
+
| **Unclear** | Read what information was missing. |
|
|
47
|
+
| **No problem found** | Check execution findings before treating it as a runtime validation. |
|
|
48
|
+
| Failed task check | Read its failure; this is a failed review, not a task-quality result. |
|
|
49
|
+
|
|
50
|
+
The chip is a summary the dashboard computes from the criterion outcomes. Under a custom rubric the chip reads **Has a problem** when any criterion fails, **Unclear** when none fails but at least one is unknown, and **No problem found** otherwise. Under a rubric with the default criterion names the same rule applies, with two exceptions: an unknown on one of the default rubric's execution criteria does not make the chip unclear but marks it not executed; and `attempt_isolation` counts only when it fails. The chip and its not executed note summarize the outcomes; neither is a separate execution audit. See [the check result](/core-concepts/check#the-result).
|
|
51
|
+
|
|
52
|
+
**[Check reference](/cli-reference/check)**
|
|
53
|
+
|
|
54
|
+
Run checks, customize the rubric, and download reports.
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Datasets"
|
|
3
|
+
description: "Find a task set, inspect its versions, and check where its tasks can run."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
The **Datasets** page shows the catalog available in your workspace. Filter **All**, **Public**, or **Private**, then open a dataset.
|
|
7
|
+
|
|
8
|
+

|
|
9
|
+
|
|
10
|
+
*The public catalog. Open a dataset to inspect its tasks and versions.*
|
|
11
|
+
|
|
12
|
+
```text
|
|
13
|
+
Dataset
|
|
14
|
+
├── Active version
|
|
15
|
+
├── Tasks
|
|
16
|
+
└── Versions
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
- **Active version:** Used when a job omits a version.
|
|
20
|
+
|
|
21
|
+
- **Tasks:** What the selected version contains.
|
|
22
|
+
|
|
23
|
+
- **Versions:** Earlier and current published versions.
|
|
24
|
+
|
|
25
|
+
## Inspect tasks
|
|
26
|
+
|
|
27
|
+
The Tasks tab shows the selected version's tasks and their requirements:
|
|
28
|
+
|
|
29
|
+
| Column | Read it for |
|
|
30
|
+
| --- | --- |
|
|
31
|
+
| Task name | The key used by include/exclude filters. |
|
|
32
|
+
| GPUs | Whether the task needs GPU resources. |
|
|
33
|
+
| Timeouts | Agent and verifier time budgets. |
|
|
34
|
+
| Skills | Task-declared skills. |
|
|
35
|
+
| Providers | Compatibility and any fallback target. |
|
|
36
|
+
|
|
37
|
+
A provider arrow names the fallback target. Hover a refusal to read its reason. These compatibility summaries are advisory; job creation and dispatch validate the actual requirements.
|
|
38
|
+
|
|
39
|
+
## Choose a version
|
|
40
|
+
|
|
41
|
+
Open **Versions** to inspect provenance and task counts. Click a version to browse its tasks.
|
|
42
|
+
|
|
43
|
+
**Activate** changes which version a bare dataset name resolves to. An explicit reference such as `my-dataset@1.0` remains pinned to that version.
|
|
44
|
+
|
|
45
|
+
**Note:**
|
|
46
|
+
|
|
47
|
+
A ready version can contain failed task builds. Inspect the task/build details before assuming every task is runnable.
|
|
48
|
+
|
|
49
|
+
## Publish or manage
|
|
50
|
+
|
|
51
|
+
Publishing starts from the CLI or SDK:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
evolve dataset publish --dir ./tasks --name my-dataset --version 1.0 --watch
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
The dashboard exposes activation and deletion where your permissions allow them. Deleting a dataset removes its versions and task records; use the confirmation details before proceeding.
|
|
58
|
+
|
|
59
|
+
**[Dataset lifecycle](/core-concepts/datasets)**
|
|
60
|
+
|
|
61
|
+
Publishing, builds, versions, and activation.
|
|
62
|
+
|
|
63
|
+
**[Task format](/core-concepts/tasks)**
|
|
64
|
+
|
|
65
|
+
The files inside each task.
|