@evolvingmachines/evolve 0.0.64 → 0.0.65

package/skills/evolve-evals/references/dashboard/checks.mdx

@@ -1,6 +1,6 @@
 ---
 title: "Checks"
-description: "Read a job's task quality checks in the dashboard."
+description: "Read task quality checks in the dashboard, on a job's CHECK tab or on the check's own page."
 ---
 
 A job's CHECK tab lists the task quality checks run on the job's datasets.
@@ -13,3 +13,9 @@ A job's CHECK tab lists the task quality checks run on the job's datasets.
 Until a check exists, the tab shows the command that would start one, `evolve check -d <dataset@version>`, with the job's dataset filled in. Checks start from the command line; see [check](/core-concepts/check).
 
 Once a dataset has been checked, the tab lists one row per task with its dataset and its result. Click a row to open the check in the [trial viewer](/dashboard/trial-viewer): the events are the checker's own trace, and its result document is on the DOWNLOAD menu.
+
+## The check page
+
+A check is a job, so it sits in the [Jobs](/dashboard/jobs) list with KIND `CHECK`, and its row opens the check's own page. The band names the check, its status, what was checked, the checker's model and effort, its cost and when it started. SHARE is the creator's and opens the share dialog; STOP, for the creator, ends every task check still queued or running while any is live. DOWNLOAD saves a completed check as a `.tar.gz` in the standard check layout.
+
+Under the band, one row per task check with its dataset and its result, the same table as a job's CHECK tab. Click a row to open it in the [trial viewer](/dashboard/trial-viewer).

package/skills/evolve-evals/references/dashboard/jobs.mdx

@@ -3,14 +3,14 @@ title: "Jobs"
 description: "Start a job from the browser, follow it, and read its results."
 ---
 
-The Jobs page lists your jobs, newest first.
+The Jobs page lists your jobs, newest first. A task quality check is a job too, so the list holds both: the KIND column tells them apart, and a check row opens the [check's page](/dashboard/checks#the-check-page).
 
 {/* retake after the UI pass, from a non-admin account */}
 <Frame caption="The jobs list">
-  <img src="/images/jobs.png" alt="The Jobs page: a NEW JOB button, a search box, a status filter, and a table of jobs with status, datasets, agents, trials, cost and creation time." />
+  <img src="/images/jobs.png" alt="The Jobs page: a NEW JOB button, a search box, a status filter, and a table of jobs and checks with kind, status, datasets, agents, trials, cost and creation time." />
 </Frame>
 
-Search by name, and filter by status: ALL, QUEUED, RUNNING, CANCELLING, COMPLETED, CANCELLED or FAILED. Each row shows the job's datasets, its agents with their models, how many trials have settled out of the total, what it has spent, and when it started. Tick rows to cancel live jobs or delete finished ones.
+Search by name, and filter by status: ALL, QUEUED, RUNNING, CANCELLING, COMPLETED, CANCELLED or FAILED. A job row shows its datasets, its agents with their models, how many trials have settled out of the total, what it has spent, and when it started; a check row shows what was checked, the checker's model and effort, how many task checks have settled out of the total, the checker's cost, and when it started. Tick rows to cancel live jobs or delete finished ones; a check row cannot be ticked, its verbs live on its page.
 
 ## New job
 
@@ -36,7 +36,7 @@ The Summary rail counts datasets × agents × attempts. VALIDATE checks the whol
   <img src="/images/job.png" alt="A job's page: the job id and status in the header, CANCEL, RESUME, DOWNLOAD and RE-RUN JOB buttons, tiles for trials, errors, average reward, cost and max cost, and the OVERVIEW tab with a progress bar and a per-agent table." />
 </Frame>
 
-The header carries the job id and its status. CANCEL stops a running job. RESUME runs a finished job's failed or stopped trials again as a new job.
+The header carries the job id and its status. CANCEL stops a running job. RESUME runs a finished job's failed or stopped trials again as a new job. Both appear for the job's creator and the organization's members; a job shared with you by address opens without them.
 
 DOWNLOAD saves a finished job's results as a `.tar.gz` in the standard job layout; while the job runs it is disabled, with the reason on hover. RE-RUN JOB opens the New job form filled in with this job's choices.
 

package/skills/evolve-evals/references/dashboard/traces.mdx

@@ -3,14 +3,14 @@ title: "Traces"
 description: "Every recorded run in one list."
 ---
 
-The Traces page lists every run the platform recorded for you, newest first: the trials of your jobs, their regrades, analyses and checks, and the managed-agent sessions the platform recorded, which these pages do not cover.
+The Traces page lists every run of the workspace you are in, newest first: the trials of its jobs, their regrades, analyses and checks, and the managed-agent sessions the platform recorded, which these pages do not cover. In a team it is one list of every member's runs, with a CREATED BY column after the trace tag. In your personal workspace it has two tabs: MINE, your own runs in any team, and SHARED WITH ME, what teammates made plus runs shared with you by email.
 
 {/* retake after the UI pass, from a non-admin account */}
 <Frame caption="The traces list">
   <img src="/images/traces.png" alt="The Traces page: a search box and a table of runs with a trace tag, the sandbox provider, the sandbox id, the agent and model, the start time, the cost and the status." />
 </Frame>
 
-Each row carries a tag naming the kind of run (EVAL for a trial, AGENT for a managed-agent session, REGRADE, ANALYZE, or CHECK), the task or session name, the dataset, the reward when there is one, the sandbox provider and sandbox id, the agent and model, when it started, what it spent, and its status. Search by name. Tick rows to stop running ones or delete finished ones.
+Each row carries a tag naming the kind of run (EVAL for a trial, AGENT for a managed-agent session, REGRADE, ANALYZE, or CHECK), the task or session name, the dataset, the reward when there is one, the sandbox provider and sandbox id, the agent and model, when it started, what it spent, and its status. Search by name. Tick rows to stop running ones or delete finished ones; in a team's list and under SHARED WITH ME only the rows you created can be ticked, since stop and delete are the creator's.
 
 Click a row to open it in the [trial viewer](/dashboard/trial-viewer).
 

package/skills/evolve-evals/references/dashboard/trial-viewer.mdx

@@ -12,7 +12,7 @@ Click a trial on a job's TRIALS tab, or a row on the Traces page, to open it.
 
 The header names the run: its kind (EVAL for a trial, AGENT for a managed-agent session, REGRADE, ANALYZE, or CHECK), its task, its agent and model, its sandbox id, what it spent, and its status.
 
-BACK returns to where you came from. RETRY runs a settled trial again as a new one-trial job. STOP ends a run that is still going.
+BACK returns to where you came from. RETRY runs a settled trial again as a new one-trial job. STOP ends a run that is still going; its creator can stop it.
 
 DOWNLOAD on a trial offers the parsed trace, the ATIF trajectory, the raw stdout, the raw stderr, the agent home folder, the verifier log, or everything at once; on an analysis or a check it offers the verdict document, the raw stdout, the raw stderr, the agent home folder, or everything.
 

package/skills/evolve-evals/references/sdk-reference/analyses.mdx

@@ -79,6 +79,33 @@ description: "Read trace-analysis runs."
   </Tab>
 </Tabs>
 
+## defaults
+
+<Tabs>
+  <Tab title="TypeScript">
+    ```ts
+    defaults(): Promise<AnalyzeDefaults>
+    ```
+
+    The defaults an analysis runs under when you name none: `model_name`, `reasoning_effort`, `sandbox_provider`, the platform's `rubric`, and `prompt`, the built-in analyze prompt as a template with `{trial_path}`, `{task_section}` and `{criteria_guidance}` unrendered. Edit the rubric or the prompt from here and pass it to `jobs().analyze()`; passed back unchanged, they run exactly as the defaults do, and the analysis records them as named, so a later change of the platform default does not move it.
+
+    ```ts
+    const { rubric } = await analyses().defaults();
+    ```
+  </Tab>
+  <Tab title="Python">
+    ```python
+    async def defaults() -> AnalyzeDefaults
+    ```
+
+    The defaults an analysis runs under when you name none: `model_name`, `reasoning_effort`, `sandbox_provider`, the platform's `rubric`, and `prompt`, the built-in analyze prompt as a template with `{trial_path}`, `{task_section}` and `{criteria_guidance}` unrendered. Edit the rubric or the prompt from here and pass it to `jobs().analyze()`; passed back unchanged, they run exactly as the defaults do, and the analysis records them as named, so a later change of the platform default does not move it.
+
+    ```python
+    rubric = (await analyses().defaults())["rubric"]
+    ```
+  </Tab>
+</Tabs>
+
 ## download
 
 <Tabs>

package/skills/evolve-evals/references/sdk-reference/jobs.mdx

@@ -98,15 +98,20 @@ description: "Start, follow, and derive jobs."
 <Tabs>
   <Tab title="TypeScript">
     ```ts
-    list(options?: { search?: string; scope?: "my" | "shared" | "org"; limit?: number; cursor?: string }): JobList
+    list(options?: { search?: string; scope?: "my" | "shared" | "org"; kind?: "job"; limit?: number; cursor?: string }): JobList
+    list(options: { kind: "check"; search?: string; scope?: "my" | "shared" | "org"; limit?: number; cursor?: string }): CheckRowList
+    list(options: { kind: "all"; search?: string; scope?: "my" | "shared" | "org"; limit?: number; cursor?: string }): JobListItemList
     ```
 
-    Your jobs, newest first. Await one page, or iterate every page.
+    Your jobs, newest first. Await one page, or iterate every page. A task quality check is a job too: `kind: "check"` lists checks alone (each row a `CheckRow`), `kind: "all"` lists jobs and checks together (each row a `JobListItem`, told apart by its `kind`); without `kind` every row is a `Job`.
 
     ```ts
     for await (const job of jobs().list({ search: "nightly" })) {
       console.log(job.id, job.status);
     }
+    for await (const row of jobs().list({ kind: "all" })) {
+      console.log(row.kind, row.id, row.status);
+    }
     ```
   </Tab>
   <Tab title="Python">
@@ -115,15 +120,18 @@ description: "Start, follow, and derive jobs."
         *,
         search: Optional[str] = None,
         scope: Optional[JobListScope] = None,
+        kind: Optional[JobListKind] = None,
         limit: Optional[int] = None,
         cursor: Optional[str] = None,
     )
     ```
 
-    Your jobs, newest first. Await one page, or iterate every page.
+    Your jobs, newest first. Await one page, or iterate every page. A task quality check is a job too: `kind='check'` lists checks alone (each row a `CheckRow`), `kind='all'` lists jobs and checks together (each row a `Job` or a `CheckRow`, told apart by its `kind`); without `kind` every row is a `Job`. Both are dataclasses, so every row reads the same way.
 
     ```python
     async for job in jobs().list(search="nightly"): print(job.id, job.status)
+    async for row in jobs().list(kind="all"):
+        print(row.kind, row.id, row.status)
     ```
   </Tab>
 </Tabs>
@@ -406,7 +414,7 @@ description: "Start, follow, and derive jobs."
     ): Promise<Job>
     ```
 
-    Enqueue one trace analysis per trial and return at once with the job, its `stats.analysis.n_pending` counting the batch. `AnalyzeConfigInput`: `model_name`, `rubric`, `prompt`, `sandbox_provider`, `reasoning_effort`, `n_concurrent`, `passing`, `failing`, `n_trials`, all optional; `{}` is every default. Calling it again is the re-analysis path, one wave at a time. The same object on `start()` as `analyze` arms the embedded trigger.
+    Enqueue one trace analysis per trial and return at once with the job, its `stats.analysis.n_pending` counting the batch. `AnalyzeConfigInput`: `model_name`, `rubric`, `prompt`, `sandbox_provider`, `reasoning_effort`, `n_concurrent`, `passing`, `failing`, `n_trials`, `trial_ids`, all optional; `{}` is every default. `trial_ids` names the trials to analyze; it combines with `passing` or `failing` and comes before `n_trials`, and an id that is not this job's is refused with `invalid_input` naming the unknown ids. Calling it again is the re-analysis path, one wave at a time. The same object on `start()` as `analyze` arms the embedded trigger, where `trial_ids` is refused because the trials do not exist yet.
 
     ```ts
     await jobs().analyze(
@@ -416,6 +424,7 @@ description: "Start, follow, and derive jobs."
         n_trials: 20,
       },
     );
+    await jobs().analyze(job.id, { trial_ids: [trial.id] });
     ```
   </Tab>
   <Tab title="Python">
@@ -432,10 +441,11 @@ description: "Start, follow, and derive jobs."
         passing=None,
         failing=None,
         n_trials=None,
+        trial_ids=None,
     ) -> Job
     ```
 
-    Enqueue one trace analysis per trial and return at once.
+    Enqueue one trace analysis per trial and return at once. `trial_ids` names the trials to analyze; an id that is not this job's is refused with `invalid_input` naming the unknown ids.
 
     ```python
     await jobs().analyze(
@@ -443,6 +453,7 @@ description: "Start, follow, and derive jobs."
         failing=True,
         n_trials=20,
     )
+    await jobs().analyze(job.id, trial_ids=[trial.id])
     ```
   </Tab>
 </Tabs>

package/skills/evolve-evals/references/sdk-reference/types.mdx

@@ -39,6 +39,7 @@ One shape from `start`, `get`, `list`, `cancel`, `resume`, `retry` and `regrade`
 | `failure` | `{ code, message } \| null` | Why the job FAILED. Null on every job today; `FAILED` is reserved. |
 | `source_jobs` | `{ action, type, job_id }[]` | `action` is `regrade`, `resume` or `retry`; `type` is always `hub`. Empty on an original job. |
 | `is_regrade` | boolean | Any `source_jobs` entry with `action: "regrade"`. |
+| `viewer` | `creator` `member` `shared` `link` \| null | How you can read this job: you made it, you belong to its organization, it was shared with your address, or you hold its link. Null on the body an acting call returns (create, analyze, cancel, resume, retry, regrade) and on list rows; `get` tells you. Analyze, cancel, resume, retry and regrade are open to `creator` and `member` only. |
 | `upload` | `UploadProvenance \| null` | Null on every job this platform ran. See [upload](/core-concepts/upload). |
 | `idempotent_replay` | boolean | True only on a response that replayed an existing job for an idempotency key. |
 | `started_at`, `updated_at` | string | |
@@ -62,7 +63,7 @@ One shape from `start`, `get`, `list`, `cancel`, `resume`, `retry` and `regrade`
 
 ### AnalyzeConfig
 
-`model_name`, `rubric` (`{ criteria: [{ name, description, guidance }] }`), `prompt` (string or null for the default body), `reasoning_effort`, `sandbox_provider`, `n_concurrent` (number or null for the organization's ceiling alone), `passing`, `failing` (both false means every analyzable trial), `n_trials` (number or null for no cap). The input shape takes every field as optional.
+`model_name`, `rubric` (`{ criteria: [{ name, description, guidance }] }`), `prompt` (string or null for the default body), `reasoning_effort`, `sandbox_provider`, `n_concurrent` (number or null for the organization's ceiling alone), `passing`, `failing` (both false means every analyzable trial), `n_trials` (number or null for no cap), `trial_ids` (always null on the job body today: the stored policy is the create-time one, which refuses the field; the trials a manual wave names are the analyses it enqueues). The input shape takes every field as optional.
 
 ### JobStats