@agentionai/fieldwork-cli 0.4.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +42 -20
- package/dist/changelog.js +113 -0
- package/dist/client.js +96 -7
- package/dist/credentials.js +78 -0
- package/dist/fieldwork-skill.md +107 -21
- package/dist/main.js +521 -85
- package/dist/select.js +124 -0
- package/dist/version.js +50 -0
- package/dist/workspace.js +72 -21
- package/package.json +28 -7
package/dist/fieldwork-skill.md
CHANGED
|
@@ -5,20 +5,21 @@ description: Use the Fieldwork CLI to manage products, campaign goals, experimen
|
|
|
5
5
|
|
|
6
6
|
# Fieldwork research bookkeeping
|
|
7
7
|
|
|
8
|
-
Use this skill when a user asks you to organize or report long-running research in Fieldwork. Fieldwork stores intent, hypotheses, configuration, and execution state. It does NOT launch, schedule, monitor, or stop processes. Run actual work with separately authorized tools; only record observed state in Fieldwork.
|
|
8
|
+
Use this skill when a user asks you to organize or report long-running research in Fieldwork. Fieldwork stores intent, hypotheses, configuration, and execution state. It does NOT launch, schedule, monitor, or stop processes. Run actual work with separately authorized tools; only record observed state in Fieldwork. It is a ledger of experiments, not a log or metrics store: see "What to record" below.
|
|
9
9
|
|
|
10
10
|
## Prerequisites and invocation
|
|
11
11
|
|
|
12
|
-
You need Node.js 22+, the `@agentionai/fieldwork-cli` package installed on PATH, and access to a running compatible Fieldwork API server. The CLI package contains no server or web app and does not start either.
|
|
12
|
+
You need Node.js 22+, the `@agentionai/fieldwork-cli` package installed on PATH, and access to a running compatible Fieldwork API server. The CLI package contains no server or web app and does not start either. Install it with `npm install --global @agentionai/fieldwork-cli@0.7.0`, or a local archive with `npm install --global ./agentionai-fieldwork-cli-0.7.0.tgz`. Installing from npm does not require pnpm or a checkout.
|
|
13
13
|
|
|
14
14
|
```sh
|
|
15
15
|
fieldwork --help
|
|
16
|
-
fieldwork
|
|
16
|
+
fieldwork auth status
|
|
17
|
+
fieldwork products list
|
|
17
18
|
```
|
|
18
19
|
|
|
19
20
|
For source development only: install with `pnpm install`, build with `pnpm --filter @agentionai/fieldwork-cli build`, and substitute `node apps/cli/dist/main.js` for `fieldwork` from the checkout root. Outside the checkout use its actual path, never an invented path. Package-manager wrappers may add output.
|
|
20
21
|
|
|
21
|
-
Server precedence is `--url`, `FIELDWORK_URL`, legacy `LAB_URL`, nearest workspace config, then `http://127.0.0.1:4310`. Supply an HTTP(S) origin, NOT a URL ending in `/api/v1` or `/products/ID`; credentials, query strings and fragments are not accepted. Workspace/tenant is configured on the server; product scoping is not authentication. Confirm the intended server and product before writing. Do not start a second server or change its database to work around connection errors.
|
|
22
|
+
Server precedence is `--url`, `FIELDWORK_URL`, legacy `LAB_URL`, nearest workspace config, then the hosted service `https://app.fieldworkledger.com`. A local server must be named explicitly, for example `--url http://127.0.0.1:4310`. The hosted service requires a credential: an agent credential issued from the web app's Account page, stored once with `fieldwork auth login --token TOKEN` (per server; `auth status` shows which credential is in use). Supply an HTTP(S) origin, NOT a URL ending in `/api/v1` or `/products/ID`; credentials, query strings and fragments are not accepted. Workspace/tenant is configured on the server; product scoping is not authentication. Confirm the intended server and product before writing. Do not start a second server or change its database to work around connection errors.
|
|
22
23
|
|
|
23
24
|
## Hierarchy and scope
|
|
24
25
|
|
|
@@ -33,6 +34,34 @@ The intended CLI interface is stub-first: readable references such as `model-a`,
|
|
|
33
34
|
|
|
34
35
|
Current compatibility: the CLI resolves stubs for campaigns, products, experiments, and runs within their parent scope; UUIDs remain accepted everywhere. Discover records with list commands when the stub is unknown. Do not pass display names as stubs. Discover existing records before creating duplicates. There is no automatic idempotency key; do not blindly retry creates after ambiguous network failures.
|
|
35
36
|
|
|
37
|
+
## What to record: a ledger, not a log
|
|
38
|
+
|
|
39
|
+
A run records what was tried, under what conditions, and the few numbers that decide a comparison -- so that months later someone can still tell what was measured and whether two results are comparable. It is not where logs, traces, per-step metrics, raw model outputs or datasets go.
|
|
40
|
+
|
|
41
|
+
- Record: the parameters that varied, the comparison context that must match for results to compare (hardware, build or version, git commit, dataset and its version, harness, time or budget caps), and summary observations (a mean, a p95, a score, a size) with their sample count where it matters.
|
|
42
|
+
- Link, do not paste: put a log or output location in `logsUri`, and files that define the work (recipes, datasets, heads) in artifacts. A path is a reference, not a copy.
|
|
43
|
+
- Aggregate before recording: one run per configuration measured, with its summary numbers -- not one run per request, step or sample.
|
|
44
|
+
- Forgot something? A parameter or context field a run left empty can be filled in later with `fieldwork runs backfill`, visibly and with provenance; do not re-record the run. Measurements taken later are a new run.
|
|
45
|
+
|
|
46
|
+
Work that is finished or no longer relevant is archived, not deleted: `fieldwork campaigns archive REF` (or `products`, `experiments`) takes it out of lists and closes it to new work, and `restore` brings it back; nothing is lost. Deleting (`delete ... --confirm NAME`) removes a unit and everything in it for good -- do it only when the user asks for that, and never to get around a limit or an archived refusal (`ARCHIVED`: restore it, or record elsewhere). A finished run is never deleted: mark it superseded or its experiment abandoned.
|
|
47
|
+
|
|
48
|
+
On the hosted service an organization's plan may limit how many products, active campaigns, active experiments per campaign, agent credentials and members it has. A refusal is `PLAN_LIMIT`, with a hint: tell the user, and suggest archiving finished work (archived work does not count) rather than deleting anything or creating work elsewhere to get around it.
|
|
49
|
+
|
|
50
|
+
Keep the CLI current. It names its version to the server on every request; when a newer release exists, a successful command prints one `{"notice":"UPDATE_AVAILABLE",...}` line on stderr (at most once a day), and a failed command's error carries an `update` object. When it says `"required": true`, update before retrying: the failure may be the version, and retrying will not change that. Install the version it names with `npm install --global @agentionai/fieldwork-cli@<latest>`. Stdout is never affected.
|
|
51
|
+
|
|
52
|
+
The hosted service enforces limits sized for that, and each refusal carries a `hint` saying what to do instead. Read it, and change what you record rather than retrying:
|
|
53
|
+
|
|
54
|
+
| Limit | Value | Refusal |
|
|
55
|
+
| ------------------------------------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
56
|
+
| One run's or experiment's own data (parameters, observations, context, environment, input refs, extras) | 64 KB of JSON | `RECORD_TOO_LARGE` (413) |
|
|
57
|
+
| Observations on one record | 200 | `TOO_MANY_OBSERVATIONS` (413) |
|
|
58
|
+
| Any request body | 256 KB | `PAYLOAD_TOO_LARGE` (413) |
|
|
59
|
+
| Runs in one experiment | 5,000 | `EXPERIMENT_FULL` (409): if you are looping, stop; otherwise split the study into experiments by what varies |
|
|
60
|
+
| Runs in one shared snapshot | 1,000 | `SHARE_TOO_LARGE` (413) |
|
|
61
|
+
| Writes by one caller (member or agent credential) | 120 a minute | `RATE_LIMITED` (429): wait; reads are never limited |
|
|
62
|
+
|
|
63
|
+
Reaching a limit in ordinary use usually means the record is carrying something that belongs elsewhere. Organizations with a genuine need for more may be offered higher limits (enterprise accounts are planned); until then, do not work around a limit by splitting one record's data across several.
|
|
64
|
+
|
|
36
65
|
## Local campaign workspace (available via `setup campaign`)
|
|
37
66
|
|
|
38
67
|
Attach a directory to a campaign so commands work relative to its product/campaign context without repeating IDs:
|
|
@@ -61,7 +90,7 @@ Use setup to generate config; do not infer bindings from directory names. Run cr
|
|
|
61
90
|
|
|
62
91
|
## Typed experiment schemas
|
|
63
92
|
|
|
64
|
-
Comparable experiments need declared shapes. A schema version defines `parameters`, `observations`, and `comparisonContext` fields with type, unit, direction, bounds, allowed values, and `compare` flags. Publish on a product for reuse, a campaign for study-specific fields (including product-linked campaigns), or an existing experiment for specialized fields. A scope may use its own schemas and its ancestors’ schemas, never a sibling’s. Schemas are complete definitions, not silently merged overlays.
|
|
93
|
+
Comparable experiments need declared shapes. A schema version defines `parameters`, `observations`, and `comparisonContext` fields with type (`number`, `integer`, `string`, `boolean`, `enum`, `ref`), unit, direction, bounds, allowed values, `refKind`, and `compare` flags. Publish on a product for reuse, a campaign for study-specific fields (including product-linked campaigns), or an existing experiment for specialized fields. A scope may use its own schemas and its ancestors’ schemas, never a sibling’s. Schemas are complete definitions, not silently merged overlays.
|
|
65
94
|
|
|
66
95
|
```sh
|
|
67
96
|
fieldwork schemas publish --product model-a --json '{"stub":"quant-study","version":1,"definition":{"parameters":{"bits":{"type":"enum","values":[4,8],"required":true},"group_size":{"type":"integer","minimum":1}},"observations":{"memory_gib":{"type":"number","unit":"GiB","direction":"minimize","required":true}},"comparisonContext":{"hardware":{"type":"string","compare":true}}}}'
|
|
@@ -74,23 +103,23 @@ Workflow: publish or reuse a version, pin it on the experiment (`schemaVersionId
|
|
|
74
103
|
|
|
75
104
|
Validation is strict about meaning, flexible about completeness: unknown fields are rejected (move exploratory data to `extras`), required fixed parameters block experiment readiness; required varying parameters may be deferred to runs, where all required execution fields must resolve before start, missing observations are allowed on incomplete runs, and failed runs may lack measurements. No string coercion or unit conversion happens. Research-value validation rejections include per-path `code`, `expected`, `receivedType`, `message`, and an actionable `hint`; the CLI prints the server's issue list on stderr. Check payloads with `schemas validate` before writing records.
|
|
76
105
|
|
|
77
|
-
|
|
78
106
|
## CLI reference
|
|
79
107
|
|
|
80
108
|
REF means a stub or UUID. Experiment/run stubs require `--campaign` or a local workspace. JSON fields ending in `Id` still require UUIDs; use `--experiment` for a readable reference. Campaign list/create do not inherit product scope: specify `--product` when needed. Other commands inherit the server from local config.
|
|
81
109
|
|
|
82
110
|
Commands below use `fieldwork` as the executable. `--help` is available on every command.
|
|
83
111
|
|
|
84
|
-
| Group
|
|
85
|
-
|
|
|
86
|
-
| products
|
|
87
|
-
| campaigns
|
|
88
|
-
| experiments | list; get REF; context REF; create --json JSON; update REF --json JSON; delete REF --revision N; all accept [--campaign REF]
|
|
89
|
-
| runs
|
|
90
|
-
| setup
|
|
91
|
-
| context
|
|
92
|
-
|
|
|
93
|
-
|
|
|
112
|
+
| Group | Commands and required options |
|
|
113
|
+
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
114
|
+
| products | list; get REF; create --json JSON; update REF --json JSON; delete REF |
|
|
115
|
+
| campaigns | list [--product REF]; get REF; context REF; create [--product REF] --json JSON; update REF --json JSON; delete REF --revision N |
|
|
116
|
+
| experiments | list [--where EXPR] [--fields PATHS] [--format json\|tsv] [--include-superseded] [--include-abandoned]; get REF; context REF; create --json JSON; update REF --json JSON; delete REF --revision N; all accept [--campaign REF] |
|
|
117
|
+
| runs | list [--experiment REF] [--where EXPR] [--fields PATHS] [--format json\|tsv] [--include-superseded] [--include-abandoned]; get REF; context REF; create [--experiment REF] --json JSON; record REF --json JSON; update REF --json JSON; delete REF --revision N; all accept [--campaign REF] |
|
|
118
|
+
| setup | campaign --campaign REF [--product REF] [--create --goal TEXT] |
|
|
119
|
+
| context | [--campaign REF] |
|
|
120
|
+
| changelog | [--since VERSION] [--release VERSION]; offline, never contacts the API |
|
|
121
|
+
| schemas | list [--inherited]; publish --json JSON; default; set-default --json JSON; these accept product/campaign/experiment scope; get VERSION_ID; template VERSION_ID; validate VERSION_ID [--ready] --json JSON |
|
|
122
|
+
| charts | fields; list; create --json JSON; these accept [--campaign REF] [--experiment REF]; get CHART_ID; data CHART_ID; series CHART_ID --json JSON; frontier CHART_ID --json JSON; delete CHART_ID |
|
|
94
123
|
|
|
95
124
|
`--json -` reads a JSON object from stdin. Use it for multiline text and configuration files rather than constructing shell strings from untrusted text. Successful data commands emit JSON on stdout. Failures emit JSON on stderr and exit nonzero. Help/version are human-readable. Direct invocation avoids package-manager output mixed into machine-readable streams.
|
|
96
125
|
|
|
@@ -140,8 +169,9 @@ fieldwork experiments update EXPERIMENT_ID --json '{"revision":1,"status":"compl
|
|
|
140
169
|
|
|
141
170
|
- `products get`: product object.
|
|
142
171
|
- `campaigns get` / `context`: campaign fields at the top level, plus `context.product`, `context.generatedAt`, `experiments`, and `runs`.
|
|
143
|
-
- `experiments get` / `context`: `{ entity, context, runs }` where runs
|
|
144
|
-
- `runs get` / `context`: `{ entity, context, runs: [] }`.
|
|
172
|
+
- `experiments get` / `context`: `{ entity, context, runs }` where `runs` is the first page of the experiment's children, `{ items, nextCursor }`. Use `runs list --experiment REF` for all of them.
|
|
173
|
+
- `runs get` / `context`: `{ entity, context, runs: { items: [], nextCursor: null } }`.
|
|
174
|
+
- `campaigns get` / `context`: `experiments` and `runs` are likewise first pages, `{ items, nextCursor }`.
|
|
145
175
|
- Experiment/run live `context` contains `product`, `campaign`, `experiment`, and `generatedAt`; parents include revisions. An independent campaign has `product: null`.
|
|
146
176
|
- A run's `entity.contextSnapshot` captures context at creation. Legacy runs can have `null`; never reconstruct a historical snapshot and present it as known truth.
|
|
147
177
|
- Run config initially shallow-merges experiment parameters with explicit config overrides. Nested objects are replaced at the top level, not deep-merged. Later parent edits do not update run config or snapshots. A config update replaces the entire config object; send the complete intended value.
|
|
@@ -183,7 +213,7 @@ fieldwork schemas set-default --campaign memory-study --json '{"schemaVersionId"
|
|
|
183
213
|
|
|
184
214
|
Replace IDs and revisions with returned values. Product/campaign defaults have their own revision counter (initially 0), separate from the entity revision. `schemas default` reports the local selection, revision, effective schema ID, and source scope. Setting `schemaVersionId: null` clears the local product/campaign selection and restores ancestor fallback; it does not disable typing when an ancestor has a default. Stale writes return 409. Published versions and source data are never edited by this operation.
|
|
185
215
|
|
|
186
|
-
A new experiment pins its explicit `schemaVersionId`, otherwise the campaign default, otherwise the product default. New runs inherit their experiment’s pinned schema; direct runs (or runs of legacy unpinned experiments) resolve the nearest default if no explicit version is supplied. Existing experiments, runs and charts are not repinned when defaults change. An experiment’s pinned version is its default for runs: use `schemas set-default --experiment REF --json '{"schemaVersionId":"RETURNED_SCHEMA_UUID","revision":CURRENT_EXPERIMENT_REVISION}'` (equivalent to `experiments update`)
|
|
216
|
+
A new experiment pins its explicit `schemaVersionId`, otherwise the campaign default, otherwise the product default. New runs inherit their experiment’s pinned schema; direct runs (or runs of legacy unpinned experiments) resolve the nearest default if no explicit version is supplied. Existing experiments, runs and charts are not repinned when defaults change. A legacy experiment with existing runs can be pinned in place when every existing run validates against the proposed schema; the experiment and compatible run pins advance atomically. Breaking proposals are rejected with validation issues. An experiment’s pinned version is its default for runs: use `schemas set-default --experiment REF --json '{"schemaVersionId":"RETURNED_SCHEMA_UUID","revision":CURRENT_EXPERIMENT_REVISION}'` (equivalent to `experiments update`). Experiment pins cannot be cleared with null. To define an experiment-local schema: create the experiment, publish on it, then explicitly pin that version.
|
|
187
217
|
|
|
188
218
|
API: `GET/POST /api/v1/{products|campaigns|experiments}/:id/schemas`; `GET .../schemas?inherited=true`; `GET .../schema-default`; `PUT /api/v1/{products|campaigns}/:id/schema-default` with `{schemaVersionId,revision}`. Experiment pin updates use `PATCH /api/v1/experiments/:id`. Campaign chart field discovery includes schemas owned by descendant experiments, but campaign charts still select exactly one immutable version. Experiment charts cannot use a sibling experiment’s schema. No automatic cross-version merging or unit conversion is performed.
|
|
189
219
|
|
|
@@ -206,11 +236,15 @@ Every point is one successful run with the exact pinned schema version. Missing
|
|
|
206
236
|
|
|
207
237
|
Chart data is live, not an immutable evidence snapshot. `charts data` returns points with source run IDs/revisions, experiment IDs, context-series labels, axis labels/units, exclusions, and `generatedAt`. Use these for provenance; no chart implies statistical significance or a proven hypothesis. The UI refresh button reloads definitions and observations. More than 500 eligible points are available in the data table/API but not drawn.
|
|
208
238
|
|
|
239
|
+
A scatter chart without aggregation can carry a frontier: the runs to join with a dashed line, drawn in X order. It is chosen, never computed -- which runs are comparable, and which trade-off is worth drawing, is a judgement. Set it with `fieldwork charts frontier CHART_ID --json '{"runIds":["RUN_ID",...]}'`; the list replaces any earlier choice, `[]` clears it, and only runs the chart currently plots are accepted (`INVALID_FRONTIER` otherwise). Do not put runs from different comparison contexts on one frontier unless the campaign says they are comparable.
|
|
240
|
+
|
|
241
|
+
By default a raw chart has one series (colour) per comparison context. `fieldwork charts series CHART_ID --json '{"series":{"by":"field","section":"parameters","field":"recipe_family"}}'` colours it by a typed parameter or context field instead; `{"series":{"by":"groups","groups":[{"label":"Ours","runIds":[...]}],"otherLabel":"Shipped"}}` defines the series as groups of runs, for a distinction no single field records. `{"series":null}` restores the default; `charts create` accepts the same `series`. Each point keeps its comparison context in `context`, so colour never hides a comparability difference. Aggregated charts cannot take a series; `INVALID_SERIES` names what was wrong.
|
|
242
|
+
|
|
209
243
|
API equivalents: `GET /api/v1/{campaigns|experiments}/:id/charts/fields`, `GET/POST /api/v1/{campaigns|experiments}/:id/charts`, `GET/DELETE /api/v1/charts/:id`, `GET /api/v1/charts/:id/data`. CLI and API are available; the MCP adapter remains a scaffold.
|
|
210
244
|
|
|
211
245
|
## Fieldwork identity and compatibility
|
|
212
246
|
|
|
213
|
-
Product: Agention Fieldwork. npm package: `@agentionai/fieldwork-cli`. Executable: `fieldwork`.
|
|
247
|
+
Product: Agention Fieldwork. npm package: `@agentionai/fieldwork-cli`. Executable: `fieldwork`. It is published on npm; repository `install.sh --help` also describes local tarball installation. No service is installed or started.
|
|
214
248
|
|
|
215
249
|
New setup uses `.fieldwork/workspace.json` and `fieldwork-skill.md`. Existing `.lab/workspace.json` bindings and `LAB_URL` remain supported; `FIELDWORK_URL` takes precedence over the legacy variable. Existing files are never renamed automatically. The server database location is unchanged.
|
|
216
250
|
|
|
@@ -218,7 +252,7 @@ New setup uses `.fieldwork/workspace.json` and `fieldwork-skill.md`. Existing `.
|
|
|
218
252
|
|
|
219
253
|
Always capture exit status, stdout, and stderr. A nonzero exit with empty stdout means failure, not `None` or a successful empty record; preserve the structured stderr error. Parse stdout as a success value only after checking exit status. `schemas validate` deliberately also returns the invalid report on stdout when it exits 1. Do not retry a create blindly after an ambiguous transport failure: inspect the intended server/campaign first.
|
|
220
254
|
|
|
221
|
-
|
|
255
|
+
The CLI's experiment/run lists are arrays of record objects regardless of workspace or explicit campaign scope; the CLI walks the API's pages (`{ items, nextCursor }`) for you. Their get/context results wrap the record in `entity`; create/update return the record directly. Do not guess shapes or treat empty stdout as `[]`. If an external wrapper returns strings instead, retain the raw command, version, status, and both streams to diagnose the difference.
|
|
222
256
|
|
|
223
257
|
Experiment `method`, `hypothesis`, `objective`, and `conclusion` have a 4000-character limit; name has a 120-character limit. For longer methods, reference a versioned file and include the exact extraction command and source field in `method`. `comparisonContext`, parameters/config, observations, and extras must be objects where supported, not JSON strings.
|
|
224
258
|
|
|
@@ -240,6 +274,8 @@ Extension accepts new optional observations/parameters, enum expansion, relaxed
|
|
|
240
274
|
|
|
241
275
|
This is the exception to ordinary pin immutability: a new immutable successor is created, with `extendedFrom` and `extensionImpact`, and all matching experiment/run/default/chart pins advance atomically. Old schema definitions, recorded values, execution dates, and creation snapshots remain intact. Revisions increment: reload affected records before writing. Existing terminal runs may then receive optional observations via normal revision-checked updates. No run recreation or fabricated lifecycle is necessary. Dry run writes nothing; apply revalidates and is not reserved by the preview. Stale-source errors require inspection, not a blind retry. Independently published versions are not auto-merged, and breaking re-pinning is still blocked.
|
|
242
276
|
|
|
277
|
+
Widening an enum (adding values) is an ordinary compatible extension. A parameter or comparison-context field added to the schema after runs were recorded can be filled in on those runs, finished ones included, in any later extension: `"backfill":{"packager":{"RUN_ID":"unsloth"}}` (or `"parameters.packager"` / `"comparisonContext.driver"` when a bare name is ambiguous). Backfill accepts runs re-pinned by this extension that left the field empty -- whether it is new or was forgotten -- and never replaces a recorded value; values are validated against the new definition, applied in the same atomic write, and listed under `impact.backfilled` (dry runs included). Each run notes them in `research.backfilled` with the supplying schema version, time and author. A backfilled value describes a run rather than records how it ran, so it stays correctable with a revision-checked `runs update`; executed configuration and context stay frozen. Do not re-record finished runs just to add a field. When the field already exists in the run's schema and was simply not logged -- a git commit, a build -- fill it in directly: `fieldwork runs backfill RUN --json '{"comparisonContext":{"git_commit":"a1b2c3d"}}'`.
|
|
278
|
+
|
|
243
279
|
Recipe/artifact commands are available in CLI 0.4.0 with the updated API. Use the explicit artifacts reference array; a config stub alone is not an enforced recipe link.
|
|
244
280
|
|
|
245
281
|
## Recipes and artifacts (updated server/client required)
|
|
@@ -255,3 +291,53 @@ Do not over-claim: a stored URI/hash records provenance but does not attest that
|
|
|
255
291
|
Chart creation now accepts `aggregation: {groupBy:[{section:"parameters",field:"variant"}],metric:"mean",spread:"band"}`. Supply ordinary typed X/Y axes as before. `metric` supports mean or sample stdev; `spread` supports none, mean±SD whiskers, or mean±SD shaded bands for line charts. Saved grouping/metric comes from chart creation; the web spread selector only changes that view. Discover fields first. No free-form math expressions run.
|
|
256
292
|
|
|
257
293
|
Aggregation buckets preserve X plus existing context boundaries (experiment/context/input/environment) and add the requested grouping fields. For a category mean choose the category as bar X; for a line choose a numeric X and a variant grouping field. Replicate runs get equal weight. Missing group fields are excluded explicitly. Source points remain in the API and table; `aggregates` includes count, mean, sample SD, bounds and contributing IDs/revisions. SD uses n−1, is null for n<2, and is never imputed as zero. Bands are ±1 SD, not confidence intervals or proof of a meaningful effect; singleton bands are absent. Include relevant non-varying parameters in grouping, inspect provenance, and do not confuse identical metadata with experimental equivalence. CLI 0.4.0 includes this reference; aggregation requires the separately updated API and web app.
|
|
294
|
+
|
|
295
|
+
## Selecting and projecting records (updated client required)
|
|
296
|
+
|
|
297
|
+
`experiments list` and `runs list` select and project rows; they never compute new values. Compute in Python, jq or your own code from the JSON.
|
|
298
|
+
|
|
299
|
+
```sh
|
|
300
|
+
fieldwork runs list --experiment output-verbosity --where 'model=qwopus' --fields parameters.n,observations.accuracy --format tsv
|
|
301
|
+
fieldwork runs list --where 'parameters.n>=8' --where 'eval_set=gsm8k'
|
|
302
|
+
fieldwork experiments list --fields stub,status --format tsv
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
`--where` takes `FIELD=VALUE` or `!=`, `>=`, `<=`, `>`, `<`; repeat it for AND. Paths are `section.field` (`parameters`, `observations`, `comparisonContext`, `extras`, `inputRefs`, `environment`) or a record field such as `status`; a bare name is searched across the value sections and an ambiguous one is rejected rather than guessed. A run's parameters are its `config`. Comparisons never coerce across types: `n=4` matches the number 4 and the string "4", but not "four". An empty operand means "no recorded value", so `--where 'extras.superseded_by='` keeps records without one. `>`/`<` compare numbers numerically and everything else as text, so ISO dates order correctly. A path that no returned record has is an error, not a blank column.
|
|
306
|
+
|
|
307
|
+
`--fields` prints only those paths, `--format tsv` prints a header row and tab-separated values with tabs and newlines escaped. JSON output stays an array of objects, projected or whole.
|
|
308
|
+
|
|
309
|
+
**Records superseded by a replacement (`extras.superseded_by`) and work under abandoned experiments are left out by default.** Add `--include-superseded` and `--include-abandoned` to see them. This is a correctness default: those records are not live evidence and averaging them in produces a wrong answer. Say which default applied when you report counts.
|
|
310
|
+
|
|
311
|
+
## Recording an outcome in one call (updated server/client required)
|
|
312
|
+
|
|
313
|
+
`runs create` accepts `status`, `stub`, `startedAt`, `finishedAt` and `errorSummary` alongside observations, so a finished or historical run is one call rather than a create plus two patches. Supplied dates are validated for order and are never invented for you.
|
|
314
|
+
|
|
315
|
+
`fieldwork runs record REF --json '{"status":"succeeded","observations":{"accuracy":0.9}}'` records an outcome on an existing run. The server reads the current revision itself, so there is no read-modify-write race; pass `revision` when you do want the optimistic check. A result may be recorded directly from `planned` — recording an outcome is not a lifecycle transition — but a run that already has a terminal status keeps it: create a new run for another attempt. `record` sets `finishedAt` only for a run this server saw `running`; it never fabricates a date for a historical outcome. Comparison context stays frozen after a run leaves planned.
|
|
316
|
+
|
|
317
|
+
Experiment and run responses now repeat `schemaVersionId`, `varying`, `observations`, `comparisonContext`, `extras` and `artifactIds` at the top level, mirroring what create and update accept. `research` remains the canonical location and holds the same values; an unset `schemaVersionId` echoes as `null`.
|
|
318
|
+
|
|
319
|
+
## Typed artifact references (updated server/client required)
|
|
320
|
+
|
|
321
|
+
A field of type `ref` records an artifact stub (or ID) and is validated against the product's artifact registry, with optional `refKind: recipe | file`. Use it wherever the set of values is an open, growing identity — a head recipe, a corpus, a build — instead of an enum that must be republished for every new member.
|
|
322
|
+
|
|
323
|
+
```sh
|
|
324
|
+
fieldwork artifacts create --product signal --json '{"stub":"head-v6a","name":"Signal v6a head","definition":{"corpus":"wiki-2026","longAnswerWeight":0.5},"files":[{"role":"head","uri":"file://heads/v6a.safetensors","sha256":"..."}]}'
|
|
325
|
+
fieldwork schemas publish --json '{"stub":"style-eval","version":5,"definition":{"parameters":{"recipe":{"type":"ref","refKind":"recipe","required":true}}}}'
|
|
326
|
+
fieldwork runs create --experiment heads --json '{"title":"v6a on gsm8k","status":"succeeded","config":{"recipe":"head-v6a"},"observations":{"accuracy":0.68}}'
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
An unregistered stub, a wrong-case stub and a free-text name are all rejected (`UNKNOWN_REF` or `INVALID_VALUE`), so refs keep the typo-safety an enum gave you without its closed value list. Recording a ref links the artifact into `research.artifactIds` and freezes it, so the measurement carries the exact recipe, corpus, flags and file hashes it was produced with. Ref fields work as a bar X axis and as a `groupBy` key exactly like enums. A new variant is then a new artifact, never a new schema version. Changing an existing field to or from `ref` is a breaking publication, not an extension.
|
|
330
|
+
|
|
331
|
+
## Charts across schema versions (updated server/client required)
|
|
332
|
+
|
|
333
|
+
A chart pins one immutable version for its axes, units, labels and required comparison context. By default it also admits runs pinned to **other versions of that same schema** when every field the chart reads still records the same kind of value in the same unit and direction. Allowed values, bounds, `refKind` and requiredness may differ: those constrain what a run could record, not what a recorded value means. `enum`, `ref` and `string` count as the same kind, as do `integer` and `number`.
|
|
334
|
+
|
|
335
|
+
Appending enum values, adding optional fields, or retiring an enum in favour of a `ref` therefore no longer strands the history that the new runs exist to be compared against. Pass `"schemaVersions":"pinned"` when creating a chart to demand exactly one version.
|
|
336
|
+
|
|
337
|
+
`charts data` returns `schemaVersions`: every candidate version with its `runs` count and either `included: true` or the `reason` it was refused. Report which versions a figure actually mixes; a unit or direction change is refused there and must stay refused. Schemas with a different stub or owner never join, and a run with no pinned schema is still excluded.
|
|
338
|
+
|
|
339
|
+
## Knowing what this CLI can do (updated client required)
|
|
340
|
+
|
|
341
|
+
`fieldwork changelog` lists recent releases of the installed CLI without contacting the API. Each change carries `requiresApi`: `false` means it works against any compatible server, `true` means it needs server behaviour from the corresponding baseline. Use it when a command is missing (`commander.unknownCommand` means the CLI is too old) or when a command exists but the server rejects it (the API is older than the client).
|
|
342
|
+
|
|
343
|
+
`--since VERSION` and `--release VERSION` narrow the output. The package bundles recent releases only; `oldestBundled` and `fullHistory` in the response say so, and `truncated: true` appears when `--since` reaches past the bundled window. Do not read an absent version as "nothing changed". The command reports what a change requires, never what a server actually provides: there is no compatibility handshake, and installing a newer client does not upgrade an API.
|