@toloka-platform/sdk 0.2.0-preview.3 → 0.2.0-preview.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/toloka.js +136 -136
- package/dist/cli/toloka.js.jsc +0 -0
- package/dist/cli/toloka.js.jsc.sha256 +1 -1
- package/dist/cli/toloka.js.sha256 +1 -1
- package/docs/cli-v2.md +3 -2
- package/docs/cli.md +32 -4
- package/openapi/gen/models/baseModel.ts +1 -0
- package/openapi/gen/models/currentUser.ts +2 -0
- package/openapi/gen/models/currentUserTeamsItem.ts +6 -0
- package/openapi/gen/models/gistingTrainingConfiguration.ts +6 -2
- package/openapi/gen/models/gistingTrainingConfigurationBatchSize.ts +2 -0
- package/openapi/gen/models/gistingTrainingConfigurationGistTokens.ts +2 -0
- package/openapi/gen/models/gistingTrainingConfigurationInput.ts +6 -2
- package/openapi/gen/models/gistingTrainingConfigurationInputBatchSize.ts +2 -0
- package/openapi/gen/models/gistingTrainingConfigurationInputGistTokens.ts +2 -0
- package/openapi/gen/models/index.ts +9 -4
- package/openapi/gen/models/sftTrainingConfiguration.ts +6 -3
- package/openapi/gen/models/sftTrainingConfigurationBatchSize.ts +2 -0
- package/openapi/gen/models/sftTrainingConfigurationInput.ts +6 -3
- package/openapi/gen/models/sftTrainingConfigurationInputBatchSize.ts +2 -0
- package/openapi/gen/models/sftTrainingConfigurationInputLoraRank.ts +2 -0
- package/openapi/gen/models/sftTrainingConfigurationLoraRank.ts +2 -0
- package/package.json +1 -1
- package/skills/catalog.json +10 -10
- package/skills/toloka-build/SKILL.md +28 -17
- package/skills/toloka-build/references/node-semantics.md +7 -7
- package/skills/toloka-build/references/workflow-and-cli.md +13 -10
- package/skills/toloka-fine-tuning/SKILL.md +145 -62
- package/skills/toloka-get-started/SKILL.md +35 -20
- package/skills/toloka-gisting/SKILL.md +156 -66
- package/src/cli/analytics.ts +50 -23
- package/src/cli/auth.ts +31 -12
- package/src/cli/auto-update.ts +12 -6
- package/src/cli/check.ts +76 -29
- package/src/cli/cli-entry.ts +3 -2
- package/src/cli/cli-error.ts +43 -0
- package/src/cli/commands/dataset.ts +13 -3
- package/src/cli/commands/models.ts +6 -7
- package/src/cli/commands/pipeline.ts +6 -7
- package/src/cli/commands/project.ts +9 -16
- package/src/cli/commands/report.ts +6 -2
- package/src/cli/commands/secret.ts +5 -4
- package/src/cli/commands/source.ts +4 -4
- package/src/cli/commands/status.ts +2 -3
- package/src/cli/commands/training.ts +50 -17
- package/src/cli/config.ts +2 -2
- package/src/cli/context.ts +55 -17
- package/src/cli/credentials.ts +12 -4
- package/src/cli/data.ts +96 -34
- package/src/cli/delivery-orders.ts +18 -1
- package/src/cli/file-system.ts +6 -2
- package/src/cli/format-error.ts +45 -2
- package/src/cli/init.ts +13 -7
- package/src/cli/models.ts +43 -12
- package/src/cli/program.ts +4 -55
- package/src/cli/project-context.ts +117 -32
- package/src/cli/project.ts +80 -28
- package/src/cli/report.ts +24 -9
- package/src/cli/runs.ts +33 -8
- package/src/cli/settings.ts +13 -5
- package/src/cli/skill.ts +61 -22
- package/src/cli/source.ts +3 -2
- package/src/cli/terminal-io.ts +13 -6
- package/src/cli/trainings.ts +129 -35
- package/src/cli/version.ts +2 -2
- package/src/client.ts +3 -9
- package/src/transport.ts +13 -1
- package/openapi/gen/models/gistingTrainingConfigurationInputRegime.ts +0 -2
- package/openapi/gen/models/gistingTrainingConfigurationRegime.ts +0 -2
- package/openapi/gen/models/sftTrainingConfigurationInputRegime.ts +0 -2
- package/openapi/gen/models/sftTrainingConfigurationRegime.ts +0 -2
package/dist/cli/toloka.js.jsc
CHANGED
|
Binary file
|
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
ee914e999937f98c40a657d10640ea64acda93b2e50f4a9306e25453c8897248 toloka.js.jsc
|
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
fabf11ce7545f788c03870778c18deda006d89c95c839ae8c1592f4eaa287366 toloka.js
|
package/docs/cli-v2.md
CHANGED
|
@@ -86,14 +86,14 @@ toloka project remove cat-labeling # local unbind only
|
|
|
86
86
|
|
|
87
87
|
`project add` with no arguments prompts for alias and id, like `wmill workspace add`. The new alias becomes `current`.
|
|
88
88
|
|
|
89
|
-
Create a live project and bind it in one step.
|
|
89
|
+
Create a live project and bind it in one step. The delivery order follows the owning team, not the account (`toloka whoami` marks internal teams `(internal)` on its `Teams:` line): a project in an internal team must carry an active id from `toloka delivery-order list`, and a project in an external team must not carry one, even when the account is internal:
|
|
90
90
|
|
|
91
91
|
```sh
|
|
92
92
|
toloka project create cat-labeling --team-id <teamId>
|
|
93
93
|
toloka project create cat-labeling --team-id <teamId> --delivery-order-id <orderId>
|
|
94
94
|
```
|
|
95
95
|
|
|
96
|
-
`POST /projects` requires `teamId`.
|
|
96
|
+
`POST /projects` requires `teamId`. `GET /me` lists the caller's teams with their write access, so `--team-id` is optional when the account has exactly one writable team; with several, the CLI refuses and names them.
|
|
97
97
|
|
|
98
98
|
`project create` writes the machine alias, sets `current`, and if cwd has no `toloka.json`, writes one.
|
|
99
99
|
|
|
@@ -126,6 +126,7 @@ Resource commands never search the disk for a pipeline first. They load the proj
|
|
|
126
126
|
| `plan` / `apply` / `validate` | whole project | no |
|
|
127
127
|
| `secret set` | project | no |
|
|
128
128
|
| `dataset list` | project | no |
|
|
129
|
+
| `dataset create <name>` | project | no |
|
|
129
130
|
| `dataset fields <name>` | named dataset in this project | no |
|
|
130
131
|
| `dataset load <name>` | named dataset in this project | no |
|
|
131
132
|
| `dataset upload <name>` | named dataset in this project | no |
|
package/docs/cli.md
CHANGED
|
@@ -49,6 +49,10 @@ The root program accepts these flags before any command:
|
|
|
49
49
|
|
|
50
50
|
Commands resolve the project root by walking up from the working directory until they find a `toloka.json` file. The CLI reports that directory on stderr when it differs from the working directory. Passing `--source-root` disables discovery. With no marker found, the working directory is used. `init` and `report` do not walk up: they use the working directory or `--source-root` only. The authentication commands `login`, `logout`, and `auth-status` do not need a project root.
|
|
51
51
|
|
|
52
|
+
## Errors and hints
|
|
53
|
+
|
|
54
|
+
A failing command prints its message to stderr and exits `1`. Most messages are followed by a `Hint: …` line with the next command to run; messages that already spell out the remedy carry no hint. Platform failures print `Potato API request failed with status <code> (request id: <id>)`, the response body, and a status-based hint: `401`/`403` point at `toloka whoami` and `toloka login`, `404` at the selected project and the matching list command, `429` asks you to wait, and `5xx` asks you to retry and then send a report with `toloka report <report.md>`.
|
|
55
|
+
|
|
52
56
|
## Resources and Base Actions
|
|
53
57
|
|
|
54
58
|
The CLI is resource-first: `toloka <resource> <action>`. A Base Action has one meaning wherever a resource implements it. A resource exposes only the Base Actions that have real behavior; there are no placeholder actions.
|
|
@@ -86,7 +90,7 @@ toloka project create [alias] [--name <name>] [--team-id <id>] [--delivery-order
|
|
|
86
90
|
toloka project untrack <pipeline|dataset> <name> [--confirm]
|
|
87
91
|
```
|
|
88
92
|
|
|
89
|
-
`add`, `switch`, `remove`, and `create` write the machine registry in the credentials file. They refuse `--api-key` / `--base-url` / `TOLOKA_*` when those would select a different key or platform than the stored file. `project list` prints project id, name, and team id. `create`
|
|
93
|
+
`add`, `switch`, `remove`, and `create` write the machine registry in the credentials file. They refuse `--api-key` / `--base-url` / `TOLOKA_*` when those would select a different key or platform than the stored file. `project list` prints project id, name, and team id. `create` uses the account's only writable team (`GET /me` lists the teams; `whoami` prints them) and asks for `--team-id` when there are several. A project owned by an internal team also needs `--delivery-order-id`; `toloka delivery-order list` shows the orders and answers with an explanation for accounts without internal teams.
|
|
90
94
|
|
|
91
95
|
`project untrack` removes one pipeline or dataset binding from `toloka.json` without deleting the live resource. It previews the name and id in a terminal; outside a terminal, pass `--confirm` after reviewing the state file.
|
|
92
96
|
|
|
@@ -96,7 +100,7 @@ toloka project untrack <pipeline|dataset> <name> [--confirm]
|
|
|
96
100
|
|
|
97
101
|
## delivery-order
|
|
98
102
|
|
|
99
|
-
|
|
103
|
+
Lists the Sunday delivery orders this key can attach on `project create`. The command is always registered; the platform answers `403` for accounts without an internal team, which the CLI reports as `Delivery orders are available to internal teams only.`
|
|
100
104
|
|
|
101
105
|
```sh
|
|
102
106
|
toloka delivery-order list
|
|
@@ -135,7 +139,7 @@ Shows which platform and key the CLI would use. Alias: `whoami`.
|
|
|
135
139
|
toloka auth-status [--show-token]
|
|
136
140
|
```
|
|
137
141
|
|
|
138
|
-
On success it prints the platform URL with its source, the API key masked unless `--show-token` is passed, the selected project, `Reachable: N project(s) visible.`,
|
|
142
|
+
On success it prints the platform URL with its source, the API key masked unless `--show-token` is passed, the selected project, `Reachable: N project(s) visible.`, `Internal: true` or `Internal: false` for the account, and a `Teams:` line that marks each team `(internal)` and `(read-only)` where that applies. The project line is printed even when the probe fails. It exits `1` when no key is in effect, when the platform rejects the key (`Rejected: …`, followed by the advice to run `toloka login`), when the platform does not answer (`Not reachable: …`; the key was not evaluated), or when the platform fails (`Platform error: …`).
|
|
139
143
|
|
|
140
144
|
## config
|
|
141
145
|
|
|
@@ -267,7 +271,7 @@ toloka base-model list [--json]
|
|
|
267
271
|
toloka base-model get <id> [--json]
|
|
268
272
|
```
|
|
269
273
|
|
|
270
|
-
Human-readable output is the default. `base-model list` prints aligned ID, name, provider, and upstream-
|
|
274
|
+
Human-readable output is the default. `base-model list` prints aligned ID, name, provider, upstream-model, revision (first 12 characters) and upstream-snapshot URL columns; `base-model get` prints labeled fields including the full pinned revision and the `https://huggingface.co/<model>/tree/<revision>` snapshot URL. Pass `--json` for machine-readable output.
|
|
271
275
|
|
|
272
276
|
## model
|
|
273
277
|
|
|
@@ -283,12 +287,34 @@ toloka model archive <name-or-id> [--json]
|
|
|
283
287
|
|
|
284
288
|
`model create` creates project-owned Model metadata linked to one catalog Base Model; it does not create a Model Version. Human-readable output is the default for every command in this group. Pass `--json` for machine-readable objects or arrays. Weight upload, validation, download, and Model Version creation are deferred.
|
|
285
289
|
|
|
290
|
+
## training
|
|
291
|
+
|
|
292
|
+
Create a reusable Training draft, price it, and run it.
|
|
293
|
+
|
|
294
|
+
```sh
|
|
295
|
+
toloka training list [--json]
|
|
296
|
+
toloka training get <name-or-id> [--json]
|
|
297
|
+
toloka training create <name> [--method sft|gisting] (--base-model <id> | --model-version <id>) --dataset <id> [--train-filter field=value] [--validation-dataset <id> [--validation-filter field=value]] [--prompt-file <path>] [--lora-rank 8|16|32 | --gist-tokens 8|32|64] [--epochs 1..5] [--learning-rate <rate>] [--batch-size 8|16|32|64] [--max-sequence-length <tokens>] [--json]
|
|
298
|
+
toloka training update <name-or-id> [--base-model <id> | --model-version <id>] [--dataset <id>] [--train-filter field=value] [--validation-dataset <id> [--validation-filter field=value]] [--prompt-file <path>] [--lora-rank 8|16|32 | --gist-tokens 8|32|64] [--epochs 1..5] [--learning-rate <rate>] [--batch-size 8|16|32|64] [--max-sequence-length <tokens>] [--json]
|
|
299
|
+
toloka training cost <name-or-id> [--idempotency-key <key>] [--json]
|
|
300
|
+
toloka training run <name-or-id> [--estimate-file <path>] [--version-name <name>] [--idempotency-key <key>] [--json]
|
|
301
|
+
toloka training runs <name-or-id> [--json]
|
|
302
|
+
toloka training status <training-run-id> [--json]
|
|
303
|
+
toloka training resume <training-run-id> [--json]
|
|
304
|
+
toloka training cancel <training-run-id> [--json]
|
|
305
|
+
toloka training logs <training-run-id>
|
|
306
|
+
toloka training metrics <training-run-id> [--key <metric>] [--resolution <n>] [--json]
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
Hyperparameters are optional flags on `training create` and `training update`: `--epochs 1..5` (default 3), `--learning-rate` (SFT 0.00001–0.001, default 0.00015; gisting 0.001–0.3, default 0.03), `--batch-size 8|16|32|64` (default 8; sequences per optimizer step across all GPUs) and `--max-sequence-length 64..16384` (default 12288 for SFT, 3072 for gisting) apply to both methods; `--lora-rank 8|16|32` (default 16) is SFT-only and `--gist-tokens 8|32|64` (default 32) is gisting-only — passing one to the other method fails before any request. `--regime` and `--eval-steps` no longer exist. `training update` has no `--method` flag: the method stays as created, and every flag you omit keeps its current value. `training cost` is blocked when the longest example exceeds `--max-sequence-length` or when the training selection leaves fewer rows than `--batch-size` after the 10% holdout. `training get` prints the resolved recipe, so omitted flags are visible as their defaults.
|
|
310
|
+
|
|
286
311
|
## dataset
|
|
287
312
|
|
|
288
313
|
Catalog and row commands for datasets in the selected project. Name or id is accepted; duplicate names must be addressed by id.
|
|
289
314
|
|
|
290
315
|
```sh
|
|
291
316
|
toloka dataset list
|
|
317
|
+
toloka dataset create <name>
|
|
292
318
|
toloka dataset fields <name>
|
|
293
319
|
toloka dataset load <name> --out <file> [--limit <n>] [--timeout <ms>]
|
|
294
320
|
toloka dataset upload <name> <file>
|
|
@@ -296,6 +322,8 @@ toloka dataset upload <name> <file>
|
|
|
296
322
|
|
|
297
323
|
`dataset list` prints live `name` and id. It does not page items.
|
|
298
324
|
|
|
325
|
+
`dataset create` makes an empty dataset in the selected project; the first `dataset upload` infers its columns from the rows.
|
|
326
|
+
|
|
299
327
|
`dataset fields` prints inferred `field()` TypeScript, or `Dataset <name> has no inferred fields yet.`
|
|
300
328
|
|
|
301
329
|
`dataset load` starts a platform export, waits for it (default 60s; pass `--timeout`), downloads the JSON array, and writes JSONL to `--out`. `--limit` slices after download; it still waits for the full export. On timeout the message names the export id. Failed downloads and invalid payloads leave `--out` untouched.
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import type { TrainingModelSource } from './trainingModelSource';
|
|
2
2
|
import type { TrainingDatasetSelection } from './trainingDatasetSelection';
|
|
3
|
-
import type {
|
|
3
|
+
import type { GistingTrainingConfigurationGistTokens } from './gistingTrainingConfigurationGistTokens';
|
|
4
|
+
import type { GistingTrainingConfigurationBatchSize } from './gistingTrainingConfigurationBatchSize';
|
|
4
5
|
|
|
5
6
|
export interface GistingTrainingConfiguration {
|
|
6
7
|
method: 'gisting';
|
|
@@ -8,6 +9,9 @@ export interface GistingTrainingConfiguration {
|
|
|
8
9
|
training: TrainingDatasetSelection;
|
|
9
10
|
validation?: TrainingDatasetSelection;
|
|
10
11
|
promptFileId: string;
|
|
11
|
-
|
|
12
|
+
gistTokens: GistingTrainingConfigurationGistTokens;
|
|
12
13
|
epochs: number;
|
|
14
|
+
learningRate: number;
|
|
15
|
+
batchSize: GistingTrainingConfigurationBatchSize;
|
|
16
|
+
maxSequenceLength: number;
|
|
13
17
|
}
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import type { TrainingModelSource } from './trainingModelSource';
|
|
2
2
|
import type { TrainingDatasetSelectionInput } from './trainingDatasetSelectionInput';
|
|
3
|
-
import type {
|
|
3
|
+
import type { GistingTrainingConfigurationInputGistTokens } from './gistingTrainingConfigurationInputGistTokens';
|
|
4
|
+
import type { GistingTrainingConfigurationInputBatchSize } from './gistingTrainingConfigurationInputBatchSize';
|
|
4
5
|
|
|
5
6
|
export interface GistingTrainingConfigurationInput {
|
|
6
7
|
method: 'gisting';
|
|
@@ -8,6 +9,9 @@ export interface GistingTrainingConfigurationInput {
|
|
|
8
9
|
training: TrainingDatasetSelectionInput;
|
|
9
10
|
validation?: TrainingDatasetSelectionInput;
|
|
10
11
|
promptFileId: string;
|
|
11
|
-
|
|
12
|
+
gistTokens?: GistingTrainingConfigurationInputGistTokens;
|
|
12
13
|
epochs?: number;
|
|
14
|
+
learningRate?: number;
|
|
15
|
+
batchSize?: GistingTrainingConfigurationInputBatchSize;
|
|
16
|
+
maxSequenceLength?: number;
|
|
13
17
|
}
|
|
@@ -110,6 +110,7 @@ export * from './createTrainingRunBody';
|
|
|
110
110
|
export * from './createdDatasetFields';
|
|
111
111
|
export * from './currentUser';
|
|
112
112
|
export * from './currentUserEmail';
|
|
113
|
+
export * from './currentUserTeamsItem';
|
|
113
114
|
export * from './dataset';
|
|
114
115
|
export * from './datasetExportReport';
|
|
115
116
|
export * from './datasetExportReportErrorsItem';
|
|
@@ -264,9 +265,11 @@ export * from './getDatasetInferredFields200';
|
|
|
264
265
|
export * from './getTrainingRunLogsParams';
|
|
265
266
|
export * from './getTrainingRunMetricHistoryParams';
|
|
266
267
|
export * from './gistingTrainingConfiguration';
|
|
268
|
+
export * from './gistingTrainingConfigurationBatchSize';
|
|
269
|
+
export * from './gistingTrainingConfigurationGistTokens';
|
|
267
270
|
export * from './gistingTrainingConfigurationInput';
|
|
268
|
-
export * from './
|
|
269
|
-
export * from './
|
|
271
|
+
export * from './gistingTrainingConfigurationInputBatchSize';
|
|
272
|
+
export * from './gistingTrainingConfigurationInputGistTokens';
|
|
270
273
|
export * from './importPipelineBody';
|
|
271
274
|
export * from './inferredField';
|
|
272
275
|
export * from './inferredFieldJsonSchema';
|
|
@@ -821,9 +824,11 @@ export * from './sampleJobStatus';
|
|
|
821
824
|
export * from './sampleResult';
|
|
822
825
|
export * from './setProjectSecretBody';
|
|
823
826
|
export * from './sftTrainingConfiguration';
|
|
827
|
+
export * from './sftTrainingConfigurationBatchSize';
|
|
824
828
|
export * from './sftTrainingConfigurationInput';
|
|
825
|
-
export * from './
|
|
826
|
-
export * from './
|
|
829
|
+
export * from './sftTrainingConfigurationInputBatchSize';
|
|
830
|
+
export * from './sftTrainingConfigurationInputLoraRank';
|
|
831
|
+
export * from './sftTrainingConfigurationLoraRank';
|
|
827
832
|
export * from './training';
|
|
828
833
|
export * from './trainingArchivedAt';
|
|
829
834
|
export * from './trainingConfiguration';
|
|
@@ -1,13 +1,16 @@
|
|
|
1
1
|
import type { TrainingModelSource } from './trainingModelSource';
|
|
2
2
|
import type { TrainingDatasetSelection } from './trainingDatasetSelection';
|
|
3
|
-
import type {
|
|
3
|
+
import type { SftTrainingConfigurationLoraRank } from './sftTrainingConfigurationLoraRank';
|
|
4
|
+
import type { SftTrainingConfigurationBatchSize } from './sftTrainingConfigurationBatchSize';
|
|
4
5
|
|
|
5
6
|
export interface SftTrainingConfiguration {
|
|
6
7
|
method: 'sft';
|
|
7
8
|
source: TrainingModelSource;
|
|
8
9
|
training: TrainingDatasetSelection;
|
|
9
10
|
validation?: TrainingDatasetSelection;
|
|
10
|
-
|
|
11
|
+
loraRank: SftTrainingConfigurationLoraRank;
|
|
11
12
|
epochs: number;
|
|
12
|
-
|
|
13
|
+
learningRate: number;
|
|
14
|
+
batchSize: SftTrainingConfigurationBatchSize;
|
|
15
|
+
maxSequenceLength: number;
|
|
13
16
|
}
|
|
@@ -1,13 +1,16 @@
|
|
|
1
1
|
import type { TrainingModelSource } from './trainingModelSource';
|
|
2
2
|
import type { TrainingDatasetSelectionInput } from './trainingDatasetSelectionInput';
|
|
3
|
-
import type {
|
|
3
|
+
import type { SftTrainingConfigurationInputLoraRank } from './sftTrainingConfigurationInputLoraRank';
|
|
4
|
+
import type { SftTrainingConfigurationInputBatchSize } from './sftTrainingConfigurationInputBatchSize';
|
|
4
5
|
|
|
5
6
|
export interface SftTrainingConfigurationInput {
|
|
6
7
|
method: 'sft';
|
|
7
8
|
source: TrainingModelSource;
|
|
8
9
|
training: TrainingDatasetSelectionInput;
|
|
9
10
|
validation?: TrainingDatasetSelectionInput;
|
|
10
|
-
|
|
11
|
+
loraRank?: SftTrainingConfigurationInputLoraRank;
|
|
11
12
|
epochs?: number;
|
|
12
|
-
|
|
13
|
+
learningRate?: number;
|
|
14
|
+
batchSize?: SftTrainingConfigurationInputBatchSize;
|
|
15
|
+
maxSequenceLength?: number;
|
|
13
16
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@toloka-platform/sdk",
|
|
3
|
-
"version": "0.2.0-preview.
|
|
3
|
+
"version": "0.2.0-preview.4",
|
|
4
4
|
"private": false,
|
|
5
5
|
"description": "Beta. Build, inspect, and operate Toloka pipeline projects as TypeScript source. The public API may change.",
|
|
6
6
|
"license": "Apache-2.0",
|
package/skills/catalog.json
CHANGED
|
@@ -11,8 +11,8 @@
|
|
|
11
11
|
"testing",
|
|
12
12
|
"checks"
|
|
13
13
|
],
|
|
14
|
-
"version": "0.2.0-preview.
|
|
15
|
-
"digest": "
|
|
14
|
+
"version": "0.2.0-preview.4",
|
|
15
|
+
"digest": "2d27360563da6ff5ef41d264982d89232087cf7c76393022c83a66c5301752b3",
|
|
16
16
|
"files": [
|
|
17
17
|
"SKILL.md",
|
|
18
18
|
"references/node-semantics.md",
|
|
@@ -29,7 +29,7 @@
|
|
|
29
29
|
"bug",
|
|
30
30
|
"report"
|
|
31
31
|
],
|
|
32
|
-
"version": "0.2.0-preview.
|
|
32
|
+
"version": "0.2.0-preview.4",
|
|
33
33
|
"digest": "76229bae596f8a1722ce7fc55bea3dd66698df5b61baa31f6cb98ca2b5fa03e8",
|
|
34
34
|
"files": [
|
|
35
35
|
"SKILL.md",
|
|
@@ -39,15 +39,15 @@
|
|
|
39
39
|
{
|
|
40
40
|
"id": "toloka-get-started",
|
|
41
41
|
"name": "Get started with Toloka",
|
|
42
|
-
"description": "
|
|
42
|
+
"description": "Orient a Toloka session to the user's outcome and safely reuse or establish the needed CLI, authentication, project, workflow, and approval context. Use when the user is new to Toloka, asks what to do first, or has not selected a pipeline, fine-tuning, or gisting workflow.",
|
|
43
43
|
"tags": [
|
|
44
44
|
"toloka",
|
|
45
45
|
"onboarding",
|
|
46
46
|
"getting-started",
|
|
47
47
|
"agent"
|
|
48
48
|
],
|
|
49
|
-
"version": "0.2.0-preview.
|
|
50
|
-
"digest": "
|
|
49
|
+
"version": "0.2.0-preview.4",
|
|
50
|
+
"digest": "f86ba45aba0bf2b06bc02b65c6f3dbb7b9b4cb28630191bafa62b6e627448138",
|
|
51
51
|
"files": [
|
|
52
52
|
"SKILL.md"
|
|
53
53
|
]
|
|
@@ -63,8 +63,8 @@
|
|
|
63
63
|
"sft",
|
|
64
64
|
"models"
|
|
65
65
|
],
|
|
66
|
-
"version": "0.2.0-preview.
|
|
67
|
-
"digest": "
|
|
66
|
+
"version": "0.2.0-preview.4",
|
|
67
|
+
"digest": "81c43af9e78e1df322085dd2ec923e85a7be0d084c689906aa5db6cdbd03c9e3",
|
|
68
68
|
"files": [
|
|
69
69
|
"SKILL.md"
|
|
70
70
|
]
|
|
@@ -80,8 +80,8 @@
|
|
|
80
80
|
"prompt",
|
|
81
81
|
"models"
|
|
82
82
|
],
|
|
83
|
-
"version": "0.2.0-preview.
|
|
84
|
-
"digest": "
|
|
83
|
+
"version": "0.2.0-preview.4",
|
|
84
|
+
"digest": "ee30f006d8f7995a825db5e7c74732e35517d88b43a4f6671d6439f1c333c815",
|
|
85
85
|
"files": [
|
|
86
86
|
"SKILL.md"
|
|
87
87
|
]
|
|
@@ -7,16 +7,25 @@ description: Build, test, inspect, or change Toloka platform pipeline projects a
|
|
|
7
7
|
|
|
8
8
|
This SDK is Beta software. The authoring API, the CLI, and generated source can change before general availability. Do not invent helpers or flags that are not in the installed package.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Help the user get a working pipeline that fits their task: understand the data and desired output, choose a suitable design, verify it, and apply or run it when requested. Work in the project source tree; TypeScript is the pipeline's desired state, not a transcription of direct platform API calls.
|
|
11
11
|
|
|
12
12
|
## Enter the pipeline workflow
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
For first-time setup or help choosing a workflow, install and read `toloka-get-started` and consult the matching public runbook. Use its onboarding steps where needed; an existing project or a focused inspection request does not need the full first-run journey. A demo is useful for learning, not a prerequisite for the user's own task.
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
Use your expertise for routine choices, and briefly explain decisions that affect quality, cost, or maintainability. Ask a focused question when missing requirements or conflicting constraints would materially change the result; do not ask the user to choose SDK mechanics you can determine yourself. Preserve explicit requirements and existing project intent. Credentials belong in the user's `toloka login` prompt, paid runs need cost approval, and destructive or out-of-scope changes need agreement before proceeding.
|
|
17
|
+
|
|
18
|
+
## Choose a design that fits the task
|
|
19
|
+
|
|
20
|
+
- Start with representative inputs and the output the user needs. Reuse an existing pipeline or node when it already serves that purpose; prefer the simplest graph that meets the requirements.
|
|
21
|
+
- Use `code` for deterministic transformations, scripts, or programmatic API/LLM calls; `generation` for the platform's human or synthetic generation journey, including original content collection; `quorum` for independent annotations where agreement or disagreement matters; and `match` for conditional routing. `root` defines the input/output boundary.
|
|
22
|
+
- Choose data contracts, audience, instructions, and quality criteria together: the person doing the task should be able to produce the declared output and understand how it will be judged. Let the task and observed data guide these choices rather than copying an example's settings.
|
|
23
|
+
- Treat quality, turnaround, and cost as tradeoffs. Explain meaningful assumptions, and seek clarification for unresolved acceptance criteria rather than inventing business rules. Read [node semantics](references/node-semantics.md) for each selected node's actual capabilities and constraints.
|
|
17
24
|
|
|
18
25
|
## Follow the reconciliation loop
|
|
19
26
|
|
|
27
|
+
For an applied change, the dependencies are read → edit → check/validate → plan → apply → confirm convergence. Iterate locally as needed. For inspection, design, or check-only requests, stop at the requested deliverable rather than creating resources, applying changes, or starting a run merely to finish this sequence.
|
|
28
|
+
|
|
20
29
|
1. Read `toloka.json` when present, then read the relevant `pipelines/**/pipeline.ts` module and everything it imports.
|
|
21
30
|
2. Edit the TypeScript source. Treat it as authoritative desired state.
|
|
22
31
|
3. Run the complete local gate, then validate each changed pipeline against the platform:
|
|
@@ -25,7 +34,7 @@ Do not replace the selected runbook's quick demo with another example, and do no
|
|
|
25
34
|
toloka validate
|
|
26
35
|
```
|
|
27
36
|
4. Run `toloka plan`. Read the complete diff and resolve every conflict or missing prerequisite.
|
|
28
|
-
5.
|
|
37
|
+
5. Show the plan and run `toloka apply` when its changes match the requested work. Resolve removals, immutable-field conflicts, or changes outside that scope with the user first. Apply reruns the complete local gate and proceeds only when it passes.
|
|
29
38
|
6. Run `toloka plan` again and require `No changes.`
|
|
30
39
|
|
|
31
40
|
The CLI reads source, compares it with live state, and reconciles the platform. Edit files; never mutate a pipeline through raw HTTP, SDK client methods, or internal platform tool calls. Do not make a live edit and then reverse-engineer it into source.
|
|
@@ -37,13 +46,15 @@ Use the directory containing `toloka.json` as the source root. Commands default
|
|
|
37
46
|
- If `toloka.json` exists, read it as CLI-managed identity state and confirm its `projectId` matches `toloka project current` before any plan or apply. It maps the project, pipeline source directories, and dataset names to live IDs. Do not hand-edit it.
|
|
38
47
|
- If it is absent and the user is adopting an existing project, run `toloka project add <alias> <projectId>`, then run `toloka import <projectId>` in the intended source root before `toloka init`. Import refuses an existing `pipelines/` directory unless you pass `--force`.
|
|
39
48
|
- After importing an existing project, run `toloka plan` before editing and require `No changes.`. Resolve importer or live-state drift separately so it cannot be mistaken for your change.
|
|
40
|
-
- If it is absent and the user is authoring from scratch,
|
|
49
|
+
- If it is absent and the user is authoring from scratch, create a project with a task-derived name when a new project is needed. For an intended existing empty project, use `toloka project add <alias> <projectId>` and `toloka --project-id <projectId> init`. Confirm `toloka project current` and the generated `toloka.json` name the intended project before authoring; do not create a duplicate merely because the checkout is unbound.
|
|
41
50
|
|
|
42
51
|
If `toloka project current` reports `from toloka.json` for a project you did not select in this checkout, stop: project create/add/switch does not rebind an existing checkout. A newly generated state file with empty `pipelines` and `datasets` can be removed before selecting the intended project and rerunning `toloka init`; any state file with live bindings may be someone else's checkout and must not be deleted without user approval.
|
|
43
52
|
|
|
44
53
|
Never assume a project ID from conversation context. Resolve it from `--project-id`, `TOLOKA_PROJECT_ID`, `toloka.json`, or `toloka project current`.
|
|
45
54
|
|
|
46
|
-
|
|
55
|
+
An unverified trial account holds one project and one pipeline, and cannot deploy or run. `project create` then fails with `Your trial allows a single project.` and pipeline creation with `Your trial allows a single pipeline.` Adopt the project that already exists instead: register it with `toloka project add <alias> <projectId>`, bind the checkout with `toloka import <projectId>`, and author inside its existing pipeline; the platform has no project delete, so there is nothing to clear. Both limits lift when the user completes identity or business verification from the banner in the platform.
|
|
56
|
+
|
|
57
|
+
`project create`, `project add`, and `project switch` also change the machine-level current-project registry used outside a source root. For temporary or evaluation projects, record the previous alias and restore it before finishing. Without `--team-id`, create uses the account's only writable team; when `toloka whoami` lists several teams, ask the user which one owns the project and pass it explicitly. Confirm the new project's team in `toloka project url`.
|
|
47
58
|
|
|
48
59
|
## Author with the typed graph API
|
|
49
60
|
|
|
@@ -51,23 +62,23 @@ Import authoring helpers from `@toloka-platform/sdk`.
|
|
|
51
62
|
|
|
52
63
|
- Declare every data contract once with `field(name, jsonSchema)`.
|
|
53
64
|
- Decide entity cardinality from the task before choosing field schemas or Task UI components. One logical output from one labeler is a static entity, even when a component represents it with an array; only multiple independent outputs are dynamic. For dynamic entities, choose a tight task-grounded `maxEntities` instead of defaulting to 50.
|
|
54
|
-
-
|
|
55
|
-
-
|
|
65
|
+
- Ground enum, length, range, format, label-precedence, tie-break, default, exact-phrase, and conditional-requiredness rules in the request, observed data, or documented platform requirements. Derive criteria, quiz answers, and UI validation from that evidence. Surface assumptions that affect accepted outputs before apply; if materially different interpretations remain, ask instead of silently deciding the user's business rules.
|
|
66
|
+
- The supported node constructors are `root`, `code`, `generation`, `match`, and `quorum`.
|
|
56
67
|
- Connect typed ports with `connect(from, to)`: `started`, `completed`, `input`, `output`, `matchNode.matched(0)`, and `default`.
|
|
57
68
|
- Export a default `definePipeline({ dataset, fields, nodes, connections })` from every `pipelines/**/pipeline.ts`.
|
|
58
69
|
- Keep the graph acyclic. Put review, retry, training, exam, appeal, and payment behavior in the owning human-work node rather than adding loop nodes.
|
|
59
70
|
|
|
60
71
|
Let constructors mint IDs for new declarations. Preserve `id` options generated by `toloka import`, but never invent or hand-edit raw `fp_…`, `ds.…`, `pl.…`, or `pp.…` values in authored settings. Never add a `layout` block: layout is not authored, existing layout is preserved, and the server supplies defaults.
|
|
61
72
|
|
|
62
|
-
|
|
73
|
+
Use `node_modules/@toloka-platform/sdk/examples/face-image-collection/` as a concrete authoring pattern when relevant; borrow its structure, not its task-specific choices.
|
|
63
74
|
|
|
64
75
|
## Get human-work settings right the first time
|
|
65
76
|
|
|
66
|
-
|
|
77
|
+
Distinguish server requirements from optional design choices. The following constraints and defaults are not all checked by `definePipeline`:
|
|
67
78
|
|
|
68
79
|
- **Do not pair `projectType: 'general'` with `verifiedExperts: false`.** The server answers `General projects require an ID-verified audience`. Specializations also require a verified audience.
|
|
69
80
|
- **Training is part of the node, not extra.** A quorum node requires `onboarding`, and while it is enabled the platform wants at least three steps and at least one quiz block. Preserve an explicit `enabled: false` that a user or `toloka import` put there; never switch Training on to satisfy your own sense of completeness.
|
|
70
|
-
- **
|
|
81
|
+
- **Optional settings should serve the task.** Exams, non-default QA, audience reuse, and smart search can be useful when the requirements call for them. Omit blocks that add no value rather than copying placeholders; an empty disabled block is still authored state. Preserve imported choices unless changing them is part of the task, and explain additions that affect cost or expert eligibility.
|
|
71
82
|
- **Run scope belongs to the platform.** `itemFilter` and `itemLimit` on the root node are set from the platform, and apply preserves whatever is live unless source declares its own value. Do not add an item limit to source to "restore" one you saw in the UI.
|
|
72
83
|
|
|
73
84
|
## Know the validation boundary
|
|
@@ -82,7 +93,7 @@ Before choosing Task UI components, call `ui.componentDocumentation()` from a lo
|
|
|
82
93
|
|
|
83
94
|
`ui.code` accepts TSX, including TypeScript annotations. `preprocess` and `postprocess` are JavaScript function bodies; do not put TypeScript-only syntax in either transform.
|
|
84
95
|
|
|
85
|
-
Register code behavior with `test.code(pipeline, node, name, callback)` and execute the bound `run` helper. Add at least one `test.pipeline(pipeline, name, callback)` scenario.
|
|
96
|
+
Register code behavior with `test.code(pipeline, node, name, callback)` and execute the bound `run` helper. Add at least one `test.pipeline(pipeline, name, callback)` scenario. Choose representative fixtures from available data, redacting sensitive values while preserving field names, nesting, optionality, and media/value shapes. Use synthetic cases when needed for privacy or to exercise boundaries absent from the sample. Treat the fixture as `root.started` item state, execute connected nodes in order with the callback's bound code and UI helpers, merge only each node's declared outputs, and assert the final `root.completed` fields. Cover meaningful match branches, optional paths, and failures as well as the happy path.
|
|
86
97
|
|
|
87
98
|
## Keep data and credentials out of source
|
|
88
99
|
|
|
@@ -94,7 +105,7 @@ Register code behavior with `test.code(pipeline, node, name, callback)` and exec
|
|
|
94
105
|
`toloka login` verifies an API key and stores it with owner-only permissions, so no later command needs
|
|
95
106
|
`--api-key`. Precedence is flag, then environment variable (`TOLOKA_API_KEY`, `TOLOKA_BASE_URL`), then
|
|
96
107
|
that file. `toloka auth-status` reports what is in effect and where it came from; `toloka logout`
|
|
97
|
-
removes it. If
|
|
108
|
+
removes it. If credentials are missing, ask the user to run `toloka login` in their terminal; never request a key in chat.
|
|
98
109
|
Anonymous command telemetry is on by default; set `TOLOKA_TELEMETRY=0` or `DO_NOT_TRACK=1` to disable it.
|
|
99
110
|
A globally installed CLI auto-updates by default; turn that off with `toloka config set auto-update false`.
|
|
100
111
|
|
|
@@ -107,14 +118,14 @@ A globally installed CLI auto-updates by default; turn that off with `toloka con
|
|
|
107
118
|
|
|
108
119
|
## Run a deployed pipeline
|
|
109
120
|
|
|
110
|
-
Before starting a run, require a passing `toloka check` and no blocking error from `toloka validate`. A warning-severity `not_ran_node_check` concerns a separate Remote Node Check and does not block the Local Check Gate; an error-severity instance remains a blocking remote-validation result. Run `toloka pipeline cost <name>`
|
|
121
|
+
Before starting a run, require a passing `toloka check` and no blocking error from `toloka validate`. A warning-severity `not_ran_node_check` concerns a separate Remote Node Check and does not block the Local Check Gate; an error-severity instance remains a blocking remote-validation result. Run `toloka pipeline cost <name>` and present the forecast for user approval before starting the run. If it skips a billed node, report the authored rate and a labeled labor floor instead of presenting `$0.00` as the price; make the missing pricing evidence clear when seeking approval.
|
|
111
122
|
|
|
112
123
|
Start a run with `toloka pipeline run <name> --wait --timeout <ms>`, then read its exit code: `0` succeeded, `1` reached a failed terminal state, `2` still running when the window closed. Without `--wait`, exit `0` only means the run was accepted. Nothing reports back later, so never describe a started run as a finished one. Name the live pipeline; there is no implicit only-pipeline.
|
|
113
124
|
|
|
114
|
-
|
|
125
|
+
Keep monitoring bounded so the user receives a useful update even when expert work or queues take time. On exit `2`, report the observed item counts and states and say the run is still in progress; choose a further bounded check only when it serves the request. Use `toloka pipeline status <name>` and `toloka pipeline results <runId> --failed` for counts, states, and errors. `toloka status` (no prefix) is project health, not the latest run.
|
|
115
126
|
|
|
116
127
|
## Report with evidence
|
|
117
128
|
|
|
118
|
-
|
|
129
|
+
Explain what changed, how it meets the task, what you verified, and what remains unresolved. Include relevant project/pipeline links, meaningful design assumptions, and observed run state and cost. Distinguish applied, validated, started, and completed; do not promise a later update unless a real reporting mechanism exists.
|
|
119
130
|
|
|
120
|
-
|
|
131
|
+
Keep enough command output and exit-status evidence to support claims about checks, mutations, and convergence, redacting secrets and sensitive data. For an audited task, retain the requested detailed records, including skill versions and project-registry changes. State when evidence is missing, and limit visual claims to surfaces you actually inspected. Use `toloka-report-issue` for product defects rather than presenting a workaround as normal workflow.
|
|
@@ -68,7 +68,7 @@ The dataset name belongs in `definePipeline({ dataset: 'name', ... })`, not in t
|
|
|
68
68
|
|
|
69
69
|
Use `code` whenever the result can be computed from inputs without a human task. This includes deterministic transforms, ETL, filtering, parsing, arithmetic, string manipulation, scripts, HTTP APIs, and LLM API calls. An LLM call made programmatically is still `code`; choose `generation` only when the platform's human or synthetic generation journey is wanted.
|
|
70
70
|
|
|
71
|
-
Use a downstream `code` node after `quorum`
|
|
71
|
+
Use a downstream `code` node after `quorum` when the requested deliverable needs a derived consensus value such as majority vote, average, or deduplication. Quorum itself returns raw collected annotations. Choose an aggregation rule grounded in the task, and clarify material ambiguity rather than inventing label precedence.
|
|
72
72
|
|
|
73
73
|
### Configure it
|
|
74
74
|
|
|
@@ -158,17 +158,17 @@ Write at least one criterion for every output entity. Each criterion needs an au
|
|
|
158
158
|
|
|
159
159
|
Use `llmQaInstruction` only when the evaluator needs a specific verification method, authoritative source, tool, or pinned edge case beyond the thresholds. It is evaluator guidance, not labeler guidance, and has no effect in `human_only` mode.
|
|
160
160
|
|
|
161
|
-
Default QA is `qualityAssurance: { qaMode: 'llm_only' }`.
|
|
161
|
+
Default QA is `qualityAssurance: { qaMode: 'llm_only' }`. Keep it when it fits the task. `hybrid` adds sampled Human QA while retaining LLM review; `human_only` is appropriate when LLM grading is prohibited or human review alone is required. Recommend a mode based on the user's quality and privacy requirements, explaining cost and turnaround implications before changing it. Human-bearing modes require `humanQaTimeForTask` of at least 10 seconds; `smartSamplingPercent` defaults to 30 for hybrid and 100 for human-only. Use those defaults unless the task justifies a different rate. QA mode applies to the whole node, not individual criteria. The redo cap is platform-controlled and is not an SDK setting.
|
|
162
162
|
|
|
163
163
|
Keep `instruction` actionable and in English. Include the workflow, concepts, quality rules, and at least one pass and fail example grounded in the task. Do not mention project schedules, dataset volume, workforce requirements, or compensation in labeler instructions.
|
|
164
164
|
|
|
165
165
|
#### Audience and pricing
|
|
166
166
|
|
|
167
|
-
For humans, set `audienceType: 'human'`, `verifiedExperts`, all three audience filters, `pricePerTask`, and `timeForTask`. The public shapes differ by filter: languages use `{ type: 'and' | 'or', values: [{ value: 'English' }] }`; specializations use `{ type: 'and' | 'or', values: [{ value, qualificationId }] }`; countries use ISO 3166-1 alpha-2 codes, for example `{ type: 'or', values: ['US'] }`. Verify every country value matches `/^[A-Z]{2}$/`; apply currently accepts arbitrary strings that can match no experts.
|
|
167
|
+
For humans, set `audienceType: 'human'`, `verifiedExperts`, all three audience filters, `pricePerTask`, and `timeForTask`. The public shapes differ by filter: languages use `{ type: 'and' | 'or', values: [{ value: 'English' }] }`; specializations use `{ type: 'and' | 'or', values: [{ value, qualificationId }] }`; countries use ISO 3166-1 alpha-2 codes, for example `{ type: 'or', values: ['US'] }`. Verify every country value matches `/^[A-Z]{2}$/`; apply currently accepts arbitrary strings that can match no experts. Derive languages and expertise from the task and its inputs, leaving specialization and country filters unrestricted unless needed. English is a reasonable starting language for an English-language task, not a universal audience default. Ask when an unresolved audience requirement changes who can do the work. Do not put specializations on an unverified audience. A specialization's `qualificationId` is platform identity: preserve an imported value, and do not invent one when the SDK cannot discover it.
|
|
168
168
|
|
|
169
|
-
|
|
169
|
+
`$0.20` and 120 seconds are a starting point for a short task, not a universal rate. Estimate effort from the work and required expertise, respect explicit rates, and explain meaningful adjustments in the cost forecast. The per-expert cap is normally off. For collection tasks, choose a cap based on contributor diversity and repetition requirements; 1 fits one-response-per-expert collection. Avoid copying an arbitrary cap into a task that does not need one.
|
|
170
170
|
|
|
171
|
-
|
|
171
|
+
Choose `audienceType: 'synthetic'` when model-generated data serves the requested outcome; preserve an explicit requirement for human judgments or original human content. Explain the quality and cost tradeoff if recommending synthetic generation. Set the required `maxCostPerAttempt`, the per-attempt ceiling in charged US dollars across fresh input, cached input, and output together, and configure the optional `generationModelId`, `reasoningEffort`, and `llmInstruction`; do not include human pricing or audience fields. Take `generationModelId` only from the union the SDK exports. Use a supported model suited to the task when there is evidence for that choice, or omit the property to use the platform default; do not guess IDs or capabilities. Synthetic generation is incompatible with `data_collection`, human QA, audience reuse, expert sourcing, and Training.
|
|
172
172
|
|
|
173
173
|
#### Training, Exam, UI, and advanced settings
|
|
174
174
|
|
|
@@ -218,7 +218,7 @@ Configure entities for one labeler's contribution, not for the outer array. A si
|
|
|
218
218
|
|
|
219
219
|
When a quorum output entity is `optional: true`, its outer field's `items` schema must also accept `null`, for example `{ type: 'array', items: { type: ['string', 'null'] } }`. The platform reserves a null annotation slot when that labeler omits the value and rejects an optional entity backed by non-nullable items.
|
|
220
220
|
|
|
221
|
-
Quorum does not aggregate. It writes all raw annotations back, including rejected ones. Add a downstream `code` node
|
|
221
|
+
Quorum does not aggregate. It writes all raw annotations back, including rejected ones. Add a downstream `code` node when the requested deliverable needs one consensus value, and make it tolerate arrays longer than `overlap`.
|
|
222
222
|
|
|
223
223
|
#### Explanation-driven quality
|
|
224
224
|
|
|
@@ -232,7 +232,7 @@ Default to `qualityAssurance: { qaMode: 'llm_only' }`. The LLM accepts or reject
|
|
|
232
232
|
|
|
233
233
|
**Quorum has no redo cycle.** A rejected annotation is never reopened or sent back to the same expert. Escalation adds a fresh annotation, and all collected annotations—accepted and rejected—remain in the output arrays. Hitting the cap with zero accepted annotations still returns the collected results.
|
|
234
234
|
|
|
235
|
-
Use `qaMode: 'none'`
|
|
235
|
+
Use `qaMode: 'none'` when agreement-based quality control fits the task better than LLM grading, explaining the tradeoff before changing an existing QA strategy. This mode requires:
|
|
236
236
|
|
|
237
237
|
- set `overlap` to at least 3;
|
|
238
238
|
- disable the per-expert cap;
|