@toloka-platform/sdk 0.12.0 → 0.12.2

package/dist/cli/toloka.js.jsc.sha256

@@ -1 +1 @@
-305f65ba5180216b7aff18c19fb093165f8bab01a8b33e5899f1ece42db9af8b  toloka.js.jsc
+8ed8ac9a70b9d78e84e61a5ab6f59647b65043707f6fd5a09e394e648151df5f  toloka.js.jsc

package/dist/cli/toloka.js.sha256

@@ -1 +1 @@
-7b5e7b3dc4fabe6f103616a89cf7418aed62482596e9b8482e8e88915e7a967d  toloka.js
+b5ed5a56f22841b7d917e9f4f7e974cb1a7d3c9fe4a0e58a1df5f8967a4be250  toloka.js

package/docs/cli.md

@@ -398,7 +398,7 @@ toloka training metrics <training-run-id> [--key <metric>] [--resolution <n>] [-
 
 Hyperparameters are optional flags on `training create` and `training update`: `--epochs 1..5` (default 3), `--learning-rate` (SFT 0.00001–0.001, default 0.00015; gisting 0.001–0.3, default 0.03), `--batch-size 8|16|32|64` (default 8; sequences per optimizer step across all GPUs) and `--max-sequence-length 64..16384` (default 12288 for SFT, 3072 for gisting) apply to both methods; `--lora-rank 8|16|32` (default 16) is SFT-only and `--gist-tokens` (default `1/8`) is gisting-only — passing one to the other method fails before any request. `--gist-tokens` takes an absolute count `8|32|64` or a fraction of the prompt being compressed (`1/4`, `1/8`, `1/16`, `1/32`), which the run resolves to `ceil(prompt tokens / N)` clamped to 8..256. `--regime` and `--eval-steps` no longer exist. `training update` has no `--method` flag: the method stays as created, and every flag you omit keeps its current value. `training cost` is blocked when the longest example exceeds `--max-sequence-length` or when the training selection leaves fewer rows than `--batch-size` after the 10% holdout. `training get` prints the resolved recipe, so omitted flags are visible as their defaults.
 
-A Training learns on a ready dataset slice that freezes a split column, and `training create` needs `--dataset` or `--slice`. `--dataset <name-or-id>` fills the empty cells of the dataset's split column (a new `split` column, 80/10/10, when it has none), freezes a slice named after the Training, and trains on it; it waits up to 10 minutes for each step and prints one line per step before `Created training`. A split column adopted from an ordinary column has no shares, so `--dataset` does not fill it; give it shares first with `toloka dataset split <dataset> --column <name> --parts <name>=<roles>:<percent>,...`. `--slice <id>` takes a slice frozen with `toloka dataset slice create`. With several split columns, `--split-column` picks the one to train with. The Training learns on the column's train parts, or the ones `--parts` names, and validates on its validate parts, including a part that both validates and tests; without validation rows, the run holds out 10% of the training rows. Rows without a part are never trained on. `training update` keeps the slice, split column and train parts. A new `--dataset` or `--slice` resets the split column and train parts unless you pass them too; `--dataset` splits again and freezes a new slice named after the Training. `training get` prints the slice, split column and train parts. `training cost` leaves out rows that any split column now puts in a test part, a validate + test part of the Training's own column apart, and prints how many as `Held-out test rows`. A Training saved before split slices were required is refused by `training cost` until `training update` gives it one.
+A Training learns on a ready dataset slice that freezes a split column, and `training create` needs `--dataset` or `--slice`. `--dataset <name-or-id>` fills the empty cells of the dataset's split column (a new `split` column, 80/10/10, when it has none), freezes a slice named after the Training, and trains on it; it waits up to 10 minutes for each step and prints one line per step before `Created training`. A split column adopted from an ordinary column has no shares, so `--dataset` does not fill it; give it shares first with `toloka dataset split <dataset> --column <name> --parts <name>=<roles>:<percent>,...`. `--slice <id>` takes a slice frozen with `toloka dataset slice create`. With several split columns, `--split-column` picks the one to train with. The Training learns on the column's train parts, or the ones `--parts` names, and validates on its validate parts, including a part that both validates and tests; without validation rows, the run holds out 10% of the training rows. Rows without a part are never trained on. `training update` keeps the slice, split column and train parts. A new `--dataset` or `--slice` resets the split column and train parts unless you pass them too; `--dataset` splits again and freezes a new slice named after the Training. `training get` prints the slice, split column and train parts. `training cost` leaves out rows that any split column now puts in a test part, a validate + test part of the Training's own column apart, and, when there are any, prints how many as `Left out as test`. The test parts of the Training's own column are outside the train parts to begin with, so they are not counted there. A Training saved before split slices were required is refused by `training cost` until `training update` gives it one.
 
 ## dataset
 
@@ -408,8 +408,9 @@ Catalog and row commands for datasets in the selected project. Name or id is acc
 toloka dataset list
 toloka dataset create <name>
 toloka dataset fields <name>
+toloka dataset fields add <name> (<field>… | --all) [--no-wait] [--timeout <ms>] [--json]
 toloka dataset columns <name> [--json]
-toloka dataset load <name> --out <file> [--limit <n>] [--timeout <ms>]
+toloka dataset load <name> --out <file> [--fields <fields>] [--limit <n>] [--timeout <ms>]
 toloka dataset upload <name> <file>
 toloka dataset split <name> [--column <name>] [--ratios 80/10/10 [--validate-on-test] | --parts <name[=roles][:percent],…>] [--balance-by <field>] [--seed <n>] [--filter field=value… | --slice <slice-id>] [--no-wait] [--timeout <ms>] [--json]
 toloka dataset slice create <name> [--name <name>] [--filter field=value]… [--freeze <columns>] [--no-wait] [--timeout <ms>] [--json]
@@ -420,21 +421,23 @@ toloka dataset slice show <slice-id> [--json]
 
 `dataset list` prints live `name` and id. It does not page items.
 
-`dataset create` makes an empty dataset in the selected project; the first `dataset upload` infers its columns from the rows.
+`dataset create` makes an empty dataset in the selected project. The first `dataset upload` stores each row whole in the `uploaded` field and infers the fields of the rows; `dataset fields add` adds them as dataset fields.
 
 `dataset fields` prints inferred `field()` TypeScript, or `Dataset <name> has no inferred fields yet.`
 
-`dataset columns` prints each visible column with its id and, for a split column, its status and its parts with shares and roles, for example `ready · train 80% (train), validate 10% (validate), test 10% (test)`. `--json` prints every field with its `split`.
+`dataset fields add` adds a dataset field for each named top-level field of the uploaded rows, with the type inferred from them, so `--filter`, `--balance-by` and `dataset split --column` can use its values; `--all` takes every top-level field that is not a dataset field yet. A dataset field name takes lowercase letters, numbers and underscores, starts with a letter, ends with a letter or number and has at most 128 characters: a named field that breaks the rule is refused, and `--all` skips such fields and prints which. The values are copied from the uploaded rows in the background, and from every later upload. The command waits up to 60 seconds for the copy (`--timeout <ms>` sets the wait) and prints the fields with their types; `--no-wait` returns at once. A slice or a split that reads a field refuses to start while the field is still being copied, and `dataset slice create` and `dataset split` wait for it first unless `--no-wait` is passed; `training create --dataset` always waits.
 
-`dataset load` starts a platform export, waits for it (default 60s; pass `--timeout`), downloads the JSON array, and writes JSONL to `--out`. `--limit` slices after download; it still waits for the full export. On timeout the message names the export id. Failed downloads and invalid payloads leave `--out` untouched.
+`dataset columns` prints each visible column with its id and, for a split column, its status and its parts with shares and roles, for example `ready · train 80% (train), validate 10% (validate), test 10% (test)`. `--json` prints every field with its `split` and `operations`, the background operations writing into it: `copy` while values are copied in from the field `sourceFieldId` names, and `split_fill` while a split column fills.
 
-`dataset upload` accepts `.json` arrays, `.jsonl` files with one object per line, and `.csv` files with a header row. The CLI infers the format from the extension and uploads the original bytes without conversion. If the blob transfer or finalization fails, the CLI cancels the started upload before returning the error, so another upload can start immediately and may reuse the same filename. Field schemas are immutable after the first successful upload: you can add columns, never alter or remove one. See [datasets](./datasets.md).
+`dataset load` starts a platform export, waits for it (default 60s; pass `--timeout`), downloads the JSON array, and writes JSONL to `--out`: one object per row with `index` and every visible field, or only the fields `--fields` names, for example `--fields uploaded,split`. `uploaded` holds each uploaded row whole, as one value. `--limit` slices after download; it still waits for the full export. On timeout the message names the export id. Failed downloads and invalid payloads leave `--out` untouched. A dataset without fields has no rows yet, and the command exits with 1.
 
-`dataset split` fills a split column: every row that has no value in it gets a part, sampled by share, and rows that already have a part keep it, so a test row stays a test row as the dataset grows. `--column` names the column (default `split`). A new name creates it; the name of a split column fills its empty cells with the shares, balancing column and seed of its last fill; the name of an ordinary column whose values name the parts makes it a split column without placing anything. `--ratios` takes whole percentages for `train`, `validate` and `test` that add up to 100 (default `80/10/10` for a new column); a 0% part is left out. `--parts` takes `name[=roles][:percent]` entries with roles `train`, `validate`, `test`, `validate+test` or `none`, for example `train:70,dev=validate:10,holdout=test:20`; a part named after a role takes it without `=roles`. To make an ordinary column a split column, `--parts` without percentages names the parts its values hold; without `--parts`, roles follow the values `train`, `validation`, `validate`, `val`, `dev` and `test`, and every value must be one of them. A part keeps its roles across fills, and the column keeps the shares of its last fill. `--validate-on-test` makes the test part the validation set too, the usual split when data is scarce; it needs a validate ratio of 0, for example `--ratios 80/0/20 --validate-on-test`, and the test score is then not fully held out. `--balance-by` shares out the rows of every label value on their own, so each part gets about the same mix of labels, though a label with only a few rows can still miss a part; a column with more than 1,000 distinct values fails the fill. `--seed` fixes the row order; it is generated when omitted and kept by the column. Shares carry over: rows that already have a part count toward it, so the column keeps its proportions as rows are added. `--filter field=value` (repeatable) places only the rows that match when the fill starts, and `--slice` only the rows of a slice, waiting while it is still freezing. Rows uploaded during a fill wait for the next one. The command waits up to 60 seconds for the rows to be placed (`--timeout <ms>` sets the wait), prints the column's parts and the rows each holds, and exits with the error when the fill fails; when the wait runs out the fill keeps running. `--no-wait` returns at once, and `dataset columns` shows the progress. A fill is refused while the same column is still filling.
+`dataset upload` accepts `.json` arrays, `.jsonl` files with one object per line, and `.csv` files with a header row. The CLI infers the format from the extension and uploads the original bytes without conversion. If the blob transfer or finalization fails, the CLI cancels the started upload before returning the error, so another upload can start immediately and may reuse the same filename. Field schemas are immutable after the first successful upload: you can add columns, never alter or remove one. When the dataset has no fields besides `uploaded` yet, the CLI prints the `dataset fields add` command that adds the uploaded fields as dataset fields; when it has fields copied from the uploaded rows, it waits until the new rows' values are copied too. See [datasets](./datasets.md).
+
+`dataset split` fills a split column: every row that has no value in it gets a part, sampled by share, and rows that already have a part keep it, so a test row stays a test row as the dataset grows. `--column` names the column (default `split`). A new name creates it; the name of a split column fills its empty cells with the shares, balancing column and seed of its last fill; the name of an ordinary column whose values name the parts makes it a split column without placing anything, and naming it again changes nothing; since nothing is placed, `--seed`, `--balance-by`, `--filter` and `--slice` are refused there unless `--parts` gives shares. A name that is a field of the uploaded rows but not a dataset field is refused with the `dataset fields add` command that adds it, so a Hugging Face `split` field becomes a split column with `toloka dataset fields add <name> split` and then `toloka dataset split <name> --column split`. `--ratios` takes whole percentages for `train`, `validate` and `test` that add up to 100 (default `80/10/10` for a new column); a 0% part is left out. `--parts` takes `name[=roles][:percent]` entries with roles `train`, `validate`, `test`, `validate+test` or `none`, for example `train:70,dev=validate:10,holdout=test:20`; a part named after a role takes it without `=roles`. To make an ordinary column a split column, `--parts` without percentages names the parts its values hold; without `--parts`, roles follow the values `train`, `validation`, `validate`, `val`, `dev` and `test`, and every value must be one of them. A part keeps its roles across fills, and the column keeps the shares of its last fill. On a split column that has shares, `--parts` without percentages is refused; omit `--parts` to repeat the last shares. `--validate-on-test` makes the test part the validation set too, the usual split when data is scarce; it needs a validate ratio of 0, for example `--ratios 80/0/20 --validate-on-test`, and the test score is then not fully held out. `--balance-by` shares out the rows of every label value on their own, so each part gets about the same mix of labels, though a label with only a few rows can still miss a part; a column with more than 1,000 distinct values fails the fill. `--seed` fixes the row order: rows are ranked by a hash of the seed and each row's position in the dataset, so the same seed puts the same rows, uploaded in the same order, in the same parts in any dataset. It is generated when omitted and kept by the column. Shares carry over: rows that already have a part count toward it, so the column keeps its proportions as rows are added. `--filter field=value` (repeatable) places only the rows that match when the fill starts, and `--slice` only the rows of a slice, waiting while it is still freezing. Rows uploaded during a fill wait for the next one. The command waits up to 60 seconds for the rows to be placed (`--timeout <ms>` sets the wait), prints the column's parts and the rows each holds, and exits with the error when the fill fails; when the wait runs out the fill keeps running. `--no-wait` returns at once, and `dataset columns` shows the progress. A fill is refused while the same column is still filling, or while a field it reads, such as the `--balance-by` field, is still being copied from the uploaded rows or filled by another split; without `--no-wait` the command first waits until no field is being copied and no split column is filling.
 
 A dataset can have several split columns. A row that any of them puts in a test part never reaches training, nor validation unless it sits in a validate + test part of the Training's own split column, so a column such as `--column holdout --parts holdout=test:10,rest=none:90` holds rows out of every Training. Name the column to train with through `training create --split-column`.
 
-`dataset slice create` freezes rows at a cutoff: the rows that match every `--filter field=value` (the top-level field equals the value; repeat the flag for more fields), or every row. The slice's id (`dsl.…`) always selects the same rows, and its frozen columns always read as they were just before the cutoff. `--name` tells slices apart (default `Slice 1`, `Slice 2` and so on). Rows uploaded later are never in it, and it is refused while an upload to the dataset is still running or a split column is still filling, so run it after `dataset upload` and `dataset split` finish. `--freeze` names the columns to freeze (default: every column); split columns are always frozen, so a slice records the parts as they were. The other columns read their latest values, and so do extra filters on a slice, such as field values or excluded parts. `--from <slice-id>` takes a ready slice again: its filters and, unless `--freeze` says otherwise, its frozen columns at a new cutoff, so rows added since join, rows that no longer match drop out, and frozen columns take their current values. `--from` refuses `--filter`. The command waits up to 60 seconds for the rows to be frozen (`--timeout <ms>` sets the wait; the slice keeps freezing when it runs out) and exits with the error when the slice fails, for example when no row matches; `--no-wait` returns at once, and `dataset slice show` reports the name, status, row count, filters and frozen columns. A slice that matches no row fails.
+`dataset slice create` freezes rows at a cutoff: the rows that match every `--filter field=value` (the top-level field equals the value; repeat the flag for more fields), or every row. The slice's id (`dsl.…`) always selects the same rows, and its frozen columns always read as they were just before the cutoff. `--name` tells slices apart (default `Slice 1`, `Slice 2` and so on). Rows uploaded later are never in it, and it is refused while an upload to the dataset is still running, a split column is still filling, or a field it freezes is still being copied from the uploaded rows; without `--no-wait` the command first waits until no field is being copied and no split column is filling. A dataset without rows has nothing to freeze, so the command refuses it. `--freeze` names the columns to freeze (default: every column); split columns are always frozen, so a slice records the parts as they were. The other columns read their latest values, and so do extra filters on a slice, such as field values or excluded parts. `--from <slice-id>` takes a ready slice again: its filters and, unless `--freeze` says otherwise, its frozen columns at a new cutoff, so rows added since join, rows that no longer match drop out, and frozen columns take their current values. `--from` refuses `--filter`. The command waits up to 60 seconds for the rows to be frozen (`--timeout <ms>` sets the wait; the slice keeps freezing when it runs out) and exits with the error when the slice fails, for example when no row matches; `--no-wait` returns at once, and `dataset slice show` reports the name, status, row count, filters and frozen columns. A slice that matches no row fails.
 
 `dataset slice list` prints the dataset's slices, newest first, with their names and the slice each one was taken from. Every slice stays readable by id.
 

package/docs/datasets.md

@@ -93,11 +93,14 @@ There is no flag to force an in-place change. The schema you declare on first ap
 toloka dataset list
 toloka dataset fields face-images
 toloka dataset load face-images --out data/face_images.jsonl
-toloka dataset load face-images --limit 100 --out data/face_images.sample.jsonl
+toloka dataset load face-images --fields image_url,faces --limit 100 --out data/face_images.sample.jsonl
 toloka dataset upload face-images data/face_images.jsonl
+toloka dataset fields add face-images --all
 ```
 
-`load` starts a platform export, waits (default 60s), downloads a JSON array, and writes JSONL to `--out`. `--limit` slices after that download. `--timeout` raises the wait. Failed downloads leave `--out` untouched.
+`load` starts a platform export, waits (default 60s), downloads a JSON array, and writes JSONL to `--out`: one object per row with every visible field, or only the fields `--fields` names. `uploaded` holds each uploaded row whole, as one value. `--limit` slices after that download. `--timeout` raises the wait. Failed downloads leave `--out` untouched.
+
+A dataset that no pipeline declares keeps each uploaded row whole in its `uploaded` field, so the rows' own fields are not dataset fields yet: `--filter`, `--balance-by` and `dataset split --column` need dataset fields. `toloka dataset fields add <dataset> <field>…` adds top-level fields of the uploaded rows as dataset fields, `--all` every one, and copies their values from the uploaded rows and from every later upload. A dataset field name takes lowercase letters, numbers and underscores, starts with a letter, ends with a letter or number and has at most 128 characters; `--all` skips fields whose names break the rule and says which. A slice or a split waits while a field it reads is still being copied.
 
 ## Load rows
 
@@ -157,9 +160,9 @@ toloka dataset slice create banking77 --name "intent eval" --freeze text,intent
 
 `dataset split` fills a split column, `split` unless `--column` names another: every row without a part gets one, sampled by share. Each part has roles. Trainings learn on `train` parts and validate on `validate` parts; `test` parts are held out: no training learns on a row that any split column puts in a test part, and none validates on one unless it sits in a validate + test part of the training's own split column. A part with no roles is for anything else. A part can both validate and test, the usual split when data is scarce (`--ratios 80/0/20 --validate-on-test`); its test score is then not fully held out. A train part has no other role. `--ratios` covers `train`, `validate` and `test`; `--parts train:70,calibration=none:10,validate:10,test:10` names other parts. `toloka dataset columns banking77` shows the split columns with their parts and roles.
 
-A split only fills empty cells. After an upload, run it again: the new rows join the parts so that the column keeps its shares, balanced by the same column with the same seed, and every row that already had a part keeps it, so no test row moves into training. Rows without a part are never trained on. To adopt a column that already names each row's part, such as a top-level Hugging Face `split` column, pass its name: `toloka dataset split banking77 --column split`. Roles follow the values `train`, `validation`, `validate`, `val`, `dev` and `test`; `--parts train,dev=validate,test` maps other names.
+A split only fills empty cells. After an upload, run it again: the new rows join the parts so that the column keeps its shares, balanced by the same column with the same seed, and every row that already had a part keeps it, so no test row moves into training. Rows without a part are never trained on. To adopt a column that already names each row's part, such as a top-level Hugging Face `split` column, pass its name: `toloka dataset split banking77 --column split`. When `split` is a field of the uploaded rows rather than a dataset field, add it first with `toloka dataset fields add banking77 split`; `dataset split` refuses a field of the uploaded rows that is not a dataset field, instead of sampling a new column under that name. The seed ranks rows by their position in the dataset, so the same file uploaded to two datasets splits the same way with the same seed. Roles follow the values `train`, `validation`, `validate`, `val`, `dev` and `test`; `--parts train,dev=validate,test` maps other names.
 
-A slice takes the rows that match its filters (`--filter intent=card_arrival`, repeated for more fields), or every row, at a cutoff. Its id (`dsl.…`) always selects the same rows, and it reads its frozen columns as they were just before the cutoff, so those values never change either. By default every column is frozen; `--freeze text,intent` picks the columns instead, split columns are always added, and the others read their latest values. Rows uploaded later are never in a slice, and a slice cannot be taken while an upload to the dataset is still running or a split column is still filling. Reading through a slice, the rows and its frozen columns, split parts included, are as of the slice cutoff; extra filters you add, such as field values or excluded parts, match the latest values. `--from <slice-id>` takes a slice again at a new cutoff: rows added since join and rows that no longer match drop out.
+A slice takes the rows that match its filters (`--filter intent=card_arrival`, repeated for more fields), or every row, at a cutoff. Its id (`dsl.…`) always selects the same rows, and it reads its frozen columns as they were just before the cutoff, so those values never change either. By default every column is frozen; `--freeze text,intent` picks the columns instead, split columns are always added, and the others read their latest values. Rows uploaded later are never in a slice, and a slice cannot be taken while an upload to the dataset is still running, a split column is still filling, or a column it freezes is still being copied from the uploaded rows. Reading through a slice, the rows and its frozen columns, split parts included, are as of the slice cutoff; extra filters you add, such as field values or excluded parts, match the latest values. `--from <slice-id>` takes a slice again at a new cutoff: rows added since join and rows that no longer match drop out.
 
 A Training learns on a slice that freezes a split column: pass `--slice <id>` (`sliceId` in the API), or `--dataset <name>` to split the dataset and freeze a slice named after the Training in one step. It learns on the train parts and validates on the validate parts of that column; `--split-column` (`splitFieldId`) picks the column when the slice freezes several, and `--parts` (`trainParts`) picks among the train parts.
 
@@ -181,7 +184,16 @@ const slice = await client.datasets.createSlice({ datasetId, name: "intent eval"
 const ready = await client.waitFor({ type: "datasetSlice", sliceId: slice.id });
 ```
 
-`split` returns the column, a dataset field whose `split` has the parts, the recipe and a `status` of `filling`, `ready` or `failed`. `waitFor({ type: "datasetSplit", datasetId, column })` resolves once the column stops filling, with `split.lastFill` counting the rows it placed, the rows that already had a part, and the rows per part. Pass `slice: sliceId` or an `itemFilter` to place only those rows. To adopt a column, pass its name and parts without shares: `{ datasetId, column: "split", parts: [{ name: "train", roles: ["train"] }, { name: "validation", roles: ["validate"] }, { name: "test", roles: ["test"] }] }`, or omit `parts` to infer the roles from the values. To take a slice again, pass `{ datasetId, fromSlice: sliceId }` to `createSlice`.
+`split` returns the column, a dataset field whose `split` has the parts, the recipe and a `status` of `filling`, `ready` or `failed`. `waitFor({ type: "datasetSplit", datasetId, column })` resolves once the column stops filling, with `split.lastFill` counting the rows it placed, the rows that already had a part, and the rows per part. Pass `slice: sliceId` or an `itemFilter` to place only those rows. To adopt a column, pass its name and parts without shares: `{ datasetId, column: "split", parts: [{ name: "train", roles: ["train"] }, { name: "validation", roles: ["validate"] }, { name: "test", roles: ["test"] }] }`, or omit `parts` to infer the roles from the values. An adoption places no rows, so it refuses `seed`, `balanceBy`, `slice` and `itemFilter`. To take a slice again, pass `{ datasetId, fromSlice: sliceId }` to `createSlice`.
+
+To add fields of the uploaded rows as dataset fields from code, and wait until their values are copied:
+
+```ts
+const created = await client.datasets.createFieldsFromUploadedRows({ datasetId, names: ["intent", "split"] });
+const fields = await client.waitFor({ type: "datasetFields", datasetId, names: ["intent", "split"] });
+```
+
+`createFieldsFromUploadedRows` adds a dataset field for each named top-level field of the uploaded rows, with the schema inferred from them, and returns the new fields, each with a `copy` operation in `operations` while the values are copied in the background. `waitFor({ type: "datasetFields", datasetId, names })` polls the dataset's fields until none of the named fields, or none at all when `names` is omitted, has a blocking operation (`copy` or `split_fill`), and resolves with every field. Like the other waits, it gives up after 60 seconds unless `timeoutMs` says otherwise and throws a `PotatoApiError` with code `wait_timeout`; the copy keeps running. `split` and `createSlice` are refused with a conflict (HTTP 409) while a field they read is still being copied.
 
 Pipeline runs accept `slice: { sliceId }`, `parts: { fieldId, names: ["test"] }` and `excludeParts: [{ fieldId, names }]` in their row filter. `parts` reads the column as it was at the slice's cutoff when the same filter selects a slice that freezes it. A run that selects through a slice reads the slice's frozen columns as they were at the cutoff and every other column as usual.
 

package/openapi/gen/api.ts

@@ -11,6 +11,7 @@ import type {
   CreateDatasetExportBody,
   CreateDatasetField200,
   CreateDatasetFieldBody,
+  CreateDatasetFieldsFromUploadedRowsBody,
   CreateDatasetSliceBody,
   CreateDatasetUpload200,
   CreateDatasetUploadBody,
@@ -22,6 +23,7 @@ import type {
   CreateProjectTrainingBody,
   CreateScopedTokenBody,
   CreateTrainingRunBody,
+  CreatedDatasetFields,
   CurrentUser,
   Dataset,
   DatasetField,
@@ -1658,6 +1660,29 @@ export const getDatasetExport = async (exportId: ExportId, options?: RequestInit
 
 
 
+export const getCreateDatasetFieldsFromUploadedRowsUrl = (datasetId: DatasetId,) => {
+
+
+  
+
+  return `/api/v2-beta/datasets/${datasetId}/fields/from-uploaded`
+}
+
+export const createDatasetFieldsFromUploadedRows = async (datasetId: DatasetId,
+    createDatasetFieldsFromUploadedRowsBody: CreateDatasetFieldsFromUploadedRowsBody, options?: RequestInit): Promise<CreatedDatasetFields> => {
+  
+  return potatoFetch<CreatedDatasetFields>(getCreateDatasetFieldsFromUploadedRowsUrl(datasetId),
+  {      
+    ...options,
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json', ...options?.headers },
+    body: JSON.stringify(
+      createDatasetFieldsFromUploadedRowsBody,)
+  }
+);}
+
+
+
 export const getGetPipelineGraphSnapshotUrl = (pipelineId: PipelineId,) => {
 
 

package/openapi/gen/models/createDatasetFieldsFromUploadedRowsBody.ts

@@ -0,0 +1,4 @@
+
+export type CreateDatasetFieldsFromUploadedRowsBody = {
+  names: string[];
+};

package/openapi/gen/models/datasetExportReportTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type DatasetExportReportTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type DatasetExportReportTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/datasetField.ts

@@ -2,6 +2,7 @@ import type { DatasetFieldId } from './datasetFieldId';
 import type { DatasetFieldDescription } from './datasetFieldDescription';
 import type { DatasetFieldJsonSchema } from './datasetFieldJsonSchema';
 import type { DatasetFieldSplit } from './datasetFieldSplit';
+import type { DatasetFieldOperation } from './datasetFieldOperation';
 
 export interface DatasetField {
   id: DatasetFieldId;
@@ -10,5 +11,6 @@ export interface DatasetField {
   jsonSchema: DatasetFieldJsonSchema;
   internal: boolean;
   split: DatasetFieldSplit;
+  operations: DatasetFieldOperation[];
   createdAt: string;
 }

package/openapi/gen/models/datasetFieldCopyOperation.ts

@@ -0,0 +1,7 @@
+import type { DatasetFieldId } from './datasetFieldId';
+
+export interface DatasetFieldCopyOperation {
+  type: 'copy';
+  sourceFieldId: DatasetFieldId;
+  blocking: boolean;
+}

package/openapi/gen/models/datasetFieldOperation.ts

@@ -0,0 +1,4 @@
+import type { DatasetFieldCopyOperation } from './datasetFieldCopyOperation';
+import type { DatasetFieldSplitFillOperation } from './datasetFieldSplitFillOperation';
+
+export type DatasetFieldOperation = DatasetFieldCopyOperation | DatasetFieldSplitFillOperation;

package/openapi/gen/models/datasetFieldSplitFillOperation.ts

@@ -0,0 +1,5 @@
+
+export interface DatasetFieldSplitFillOperation {
+  type: 'split_fill';
+  blocking: boolean;
+}

package/openapi/gen/models/index.ts

@@ -95,6 +95,7 @@ export * from './createDatasetFieldBodyAnyOfSixFieldsItemJsonSchema';
 export * from './createDatasetFieldBodyAnyOfSixFieldsItemJsonSchemaAdditionalProperties';
 export * from './createDatasetFieldBodyAnyOfSixFieldsItemJsonSchemaType';
 export * from './createDatasetFieldBodyAnyOfSixFieldsItemJsonSchemaTypeAnyOfItem';
+export * from './createDatasetFieldsFromUploadedRowsBody';
 export * from './createDatasetSliceBody';
 export * from './createDatasetSliceBodyAnyOf';
 export * from './createDatasetSliceBodyAnyOfItemFilter';
@@ -141,12 +142,14 @@ export * from './datasetExportReportTokens';
 export * from './datasetExportReportTokensEstimatorVersion';
 export * from './datasetExportReportTokensTotal';
 export * from './datasetField';
+export * from './datasetFieldCopyOperation';
 export * from './datasetFieldDescription';
 export * from './datasetFieldId';
 export * from './datasetFieldJsonSchema';
 export * from './datasetFieldJsonSchemaAdditionalProperties';
 export * from './datasetFieldJsonSchemaType';
 export * from './datasetFieldJsonSchemaTypeAnyOfItem';
+export * from './datasetFieldOperation';
 export * from './datasetFieldSplit';
 export * from './datasetFieldSplitAnyOf';
 export * from './datasetFieldSplitAnyOfError';
@@ -164,6 +167,7 @@ export * from './datasetFieldSplitAnyOfRecipeAnyOf';
 export * from './datasetFieldSplitAnyOfRecipeAnyOfBalanceByFieldId';
 export * from './datasetFieldSplitAnyOfRecipeAnyOfSharesItem';
 export * from './datasetFieldSplitAnyOfStatus';
+export * from './datasetFieldSplitFillOperation';
 export * from './datasetId';
 export * from './datasetItem';
 export * from './datasetItemCurrentSubmissionId';

package/openapi/gen/models/trainingCostEstimateOneOfInputsTrainingReportAnyOfTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingCostEstimateOneOfInputsTrainingReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingCostEstimateOneOfInputsTrainingReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/trainingCostEstimateOneOfInputsValidationAnyOfReportAnyOfTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingCostEstimateOneOfInputsValidationAnyOfReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingCostEstimateOneOfInputsValidationAnyOfReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/trainingCostEstimateOneOfTwosevenInputsTrainingReportAnyOfTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingCostEstimateOneOfTwosevenInputsTrainingReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingCostEstimateOneOfTwosevenInputsTrainingReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/trainingCostEstimateOneOfTwosevenInputsValidationAnyOfReportAnyOfTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingCostEstimateOneOfTwosevenInputsValidationAnyOfReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingCostEstimateOneOfTwosevenInputsValidationAnyOfReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/trainingRunInputsAnyOfTrainingReportTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingRunInputsAnyOfTrainingReportTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingRunInputsAnyOfTrainingReportTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/trainingRunInputsAnyOfValidationAnyOfReportTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingRunInputsAnyOfValidationAnyOfReportTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingRunInputsAnyOfValidationAnyOfReportTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/trainingRunQuoteInputsTrainingReportAnyOfTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingRunQuoteInputsTrainingReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingRunQuoteInputsTrainingReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/openapi/gen/models/trainingRunQuoteInputsValidationAnyOfReportAnyOfTokensEstimatorVersion.ts

@@ -1,2 +1,2 @@
 
-export type TrainingRunQuoteInputsValidationAnyOfReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | null;
+export type TrainingRunQuoteInputsValidationAnyOfReportAnyOfTokensEstimatorVersion = 'utf16-code-units-div-4-v1' | 'script-weighted-v1' | null;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@toloka-platform/sdk",
-  "version": "0.12.0",
+  "version": "0.12.2",
   "private": false,
   "description": "Build, inspect, and operate Toloka pipeline projects as TypeScript source. The public API may change between minor versions before 1.0.",
   "license": "Apache-2.0",
@@ -77,6 +77,6 @@
     "bun": ">=1.4.0"
   },
   "tolokaBuild": {
-    "commit": "cb78236aa061ea2ad293092b1059d5f93244fcdf"
+    "commit": "72b9ea77f616c7f9c3c564c550eac42f703929b2"
   }
 }

package/skills/catalog.json

@@ -11,8 +11,8 @@
         "testing",
         "checks"
       ],
-      "version": "0.12.0",
-      "digest": "c7a14e0a975a69a93e0186e26bf69b836c9c877c15546b08521b728eeade6a2d",
+      "version": "0.12.2",
+      "digest": "7e6b79b9b85fbfc86cf98f33ebd30fdb4617707d22f7c24440bee2fedecb5f71",
       "files": [
         "SKILL.md",
         "references/human-work.md",
@@ -33,7 +33,7 @@
         "bug",
         "report"
       ],
-      "version": "0.12.0",
+      "version": "0.12.2",
       "digest": "70f0bd2c80b32f87896aeb27c363b8a2d37f40e22424f57fe1aad142e125ad67",
       "files": [
         "SKILL.md",
@@ -50,7 +50,7 @@
         "getting-started",
         "agent"
       ],
-      "version": "0.12.0",
+      "version": "0.12.2",
       "digest": "2278adeb46a792356579395fcf73c42322cb23bcd0c61e20856ccabdad6762d1",
       "files": [
         "SKILL.md"
@@ -67,8 +67,8 @@
         "sft",
         "models"
       ],
-      "version": "0.12.0",
-      "digest": "def28cd22976477fff51ca99b12090b245621659c616ab9e67ba80b742fd4bfd",
+      "version": "0.12.2",
+      "digest": "3ce10dee5d3c283d8b5fdea15134a58fc738c7af2c1bb19eb5a0bf70ea0bca13",
       "files": [
         "SKILL.md"
       ]
@@ -84,8 +84,8 @@
         "prompt",
         "models"
       ],
-      "version": "0.12.0",
-      "digest": "4644cc1d354d8e04cb41798bdee46dfeb81cd109221971328e926cdca7b8fe9b",
+      "version": "0.12.2",
+      "digest": "be8bce8d9cf7241ccb60d2b2a9b7cc07d7755a76e8ff8ec1b0847c49efbb3469",
       "files": [
         "SKILL.md"
       ]

package/skills/toloka-build/references/workflow-and-cli.md

@@ -34,7 +34,7 @@ Run `toloka --help` and `toloka <command> --help` for the exact commands and opt
 - projects and tokens: `project` (`list`, `add`, `switch`, `current`, `url`, `remove`, `create`, `untrack`, `token`, `tokens`, `revoke-token`, `revoke-tokens`), `delivery-order list`;
 - source reconciliation: `init`, `import`, `check`, `validate`, `plan`, `apply`, `status`, `secret set`;
 - catalogs: `audience specialization list`, `generation-model list`, `ui components`;
-- data and runs: `dataset` (`list`, `create`, `fields`, `columns`, `load`, `upload`, `split`, `slice`), `pipeline` (`list`, `run`, `status`, `runs`, `cost`, `results`, `cancel`);
+- data and runs: `dataset` (`list`, `create`, `fields`, `fields add`, `columns`, `load`, `upload`, `split`, `slice`), `pipeline` (`list`, `run`, `status`, `runs`, `cost`, `results`, `cancel`);
 - models and training: `base-model`, `model` (including `model version download`), `training`;
 - skills and reports: `skill` (`list`, `search`, `install`), `report`.
 
@@ -54,7 +54,7 @@ The global Base Model Catalog is read-only and uses `toloka base-model`. `base-m
 
 Hyperparameters are optional flags on `training create` and `training update`; the `toloka-fine-tuning` and `toloka-gisting` skills list their ranges and defaults. `--lora-rank` is SFT-only and `--gist-tokens` gisting-only: passing one to the other method fails before any request. `training update` has no `--method` flag: the method stays as created, and every flag you omit keeps its current value. `training cost` is blocked when the longest example exceeds `--max-sequence-length`, or when the training selection leaves fewer rows than `--batch-size` after the 10% holdout (taken only when there is no validation selection).
 
-`--dataset <name-or-id>` fills the empty cells of the Dataset's split column (a new `split` column, 80/10/10, when it has none), freezes a slice named after the Training, and trains on it. A split column adopted from an ordinary column has no shares, so `--dataset` does not fill it; give it shares first with `toloka dataset split <dataset> --column <name> --parts <name>=<roles>:<percent>,...`. `--slice <id>` takes a ready slice frozen with `toloka dataset slice create`. The Training learns on the split column's train parts (`--parts` picks some of them) and validates on its validate parts, including a part that also tests, or takes the 10% holdout when they are empty; `--split-column` picks the column when there are several. Rows without a part are never trained on. Rows that any split column puts in a test part stay out of training, and out of validation unless they sit in a validate + test part of the Training's own split column: `training cost` leaves them out and reports them as held-out test rows. `training update` keeps the slice, split column and train parts; a new `--dataset` or `--slice` resets the split column and train parts unless you pass them too. `training get` shows them in `configuration.sliceId`, `configuration.splitFieldId` and `configuration.trainParts`.
+`--dataset <name-or-id>` fills the empty cells of the Dataset's split column (a new `split` column, 80/10/10, when it has none), freezes a slice named after the Training, and trains on it. A split column adopted from an ordinary column has no shares, so `--dataset` does not fill it; give it shares first with `toloka dataset split <dataset> --column <name> --parts <name>=<roles>:<percent>,...`. When the uploaded rows have a top-level `split` field that is not a dataset field yet, `--dataset` refuses and prints the `toloka dataset fields add` command that adds it. `--slice <id>` takes a ready slice frozen with `toloka dataset slice create`. The Training learns on the split column's train parts (`--parts` picks some of them) and validates on its validate parts, including a part that also tests, or takes the 10% holdout when they are empty; `--split-column` picks the column when there are several. Rows without a part are never trained on. Rows that any split column puts in a test part stay out of training, and out of validation unless they sit in a validate + test part of the Training's own split column: `training cost` leaves them out and, when there are any, reports how many as `Left out as test`. `training update` keeps the slice, split column and train parts; a new `--dataset` or `--slice` resets the split column and train parts unless you pass them too. `training get` shows them in `configuration.sliceId`, `configuration.splitFieldId` and `configuration.trainParts`.
 
 Use `training cost` to validate inputs and show the maximum charge before `training run`. A blocked estimate exits 1 and lists the issues on stderr instead of printing the estimate; fix them and recalculate. `training run` accepts that ready estimate JSON through `--estimate-file`; without it, the command calculates a fresh ready estimate itself before creating the run. `--version-name` defaults to the Training name; pass it when a Model Version with that name already exists. Use `training status`, `training logs`, `training metrics`, `training resume`, and `training cancel` with the returned Training Run ID. `training metrics` lists each metric's latest value while the run is still training; `--key` prints one metric's point history instead. `training cancel` before training starts is free; on a run that is already training it stops the GPU job, and the run is charged only for the GPU seconds used until then and produces no Model Version.
 
@@ -119,9 +119,18 @@ toloka dataset fields <dataset>
 
 That prints the same inferred types as `field()` declarations for the top-level columns. Use them to recover stored names and base types, never to replace stricter source schemas. The command does not write source or mutate the dataset.
 
+A dataset that no pipeline declares keeps each uploaded row whole in its `uploaded` field, so the rows' own fields are not dataset fields yet, and `--filter`, `--balance-by` and `dataset split --column` need dataset fields. Add top-level fields of the uploaded rows as dataset fields:
+
+```sh
+toloka dataset fields add <dataset> <field>…
+toloka dataset fields add <dataset> --all
+```
+
+Each new dataset field takes the type inferred from the rows, and its values are copied from the uploaded rows and from every later upload. `--all` takes every top-level field that is not a dataset field yet. A dataset field name takes lowercase letters, numbers and underscores, starts with a letter, ends with a letter or number and has at most 128 characters: a named field that breaks the rule is refused, and `--all` skips such fields and says which. The command waits up to 60 seconds for the values to be copied (`--timeout <ms>` sets the wait, `--no-wait` returns at once); until the copy finishes, `toloka dataset columns <dataset> --json` shows a `copy` operation in the field's `operations`, and a split or slice that reads it waits for it.
+
 Keep item files outside the source tree or gitignored. Never embed sample or production rows in `pipeline.ts` to make reconciliation create them.
 
-Pull live rows with `toloka dataset load <name> --out <file> [--limit <n>]`. `--limit` slices after the export downloads; omit it only for the whole table.
+Pull live rows with `toloka dataset load <name> --out <file> [--fields <fields>] [--limit <n>]`. Each JSONL line is one row with `index` and every visible field, or only the fields `--fields` names, for example `--fields uploaded,split`; `uploaded` holds each uploaded row whole, as one nested value. `--limit` slices after the export downloads; omit it only for the whole table.
 
 ## Split a dataset and freeze a slice
 
@@ -132,11 +141,11 @@ toloka dataset split <dataset> --ratios 80/10/10 --balance-by <label-field>
 toloka dataset slice create <dataset> --name "<slice-name>"
 ```
 
-`dataset split` places every row that has no part yet and never moves a row that has one; `--filter field=value` or `--slice <slice-id>` limits it to those rows. It waits up to 60 seconds for the rows to be placed (`--timeout <ms>` sets the wait, `--no-wait` skips it); when the wait runs out the fill keeps running, and `toloka dataset columns <dataset>` shows its progress. `--ratios` takes whole percentages that add up to 100 (default `80/10/10` for a new column). `--balance-by` shares out each label's rows on their own; a label with only a few rows can still miss a part. `--seed` makes the placement repeatable, and the column keeps it. When data is scarce, `--ratios 80/0/20 --validate-on-test` makes the test part the validation set too; say that its test score is then not fully held out. `--parts name[=roles][:percent],…` names other parts, for example `train:70,dev=validate:10,holdout=test:20`. `--column <name>` picks the column (default `split`); several split columns can live on one Dataset. When the rows already carry a top-level column that names their part, `--column <field>` adopts it: roles follow the values `train`, `validation`/`validate`/`val`/`dev` and `test`, and `--parts train,dev=validate,test` maps other names. Check `toloka dataset columns <dataset>` first: a name that is not a column yet creates a new, sampled split column.
+`dataset split` places every row that has no part yet and never moves a row that has one; `--filter field=value` or `--slice <slice-id>` limits it to those rows. It waits up to 60 seconds for the rows to be placed (`--timeout <ms>` sets the wait, `--no-wait` skips it); when the wait runs out the fill keeps running, and `toloka dataset columns <dataset>` shows its progress. `--ratios` takes whole percentages that add up to 100 (default `80/10/10` for a new column). `--balance-by` shares out each label's rows on their own; a label with only a few rows can still miss a part. `--seed` makes the placement repeatable, and the column keeps it: rows are ranked by the seed and their position in the dataset, so the same file uploaded in the same order splits the same way. When data is scarce, `--ratios 80/0/20 --validate-on-test` makes the test part the validation set too; say that its test score is then not fully held out. `--parts name[=roles][:percent],…` names other parts, for example `train:70,dev=validate:10,holdout=test:20`. On a split column that already has shares, `--parts` without percentages is refused: give the percentages, adding up to 100, or omit `--parts` to repeat the column's last shares. `--column <name>` picks the column (default `split`); several split columns can live on one Dataset. When a column already names each row's part, `--column <column>` adopts it: roles follow the values `train`, `validation`/`validate`/`val`/`dev` and `test`, and `--parts train,dev=validate,test` maps other names; naming it again changes nothing. Adoption places no rows, so `--seed`, `--balance-by`, `--filter` and `--slice` are refused with it. When the part is a field of the uploaded rows, such as a Hugging Face `split` field, add it as a dataset field first with `toloka dataset fields add <dataset> split`: `dataset split` refuses a name that is a field of the uploaded rows but not a dataset field, and prints that command. Any other new name creates a new, sampled split column.
 
 When rows are added later, run `dataset split` again: it places only the new rows, with the column's last shares, balancing column and seed, so the column keeps its proportions and no test row moves into training. Then take a new slice, for example with `dataset slice create <dataset> --from <slice-id>`, which reruns the slice's filters at a new cutoff with the same frozen columns.
 
-`dataset slice create` freezes every row, or only the rows that match each `--filter field=value`. `--freeze <columns>` picks the frozen columns (default: every column); split columns are always frozen. `--name` tells slices apart (default `Slice 1`, `Slice 2` and so on). Rows uploaded later are never in the slice, and `slice create` is refused while an upload is running or a split column is filling, so run it after `dataset upload` and `dataset split` finish. Both commands wait until the rows are placed or frozen and exit with the error when the fill or slice fails; `--no-wait` returns at once, so check `dataset columns` or `dataset slice show`.
+`dataset slice create` freezes every row, or only the rows that match each `--filter field=value`. `--freeze <columns>` picks the frozen columns (default: every column); split columns are always frozen. `--name` tells slices apart (default `Slice 1`, `Slice 2` and so on). Rows uploaded later are never in the slice, and `slice create` is refused while an upload is running, a split column is filling, or a field it freezes is still being copied from the uploaded rows, so run it after `dataset upload` and `dataset split` finish; a dataset without rows has nothing to freeze, so it is refused too. Both commands first wait for fields still being copied, and both wait until the rows are placed or frozen and exit with the error when the fill or slice fails; `--no-wait` returns at once, so check `dataset columns` or `dataset slice show`.
 
 ## Run a pipeline and report results
 

package/skills/toloka-fine-tuning/SKILL.md

@@ -13,12 +13,12 @@ For first-time setup details, read `https://platform.toloka.ai/agents/fine-tunin
 
 ## Domain model
 
-- **SFT** adapts model weights from representative chat examples. Each training row must follow the platform chat contract and end with the assistant response the model should learn to produce.
+- **SFT** adapts model weights from representative chat examples. Each training row must follow the platform chat contract and end with the assistant response the model should learn to produce. Rows with tool calls also need `tools` declaring every function they call. Each call's `arguments` must be a string holding a JSON object (it is not checked against the function's `parameters`), call ids must be unique within a turn, and the `tool` replies must answer every call before the next turn. Cost estimation rejects rows that break these rules and names the first few in its error.
 - A **Training** is an editable project-owned draft. A **Training Run** is one immutable, billable execution of a frozen draft and the rows of its slice.
 - A run can start from a catalog **Base Model** or an exact existing **Model Version**. An explicit Model Version is the source; do not also select a Base Model.
 - A completed run publishes a durable Model Version. It does not prove deployment or inference readiness.
 - Training and validation are data roles, not file-layout rules. A **split column** puts each Dataset row in a part such as `train`, `validate` or `test`, and a **slice** freezes rows together with their split columns. A Training learns on a slice's rows in the train parts of one split column and validates on its validate parts, which may include a part that also tests. Without validation rows, the provider reserves 10% of the training rows as a holdout.
-- A Training needs `--dataset <dataset>` or `--slice <id>`. `--dataset` fills the empty cells of the Dataset's split column (a new `split` column, 80/10/10, when it has none) and freezes a slice named after the Training; `--slice` takes a slice frozen with `toloka dataset slice create`, which must freeze a split column. Rows without a part are never trained on. Rows that any split column puts in a test part never reach training, nor validation unless they sit in a validate + test part of the Training's own split column; the cost estimate reports them as held-out test rows. `toloka dataset split <dataset>` places only rows without a part, so after adding rows, run it again and take a new slice: no test row moves into training. `toloka dataset columns <dataset>` shows the split columns and their roles.
+- A Training needs `--dataset <dataset>` or `--slice <id>`. `--dataset` fills the empty cells of the Dataset's split column (a new `split` column, 80/10/10, when it has none) and freezes a slice named after the Training; `--slice` takes a slice frozen with `toloka dataset slice create`, which must freeze a split column. Rows without a part are never trained on. Rows that any split column puts in a test part never reach training, nor validation unless they sit in a validate + test part of the Training's own split column; the cost estimate reports any train rows they remove as `Left out as test`. `toloka dataset split <dataset>` places only rows without a part, so after adding rows, run it again and take a new slice: no test row moves into training. `toloka dataset columns <dataset>` shows the split columns and their roles.
 
 ## Working posture
 
@@ -69,7 +69,7 @@ toloka dataset fields <dataset-name>
 
 When the user supplied separate training and validation files, give each file its role before uploading the next one: `toloka dataset split <dataset-name> --parts train:100 --json` after the training file, then `toloka dataset split <dataset-name> --parts validate:100 --json` after the validation file. Each fill places only rows without a part, so a supplied validation set never mixes into training; pass `--parts` on every such fill, because a fill without it repeats the column's last shares. For one undivided file, let the split sample it: `toloka dataset split <dataset-name> --ratios 80/10/10 [--balance-by <label-field>] --json`, or leave it to `training create --dataset`, which splits 80/10/10.
 
-CLI uploads to an otherwise undeclared Dataset store each row whole. A nested `split` value in that uploaded object is not a top-level Dataset field, so it cannot name parts; use the fills above. An existing top-level column whose values name the parts becomes a split column with `toloka dataset split <dataset> --column <field>`: roles follow the values `train`, `validation`/`validate`/`val`/`dev` and `test`, and `--parts train,dev=validate,test` maps other names. Check `toloka dataset columns <dataset>` first, because a name that is not a column yet creates a new, sampled split column.
+CLI uploads to an otherwise undeclared Dataset store each row whole in its `uploaded` field, so the rows' own fields are not dataset fields yet. `toloka dataset fields add <dataset> <field>…` adds top-level fields of the uploaded rows as dataset fields, for filters and `--balance-by`, and copies their values from every upload. When the rows carry their own role, such as a Hugging Face `split` field, add it, then adopt it: `toloka dataset fields add <dataset> split`, then `toloka dataset split <dataset> --column split`. Roles follow the values `train`, `validation`/`validate`/`val`/`dev` and `test`, and `--parts train,dev=validate,test` maps other names. `dataset split` and `training create --dataset` refuse a split column name that is a field of the uploaded rows but not a dataset field, and print the `fields add` command.
 
 To train on a subset or reuse rows across Trainings, freeze a slice once every upload and split has finished; `--filter <field>=<value>` keeps only matching rows, and split columns are always frozen:
 
@@ -129,7 +129,7 @@ A blocked estimate exits 1, leaves the redirected file empty, and prints `Traini
 
 For a duration issue, use the reported rows, tokens, epochs, GPU count, and wall time to compare supported options. `qwen3-32b` and `qwen3-235b` use 8 GPUs and can reduce wall time, but changing the model also changes quality, serving constraints, and cost; reducing rows can weaken coverage; reducing epochs can weaken adaptation. Do not change the requested model, silently discard data, or alter epochs merely to make the estimate pass. Present the tradeoff, obtain direction when it is material, then update the draft, re-estimate, and obtain approval for the new quote.
 
-For a ready estimate, report the source, slice and split column, training and validation row counts, held-out test rows, token totals, maximum GPU seconds, `maximumCharge`, currency, and fingerprint. If the user asked only for an estimate, stop here.
+For a ready estimate, report the source, slice and split column, training and validation row counts, any rows left out as test, token totals, maximum GPU seconds, `maximumCharge`, currency, and fingerprint. If the user asked only for an estimate, stop here.
 
 ### 6. Run only after explicit cost approval
 
@@ -163,6 +163,7 @@ Control and billing rules:
 - `payment_window_expired` is final: the run is `failed`, cannot resume, and was not charged, even though `billingStatus` remains `payment_required`. A replacement run needs funds and new cost approval.
 - Cancel only on explicit request, before output checks begin. Before training starts, cancellation is free. During training, it stops the GPU job and charges for GPU seconds used, with no Model Version. A startup cancellation may remain `stopping` until the job stops; wait rather than retrying cancel.
 - `provider_failed` and `provider_timeout` are platform failures: no charge, full hold returned. Reported GPU usage remains visible.
+- A `failed` or `canceled` run frees its estimate: the same estimate file can start one new run at the same `maximumCharge` until the estimate expires. Retry only an error with `retryable: true` (network, storage, or GPU capacity), and only after the user agrees.
 
 Do not duplicate a slow run. Progress `queued` or `pending` means it is waiting for GPUs; training starts automatically. Training may also exceed the estimated duration. The charge stays capped at the approved `maximumCharge`; successful runs still publish a Model Version.
 

package/skills/toloka-gisting/SKILL.md

@@ -18,7 +18,7 @@ For first-time setup details, read `https://platform.toloka.ai/agents/gisting.md
 - A run can start from a catalog **Base Model** or an exact existing **Model Version**. An explicit Model Version is the source; do not also select a Base Model.
 - A completed run publishes a durable Model Version. It does not prove deployment or inference readiness.
 - Training and validation are data roles, not file-layout rules. A **split column** puts each Dataset row in a part such as `train`, `validate` or `test`, and a **slice** freezes rows together with their split columns. A Training learns on a slice's rows in the train parts of one split column and validates on its validate parts, which may include a part that also tests. Without validation rows, the provider reserves 10% of the training rows as a holdout.
-- A Training needs `--dataset <dataset>` or `--slice <id>`. `--dataset` fills the empty cells of the Dataset's split column (a new `split` column, 80/10/10, when it has none) and freezes a slice named after the Training; `--slice` takes a slice frozen with `toloka dataset slice create`, which must freeze a split column. Rows without a part are never trained on. Rows that any split column puts in a test part never reach training, nor validation unless they sit in a validate + test part of the Training's own split column; the cost estimate reports them as held-out test rows. `toloka dataset split <dataset>` places only rows without a part, so after adding rows, run it again and take a new slice: no test row moves into training. `toloka dataset columns <dataset>` shows the split columns and their roles.
+- A Training needs `--dataset <dataset>` or `--slice <id>`. `--dataset` fills the empty cells of the Dataset's split column (a new `split` column, 80/10/10, when it has none) and freezes a slice named after the Training; `--slice` takes a slice frozen with `toloka dataset slice create`, which must freeze a split column. Rows without a part are never trained on. Rows that any split column puts in a test part never reach training, nor validation unless they sit in a validate + test part of the Training's own split column; the cost estimate reports any train rows they remove as `Left out as test`. `toloka dataset split <dataset>` places only rows without a part, so after adding rows, run it again and take a new slice: no test row moves into training. `toloka dataset columns <dataset>` shows the split columns and their roles.
 
 ## Working posture
 
@@ -70,7 +70,7 @@ toloka dataset fields <dataset-name>
 
 When the user supplied separate training and validation files, give each file its role before uploading the next one: `toloka dataset split <dataset-name> --parts train:100 --json` after the training file, then `toloka dataset split <dataset-name> --parts validate:100 --json` after the validation file. Each fill places only rows without a part, so a supplied validation set never mixes into training; pass `--parts` on every such fill, because a fill without it repeats the column's last shares. For one undivided file, let the split sample it: `toloka dataset split <dataset-name> --ratios 80/10/10 [--balance-by <label-field>] --json`, or leave it to `training create --dataset`, which splits 80/10/10.
 
-CLI uploads to an otherwise undeclared Dataset store each row whole. A nested `split` value in that uploaded object is not a top-level Dataset field, so it cannot name parts; use the fills above. An existing top-level column whose values name the parts becomes a split column with `toloka dataset split <dataset> --column <field>`: roles follow the values `train`, `validation`/`validate`/`val`/`dev` and `test`, and `--parts train,dev=validate,test` maps other names. Check `toloka dataset columns <dataset>` first, because a name that is not a column yet creates a new, sampled split column.
+CLI uploads to an otherwise undeclared Dataset store each row whole in its `uploaded` field, so the rows' own fields are not dataset fields yet. `toloka dataset fields add <dataset> <field>…` adds top-level fields of the uploaded rows as dataset fields, for filters and `--balance-by`, and copies their values from every upload. When the rows carry their own role, such as a Hugging Face `split` field, add it, then adopt it: `toloka dataset fields add <dataset> split`, then `toloka dataset split <dataset> --column split`. Roles follow the values `train`, `validation`/`validate`/`val`/`dev` and `test`, and `--parts train,dev=validate,test` maps other names. `dataset split` and `training create --dataset` refuse a split column name that is a field of the uploaded rows but not a dataset field, and print the `fields add` command.
 
 To train on a subset or reuse rows across Trainings, freeze a slice once every upload and split has finished; `--filter <field>=<value>` keeps only matching rows, and split columns are always frozen:
 
@@ -78,7 +78,7 @@ To train on a subset or reuse rows across Trainings, freeze a slice once every u
 toloka dataset slice create <dataset-name> --name "<slice-name>" --json
 ```
 
-Check that the examples demonstrate the desired behavior under the prompt: dynamic input in user messages, target output in the final assistant message, representative edge cases, and no prompt or answer leakage across validation.
+Check that the examples demonstrate the desired behavior under the prompt: dynamic input in user messages, target output in the final assistant message, representative edge cases, and no prompt or answer leakage across validation. Rows with tool calls also need `tools` declaring every function they call. Each call's `arguments` must be a string holding a JSON object (it is not checked against the function's `parameters`), call ids must be unique within a turn, and the `tool` replies must answer every call before the next turn. Cost estimation rejects rows that break these rules and names the first few in its error.
 
 ### 3. Preserve the prompt and choose the source
 
@@ -136,7 +136,7 @@ A blocked estimate exits 1, leaves the redirected file empty, and prints `Traini
 
 For a duration issue, use the reported rows, tokens, epochs, GPU count, and wall time to compare supported options. Gisting publishes full model weights, so only `qwen3-4b` and `qwen3-8b` are supported; switching to the larger 8-GPU tiers is not a valid workaround. Reducing rows can weaken coverage and reducing epochs can weaken compression quality. Do not silently change the requested model, discard data, or alter epochs merely to make the estimate pass. Explain the tradeoff, obtain direction when material, update the draft, re-estimate, and obtain approval for the new quote.
 
-For a ready estimate, report the source, prompt, slice and split column, training and validation row counts, held-out test rows, token totals, maximum GPU seconds, `maximumCharge`, currency, and fingerprint. If the user asked only for an estimate, stop here.
+For a ready estimate, report the source, prompt, slice and split column, training and validation row counts, any rows left out as test, token totals, maximum GPU seconds, `maximumCharge`, currency, and fingerprint. If the user asked only for an estimate, stop here.
 
 ### 6. Run only after explicit cost approval
 
@@ -170,6 +170,7 @@ Control and billing rules:
 - `payment_window_expired` is final: the run is `failed`, cannot resume, and was not charged, even though `billingStatus` remains `payment_required`. A replacement run needs funds and new cost approval.
 - Cancel only on explicit request, before output checks begin. Before training starts, cancellation is free. During training, it stops the GPU job and charges for GPU seconds used, with no Model Version. A startup cancellation may remain `stopping` until the job stops; wait rather than retrying cancel.
 - `provider_failed` and `provider_timeout` are platform failures: no charge, full hold returned. Reported GPU usage remains visible.
+- A `failed` or `canceled` run frees its estimate: the same estimate file can start one new run at the same `maximumCharge` until the estimate expires. Retry only an error with `retryable: true` (network, storage, or GPU capacity), and only after the user agrees.
 
 Do not duplicate a slow run. Progress `queued` or `pending` means it is waiting for GPUs; training starts automatically. Training may also exceed the estimated duration. The charge stays capped at the approved `maximumCharge`; successful runs still publish a Model Version.
 
@@ -182,4 +183,4 @@ toloka model versions <model-id> --json
 toloka model version download <model-version-id> --out <absolute-output-directory>
 ```
 
-The downloaded folder contains full model weights (`config.json` and `model-*.safetensors`) and tokenizer files, including supported text sidecars such as `merges.txt` and `chat_template.jinja`. Keep the files together. The learned prompt is represented by `<|gist_…|>` tokens that prefix requests in place of the original prompt. Verify serving separately with a compatible runtime; a successful download alone does not establish inference readiness.
+The downloaded folder contains full model weights (`config.json` and `model-*.safetensors`) and tokenizer files, including supported text sidecars such as `merges.txt` and `chat_template.jinja`. Keep the files together. The learned prompt is represented by the special tokens `<|gist_0|>`, `<|gist_1|>`, and so on. A request must match training: the gist tokens first as plain text, then the conversation without the final assistant turn rendered by the tokenizer's chat template with `add_generation_prompt=True` and `enable_thinking=False`, without the original prompt. Send that string to a completions endpoint, because a chat endpoint wraps it in message markers; a system message holding only the gist tokens is the closest chat-only fallback and must be checked on validation rows. Verify serving separately with a compatible runtime; a successful download alone does not establish inference readiness.