@withgateway/sdk 0.2.6 → 0.2.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (110) hide show
  1. package/README.md +24 -8
  2. package/dist/cli.js +15468 -11263
  3. package/dist/cli.js.map +1 -1
  4. package/dist/cli.mjs +15233 -11028
  5. package/dist/cli.mjs.map +1 -1
  6. package/dist/files/index.d.mts +1 -1
  7. package/dist/files/index.d.ts +1 -1
  8. package/dist/files/index.js +291 -40
  9. package/dist/files/index.js.map +1 -1
  10. package/dist/files/index.mjs +284 -37
  11. package/dist/files/index.mjs.map +1 -1
  12. package/dist/hooks/index.d.mts +1 -1
  13. package/dist/hooks/index.d.ts +1 -1
  14. package/dist/hooks/index.js +203 -47
  15. package/dist/hooks/index.js.map +1 -1
  16. package/dist/hooks/index.mjs +198 -42
  17. package/dist/hooks/index.mjs.map +1 -1
  18. package/dist/{index-BaKu5ZSU.d.mts → index-C3UzD0G3.d.mts} +75 -1
  19. package/dist/{index-BaKu5ZSU.d.ts → index-C3UzD0G3.d.ts} +75 -1
  20. package/dist/{index-CLkMr6VG.d.ts → index-D-_4Ibcn.d.ts} +36 -4
  21. package/dist/{index-CdbmO4aX.d.mts → index-FfK4Nump.d.mts} +36 -4
  22. package/dist/index.d.mts +3 -3
  23. package/dist/index.d.ts +3 -3
  24. package/dist/index.js +2149 -615
  25. package/dist/index.js.map +1 -1
  26. package/dist/index.mjs +2053 -566
  27. package/dist/index.mjs.map +1 -1
  28. package/dist/traces/index.js +268 -31
  29. package/dist/traces/index.js.map +1 -1
  30. package/dist/traces/index.mjs +262 -28
  31. package/dist/traces/index.mjs.map +1 -1
  32. package/dist/tracing/index.js +19 -11
  33. package/dist/tracing/index.js.map +1 -1
  34. package/dist/tracing/index.mjs +19 -11
  35. package/dist/tracing/index.mjs.map +1 -1
  36. package/dist/worlds/index.d.mts +886 -117
  37. package/dist/worlds/index.d.ts +886 -117
  38. package/dist/worlds/index.js +1770 -452
  39. package/dist/worlds/index.js.map +1 -1
  40. package/dist/worlds/index.mjs +1752 -434
  41. package/dist/worlds/index.mjs.map +1 -1
  42. package/package.json +14 -10
  43. package/runtime/gatewaysdk/environment/_toml.py +3 -10
  44. package/runtime/gatewaysdk/environment/_world.py +95 -28
  45. package/runtime/gatewaysdk/environment/runtime.py +10 -6
  46. package/runtime/gatewaysdk/environment/schema/__main__.py +35 -0
  47. package/runtime/gatewaysdk/environment/schema/api.py +19 -23
  48. package/runtime/gatewaysdk/environment/schema/batch.py +68 -14
  49. package/runtime/gatewaysdk/environment/schema/bundle_runtime.py +4 -2
  50. package/runtime/gatewaysdk/environment/schema/cast.py +264 -0
  51. package/runtime/gatewaysdk/environment/schema/compiler.py +35 -1
  52. package/runtime/gatewaysdk/environment/schema/conform.py +28 -10
  53. package/runtime/gatewaysdk/environment/schema/connector.py +59 -14
  54. package/runtime/gatewaysdk/environment/schema/connectors/custom/handlers.py +38 -1
  55. package/runtime/gatewaysdk/environment/schema/connectors/google-drive/readers/merge.toml +3 -1
  56. package/runtime/gatewaysdk/environment/schema/credentials.py +20 -0
  57. package/runtime/gatewaysdk/environment/schema/csv_rows.py +117 -0
  58. package/runtime/gatewaysdk/environment/schema/cycles.py +121 -0
  59. package/runtime/gatewaysdk/environment/schema/data.py +21 -5
  60. package/runtime/gatewaysdk/environment/schema/grading.py +95 -13
  61. package/runtime/gatewaysdk/environment/schema/handler_errors.py +44 -0
  62. package/runtime/gatewaysdk/environment/schema/handler_scan.py +175 -0
  63. package/runtime/gatewaysdk/environment/schema/host.py +68 -115
  64. package/runtime/gatewaysdk/environment/schema/host_requests.py +78 -0
  65. package/runtime/gatewaysdk/environment/schema/host_surfaces.py +12 -0
  66. package/runtime/gatewaysdk/environment/schema/ingest.py +5 -18
  67. package/runtime/gatewaysdk/environment/schema/local.py +7 -10
  68. package/runtime/gatewaysdk/environment/schema/openapi_document.py +322 -0
  69. package/runtime/gatewaysdk/environment/schema/openapi_import.py +382 -0
  70. package/runtime/gatewaysdk/environment/schema/openapi_routes.py +441 -0
  71. package/runtime/gatewaysdk/environment/schema/openapi_schemas.py +629 -0
  72. package/runtime/gatewaysdk/environment/schema/openapi_source.py +210 -0
  73. package/runtime/gatewaysdk/environment/schema/platform.py +110 -299
  74. package/runtime/gatewaysdk/environment/schema/platform_import.py +182 -0
  75. package/runtime/gatewaysdk/environment/schema/platform_query.py +95 -0
  76. package/runtime/gatewaysdk/environment/schema/route_bindings.py +176 -0
  77. package/runtime/gatewaysdk/environment/schema/runtime.py +56 -7
  78. package/runtime/gatewaysdk/environment/schema/scaffold.py +64 -5
  79. package/runtime/gatewaysdk/environment/schema/skills/benchmarks-getting-started/SKILL.md +70 -22
  80. package/runtime/gatewaysdk/environment/schema/skills/connector-schema-authoring/SKILL.md +54 -4
  81. package/runtime/gatewaysdk/environment/schema/skills/hooks-getting-started/SKILL.md +27 -2
  82. package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/ROWS.md +62 -0
  83. package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md +91 -54
  84. package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/SKILL.md +146 -146
  85. package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/cli-mcp-map.md +11 -7
  86. package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/gotchas.md +30 -0
  87. package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/new-world-sandbox.md +43 -0
  88. package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/openapi-import.md +90 -0
  89. package/runtime/gatewaysdk/environment/schema/store.py +15 -6
  90. package/runtime/gatewaysdk/environment/schema/store_contract.py +119 -0
  91. package/runtime/gatewaysdk/environment/schema/tasks.py +121 -32
  92. package/runtime/gatewaysdk/environment/schema/tools_world.py +32 -10
  93. package/runtime/gatewaysdk/environment/schema/tree_import.py +228 -0
  94. package/runtime/gatewaysdk/environment/schema/ui.py +18 -2
  95. package/runtime/gatewaysdk/environment/schema/validation.py +35 -0
  96. package/runtime/gatewaysdk/environment/schema/verbs.py +8 -2
  97. package/runtime/gatewaysdk/environment/world.py +42 -3
  98. package/runtime/gatewaysdk/environment/world_session_contract.py +57 -0
  99. package/skills/benchmarks-getting-started/SKILL.md +70 -22
  100. package/skills/connector-schema-authoring/SKILL.md +54 -4
  101. package/skills/connector-schema-authoring/connector-worlds.md +21 -7
  102. package/skills/connector-schema-authoring/world-schema.md +144 -83
  103. package/skills/hooks-getting-started/SKILL.md +27 -2
  104. package/skills/world-data-ingestion/ROWS.md +62 -0
  105. package/skills/world-data-ingestion/SKILL.md +91 -54
  106. package/skills/worlds-getting-started/SKILL.md +146 -146
  107. package/skills/worlds-getting-started/references/cli-mcp-map.md +11 -7
  108. package/skills/worlds-getting-started/references/gotchas.md +30 -0
  109. package/skills/worlds-getting-started/references/new-world-sandbox.md +43 -0
  110. package/skills/worlds-getting-started/references/openapi-import.md +90 -0
@@ -1,18 +1,10 @@
1
1
  ---
2
2
  name: world-data-ingestion
3
- description: Put rows into a world from any agent. Use when uploading data to a world, seeding a world, importing rows (a few or millions), adding records to a schema world or mock-tools world, seeding a live session, exporting a session's calls, turning a vendor export or a real run into world data, turning an agent's real tool calls, a trace's TOOL observations or a saved ClickHouse result into rows (`gateway worlds data calls pull`, then `data ingest --transform` with your own code or `--map ingest.toml`, `captureToolCalls`), or asking why a served world starts from a snapshot. Covers the rows file contract (JSON object, JSONL, --entity), the tool-call record contract, the transform contract, the `ingest.toml` grammar and `map init`, append vs replace, primary-key upsert, `gateway worlds describe`, `gateway worlds data show|import|check|extract|calls pull|ingest|status|batches|publish`, chunked resumable uploads, refusal reports, data snapshots (`db/state.sqlite`, `WORLD_SNAPSHOT_*`), `gateway worlds session seed|export`, the MCP twins import_world_data / seed_world_session / export_world_session, the refusals, and the definition of done.
3
+ description: Put rows into a world from any agent. Use when uploading data to a world, seeding a world, importing rows (a few or millions), adding records to a schema world or mock-tools world, pulling a connector's data through an active connection (`gateway worlds data pull --connection`), seeding a live session, exporting a session's calls, turning a vendor export or a real run into world data, turning an agent's real tool calls, a trace's TOOL observations or a saved ClickHouse result into rows (`gateway worlds data calls pull`, then `data ingest --transform` with your own code or `--map ingest.toml`, `captureToolCalls`), or asking why a served world starts from a snapshot. Covers the rows file contract (JSON object, JSONL, --entity), the tool-call record contract, the transform contract, the `ingest.toml` grammar and `map init`, append vs replace, primary-key upsert, `gateway worlds describe`, `gateway worlds data show|import|pull|check|extract|calls pull|ingest|status|batches|publish`, chunked resumable uploads, refusal reports, data snapshots (`db/state.sqlite`, `WORLD_SNAPSHOT_*`), `gateway worlds session seed|export`, the MCP twins import_world_data / pull_world_data / seed_world_session / export_world_session, the refusals, and the definition of done.
4
4
  ---
5
5
 
6
6
  # World data ingestion
7
7
 
8
- > **Two copies, kept identical.** This file ships inside the `gatewaysdk` Python package
9
- > (`gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md`) and the `@withgateway/sdk`
10
- > npm package (copied into `skills/` at build) and is what
11
- > `gateway worlds skill world-data-ingestion` prints and `--install DIR` writes into a
12
- > repository's `.claude/skills/`. The repository copy is `.claude/skills/world-data-ingestion/SKILL.md`.
13
- > Edit one, copy to the other; a test compares them.
14
-
15
-
16
8
  > **Two copies, kept identical.** This file ships inside the `gatewaysdk` Python package
17
9
  > (`gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md`) and the `@withgateway/sdk`
18
10
  > npm package (copied into `skills/` at build) and is what
@@ -48,13 +40,23 @@ For the end-to-end path (connector, world, session, runs) read `worlds-getting-s
48
40
 
49
41
  ## The rows file
50
42
 
51
- Three accepted shapes. Pick one per file; a command may take several files.
43
+ Four accepted shapes. Pick one per file; a command may take several files.
52
44
 
53
45
  | Shape | Looks like | When |
54
46
  | --- | --- | --- |
55
47
  | JSON object | `{"users": [{...}, {...}], "channels": [{...}]}` | several entities in one file |
56
48
  | JSONL, tagged | one line per row: `{"entity": "users", "row": {...}}` | streaming, mixed entities |
57
49
  | Bare rows + `--entity NAME` | `[{...}, {...}]` or one bare `{...}` per line | one entity, rows straight from a query |
50
+ | CSV/TSV + `--entity NAME` | `id,subject,status` header row, one row per line (`.csv`, `.tsv`, `.gz`) | a spreadsheet or vendor export of one entity |
51
+
52
+ A CSV/TSV is decided by its name and read as a stream (bounded memory at any size).
53
+ `--delimiter <char>` splits the cells (one character, `\t` for a tab; default `,`, tab for a
54
+ `.tsv`). The header row names the fields and **every cell is cast by the entity's schema
55
+ type**: `integer`/`number` parse as numbers, `boolean` takes true/false/1/0, `array`/`object`
56
+ parse as JSON, `string` stays text; an empty cell is null when the field admits null, left out
57
+ when optional, refused when required. A cell that does not fit is a row-level refusal naming
58
+ the field and the cell (`/taxi_trip/3: fare = "n/a" is not a number`). Same rule in both CLIs,
59
+ pinned by `tests/conformance/csv_cast.json`. Without `--entity` a CSV is refused.
58
60
 
59
61
  Rules:
60
62
 
@@ -92,7 +94,10 @@ file the agent should edit.
92
94
  Every CLI step has its MCP twin on the next line. `<WORLD>` is a local world directory, else a
93
95
  platform world slug (the CLI pulls it to a temp dir, imports, and pushes a new version unless
94
96
  `--no-push`). Auth and host: `GATEWAY_HOST`, `GATEWAY_PUBLIC_KEY`, `GATEWAY_SECRET_KEY`, or
95
- `gateway auth login`.
97
+ `gateway auth login`. Third place: a CLI sandbox (`worlds sandbox open <slug>@main`) has `gateway` + `sqlite3` on PATH, so
98
+ `sandbox exec <id> 'gateway worlds data import . rows.json'` (`show`, `query`) runs on the uncommitted copy with nothing local; `check` then `commit` lands it. Same refusals as a directory (exit 2).
99
+ The sandbox holds **no platform credentials, by design**: never write keys into its tree (`.gateway-env`, `*.env` are held out of every
100
+ commit, with a warning); bulk rows reach the platform world through `worlds sandbox import <id> <path>` from where the keys are (2d).
96
101
 
97
102
  ### 1. See what the world holds
98
103
 
@@ -109,8 +114,9 @@ gateway worlds data query <WORLD> users --where email=jo@acme.test --resolve #
109
114
  `[redact]` policy before matching, and prints `resolved <field> -> <form>`; a dropped field
110
115
  resolves to nothing, so zero rows there is the policy, not a missing import. MCP twin:
111
116
  `query_world_data({ slug, entity, where, resolve, across?, compare? })`. A snapshot-backed world
112
- answers it without a session. `--across` / `across: true` asks every linked world too
113
- (`[dependencies.<alias>.entities]` mappings with their transforms); `--compare` reports parity.
117
+ answers it without a session at any size (the platform opens its tree + `db/state.sqlite` in a prepared
118
+ directory it owns; nothing is bundled into the request). `--across` / `across: true` asks every linked
119
+ world too (`[dependencies.<alias>.entities]` mappings with their transforms); `--compare` reports parity.
114
120
 
115
121
  `describe` marks each field `[hash]`, `[preserve]`, `[round:n]` or `[drop]` from the world's `[redact]`
116
122
  policy. Rows you import are stored as sent; rows the world captured from a vendor arrive already
@@ -118,8 +124,7 @@ redacted, and a hashed field is still found by its plaintext because the stored
118
124
  policy (`sha256:` of `<slug>:<value>`; `storedForms` in `@withgateway/sdk/worlds`). Never import
119
125
  plaintext into a field the policy hashes: check `describe` first.
120
126
 
121
- MCP: no counts tool. `get_environment({ containerId })` lists versions; `import_world_data`
122
- returns `entityCounts` after a write. Use the CLI to look before writing.
127
+ MCP: no counts tool. `get_environment({ containerId })` lists versions; `import_world_data` returns `entityCounts` after a write.
123
128
 
124
129
  ### 2. Import
125
130
 
@@ -135,15 +140,19 @@ gateway worlds data import <slug> export.jsonl --resume # pick a d
135
140
  gateway worlds data check <WORLD> rows.json --report refusals.jsonl # the dry run on its own; exit 2 on refusals
136
141
  ```
137
142
 
138
- **A directory** takes small inputs only (at most 5,000 rows / 4 MiB, the inline cap): the rows
139
- land in the tree and the snapshot is rebuilt. Prints `{world, kind, entity_counts, changed}`.
140
- Above the cap it refuses with `import into the platform world <slug>`.
143
+ **A directory** streams the files in place (the bundled Python runtime reads one row at a
144
+ time) into the tree and rebuilds the snapshot. Prints `{world, kind, entity_counts, changed}`.
145
+ The tree is a source file publication reads whole (refused over 4 MiB), so the door has the
146
+ same cap — **4 MiB / 100,000 rows** in `data/initial.json`; above it the refusal names the bulk door: `sandbox
147
+ import` from a sandbox, `data import <slug>` where the keys are. Without `python3` the rows
148
+ travel whole with the tree: 5,000 rows / 4 MiB, and no `.csv`.
141
149
 
142
150
  **A slug** streams the files (never loaded whole; `-` is stdin, `.gz` is fine) into gzipped
143
151
  JSONL chunks of `--chunk-bytes` (64 MiB default) or `--chunk-rows N`, uploads them
144
152
  `--parallel 4` at a time straight to the bucket, and completes the batch; the platform
145
153
  validates every row against the contract, applies the batch onto the world's data snapshot
146
- and, with `--publish now` (default), cuts a data-only version (no image rebuild). `--wait`
154
+ and, with `--publish now` (default), cuts a data-only version (no image rebuild) on your open edit session
155
+ (`editSessionId`; only your credential reads it as `main` until `worlds edit close <slug>` or 60 idle minutes seal it). `--wait`
147
156
  (default) polls to the end and prints the batch: `{batchId, state, rows, refusalCount,
148
157
  refusals[:200], entityCounts, versionId?, batchDigest?}`. `--no-wait` prints `{batchId,
149
158
  state}` and `gateway worlds data status <batchId>` reads it later. Exit codes: 0 applied or
@@ -151,11 +160,9 @@ published, 2 refused (the report is written when `--report` is given: JSONL of r
151
160
  records, at most 10,000 then `{"truncated": true, "total": N}`), 1 error.
152
161
 
153
162
  A manifest at `.gateway/imports/<batchId>.json` records every chunk's digest as it lands;
154
- `--resume` reopens the newest incomplete batch for that slug and files, re-registers the
155
- chunks (the platform answers `uploaded: true` for digests it already holds) and uploads only
156
- the rest. `--publish later` applies without a version; `gateway worlds data publish <slug>`
157
- folds every pending batch into one. `gateway worlds data batches <slug>` lists them.
158
-
163
+ `--resume` reopens the newest incomplete batch for that slug and files and uploads only the
164
+ chunks the platform does not already hold. `--publish later` applies without a version;
165
+ `gateway worlds data publish <slug>` folds every pending batch into one; `data batches <slug>` lists them.
159
166
  A refusal record, everywhere (dry run, report, status, MCP):
160
167
  `{"chunk", "index", "entity", "primaryKey", "field", "pointer", "detail"}`, `pointer` being
161
168
  `/<entity>/<index>/<field>` inside the chunk.
@@ -165,6 +172,33 @@ object shape (`{entity: [rows]}`), at most 5,000 rows and 4 MiB per call: the in
165
172
  one chunk through the same batch job. Returns `{ versionId, entityCounts, deduped }`; a
166
173
  refused row is a tool error naming the field, and nothing is written.
167
174
 
175
+ ### 2a. Pull a connector's data through an active connection
176
+
177
+ For a schema world created from a connector template (`worlds create --from-connection`, or
178
+ the Connectors page), the provider is the source — no rows file:
179
+
180
+ ```bash
181
+ gateway worlds data pull <slug> --connection google_drive # connector slug when the project has one connection for it
182
+ gateway worlds data pull <slug> --connection <connectionId> --mode append -m "nightly"
183
+ gateway worlds data pull <slug> --connection <connectionId> --json
184
+ ```
185
+
186
+ `POST /api/public/worlds/{slug}/data/pull` with `{ connection, mode = "replace", message? }`.
187
+ What is read is the template's installed reader plan (`readers/merge.toml` + `merge.ingest.toml`
188
+ in the connector package), never a tool or argument you name; the rows land through the world's
189
+ contract like `data import`. `replace` (default) rebuilds the mapped entities from the provider;
190
+ `append` upserts onto the existing rows and keeps what the provider no longer returns. Prints
191
+ `pulled|unchanged <slug> from <connector> connection <id> (<mode>, <n> calls[, k unmatched (tool…)])
192
+ <hash> [entity=count …]`; `changed: false` in `--json` when the pull reproduced the tip (no new
193
+ version). The key reaches the connections shared with its project; a personal connection serves
194
+ its owner alone (the Connectors page, or the MCP tool as that user) and is refused here. An
195
+ inactive connection, a world without a contract, or a template that does not read the connector
196
+ is a refusal (exit 1) and nothing is written; a provider failure is reported as
197
+ `provider tool failed (HTTP n)` with no provider body.
198
+
199
+ MCP: `pull_world_data({ slug, connection, mode?, message? })` — same request; the acting user is
200
+ the MCP session's (a PAT reaches its own personal connection).
201
+
168
202
  ### 2b. Extract rows from what you already have
169
203
 
170
204
  ```bash
@@ -180,17 +214,15 @@ under the world's `[api].envelope` key, the bare array when it is `""`), `sessio
180
214
  and empty-string fields, strips fields the entity does not declare (counted on stderr, never
181
215
  silently), dedupes by primary key (`--merge fill` default) and applies the world's `[redact]`
182
216
  policy (`drop`, `hash` as `sha256:<hex>` of `<slug>:<value>`, `preserve` as a same-length,
183
- same-character-class hash for identifiers, `round`) unless `--no-redact`.
184
- Output is tagged JSONL; the summary `{"read", "rows", "unknownFields", "redacted"}` is on
185
- stderr.
217
+ same-character-class hash for identifiers, `round`) unless `--no-redact`. Output is tagged
218
+ JSONL; the summary `{"read", "rows", "unknownFields", "redacted"}` is on stderr.
186
219
 
187
220
  Platform agent (console): the same `import_world_data`, plus `sandboxPath` for a rows file
188
221
  already in the sandbox (any ROWS form above; `entity` for bare arrays/JSONL) instead of
189
- `rows`. Build the file with code, then import — do not paste 4 MiB through `rows`.
222
+ `rows` — build the file with code, then import; do not paste 4 MiB through `rows`.
190
223
  `run_gateway_cli({ command: "data show" | "data check" | "data import" | "data extract", args: ["/workspace/<world>", ...] })`
191
- is the local-directory form for a world you `load_world_to_sandbox`ed — the same TypeScript
192
- `gateway worlds data …` commands and flags as above (`--entity`, `--mode`, `--merge`, `--dry-run`,
193
- `--report`, `--shape`), no remote slugs; land the result with `ingest_to_world`. Capture pages are not rows: `get_connector_capture(downloadTo)` →
224
+ is the local-directory form for a world you `load_world_to_sandbox`ed (same flags as above, no remote
225
+ slugs); land the result with `ingest_to_world`. Capture pages are not rows: `get_connector_capture(downloadTo)` →
194
226
  `run_gateway_cli connector ingest` → `ingest_to_world`.
195
227
 
196
228
  ### 2c. Turn tool calls into rows: pull, transform, ingest
@@ -209,7 +241,8 @@ gateway worlds describe <WORLD>; gateway worlds data query <WORLD> <entity> --li
209
241
  upload rows.json --to seeds/rows.json` then `gateway worlds data import <WORLD> drive:seeds/rows.json`
210
242
  is the route from a laptop (or an agent's `request_file_upload`) to a world with nothing staged
211
243
  locally: the platform reads the drive file itself (`.json`, `.jsonl`, `.ndjson`, optionally `.gz`;
212
- convert CSV first) and stages it as one chunk — nothing comes down to your machine and nothing is
244
+ a `.csv` on the drive is refused — import it from the machine that holds it with `--entity`, or
245
+ convert it first) and stages it as one chunk — nothing comes down to your machine and nothing is
213
246
  re-uploaded, so `drive:` needs a world slug and no `--dry-run`. The report keeps the drive path under
214
247
  `inputs` and the manifest records the digest, size and rows the platform derived. `--to <path>` on `calls pull`, `extract`
215
248
  and `session export` lands the output in the drive (`--from drive:<path>` on `calls pull` and
@@ -329,6 +362,22 @@ redact="off", dry_run=False)` and `ingest_tool_calls`; `capture_tool_calls`. Bri
329
362
  rows (+ `files` when the tree changed); transforms never run in the bridge. No MCP twin yet;
330
363
  the console agent has `run_gateway_cli data ingest` on a sandboxed directory.
331
364
 
365
+ ### 2d. Rows generated or downloaded inside a sandbox
366
+
367
+ ```bash
368
+ gateway worlds sandbox exec <id> 'curl -sSL https://example.test/trips.csv -o data/trips.csv' # or a script that writes rows
369
+ gateway worlds sandbox import <id> data/trips.csv --entity taxi_trip [--delimiter ...] [--mode append|replace] [--dry-run]
370
+ gateway worlds sandbox commit <id> -m "tools" # the SOURCE tree only; rows are not in a commit
371
+ ```
372
+
373
+ `sandbox import` runs where the project keys are; the **platform** reads the file out of the
374
+ sandbox (`POST /api/public/benchmark-containers/sandboxes/{id}/import`; the sandbox's `gateway`
375
+ chunks it to JSONL in a scratch dir the platform reads and removes) and streams it through the
376
+ same ledger as `data import <slug>`: contract, refusals with pointers, `--mode`, `--merge`,
377
+ `--redact`, `--per-chunk`, `-m`. Prints `{versionId, entityCounts, rowCount}`, or `{batchId,
378
+ state}` to poll with `data status`. A new-world sandbox commits once first. MCP:
379
+ `import_world_sandbox_data({ sandboxId, path, entity?, mode? })`. `commit` never carries rows.
380
+
332
381
  ### 3. Verify on a live session
333
382
 
334
383
  Open a session, seed it, hit it. Live seeding does not change the world's version; it
@@ -391,9 +440,8 @@ mapping), or with neither the operations projection for tools whose result is th
391
440
  response body. The schema, not the tool, decides the shape; `--dry-run` shows the rows before
392
441
  anything is written.
393
442
 
394
- Sanitize before importing: hash or drop personal fields the world's redaction policy
395
- (`[redact]` in `connector.toml`) would have hashed or dropped. Customer rows never enter a
396
- shared template.
443
+ Sanitize before importing: hash or drop personal fields the world's `[redact]` policy
444
+ (`connector.toml`) would have hashed or dropped. Customer rows never enter a shared template.
397
445
 
398
446
  ## Failure modes
399
447
 
@@ -408,7 +456,8 @@ shared template.
408
456
  | a contract error at `/<entity>/<index>/<field>` | type, enum, required or reference broken | fix that row; nothing was written |
409
457
  | refused, kind `api` (or `harbor`), naming the file to edit | the world is a generated bundle or a task suite | edit the named source, `bench push` again |
410
458
  | `version` missing from the result | a local directory was given | push with `gateway bench push <dir>` when ready |
411
- | `exceed the inline cap ... import into the platform world <slug>` | a directory import above 5,000 rows / 4 MiB | import into the platform slug (chunked), or `--dry-run` to check locally |
459
+ | `... is over what a world's source tree carries as data (4 MiB / 100000 rows ...)`; `data/initial.json would be N MiB with these rows, over the 4 MiB publication reads a source file whole`; `exceed the inline cap ... without the bundled runtime` | a directory import above the tree cap; or no `python3`, so the rows travel whole (5,000 / 4 MiB) | `worlds sandbox import` from a sandbox, `data import <slug>` with keys (never keys into the sandbox); install python3 |
460
+ | `/<entity>/<i>: <field> = "<cell>" is not a number` (integer, boolean, array, object) | a CSV cell the field's schema type does not fit | fix the cell or the header; nothing was written |
412
461
  | exit 2, `state: refused` | the platform refused the batch; `--report` has every record | fix the rows named, `--resume` is not for this: run the import again |
413
462
  | `nothing to resume` / `only an open or uploading batch resumes` | no incomplete manifest, or the batch already ran | run the import without `--resume` |
414
463
  | `409 base_mismatch` from `/__world/seed` | the overlay named another snapshot digest than the pod is on | the runner posts the digest of the version's snapshot; a mismatch is a version mismatch, never reseed silently |
@@ -420,29 +469,17 @@ shared template.
420
469
  | `refused [i] x: no [[sources]] named 'x'` / `… selects or matches records[i]` | a mapping: no source claims the record (no source of that name, or every `when` failed / `select` found nothing) | add the source, or `[ingest] unmatched = "skip"` to count them instead; nothing was written |
421
470
  | `<script> exited N` / `timed out after Ns` / `must return {entity: [rows]}` (exit 1) | the transform failed, ran past `--timeout`, or returned the wrong shape | read its stderr (shown prefixed `transform:`); a transform cannot reach the network by design |
422
471
  | `accounts has no field nickname; its fields are …` / `carry names parent field id, which the source does not map` | the mapping names what the contract does not have, or a carry the parent never sets | `gateway worlds describe` lists the fields; map the parent field the carry needs |
423
- | `--from csv needs --transform or --map` | a CSV says nothing about its entity or types | declare the mapping (`type = "number"` for numeric columns) or write the transform |
472
+ | `<file>.csv is a CSV; say which entity its columns fill with --entity` / `--delimiter is one character or \t` | a CSV/TSV without `--entity`; a multi-character delimiter | pass `--entity <name>` (cells cast by its field types); one character or `\t` (a `.tsv` needs none) |
473
+ | `--from csv needs --transform or --map` | `data ingest`: a CSV says nothing about its entity or types | declare the mapping (`type = "number"` for numeric columns) or write the transform |
424
474
  | `--from traces needs --trace, --session or --since` | no selection for the observations | say which calls: trace ids, a session, or a time window |
425
475
  | `data ingest` reports `skipped: operation x lands no entity` | the operations route: the operation has no `entity` / `results` | give it one in `connector.toml`, or map that tool in `ingest.toml` / the transform |
426
476
  | session stays `PROVISIONING` | cold pod on a fresh version | wait up to twenty minutes; `session status` |
427
477
  | duplicate rows after a live append on an api world | the world declares no primary keys, so append adds without upserting | use `--mode replace`, or declare keys in `world.json` |
428
478
 
429
- ## Data snapshots (why a served world starts in under a second)
430
-
431
- A schema world's validated database is kept, not rebuilt. `gateway worlds schema compile`
432
- builds `db/state.sqlite` (the default snapshot: `data/initial.json` applied through the
433
- contract) and `scenarios/<name>/state.sqlite` (the default with the scenario's overlays),
434
- each with a `state.json` sidecar `{contractHash, batchDigest, entityCounts, builtAt,
435
- runtime, fileSha256}`; the same key is never rebuilt, `schema check` verifies the sidecars,
436
- and a data batch the contract refuses leaves no snapshot behind. `gateway worlds serve` and
437
- the api bundle open the snapshot (copy + `SQLiteStore.open`, no row replay) and reset by
438
- copying it again; without one the JSON is replayed as before. A hosted pod gets
439
- `WORLD_SNAPSHOT_URL` (presigned GET, `.gz` or `Content-Encoding: gzip` is gunzipped),
440
- `WORLD_SNAPSHOT_SHA256` and `WORLD_SNAPSHOT_DIGEST`; the server downloads, verifies, writes
441
- the sidecar and only then serves; a mismatch exits 2 naming the digest. `/__world/state`
442
- reports `"snapshot": {batchDigest, contractHash}` (or null) and `/__world/seed` takes the
443
- overlay form `{"base": digest, "rows": {...}, "mode": append|replace}` beside the legacy
444
- whole-state body. Snapshot files are build outputs: gitignore `db/state.sqlite`,
445
- `db/state.json`, `scenarios/*/state.*`.
479
+ ## Data snapshots
480
+
481
+ Why a served world starts in under a second (`db/state.sqlite`, `WORLD_SNAPSHOT_*`, the
482
+ `/__world/seed` overlay form): [ROWS.md — Data snapshots](ROWS.md#data-snapshots).
446
483
 
447
484
  ## Definition of done
448
485