@withgateway/sdk 0.2.6 → 0.2.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +24 -8
- package/dist/cli.js +15468 -11263
- package/dist/cli.js.map +1 -1
- package/dist/cli.mjs +15233 -11028
- package/dist/cli.mjs.map +1 -1
- package/dist/files/index.d.mts +1 -1
- package/dist/files/index.d.ts +1 -1
- package/dist/files/index.js +291 -40
- package/dist/files/index.js.map +1 -1
- package/dist/files/index.mjs +284 -37
- package/dist/files/index.mjs.map +1 -1
- package/dist/hooks/index.d.mts +1 -1
- package/dist/hooks/index.d.ts +1 -1
- package/dist/hooks/index.js +203 -47
- package/dist/hooks/index.js.map +1 -1
- package/dist/hooks/index.mjs +198 -42
- package/dist/hooks/index.mjs.map +1 -1
- package/dist/{index-BaKu5ZSU.d.mts → index-C3UzD0G3.d.mts} +75 -1
- package/dist/{index-BaKu5ZSU.d.ts → index-C3UzD0G3.d.ts} +75 -1
- package/dist/{index-CLkMr6VG.d.ts → index-D-_4Ibcn.d.ts} +36 -4
- package/dist/{index-CdbmO4aX.d.mts → index-FfK4Nump.d.mts} +36 -4
- package/dist/index.d.mts +3 -3
- package/dist/index.d.ts +3 -3
- package/dist/index.js +2149 -615
- package/dist/index.js.map +1 -1
- package/dist/index.mjs +2053 -566
- package/dist/index.mjs.map +1 -1
- package/dist/traces/index.js +268 -31
- package/dist/traces/index.js.map +1 -1
- package/dist/traces/index.mjs +262 -28
- package/dist/traces/index.mjs.map +1 -1
- package/dist/tracing/index.js +19 -11
- package/dist/tracing/index.js.map +1 -1
- package/dist/tracing/index.mjs +19 -11
- package/dist/tracing/index.mjs.map +1 -1
- package/dist/worlds/index.d.mts +886 -117
- package/dist/worlds/index.d.ts +886 -117
- package/dist/worlds/index.js +1770 -452
- package/dist/worlds/index.js.map +1 -1
- package/dist/worlds/index.mjs +1752 -434
- package/dist/worlds/index.mjs.map +1 -1
- package/package.json +14 -10
- package/runtime/gatewaysdk/environment/_toml.py +3 -10
- package/runtime/gatewaysdk/environment/_world.py +95 -28
- package/runtime/gatewaysdk/environment/runtime.py +10 -6
- package/runtime/gatewaysdk/environment/schema/__main__.py +35 -0
- package/runtime/gatewaysdk/environment/schema/api.py +19 -23
- package/runtime/gatewaysdk/environment/schema/batch.py +68 -14
- package/runtime/gatewaysdk/environment/schema/bundle_runtime.py +4 -2
- package/runtime/gatewaysdk/environment/schema/cast.py +264 -0
- package/runtime/gatewaysdk/environment/schema/compiler.py +35 -1
- package/runtime/gatewaysdk/environment/schema/conform.py +28 -10
- package/runtime/gatewaysdk/environment/schema/connector.py +59 -14
- package/runtime/gatewaysdk/environment/schema/connectors/custom/handlers.py +38 -1
- package/runtime/gatewaysdk/environment/schema/connectors/google-drive/readers/merge.toml +3 -1
- package/runtime/gatewaysdk/environment/schema/credentials.py +20 -0
- package/runtime/gatewaysdk/environment/schema/csv_rows.py +117 -0
- package/runtime/gatewaysdk/environment/schema/cycles.py +121 -0
- package/runtime/gatewaysdk/environment/schema/data.py +21 -5
- package/runtime/gatewaysdk/environment/schema/grading.py +95 -13
- package/runtime/gatewaysdk/environment/schema/handler_errors.py +44 -0
- package/runtime/gatewaysdk/environment/schema/handler_scan.py +175 -0
- package/runtime/gatewaysdk/environment/schema/host.py +68 -115
- package/runtime/gatewaysdk/environment/schema/host_requests.py +78 -0
- package/runtime/gatewaysdk/environment/schema/host_surfaces.py +12 -0
- package/runtime/gatewaysdk/environment/schema/ingest.py +5 -18
- package/runtime/gatewaysdk/environment/schema/local.py +7 -10
- package/runtime/gatewaysdk/environment/schema/openapi_document.py +322 -0
- package/runtime/gatewaysdk/environment/schema/openapi_import.py +382 -0
- package/runtime/gatewaysdk/environment/schema/openapi_routes.py +441 -0
- package/runtime/gatewaysdk/environment/schema/openapi_schemas.py +629 -0
- package/runtime/gatewaysdk/environment/schema/openapi_source.py +210 -0
- package/runtime/gatewaysdk/environment/schema/platform.py +110 -299
- package/runtime/gatewaysdk/environment/schema/platform_import.py +182 -0
- package/runtime/gatewaysdk/environment/schema/platform_query.py +95 -0
- package/runtime/gatewaysdk/environment/schema/route_bindings.py +176 -0
- package/runtime/gatewaysdk/environment/schema/runtime.py +56 -7
- package/runtime/gatewaysdk/environment/schema/scaffold.py +64 -5
- package/runtime/gatewaysdk/environment/schema/skills/benchmarks-getting-started/SKILL.md +70 -22
- package/runtime/gatewaysdk/environment/schema/skills/connector-schema-authoring/SKILL.md +54 -4
- package/runtime/gatewaysdk/environment/schema/skills/hooks-getting-started/SKILL.md +27 -2
- package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/ROWS.md +62 -0
- package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md +91 -54
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/SKILL.md +146 -146
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/cli-mcp-map.md +11 -7
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/gotchas.md +30 -0
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/new-world-sandbox.md +43 -0
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/openapi-import.md +90 -0
- package/runtime/gatewaysdk/environment/schema/store.py +15 -6
- package/runtime/gatewaysdk/environment/schema/store_contract.py +119 -0
- package/runtime/gatewaysdk/environment/schema/tasks.py +121 -32
- package/runtime/gatewaysdk/environment/schema/tools_world.py +32 -10
- package/runtime/gatewaysdk/environment/schema/tree_import.py +228 -0
- package/runtime/gatewaysdk/environment/schema/ui.py +18 -2
- package/runtime/gatewaysdk/environment/schema/validation.py +35 -0
- package/runtime/gatewaysdk/environment/schema/verbs.py +8 -2
- package/runtime/gatewaysdk/environment/world.py +42 -3
- package/runtime/gatewaysdk/environment/world_session_contract.py +57 -0
- package/skills/benchmarks-getting-started/SKILL.md +70 -22
- package/skills/connector-schema-authoring/SKILL.md +54 -4
- package/skills/connector-schema-authoring/connector-worlds.md +21 -7
- package/skills/connector-schema-authoring/world-schema.md +144 -83
- package/skills/hooks-getting-started/SKILL.md +27 -2
- package/skills/world-data-ingestion/ROWS.md +62 -0
- package/skills/world-data-ingestion/SKILL.md +91 -54
- package/skills/worlds-getting-started/SKILL.md +146 -146
- package/skills/worlds-getting-started/references/cli-mcp-map.md +11 -7
- package/skills/worlds-getting-started/references/gotchas.md +30 -0
- package/skills/worlds-getting-started/references/new-world-sandbox.md +43 -0
- package/skills/worlds-getting-started/references/openapi-import.md +90 -0
|
@@ -1,18 +1,10 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: world-data-ingestion
|
|
3
|
-
description: Put rows into a world from any agent. Use when uploading data to a world, seeding a world, importing rows (a few or millions), adding records to a schema world or mock-tools world, seeding a live session, exporting a session's calls, turning a vendor export or a real run into world data, turning an agent's real tool calls, a trace's TOOL observations or a saved ClickHouse result into rows (`gateway worlds data calls pull`, then `data ingest --transform` with your own code or `--map ingest.toml`, `captureToolCalls`), or asking why a served world starts from a snapshot. Covers the rows file contract (JSON object, JSONL, --entity), the tool-call record contract, the transform contract, the `ingest.toml` grammar and `map init`, append vs replace, primary-key upsert, `gateway worlds describe`, `gateway worlds data show|import|check|extract|calls pull|ingest|status|batches|publish`, chunked resumable uploads, refusal reports, data snapshots (`db/state.sqlite`, `WORLD_SNAPSHOT_*`), `gateway worlds session seed|export`, the MCP twins import_world_data / seed_world_session / export_world_session, the refusals, and the definition of done.
|
|
3
|
+
description: Put rows into a world from any agent. Use when uploading data to a world, seeding a world, importing rows (a few or millions), adding records to a schema world or mock-tools world, pulling a connector's data through an active connection (`gateway worlds data pull --connection`), seeding a live session, exporting a session's calls, turning a vendor export or a real run into world data, turning an agent's real tool calls, a trace's TOOL observations or a saved ClickHouse result into rows (`gateway worlds data calls pull`, then `data ingest --transform` with your own code or `--map ingest.toml`, `captureToolCalls`), or asking why a served world starts from a snapshot. Covers the rows file contract (JSON object, JSONL, --entity), the tool-call record contract, the transform contract, the `ingest.toml` grammar and `map init`, append vs replace, primary-key upsert, `gateway worlds describe`, `gateway worlds data show|import|pull|check|extract|calls pull|ingest|status|batches|publish`, chunked resumable uploads, refusal reports, data snapshots (`db/state.sqlite`, `WORLD_SNAPSHOT_*`), `gateway worlds session seed|export`, the MCP twins import_world_data / pull_world_data / seed_world_session / export_world_session, the refusals, and the definition of done.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# World data ingestion
|
|
7
7
|
|
|
8
|
-
> **Two copies, kept identical.** This file ships inside the `gatewaysdk` Python package
|
|
9
|
-
> (`gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md`) and the `@withgateway/sdk`
|
|
10
|
-
> npm package (copied into `skills/` at build) and is what
|
|
11
|
-
> `gateway worlds skill world-data-ingestion` prints and `--install DIR` writes into a
|
|
12
|
-
> repository's `.claude/skills/`. The repository copy is `.claude/skills/world-data-ingestion/SKILL.md`.
|
|
13
|
-
> Edit one, copy to the other; a test compares them.
|
|
14
|
-
|
|
15
|
-
|
|
16
8
|
> **Two copies, kept identical.** This file ships inside the `gatewaysdk` Python package
|
|
17
9
|
> (`gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md`) and the `@withgateway/sdk`
|
|
18
10
|
> npm package (copied into `skills/` at build) and is what
|
|
@@ -48,13 +40,23 @@ For the end-to-end path (connector, world, session, runs) read `worlds-getting-s
|
|
|
48
40
|
|
|
49
41
|
## The rows file
|
|
50
42
|
|
|
51
|
-
|
|
43
|
+
Four accepted shapes. Pick one per file; a command may take several files.
|
|
52
44
|
|
|
53
45
|
| Shape | Looks like | When |
|
|
54
46
|
| --- | --- | --- |
|
|
55
47
|
| JSON object | `{"users": [{...}, {...}], "channels": [{...}]}` | several entities in one file |
|
|
56
48
|
| JSONL, tagged | one line per row: `{"entity": "users", "row": {...}}` | streaming, mixed entities |
|
|
57
49
|
| Bare rows + `--entity NAME` | `[{...}, {...}]` or one bare `{...}` per line | one entity, rows straight from a query |
|
|
50
|
+
| CSV/TSV + `--entity NAME` | `id,subject,status` header row, one row per line (`.csv`, `.tsv`, `.gz`) | a spreadsheet or vendor export of one entity |
|
|
51
|
+
|
|
52
|
+
A CSV/TSV is decided by its name and read as a stream (bounded memory at any size).
|
|
53
|
+
`--delimiter <char>` splits the cells (one character, `\t` for a tab; default `,`, tab for a
|
|
54
|
+
`.tsv`). The header row names the fields and **every cell is cast by the entity's schema
|
|
55
|
+
type**: `integer`/`number` parse as numbers, `boolean` takes true/false/1/0, `array`/`object`
|
|
56
|
+
parse as JSON, `string` stays text; an empty cell is null when the field admits null, left out
|
|
57
|
+
when optional, refused when required. A cell that does not fit is a row-level refusal naming
|
|
58
|
+
the field and the cell (`/taxi_trip/3: fare = "n/a" is not a number`). Same rule in both CLIs,
|
|
59
|
+
pinned by `tests/conformance/csv_cast.json`. Without `--entity` a CSV is refused.
|
|
58
60
|
|
|
59
61
|
Rules:
|
|
60
62
|
|
|
@@ -92,7 +94,10 @@ file the agent should edit.
|
|
|
92
94
|
Every CLI step has its MCP twin on the next line. `<WORLD>` is a local world directory, else a
|
|
93
95
|
platform world slug (the CLI pulls it to a temp dir, imports, and pushes a new version unless
|
|
94
96
|
`--no-push`). Auth and host: `GATEWAY_HOST`, `GATEWAY_PUBLIC_KEY`, `GATEWAY_SECRET_KEY`, or
|
|
95
|
-
`gateway auth login`.
|
|
97
|
+
`gateway auth login`. Third place: a CLI sandbox (`worlds sandbox open <slug>@main`) has `gateway` + `sqlite3` on PATH, so
|
|
98
|
+
`sandbox exec <id> 'gateway worlds data import . rows.json'` (`show`, `query`) runs on the uncommitted copy with nothing local; `check` then `commit` lands it. Same refusals as a directory (exit 2).
|
|
99
|
+
The sandbox holds **no platform credentials, by design**: never write keys into its tree (`.gateway-env`, `*.env` are held out of every
|
|
100
|
+
commit, with a warning); bulk rows reach the platform world through `worlds sandbox import <id> <path>` from where the keys are (2d).
|
|
96
101
|
|
|
97
102
|
### 1. See what the world holds
|
|
98
103
|
|
|
@@ -109,8 +114,9 @@ gateway worlds data query <WORLD> users --where email=jo@acme.test --resolve #
|
|
|
109
114
|
`[redact]` policy before matching, and prints `resolved <field> -> <form>`; a dropped field
|
|
110
115
|
resolves to nothing, so zero rows there is the policy, not a missing import. MCP twin:
|
|
111
116
|
`query_world_data({ slug, entity, where, resolve, across?, compare? })`. A snapshot-backed world
|
|
112
|
-
answers it without a session
|
|
113
|
-
|
|
117
|
+
answers it without a session at any size (the platform opens its tree + `db/state.sqlite` in a prepared
|
|
118
|
+
directory it owns; nothing is bundled into the request). `--across` / `across: true` asks every linked
|
|
119
|
+
world too (`[dependencies.<alias>.entities]` mappings with their transforms); `--compare` reports parity.
|
|
114
120
|
|
|
115
121
|
`describe` marks each field `[hash]`, `[preserve]`, `[round:n]` or `[drop]` from the world's `[redact]`
|
|
116
122
|
policy. Rows you import are stored as sent; rows the world captured from a vendor arrive already
|
|
@@ -118,8 +124,7 @@ redacted, and a hashed field is still found by its plaintext because the stored
|
|
|
118
124
|
policy (`sha256:` of `<slug>:<value>`; `storedForms` in `@withgateway/sdk/worlds`). Never import
|
|
119
125
|
plaintext into a field the policy hashes: check `describe` first.
|
|
120
126
|
|
|
121
|
-
MCP: no counts tool. `get_environment({ containerId })` lists versions; `import_world_data`
|
|
122
|
-
returns `entityCounts` after a write. Use the CLI to look before writing.
|
|
127
|
+
MCP: no counts tool. `get_environment({ containerId })` lists versions; `import_world_data` returns `entityCounts` after a write.
|
|
123
128
|
|
|
124
129
|
### 2. Import
|
|
125
130
|
|
|
@@ -135,15 +140,19 @@ gateway worlds data import <slug> export.jsonl --resume # pick a d
|
|
|
135
140
|
gateway worlds data check <WORLD> rows.json --report refusals.jsonl # the dry run on its own; exit 2 on refusals
|
|
136
141
|
```
|
|
137
142
|
|
|
138
|
-
**A directory**
|
|
139
|
-
|
|
140
|
-
|
|
143
|
+
**A directory** streams the files in place (the bundled Python runtime reads one row at a
|
|
144
|
+
time) into the tree and rebuilds the snapshot. Prints `{world, kind, entity_counts, changed}`.
|
|
145
|
+
The tree is a source file publication reads whole (refused over 4 MiB), so the door has the
|
|
146
|
+
same cap — **4 MiB / 100,000 rows** in `data/initial.json`; above it the refusal names the bulk door: `sandbox
|
|
147
|
+
import` from a sandbox, `data import <slug>` where the keys are. Without `python3` the rows
|
|
148
|
+
travel whole with the tree: 5,000 rows / 4 MiB, and no `.csv`.
|
|
141
149
|
|
|
142
150
|
**A slug** streams the files (never loaded whole; `-` is stdin, `.gz` is fine) into gzipped
|
|
143
151
|
JSONL chunks of `--chunk-bytes` (64 MiB default) or `--chunk-rows N`, uploads them
|
|
144
152
|
`--parallel 4` at a time straight to the bucket, and completes the batch; the platform
|
|
145
153
|
validates every row against the contract, applies the batch onto the world's data snapshot
|
|
146
|
-
and, with `--publish now` (default), cuts a data-only version (no image rebuild)
|
|
154
|
+
and, with `--publish now` (default), cuts a data-only version (no image rebuild) on your open edit session
|
|
155
|
+
(`editSessionId`; only your credential reads it as `main` until `worlds edit close <slug>` or 60 idle minutes seal it). `--wait`
|
|
147
156
|
(default) polls to the end and prints the batch: `{batchId, state, rows, refusalCount,
|
|
148
157
|
refusals[:200], entityCounts, versionId?, batchDigest?}`. `--no-wait` prints `{batchId,
|
|
149
158
|
state}` and `gateway worlds data status <batchId>` reads it later. Exit codes: 0 applied or
|
|
@@ -151,11 +160,9 @@ published, 2 refused (the report is written when `--report` is given: JSONL of r
|
|
|
151
160
|
records, at most 10,000 then `{"truncated": true, "total": N}`), 1 error.
|
|
152
161
|
|
|
153
162
|
A manifest at `.gateway/imports/<batchId>.json` records every chunk's digest as it lands;
|
|
154
|
-
`--resume` reopens the newest incomplete batch for that slug and files
|
|
155
|
-
chunks
|
|
156
|
-
|
|
157
|
-
folds every pending batch into one. `gateway worlds data batches <slug>` lists them.
|
|
158
|
-
|
|
163
|
+
`--resume` reopens the newest incomplete batch for that slug and files and uploads only the
|
|
164
|
+
chunks the platform does not already hold. `--publish later` applies without a version;
|
|
165
|
+
`gateway worlds data publish <slug>` folds every pending batch into one; `data batches <slug>` lists them.
|
|
159
166
|
A refusal record, everywhere (dry run, report, status, MCP):
|
|
160
167
|
`{"chunk", "index", "entity", "primaryKey", "field", "pointer", "detail"}`, `pointer` being
|
|
161
168
|
`/<entity>/<index>/<field>` inside the chunk.
|
|
@@ -165,6 +172,33 @@ object shape (`{entity: [rows]}`), at most 5,000 rows and 4 MiB per call: the in
|
|
|
165
172
|
one chunk through the same batch job. Returns `{ versionId, entityCounts, deduped }`; a
|
|
166
173
|
refused row is a tool error naming the field, and nothing is written.
|
|
167
174
|
|
|
175
|
+
### 2a. Pull a connector's data through an active connection
|
|
176
|
+
|
|
177
|
+
For a schema world created from a connector template (`worlds create --from-connection`, or
|
|
178
|
+
the Connectors page), the provider is the source — no rows file:
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
gateway worlds data pull <slug> --connection google_drive # connector slug when the project has one connection for it
|
|
182
|
+
gateway worlds data pull <slug> --connection <connectionId> --mode append -m "nightly"
|
|
183
|
+
gateway worlds data pull <slug> --connection <connectionId> --json
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
`POST /api/public/worlds/{slug}/data/pull` with `{ connection, mode = "replace", message? }`.
|
|
187
|
+
What is read is the template's installed reader plan (`readers/merge.toml` + `merge.ingest.toml`
|
|
188
|
+
in the connector package), never a tool or argument you name; the rows land through the world's
|
|
189
|
+
contract like `data import`. `replace` (default) rebuilds the mapped entities from the provider;
|
|
190
|
+
`append` upserts onto the existing rows and keeps what the provider no longer returns. Prints
|
|
191
|
+
`pulled|unchanged <slug> from <connector> connection <id> (<mode>, <n> calls[, k unmatched (tool…)])
|
|
192
|
+
<hash> [entity=count …]`; `changed: false` in `--json` when the pull reproduced the tip (no new
|
|
193
|
+
version). The key reaches the connections shared with its project; a personal connection serves
|
|
194
|
+
its owner alone (the Connectors page, or the MCP tool as that user) and is refused here. An
|
|
195
|
+
inactive connection, a world without a contract, or a template that does not read the connector
|
|
196
|
+
is a refusal (exit 1) and nothing is written; a provider failure is reported as
|
|
197
|
+
`provider tool failed (HTTP n)` with no provider body.
|
|
198
|
+
|
|
199
|
+
MCP: `pull_world_data({ slug, connection, mode?, message? })` — same request; the acting user is
|
|
200
|
+
the MCP session's (a PAT reaches its own personal connection).
|
|
201
|
+
|
|
168
202
|
### 2b. Extract rows from what you already have
|
|
169
203
|
|
|
170
204
|
```bash
|
|
@@ -180,17 +214,15 @@ under the world's `[api].envelope` key, the bare array when it is `""`), `sessio
|
|
|
180
214
|
and empty-string fields, strips fields the entity does not declare (counted on stderr, never
|
|
181
215
|
silently), dedupes by primary key (`--merge fill` default) and applies the world's `[redact]`
|
|
182
216
|
policy (`drop`, `hash` as `sha256:<hex>` of `<slug>:<value>`, `preserve` as a same-length,
|
|
183
|
-
same-character-class hash for identifiers, `round`) unless `--no-redact`.
|
|
184
|
-
|
|
185
|
-
stderr.
|
|
217
|
+
same-character-class hash for identifiers, `round`) unless `--no-redact`. Output is tagged
|
|
218
|
+
JSONL; the summary `{"read", "rows", "unknownFields", "redacted"}` is on stderr.
|
|
186
219
|
|
|
187
220
|
Platform agent (console): the same `import_world_data`, plus `sandboxPath` for a rows file
|
|
188
221
|
already in the sandbox (any ROWS form above; `entity` for bare arrays/JSONL) instead of
|
|
189
|
-
`rows
|
|
222
|
+
`rows` — build the file with code, then import; do not paste 4 MiB through `rows`.
|
|
190
223
|
`run_gateway_cli({ command: "data show" | "data check" | "data import" | "data extract", args: ["/workspace/<world>", ...] })`
|
|
191
|
-
is the local-directory form for a world you `load_world_to_sandbox`ed
|
|
192
|
-
|
|
193
|
-
`--report`, `--shape`), no remote slugs; land the result with `ingest_to_world`. Capture pages are not rows: `get_connector_capture(downloadTo)` →
|
|
224
|
+
is the local-directory form for a world you `load_world_to_sandbox`ed (same flags as above, no remote
|
|
225
|
+
slugs); land the result with `ingest_to_world`. Capture pages are not rows: `get_connector_capture(downloadTo)` →
|
|
194
226
|
`run_gateway_cli connector ingest` → `ingest_to_world`.
|
|
195
227
|
|
|
196
228
|
### 2c. Turn tool calls into rows: pull, transform, ingest
|
|
@@ -209,7 +241,8 @@ gateway worlds describe <WORLD>; gateway worlds data query <WORLD> <entity> --li
|
|
|
209
241
|
upload rows.json --to seeds/rows.json` then `gateway worlds data import <WORLD> drive:seeds/rows.json`
|
|
210
242
|
is the route from a laptop (or an agent's `request_file_upload`) to a world with nothing staged
|
|
211
243
|
locally: the platform reads the drive file itself (`.json`, `.jsonl`, `.ndjson`, optionally `.gz`;
|
|
212
|
-
|
|
244
|
+
a `.csv` on the drive is refused — import it from the machine that holds it with `--entity`, or
|
|
245
|
+
convert it first) and stages it as one chunk — nothing comes down to your machine and nothing is
|
|
213
246
|
re-uploaded, so `drive:` needs a world slug and no `--dry-run`. The report keeps the drive path under
|
|
214
247
|
`inputs` and the manifest records the digest, size and rows the platform derived. `--to <path>` on `calls pull`, `extract`
|
|
215
248
|
and `session export` lands the output in the drive (`--from drive:<path>` on `calls pull` and
|
|
@@ -329,6 +362,22 @@ redact="off", dry_run=False)` and `ingest_tool_calls`; `capture_tool_calls`. Bri
|
|
|
329
362
|
rows (+ `files` when the tree changed); transforms never run in the bridge. No MCP twin yet;
|
|
330
363
|
the console agent has `run_gateway_cli data ingest` on a sandboxed directory.
|
|
331
364
|
|
|
365
|
+
### 2d. Rows generated or downloaded inside a sandbox
|
|
366
|
+
|
|
367
|
+
```bash
|
|
368
|
+
gateway worlds sandbox exec <id> 'curl -sSL https://example.test/trips.csv -o data/trips.csv' # or a script that writes rows
|
|
369
|
+
gateway worlds sandbox import <id> data/trips.csv --entity taxi_trip [--delimiter ...] [--mode append|replace] [--dry-run]
|
|
370
|
+
gateway worlds sandbox commit <id> -m "tools" # the SOURCE tree only; rows are not in a commit
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
`sandbox import` runs where the project keys are; the **platform** reads the file out of the
|
|
374
|
+
sandbox (`POST /api/public/benchmark-containers/sandboxes/{id}/import`; the sandbox's `gateway`
|
|
375
|
+
chunks it to JSONL in a scratch dir the platform reads and removes) and streams it through the
|
|
376
|
+
same ledger as `data import <slug>`: contract, refusals with pointers, `--mode`, `--merge`,
|
|
377
|
+
`--redact`, `--per-chunk`, `-m`. Prints `{versionId, entityCounts, rowCount}`, or `{batchId,
|
|
378
|
+
state}` to poll with `data status`. A new-world sandbox commits once first. MCP:
|
|
379
|
+
`import_world_sandbox_data({ sandboxId, path, entity?, mode? })`. `commit` never carries rows.
|
|
380
|
+
|
|
332
381
|
### 3. Verify on a live session
|
|
333
382
|
|
|
334
383
|
Open a session, seed it, hit it. Live seeding does not change the world's version; it
|
|
@@ -391,9 +440,8 @@ mapping), or with neither the operations projection for tools whose result is th
|
|
|
391
440
|
response body. The schema, not the tool, decides the shape; `--dry-run` shows the rows before
|
|
392
441
|
anything is written.
|
|
393
442
|
|
|
394
|
-
Sanitize before importing: hash or drop personal fields the world's
|
|
395
|
-
(`
|
|
396
|
-
shared template.
|
|
443
|
+
Sanitize before importing: hash or drop personal fields the world's `[redact]` policy
|
|
444
|
+
(`connector.toml`) would have hashed or dropped. Customer rows never enter a shared template.
|
|
397
445
|
|
|
398
446
|
## Failure modes
|
|
399
447
|
|
|
@@ -408,7 +456,8 @@ shared template.
|
|
|
408
456
|
| a contract error at `/<entity>/<index>/<field>` | type, enum, required or reference broken | fix that row; nothing was written |
|
|
409
457
|
| refused, kind `api` (or `harbor`), naming the file to edit | the world is a generated bundle or a task suite | edit the named source, `bench push` again |
|
|
410
458
|
| `version` missing from the result | a local directory was given | push with `gateway bench push <dir>` when ready |
|
|
411
|
-
| `exceed the inline cap ...
|
|
459
|
+
| `... is over what a world's source tree carries as data (4 MiB / 100000 rows ...)`; `data/initial.json would be N MiB with these rows, over the 4 MiB publication reads a source file whole`; `exceed the inline cap ... without the bundled runtime` | a directory import above the tree cap; or no `python3`, so the rows travel whole (5,000 / 4 MiB) | `worlds sandbox import` from a sandbox, `data import <slug>` with keys (never keys into the sandbox); install python3 |
|
|
460
|
+
| `/<entity>/<i>: <field> = "<cell>" is not a number` (integer, boolean, array, object) | a CSV cell the field's schema type does not fit | fix the cell or the header; nothing was written |
|
|
412
461
|
| exit 2, `state: refused` | the platform refused the batch; `--report` has every record | fix the rows named, `--resume` is not for this: run the import again |
|
|
413
462
|
| `nothing to resume` / `only an open or uploading batch resumes` | no incomplete manifest, or the batch already ran | run the import without `--resume` |
|
|
414
463
|
| `409 base_mismatch` from `/__world/seed` | the overlay named another snapshot digest than the pod is on | the runner posts the digest of the version's snapshot; a mismatch is a version mismatch, never reseed silently |
|
|
@@ -420,29 +469,17 @@ shared template.
|
|
|
420
469
|
| `refused [i] x: no [[sources]] named 'x'` / `… selects or matches records[i]` | a mapping: no source claims the record (no source of that name, or every `when` failed / `select` found nothing) | add the source, or `[ingest] unmatched = "skip"` to count them instead; nothing was written |
|
|
421
470
|
| `<script> exited N` / `timed out after Ns` / `must return {entity: [rows]}` (exit 1) | the transform failed, ran past `--timeout`, or returned the wrong shape | read its stderr (shown prefixed `transform:`); a transform cannot reach the network by design |
|
|
422
471
|
| `accounts has no field nickname; its fields are …` / `carry names parent field id, which the source does not map` | the mapping names what the contract does not have, or a carry the parent never sets | `gateway worlds describe` lists the fields; map the parent field the carry needs |
|
|
423
|
-
|
|
|
472
|
+
| `<file>.csv is a CSV; say which entity its columns fill with --entity` / `--delimiter is one character or \t` | a CSV/TSV without `--entity`; a multi-character delimiter | pass `--entity <name>` (cells cast by its field types); one character or `\t` (a `.tsv` needs none) |
|
|
473
|
+
| `--from csv needs --transform or --map` | `data ingest`: a CSV says nothing about its entity or types | declare the mapping (`type = "number"` for numeric columns) or write the transform |
|
|
424
474
|
| `--from traces needs --trace, --session or --since` | no selection for the observations | say which calls: trace ids, a session, or a time window |
|
|
425
475
|
| `data ingest` reports `skipped: operation x lands no entity` | the operations route: the operation has no `entity` / `results` | give it one in `connector.toml`, or map that tool in `ingest.toml` / the transform |
|
|
426
476
|
| session stays `PROVISIONING` | cold pod on a fresh version | wait up to twenty minutes; `session status` |
|
|
427
477
|
| duplicate rows after a live append on an api world | the world declares no primary keys, so append adds without upserting | use `--mode replace`, or declare keys in `world.json` |
|
|
428
478
|
|
|
429
|
-
## Data snapshots
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
contract) and `scenarios/<name>/state.sqlite` (the default with the scenario's overlays),
|
|
434
|
-
each with a `state.json` sidecar `{contractHash, batchDigest, entityCounts, builtAt,
|
|
435
|
-
runtime, fileSha256}`; the same key is never rebuilt, `schema check` verifies the sidecars,
|
|
436
|
-
and a data batch the contract refuses leaves no snapshot behind. `gateway worlds serve` and
|
|
437
|
-
the api bundle open the snapshot (copy + `SQLiteStore.open`, no row replay) and reset by
|
|
438
|
-
copying it again; without one the JSON is replayed as before. A hosted pod gets
|
|
439
|
-
`WORLD_SNAPSHOT_URL` (presigned GET, `.gz` or `Content-Encoding: gzip` is gunzipped),
|
|
440
|
-
`WORLD_SNAPSHOT_SHA256` and `WORLD_SNAPSHOT_DIGEST`; the server downloads, verifies, writes
|
|
441
|
-
the sidecar and only then serves; a mismatch exits 2 naming the digest. `/__world/state`
|
|
442
|
-
reports `"snapshot": {batchDigest, contractHash}` (or null) and `/__world/seed` takes the
|
|
443
|
-
overlay form `{"base": digest, "rows": {...}, "mode": append|replace}` beside the legacy
|
|
444
|
-
whole-state body. Snapshot files are build outputs: gitignore `db/state.sqlite`,
|
|
445
|
-
`db/state.json`, `scenarios/*/state.*`.
|
|
479
|
+
## Data snapshots
|
|
480
|
+
|
|
481
|
+
Why a served world starts in under a second (`db/state.sqlite`, `WORLD_SNAPSHOT_*`, the
|
|
482
|
+
`/__world/seed` overlay form): [ROWS.md — Data snapshots](ROWS.md#data-snapshots).
|
|
446
483
|
|
|
447
484
|
## Definition of done
|
|
448
485
|
|