gatewaysdk 0.3.2__py3-none-any.whl

gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md

@@ -0,0 +1,293 @@
+---
+name: world-data-ingestion
+description: Put rows into a world from any agent. Use when uploading data to a world, seeding a world, importing rows (a few or millions), adding records to a schema world or mock-tools world, seeding a live session, exporting a session's calls, turning a vendor export or a real run into world data, or asking why a served world starts from a snapshot. Covers the rows file contract (JSON object, JSONL, --entity), append vs replace, primary-key upsert, `gateway worlds data show|import|check|extract|status|batches|publish`, chunked resumable uploads, refusal reports, data snapshots (`db/state.sqlite`, `WORLD_SNAPSHOT_*`), `gateway worlds session seed|export`, the MCP twins import_world_data / seed_world_session / export_world_session, the refusals, and the definition of done.
+---
+
+# World data ingestion
+
+> **Two copies, kept identical.** This file ships inside the `gatewaysdk` Python package
+> (`gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md`) and the `@withgateway/sdk`
+> npm package (copied into `skills/` at build) and is what
+> `gateway worlds skill world-data-ingestion` prints and `--install DIR` writes into a
+> repository's `.claude/skills/`. The repository copy is `.claude/skills/world-data-ingestion/SKILL.md`.
+> Edit one, copy to the other; a test compares them.
+
+
+> **Two copies, kept identical.** This file ships inside the `gatewaysdk` Python package
+> (`gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md`) and the `@withgateway/sdk`
+> npm package (copied into `skills/` at build) and is what
+> `gateway worlds skill world-data-ingestion` prints and `--install DIR` writes into a
+> repository's `.claude/skills/`. The repository copy is `.claude/skills/world-data-ingestion/SKILL.md`.
+> Edit one, copy to the other; a test compares them.
+
+## Purpose
+
+Get rows into a world without knowing the world's file layout. One rows file, one
+command, one MCP tool. The world's contract decides whether a row is accepted; the
+command never writes a partial result.
+
+Read [ROWS.md](ROWS.md) for the full rows file reference and worked examples.
+
+## When to use
+
+- An agent has records (fixtures, a customer's sanitized export, rows harvested from a real
+  run) and a world that should contain them.
+- A world exists but is empty or thin and a scenario needs data behind it.
+- A live session should start from, or be extended with, specific rows.
+- A real run's tool calls should become the seed of the next version.
+
+Do not use this skill to author the world's data model; that is
+`connector-schema-authoring`. Do not use it for vendor captures; that is
+`gateway worlds connector capture` + `ingest` (see `documentation/features/connector-worlds.md`).
+For the end-to-end path (connector, world, session, runs) read `worlds-getting-started`.
+
+## The rows file
+
+Three accepted shapes. Pick one per file; a command may take several files.
+
+| Shape | Looks like | When |
+| --- | --- | --- |
+| JSON object | `{"users": [{...}, {...}], "channels": [{...}]}` | several entities in one file |
+| JSONL, tagged | one line per row: `{"entity": "users", "row": {...}}` | streaming, mixed entities |
+| Bare rows + `--entity NAME` | `[{...}, {...}]` or one bare `{...}` per line | one entity, rows straight from a query |
+
+Rules:
+
+- Entity names are the world's own: the `entities` keys of `schema/world.json` for a schema
+  world, the table names of `db/schema.sql` for a mock-tools world. Unknown entity: refused.
+- A row is a flat JSON object with the entity's field names in the vendor's casing. Unknown
+  field, wrong type, missing required field, broken reference: refused, field named.
+- **Mode `append` (default)** adds rows. A row whose primary key already exists is an
+  update, not a duplicate (upsert by primary key). **Mode `replace`** replaces every row of
+  each entity named in the file; entities the file does not name are untouched.
+- The whole file is checked before anything is written. One bad row means nothing is written
+  (`--atomic`, the default); `--per-chunk` keeps the good chunks of a large upload and lists
+  the refused ones.
+- Inside one batch, two rows with the same key merge by `--merge`: `upsert` (default, the
+  later row replaces the earlier) or `fill` (the first row wins, later rows only add fields
+  it lacks).
+
+## Decision table by world kind
+
+`gateway worlds data show <WORLD>` prints the kind first. Then:
+
+| Kind | How the import lands | What you edit instead |
+| --- | --- | --- |
+| `schema` (has `schema/world.json`) | Directory: a contract-checked batch appended to `data/initial.json`, proven by building the snapshot, then `compile` + `check`. Slug: a data batch on the platform, applied onto the world's snapshot | nothing; the batch is the source |
+| `mock-tools` (has `gateway-env.toml` + `db/schema.sql`) | `INSERT` statements appended to `db/seed.sql` for tables found in `db/schema.sql`; columns come from the table | nothing; `seed.sql` is the source |
+| `api` (has `gateway-api-world.toml` at the root) | Refused: the api bundle is generated from a schema world | the schema world it was packaged from, then `bench push` again |
+| `harbor` (task dirs) | Refused | the task's own fixtures under its task dir |
+| other | Refused, naming the kind and the file to edit | the file named in the refusal |
+
+The refusal for `api`/`harbor` is a result, not an error: it names the world kind and the
+file the agent should edit.
+
+## Commands, in order
+
+Every CLI step has its MCP twin on the next line. `<WORLD>` is a local world directory, else a
+platform world slug (the CLI pulls it to a temp dir, imports, and pushes a new version unless
+`--no-push`). Auth and host: `GATEWAY_HOST`, `GATEWAY_PUBLIC_KEY`, `GATEWAY_SECRET_KEY`, or
+`gateway auth login`.
+
+### 1. See what the world holds
+
+```bash
+gateway worlds data show <WORLD>                    # {world, kind, entity_counts}
+gateway worlds data show <WORLD> --entity users     # the rows of one entity
+```
+
+MCP: no counts tool. `get_environment({ containerId })` lists versions; `import_world_data`
+returns `entityCounts` after a write. Use the CLI to look before writing.
+
+### 2. Import
+
+```bash
+gateway worlds data import <WORLD> rows.json                          # append (default)
+gateway worlds data import <WORLD> users.jsonl --entity users         # bare rows, one entity
+gateway worlds data import <WORLD> rows.json --mode replace           # replace the named entities
+gateway worlds data import <WORLD> rows.json --dry-run                # check only, write nothing
+gateway worlds data import <slug> export.jsonl -m "Q3 tenants"        # platform: chunked upload, new version
+gateway worlds data import <slug> export.jsonl --per-chunk --publish later --report refusals.jsonl
+gateway worlds data import <slug> export.jsonl --resume               # pick a dead upload back up
+gateway worlds data check <WORLD> rows.json --report refusals.jsonl   # the dry run on its own; exit 2 on refusals
+```
+
+**A directory** takes small inputs only (at most 5,000 rows / 4 MiB, the inline cap): the rows
+land in the tree and the snapshot is rebuilt. Prints `{world, kind, entity_counts, changed}`.
+Above the cap it refuses with `import into the platform world <slug>`.
+
+**A slug** streams the files (never loaded whole; `-` is stdin, `.gz` is fine) into gzipped
+JSONL chunks of `--chunk-bytes` (64 MiB default) or `--chunk-rows N`, uploads them
+`--parallel 4` at a time straight to the bucket, and completes the batch; the platform
+validates every row against the contract, applies the batch onto the world's data snapshot
+and, with `--publish now` (default), cuts a data-only version (no image rebuild). `--wait`
+(default) polls to the end and prints the batch: `{batchId, state, rows, refusalCount,
+refusals[:200], entityCounts, versionId?, batchDigest?}`. `--no-wait` prints `{batchId,
+state}` and `gateway worlds data status <batchId>` reads it later. Exit codes: 0 applied or
+published, 2 refused (the report is written when `--report` is given: JSONL of refusal
+records, at most 10,000 then `{"truncated": true, "total": N}`), 1 error.
+
+A manifest at `.gateway/imports/<batchId>.json` records every chunk's digest as it lands;
+`--resume` reopens the newest incomplete batch for that slug and files, re-registers the
+chunks (the platform answers `uploaded: true` for digests it already holds) and uploads only
+the rest. `--publish later` applies without a version; `gateway worlds data publish <slug>`
+folds every pending batch into one. `gateway worlds data batches <slug>` lists them.
+
+A refusal record, everywhere (dry run, report, status, MCP):
+`{"chunk", "index", "entity", "primaryKey", "field", "pointer", "detail"}`, `pointer` being
+`/<entity>/<index>/<field>` inside the chunk.
+
+MCP: `import_world_data({ containerId | slug, rows, mode?, message? })`. `rows` is the JSON
+object shape (`{entity: [rows]}`), at most 5,000 rows and 4 MiB per call: the inline lane,
+one chunk through the same batch job. Returns `{ versionId, entityCounts, deduped }`; a
+refused row is a tool error naming the field, and nothing is written.
+
+### 2b. Extract rows from what you already have
+
+```bash
+gateway worlds data extract <WORLD> calls.jsonl --shape session-export --entity users --out users.jsonl
+gateway worlds data extract <WORLD> response.json --shape vendor-envelope --entity records | gateway worlds data import <slug> - 
+gateway worlds data extract <WORLD> tool-results.jsonl --shape tool-results --entity records --merge upsert --no-redact
+```
+
+Three shapes: `tool-results` (`raw_output.results[]` or `.items[]` of each line, rows that
+carry the key), `vendor-envelope` (the vendor's response, one per line or one document; rows
+under the world's `[api].envelope` key, the bare array when it is `""`), `session-export`
+(`calls.jsonl`; `result.<envelope>[]` of successful `call` records). Every adapter drops null
+and empty-string fields, strips fields the entity does not declare (counted on stderr, never
+silently), dedupes by primary key (`--merge fill` default) and applies the world's `[redact]`
+policy (`drop`, `hash` as `sha256:<hex>` of `<slug>:<value>`, `round`) unless `--no-redact`.
+Output is tagged JSONL; the summary `{"read", "rows", "unknownFields", "redacted"}` is on
+stderr.
+
+Platform agent (console): the same `import_world_data`, plus `sandboxPath` for a rows file
+already in the sandbox (any ROWS form above; `entity` for bare arrays/JSONL) instead of
+`rows`. Build the file with code, then import — do not paste 4 MiB through `rows`.
+`run_gateway_cli({ command: "data show" | "data check" | "data import" | "data extract", args: ["/workspace/<world>", ...] })`
+is the local-directory form for a world you `load_world_to_sandbox`ed — the same TypeScript
+`gateway worlds data …` commands and flags as above (`--entity`, `--mode`, `--merge`, `--dry-run`,
+`--report`, `--shape`), no remote slugs; land the result with `ingest_to_world`. Capture pages are not rows: `get_connector_capture(downloadTo)` →
+`run_gateway_cli connector ingest` → `ingest_to_world`.
+
+### 3. Verify on a live session
+
+Open a session, seed it, hit it. Live seeding does not change the world's version; it
+changes this session only.
+
+```bash
+gateway worlds session open <slug> -t <scenario> --surface api        # prints sessionId and api.url
+gateway worlds session seed <SESSION_ID> rows.json                    # kind:"seed" call, waits for the result
+gateway worlds session seed <SESSION_ID> rows.json --mode replace     # swap the named entities
+curl -H 'x-api-key: any' "$API_URL/<route>"                           # read it back through the vendor's route
+```
+
+Prints `{sessionId, mode, result}`. MCP: `seed_world_session({ sessionId, rows, mode? })` waits
+up to 60 seconds for the engine and otherwise returns a `callId` to poll. Same caps as the
+import. Public API: `POST /api/public/world-sessions/{sessionId}/calls` with
+`{ kind: "seed", rows, mode }`.
+
+The tools surface is open by default; `--surface api` adds the vendor's HTTP routes. A fresh
+version's first session takes about fifteen minutes to come up (`PROVISIONING`, 503 on the
+URL); that is a cold pod, not a fault. Pass `--no-wait` and poll
+`gateway worlds session status <SESSION_ID>`.
+
+Which kinds can seed live: schema worlds (the store applies the batch through the contract),
+api worlds (the runner merges the rows into the whole state and posts it to the world's seed
+route; `append` upserts where the world declares keys), mock-tools task worlds with an
+`environment/schema.sql` (INSERTs against the live SQLite). Any other kind answers
+`this world kind cannot be seeded live`.
+
+Iterate here: seed, run the agent, adjust the rows, seed again with `--mode replace`. When
+the rows are right, `data import` them so the next version carries them.
+
+**Rows that belong to one task, not the world.** A World Host session can take its seed with
+the open: `gateway worlds session open <slug> --task-file task.json`, where the file is
+`{instruction, seed: {mode?, rows}, grader?, metadata?, name?}` and `seed.rows` is this same
+rows contract (`{entity: [row, …]}`, primary-key upsert on `append`). The host lays the rows on
+the fresh clone before the first call, and `session reset <SESSION_ID>` re-applies them, so an
+episode restarts at the task's rows rather than the bare snapshot. Caps: 5,000 rows / 4 MiB per
+task seed; a row for an entity the version's contract does not declare is refused at open with
+the entity's name. SDK: `openSession(slug, { task: { instruction, seed } })` /
+`open_session(slug, {"instruction": …, "seed": …})`. `session state <SESSION_ID>` prints
+every entity's rows as the world holds them now (the dump a grader reads) — the quickest way
+to see what a seed did.
+
+### 4. Export a real run as rows
+
+```bash
+gateway worlds session export <SESSION_ID> --out calls.jsonl
+```
+
+Each line is `{seq, kind, tool, args, result, error, state, createdAt, completedAt}`, the
+session's call log oldest first; `kind` is one of `call`, `grade`, `seed`, `state`, `reset`, `close`.
+MCP: `export_world_session({ sessionId, limit? })` returns the same records. Public API:
+`GET /api/public/world-sessions/{sessionId}/calls` (paged by `cursor` and `limit`).
+
+To turn it into a rows file, keep successful calls whose results are rows of one entity and
+map them to the entity's fields. The schema, not the tool, decides the shape:
+
+```bash
+jq -c 'select(.kind == "call" and .error == null and .tool == "list_users")
+       | .result.users[]? | {entity: "users", row: .}' calls.jsonl > users.jsonl
+gateway worlds data import <WORLD> users.jsonl
+```
+
+Sanitize before importing: hash or drop personal fields the world's redaction policy
+(`[redact]` in `connector.toml`) would have hashed or dropped. Customer rows never enter a
+shared template.
+
+## Failure modes
+
+| You see | Cause | Do |
+| --- | --- | --- |
+| `<file> is a bare array; say which entity it holds with --entity` | bare rows without `--entity` | add `--entity NAME` |
+| `<file> maps entities; drop --entity or make the file a bare array` | object shape given with `--entity` | drop the flag |
+| `Unknown entity X; the world has a, b, c` | name is not in `schema/world.json` | `data show` lists entities; fix the name |
+| `No table X in db/schema.sql; tables: ...` / `Column c is not in table t; columns: ...` | mock-tools name mismatch | fix the name; nested values must be encoded as text yourself |
+| `Row is missing primary key field 'id'` at `/users/3` | a row lacks its key | add the key; nothing was written |
+| a contract error at `/<entity>/<index>/<field>` | type, enum, required or reference broken | fix that row; nothing was written |
+| refused, kind `api` (or `harbor`), naming the file to edit | the world is a generated bundle or a task suite | edit the named source, `bench push` again |
+| `version` missing from the result | a local directory was given | push with `gateway bench push <dir>` when ready |
+| `exceed the inline cap ... import into the platform world <slug>` | a directory import above 5,000 rows / 4 MiB | import into the platform slug (chunked), or `--dry-run` to check locally |
+| exit 2, `state: refused` | the platform refused the batch; `--report` has every record | fix the rows named, `--resume` is not for this: run the import again |
+| `nothing to resume` / `only an open or uploading batch resumes` | no incomplete manifest, or the batch already ran | run the import without `--resume` |
+| `409 base_mismatch` from `/__world/seed` | the overlay named another snapshot digest than the pod is on | the runner posts the digest of the version's snapshot; a mismatch is a version mismatch, never reseed silently |
+| `world snapshot refused: ... sha256` at pod start | `WORLD_SNAPSHOT_SHA256` does not match the download | the platform must hand out the snapshot's own sha256; the pod exits 2 rather than serve the wrong data |
+| `this world kind cannot be seeded live` | the session's engine has no seed door | import into the world instead and open a new session |
+| `A seed call carries at most 5000 rows` / `exceed 4 MiB` | one call too large | split the file |
+| session stays `PROVISIONING` | cold pod on a fresh version | wait up to twenty minutes; `session status` |
+| duplicate rows after a live append on an api world | the world declares no primary keys, so append adds without upserting | use `--mode replace`, or declare keys in `world.json` |
+
+## Data snapshots (why a served world starts in under a second)
+
+A schema world's validated database is kept, not rebuilt. `gateway worlds schema compile`
+builds `db/state.sqlite` (the default snapshot: `data/initial.json` applied through the
+contract) and `scenarios/<name>/state.sqlite` (the default with the scenario's overlays),
+each with a `state.json` sidecar `{contractHash, batchDigest, entityCounts, builtAt,
+runtime, fileSha256}`; the same key is never rebuilt, `schema check` verifies the sidecars,
+and a data batch the contract refuses leaves no snapshot behind. `gateway worlds serve` and
+the api bundle open the snapshot (copy + `SQLiteStore.open`, no row replay) and reset by
+copying it again; without one the JSON is replayed as before. A hosted pod gets
+`WORLD_SNAPSHOT_URL` (presigned GET, `.gz` or `Content-Encoding: gzip` is gunzipped),
+`WORLD_SNAPSHOT_SHA256` and `WORLD_SNAPSHOT_DIGEST`; the server downloads, verifies, writes
+the sidecar and only then serves; a mismatch exits 2 naming the digest. `/__world/state`
+reports `"snapshot": {batchDigest, contractHash}` (or null) and `/__world/seed` takes the
+overlay form `{"base": digest, "rows": {...}, "mode": append|replace}` beside the legacy
+whole-state body. Snapshot files are build outputs: gitignore `db/state.sqlite`,
+`db/state.json`, `scenarios/*/state.*`.
+
+## Definition of done
+
+- [ ] `gateway worlds data show <WORLD>` before and after; the counts moved by what you sent.
+- [ ] The import printed `entity_counts` (directory) or `state: applied|published` with a `versionId` (slug); a refused batch has its `--report`.
+- [ ] A session opened on that version answers the rows (tools surface or vendor route).
+- [ ] Rows carry no unsanitized customer data; `captures/` and `data/` stay out of templates.
+- [ ] If the rows came from a run: `calls.jsonl` is kept beside the rows file with the session id.
+
+## Related
+
+- `worlds-getting-started`: the whole path, auth to runs, CLI with MCP twins.
+- `connector-schema-authoring`: the data model the rows are checked against.
+- `documentation/features/world-data-ingestion.md`: what exists, the contract, code pointers.
+- `documentation/features/connector-worlds.md`: captures, ingest, serving the vendor's API.
+- Consumer page: `gateway_documentation/pages/worlds/data.mdx`.