gatewaysdk 0.3.2__py3-none-any.whl

gatewaysdk/environment/schema/skills/connector-schema-authoring/SKILL.md

@@ -0,0 +1,313 @@
+---
+name: connector-schema-authoring
+description: Checklist for creating or updating an SDK connector schema contract (Microsoft Teams, Slack, GitHub, Salesforce, Microsoft 365 / Entra ID, or any connector). Use when adding templates, authoring vendor schemas, choosing fidelity shape, recording provenance, or validating fixtures through the schema-aware store. Also covers optional tenant overlays, tools, MCP comparison, live evidence, and later platform integration, distinguishing implemented tooling from future work. Covers gateway-world/1, schema init/compile/check, store.ingest, honest fidelity assessment, and the definition of done.
+---
+
+> **For agents.** This copy ships inside the `gatewaysdk` Python package and is installed
+> into a repository by `gateway worlds connector skill --install .` (with the
+> `connector-template-author` agent under `.claude/agents/`). Author the template with the
+> checklist below; when it compiles and conforms, `gateway worlds connector publish <dir>`
+> turns it into a workspace connector — an org-owned, versioned object on the platform that
+> worlds are initialized from (`gateway worlds schema init ./x --connector workspace:<slug>`)
+> and that can be published to every organization and cloned. The `briefs/` directory and
+> the repository-relative links below are not shipped; the checklist stands on its own.
+
+# Connector Schema Authoring
+
+## Purpose
+
+Author an SDK schema contract with documented scope and fidelity evidence.
+The **data model is the deliverable**; tools are optional consumers.
+The checklist distinguishes implemented commands from future tooling.
+
+Read first, in this order:
+
+1. [world-schema.md](../../../documentation/features/world-schema.md): the `gateway-world/1` contract, store API, raw ingestion, templates.
+2. [connector-capture-and-schema-handoff.md](../../../documentation/architecture/connector-capture-and-schema-handoff.md): the implemented capture boundary and future mapper contract.
+3. [CONNECTORS.md](CONNECTORS.md) in this folder: per-connector shape, model source, tenant channel, sandbox.
+4. [briefs/](briefs/README.md): one self-contained work order per connector (sources to pin, entities, tools, errors, sandbox, definition of done); the `connector-template-author` agent implements one brief.
+5. [CONVENTIONS.md](CONVENTIONS.md) in this folder: the generation standard every template follows (identity, prefixes, keys, types, tools, data, layout), enforced by `test_committed_templates_follow_the_generation_conventions`.
+6. [gateway-tool-access.md](../../../documentation/integrations/gateway-tool-access.md) only if the world declares an MCP surface.
+
+The formerly referenced `connector-fidelity-scope.md` and
+`connector-platform-infra-plan.md` are absent from this checkout. Do not reconstruct
+them or treat their old work-item identifiers as authoritative. Verify current code
+and scope the milestone explicitly: SDK schema authoring does not require provider
+registration, a platform builder, live ingestion, tools, or a registry framework.
+
+## The rules that never bend
+
+- **The schema is the deliverable.** A world with a faithful model and no tools is complete. Tools are added when wanted, by whoever wants them.
+- **Every write goes through the schema.** Handlers use `gateway_env.store`; ingestion uses `store.apply_batch` or `store.ingest`; nothing else writes. Out-of-contract writes are rejected with the reason.
+- **Check the right boundary.** Compile/check validate compiled artifacts; materialization and store operations validate persisted data. Publication checks actual bundle bytes, handler declarations and initial state through the trusted runtime. Fidelity evidence and Environment Actions remain advisory.
+- **Fidelity shape is decided by the vendor, not by a house rule.** A **base** connector is prebaked once (Slack). A **tenant-shaped** connector's model is mostly customer-defined, so describe at bind time is required (Salesforce, NetSuite, Workday). A **mixed** connector is a base plus an optional overlay (Okta, Entra, Google, LDAP, Jamf).
+- **Surface parity is advisory.** If the world declares it mirrors an MCP surface, produce the parity report and show it. Never block on it, never shrink or bend the model to satisfy it.
+- **The label is honest.** Compilation and synthetic fixtures do not establish live fidelity or behavioral equivalence. Use a provisional `subset` assessment when coverage or evidence is incomplete; identify synthetic fixtures as synthetic. Never invent replay scores or claim `mimic` without measured supporting evidence. A check that could not run is `skipped`, never `passed`.
+- **Customer data never enters a shared template.** Sanitize captured goldens; keep tenant overlays private.
+- **Anyone can author a contract.** SDK-shipped and user-authored contracts use the same compiler/store checks. Platform registration is a separate milestone.
+- **A customer's connectors are the customer's.** A world we build for a customer against their third-party providers is their custom connector: it is authored from `--connector custom` (or their own template) in their project or untracked staging, pushed to their project, and never merged as an SDK template, catalog row or reserved prefix. The repo ships the abstraction (`connector.toml`, capture, ingest, the world's API), not their worlds.
+
+## Tooling status (2026-09-07)
+
+| Command or path | Status |
+| --- | --- |
+| `gateway worlds schema init PATH --connector <slug> [--name NAME]` | exists (#219); `--connector custom` is the blank starting point for an arbitrary connector |
+| `gateway worlds connector validate\|capture\|ingest PATH` | exists; a `connector.toml` in the world describes base URL, auth scheme, pacing, redaction and operations; `capture` records paced, redacted pages (errors included) under `captures/` with the secret never written; `ingest` seeds `data/initial.json` through the contract in a scratch copy first |
+| `gateway worlds connector conform PATH --openapi vendor.yaml` | exists; serves the world in-process, calls every declared operation with arguments from its own seed, and checks status + body against the vendor's OpenAPI response schemas; non-zero exit on any mismatch. Run it before calling a world 1:1; its FAIL list is the work list (unseeded entities → 404s, creates the contract rejects → 4xx, honest 501s) |
+| `gateway worlds connector capture PATH --op … --remote` | exists; the platform runs the capture (web pod egress = the allow-listed NAT IP) with `gateway bench secrets set` credentials; pages land in the drive under `worlds/<slug>/captures/` and are pulled into the world's `captures/` with the manifest merged |
+| `gateway worlds serve PATH` / `gateway bench push PATH` | exists; `serve` answers the vendor's routes from the world's store locally (stdlib only); `bench push` on a world carrying `connector.toml` packages the api-kind bundle the platform hosts itself (`gateway worlds schema api-bundle` writes it explicitly) |
+| `gateway worlds schema compile \| check \| describe PATH` | exists (#216) |
+| `gateway worlds schema entity add PATH NAME --ref D \| --fields "a:string,b:integer?" \| --schema JSON --key F [--unique F,F] [--relationship F->entity]` | exists (TS CLI, local); declares one entity in `schema/world.json` without hand-editing the document: keeps `$defs`, package metadata and other entities byte-for-byte in meaning, refuses duplicate names, non-required or non-string/integer keys, unknown `$defs` and undeclared relationship targets, and writes nothing on refusal |
+| `gateway worlds connector route add PATH NAME --path /v1/x/{id} [-X POST] [--param q] [--entity E] [--results r] [--action create] [--filter id=id] [--body f=p] [--handler fn]` | exists (TS CLI, local); appends one `[[operations]]` (+ `[operations.api]`) to `connector.toml` leaving existing bytes untouched; validates name, method, path, entity against `schema/world.json`, filter keys against `{args}`/params, and the handler/action pairing |
+| `gateway worlds connector publish DIR [--slug S] [-m MSG] [--tag T] [--workspace-visible] [--project-scope]` | exists; saves a template directory, or the world it was authored in, as a workspace connector (Context Hub `connector` repo, org-owned, versioned); the platform compiles the template before accepting it and answers 400 with the reason; a re-publish is a new commit and paths no longer present are removed; `data/`, `captures/`, `db/` and generated files stay behind |
+| `gateway worlds connector list [--hub] [--query Q]` / `pull SLUG [--out DIR] [--version V]` / `clone SLUG-OR-ID [--name N] [--slug S]` | exists; `list` shows this org's connectors (vendor, auth, entities, operations), `--hub` adds every connector another org published; `pull` writes the template files; `clone` copies a published connector into this org with provenance — a live reference is never made, so the publisher's later edits need a re-clone |
+| `gateway worlds schema init PATH --connector workspace:<slug>` | exists; initializes from the platform's copy through `init_bundle_from_files` (needs `GATEWAY_*` keys); the offline `python -m gatewaysdk.environment.schema init` refuses `workspace:` and names this command |
+| `gateway worlds connector skill [--install DIR] [--force]` | exists; prints this skill, or installs it (SKILL.md, CONVENTIONS.md, CONNECTORS.md) plus the `connector-template-author` agent under `DIR/.claude/`; never overwrites a differing file without `--force`; the packaged copy is `gatewaysdk/environment/schema/skills/` and a test keeps it in sync with this file — re-copy it (with its "For agents" note) after editing here |
+| `world.store.ingest(...)`, `store.apply_batch(...)` | exists (#219, #216) |
+| Connector templates under `gatewaysdk/sdk/src/gatewaysdk/environment/schema/connectors/` | authored templates; see [CONNECTORS.md](CONNECTORS.md), including the schema-only Microsoft Teams base |
+| `gateway worlds schema import PATH --from <kind> --spec <url\|path> [--scope scope.toml]` | not implemented; author the contract and provenance manually |
+| `gateway worlds schema parity PATH --surface <tools-list.json> \| --connection <id>` (advisory report) | not implemented |
+| `gateway worlds playground <connector>` | not implemented |
+| Platform upload of a JSON Schema, OpenAPI, or DDL as a new connector | not implemented as a UI upload; the platform agent's `create_custom_world` takes an authored `gateway-world/1` document, inline files, a sandbox tree or `--connector`-style `template` and publishes it through the trusted `compile` bridge (2026-09-09) |
+| Platform agent record ingestion (`ingest_records_to_world` → `platform.py apply-batch`, `ingest_to_world` file writes with recompile, `describe_world_schema`, `load_world_to_sandbox`) | exists; schema-gated, `batchId` idempotent, one immutable READY version per call, project-scoped; see `documentation/features/agent.md` |
+| SDK/public and platform publication | validates assembled files and actual stored archive bytes through the trusted compiler/materializer, including READY/dedup/ref paths |
+| Connectors Slack world creation | durable structured capture → shared Python mapper → immutable publication; explicit synthetic mode |
+| Worker schema tests/workbench | shared validation and isolated E2B author execution; missing isolation is non-green |
+| Hosted schema tools sessions | standalone `mock-tools` worlds use the shared `World` child for calls, grading and reset; composed dependencies and API/Harbor/verifiers schema engines remain unsupported |
+
+Do only the manual work authorized by the milestone; record unavailable tooling and
+skipped live checks without expanding a schema task into connector implementation.
+
+## The checklist
+
+Copy the applicable items into the PR description. Sections 2, 3, 5, 6, and platform
+registration apply only when the milestone includes those capabilities. A schema-only
+deliverable can finish with synthetic fixtures and explicitly skipped live validation.
+
+### 0. Decide the shape
+
+- [ ] **Fidelity shape**: base, tenant-shaped, or mixed. See [CONNECTORS.md](CONNECTORS.md). Tenant-shaped means describe-on-bind is a requirement of the world, and the shared template is only a skeleton.
+- [ ] **Model source**: authored by hand (start from `--connector custom`), imported (`openapi`, `odata-csdl`, `discovery`, `wsdl-xsd`, `ldap-subschema`, `describe`, `docs`), or uploaded. Pin the spec URL and version now.
+- [ ] **Scope of the model**: which entities and fields, decided from what the connector's users need the world to be faithful about. Record the reasoning in `scope.md` or `scope.toml`; coverage of the vendor's full model is reported, not required.
+- [ ] **Sandbox**: record whether live validation is available and in scope. If it is not, record the reason and mark live checks `skipped`; a schema foundation can still ship as provisional `subset`.
+- [ ] **Tools and surface** (optional): none, generated defaults, hand-written, user's custom, or mapped to an MCP surface. Decide now so the entity scope covers what those tools touch.
+
+### 1. Build the model
+
+- [ ] Author `schema/world.json` and reusable sibling definitions from pinned vendor sources, following the type-mapping notes in [CONNECTORS.md](CONNECTORS.md). `gateway worlds schema entity add` declares an entity with its key, uniques and relationships in one command so the declaration shape is never wrong; hand-author only the payload definitions (`--schema` or `$defs`). Record source URLs, versions/retrieval dates, and hashes of the actual retrieved bytes in `schema/provenance.json`.
+- [ ] Entities have the vendor's keys, relationships, enums, and nullability, and follow [CONVENTIONS.md](CONVENTIONS.md): one reserved prefix per template, plural snake_case names, vendor casing on fields, shared `base.json` primitives. Reserved prefixes and the `payload` name are avoided.
+- [ ] Enforce supported required fields, types, enums, key uniqueness, and reference integrity. Document vendor rules the dialect cannot express, normalization choices, and constraints the future mapper must enforce; structural validation is not vendor behavior.
+- [ ] `gateway worlds schema compile PATH` then `check PATH` pass. Never edit generated files; SDK templates contain authored inputs and generate bundle artifacts at `init` time.
+- [ ] Shared templates and committed fixtures hold synthetic data only. Tenant captures stay in their owning project; initial batches and refresh journals retain explicit capture/synthetic provenance.
+
+### 2. Tenant overlay (required for tenant-shaped, optional for mixed, absent for base)
+
+- [ ] No automatic overlay/describe-on-bind implementation exists. When tenant integration is in scope, run the vendor describe manually through an authorized connection and document the authored overlay process.
+- [ ] The overlay is additive only: new entities, fields, enum values. A retype or removal is drift and becomes a proposal on the template.
+- [ ] The overlaid world compiles and passes `check`; its lock differs from the template's.
+- [ ] For tenant-shaped connectors, the badge states that an unbound world is a skeleton.
+
+### 3. Playground and evidence
+
+- [ ] Keep sandbox credentials in authorized secret storage, never in a shared template or bundle. Do not assume an internal playground project has been provisioned.
+- [ ] Seed synthetic records through the vendor write API where the sandbox allows it (seedable table in [CONNECTORS.md](CONNECTORS.md)); seeding is idempotent.
+- [ ] Pull records back through the vendor API and land them with `store.ingest(source=..., mapping=...)`; `audit()` passes with zero violations.
+- [ ] Distinguish captured, sanitized vendor goldens from wholly synthetic fixtures. Record actual replay methods and results when available; no replay runner or universal score threshold is implemented here.
+- [ ] Production evidence requires an authorized capture workflow and sanitized records; do not infer that a generic grant flag or vendor mapper already exists.
+
+### 4. Fidelity report
+
+- [ ] `fidelity.json` records shape, assessment, scope coverage, source identity, and measured or skipped checks with reasons. Include a replay score only when replay ran; include parity only when a surface is declared.
+- [ ] The label follows the checks. Reasons for `subset` are listed.
+- [ ] A check that could not run is `skipped`.
+
+### 5. Tools (only if the world ships tools)
+
+- [ ] Every tool in `x-gateway.tools` has exactly one handler in `tools/*.py` (the loader refuses a mismatch); generated defaults are fine.
+- [ ] Handler reads use `gateway_env.store.get` / `list`; writes use `insert` / `update` / `delete`. Read-only SQL uses `gateway_env.db.one/all`; raw SQL writes are denied. The host's `SQLiteStore.query` is not exposed by the handler store proxy.
+- [ ] Handlers never read `benchmark_runtime_context` or `benchmark_conversation_events`.
+- [ ] A user attaching custom tools to this world needs nothing from this checklist beyond the store API; document any conventions the model expects (key formats, enum values).
+
+### 6. Surface binding (only if the world declares it mirrors an MCP surface)
+
+- [ ] Export a real connection's filtered `tools/list` through the MCP proxy under a grant, or inspect its `AgentSecurityResource` tool rows. Native Slack source capture has no live MCP catalog; its schema template tools are separate offline behavior.
+- [ ] Produce an advisory comparison of names and input schemas, recording coverage and mismatches in `parity.json`. Do not alter the model to satisfy it; report unsupported tool-contract constructs explicitly.
+- [ ] Compare read-only tools manually against a live connection only when such comparison is authorized and in scope.
+- [ ] Native manifest changes bump `TOOL_VERSIONS`.
+
+### 7. Ship the template; register platform integration only when in scope
+
+- [ ] SDK-shipped: commit authored files under `gatewaysdk/sdk/src/gatewaysdk/environment/schema/connectors/<slug>/`; `list_templates()` and `test_schema_scaffold.py` cover the slug and preserve its authored dependencies/provenance during initialization. Schema-only templates declare an empty tools map and have no operation handlers.
+- [ ] Platform integration, if separately in scope: map an existing source kind in `schemaWorldTemplates.ts` so "Build from connector" offers it. Do not register a native provider or invent a source kind for an SDK-only schema milestone.
+- [ ] Treat `connectionMode` as the primary authentication path, not the only setup path. A managed connector may retain `remoteMcpPreset` as a fallback; render that action from the preset's presence. For native source connectors such as Slack, the source-backed Link action is the managed authentication path because native-provider dispatch intentionally precedes Merge.
+- [ ] User-added templates can use `--connector custom` and SDK publication. Generic schema upload and shared/project registry infrastructure remain unimplemented.
+- [ ] Record shape, assessment, coverage, source hashes, and limitations in the PR and authored metadata.
+- [ ] Docs updated: `world-schema.md` template list, `environment-files.md` if the panel changed, [CONNECTORS.md](CONNECTORS.md) row.
+- [ ] Pin source versions/retrieval dates and document a revalidation procedure; automated vendor drift checks are not supplied by this checklist.
+
+### 8. Definition of done
+
+- [ ] Relevant schema test suites and required scoped checks pass, including the slug in the scaffold suite if SDK-shipped. Report exact commands and results.
+- [ ] Synthetic fixture round trips exercise the store boundary. Live round trips either have recorded results or a dated `skipped` reason; never claim them from offline tests.
+- [ ] For tenant-shaped integration milestones, an overlay comes from an authorized tenant; a schema-only skeleton states that limitation. Base connectors require no tenant overlay.
+- [ ] Fixtures identify their origin; captured goldens are sanitized and distinct from synthetic examples. No customer data enters the shared template.
+- [ ] Shape, label and coverage in the PR description; parity summary if a surface is declared.
+- [ ] Review flags resolved; report observed CI status without claiming checks that did not run; docs and [CONNECTORS.md](CONNECTORS.md) updated.
+
+## Runtime authoring boundary
+
+When adding a bridge command to `schema/platform.py`, mirror it as a typed wrapper in
+`worldSchemaRuntime.ts` (`compileWorldFiles` / `applyWorldBatch` are the pattern: file map
+in, `{entries, report, ...}` out, errors as `SchemaBridgeError`), cover it in
+`tests/environment/test_schema_platform.py`, and let the agent tool call the wrapper, never
+the subprocess. The bridge needs Python ≥ 3.11 (`tomllib`); set `GATEWAY_SCHEMA_PYTHON`
+when the default interpreter is older.
+
+Use `worldSchemaRuntime.ts` for platform validation and `schema.platform` for
+the Python bridge. Publication parses bundle Python without importing it on the
+web or worker host. Schema workbench edits use `/workspace/schema_edit.py` with
+an explicit batch; publish the resulting ordered `data/batches/*.json` journal,
+not a raw database dump. The hosted session owns an isolated shared `World` and
+receives no provider tokens. API/Harbor/verifiers schema runtimes and hosted
+dependency composition retain explicit rejection.
+
+## Implementing many connectors in parallel
+
+Once a brief exists for every connector, spawn one `connector-template-author` agent
+per brief (`.claude/agents/connector-template-author.md`). Overlap is prevented by
+ownership, not by coordination:
+
+- Each agent writes only `connectors/<slug>/`, its own rows in `CONNECTORS.md`, and its
+  own line in `PREFIXES`. Prefixes for the whole catalog are reserved in
+  `test_schema_scaffold.py` before any agent starts.
+- Shared files (`base.json`, `scaffold.py`, compiler, store, `schemaWorldTemplates.ts`)
+  are orchestrator-only. An agent that needs a shared change stops and reports it.
+- The gate is the same for every branch: `test_committed_templates_follow_the_generation_conventions`
+  plus the full `tests/environment` suite. One PR per connector, stacked or independent.
+- The orchestrator merges in wave order, re-runs the suite after each merge, and reconciles
+  `CONNECTORS.md` if two agents touched adjacent rows.
+
+Full procedure: `.llm_artifacts/plans/connector-implementation-orchestration.md`.
+
+## Common mistakes
+
+- Treating the MCP tool list as the model. The model comes from the vendor's schema and the tenant; tools are mapped onto it afterwards, if at all.
+- Blocking or bending the model because a declared surface does not match. Report the mismatch; the model stays faithful to the system.
+- Shipping a tenant-shaped connector (Salesforce, NetSuite, Workday) as if its skeleton were the model. Describe-on-bind is required there.
+- Adding a required overlay step to a base connector. Slack and the Microsoft Teams collaboration base do not require tenant schema discovery.
+- Skipping the round trip silently when there is no sandbox. Record the blocker and set the label.
+- Committing captured production records as goldens without sanitizing them.
+- Assuming a planned command exists. Check current tooling and perform only in-scope manual work.
+- Treating descriptive resources as schema authority. `init` preserves flat supporting `.json`, `.md` and `.toml` files; only referenced schemas become compiler inputs. Slack capability defaults expand into self-contained entries; see [world-schema.md](../../../documentation/features/world-schema.md#connector-templates-and-init).
+- Converting Slack message timestamps to floats or RFC3339. Preserve exact `(workspace_id, channel_id, ts)` keys; see the [Slack model guide](../../../gatewaysdk/sdk/src/gatewaysdk/environment/schema/connectors/slack/MODEL.md) for normalization and limits.
+
+## Where this skill is discoverable
+
+This file is the source. It is not published to the platform Context Hub, but it ships with the
+Python SDK as package data so an agent in any repository can install it.
+
+| Harness | How it finds this skill |
+| --- | --- |
+| Claude Code | `skill-rules.json` triggers (prompt keywords, intent patterns, file paths) and the Skills table in `CLAUDE.md` |
+| Codex | the Skills table in `AGENTS.md` |
+| Cursor | `.cursor/rules/connector-schema-authoring.mdc` (agent-requestable, points here) |
+| Any harness, any repository | `gateway worlds connector skill --install .` writes the packaged copy (this file + CONVENTIONS.md + CONNECTORS.md and the `connector-template-author` agent) under that repository's `.claude/`; `briefs/` is not shipped |
+
+## Related
+
+- [session-trace-parity](../session-trace-parity/SKILL.md): unrelated parity rule for sessions and traces.
+- [Microsoft Teams schema](../../../documentation/features/microsoft-teams-schema.md): schema-only collaboration scope, source provenance, and future mapper obligations.
+- [prisma-migration-patterns](../prisma-migration-patterns/SKILL.md): only when separately authorized platform infrastructure requires database migrations.
+- [backend-dev-guidelines](../backend-dev-guidelines/SKILL.md): for the tRPC route and worker queue conventions the platform pieces follow.
+
+## Source verification
+
+Revalidated 2026-09-07 while integrating `ba89fe16` with `015d6b25`:
+`schema/scaffold.py`, `schema/platform.py`, `tests/environment/test_schema_scaffold.py`,
+`worldSchemaPublication.ts`, `worldSchemaRuntime.ts`, `environmentBundleTester.ts`,
+`environmentBuildWorkbench.ts`, `environmentSchemaSandbox.ts`,
+`NativeSlackService.ts`, and `deploy/benchmark-runner/engines/{session_server,schema_session}.py`.
+Full source paths and runtime boundaries are listed in [world-schema.md](../../../documentation/features/world-schema.md).
+These source checks do not imply live-provider validation or executed fidelity scores.
+
+## Custom connections (any vendor with an API key)
+
+A world can describe the vendor it mirrors in `connector.toml` at its root: base URL,
+auth scheme (`header`, `bearer`, `basic`, `query`, `oauth2`, `none`) with the secret named by an
+environment variable, capture pacing, a redaction policy (`hash` or `drop` fields at any
+depth), and the operations the agent actually calls (method, path with `{args}`, query
+params, and the entity plus results path their rows land in). The same file is meant to
+drive the platform's custom source later, so describe once.
+
+- `gateway worlds connector validate PATH` parses the spec and lists operations.
+- `gateway worlds connector capture PATH --op NAME --arg k=v --param k=v` calls the live
+  connection, follows the spec's cursor up to `max_pages`, and writes every page under
+  `captures/<op>/` plus `captures/manifest.json` (sha256 per file). The recorded request
+  never carries the secret; bodies are redacted before they touch disk; non-2xx replies are
+  recorded too, since the vendor's error bodies are part of the contract, and end the run.
+- `gateway worlds connector ingest PATH` turns the successful captures into the world's
+  `data/initial.json` batch (rows deduped by primary key against what the batch already
+  holds: new rows insert, changed rows update, unchanged rows are skipped), proves the batch
+  applies in a scratch copy, then writes it and recompiles. Nothing new means nothing written.
+
+Three refinements the first three vendors forced:
+
+- **Vendors inline what the convention splits.** A user row that carries `channels[]`
+  does not fit a closed `users` entity when memberships are their own entity. Per
+  operation, `[operations.ingest] drop = ["channels"]` removes the inline collection from
+  the parent, and `[[operations.ingest.explode]]` turns each item into a child row:
+  `field`, `entity`, `carry = { user_id = "id" }` (child field from the parent row) and
+  `take = { channel_id = "entity_id" }` (child field from the item). Children are inserted
+  first; keys dedupe against the batch like any other row.
+- **Multi-part credentials.** `[auth] headers = { "X-App-Token" = "VENDOR_APP_TOKEN" }`
+  adds credential headers whose values come from the named env vars, beside the primary
+  secret. A missing variable refuses the capture rather than sending a partial credential.
+- **Query-driven lookups.** `[operations.api] filter` may name a declared query parameter
+  as well as a path argument; a parameter the caller leaves out does not narrow.
+
+Be sparing with live calls; providers ask for it and `requests_per_minute` is the knob.
+Commit `connector.toml`; commit captures only after checking the redaction policy covers
+every PII field the vendor returns. The platform half exists: the key is held encrypted on
+the world (Secrets tab / `gateway bench secrets set`), `gateway worlds connector capture --remote`
+runs the same operations server-side into the project's file drive, and the CLI pulls the
+redacted pages back into `captures/`.
+
+### From the platform agent (no checkout)
+
+The console agent has this loop as tools, all discoverable with `search_tools`:
+`get_connector_authoring_skill` (this file) → write `connector.toml` + `schema/` in the
+sandbox → `run_gateway_cli({ command: "connector validate" | "connector conform" | "schema check", args: [...] })`
+→ `validate_connector_template({ files })` → `publish_connector({ slug, files, parentCommit? })`
+(needs `hub:write` and the session's `allowConnectorWrites`, else `confirm: true`; a moved
+head is refused by `parentCommit`, re-read with `get_connector`) → `init_world_from_connector`
+→ `start_connector_capture({ containerId, op, args, params })` → `get_connector_capture({ …, downloadTo })`
+→ `run_gateway_cli connector ingest` → `ingest_to_world`. `list_world_secrets` shows the env
+var names the world holds, never values; a `connector.toml` with an inline credential is
+refused before anything is queued. `run_gateway_cli` runs the TypeScript `gateway worlds …` CLI
+(the same flags as this page) staged into the sandbox, local-only (paths under `/workspace`,
+no `--host`/`--remote`/`workspace:<slug>`, platform env stripped, pinned to its local runtime),
+so publishing always goes through the tools above. To parallelise several connectors, `spawn_subagent` with a brief
+naming only the tools that child needs (`allowSubagents` on the session).
+
+### Serving the vendor's API from the world
+
+A provider's client speaks the vendor's REST paths, not tool calls, so the world answers
+those paths itself. `connector.toml` already names them; add what the world needs to serve
+them:
+
+- `[api] envelope = "results"` wraps rows as `{"results": [...], "hits": N, "cursor": ""}`
+- `[operations.api] action` makes a route a `create`, `update`, `delete` or `handler`; handlers live in the world's `api/handlers.py` and carry computed results and workflow state (`ctx.ladder` on the clock). A world is read *and* write: an agent that creates a search and gets 501 is not testing anything. Full reference: `documentation/features/connector-worlds.md`.
+  (empty means a bare array); `unauthorized` / `not_found` are JSON strings replayed on
+  401 / 404, so the vendor's own error bodies come back.
+- `[operations.api] filter = { email = "attributes.email" }` maps a path argument to the
+  row field it selects on, matched case-insensitively at any depth. An operation without
+  an `entity` answers 501 `not_modelled` rather than inventing a response.
+
+`gateway worlds serve PATH --port 8080 [--api-key K]` runs it locally; the runner's admin
+endpoints (`/healthz`, `/__world/seed`, `/__world/reset`, `/__world/state`) sit beside the
+vendor routes. `gateway bench push PATH` publishes the world: for a world carrying
+`connector.toml` it packages the api-kind bundle the platform hosts on the way (root
+legacy-shaped on purpose, since the api runner refuses a schema bundle there; `server/`
+needs nothing installed). Open a session with `--surface api` and hand the client that
+URL; the vendor's auth scheme applies (`WORLD_API_KEY` on the host pins the credential,
+otherwise any non-empty one).

gatewaysdk/environment/schema/skills/world-data-ingestion/ROWS.md

@@ -0,0 +1,223 @@
+# Rows file reference
+
+Companion to [SKILL.md](SKILL.md). The contract is fixed in
+`.llm_artifacts/plans/world-data-ingestion-plan.md`; this page is the worked form.
+
+## Contents
+
+1. [Shapes](#shapes)
+2. [Modes](#modes)
+3. [Keys and upserts](#keys-and-upserts)
+4. [Schema worlds](#schema-worlds)
+5. [Mock-tools worlds](#mock-tools-worlds)
+6. [Live seeding](#live-seeding)
+7. [From a call log to rows](#from-a-call-log-to-rows)
+8. [MCP payloads](#mcp-payloads)
+9. [Chunks, batches and refusal records](#chunks-batches-and-refusal-records)
+10. [Extract adapters](#extract-adapters)
+
+## Shapes
+
+**JSON object, entity to rows.** One file, many entities.
+
+```json
+{
+  "users": [
+    {"id": "U1", "name": "Ada", "email": "ada@example.com"},
+    {"id": "U2", "name": "Lin", "email": "lin@example.com"}
+  ],
+  "channels": [
+    {"id": "C1", "name": "general", "created_by": "U1"}
+  ]
+}
+```
+
+**JSONL, tagged.** One row per line, each line names its entity.
+
+```jsonl
+{"entity": "users", "row": {"id": "U1", "name": "Ada", "email": "ada@example.com"}}
+{"entity": "channels", "row": {"id": "C1", "name": "general", "created_by": "U1"}}
+```
+
+**Bare rows with `--entity NAME`.** A JSON array, or JSONL of bare objects.
+
+```bash
+gateway worlds data import ./world users.json --entity users
+gateway worlds data import ./world users.jsonl --entity users
+```
+
+A file that mixes shapes is refused. Several files on one command are applied as one batch:
+all or nothing (`--per-chunk` on a platform import relaxes that to one chunk at a time). Files
+are streamed, so size is not a concern: pass `-` for stdin, `.gz` files are gunzipped.
+
+## Modes
+
+| Mode | Named entities | Other entities |
+| --- | --- | --- |
+| `append` (default) | rows added; existing keys updated | untouched |
+| `replace` | every existing row removed, then the file's rows inserted | untouched |
+
+`replace` on a live session swaps the named entities in the running state; the rest stays.
+`replace` on a world rewrites only the named entities in `data/initial.json` (schema) or
+`db/seed.sql` (mock-tools).
+
+## Keys and upserts
+
+The primary key is the entity's `primaryKey` in `schema/world.json`, or the table's
+`PRIMARY KEY` in `db/schema.sql`. On `append`, a row whose key already exists updates the
+existing row; an identical row is skipped; a new key inserts. Order inside a file does not
+matter for keys, but referenced rows must exist by the end of the batch (a `channels` row
+whose `created_by` names an unknown user is refused).
+
+A row that lacks its entity's key field is refused (`Row is missing primary key field 'id'`).
+Where the contract declares no key at all (a hand-built api world), a live `append` adds
+without upserting; use `replace` there.
+
+## Schema worlds
+
+The import builds an `apply_batch` payload and proves it in a scratch copy first:
+
+```json
+{
+  "id": "<package id>.initial.v<N>",
+  "target": "<contract target>",
+  "mapping": {"id": "data-import", "version": "1"},
+  "coverage": "partial",
+  "mutations": [
+    {"op": "insert", "entity": "users", "value": {"id": "U1", "...": "..."}, "source": {"record": "data-import:users:<identity>"}}
+  ]
+}
+```
+
+On success the batch is appended to `data/initial.json`, then `compile` (which rebuilds the
+world's snapshot, `db/state.sqlite`, and every scenario's) and `check` run. A refusal points at
+the row and field (`/users/3/email`) with the store's reason. Nothing is written on refusal.
+A directory takes at most 5,000 rows / 4 MiB; a platform slug takes any size as chunks. Rows already held with the same key and content are skipped, so the
+result says `changed: false` when nothing was new.
+
+The same batch is what `gateway worlds connector ingest` builds from captures; the two
+commands share the check. Source: `gatewaysdk/environment/schema/data.py` (`import_rows`,
+`seed_batch`) beside `connector.py` (`ingest`).
+
+## Mock-tools worlds
+
+Tables come from `db/schema.sql`; columns from each table's `CREATE TABLE`. Rows become
+statements appended to `db/seed.sql`: `INSERT OR REPLACE` on `append` (an upsert by the
+table's own key), plain `INSERT` on `replace`. A column the table does not have is refused
+by name (`Column c is not in table t; columns: ...`). Values are scalars quoted as SQLite
+literals; a nested object or array is refused (`takes a scalar; encode nested values as
+text yourself`).
+
+Materialization (`gatewaysdk/environment/_materialize.py`) runs `schema.sql` then
+`seed.sql`, so the appended block is part of every future session's starting state.
+
+## Live seeding
+
+`gateway worlds session seed <SESSION_ID> rows.json [--mode]` posts a session call:
+
+```json
+{"kind": "seed", "args": {"mode": "append", "rows": {"users": [{"id": "U9", "name": "New"}]}}}
+```
+
+and waits for the result. Schema worlds: the engine applies one batch through the contract
+(`seed_batch`), so a refused row leaves the session as it was. Api worlds: the runner reads
+the world's state, merges the rows in (`merge_seed`: `replace` swaps the named entities,
+`append` upserts where keys are known) and posts the whole state to the world's seed route,
+restoring the previous state if the world refuses. Mock-tools task worlds: the session
+server executes the INSERTs against the live SQLite. Any other kind answers
+`this world kind cannot be seeded live`. A call carries at most 5,000 rows and 4 MiB. A live
+seed never changes the world's version.
+
+## From a call log to rows
+
+`gateway worlds session export <SESSION_ID> --out calls.jsonl` writes one line per call:
+
+```jsonl
+{"seq": 1, "kind": "call", "tool": "list_users", "args": {}, "result": {"users": [{"id": "U1"}]}, "error": null, "state": "done", "createdAt": "2026-09-11T10:00:00Z", "completedAt": "2026-09-11T10:00:00Z"}
+{"seq": 2, "kind": "call", "tool": "post_message", "args": {"channel": "C1", "text": "hi"}, "result": {"ok": true, "ts": "1.0"}, "error": null, "state": "done", "createdAt": "2026-09-11T10:00:01Z", "completedAt": "2026-09-11T10:00:01Z"}
+```
+
+`kind` is `call`, `grade`, `seed`, `reset` or `close`; seeds and resets are in the log too, so
+filter on `kind == "call"`.
+
+Two ways to make rows from it:
+
+- **Reads become seed.** Results of list/get tools already are rows of an entity. Select the
+  tool, unwrap the result, tag the entity.
+- **Writes become expected state.** A `post_message` call's `args` plus the ids in its
+  `result` are the row the world should contain after the run. Build the row from both.
+
+```bash
+jq -c 'select(.kind == "call" and .error == null and .tool == "post_message")
+       | {entity: "messages", row: {channel_id: .args.channel, text: .args.text, ts: .result.ts}}' \
+   calls.jsonl > messages.jsonl
+gateway worlds data import ./world messages.jsonl
+```
+
+Hash or drop personal fields before importing. Keep `calls.jsonl` beside the rows with the
+session id so the provenance survives.
+
+## MCP payloads
+
+```json
+{"tool": "import_world_data", "input": {"slug": "vendor-world", "rows": {"users": [{"id": "U1", "name": "Ada"}]}, "mode": "append", "message": "seed users"}}
+{"tool": "seed_world_session", "input": {"sessionId": "<id>", "rows": {"users": [{"id": "U9"}]}, "mode": "replace"}}
+{"tool": "export_world_session", "input": {"sessionId": "<id>", "limit": 500}}
+```
+
+`import_world_data` accepts `containerId` or `slug`; `rows` is always the JSON object shape
+(5,000 rows / 4 MiB per call). It returns `{ versionId, entityCounts, deduped }`;
+`seed_world_session` returns the engine's `{ seeded, mode, entity_counts? }` or a `callId` to
+poll when the engine takes longer than 60 seconds. A refused row is a tool error naming the
+row and field; nothing is written.
+
+## Chunks, batches and refusal records
+
+A platform import (`data import <slug>`) cuts the row stream into chunks: gzip over JSONL,
+one row per line, either tagged `{"entity": "users", "row": {...}}` or, when the chunk was
+declared with `--entity`, the bare row. A line is
+`json.dumps(row, ensure_ascii=False, separators=(",", ":"))`. A chunk closes on a row
+boundary when the next line would exceed `--chunk-bytes` (64 MiB) or at `--chunk-rows`.
+Chunks are addressed by the sha256 of their gzipped bytes; a batch by
+`sha256(parent + "\n" + batchId + "\n" + sorted chunk digests joined by "\n")`.
+
+The manifest `.gateway/imports/<batchId>.json`:
+
+```json
+{"version": 1, "batchId": "...", "slug": "...", "host": "...",
+ "flags": {"mode": "append", "merge": "upsert", "atomic": true, "publish": "now", "dryRun": false, "chunkBytes": 67108864, "chunkRows": null, "entity": null},
+ "inputs": ["export.jsonl"],
+ "chunks": [{"ordinal": 0, "digest": "...", "bytes": 123, "rows": 50000, "entity": null, "chunkId": "...", "uploaded": true}],
+ "completed": false, "createdAt": "...", "updatedAt": "..."}
+```
+
+A refusal record (dry run, `--report`, `status`, MCP):
+
+```json
+{"chunk": 0, "index": 17, "entity": "users", "primaryKey": {"id": "u17"}, "field": "email", "pointer": "/users/17/email", "detail": "String length is outside the declared bound"}
+```
+
+`chunk` is the chunk ordinal, `index` the row's line inside it, `primaryKey` null when the
+row lacks its key, `field` null for a row-level refusal (a missing required property, an
+unknown property; `detail` names it). Reports are JSONL of these, at most 10,000 records
+then `{"truncated": true, "total": N}`. `data check` (or `import --dry-run`) produces the
+same records locally against the world's contract: shape first (type, bounds, required,
+unknown property, key present), then references and key conflicts in a scratch apply.
+
+## Extract adapters
+
+`gateway worlds data extract <WORLD> <file|-> --shape S --entity E [--out F] [--merge fill|upsert] [--no-redact]`
+
+| Shape | Input | Rows |
+| --- | --- | --- |
+| `tool-results` | one `{..., "raw_output": {...}}` per line | every object of `raw_output.results[]` (else `.items[]`) carrying the entity's key |
+| `vendor-envelope` | the vendor's response, one per line or one document | under the world's `[api].envelope` key; the bare array when it is `""` |
+| `session-export` | `calls.jsonl` from `session export` | `result.<envelope>[]` of `kind == "call"` records with no error |
+
+Then, in order: null and `""` values dropped; fields the entity does not declare stripped
+and counted; rows deduped by primary key in first-seen order (`fill`: first wins, later
+rows add missing fields; `upsert`: last wins); the world's `[redact]` policy applied
+(`drop` removed, `hash` replaced by `"sha256:" + sha256("<slug>:<value>")`, lists element
+by element, `round` rounded). Output is tagged JSONL; stderr gets
+`{"read": N, "rows": M, "unknownFields": {name: n}, "redacted": {"dropped": n, "hashed": n}}`.
+The fixtures under `gatewaysdk/fixtures/world-data/extract/` are the worked examples.