@withgateway/sdk 0.2.5 → 0.2.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (75) hide show
  1. package/dist/cli.js +3121 -1283
  2. package/dist/cli.js.map +1 -1
  3. package/dist/cli.mjs +3185 -1347
  4. package/dist/cli.mjs.map +1 -1
  5. package/dist/files/index.js +9 -0
  6. package/dist/files/index.js.map +1 -1
  7. package/dist/files/index.mjs +9 -0
  8. package/dist/files/index.mjs.map +1 -1
  9. package/dist/hooks/index.d.mts +1 -1
  10. package/dist/hooks/index.d.ts +1 -1
  11. package/dist/hooks/index.js +280 -8
  12. package/dist/hooks/index.js.map +1 -1
  13. package/dist/hooks/index.mjs +272 -8
  14. package/dist/hooks/index.mjs.map +1 -1
  15. package/dist/{index-1xnxt0Cc.d.ts → index-BO2SNSgy.d.ts} +12 -1
  16. package/dist/{index-B0R3IgsV.d.mts → index-BaKu5ZSU.d.mts} +124 -2
  17. package/dist/{index-B0R3IgsV.d.ts → index-BaKu5ZSU.d.ts} +124 -2
  18. package/dist/{index-C1wDEwH4.d.mts → index-Cz8Q6Keo.d.mts} +12 -1
  19. package/dist/index.d.mts +4 -4
  20. package/dist/index.d.ts +4 -4
  21. package/dist/index.js +726 -167
  22. package/dist/index.js.map +1 -1
  23. package/dist/index.mjs +718 -167
  24. package/dist/index.mjs.map +1 -1
  25. package/dist/traces/index.js +9 -0
  26. package/dist/traces/index.js.map +1 -1
  27. package/dist/traces/index.mjs +9 -0
  28. package/dist/traces/index.mjs.map +1 -1
  29. package/dist/tracing/index.d.mts +1 -1
  30. package/dist/tracing/index.d.ts +1 -1
  31. package/dist/tracing/index.js +97 -61
  32. package/dist/tracing/index.js.map +1 -1
  33. package/dist/tracing/index.mjs +97 -61
  34. package/dist/tracing/index.mjs.map +1 -1
  35. package/dist/worlds/index.d.mts +154 -4
  36. package/dist/worlds/index.d.ts +154 -4
  37. package/dist/worlds/index.js +348 -97
  38. package/dist/worlds/index.js.map +1 -1
  39. package/dist/worlds/index.mjs +348 -97
  40. package/dist/worlds/index.mjs.map +1 -1
  41. package/package.json +1 -1
  42. package/runtime/gatewaysdk/environment/_bundle.py +1 -1
  43. package/runtime/gatewaysdk/environment/_materialize.py +53 -9
  44. package/runtime/gatewaysdk/environment/_toml.py +13 -0
  45. package/runtime/gatewaysdk/environment/schema/api.py +3 -7
  46. package/runtime/gatewaysdk/environment/schema/batch.py +15 -11
  47. package/runtime/gatewaysdk/environment/schema/bundle_runtime.py +24 -0
  48. package/runtime/gatewaysdk/environment/schema/compiler.py +60 -28
  49. package/runtime/gatewaysdk/environment/schema/conform.py +1 -1
  50. package/runtime/gatewaysdk/environment/schema/connector.py +1 -1
  51. package/runtime/gatewaysdk/environment/schema/connectors/google-drive/readers/merge.ingest.toml +25 -0
  52. package/runtime/gatewaysdk/environment/schema/connectors/google-drive/readers/merge.toml +17 -0
  53. package/runtime/gatewaysdk/environment/schema/extract.py +1 -5
  54. package/runtime/gatewaysdk/environment/schema/host_surfaces.py +1 -5
  55. package/runtime/gatewaysdk/environment/schema/mapping.py +1 -1
  56. package/runtime/gatewaysdk/environment/schema/platform.py +16 -4
  57. package/runtime/gatewaysdk/environment/schema/redaction.py +1 -1
  58. package/runtime/gatewaysdk/environment/schema/scaffold.py +9 -2
  59. package/runtime/gatewaysdk/environment/schema/skills/benchmarks-getting-started/SKILL.md +67 -9
  60. package/runtime/gatewaysdk/environment/schema/skills/hooks-getting-started/SKILL.md +61 -2
  61. package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md +4 -1
  62. package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/SKILL.md +11 -11
  63. package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/cli-mcp-map.md +17 -1
  64. package/runtime/gatewaysdk/environment/schema/snapshot.py +1 -5
  65. package/runtime/gatewaysdk/environment/schema/tools_world.py +1 -5
  66. package/runtime/gatewaysdk/environment/schema/ui.py +1 -5
  67. package/runtime/gatewaysdk/environment/schema/world_tests.py +1 -1
  68. package/runtime/gatewaysdk/environment/world.py +120 -4
  69. package/skills/benchmarks-getting-started/SKILL.md +67 -9
  70. package/skills/connector-schema-authoring/connector-worlds.md +8 -1
  71. package/skills/connector-schema-authoring/world-schema.md +1 -1
  72. package/skills/hooks-getting-started/SKILL.md +61 -2
  73. package/skills/world-data-ingestion/SKILL.md +4 -1
  74. package/skills/worlds-getting-started/SKILL.md +11 -11
  75. package/skills/worlds-getting-started/references/cli-mcp-map.md +17 -1
@@ -135,10 +135,17 @@ the TOML is parsed only to validate). Flags map onto the spec's keys:
135
135
  | `--method`/`-X` (default `GET`), `--path`/`-p` (must start with `/`, no query string), `--param` (repeatable) | `method`, `path`, `params` |
136
136
  | `--entity`, `--results` | `entity` (must be declared in `schema/world.json`), `results` |
137
137
  | `--action`, `--status`, `--single`, `--filter arg=row.field` (repeatable), `--body row.field=request.path` (repeatable), `--handler` | `[operations.api]` |
138
+ | `--default row.field=value` (repeatable), `--generate row.field=uuid\|int\|now` (repeatable) | `[operations.api].defaults`, `[operations.api].generate` |
139
+
140
+ A create route whose entity has a single-field primary key gets `generate` for that key on
141
+ its own (`uuid` for a string key, `int` for an integer one) unless `--body` maps it or
142
+ `--generate` names it, so a POST that leaves the id out is created rather than refused with
143
+ "Missing required property id". Composite keys are never minted.
138
144
 
139
145
  Refusals: a duplicate or malformed name, an unknown method, an undeclared entity, `--results`
140
146
  or a store action without `--entity`, a `--filter` key that is neither a `{path}` argument
141
- nor a `--param`, `action = "handler"` without `--handler`, a status outside 2xx, or a file
147
+ nor a `--param`, `action = "handler"` without `--handler`, a status outside 2xx, a `--generate`
148
+ kind other than `uuid`/`int`/`now`, `--default`/`--generate` on a read route, or a file
142
149
  with no `[connector]` table. Source: `gatewaysdk/sdk-ts/src/cli/runtime/scaffold.ts`.
143
150
 
144
151
  ## Writes and logic
@@ -117,7 +117,7 @@ every other entity:
117
117
  gateway worlds schema entity add ./x crm_accounts --ref CrmAccount --key id
118
118
  gateway worlds schema entity add ./x crm_contacts \
119
119
  --fields "id:string,account_id:string,email:string?,created_at:integer" \
120
- --key id --unique email --relationship account_id->crm_accounts
120
+ --key id --unique email --relationship 'account_id->crm_accounts'
121
121
  gateway worlds schema entity add ./x crm_notes --schema '{"type":"object",...}' --key id
122
122
  ```
123
123
 
@@ -76,8 +76,10 @@ Kinds: `person_name`, `email`, `username`, `phone`, `company`, `domain`, `ip`, `
76
76
  `machine`, `log`; anything else gets a `REF-nnnn` stand-in. `identity` ties attributes to one
77
77
  person so they stay consistent; `role: "public"` is never replaced. Every variant of a value is
78
78
  tracked (bare surname, handle spellings, `(at)` emails, phone formats, percent-encoding, a
79
- world's `sha256:` stored form); what slips through is counted (`leak_through_inbound`), and a
80
- real value in the outgoing report fails the run (`leaks_outbound`).
79
+ world's `sha256:` stored form), and in any case: `Alexr` or `ALEXR` for a seeded `alexr` is swapped
80
+ where it stands alone, and the stand-in keeps the case it arrived in. The door swaps and never
81
+ blocks. What slips through is counted (`leak_through_inbound`), and a real value in the outgoing
82
+ report fails the run (`leaks_outbound`).
81
83
 
82
84
  ## Identities that arrive, not seeded
83
85
 
@@ -127,6 +129,63 @@ spelling back in a JSON or text file; a `[redacted …]` token stays — redacti
127
129
  value, and its raw copy is a `record` sink placed before the `redact` rule. The legend maps
128
130
  personas to people: keep it with the run key, out of version control, never beside world data.
129
131
 
132
+ ## Did anything real get through? (the audit)
133
+
134
+ The door's own meters only count values it knew about, so they read zero for anything it never
135
+ recognised. After a run, the audit checks from the other side: it compares what your tools
136
+ **returned** with what the model was **shown**, and lists every real identifier that survived.
137
+
138
+ Write both sides with two `record` rules: one beside the inbound `alias` rule (raw), one after it
139
+ (shown). The outbound `alias` rule stays as you have it, so the tool is called with real values.
140
+
141
+ ```toml
142
+ [[hooks.pre_tool_call]]
143
+ matcher = "*"
144
+ handlers = [{ type = "alias" }]
145
+
146
+ [[hooks.post_tool_call]]
147
+ matcher = "*"
148
+ handlers = [{ type = "record", sink = "{run_dir}/raw.jsonl" }, { type = "alias" }]
149
+
150
+ [[hooks.post_tool_call]]
151
+ matcher = "*"
152
+ handlers = [{ type = "record", sink = "{run_dir}/shown.jsonl" }]
153
+ ```
154
+
155
+ ```python
156
+ from gatewaysdk.hooks import RedactionAudit
157
+
158
+ audit = RedactionAudit.for_run(
159
+ legend=run.alias.legend(), # your subjects and found personas
160
+ field_kinds={"*": {"full_name": "person_name", "email": "email", "handle": "username"}},
161
+ )
162
+ report = audit.run(raw_records, shown_records) # lists of the JSONL lines
163
+ report.ok # False: something real reached the model
164
+ report.to_dict() # findings, census, detectors, coverage
165
+ report.review_sample(10, seed=1) # unflagged text for a person to read
166
+ ```
167
+
168
+ TypeScript is the same: `RedactionAudit.forRun({ legend, fieldKinds }).run(raw, shown)`.
169
+
170
+ From the command line, when the run directory holds `raw.jsonl`, `shown.jsonl` and `legend.json`:
171
+
172
+ ```bash
173
+ gateway hooks audit runs/<run id>/ --field-kinds fields.json --sample 10 # exit 1 when anything leaked
174
+ gateway hooks audit runs/<run id>/ --json # the same report, for a script
175
+ ```
176
+
177
+ `fields.json` is `{"*": {"full_name": "person_name", "email": "email"}}`, or keyed by tool name.
178
+
179
+ It knows a value is real from three places, strongest first: your legend; every value of a field
180
+ whose kind you declared (a name in a vendor row that reappears in a web snippet is caught for
181
+ certain); and what a pattern finds (`email`, `phone`, `ip`). A finding gives the kind, the tool, the
182
+ record and a 3-character prefix, never the value.
183
+
184
+ Read `ok` as "no **detected** leaks with these detectors", not as proof: a name that is in no list
185
+ and matches no pattern is invisible to it. That is what `review_sample` is for. When you find a
186
+ miss, add it to the task's `entities` or `redact` values and run again; the raw file is real data,
187
+ keep it with the legend.
188
+
130
189
  ## Wiring
131
190
 
132
191
  - **Agent SDK** — open the file (`from gatewaysdk.hooks import HookRun; run = HookRun("gateway-hooks.toml")`
@@ -208,7 +208,10 @@ gateway worlds describe <WORLD>; gateway worlds data query <WORLD> <entity> --li
208
208
  **The drive.** Every command here takes `drive:<path>` where it takes a local file: `gateway files
209
209
  upload rows.json --to seeds/rows.json` then `gateway worlds data import <WORLD> drive:seeds/rows.json`
210
210
  is the route from a laptop (or an agent's `request_file_upload`) to a world with nothing staged
211
- locally; the report keeps the drive path under `inputs`. `--to <path>` on `calls pull`, `extract`
211
+ locally: the platform reads the drive file itself (`.json`, `.jsonl`, `.ndjson`, optionally `.gz`;
212
+ convert CSV first) and stages it as one chunk — nothing comes down to your machine and nothing is
213
+ re-uploaded, so `drive:` needs a world slug and no `--dry-run`. The report keeps the drive path under
214
+ `inputs` and the manifest records the digest, size and rows the platform derived. `--to <path>` on `calls pull`, `extract`
212
215
  and `session export` lands the output in the drive (`--from drive:<path>` on `calls pull` and
213
216
  `ingest` reads it back). `gateway traces export drive:<path> --observations --type TOOL` puts a
214
217
  trace export there for the same route. `gateway files --help` lists the drive commands.
@@ -132,7 +132,7 @@ definitions and handler bodies are hand-written:
132
132
 
133
133
  ```bash
134
134
  gateway worlds schema entity add ./x crm_accounts --fields "id:string,name:string,tier:string?" --key id
135
- gateway worlds schema entity add ./x crm_contacts --ref CrmContact --key id --relationship account_id->crm_accounts
135
+ gateway worlds schema entity add ./x crm_contacts --ref CrmContact --key id --relationship 'account_id->crm_accounts' # quote it: a bare > is a shell redirect
136
136
  gateway worlds connector route add ./x list_accounts --path /v1/accounts --param page --entity crm_accounts --results data
137
137
  gateway worlds connector route add ./x get_account --path /v1/accounts/{id} --entity crm_accounts --filter id=id --single
138
138
  gateway worlds connector route add ./x create_account -X POST --path /v1/accounts --entity crm_accounts --action create --body name=name
@@ -140,8 +140,8 @@ gateway worlds connector route add ./x create_account -X POST --path /v1/account
140
140
 
141
141
  Each refuses (and writes nothing) on a duplicate name, an undeclared entity or `$defs`
142
142
  reference, a key that is not a required string/integer field, or a filter on an unknown
143
- argument. A custom world's first `route add` creates `connector.toml`; `schema init` prints `kind: "schema"`, `api` and
144
- the `next` commands in order (custom: entity add → route add → task add → compile → check → import → serve → `worlds create`).
143
+ argument. A create route mints its single-field primary key (`uuid` for strings, `int` for integers) unless `--body` maps it;
144
+ `--generate field=uuid|int|now` and `--default field=value` fill the rest. A custom world's first `route add` creates `connector.toml`; `schema init` prints `kind: "schema"`, `api` and the `next` commands in order (custom: entity add → route add → task add → compile → check → import → serve → `worlds create`).
145
145
  Manifest `kind` is `schema` (older trees: `mock-tools`). Handlers: `def name(ctx, args, query, body)`; `ctx` reads and writes the store and keeps the clock.
146
146
 
147
147
  MCP: `get_connector_authoring_skill({})` returns the authoring skill and conventions.
@@ -446,10 +446,13 @@ gateway bench fork x --into x-experiment # a new container sh
446
446
  gateway bench diff <base-version-id> <head-version-id> # per-file diff
447
447
  gateway bench propose ./x --title t -m why # a change proposal instead of a push
448
448
  gateway bench log x
449
+ gateway worlds sandbox open x@main # no local checkout: edit the source tree server-side,
450
+ gateway worlds sandbox write|exec|check|commit|close <id> # nothing on disk; `bench sandbox` is the same set
449
451
  ```
450
452
 
451
453
  `worlds run --path` on a clean `bench pull` checkout files runs pinned to the base version;
452
454
  a drifted directory runs unfiled. Push a new version when the change is meant to stick.
455
+ Sandbox twins (`open_world_sandbox` … `close_world_sandbox`): `references/cli-mcp-map.md`.
453
456
 
454
457
  MCP: `env_diff({ containerId })`, `env_propose({ containerId, ... })`,
455
458
  `env_list_proposals`, `env_get_proposal`, `env_apply_proposal`, `env_dismiss_proposal`,
@@ -462,9 +465,8 @@ tasks): `references/ui-map.md`.
462
465
 
463
466
  ## CLI to MCP map
464
467
 
465
- Every CLI step with its MCP twin, the steps that have none, and how the platform (console)
466
- agent reaches the same surface (`run_gateway_cli` for the local rows, `spawn_subagent` for
467
- one world per child): `references/cli-mcp-map.md`.
468
+ Every CLI step with its MCP twin, the steps that have none, and how the platform (console) agent
469
+ reaches the same surface (`run_gateway_cli`, `spawn_subagent`): `references/cli-mcp-map.md`.
468
470
 
469
471
  ## Gotchas
470
472
 
@@ -489,12 +491,10 @@ one world per child): `references/cli-mcp-map.md`.
489
491
  ## Related
490
492
 
491
493
  - `benchmarks-getting-started`: a benchmark is a world with tasks — init, check, push, run (models × tasks × rollouts), results.
492
- - `world-data-ingestion`: rows in, calls out.
493
- - `connector-schema-authoring`: the contract, fidelity shapes, conventions, definition of done.
494
+ - `world-data-ingestion`: rows in, calls out. `connector-schema-authoring`: the contract, fidelity shapes, definition of done.
494
495
  - `documentation/features/connector-worlds.md`: `connector.toml` in full, capture, conform, serving.
495
496
  - `documentation/features/workspace-connectors.md`: the connector object, sharing, MCP surface.
496
497
  - `documentation/features/world-schema.md`: the `gateway-world/1` contract and store API.
497
- - Outside the repository, `connector-worlds.md` and `world-schema.md` ship with the npm package
498
- beside the authoring skill: `gateway worlds skill connector-schema-authoring --install .` lands
499
- them under `.claude/skills/connector-schema-authoring/`.
498
+ - Outside the repository, `connector-worlds.md` and `world-schema.md` ship with the npm package beside
499
+ the authoring skill: `gateway worlds skill connector-schema-authoring --install .` installs them.
500
500
  - Consumer pages: `gateway_documentation/pages/worlds/{getting-started,data,custom-connections,slack}.mdx`.
@@ -27,16 +27,32 @@ Reference for `worlds-getting-started`: every CLI step and its MCP twin, then ho
27
27
  | `files connectors list\|sync <id>` | `list_file_connectors` / `sync_file_connector` (`GET\|POST /api/public/files/connectors[/sync]`) |
28
28
  | `traces list\|get\|export [--observations]` | `get_trace`, `get_observations` (`GET /api/public/traces[/{id}]`, `/observations`) |
29
29
  | `traces sessions list\|get [--io]` | `get_session` (`GET /api/public/sessions[/{id}?includeIO=true]`) |
30
- | `worlds data … drive:<path>` / `--to <path>` (any data command) | `get_file` + `import_world_data` (the CLI downloads, then runs the command as on a local file) |
30
+ | `worlds data import <slug> drive:<path>` | `POST /api/public/worlds/{slug}/data/batches/{id}/chunks` with `{ordinal, drive}` (the platform stages the drive file itself; nothing is downloaded or re-uploaded) |
31
+ | `worlds data … drive:<path>` / `--to <path>` (other data commands) | `get_file` + the command (the CLI downloads, then runs the command as on a local file) |
31
32
  | `environment run` / `worlds run` | `run_evaluation` |
32
33
  | `environment link-task-set` / `create-task` | `create_task_set`, `create_task`, `link_task_set` |
33
34
  | `bench diff` / `propose` | `env_diff`, `env_propose` |
35
+ | `worlds\|bench sandbox open\|status\|read\|write\|exec\|check\|commit\|close` | `open_world_sandbox`, `get_world_sandbox`, `read_world_sandbox_file`, `write_world_sandbox_file` (base64), `exec_in_world_sandbox`, `check_world_sandbox`, `commit_world_sandbox`, `close_world_sandbox` (`/api/public/benchmark-containers/sandboxes[/{id}[/file\|exec\|check\|commit]]`; the source tree only — no data snapshot, nothing on the caller's disk; `commit` lands on the recorded base) |
34
36
  | `worlds skill <name>` / `worlds connector skill` | `get_connector_authoring_skill`, `guide` |
35
37
  | `worlds connector capture --remote` / poll + download | `start_connector_capture`, `get_connector_capture` (platform agent only) |
36
38
 
37
39
  Steps with no MCP tool: opening a session (`worlds session open`) — use the CLI or the public
38
40
  sessions route.
39
41
 
42
+ ## Size and time limits (one source: `packages/shared/src/world/limits.ts`)
43
+
44
+ Every door enforces the same number and names it in the refusal; the CLI checks the file
45
+ before it makes a request.
46
+
47
+ | Door | Limit | Over it |
48
+ |---|---|---|
49
+ | `sandbox write`, `write_world_sandbox_file`, `env_write_file`, `propose_task_fix` | 256 KB per file | refused, naming the limit; rows → `worlds data import <slug> <file>`; a big source file → `gateway files upload`, then fetch it inside the sandbox with `sandbox exec`; the whole tree → `bench push` |
50
+ | `sandbox exec --timeout`, `exec_in_world_sandbox timeoutMs` | default 60 s, at most 600 s | refused, not clamped |
51
+ | Sandbox tree at `check`/`commit`, `bench push` unpacked | 128 MB of source | refused, naming the limit; move data to `worlds data import`, delete scratch files |
52
+ | `bench push` / `worlds create --from` compressed bundle | 64 MB (16 MB from the browser) | refused, naming the limit |
53
+ | A single manifest / metadata file in a bundle | 1 MB | refused |
54
+ | `sandbox write` HTTP body | 1 MB (framework cap; only reachable by a body no schema would accept) | 413 |
55
+
40
56
  ## The platform agent
41
57
 
42
58
  The platform (console) agent has every MCP tool in this table as an agent tool of the