@withgateway/sdk 0.2.5 → 0.2.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli.js +3121 -1283
- package/dist/cli.js.map +1 -1
- package/dist/cli.mjs +3185 -1347
- package/dist/cli.mjs.map +1 -1
- package/dist/files/index.js +9 -0
- package/dist/files/index.js.map +1 -1
- package/dist/files/index.mjs +9 -0
- package/dist/files/index.mjs.map +1 -1
- package/dist/hooks/index.d.mts +1 -1
- package/dist/hooks/index.d.ts +1 -1
- package/dist/hooks/index.js +280 -8
- package/dist/hooks/index.js.map +1 -1
- package/dist/hooks/index.mjs +272 -8
- package/dist/hooks/index.mjs.map +1 -1
- package/dist/{index-1xnxt0Cc.d.ts → index-BO2SNSgy.d.ts} +12 -1
- package/dist/{index-B0R3IgsV.d.mts → index-BaKu5ZSU.d.mts} +124 -2
- package/dist/{index-B0R3IgsV.d.ts → index-BaKu5ZSU.d.ts} +124 -2
- package/dist/{index-C1wDEwH4.d.mts → index-Cz8Q6Keo.d.mts} +12 -1
- package/dist/index.d.mts +4 -4
- package/dist/index.d.ts +4 -4
- package/dist/index.js +726 -167
- package/dist/index.js.map +1 -1
- package/dist/index.mjs +718 -167
- package/dist/index.mjs.map +1 -1
- package/dist/traces/index.js +9 -0
- package/dist/traces/index.js.map +1 -1
- package/dist/traces/index.mjs +9 -0
- package/dist/traces/index.mjs.map +1 -1
- package/dist/tracing/index.d.mts +1 -1
- package/dist/tracing/index.d.ts +1 -1
- package/dist/tracing/index.js +97 -61
- package/dist/tracing/index.js.map +1 -1
- package/dist/tracing/index.mjs +97 -61
- package/dist/tracing/index.mjs.map +1 -1
- package/dist/worlds/index.d.mts +154 -4
- package/dist/worlds/index.d.ts +154 -4
- package/dist/worlds/index.js +348 -97
- package/dist/worlds/index.js.map +1 -1
- package/dist/worlds/index.mjs +348 -97
- package/dist/worlds/index.mjs.map +1 -1
- package/package.json +1 -1
- package/runtime/gatewaysdk/environment/_bundle.py +1 -1
- package/runtime/gatewaysdk/environment/_materialize.py +53 -9
- package/runtime/gatewaysdk/environment/_toml.py +13 -0
- package/runtime/gatewaysdk/environment/schema/api.py +3 -7
- package/runtime/gatewaysdk/environment/schema/batch.py +15 -11
- package/runtime/gatewaysdk/environment/schema/bundle_runtime.py +24 -0
- package/runtime/gatewaysdk/environment/schema/compiler.py +60 -28
- package/runtime/gatewaysdk/environment/schema/conform.py +1 -1
- package/runtime/gatewaysdk/environment/schema/connector.py +1 -1
- package/runtime/gatewaysdk/environment/schema/connectors/google-drive/readers/merge.ingest.toml +25 -0
- package/runtime/gatewaysdk/environment/schema/connectors/google-drive/readers/merge.toml +17 -0
- package/runtime/gatewaysdk/environment/schema/extract.py +1 -5
- package/runtime/gatewaysdk/environment/schema/host_surfaces.py +1 -5
- package/runtime/gatewaysdk/environment/schema/mapping.py +1 -1
- package/runtime/gatewaysdk/environment/schema/platform.py +16 -4
- package/runtime/gatewaysdk/environment/schema/redaction.py +1 -1
- package/runtime/gatewaysdk/environment/schema/scaffold.py +9 -2
- package/runtime/gatewaysdk/environment/schema/skills/benchmarks-getting-started/SKILL.md +67 -9
- package/runtime/gatewaysdk/environment/schema/skills/hooks-getting-started/SKILL.md +61 -2
- package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md +4 -1
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/SKILL.md +11 -11
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/references/cli-mcp-map.md +17 -1
- package/runtime/gatewaysdk/environment/schema/snapshot.py +1 -5
- package/runtime/gatewaysdk/environment/schema/tools_world.py +1 -5
- package/runtime/gatewaysdk/environment/schema/ui.py +1 -5
- package/runtime/gatewaysdk/environment/schema/world_tests.py +1 -1
- package/runtime/gatewaysdk/environment/world.py +120 -4
- package/skills/benchmarks-getting-started/SKILL.md +67 -9
- package/skills/connector-schema-authoring/connector-worlds.md +8 -1
- package/skills/connector-schema-authoring/world-schema.md +1 -1
- package/skills/hooks-getting-started/SKILL.md +61 -2
- package/skills/world-data-ingestion/SKILL.md +4 -1
- package/skills/worlds-getting-started/SKILL.md +11 -11
- package/skills/worlds-getting-started/references/cli-mcp-map.md +17 -1
|
@@ -70,9 +70,9 @@ Expected output (JSON): `"kind": "harbor"`, `"tasks": ["hello"]`, the files, and
|
|
|
70
70
|
code-bench/
|
|
71
71
|
benchmark.json # kind, models, rollouts: the defaults `benchmarks run` reads
|
|
72
72
|
pyproject.toml # [project] name + version: the slug and the version `push` records; not a Python package
|
|
73
|
-
tasks/hello/task.toml # version, [metadata], [agent] timeout_sec, [verifier] timeout_sec
|
|
73
|
+
tasks/hello/task.toml # version, [metadata], [agent] timeout_sec, [verifier] timeout_sec; docker_image only as a comment
|
|
74
74
|
tasks/hello/instruction.md # what the agent is told
|
|
75
|
-
tasks/hello/environment/Dockerfile # the task's container (FROM
|
|
75
|
+
tasks/hello/environment/Dockerfile # the task's container, built by the runner (FROM --image; WORKDIR /app): the ONE image source
|
|
76
76
|
tasks/hello/tests/test.sh # the grader: runs after the agent, writes /logs/verifier/reward.json
|
|
77
77
|
tasks/hello/solution/solve.sh # the reference solution; must score 1.0 (harbor's oracle agent runs it)
|
|
78
78
|
```
|
|
@@ -82,7 +82,40 @@ The reward contract is the only thing the platform reads: `tests/test.sh` writes
|
|
|
82
82
|
other key a sub-reward — or a bare number to `/logs/verifier/reward.txt`. Any language, any method.
|
|
83
83
|
|
|
84
84
|
Add a task: copy `tasks/hello` to `tasks/<name>`, edit `instruction.md`, `tests/test.sh`,
|
|
85
|
-
`solution/solve.sh`, and the image
|
|
85
|
+
`solution/solve.sh`, and the image. Two tasks minimum.
|
|
86
|
+
|
|
87
|
+
**One image source.** Harbor's rule (`should_use_prebuilt_docker_image`): when `task.toml` names
|
|
88
|
+
`[environment] docker_image`, the runner pulls that prebuilt image and runs it as it is —
|
|
89
|
+
`environment/Dockerfile` is **never built**, so anything it installs is absent on the runner (a tool the
|
|
90
|
+
agent needs, a pip package) while a Dockerfile-first local run passes. Without `docker_image` the
|
|
91
|
+
runner builds the Dockerfile. The scaffold therefore writes only the Dockerfile; put what the agent
|
|
92
|
+
needs in it. To run a prebuilt image instead, delete the Dockerfile and set `docker_image`.
|
|
93
|
+
`validate` fails a task that has both when the Dockerfile adds anything beyond `FROM`/`WORKDIR`
|
|
94
|
+
(a note when it does not), and `dry-run` follows the runner's choice (`image … (prebuilt: docker_image)`,
|
|
95
|
+
`pull`, no `build`).
|
|
96
|
+
|
|
97
|
+
**Secrets (an API key the task needs).** Declare it in the task's `environment/docker-compose.yaml`
|
|
98
|
+
— an overlay harbor applies on top of the image, prebuilt or built — and set it on the hosted
|
|
99
|
+
environment:
|
|
100
|
+
|
|
101
|
+
```yaml
|
|
102
|
+
# tasks/q1/environment/docker-compose.yaml
|
|
103
|
+
services:
|
|
104
|
+
main:
|
|
105
|
+
environment:
|
|
106
|
+
EXA_API_KEY: ${EXA_API_KEY:?set with gateway bench secrets set}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
gateway bench secrets set --env code-bench --key EXA_API_KEY # paste the value, then Ctrl-D; never on the command line
|
|
111
|
+
gateway bench secrets list --env code-bench
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
`${NAME:?…}` (and a bare `${NAME}`) is a requirement; `${NAME:-default}` is not. `validate` lists a
|
|
115
|
+
task's required names as `secrets …`; `run` refuses before dispatch when one is not set on the
|
|
116
|
+
environment, naming the exact `bench secrets set` command. `gateway secret …` is a different store
|
|
117
|
+
(the platform's secret vault for agents and hooks) and does **not** feed hosted harbor runs.
|
|
118
|
+
`run_parameters.json` does not either. Locally, `dry-run` exports what your shell holds — see A3.
|
|
86
119
|
|
|
87
120
|
MCP: no twin for a local scaffold; on an existing environment, `env_write_file({ containerId, path,
|
|
88
121
|
newContent })` for each file, then `env_commit`.
|
|
@@ -97,15 +130,18 @@ Expected output:
|
|
|
97
130
|
|
|
98
131
|
```
|
|
99
132
|
./code-bench: harbor benchmark — runner reads it as harbor — ok
|
|
100
|
-
ok hello: image
|
|
133
|
+
ok hello: image build tasks/hello/environment/Dockerfile; instruction yes; tests/test.sh yes; solution yes
|
|
101
134
|
benchmark ok (harbor): 1 task(s): hello; next: gateway benchmarks dry-run ./code-bench
|
|
102
135
|
```
|
|
103
136
|
|
|
104
|
-
Per task: `task.toml` present,
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
137
|
+
Per task: `task.toml` present, ONE image source (`[environment] docker_image` → `image <ref> (prebuilt:
|
|
138
|
+
docker_image)`, else `environment/Dockerfile` → `image build …`; both is a problem when the Dockerfile
|
|
139
|
+
adds anything), the compose overlay and its `secrets` when the task has one, `instruction.md`
|
|
140
|
+
non-empty, `tests/test.sh` present and writing the reward file, `solution/solve.sh` noted when absent.
|
|
141
|
+
Tree-level: `pyproject.toml` (a note when absent) and that the runner's own dispatch rule
|
|
142
|
+
(`detect_env_kind`) reads the tree as harbor — a `gateway-env.toml` at the root would make it a world
|
|
143
|
+
to the runner. Exit 2 on any problem; notes never fail it. The closing line counts the tasks and names
|
|
144
|
+
the first five (`24 tasks: a, b, c, d, e, … (19 more)`); `--json` lists them all.
|
|
109
145
|
|
|
110
146
|
### A3. Dry-run one task
|
|
111
147
|
|
|
@@ -131,6 +167,16 @@ runner's `harbor run` line, the static check of `tests/test.sh`) followed by
|
|
|
131
167
|
`not executed: … is not installed; execution needs Docker — the plan is what would run`, exit 0.
|
|
132
168
|
`GATEWAY_DOCKER` names the docker binary.
|
|
133
169
|
|
|
170
|
+
The image is chosen by harbor's rule (A1): `docker_image` set → `docker pull`, the Dockerfile is not
|
|
171
|
+
built and the plan says so; otherwise `docker build`. A task with `environment/docker-compose.yaml`
|
|
172
|
+
runs through `docker compose` instead of `docker run` — harbor's base for `main` (the image, the
|
|
173
|
+
mounts, written to `<tmp>/docker-compose.base.yaml`) plus the task's file, then `compose down` — so
|
|
174
|
+
its extra services and `${NAME:?…}` secrets are exercised locally. The names must be exported in
|
|
175
|
+
your shell (`export EXA_API_KEY=…`); a missing one is reported as `not executed: the task's
|
|
176
|
+
docker-compose file requires EXA_API_KEY … export EXA_API_KEY here for the dry-run. Hosted runs read
|
|
177
|
+
it from … gateway bench secrets set --env code-bench --key EXA_API_KEY`, exit 0, nothing spawned.
|
|
178
|
+
Values never appear in the printed argv.
|
|
179
|
+
|
|
134
180
|
### A4. Push
|
|
135
181
|
|
|
136
182
|
```bash
|
|
@@ -146,6 +192,18 @@ tree under a world's slug (exit 2, nothing sent); the hub's versions door refuse
|
|
|
146
192
|
`benchmark_substrate_mismatch`. Later pushes are new content-hashed versions (`gateway bench log`,
|
|
147
193
|
`diff`, `branch`, `tag`).
|
|
148
194
|
|
|
195
|
+
**Which slug.** stderr says where it came from: `slug code-bench (from benchmark.json)`. Order:
|
|
196
|
+
`--slug` > the dir's `.gateway/.env-metadata.json` (written by every push and pull, so a dir keeps
|
|
197
|
+
pushing to the slug it became) > `benchmark.json` > `pyproject.toml [project].name` > the directory
|
|
198
|
+
name. A tracked slug that disagrees with what the tree declares is refused with exit 2 and nothing
|
|
199
|
+
sent — `./vendor tracks slug websearch-bench-e2e (.gateway/.env-metadata.json, from an earlier push)
|
|
200
|
+
but its benchmark.json says websearch-bench` — because that is how one `--slug X` push made every
|
|
201
|
+
later bare push land on X. Settle it with `--slug <the one you mean>` (the dir retracks) or delete the
|
|
202
|
+
metadata file. The platform's collision suffix (`code-bench-2` minted for a taken `code-bench`) is
|
|
203
|
+
not a disagreement. A 409 `Branch "main" has moved` (another push landed on the slug since this dir's
|
|
204
|
+
base) names its way out: `gateway benchmarks push ./code-bench --force` moves the branch anyway;
|
|
205
|
+
`gateway bench pull code-bench` into a fresh dir builds on the tip.
|
|
206
|
+
|
|
149
207
|
MCP: `create_environment_version` (or `env_write_file` + `env_commit`).
|
|
150
208
|
|
|
151
209
|
### A5. Run: models × tasks × rollouts
|
|
@@ -76,8 +76,10 @@ Kinds: `person_name`, `email`, `username`, `phone`, `company`, `domain`, `ip`, `
|
|
|
76
76
|
`machine`, `log`; anything else gets a `REF-nnnn` stand-in. `identity` ties attributes to one
|
|
77
77
|
person so they stay consistent; `role: "public"` is never replaced. Every variant of a value is
|
|
78
78
|
tracked (bare surname, handle spellings, `(at)` emails, phone formats, percent-encoding, a
|
|
79
|
-
world's `sha256:` stored form)
|
|
80
|
-
|
|
79
|
+
world's `sha256:` stored form), and in any case: `Alexr` or `ALEXR` for a seeded `alexr` is swapped
|
|
80
|
+
where it stands alone, and the stand-in keeps the case it arrived in. The door swaps and never
|
|
81
|
+
blocks. What slips through is counted (`leak_through_inbound`), and a real value in the outgoing
|
|
82
|
+
report fails the run (`leaks_outbound`).
|
|
81
83
|
|
|
82
84
|
## Identities that arrive, not seeded
|
|
83
85
|
|
|
@@ -127,6 +129,63 @@ spelling back in a JSON or text file; a `[redacted …]` token stays — redacti
|
|
|
127
129
|
value, and its raw copy is a `record` sink placed before the `redact` rule. The legend maps
|
|
128
130
|
personas to people: keep it with the run key, out of version control, never beside world data.
|
|
129
131
|
|
|
132
|
+
## Did anything real get through? (the audit)
|
|
133
|
+
|
|
134
|
+
The door's own meters only count values it knew about, so they read zero for anything it never
|
|
135
|
+
recognised. After a run, the audit checks from the other side: it compares what your tools
|
|
136
|
+
**returned** with what the model was **shown**, and lists every real identifier that survived.
|
|
137
|
+
|
|
138
|
+
Write both sides with two `record` rules: one beside the inbound `alias` rule (raw), one after it
|
|
139
|
+
(shown). The outbound `alias` rule stays as you have it, so the tool is called with real values.
|
|
140
|
+
|
|
141
|
+
```toml
|
|
142
|
+
[[hooks.pre_tool_call]]
|
|
143
|
+
matcher = "*"
|
|
144
|
+
handlers = [{ type = "alias" }]
|
|
145
|
+
|
|
146
|
+
[[hooks.post_tool_call]]
|
|
147
|
+
matcher = "*"
|
|
148
|
+
handlers = [{ type = "record", sink = "{run_dir}/raw.jsonl" }, { type = "alias" }]
|
|
149
|
+
|
|
150
|
+
[[hooks.post_tool_call]]
|
|
151
|
+
matcher = "*"
|
|
152
|
+
handlers = [{ type = "record", sink = "{run_dir}/shown.jsonl" }]
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
```python
|
|
156
|
+
from gatewaysdk.hooks import RedactionAudit
|
|
157
|
+
|
|
158
|
+
audit = RedactionAudit.for_run(
|
|
159
|
+
legend=run.alias.legend(), # your subjects and found personas
|
|
160
|
+
field_kinds={"*": {"full_name": "person_name", "email": "email", "handle": "username"}},
|
|
161
|
+
)
|
|
162
|
+
report = audit.run(raw_records, shown_records) # lists of the JSONL lines
|
|
163
|
+
report.ok # False: something real reached the model
|
|
164
|
+
report.to_dict() # findings, census, detectors, coverage
|
|
165
|
+
report.review_sample(10, seed=1) # unflagged text for a person to read
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
TypeScript is the same: `RedactionAudit.forRun({ legend, fieldKinds }).run(raw, shown)`.
|
|
169
|
+
|
|
170
|
+
From the command line, when the run directory holds `raw.jsonl`, `shown.jsonl` and `legend.json`:
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
gateway hooks audit runs/<run id>/ --field-kinds fields.json --sample 10 # exit 1 when anything leaked
|
|
174
|
+
gateway hooks audit runs/<run id>/ --json # the same report, for a script
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
`fields.json` is `{"*": {"full_name": "person_name", "email": "email"}}`, or keyed by tool name.
|
|
178
|
+
|
|
179
|
+
It knows a value is real from three places, strongest first: your legend; every value of a field
|
|
180
|
+
whose kind you declared (a name in a vendor row that reappears in a web snippet is caught for
|
|
181
|
+
certain); and what a pattern finds (`email`, `phone`, `ip`). A finding gives the kind, the tool, the
|
|
182
|
+
record and a 3-character prefix, never the value.
|
|
183
|
+
|
|
184
|
+
Read `ok` as "no **detected** leaks with these detectors", not as proof: a name that is in no list
|
|
185
|
+
and matches no pattern is invisible to it. That is what `review_sample` is for. When you find a
|
|
186
|
+
miss, add it to the task's `entities` or `redact` values and run again; the raw file is real data,
|
|
187
|
+
keep it with the legend.
|
|
188
|
+
|
|
130
189
|
## Wiring
|
|
131
190
|
|
|
132
191
|
- **Agent SDK** — open the file (`from gatewaysdk.hooks import HookRun; run = HookRun("gateway-hooks.toml")`
|
|
@@ -208,7 +208,10 @@ gateway worlds describe <WORLD>; gateway worlds data query <WORLD> <entity> --li
|
|
|
208
208
|
**The drive.** Every command here takes `drive:<path>` where it takes a local file: `gateway files
|
|
209
209
|
upload rows.json --to seeds/rows.json` then `gateway worlds data import <WORLD> drive:seeds/rows.json`
|
|
210
210
|
is the route from a laptop (or an agent's `request_file_upload`) to a world with nothing staged
|
|
211
|
-
locally
|
|
211
|
+
locally: the platform reads the drive file itself (`.json`, `.jsonl`, `.ndjson`, optionally `.gz`;
|
|
212
|
+
convert CSV first) and stages it as one chunk — nothing comes down to your machine and nothing is
|
|
213
|
+
re-uploaded, so `drive:` needs a world slug and no `--dry-run`. The report keeps the drive path under
|
|
214
|
+
`inputs` and the manifest records the digest, size and rows the platform derived. `--to <path>` on `calls pull`, `extract`
|
|
212
215
|
and `session export` lands the output in the drive (`--from drive:<path>` on `calls pull` and
|
|
213
216
|
`ingest` reads it back). `gateway traces export drive:<path> --observations --type TOOL` puts a
|
|
214
217
|
trace export there for the same route. `gateway files --help` lists the drive commands.
|
|
@@ -132,7 +132,7 @@ definitions and handler bodies are hand-written:
|
|
|
132
132
|
|
|
133
133
|
```bash
|
|
134
134
|
gateway worlds schema entity add ./x crm_accounts --fields "id:string,name:string,tier:string?" --key id
|
|
135
|
-
gateway worlds schema entity add ./x crm_contacts --ref CrmContact --key id --relationship account_id->crm_accounts
|
|
135
|
+
gateway worlds schema entity add ./x crm_contacts --ref CrmContact --key id --relationship 'account_id->crm_accounts' # quote it: a bare > is a shell redirect
|
|
136
136
|
gateway worlds connector route add ./x list_accounts --path /v1/accounts --param page --entity crm_accounts --results data
|
|
137
137
|
gateway worlds connector route add ./x get_account --path /v1/accounts/{id} --entity crm_accounts --filter id=id --single
|
|
138
138
|
gateway worlds connector route add ./x create_account -X POST --path /v1/accounts --entity crm_accounts --action create --body name=name
|
|
@@ -140,8 +140,8 @@ gateway worlds connector route add ./x create_account -X POST --path /v1/account
|
|
|
140
140
|
|
|
141
141
|
Each refuses (and writes nothing) on a duplicate name, an undeclared entity or `$defs`
|
|
142
142
|
reference, a key that is not a required string/integer field, or a filter on an unknown
|
|
143
|
-
argument. A
|
|
144
|
-
the `next` commands in order (custom: entity add → route add → task add → compile → check → import → serve → `worlds create`).
|
|
143
|
+
argument. A create route mints its single-field primary key (`uuid` for strings, `int` for integers) unless `--body` maps it;
|
|
144
|
+
`--generate field=uuid|int|now` and `--default field=value` fill the rest. A custom world's first `route add` creates `connector.toml`; `schema init` prints `kind: "schema"`, `api` and the `next` commands in order (custom: entity add → route add → task add → compile → check → import → serve → `worlds create`).
|
|
145
145
|
Manifest `kind` is `schema` (older trees: `mock-tools`). Handlers: `def name(ctx, args, query, body)`; `ctx` reads and writes the store and keeps the clock.
|
|
146
146
|
|
|
147
147
|
MCP: `get_connector_authoring_skill({})` returns the authoring skill and conventions.
|
|
@@ -446,10 +446,13 @@ gateway bench fork x --into x-experiment # a new container sh
|
|
|
446
446
|
gateway bench diff <base-version-id> <head-version-id> # per-file diff
|
|
447
447
|
gateway bench propose ./x --title t -m why # a change proposal instead of a push
|
|
448
448
|
gateway bench log x
|
|
449
|
+
gateway worlds sandbox open x@main # no local checkout: edit the source tree server-side,
|
|
450
|
+
gateway worlds sandbox write|exec|check|commit|close <id> # nothing on disk; `bench sandbox` is the same set
|
|
449
451
|
```
|
|
450
452
|
|
|
451
453
|
`worlds run --path` on a clean `bench pull` checkout files runs pinned to the base version;
|
|
452
454
|
a drifted directory runs unfiled. Push a new version when the change is meant to stick.
|
|
455
|
+
Sandbox twins (`open_world_sandbox` … `close_world_sandbox`): `references/cli-mcp-map.md`.
|
|
453
456
|
|
|
454
457
|
MCP: `env_diff({ containerId })`, `env_propose({ containerId, ... })`,
|
|
455
458
|
`env_list_proposals`, `env_get_proposal`, `env_apply_proposal`, `env_dismiss_proposal`,
|
|
@@ -462,9 +465,8 @@ tasks): `references/ui-map.md`.
|
|
|
462
465
|
|
|
463
466
|
## CLI to MCP map
|
|
464
467
|
|
|
465
|
-
Every CLI step with its MCP twin, the steps that have none, and how the platform (console)
|
|
466
|
-
|
|
467
|
-
one world per child): `references/cli-mcp-map.md`.
|
|
468
|
+
Every CLI step with its MCP twin, the steps that have none, and how the platform (console) agent
|
|
469
|
+
reaches the same surface (`run_gateway_cli`, `spawn_subagent`): `references/cli-mcp-map.md`.
|
|
468
470
|
|
|
469
471
|
## Gotchas
|
|
470
472
|
|
|
@@ -489,12 +491,10 @@ one world per child): `references/cli-mcp-map.md`.
|
|
|
489
491
|
## Related
|
|
490
492
|
|
|
491
493
|
- `benchmarks-getting-started`: a benchmark is a world with tasks — init, check, push, run (models × tasks × rollouts), results.
|
|
492
|
-
- `world-data-ingestion`: rows in, calls out.
|
|
493
|
-
- `connector-schema-authoring`: the contract, fidelity shapes, conventions, definition of done.
|
|
494
|
+
- `world-data-ingestion`: rows in, calls out. `connector-schema-authoring`: the contract, fidelity shapes, definition of done.
|
|
494
495
|
- `documentation/features/connector-worlds.md`: `connector.toml` in full, capture, conform, serving.
|
|
495
496
|
- `documentation/features/workspace-connectors.md`: the connector object, sharing, MCP surface.
|
|
496
497
|
- `documentation/features/world-schema.md`: the `gateway-world/1` contract and store API.
|
|
497
|
-
- Outside the repository, `connector-worlds.md` and `world-schema.md` ship with the npm package
|
|
498
|
-
|
|
499
|
-
them under `.claude/skills/connector-schema-authoring/`.
|
|
498
|
+
- Outside the repository, `connector-worlds.md` and `world-schema.md` ship with the npm package beside
|
|
499
|
+
the authoring skill: `gateway worlds skill connector-schema-authoring --install .` installs them.
|
|
500
500
|
- Consumer pages: `gateway_documentation/pages/worlds/{getting-started,data,custom-connections,slack}.mdx`.
|
|
@@ -27,16 +27,32 @@ Reference for `worlds-getting-started`: every CLI step and its MCP twin, then ho
|
|
|
27
27
|
| `files connectors list\|sync <id>` | `list_file_connectors` / `sync_file_connector` (`GET\|POST /api/public/files/connectors[/sync]`) |
|
|
28
28
|
| `traces list\|get\|export [--observations]` | `get_trace`, `get_observations` (`GET /api/public/traces[/{id}]`, `/observations`) |
|
|
29
29
|
| `traces sessions list\|get [--io]` | `get_session` (`GET /api/public/sessions[/{id}?includeIO=true]`) |
|
|
30
|
-
| `worlds data
|
|
30
|
+
| `worlds data import <slug> drive:<path>` | `POST /api/public/worlds/{slug}/data/batches/{id}/chunks` with `{ordinal, drive}` (the platform stages the drive file itself; nothing is downloaded or re-uploaded) |
|
|
31
|
+
| `worlds data … drive:<path>` / `--to <path>` (other data commands) | `get_file` + the command (the CLI downloads, then runs the command as on a local file) |
|
|
31
32
|
| `environment run` / `worlds run` | `run_evaluation` |
|
|
32
33
|
| `environment link-task-set` / `create-task` | `create_task_set`, `create_task`, `link_task_set` |
|
|
33
34
|
| `bench diff` / `propose` | `env_diff`, `env_propose` |
|
|
35
|
+
| `worlds\|bench sandbox open\|status\|read\|write\|exec\|check\|commit\|close` | `open_world_sandbox`, `get_world_sandbox`, `read_world_sandbox_file`, `write_world_sandbox_file` (base64), `exec_in_world_sandbox`, `check_world_sandbox`, `commit_world_sandbox`, `close_world_sandbox` (`/api/public/benchmark-containers/sandboxes[/{id}[/file\|exec\|check\|commit]]`; the source tree only — no data snapshot, nothing on the caller's disk; `commit` lands on the recorded base) |
|
|
34
36
|
| `worlds skill <name>` / `worlds connector skill` | `get_connector_authoring_skill`, `guide` |
|
|
35
37
|
| `worlds connector capture --remote` / poll + download | `start_connector_capture`, `get_connector_capture` (platform agent only) |
|
|
36
38
|
|
|
37
39
|
Steps with no MCP tool: opening a session (`worlds session open`) — use the CLI or the public
|
|
38
40
|
sessions route.
|
|
39
41
|
|
|
42
|
+
## Size and time limits (one source: `packages/shared/src/world/limits.ts`)
|
|
43
|
+
|
|
44
|
+
Every door enforces the same number and names it in the refusal; the CLI checks the file
|
|
45
|
+
before it makes a request.
|
|
46
|
+
|
|
47
|
+
| Door | Limit | Over it |
|
|
48
|
+
|---|---|---|
|
|
49
|
+
| `sandbox write`, `write_world_sandbox_file`, `env_write_file`, `propose_task_fix` | 256 KB per file | refused, naming the limit; rows → `worlds data import <slug> <file>`; a big source file → `gateway files upload`, then fetch it inside the sandbox with `sandbox exec`; the whole tree → `bench push` |
|
|
50
|
+
| `sandbox exec --timeout`, `exec_in_world_sandbox timeoutMs` | default 60 s, at most 600 s | refused, not clamped |
|
|
51
|
+
| Sandbox tree at `check`/`commit`, `bench push` unpacked | 128 MB of source | refused, naming the limit; move data to `worlds data import`, delete scratch files |
|
|
52
|
+
| `bench push` / `worlds create --from` compressed bundle | 64 MB (16 MB from the browser) | refused, naming the limit |
|
|
53
|
+
| A single manifest / metadata file in a bundle | 1 MB | refused |
|
|
54
|
+
| `sandbox write` HTTP body | 1 MB (framework cap; only reachable by a body no schema would accept) | 413 |
|
|
55
|
+
|
|
40
56
|
## The platform agent
|
|
41
57
|
|
|
42
58
|
The platform (console) agent has every MCP tool in this table as an agent tool of the
|
|
@@ -44,11 +44,7 @@ from datetime import datetime, timezone
|
|
|
44
44
|
from pathlib import Path
|
|
45
45
|
from typing import Any, Iterable
|
|
46
46
|
|
|
47
|
-
|
|
48
|
-
import tomllib
|
|
49
|
-
except ModuleNotFoundError: # pragma: no cover - python 3.10
|
|
50
|
-
import tomli as tomllib
|
|
51
|
-
|
|
47
|
+
from .._toml import tomllib
|
|
52
48
|
from .compiler import Contract
|
|
53
49
|
from .store import SQLiteStore
|
|
54
50
|
from .validation import ContractError, canonical, fail, read_json
|
|
@@ -47,11 +47,7 @@ from dataclasses import dataclass, field
|
|
|
47
47
|
from pathlib import Path
|
|
48
48
|
from typing import Any
|
|
49
49
|
|
|
50
|
-
|
|
51
|
-
import tomllib
|
|
52
|
-
except ModuleNotFoundError: # pragma: no cover - python 3.10
|
|
53
|
-
import tomli as tomllib
|
|
54
|
-
|
|
50
|
+
from .._toml import tomllib
|
|
55
51
|
from .. import _bundle
|
|
56
52
|
from .._materialize import materialize_template
|
|
57
53
|
from .._world import World
|
|
@@ -32,11 +32,7 @@ from dataclasses import dataclass
|
|
|
32
32
|
from pathlib import Path
|
|
33
33
|
from typing import Any
|
|
34
34
|
|
|
35
|
-
|
|
36
|
-
import tomllib
|
|
37
|
-
except ModuleNotFoundError: # pragma: no cover - python < 3.11
|
|
38
|
-
import tomli as tomllib # type: ignore[no-redef]
|
|
39
|
-
|
|
35
|
+
from .._toml import tomllib
|
|
40
36
|
from .validation import fail
|
|
41
37
|
|
|
42
38
|
STATIC_SOURCE = Path(__file__).with_name("ui_static")
|
|
@@ -64,7 +64,6 @@ import shutil
|
|
|
64
64
|
import subprocess
|
|
65
65
|
import sys
|
|
66
66
|
import tempfile
|
|
67
|
-
import tomllib
|
|
68
67
|
import urllib.error
|
|
69
68
|
import urllib.parse
|
|
70
69
|
import urllib.request
|
|
@@ -72,6 +71,7 @@ from dataclasses import asdict, dataclass, field
|
|
|
72
71
|
from pathlib import Path
|
|
73
72
|
from typing import Any
|
|
74
73
|
|
|
74
|
+
from .._toml import tomllib
|
|
75
75
|
from .api import load_api_settings
|
|
76
76
|
from .conform import PIN, _credentials
|
|
77
77
|
from .connector import load_spec
|
|
@@ -15,14 +15,20 @@ dependencies.
|
|
|
15
15
|
|
|
16
16
|
from __future__ import annotations
|
|
17
17
|
|
|
18
|
+
import hashlib
|
|
18
19
|
import os
|
|
19
20
|
import re
|
|
20
|
-
import
|
|
21
|
-
from collections.abc import Iterator
|
|
21
|
+
from collections.abc import Callable, Iterator
|
|
22
22
|
from dataclasses import dataclass
|
|
23
23
|
from functools import cached_property
|
|
24
24
|
from pathlib import Path
|
|
25
25
|
from typing import ClassVar
|
|
26
|
+
from ._toml import tomllib
|
|
27
|
+
|
|
28
|
+
try: # optional: only consulted when a tree ships a root .gitignore
|
|
29
|
+
from gitignore_parser import parse_gitignore
|
|
30
|
+
except ImportError: # pragma: no cover - the runner and the sandbox vendor this module alone
|
|
31
|
+
parse_gitignore = None # type: ignore[assignment]
|
|
26
32
|
|
|
27
33
|
# ── Markers (twin of packages/shared/src/world/markers.ts) ──────────────────
|
|
28
34
|
|
|
@@ -43,6 +49,10 @@ KIND_VERIFIERS = "verifiers"
|
|
|
43
49
|
KIND_VERIFIERS_V1 = "verifiers-v1"
|
|
44
50
|
|
|
45
51
|
_SKIPPED_DIRS = frozenset({"dist", "build", "outputs", "__pycache__"})
|
|
52
|
+
# A root Gateway marker widens the hashed root from Prime's source globs to
|
|
53
|
+
# every ordinary root file: those kinds keep executable contract at the root.
|
|
54
|
+
_BROAD_ROOT_MARKERS = frozenset({WORLD_MANIFEST, API_BUNDLE_MANIFEST})
|
|
55
|
+
_HASH_CHUNK = 1 << 20
|
|
46
56
|
_HARBOR_TASK = re.compile(r"^(?:tasks/)?([^/]+)/task\.toml$")
|
|
47
57
|
|
|
48
58
|
# The `verifiers.v1` namespace first shipped in verifiers 0.2.0, so a v1
|
|
@@ -71,6 +81,35 @@ def _skipped_dir(name: str) -> bool:
|
|
|
71
81
|
return name.startswith(".") or name in _SKIPPED_DIRS or name.endswith(".egg-info")
|
|
72
82
|
|
|
73
83
|
|
|
84
|
+
def _ordinary_root_file(name: str, broad: bool) -> bool:
|
|
85
|
+
return broad or name in ("README.md", PYPROJECT) or name in _BROAD_ROOT_MARKERS or name.endswith(".py")
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
def is_hashed_path(relative: str, broad_root: bool) -> bool:
|
|
89
|
+
"""Twin of ``worldTree.ts isHashedPath``: the path-only half of the archive
|
|
90
|
+
rule (hidden names, pruned directories, the root globs)."""
|
|
91
|
+
parts = relative.split("/")
|
|
92
|
+
name = parts[-1]
|
|
93
|
+
if name.startswith(".") or "__pycache__" in parts:
|
|
94
|
+
return False
|
|
95
|
+
directories = parts[:-1]
|
|
96
|
+
if not directories:
|
|
97
|
+
return _ordinary_root_file(name, broad_root)
|
|
98
|
+
return not any(_skipped_dir(component) for component in directories)
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
def _gitignore_matcher(root: Path) -> Callable[[str], bool] | None:
|
|
102
|
+
gitignore = root / ".gitignore"
|
|
103
|
+
if not gitignore.exists():
|
|
104
|
+
return None
|
|
105
|
+
if parse_gitignore is None:
|
|
106
|
+
raise RuntimeError(
|
|
107
|
+
"A .gitignore is present but 'gitignore_parser' is not installed. Install it (a declared "
|
|
108
|
+
"dependency of gatewaysdk) so the content hash stays compatible with the hub."
|
|
109
|
+
)
|
|
110
|
+
return parse_gitignore(str(gitignore), base_dir=str(root))
|
|
111
|
+
|
|
112
|
+
|
|
74
113
|
def pyproject_pins_v1(path: str, text: str) -> bool:
|
|
75
114
|
if path != PYPROJECT:
|
|
76
115
|
return False
|
|
@@ -179,6 +218,83 @@ class WorldTree:
|
|
|
179
218
|
def has_connector(self) -> bool:
|
|
180
219
|
return self.has(self.world_path(CONNECTOR_SPEC))
|
|
181
220
|
|
|
221
|
+
# ── The archive: the file set a version IS ───────────────────────────────
|
|
222
|
+
#
|
|
223
|
+
# Vendored from Prime Intellect's ``prime`` CLI (MIT; THIRD_PARTY_LICENSES)
|
|
224
|
+
# so a container hashed here resolves to the same logical version on their
|
|
225
|
+
# Environments Hub: SHA-256 over ``file:<relpath>`` + raw bytes, in sorted
|
|
226
|
+
# path order, over README.md / pyproject.toml / *.py at the root plus every
|
|
227
|
+
# non-hidden nested file outside dist, build, outputs, __pycache__ and
|
|
228
|
+
# *.egg-info. A root Gateway marker widens the root to every ordinary file
|
|
229
|
+
# (the marker itself must be hashed or a manifest-only edit dedups into the
|
|
230
|
+
# previous version and is silently discarded). Symlinks and ``.gitignore``-d
|
|
231
|
+
# paths are dropped. This is the version primary key, NOT a tarball digest.
|
|
232
|
+
# Twin: ``packages/shared/src/world/worldTree.ts`` (``isHashedPath`` +
|
|
233
|
+
# ``contentHash``); the corpus under ``conformance/worlds/`` pins both.
|
|
234
|
+
|
|
235
|
+
@property
|
|
236
|
+
def broad_root(self) -> bool:
|
|
237
|
+
return any(self.has(marker) for marker in _BROAD_ROOT_MARKERS)
|
|
238
|
+
|
|
239
|
+
def archive_files(self) -> list[Path]:
|
|
240
|
+
"""The hashed files, absolute, in sorted relative-path order."""
|
|
241
|
+
ignored = _gitignore_matcher(self.root)
|
|
242
|
+
broad = self.broad_root
|
|
243
|
+
found: dict[str, Path] = {}
|
|
244
|
+
|
|
245
|
+
def keep(path: Path) -> None:
|
|
246
|
+
if not path.is_file() or path.is_symlink():
|
|
247
|
+
return
|
|
248
|
+
relative = path.relative_to(self.root).as_posix()
|
|
249
|
+
if not is_hashed_path(relative, broad):
|
|
250
|
+
return
|
|
251
|
+
if ignored is not None and ignored(str(path)):
|
|
252
|
+
return
|
|
253
|
+
found[relative] = path
|
|
254
|
+
|
|
255
|
+
def pruned(directory: Path) -> bool:
|
|
256
|
+
"""A symlinked directory is not part of the tree: it can point anywhere."""
|
|
257
|
+
return (
|
|
258
|
+
directory.is_symlink()
|
|
259
|
+
or _skipped_dir(directory.name)
|
|
260
|
+
or (ignored is not None and ignored(str(directory)))
|
|
261
|
+
)
|
|
262
|
+
|
|
263
|
+
for entry in sorted(self.root.iterdir(), key=lambda p: p.name):
|
|
264
|
+
if entry.is_file():
|
|
265
|
+
keep(entry)
|
|
266
|
+
continue
|
|
267
|
+
if not entry.is_dir() or pruned(entry):
|
|
268
|
+
continue
|
|
269
|
+
for dirpath, dirnames, filenames in os.walk(entry):
|
|
270
|
+
here = Path(dirpath)
|
|
271
|
+
dirnames[:] = sorted(name for name in dirnames if not pruned(here / name))
|
|
272
|
+
for filename in sorted(filenames):
|
|
273
|
+
keep(here / filename)
|
|
274
|
+
return [found[relative] for relative in sorted(found)]
|
|
275
|
+
|
|
276
|
+
def hashed_paths(self) -> list[str]:
|
|
277
|
+
return [path.relative_to(self.root).as_posix() for path in self.archive_files()]
|
|
278
|
+
|
|
279
|
+
@property
|
|
280
|
+
def content_hash(self) -> str:
|
|
281
|
+
"""Streams every file; a large tree never has to fit in memory.
|
|
282
|
+
|
|
283
|
+
An unreadable file raises: a digest that silently omitted it would
|
|
284
|
+
name a tree the manifest and archive do not describe.
|
|
285
|
+
"""
|
|
286
|
+
digest = hashlib.sha256()
|
|
287
|
+
for path in self.archive_files():
|
|
288
|
+
relative = path.relative_to(self.root).as_posix()
|
|
289
|
+
digest.update(f"file:{relative}".encode("utf-8"))
|
|
290
|
+
try:
|
|
291
|
+
with open(path, "rb") as handle:
|
|
292
|
+
for chunk in iter(lambda: handle.read(_HASH_CHUNK), b""):
|
|
293
|
+
digest.update(chunk)
|
|
294
|
+
except OSError as error:
|
|
295
|
+
raise OSError(f"cannot read {relative} while hashing the tree: {error}") from error
|
|
296
|
+
return digest.hexdigest()
|
|
297
|
+
|
|
182
298
|
@cached_property
|
|
183
299
|
def manifest(self) -> WorldManifest:
|
|
184
300
|
path = self.world / WORLD_MANIFEST
|
|
@@ -204,8 +320,8 @@ class WorldTree:
|
|
|
204
320
|
def v1_evidence(self) -> Iterator[tuple[str, str]]:
|
|
205
321
|
"""``(relpath, text)`` of the files a verifiers package's API version is
|
|
206
322
|
read from: the root pyproject and every ``.py`` the archive rules keep
|
|
207
|
-
(``
|
|
208
|
-
|
|
323
|
+
(``archive_files``: no hidden, build, dist, outputs, pycache or egg-info
|
|
324
|
+
directories)."""
|
|
209
325
|
for dirpath, dirnames, filenames in os.walk(self.root):
|
|
210
326
|
dirnames[:] = sorted(name for name in dirnames if not _skipped_dir(name))
|
|
211
327
|
for filename in sorted(filenames):
|
|
@@ -70,9 +70,9 @@ Expected output (JSON): `"kind": "harbor"`, `"tasks": ["hello"]`, the files, and
|
|
|
70
70
|
code-bench/
|
|
71
71
|
benchmark.json # kind, models, rollouts: the defaults `benchmarks run` reads
|
|
72
72
|
pyproject.toml # [project] name + version: the slug and the version `push` records; not a Python package
|
|
73
|
-
tasks/hello/task.toml # version, [metadata], [agent] timeout_sec, [verifier] timeout_sec
|
|
73
|
+
tasks/hello/task.toml # version, [metadata], [agent] timeout_sec, [verifier] timeout_sec; docker_image only as a comment
|
|
74
74
|
tasks/hello/instruction.md # what the agent is told
|
|
75
|
-
tasks/hello/environment/Dockerfile # the task's container (FROM
|
|
75
|
+
tasks/hello/environment/Dockerfile # the task's container, built by the runner (FROM --image; WORKDIR /app): the ONE image source
|
|
76
76
|
tasks/hello/tests/test.sh # the grader: runs after the agent, writes /logs/verifier/reward.json
|
|
77
77
|
tasks/hello/solution/solve.sh # the reference solution; must score 1.0 (harbor's oracle agent runs it)
|
|
78
78
|
```
|
|
@@ -82,7 +82,40 @@ The reward contract is the only thing the platform reads: `tests/test.sh` writes
|
|
|
82
82
|
other key a sub-reward — or a bare number to `/logs/verifier/reward.txt`. Any language, any method.
|
|
83
83
|
|
|
84
84
|
Add a task: copy `tasks/hello` to `tasks/<name>`, edit `instruction.md`, `tests/test.sh`,
|
|
85
|
-
`solution/solve.sh`, and the image
|
|
85
|
+
`solution/solve.sh`, and the image. Two tasks minimum.
|
|
86
|
+
|
|
87
|
+
**One image source.** Harbor's rule (`should_use_prebuilt_docker_image`): when `task.toml` names
|
|
88
|
+
`[environment] docker_image`, the runner pulls that prebuilt image and runs it as it is —
|
|
89
|
+
`environment/Dockerfile` is **never built**, so anything it installs is absent on the runner (a tool the
|
|
90
|
+
agent needs, a pip package) while a Dockerfile-first local run passes. Without `docker_image` the
|
|
91
|
+
runner builds the Dockerfile. The scaffold therefore writes only the Dockerfile; put what the agent
|
|
92
|
+
needs in it. To run a prebuilt image instead, delete the Dockerfile and set `docker_image`.
|
|
93
|
+
`validate` fails a task that has both when the Dockerfile adds anything beyond `FROM`/`WORKDIR`
|
|
94
|
+
(a note when it does not), and `dry-run` follows the runner's choice (`image … (prebuilt: docker_image)`,
|
|
95
|
+
`pull`, no `build`).
|
|
96
|
+
|
|
97
|
+
**Secrets (an API key the task needs).** Declare it in the task's `environment/docker-compose.yaml`
|
|
98
|
+
— an overlay harbor applies on top of the image, prebuilt or built — and set it on the hosted
|
|
99
|
+
environment:
|
|
100
|
+
|
|
101
|
+
```yaml
|
|
102
|
+
# tasks/q1/environment/docker-compose.yaml
|
|
103
|
+
services:
|
|
104
|
+
main:
|
|
105
|
+
environment:
|
|
106
|
+
EXA_API_KEY: ${EXA_API_KEY:?set with gateway bench secrets set}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
gateway bench secrets set --env code-bench --key EXA_API_KEY # paste the value, then Ctrl-D; never on the command line
|
|
111
|
+
gateway bench secrets list --env code-bench
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
`${NAME:?…}` (and a bare `${NAME}`) is a requirement; `${NAME:-default}` is not. `validate` lists a
|
|
115
|
+
task's required names as `secrets …`; `run` refuses before dispatch when one is not set on the
|
|
116
|
+
environment, naming the exact `bench secrets set` command. `gateway secret …` is a different store
|
|
117
|
+
(the platform's secret vault for agents and hooks) and does **not** feed hosted harbor runs.
|
|
118
|
+
`run_parameters.json` does not either. Locally, `dry-run` exports what your shell holds — see A3.
|
|
86
119
|
|
|
87
120
|
MCP: no twin for a local scaffold; on an existing environment, `env_write_file({ containerId, path,
|
|
88
121
|
newContent })` for each file, then `env_commit`.
|
|
@@ -97,15 +130,18 @@ Expected output:
|
|
|
97
130
|
|
|
98
131
|
```
|
|
99
132
|
./code-bench: harbor benchmark — runner reads it as harbor — ok
|
|
100
|
-
ok hello: image
|
|
133
|
+
ok hello: image build tasks/hello/environment/Dockerfile; instruction yes; tests/test.sh yes; solution yes
|
|
101
134
|
benchmark ok (harbor): 1 task(s): hello; next: gateway benchmarks dry-run ./code-bench
|
|
102
135
|
```
|
|
103
136
|
|
|
104
|
-
Per task: `task.toml` present,
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
137
|
+
Per task: `task.toml` present, ONE image source (`[environment] docker_image` → `image <ref> (prebuilt:
|
|
138
|
+
docker_image)`, else `environment/Dockerfile` → `image build …`; both is a problem when the Dockerfile
|
|
139
|
+
adds anything), the compose overlay and its `secrets` when the task has one, `instruction.md`
|
|
140
|
+
non-empty, `tests/test.sh` present and writing the reward file, `solution/solve.sh` noted when absent.
|
|
141
|
+
Tree-level: `pyproject.toml` (a note when absent) and that the runner's own dispatch rule
|
|
142
|
+
(`detect_env_kind`) reads the tree as harbor — a `gateway-env.toml` at the root would make it a world
|
|
143
|
+
to the runner. Exit 2 on any problem; notes never fail it. The closing line counts the tasks and names
|
|
144
|
+
the first five (`24 tasks: a, b, c, d, e, … (19 more)`); `--json` lists them all.
|
|
109
145
|
|
|
110
146
|
### A3. Dry-run one task
|
|
111
147
|
|
|
@@ -131,6 +167,16 @@ runner's `harbor run` line, the static check of `tests/test.sh`) followed by
|
|
|
131
167
|
`not executed: … is not installed; execution needs Docker — the plan is what would run`, exit 0.
|
|
132
168
|
`GATEWAY_DOCKER` names the docker binary.
|
|
133
169
|
|
|
170
|
+
The image is chosen by harbor's rule (A1): `docker_image` set → `docker pull`, the Dockerfile is not
|
|
171
|
+
built and the plan says so; otherwise `docker build`. A task with `environment/docker-compose.yaml`
|
|
172
|
+
runs through `docker compose` instead of `docker run` — harbor's base for `main` (the image, the
|
|
173
|
+
mounts, written to `<tmp>/docker-compose.base.yaml`) plus the task's file, then `compose down` — so
|
|
174
|
+
its extra services and `${NAME:?…}` secrets are exercised locally. The names must be exported in
|
|
175
|
+
your shell (`export EXA_API_KEY=…`); a missing one is reported as `not executed: the task's
|
|
176
|
+
docker-compose file requires EXA_API_KEY … export EXA_API_KEY here for the dry-run. Hosted runs read
|
|
177
|
+
it from … gateway bench secrets set --env code-bench --key EXA_API_KEY`, exit 0, nothing spawned.
|
|
178
|
+
Values never appear in the printed argv.
|
|
179
|
+
|
|
134
180
|
### A4. Push
|
|
135
181
|
|
|
136
182
|
```bash
|
|
@@ -146,6 +192,18 @@ tree under a world's slug (exit 2, nothing sent); the hub's versions door refuse
|
|
|
146
192
|
`benchmark_substrate_mismatch`. Later pushes are new content-hashed versions (`gateway bench log`,
|
|
147
193
|
`diff`, `branch`, `tag`).
|
|
148
194
|
|
|
195
|
+
**Which slug.** stderr says where it came from: `slug code-bench (from benchmark.json)`. Order:
|
|
196
|
+
`--slug` > the dir's `.gateway/.env-metadata.json` (written by every push and pull, so a dir keeps
|
|
197
|
+
pushing to the slug it became) > `benchmark.json` > `pyproject.toml [project].name` > the directory
|
|
198
|
+
name. A tracked slug that disagrees with what the tree declares is refused with exit 2 and nothing
|
|
199
|
+
sent — `./vendor tracks slug websearch-bench-e2e (.gateway/.env-metadata.json, from an earlier push)
|
|
200
|
+
but its benchmark.json says websearch-bench` — because that is how one `--slug X` push made every
|
|
201
|
+
later bare push land on X. Settle it with `--slug <the one you mean>` (the dir retracks) or delete the
|
|
202
|
+
metadata file. The platform's collision suffix (`code-bench-2` minted for a taken `code-bench`) is
|
|
203
|
+
not a disagreement. A 409 `Branch "main" has moved` (another push landed on the slug since this dir's
|
|
204
|
+
base) names its way out: `gateway benchmarks push ./code-bench --force` moves the branch anyway;
|
|
205
|
+
`gateway bench pull code-bench` into a fresh dir builds on the tip.
|
|
206
|
+
|
|
149
207
|
MCP: `create_environment_version` (or `env_write_file` + `env_commit`).
|
|
150
208
|
|
|
151
209
|
### A5. Run: models × tasks × rollouts
|