@withgateway/sdk 0.1.19 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +141 -2
- package/dist/cli.js +8677 -1320
- package/dist/cli.js.map +1 -1
- package/dist/cli.mjs +8683 -1327
- package/dist/cli.mjs.map +1 -1
- package/dist/index.d.mts +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1005 -55
- package/dist/index.js.map +1 -1
- package/dist/index.mjs +969 -47
- package/dist/index.mjs.map +1 -1
- package/dist/tracing/index.js +9 -2
- package/dist/tracing/index.js.map +1 -1
- package/dist/tracing/index.mjs +9 -2
- package/dist/tracing/index.mjs.map +1 -1
- package/dist/worlds/index.d.mts +738 -28
- package/dist/worlds/index.d.ts +738 -28
- package/dist/worlds/index.js +966 -51
- package/dist/worlds/index.js.map +1 -1
- package/dist/worlds/index.mjs +960 -45
- package/dist/worlds/index.mjs.map +1 -1
- package/package.json +9 -2
- package/runtime/gatewaysdk/__init__.py +0 -0
- package/runtime/gatewaysdk/environment/__init__.py +108 -0
- package/runtime/gatewaysdk/environment/_bundle.py +281 -0
- package/runtime/gatewaysdk/environment/_harbor.py +276 -0
- package/runtime/gatewaysdk/environment/_materialize.py +97 -0
- package/runtime/gatewaysdk/environment/_tar.py +33 -0
- package/runtime/gatewaysdk/environment/_world.py +346 -0
- package/runtime/gatewaysdk/environment/core.py +527 -0
- package/runtime/gatewaysdk/environment/errors.py +58 -0
- package/runtime/gatewaysdk/environment/runtime.py +150 -0
- package/runtime/gatewaysdk/environment/schema/__init__.py +35 -0
- package/runtime/gatewaysdk/environment/schema/__main__.py +225 -0
- package/runtime/gatewaysdk/environment/schema/_slack_fidelity.py +116 -0
- package/runtime/gatewaysdk/environment/schema/_slack_scenario.py +183 -0
- package/runtime/gatewaysdk/environment/schema/api.py +1628 -0
- package/runtime/gatewaysdk/environment/schema/batch.py +755 -0
- package/runtime/gatewaysdk/environment/schema/compiler.py +615 -0
- package/runtime/gatewaysdk/environment/schema/conform.py +428 -0
- package/runtime/gatewaysdk/environment/schema/connector.py +777 -0
- package/runtime/gatewaysdk/environment/schema/connectors/apple-business-manager/handlers.py +17 -0
- package/runtime/gatewaysdk/environment/schema/connectors/apple-business-manager/parity.json +28 -0
- package/runtime/gatewaysdk/environment/schema/connectors/apple-business-manager/provenance.json +233 -0
- package/runtime/gatewaysdk/environment/schema/connectors/apple-business-manager/scope.toml +112 -0
- package/runtime/gatewaysdk/environment/schema/connectors/apple-business-manager/world.json +1380 -0
- package/runtime/gatewaysdk/environment/schema/connectors/base.json +28 -0
- package/runtime/gatewaysdk/environment/schema/connectors/custom/handlers.py +18 -0
- package/runtime/gatewaysdk/environment/schema/connectors/custom/world.json +123 -0
- package/runtime/gatewaysdk/environment/schema/connectors/github/handlers.py +81 -0
- package/runtime/gatewaysdk/environment/schema/connectors/github/parity.json +37 -0
- package/runtime/gatewaysdk/environment/schema/connectors/github/provenance.json +134 -0
- package/runtime/gatewaysdk/environment/schema/connectors/github/scope.toml +158 -0
- package/runtime/gatewaysdk/environment/schema/connectors/github/world.json +5869 -0
- package/runtime/gatewaysdk/environment/schema/connectors/google-calendar/handlers.py +23 -0
- package/runtime/gatewaysdk/environment/schema/connectors/google-calendar/world.json +209 -0
- package/runtime/gatewaysdk/environment/schema/connectors/google-drive/handlers.py +18 -0
- package/runtime/gatewaysdk/environment/schema/connectors/google-drive/world.json +286 -0
- package/runtime/gatewaysdk/environment/schema/connectors/jamf/handlers.py +34 -0
- package/runtime/gatewaysdk/environment/schema/connectors/jamf/parity.json +49 -0
- package/runtime/gatewaysdk/environment/schema/connectors/jamf/provenance.json +181 -0
- package/runtime/gatewaysdk/environment/schema/connectors/jamf/scope.toml +128 -0
- package/runtime/gatewaysdk/environment/schema/connectors/jamf/world.json +2218 -0
- package/runtime/gatewaysdk/environment/schema/connectors/jira/handlers.py +28 -0
- package/runtime/gatewaysdk/environment/schema/connectors/jira/world.json +389 -0
- package/runtime/gatewaysdk/environment/schema/connectors/kandji/handlers.py +38 -0
- package/runtime/gatewaysdk/environment/schema/connectors/kandji/parity.json +29 -0
- package/runtime/gatewaysdk/environment/schema/connectors/kandji/provenance.json +152 -0
- package/runtime/gatewaysdk/environment/schema/connectors/kandji/scope.toml +120 -0
- package/runtime/gatewaysdk/environment/schema/connectors/kandji/world.json +765 -0
- package/runtime/gatewaysdk/environment/schema/connectors/linear/handlers.py +20 -0
- package/runtime/gatewaysdk/environment/schema/connectors/linear/world.json +363 -0
- package/runtime/gatewaysdk/environment/schema/connectors/microsoft-teams/fidelity.json +45 -0
- package/runtime/gatewaysdk/environment/schema/connectors/microsoft-teams/handlers.py +1 -0
- package/runtime/gatewaysdk/environment/schema/connectors/microsoft-teams/provenance.json +851 -0
- package/runtime/gatewaysdk/environment/schema/connectors/microsoft-teams/scope.md +231 -0
- package/runtime/gatewaysdk/environment/schema/connectors/microsoft-teams/teams-types.json +1710 -0
- package/runtime/gatewaysdk/environment/schema/connectors/microsoft-teams/world.json +858 -0
- package/runtime/gatewaysdk/environment/schema/connectors/netsuite/handlers.py +32 -0
- package/runtime/gatewaysdk/environment/schema/connectors/netsuite/parity.json +50 -0
- package/runtime/gatewaysdk/environment/schema/connectors/netsuite/provenance.json +137 -0
- package/runtime/gatewaysdk/environment/schema/connectors/netsuite/scope.toml +78 -0
- package/runtime/gatewaysdk/environment/schema/connectors/netsuite/world.json +16243 -0
- package/runtime/gatewaysdk/environment/schema/connectors/salesforce/handlers.py +806 -0
- package/runtime/gatewaysdk/environment/schema/connectors/salesforce/parity.json +46 -0
- package/runtime/gatewaysdk/environment/schema/connectors/salesforce/provenance.json +143 -0
- package/runtime/gatewaysdk/environment/schema/connectors/salesforce/scope.toml +174 -0
- package/runtime/gatewaysdk/environment/schema/connectors/salesforce/world.json +4695 -0
- package/runtime/gatewaysdk/environment/schema/connectors/slack/MODEL.md +236 -0
- package/runtime/gatewaysdk/environment/schema/connectors/slack/capabilities.json +1039 -0
- package/runtime/gatewaysdk/environment/schema/connectors/slack/handlers.py +762 -0
- package/runtime/gatewaysdk/environment/schema/connectors/slack/parity.json +65 -0
- package/runtime/gatewaysdk/environment/schema/connectors/slack/provenance.json +294 -0
- package/runtime/gatewaysdk/environment/schema/connectors/slack/scope.toml +44 -0
- package/runtime/gatewaysdk/environment/schema/connectors/slack/world.json +4993 -0
- package/runtime/gatewaysdk/environment/schema/connectors/workday/handlers.py +75 -0
- package/runtime/gatewaysdk/environment/schema/connectors/workday/parity.json +36 -0
- package/runtime/gatewaysdk/environment/schema/connectors/workday/provenance.json +179 -0
- package/runtime/gatewaysdk/environment/schema/connectors/workday/scope.toml +180 -0
- package/runtime/gatewaysdk/environment/schema/connectors/workday/world.json +806 -0
- package/runtime/gatewaysdk/environment/schema/data.py +593 -0
- package/runtime/gatewaysdk/environment/schema/extract.py +263 -0
- package/runtime/gatewaysdk/environment/schema/host.py +2407 -0
- package/runtime/gatewaysdk/environment/schema/host_surfaces.py +275 -0
- package/runtime/gatewaysdk/environment/schema/platform.py +1272 -0
- package/runtime/gatewaysdk/environment/schema/scaffold.py +250 -0
- package/runtime/gatewaysdk/environment/schema/skill.py +118 -0
- package/runtime/gatewaysdk/environment/schema/skills/agents/connector-template-author.md +202 -0
- package/runtime/gatewaysdk/environment/schema/skills/connector-schema-authoring/CONNECTORS.md +331 -0
- package/runtime/gatewaysdk/environment/schema/skills/connector-schema-authoring/CONVENTIONS.md +100 -0
- package/runtime/gatewaysdk/environment/schema/skills/connector-schema-authoring/SKILL.md +313 -0
- package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/ROWS.md +223 -0
- package/runtime/gatewaysdk/environment/schema/skills/world-data-ingestion/SKILL.md +293 -0
- package/runtime/gatewaysdk/environment/schema/skills/worlds-getting-started/SKILL.md +390 -0
- package/runtime/gatewaysdk/environment/schema/slack.py +757 -0
- package/runtime/gatewaysdk/environment/schema/snapshot.py +479 -0
- package/runtime/gatewaysdk/environment/schema/store.py +904 -0
- package/runtime/gatewaysdk/environment/schema/tasks.py +525 -0
- package/runtime/gatewaysdk/environment/schema/tools_world.py +322 -0
- package/runtime/gatewaysdk/environment/schema/validation.py +577 -0
- package/runtime/gatewaysdk/environment/schema/worker.py +78 -0
- package/runtime/gatewaysdk/environment/schema/workers.py +500 -0
- package/runtime/gatewaysdk/environment/schema/world_tests.py +336 -0
- package/skills/agents/connector-template-author.md +202 -0
- package/skills/connector-schema-authoring/CONNECTORS.md +331 -0
- package/skills/connector-schema-authoring/CONVENTIONS.md +100 -0
- package/skills/connector-schema-authoring/SKILL.md +313 -0
- package/skills/world-data-ingestion/ROWS.md +223 -0
- package/skills/world-data-ingestion/SKILL.md +293 -0
- package/skills/worlds-getting-started/SKILL.md +390 -0
|
@@ -0,0 +1,336 @@
|
|
|
1
|
+
"""World tests: the checks a world ships for itself, run against the world it ships with.
|
|
2
|
+
|
|
3
|
+
A test is not an eval. An eval grades an AGENT against a task (scenario graders, rubric
|
|
4
|
+
judges, run configs); a test asserts the WORLD itself answers the way its author says it
|
|
5
|
+
must, the way a unit test asserts a function. They live in ``<world>/tests/`` and run on
|
|
6
|
+
every push (``gateway-env.toml`` ``[actions] on_push = ["tests"]``), locally
|
|
7
|
+
(``gateway worlds test <dir>``) and on the platform (``POST /api/public/worlds/{slug}/tests``).
|
|
8
|
+
|
|
9
|
+
Two shapes, both discovered from ``tests/``:
|
|
10
|
+
|
|
11
|
+
* ``*.json`` — HTTP cases against the served vendor API, in-process, no network::
|
|
12
|
+
|
|
13
|
+
{"cases": [{"name": "unknown IP answers empty",
|
|
14
|
+
"request": {"method": "GET", "path": "/api/v1/ips/8.8.8.8/activities", "auth": true},
|
|
15
|
+
"expect": {"status": 200, "json": []}}]}
|
|
16
|
+
|
|
17
|
+
``request``: ``method``, ``path`` (the vendor path, base-url prefix optional), ``query``,
|
|
18
|
+
``headers``, ``body`` (JSON), ``auth`` (default true: the runner presents the world's own
|
|
19
|
+
credential the way the World Host does, an oauth2 world through its own grant).
|
|
20
|
+
``expect``: ``status`` (int or list), ``json`` (a subset the answer must contain: object
|
|
21
|
+
keys recursively, list items by index), ``length`` ({``path``, ``eq``/``min``/``max``} over
|
|
22
|
+
a dotted path into the answer), ``contains`` (substrings of the body), ``headers`` (subset).
|
|
23
|
+
|
|
24
|
+
* ``test_*.py`` / ``*_test.py`` — pytest, with ``WORLD_URL`` and ``WORLD_API_KEY`` in the
|
|
25
|
+
environment, the world served on a local port for the duration. Skipped, not failed,
|
|
26
|
+
where pytest is not installed.
|
|
27
|
+
"""
|
|
28
|
+
|
|
29
|
+
from __future__ import annotations
|
|
30
|
+
|
|
31
|
+
import json
|
|
32
|
+
import os
|
|
33
|
+
import subprocess
|
|
34
|
+
import sys
|
|
35
|
+
import tempfile
|
|
36
|
+
import threading
|
|
37
|
+
import urllib.error
|
|
38
|
+
import urllib.parse
|
|
39
|
+
import urllib.request
|
|
40
|
+
from dataclasses import asdict, dataclass, field
|
|
41
|
+
from pathlib import Path
|
|
42
|
+
from typing import Any
|
|
43
|
+
|
|
44
|
+
from .api import serve
|
|
45
|
+
from .conform import PIN, _credentials
|
|
46
|
+
from .connector import load_spec
|
|
47
|
+
from .api import load_api_settings
|
|
48
|
+
from .validation import ContractError, fail
|
|
49
|
+
|
|
50
|
+
TESTS_DIR = "tests"
|
|
51
|
+
CASE_TIMEOUT_S = 30
|
|
52
|
+
PYTEST_TIMEOUT_S = 600
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
@dataclass
|
|
56
|
+
class CaseResult:
|
|
57
|
+
name: str
|
|
58
|
+
file: str
|
|
59
|
+
status: str # passed | failed | error
|
|
60
|
+
request: dict = field(default_factory=dict)
|
|
61
|
+
expected: dict = field(default_factory=dict)
|
|
62
|
+
actual: dict = field(default_factory=dict)
|
|
63
|
+
problems: list[str] = field(default_factory=list)
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
@dataclass
|
|
67
|
+
class PythonResult:
|
|
68
|
+
status: str # passed | failed | skipped | none
|
|
69
|
+
files: list[str] = field(default_factory=list)
|
|
70
|
+
tests: int = 0
|
|
71
|
+
failures: int = 0
|
|
72
|
+
errors: int = 0
|
|
73
|
+
skipped: int = 0
|
|
74
|
+
reason: str | None = None
|
|
75
|
+
output: str = ""
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
@dataclass
|
|
79
|
+
class Report:
|
|
80
|
+
world: str
|
|
81
|
+
cases: list[CaseResult]
|
|
82
|
+
python: PythonResult
|
|
83
|
+
scenario: str | None = None
|
|
84
|
+
|
|
85
|
+
@property
|
|
86
|
+
def summary(self) -> dict:
|
|
87
|
+
passed = sum(1 for c in self.cases if c.status == "passed")
|
|
88
|
+
failed = sum(1 for c in self.cases if c.status != "passed")
|
|
89
|
+
if self.python.status == "failed":
|
|
90
|
+
failed += max(1, self.python.failures + self.python.errors)
|
|
91
|
+
if self.python.status == "passed":
|
|
92
|
+
passed += self.python.tests
|
|
93
|
+
return {
|
|
94
|
+
"total": passed + failed,
|
|
95
|
+
"passed": passed,
|
|
96
|
+
"failed": failed,
|
|
97
|
+
"skipped": self.python.skipped + (1 if self.python.status == "skipped" else 0),
|
|
98
|
+
"status": "passed" if failed == 0 else "failed",
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
def to_json(self) -> dict:
|
|
102
|
+
return {
|
|
103
|
+
"world": self.world,
|
|
104
|
+
"scenario": self.scenario,
|
|
105
|
+
"summary": self.summary,
|
|
106
|
+
"cases": [asdict(c) for c in self.cases],
|
|
107
|
+
"python": asdict(self.python),
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
|
|
111
|
+
# ---------------------------------------------------------------- discovery
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
def discover(root: str | Path) -> tuple[list[tuple[str, dict]], list[Path]]:
|
|
115
|
+
"""(json cases as (file, case) pairs, python test files) under ``<root>/tests``."""
|
|
116
|
+
tests = Path(root) / TESTS_DIR
|
|
117
|
+
if not tests.is_dir():
|
|
118
|
+
return [], []
|
|
119
|
+
cases: list[tuple[str, dict]] = []
|
|
120
|
+
for path in sorted(tests.rglob("*.json")):
|
|
121
|
+
try:
|
|
122
|
+
document = json.loads(path.read_text("utf-8"))
|
|
123
|
+
except (OSError, json.JSONDecodeError) as error:
|
|
124
|
+
fail("tests", f"{path.relative_to(root)}: not valid JSON ({error})", f"/tests/{path.name}")
|
|
125
|
+
listed = document.get("cases") if isinstance(document, dict) else document
|
|
126
|
+
if not isinstance(listed, list):
|
|
127
|
+
fail("tests", f"{path.relative_to(root)}: expected a list of cases or {{\"cases\": [...]}}", f"/tests/{path.name}")
|
|
128
|
+
for index, case in enumerate(listed):
|
|
129
|
+
if not isinstance(case, dict) or not isinstance(case.get("request"), dict) or not isinstance(case.get("expect"), dict):
|
|
130
|
+
fail("tests", f"{path.relative_to(root)}[{index}]: a case is {{name, request, expect}}", f"/tests/{path.name}")
|
|
131
|
+
cases.append((str(path.relative_to(root)), case))
|
|
132
|
+
python = sorted(p for p in tests.rglob("*.py") if p.name.startswith("test_") or p.name.endswith("_test.py"))
|
|
133
|
+
return cases, python
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
# ---------------------------------------------------------------- predicates
|
|
137
|
+
|
|
138
|
+
|
|
139
|
+
def _subset(expected: Any, actual: Any, at: str, problems: list[str]) -> None:
|
|
140
|
+
if isinstance(expected, dict):
|
|
141
|
+
if not isinstance(actual, dict):
|
|
142
|
+
problems.append(f"{at}: expected an object, got {type(actual).__name__}")
|
|
143
|
+
return
|
|
144
|
+
for key, value in expected.items():
|
|
145
|
+
if key not in actual:
|
|
146
|
+
problems.append(f"{at}.{key}: missing")
|
|
147
|
+
continue
|
|
148
|
+
_subset(value, actual[key], f"{at}.{key}", problems)
|
|
149
|
+
return
|
|
150
|
+
if isinstance(expected, list):
|
|
151
|
+
if not isinstance(actual, list):
|
|
152
|
+
problems.append(f"{at}: expected a list, got {type(actual).__name__}")
|
|
153
|
+
return
|
|
154
|
+
if len(actual) < len(expected):
|
|
155
|
+
problems.append(f"{at}: expected at least {len(expected)} items, got {len(actual)}")
|
|
156
|
+
return
|
|
157
|
+
for index, value in enumerate(expected):
|
|
158
|
+
_subset(value, actual[index], f"{at}[{index}]", problems)
|
|
159
|
+
return
|
|
160
|
+
if expected != actual:
|
|
161
|
+
problems.append(f"{at}: expected {json.dumps(expected)}, got {json.dumps(actual)[:120]}")
|
|
162
|
+
|
|
163
|
+
|
|
164
|
+
def _at_path(value: Any, path: str) -> Any:
|
|
165
|
+
if path in ("", "$", "."):
|
|
166
|
+
return value
|
|
167
|
+
for part in path.lstrip("$").strip(".").split("."):
|
|
168
|
+
if isinstance(value, list) and part.isdigit():
|
|
169
|
+
value = value[int(part)] if int(part) < len(value) else None
|
|
170
|
+
elif isinstance(value, dict):
|
|
171
|
+
value = value.get(part)
|
|
172
|
+
else:
|
|
173
|
+
return None
|
|
174
|
+
return value
|
|
175
|
+
|
|
176
|
+
|
|
177
|
+
def check(expect: dict, status: int, headers: dict[str, str], raw: bytes) -> list[str]:
|
|
178
|
+
problems: list[str] = []
|
|
179
|
+
wanted = expect.get("status")
|
|
180
|
+
if wanted is not None:
|
|
181
|
+
allowed = wanted if isinstance(wanted, list) else [wanted]
|
|
182
|
+
if status not in allowed:
|
|
183
|
+
problems.append(f"status: expected {allowed if len(allowed) > 1 else allowed[0]}, got {status}")
|
|
184
|
+
body: Any = None
|
|
185
|
+
if any(k in expect for k in ("json", "length")):
|
|
186
|
+
try:
|
|
187
|
+
body = json.loads(raw) if raw else None
|
|
188
|
+
except json.JSONDecodeError:
|
|
189
|
+
problems.append("body: not JSON")
|
|
190
|
+
return problems
|
|
191
|
+
if "json" in expect:
|
|
192
|
+
_subset(expect["json"], body, "$", problems)
|
|
193
|
+
length = expect.get("length")
|
|
194
|
+
if isinstance(length, dict):
|
|
195
|
+
found = _at_path(body, str(length.get("path", "$")))
|
|
196
|
+
if not isinstance(found, (list, dict, str)):
|
|
197
|
+
problems.append(f"length {length.get('path', '$')}: nothing countable there")
|
|
198
|
+
else:
|
|
199
|
+
n = len(found)
|
|
200
|
+
if "eq" in length and n != length["eq"]:
|
|
201
|
+
problems.append(f"length {length.get('path', '$')}: expected {length['eq']}, got {n}")
|
|
202
|
+
if "min" in length and n < length["min"]:
|
|
203
|
+
problems.append(f"length {length.get('path', '$')}: expected at least {length['min']}, got {n}")
|
|
204
|
+
if "max" in length and n > length["max"]:
|
|
205
|
+
problems.append(f"length {length.get('path', '$')}: expected at most {length['max']}, got {n}")
|
|
206
|
+
for needle in expect.get("contains") or []:
|
|
207
|
+
if str(needle).encode("utf-8") not in raw:
|
|
208
|
+
problems.append(f"body: does not contain {json.dumps(needle)}")
|
|
209
|
+
for name, value in (expect.get("headers") or {}).items():
|
|
210
|
+
if headers.get(name.lower()) != value:
|
|
211
|
+
problems.append(f"header {name}: expected {json.dumps(value)}, got {json.dumps(headers.get(name.lower()))}")
|
|
212
|
+
return problems
|
|
213
|
+
|
|
214
|
+
|
|
215
|
+
# ---------------------------------------------------------------- the run
|
|
216
|
+
|
|
217
|
+
|
|
218
|
+
def run(world: str | Path, *, scenario: str | None = None) -> Report:
|
|
219
|
+
"""Serve the world on a local port and run every test it ships."""
|
|
220
|
+
root = Path(world)
|
|
221
|
+
cases, python_files = discover(root)
|
|
222
|
+
spec = load_spec(root)
|
|
223
|
+
settings = load_api_settings(root, spec)
|
|
224
|
+
server = serve(root, port=0, host="127.0.0.1", api_key=PIN, scenario=scenario)
|
|
225
|
+
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
|
226
|
+
thread.start()
|
|
227
|
+
base = f"http://127.0.0.1:{server.server_port}"
|
|
228
|
+
prefix = urllib.parse.urlsplit(spec.base_url).path.rstrip("/")
|
|
229
|
+
state = server.world_state # type: ignore[attr-defined]
|
|
230
|
+
results: list[CaseResult] = []
|
|
231
|
+
try:
|
|
232
|
+
credentials = _credentials(base, spec, settings) if cases else {}
|
|
233
|
+
for file, case in cases:
|
|
234
|
+
results.append(_run_case(base, prefix, spec, credentials, file, case))
|
|
235
|
+
python = _run_python(root, python_files, base)
|
|
236
|
+
finally:
|
|
237
|
+
server.shutdown()
|
|
238
|
+
server.server_close()
|
|
239
|
+
state.close()
|
|
240
|
+
return Report(str(root), results, python, scenario)
|
|
241
|
+
|
|
242
|
+
|
|
243
|
+
def _run_case(base: str, prefix: str, spec, credentials: dict[str, str], file: str, case: dict) -> CaseResult:
|
|
244
|
+
request = case["request"]
|
|
245
|
+
expect = case["expect"]
|
|
246
|
+
name = str(case.get("name") or f"{request.get('method', 'GET')} {request.get('path', '/')}")
|
|
247
|
+
result = CaseResult(name=name, file=file, status="error", request=request, expected=expect)
|
|
248
|
+
method = str(request.get("method") or "GET").upper()
|
|
249
|
+
path = str(request.get("path") or "/")
|
|
250
|
+
if prefix and not path.startswith(prefix + "/") and path != prefix:
|
|
251
|
+
# a case may name the vendor path bare, or with the base url's prefix
|
|
252
|
+
path = prefix + (path if path.startswith("/") else "/" + path)
|
|
253
|
+
query = dict(request.get("query") or {})
|
|
254
|
+
if request.get("auth", True) and spec.scheme == "query":
|
|
255
|
+
query[spec.auth_name or ""] = PIN
|
|
256
|
+
url = base + path + (f"?{urllib.parse.urlencode(query)}" if query else "")
|
|
257
|
+
headers = {k: str(v) for k, v in (request.get("headers") or {}).items()}
|
|
258
|
+
if request.get("auth", True):
|
|
259
|
+
headers = {**credentials, **headers}
|
|
260
|
+
body = request.get("body")
|
|
261
|
+
data = None
|
|
262
|
+
if body is not None:
|
|
263
|
+
data = json.dumps(body).encode("utf-8") if not isinstance(body, str) else body.encode("utf-8")
|
|
264
|
+
headers.setdefault("Content-Type", "application/json")
|
|
265
|
+
try:
|
|
266
|
+
req = urllib.request.Request(url, method=method, data=data, headers=headers)
|
|
267
|
+
try:
|
|
268
|
+
with urllib.request.urlopen(req, timeout=CASE_TIMEOUT_S) as reply:
|
|
269
|
+
status, raw, got = reply.status, reply.read(), dict(reply.headers.items())
|
|
270
|
+
except urllib.error.HTTPError as error:
|
|
271
|
+
status, raw, got = error.code, error.read(), dict(error.headers.items())
|
|
272
|
+
except (urllib.error.URLError, OSError) as error:
|
|
273
|
+
result.problems = [f"request failed: {error}"]
|
|
274
|
+
return result
|
|
275
|
+
lowered = {k.lower(): v for k, v in got.items()}
|
|
276
|
+
excerpt = raw[:2000].decode("utf-8", "replace")
|
|
277
|
+
result.actual = {"status": status, "body": excerpt, "truncated": len(raw) > 2000}
|
|
278
|
+
result.problems = check(expect, status, lowered, raw)
|
|
279
|
+
result.status = "passed" if not result.problems else "failed"
|
|
280
|
+
return result
|
|
281
|
+
|
|
282
|
+
|
|
283
|
+
def _run_python(root: Path, files: list[Path], base: str) -> PythonResult:
|
|
284
|
+
if not files:
|
|
285
|
+
return PythonResult(status="none")
|
|
286
|
+
names = [str(p.relative_to(root)) for p in files]
|
|
287
|
+
try:
|
|
288
|
+
import pytest # noqa: F401
|
|
289
|
+
except ImportError:
|
|
290
|
+
return PythonResult(status="skipped", files=names, reason="pytest is not installed in this runtime")
|
|
291
|
+
with tempfile.TemporaryDirectory(prefix="gateway-world-tests-") as scratch:
|
|
292
|
+
junit = Path(scratch) / "junit.xml"
|
|
293
|
+
env = {
|
|
294
|
+
**{k: v for k, v in os.environ.items() if k in ("PATH", "LANG", "LC_ALL", "TZ", "HOME", "PYTHONPATH", "VIRTUAL_ENV")},
|
|
295
|
+
"WORLD_URL": base,
|
|
296
|
+
"WORLD_API_KEY": PIN,
|
|
297
|
+
"PYTHONDONTWRITEBYTECODE": "1",
|
|
298
|
+
}
|
|
299
|
+
try:
|
|
300
|
+
proc = subprocess.run(
|
|
301
|
+
[sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", f"--junitxml={junit}", *names],
|
|
302
|
+
cwd=str(root),
|
|
303
|
+
env=env,
|
|
304
|
+
capture_output=True,
|
|
305
|
+
text=True,
|
|
306
|
+
timeout=PYTEST_TIMEOUT_S,
|
|
307
|
+
)
|
|
308
|
+
except subprocess.TimeoutExpired:
|
|
309
|
+
return PythonResult(status="failed", files=names, reason=f"pytest exceeded {PYTEST_TIMEOUT_S}s")
|
|
310
|
+
output = (proc.stdout + proc.stderr)[-4000:]
|
|
311
|
+
counts = _junit_counts(junit)
|
|
312
|
+
if counts is None:
|
|
313
|
+
return PythonResult(status="failed", files=names, reason="pytest wrote no junit report", output=output)
|
|
314
|
+
tests, failures, errors, skipped = counts
|
|
315
|
+
if proc.returncode == 5: # pytest: no tests collected
|
|
316
|
+
return PythonResult(status="none", files=names, reason="no tests collected", output=output)
|
|
317
|
+
status = "passed" if proc.returncode == 0 else "failed"
|
|
318
|
+
return PythonResult(status=status, files=names, tests=tests, failures=failures, errors=errors, skipped=skipped, output=output)
|
|
319
|
+
|
|
320
|
+
|
|
321
|
+
def _junit_counts(path: Path) -> tuple[int, int, int, int] | None:
|
|
322
|
+
if not path.is_file():
|
|
323
|
+
return None
|
|
324
|
+
import xml.etree.ElementTree as ET
|
|
325
|
+
|
|
326
|
+
try:
|
|
327
|
+
tree = ET.parse(path)
|
|
328
|
+
except ET.ParseError:
|
|
329
|
+
return None
|
|
330
|
+
total = failures = errors = skipped = 0
|
|
331
|
+
for suite in tree.iter("testsuite"):
|
|
332
|
+
total += int(suite.get("tests") or 0)
|
|
333
|
+
failures += int(suite.get("failures") or 0)
|
|
334
|
+
errors += int(suite.get("errors") or 0)
|
|
335
|
+
skipped += int(suite.get("skipped") or 0)
|
|
336
|
+
return total, failures, errors, skipped
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: connector-template-author
|
|
3
|
+
model: opus
|
|
4
|
+
description: |
|
|
5
|
+
Implements ONE connector schema template under the unified gateway-world/1
|
|
6
|
+
contract from a brief in .claude/skills/connector-schema-authoring/briefs/.
|
|
7
|
+
Owns exactly one directory (gatewaysdk/environment/schema/connectors/<slug>/)
|
|
8
|
+
plus that connector's rows in the reference tables, and nothing else, so many
|
|
9
|
+
authors run in parallel without overlap. Follows the connector-schema-authoring
|
|
10
|
+
skill checklist and the generation conventions; the conventions test is the
|
|
11
|
+
merge gate.
|
|
12
|
+
|
|
13
|
+
<example>
|
|
14
|
+
Context: Briefs exist for every connector; time to build the templates in parallel.
|
|
15
|
+
user: "Implement the Okta connector template from its brief"
|
|
16
|
+
assistant: "I'll spawn connector-template-author with briefs/okta.md; it owns connectors/okta/ only."
|
|
17
|
+
</example>
|
|
18
|
+
|
|
19
|
+
<example>
|
|
20
|
+
Context: A template exists but was hand-typed before the vendor spec was pinned.
|
|
21
|
+
user: "Re-derive the GitHub template from the published OpenAPI and pin provenance"
|
|
22
|
+
assistant: "I'll spawn connector-template-author with briefs/github.md in re-import mode."
|
|
23
|
+
</example>
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
# Connector Template Author
|
|
27
|
+
|
|
28
|
+
## Role Contract
|
|
29
|
+
|
|
30
|
+
You are: A schema author who turns one connector brief into a faithful `gateway-world/1`
|
|
31
|
+
template. The data model is the deliverable. Tools are optional and only what the brief
|
|
32
|
+
asks for.
|
|
33
|
+
|
|
34
|
+
Goal: `gatewaysdk/sdk/src/gatewaysdk/environment/schema/connectors/<slug>/` contains
|
|
35
|
+
`world.json`, `provenance.json`, `scope.toml` (if imported), and `handlers.py` (if the
|
|
36
|
+
brief ships tools), such that `init --connector <slug>` produces a world that compiles,
|
|
37
|
+
checks, runs, and passes `test_committed_templates_follow_the_generation_conventions`
|
|
38
|
+
and the scaffold suite.
|
|
39
|
+
|
|
40
|
+
Constraints:
|
|
41
|
+
- Write ONLY inside your slug's directory, plus the rows the brief names in
|
|
42
|
+
`CONNECTORS.md` (your connector's row in "Status of templates" and your dossier block)
|
|
43
|
+
and your slug's line in the `PREFIXES` table of `test_schema_scaffold.py`.
|
|
44
|
+
- Never touch `base.json`, `scaffold.py`, the compiler, the store, other templates, or
|
|
45
|
+
another connector's rows. If a shared file must change, stop and report it.
|
|
46
|
+
- Never invent vendor facts. Every entity, field, enum, and error comes from the source the
|
|
47
|
+
brief pins; anything you could not confirm is listed under `unverified` in `provenance.json`.
|
|
48
|
+
- Never put vendor or customer data in `data/records.json` or goldens; synthetic only.
|
|
49
|
+
- Keep every scratch file in your OWN subdirectory (`<scratch root>/<slug>/`). Authors run in
|
|
50
|
+
parallel and share one scratch root; generic names like `roundtrip.py` clobber a peer's work.
|
|
51
|
+
- Never bend the model to fit a tool or an MCP surface; report mismatches in the summary.
|
|
52
|
+
|
|
53
|
+
## Work Protocol
|
|
54
|
+
|
|
55
|
+
### Phase 1: Read, then confirm scope
|
|
56
|
+
|
|
57
|
+
1. Read, in order: the brief; `SKILL.md`, `CONVENTIONS.md`, `CONNECTORS.md` in
|
|
58
|
+
`.claude/skills/connector-schema-authoring/`; `documentation/features/world-schema.md`.
|
|
59
|
+
2. Read one existing template of the same shape as a pattern (the brief names it).
|
|
60
|
+
3. Fetch the pinned sources from the brief. Record URL, version, and SHA-256 for
|
|
61
|
+
`provenance.json` before reading them.
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
SCOPE CONFIRMATION:
|
|
65
|
+
- Brief: briefs/<slug>.md
|
|
66
|
+
- Shape: base | mixed | tenant-shaped
|
|
67
|
+
- Prefix: <prefix_> (reserved in PREFIXES)
|
|
68
|
+
- Files I own: connectors/<slug>/*, my rows in CONNECTORS.md, my PREFIXES line
|
|
69
|
+
- Files I will NOT touch: everything else
|
|
70
|
+
- Sources pinned: [url, version, sha256] ...
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
### Phase 2: Model
|
|
74
|
+
|
|
75
|
+
Build `world.json` entity by entity from the pinned source, in the brief's order:
|
|
76
|
+
|
|
77
|
+
- primary key = the vendor's id; composite keys in containment order
|
|
78
|
+
- relationships on the child, `on_delete: restrict`; membership as its own entity
|
|
79
|
+
- vendor casing on field names; `base.json` primitives by `$ref`
|
|
80
|
+
- nullable vs optional exactly as the vendor documents; enums verbatim, vendor order
|
|
81
|
+
- the error table from the brief's error format
|
|
82
|
+
- for tenant-shaped connectors, the skeleton only: standard objects, and a note in
|
|
83
|
+
`provenance.json` that describe-on-bind is required
|
|
84
|
+
|
|
85
|
+
Write `scope.toml` (imported models) with the allowlist, key overrides, relationship
|
|
86
|
+
hints, `polymorphic`, and the reason for every exclusion.
|
|
87
|
+
|
|
88
|
+
### Phase 2c: Connection and served API (`connector.toml`)
|
|
89
|
+
|
|
90
|
+
When the brief names a vendor API (base URL, auth scheme, operations the agent calls),
|
|
91
|
+
ship `connector.toml` in the template directory. `init` copies it to the world root, where
|
|
92
|
+
`gateway worlds connector capture|ingest`, `gateway worlds serve` and `gateway bench push` read it.
|
|
93
|
+
The format is documented in `SKILL.md` under "Custom connections" and "Serving the vendor's API":
|
|
94
|
+
|
|
95
|
+
- `[connector]` slug and `base_url`; `[auth]` scheme (`header|bearer|basic|query|oauth2|none`),
|
|
96
|
+
`name`, and `secret_env` naming the environment variable, never a value.
|
|
97
|
+
- `[capture]` pacing (`requests_per_minute` low; providers ask us to be sparing), cursor
|
|
98
|
+
pagination when the vendor has it (`cursor_field` in the body or `cursor_header`);
|
|
99
|
+
`[redact]` `hash`/`drop` for every PII field the vendor documents returning, `round` for
|
|
100
|
+
coordinates. Field names keep the vendor's casing.
|
|
101
|
+
- Every operation the vendor's client writes with is a write here: `[operations.api] action`
|
|
102
|
+
`create`/`update`/`delete` with `body`/`defaults`/`generate`, and `action = "handler"` for
|
|
103
|
+
anything computed or stateful (executes, status polls, summaries) in the world's
|
|
104
|
+
`api/handlers.py`. A read-only world is not 1:1; declare `status`, `single`, `envelope`
|
|
105
|
+
and `page` the way the vendor answers.
|
|
106
|
+
- When a vendor OpenAPI exists, `gateway worlds connector conform <world> --openapi <doc>` is the
|
|
107
|
+
definition of done: every documented operation answers the documented status with a body the
|
|
108
|
+
vendor's schema accepts. Seed every entity a by-id route reads; make every create accept the
|
|
109
|
+
minimal body the vendor's request schema requires. Report the summary line.
|
|
110
|
+
- `[[operations]]` for exactly the operations the brief lists as agent-facing: `name`
|
|
111
|
+
(the vendor's operationId where it has one), `method`, `path` with `{args}`, `params`,
|
|
112
|
+
`entity`, and `results` (the dotted path to the row array in the vendor envelope).
|
|
113
|
+
- `[api]` `envelope` matching the vendor's list envelope, and `unauthorized`/`not_found`
|
|
114
|
+
as the vendor's documented error bodies (JSON strings); `[operations.api] filter` mapping
|
|
115
|
+
each path argument to the row field it selects on.
|
|
116
|
+
|
|
117
|
+
Then prove it: `gateway worlds connector validate` on an initialized world, and
|
|
118
|
+
`gateway worlds serve` answering one route from synthetic rows you inserted during
|
|
119
|
+
verification (curl it; check the envelope and the 401 body). Never call the live vendor.
|
|
120
|
+
|
|
121
|
+
### Phase 3: Tools (only if the brief lists them)
|
|
122
|
+
|
|
123
|
+
Generate the default set the brief names (`list_<entities>`, `get_<entity>`, one write)
|
|
124
|
+
with handlers on `gateway_env.store`; raise the vendor's error string where the brief lists
|
|
125
|
+
one. If the brief says "no tools", `x-gateway.tools` is `{}` and there is no `handlers.py`.
|
|
126
|
+
|
|
127
|
+
### Phase 4: Data and verify loop
|
|
128
|
+
|
|
129
|
+
1. Build a handful of synthetic rows per entity, enough to exercise every relationship
|
|
130
|
+
(child-first ordering is fine; the store defers reference checks). A template cannot
|
|
131
|
+
ship `data/records.json` today, so insert them through `store` during verification and
|
|
132
|
+
record the coverage in `provenance.json` rather than committing a fixture file.
|
|
133
|
+
2. In a temp directory: `gateway worlds schema init <tmp> --connector <slug>` then
|
|
134
|
+
`compile`, `check`; `init_from_path`, branch, list every entity, run every tool.
|
|
135
|
+
3. `pytest tests/environment/test_schema_scaffold.py -q` and then the full
|
|
136
|
+
`tests/environment` suite. Iterate until green. Max 3 fix rounds per failure class;
|
|
137
|
+
then document.
|
|
138
|
+
|
|
139
|
+
```
|
|
140
|
+
ITERATION LOG:
|
|
141
|
+
Round 1: [change] → [result]
|
|
142
|
+
Round 2: ...
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
### Phase 5: Reference rows and provenance
|
|
146
|
+
|
|
147
|
+
- Add or update your row in `CONNECTORS.md` "Status of templates" and your dossier block
|
|
148
|
+
if the brief changed a fact.
|
|
149
|
+
- `provenance.json`: `shape`, `source` (`kind`, `url`, `version`, `sha256`),
|
|
150
|
+
`importer` or `authored_by`, `scope_hash`, `generated_at`, `unverified: []`.
|
|
151
|
+
|
|
152
|
+
### Phase 6: Self-check
|
|
153
|
+
|
|
154
|
+
```
|
|
155
|
+
SELF-CHECK BEFORE FINALIZING:
|
|
156
|
+
|
|
157
|
+
SCOPE:
|
|
158
|
+
[ ] Only my directory, my CONNECTORS.md rows, my PREFIXES line changed
|
|
159
|
+
[ ] No shared file changed (base.json, scaffold, compiler, store, other templates)
|
|
160
|
+
|
|
161
|
+
FIDELITY:
|
|
162
|
+
[ ] Every entity, field, enum, and error traces to a pinned source or is listed unverified
|
|
163
|
+
[ ] Keys, relationships, nullability follow the vendor, not my preference
|
|
164
|
+
[ ] Tenant-shaped: skeleton only, describe-on-bind noted
|
|
165
|
+
[ ] No vendor or customer data anywhere
|
|
166
|
+
|
|
167
|
+
CONVENTIONS:
|
|
168
|
+
[ ] $id, package id, prefix, plural names, verb_noun tools, closed inputs, layout
|
|
169
|
+
[ ] test_committed_templates_follow_the_generation_conventions passes
|
|
170
|
+
[ ] Full tests/environment suite passes
|
|
171
|
+
|
|
172
|
+
REPORT:
|
|
173
|
+
[ ] Summary lists: entities, tools, unverified items, mismatches with any MCP surface,
|
|
174
|
+
anything that needs a shared-file change (not done, reported)
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
## Output
|
|
178
|
+
|
|
179
|
+
A single summary in this shape, nothing else:
|
|
180
|
+
|
|
181
|
+
```
|
|
182
|
+
CONNECTOR: <slug> SHAPE: <shape> PREFIX: <prefix_>
|
|
183
|
+
FILES: connectors/<slug>/{world.json, provenance.json[, scope.toml][, handlers.py][, connector.toml]}
|
|
184
|
+
ENTITIES: name (key) [→ relationships] ...
|
|
185
|
+
TOOLS: none | list ...
|
|
186
|
+
CONNECTION: none | <n> operations, auth <scheme>, served <route> -> <status> during verification
|
|
187
|
+
SOURCES: url@version sha256 ...
|
|
188
|
+
UNVERIFIED: ...
|
|
189
|
+
TESTS: scaffold N passed; environment N passed
|
|
190
|
+
NEEDS ORCHESTRATOR: shared-file changes or conflicts, or "none"
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
## Red Lines
|
|
194
|
+
|
|
195
|
+
1. **One directory.** Writing outside your slug's directory and your named rows breaks
|
|
196
|
+
parallel safety; stop and report instead.
|
|
197
|
+
2. **No invented vendor facts.** Unconfirmed becomes `unverified`, never a guess.
|
|
198
|
+
3. **No vendor or customer data** in the template.
|
|
199
|
+
4. **Never edit generated artifacts** (`db/schema.sql`, `runtime.json`, `README.md`,
|
|
200
|
+
`tool-schemas.json`, `lock.json`); they come from `compile`.
|
|
201
|
+
5. **Never bend the model** to satisfy a tool, a surface, or a test; report the mismatch.
|
|
202
|
+
6. **Never skip the test loop** or finish with a red conventions test.
|