@sdxc/spec 0.0.0-pre.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE.md +21 -0
- package/README.md +924 -0
- package/dist/ast.d.ts +193 -0
- package/dist/ast.js +9 -0
- package/dist/builtins.d.ts +29 -0
- package/dist/builtins.js +66 -0
- package/dist/cli.d.ts +21 -0
- package/dist/cli.js +297 -0
- package/dist/diagnostics.d.ts +47 -0
- package/dist/diagnostics.js +8 -0
- package/dist/errors.d.ts +131 -0
- package/dist/errors.js +159 -0
- package/dist/executor.d.ts +66 -0
- package/dist/executor.js +320 -0
- package/dist/expectation.d.ts +61 -0
- package/dist/expectation.js +222 -0
- package/dist/index.d.ts +51 -0
- package/dist/index.js +36 -0
- package/dist/lexer.d.ts +22 -0
- package/dist/lexer.js +284 -0
- package/dist/loader.d.ts +21 -0
- package/dist/loader.js +81 -0
- package/dist/parser.d.ts +24 -0
- package/dist/parser.js +502 -0
- package/dist/permissions.d.ts +139 -0
- package/dist/permissions.js +325 -0
- package/dist/plugin.d.ts +90 -0
- package/dist/plugin.js +9 -0
- package/dist/plugins/browser.d.ts +24 -0
- package/dist/plugins/browser.js +896 -0
- package/dist/plugins/cli.d.ts +17 -0
- package/dist/plugins/cli.js +134 -0
- package/dist/plugins/db-e2e-probe.d.ts +14 -0
- package/dist/plugins/db-e2e-probe.js +112 -0
- package/dist/plugins/db.d.ts +19 -0
- package/dist/plugins/db.js +199 -0
- package/dist/plugins/demo.d.ts +17 -0
- package/dist/plugins/demo.js +70 -0
- package/dist/plugins/env.d.ts +18 -0
- package/dist/plugins/env.js +87 -0
- package/dist/plugins/fs.d.ts +16 -0
- package/dist/plugins/fs.js +415 -0
- package/dist/plugins/http.d.ts +19 -0
- package/dist/plugins/http.js +505 -0
- package/dist/plugins/jwt.d.ts +17 -0
- package/dist/plugins/jwt.js +342 -0
- package/dist/plugins/sample.d.ts +27 -0
- package/dist/plugins/sample.js +400 -0
- package/dist/plugins/url.d.ts +18 -0
- package/dist/plugins/url.js +126 -0
- package/dist/project-config.d.ts +163 -0
- package/dist/project-config.js +497 -0
- package/dist/registry.d.ts +56 -0
- package/dist/registry.js +110 -0
- package/dist/reporter.d.ts +30 -0
- package/dist/reporter.js +237 -0
- package/dist/run.d.ts +74 -0
- package/dist/run.js +179 -0
- package/dist/runner.d.ts +52 -0
- package/dist/runner.js +38 -0
- package/dist/source.d.ts +37 -0
- package/dist/source.js +31 -0
- package/dist/sources.d.ts +45 -0
- package/dist/sources.js +54 -0
- package/dist/tokens.d.ts +34 -0
- package/dist/tokens.js +25 -0
- package/dist/transport-stdio.d.ts +34 -0
- package/dist/transport-stdio.js +400 -0
- package/dist/values.d.ts +48 -0
- package/dist/values.js +52 -0
- package/dist/workers.d.ts +40 -0
- package/dist/workers.js +26 -0
- package/dist/workspace-none.d.ts +23 -0
- package/dist/workspace-none.js +33 -0
- package/dist/workspace.d.ts +47 -0
- package/dist/workspace.js +116 -0
- package/package.json +28 -0
package/README.md
ADDED
|
@@ -0,0 +1,924 @@
|
|
|
1
|
+
# @sdxc/spec
|
|
2
|
+
|
|
3
|
+
Write down how your app should behave, then run it. `spec` is an **executable
|
|
4
|
+
specification runner**: you describe behavior in `.spec` files — setup, action,
|
|
5
|
+
expectation — and it runs each one against your real app, in an isolated
|
|
6
|
+
workspace, under permissions you grant explicitly. The specs don't care _how_
|
|
7
|
+
the app is built (a CLI, an HTTP server, a database, a web page); they only
|
|
8
|
+
describe what it should do, so they stay true as the implementation changes.
|
|
9
|
+
|
|
10
|
+
## Overview
|
|
11
|
+
|
|
12
|
+
A suite is a directory of `.spec` files (conventionally `spec/`). Each file is
|
|
13
|
+
written in a deliberately tiny language — no `if`, no loops, no operators — so a
|
|
14
|
+
spec reads as a linear, diffable list of steps. A `test` declares its setup in
|
|
15
|
+
`given`, its action in `when`, and its checks in `then`. The `spec` CLI loads
|
|
16
|
+
the suite, runs every test in its **own fresh temporary directory**, and prints
|
|
17
|
+
one line per test plus a summary.
|
|
18
|
+
|
|
19
|
+
Two rules shape everything:
|
|
20
|
+
|
|
21
|
+
- **Deny by default.** A test that spawns a process, reaches the network, reads
|
|
22
|
+
an environment variable, or touches files outside its workspace needs an
|
|
23
|
+
explicit `--allow-*` grant. Every denial tells you the exact flag that would
|
|
24
|
+
allow it.
|
|
25
|
+
- **Isolated per test.** Each test gets its own workspace, created before it
|
|
26
|
+
runs and removed after. Tests never see each other's files, so order never
|
|
27
|
+
matters.
|
|
28
|
+
|
|
29
|
+
Exit codes are the contract to script against: **0** everything passed, **1** a
|
|
30
|
+
test failed, **2** a usage or load error (a bad flag, an unreadable suite, a
|
|
31
|
+
parse error).
|
|
32
|
+
|
|
33
|
+
## Quickstart
|
|
34
|
+
|
|
35
|
+
### Get the CLI
|
|
36
|
+
|
|
37
|
+
Inside this repo you can run the CLI straight from source — the dev entry point:
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
bun packages/spec/src/cli.ts run spec
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
For everyday use, compile a single self-contained executable that starts fast
|
|
44
|
+
and runs anywhere (no repo, no `node_modules` beside it):
|
|
45
|
+
|
|
46
|
+
```sh
|
|
47
|
+
cd packages/spec
|
|
48
|
+
bun run build # → packages/spec/bin/spec
|
|
49
|
+
./bin/spec run spec
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Put `bin/spec` on your `PATH` and it's just `spec`. The rest of this guide
|
|
53
|
+
writes `spec`; use whichever launcher you have.
|
|
54
|
+
|
|
55
|
+
### Write a suite
|
|
56
|
+
|
|
57
|
+
Create `spec/greeting.spec`:
|
|
58
|
+
|
|
59
|
+
```
|
|
60
|
+
use fs
|
|
61
|
+
use cli
|
|
62
|
+
|
|
63
|
+
test "the script prints its greeting" {
|
|
64
|
+
given {
|
|
65
|
+
write "index.js" "console.log(\"hello from the workspace\")"
|
|
66
|
+
}
|
|
67
|
+
when {
|
|
68
|
+
let result = run "bun" "index.js"
|
|
69
|
+
}
|
|
70
|
+
then {
|
|
71
|
+
expect result.exit_code 0
|
|
72
|
+
expect result.stdout "hello from the workspace\n"
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
test "the generated config is on disk" {
|
|
77
|
+
given {
|
|
78
|
+
write "package.json" { name: "demo", type: "module" }
|
|
79
|
+
}
|
|
80
|
+
then {
|
|
81
|
+
expect file "package.json" exists
|
|
82
|
+
expect file "package.json" contains "\"type\": \"module\""
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
### Run it
|
|
88
|
+
|
|
89
|
+
`write` creates files in the test's workspace and `run` starts processes there.
|
|
90
|
+
Spawning `bun` is a privileged act, so grant exactly that one executable:
|
|
91
|
+
|
|
92
|
+
```sh
|
|
93
|
+
spec run spec --allow-run=bun
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
```
|
|
97
|
+
✓ the script prints its greeting
|
|
98
|
+
✓ the generated config is on disk
|
|
99
|
+
|
|
100
|
+
2 passed, 0 failed (21ms)
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Drop the grant and the process-spawning test fails **before** `bun` is ever
|
|
104
|
+
launched. Every test denied for the same missing grant collapses into one block
|
|
105
|
+
that names the flag to add and lists the tests it affected:
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
✓ the generated config is on disk
|
|
109
|
+
|
|
110
|
+
✗ Permission denied: run (1 test)
|
|
111
|
+
|
|
112
|
+
The spec attempted to reach:
|
|
113
|
+
> cli.run
|
|
114
|
+
|
|
115
|
+
Re-run with an appropriate permission, for example:
|
|
116
|
+
> spec run --allow-run
|
|
117
|
+
|
|
118
|
+
Affected tests:
|
|
119
|
+
- the script prints its greeting (spec/greeting.spec:9)
|
|
120
|
+
|
|
121
|
+
1 passed, 1 failed (1ms)
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
`spec run` takes a **directory** (default `./spec`); it scans it recursively for
|
|
125
|
+
`.spec` files.
|
|
126
|
+
|
|
127
|
+
## The language by example
|
|
128
|
+
|
|
129
|
+
### Tests and phases
|
|
130
|
+
|
|
131
|
+
A `test` has up to three phase blocks, always in this order: `given` (arrange),
|
|
132
|
+
`when` (act), `then` (assert). Each is optional, but you can't reorder them.
|
|
133
|
+
Lines end statements — there are no semicolons. `#` starts a comment to
|
|
134
|
+
end of line (a `#` inside a string is just text).
|
|
135
|
+
|
|
136
|
+
```
|
|
137
|
+
test "a write is read back verbatim" {
|
|
138
|
+
given {
|
|
139
|
+
write "notes.txt" "remember the milk" # arrange
|
|
140
|
+
}
|
|
141
|
+
when {
|
|
142
|
+
let content = read "notes.txt" # act, capture a value
|
|
143
|
+
}
|
|
144
|
+
then {
|
|
145
|
+
expect content "remember the milk" # assert
|
|
146
|
+
}
|
|
147
|
+
}
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### `expect`
|
|
151
|
+
|
|
152
|
+
`expect` has three forms:
|
|
153
|
+
|
|
154
|
+
```
|
|
155
|
+
expect content "remember the milk" # two values: deep structural equality
|
|
156
|
+
expect true # one value: it must be true
|
|
157
|
+
expect file "notes.txt" exists # observable: assert straight from a capability
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
The observable form reads the world through a capability (`file … exists`,
|
|
161
|
+
`file … contains`, `directory … exists`, `browser.heading …`) and passes when
|
|
162
|
+
that observation holds.
|
|
163
|
+
|
|
164
|
+
### `let` and references
|
|
165
|
+
|
|
166
|
+
`let` binds the result of a step. Reach into a returned object with a dotted
|
|
167
|
+
reference:
|
|
168
|
+
|
|
169
|
+
```
|
|
170
|
+
when {
|
|
171
|
+
let result = run "bun" "build.js"
|
|
172
|
+
}
|
|
173
|
+
then {
|
|
174
|
+
expect result.exit_code 0
|
|
175
|
+
expect result.stdout "built\n"
|
|
176
|
+
}
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
One thing to know: a **bare word** in tool-argument position is a symbol, not a
|
|
180
|
+
variable. `write "f" content` hands the tool the literal word `content`. To pass
|
|
181
|
+
a bound value to a tool, use a dotted reference (`result.stdout`), boxing it in
|
|
182
|
+
an object if needed (`let x = { path: p }` then `write x.path …`).
|
|
183
|
+
|
|
184
|
+
A bare path on the right of `let`/`return` is a reference when its head names a
|
|
185
|
+
binding, but when the head is not a binding and the path resolves to a tool that
|
|
186
|
+
needs no arguments, it is a **zero-argument tool call** — so `let current =
|
|
187
|
+
browser.url` binds that tool's observed value. This works for any argument-less
|
|
188
|
+
tool, and the call is permission-gated like any other.
|
|
189
|
+
|
|
190
|
+
### `eventually`
|
|
191
|
+
|
|
192
|
+
Wrap an observable assertion in `eventually` to retry it until it holds or the
|
|
193
|
+
window ends — for anything that becomes true a moment later (a server coming up,
|
|
194
|
+
an async write landing). A plain assertion checks exactly once.
|
|
195
|
+
|
|
196
|
+
```
|
|
197
|
+
then {
|
|
198
|
+
eventually within 2s {
|
|
199
|
+
expect file "ready.txt" exists
|
|
200
|
+
}
|
|
201
|
+
}
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
Only assertions may be retried — an action (a mutation) inside `eventually` is
|
|
205
|
+
an error, since a retried mutation is not a retried check.
|
|
206
|
+
|
|
207
|
+
### `command` and `fixture`
|
|
208
|
+
|
|
209
|
+
A **command** is a reusable step; a **fixture** is reusable data (its value is
|
|
210
|
+
whatever it `return`s). Define them in the same file, or share them suite-wide
|
|
211
|
+
by putting them under `spec/commands/` and `spec/fixtures/`. Either way they
|
|
212
|
+
resolve **by name** from anywhere — no import, no path — because every
|
|
213
|
+
definition is registered before any test runs.
|
|
214
|
+
|
|
215
|
+
```
|
|
216
|
+
# spec/fixtures/book.spec
|
|
217
|
+
fixture book {
|
|
218
|
+
return { title: "Dune", author: "Herbert", year: 1965 }
|
|
219
|
+
}
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
```
|
|
223
|
+
# spec/commands/seed.spec
|
|
224
|
+
use fs
|
|
225
|
+
|
|
226
|
+
command seed_file(path) {
|
|
227
|
+
let target = { path: path }
|
|
228
|
+
write target.path "seeded"
|
|
229
|
+
}
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
```
|
|
233
|
+
# spec/tour.spec
|
|
234
|
+
test "commands and fixtures compose by name" {
|
|
235
|
+
given {
|
|
236
|
+
seed_file "out.txt" # run a command for its effect
|
|
237
|
+
}
|
|
238
|
+
when {
|
|
239
|
+
let record = fixture book # run a fixture for its value
|
|
240
|
+
}
|
|
241
|
+
then {
|
|
242
|
+
expect record.title "Dune"
|
|
243
|
+
expect file "out.txt" contains "seeded"
|
|
244
|
+
}
|
|
245
|
+
}
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
### `use` and namespaces
|
|
249
|
+
|
|
250
|
+
Capabilities live in namespaces (`fs`, `cli`, `http`, …). Use a fully qualified
|
|
251
|
+
name anywhere (`fs.write`, `cli.run`), or `use` a namespace at the top of a file
|
|
252
|
+
to call its tools by their bare names (`write`, `run`). `use` is **per file**;
|
|
253
|
+
suite commands and fixtures need no `use` at all. If a bare name could mean two
|
|
254
|
+
things, that's an error naming both candidates — the runtime never guesses.
|
|
255
|
+
|
|
256
|
+
```
|
|
257
|
+
use fs # now `write`, `read`, `file`, … are available unqualified
|
|
258
|
+
use cli # now `run` is too
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
## Capabilities
|
|
262
|
+
|
|
263
|
+
Capabilities are the built-in namespaces. `fs` needs no grant (it's confined to
|
|
264
|
+
the workspace); the rest are privileged and denied until you grant them.
|
|
265
|
+
|
|
266
|
+
### `fs` — the workspace filesystem
|
|
267
|
+
|
|
268
|
+
Every path is resolved inside the test's workspace, so `fs` needs no permission.
|
|
269
|
+
Tools: `write`, `read`, `mkdir`, `copy`, `remove`, and the observables `exists`,
|
|
270
|
+
`file`, `directory`. Strings are written verbatim; objects/arrays are written as
|
|
271
|
+
JSON.
|
|
272
|
+
|
|
273
|
+
```
|
|
274
|
+
use fs
|
|
275
|
+
|
|
276
|
+
test "mkdir, copy, and remove move files around the workspace" {
|
|
277
|
+
given {
|
|
278
|
+
mkdir "src"
|
|
279
|
+
write "src/index.ts" "export const answer = 42"
|
|
280
|
+
}
|
|
281
|
+
when {
|
|
282
|
+
copy "src/index.ts" "dist/index.ts"
|
|
283
|
+
remove "src/index.ts"
|
|
284
|
+
}
|
|
285
|
+
then {
|
|
286
|
+
expect directory "src" exists
|
|
287
|
+
expect file "dist/index.ts" contains "answer = 42"
|
|
288
|
+
}
|
|
289
|
+
}
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
### `cli` — run processes · `--allow-run`
|
|
293
|
+
|
|
294
|
+
`run` spawns a program in the workspace and returns `{ stdout, stderr,
|
|
295
|
+
exit_code }`. It needs `--allow-run`, scoped by executable basename. The child
|
|
296
|
+
gets a minimal environment (`PATH`/`HOME`/`TMPDIR` plus only the vars you granted
|
|
297
|
+
with `--allow-env`), so your host environment never leaks in.
|
|
298
|
+
|
|
299
|
+
```
|
|
300
|
+
use cli
|
|
301
|
+
|
|
302
|
+
test "run captures stdout and the exit code" {
|
|
303
|
+
when {
|
|
304
|
+
let result = run "echo" "hello"
|
|
305
|
+
}
|
|
306
|
+
then {
|
|
307
|
+
expect result.exit_code 0
|
|
308
|
+
expect result.stdout "hello\n"
|
|
309
|
+
}
|
|
310
|
+
}
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
```sh
|
|
314
|
+
spec run spec --allow-run=echo
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
### `http` — call an HTTP API · `--allow-net`
|
|
318
|
+
|
|
319
|
+
`get`, `post`, `put`, `patch`, `delete`, each needing `--allow-net` for the
|
|
320
|
+
URL's host (and port, if you scope one). URLs must be **absolute**. An optional
|
|
321
|
+
bare body travels as `text/plain` when it's a string, JSON otherwise. Each returns
|
|
322
|
+
`{ status, ok, headers, text, json }`; an HTTP error status is a normal value —
|
|
323
|
+
only a network-level failure is an error.
|
|
324
|
+
|
|
325
|
+
```
|
|
326
|
+
use http
|
|
327
|
+
|
|
328
|
+
test "creating a post returns 201" {
|
|
329
|
+
when {
|
|
330
|
+
let response = http.post "http://localhost:3000/api/posts" { title: "Hello" }
|
|
331
|
+
}
|
|
332
|
+
then {
|
|
333
|
+
expect response.status 201
|
|
334
|
+
}
|
|
335
|
+
}
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
```sh
|
|
339
|
+
spec run spec --allow-net=localhost:3000
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
#### Request options: headers, bodies, and credentials
|
|
343
|
+
|
|
344
|
+
After the URL, a request takes optional **word-tagged options**, in any order:
|
|
345
|
+
|
|
346
|
+
- `headers { Name: "value", … }` — request headers (an `Authorization`, an
|
|
347
|
+
`Accept`, a cookie). Numbers and booleans stringify; header names are
|
|
348
|
+
case-insensitive, and an explicit `content-type` here overrides the body's.
|
|
349
|
+
- `form { field: "value", … }` — a body sent as
|
|
350
|
+
`application/x-www-form-urlencoded` (the shape OAuth token endpoints and
|
|
351
|
+
classic form posts expect).
|
|
352
|
+
- `json <value>` — a body sent as `application/json`; the explicit form of a
|
|
353
|
+
bare non-string body.
|
|
354
|
+
- `text "<string>"` — a body sent as `text/plain`; the explicit form of a bare
|
|
355
|
+
string body.
|
|
356
|
+
- `bearer <token>` — sets `Authorization: Bearer <token>`, so a resource-server
|
|
357
|
+
call passes the access token, not a hand-built header.
|
|
358
|
+
- `basic <user> <pass>` — sets `Authorization: Basic base64(user:pass)`, the
|
|
359
|
+
`client_secret_basic` shape OAuth introspection and revocation expect. It is
|
|
360
|
+
the one option that takes two values.
|
|
361
|
+
|
|
362
|
+
They combine, so an authenticated form post is one call:
|
|
363
|
+
|
|
364
|
+
```
|
|
365
|
+
use http
|
|
366
|
+
|
|
367
|
+
test "the token endpoint rejects a bad code" {
|
|
368
|
+
when {
|
|
369
|
+
let response = http.post "http://localhost:3000/oauth/token" form {
|
|
370
|
+
grant_type: "authorization_code"
|
|
371
|
+
code: "bogus"
|
|
372
|
+
} headers { authorization: "Basic dXNlcjpwYXNz" }
|
|
373
|
+
}
|
|
374
|
+
then {
|
|
375
|
+
expect response.status 400
|
|
376
|
+
expect response.json.error "invalid_grant"
|
|
377
|
+
}
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
test "a bogus bearer token is rejected" {
|
|
381
|
+
when {
|
|
382
|
+
let who = http.get "http://localhost:3000/userinfo" bearer "bogus"
|
|
383
|
+
}
|
|
384
|
+
then {
|
|
385
|
+
expect who.status 401
|
|
386
|
+
}
|
|
387
|
+
}
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
A request carries **at most one body** (the bare body, or one of
|
|
391
|
+
`json`/`form`/`text`), **at most one** `headers` block, and **at most one** auth
|
|
392
|
+
option (`bearer` or `basic`); a second body, a body on a `GET`, both `bearer` and
|
|
393
|
+
`basic`, or an unknown tag is an error. An explicit `headers.authorization`
|
|
394
|
+
overrides `bearer`/`basic`. The two original forms — `http.get url` and
|
|
395
|
+
`http.<verb> url <body>` — are unchanged, so existing specs keep working exactly
|
|
396
|
+
as before.
|
|
397
|
+
|
|
398
|
+
### `browser` — drive a real browser · `--allow-net`
|
|
399
|
+
|
|
400
|
+
Drive a browser through its **accessibility tree**, not DOM internals: address
|
|
401
|
+
elements by role (a bare word) and accessible name (a string). Actions include
|
|
402
|
+
`open`, `navigate`, `cookie`, `ua`, `click`, `fill … with …`, `check`, `press`;
|
|
403
|
+
observables include `heading`, `link`, `button`, `text`, `checkbox`, `url`,
|
|
404
|
+
`title`. Reaching the page is the privileged act, so each tool needs
|
|
405
|
+
`--allow-net` for the target host. It's backed by a globally installed `agent-browser` CLI, loaded lazily —
|
|
406
|
+
a suite that never touches `browser.*` needs neither the grant nor the binary.
|
|
407
|
+
|
|
408
|
+
```
|
|
409
|
+
use browser
|
|
410
|
+
|
|
411
|
+
test "the sign-in form authenticates" {
|
|
412
|
+
when {
|
|
413
|
+
browser.open "http://localhost:3000/login"
|
|
414
|
+
browser.fill textbox "Email" with "user@example.com"
|
|
415
|
+
browser.fill textbox "Password" with "correct horse"
|
|
416
|
+
browser.click button "Sign in"
|
|
417
|
+
}
|
|
418
|
+
then {
|
|
419
|
+
expect browser.heading "Welcome back"
|
|
420
|
+
expect browser.title "My App"
|
|
421
|
+
expect browser.url "http://localhost:3000/home"
|
|
422
|
+
}
|
|
423
|
+
}
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
```sh
|
|
427
|
+
spec run spec --allow-net=localhost:3000
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
`browser.heading` also takes a level, for when the rung of the document outline
|
|
431
|
+
is part of the behavior: `level 3` matches an `<h3>` and equally a
|
|
432
|
+
`role="heading"` with `aria-level="3"`, because both reach the accessibility
|
|
433
|
+
tree the same way. A heading of that name at another level fails with the levels
|
|
434
|
+
it did find.
|
|
435
|
+
|
|
436
|
+
```
|
|
437
|
+
then {
|
|
438
|
+
expect browser.heading "Billing" level 2
|
|
439
|
+
}
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
Like `browser.url`, `browser.title` reads as a value too — `let name =
|
|
443
|
+
browser.title` with no argument binds the current title instead of asserting one.
|
|
444
|
+
|
|
445
|
+
`browser.cookie` seeds the session's cookie jar, so a test that isn't about
|
|
446
|
+
signing in can arrive already signed in. The `for` clause names the URL the
|
|
447
|
+
cookie belongs to, which is what lets it be set _before_ the first navigation;
|
|
448
|
+
drop the clause to set it on the page already open. Pair it with
|
|
449
|
+
[`env.get`](#env--read-a-granted-variable----allow-env) — the token belongs in
|
|
450
|
+
the environment, not in the document.
|
|
451
|
+
|
|
452
|
+
```
|
|
453
|
+
given {
|
|
454
|
+
let token = env.get "SESSION_COOKIE"
|
|
455
|
+
let jar = { session: token }
|
|
456
|
+
browser.cookie "session" jar.session for "http://localhost:3000/app"
|
|
457
|
+
}
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
`browser.ua` sets the `User-Agent` the session sends, so the app can recognize
|
|
461
|
+
its own spec run — skip a rate limiter, tag analytics, take a test-only branch.
|
|
462
|
+
Set it before `open`; it applies to requests made after it. It changes the
|
|
463
|
+
request header only: `navigator.userAgent` inside the page still reports the
|
|
464
|
+
real browser.
|
|
465
|
+
|
|
466
|
+
```
|
|
467
|
+
given {
|
|
468
|
+
browser.ua "spec-runner/1.0"
|
|
469
|
+
}
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
`browser.url` also reads as a value: `let current = browser.url` binds the
|
|
473
|
+
session's current URL, so a spec can pull the authorization `code` out of the
|
|
474
|
+
page the browser landed on. (A bare binding reaches a tool through a dotted
|
|
475
|
+
reference, so box it first — see [`let` and references](#let-and-references).)
|
|
476
|
+
|
|
477
|
+
```
|
|
478
|
+
when {
|
|
479
|
+
browser.click button "Authorize"
|
|
480
|
+
let landing = browser.url # capture the redirect URL
|
|
481
|
+
let where = { url: landing }
|
|
482
|
+
let code = url.query where.url "code" # read ?code=… out of it
|
|
483
|
+
}
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
### `db` — query a database · `--allow-env=DATABASE_URL`
|
|
487
|
+
|
|
488
|
+
`db.query` runs raw SQL on Bun's SQL client and returns `{ rows, affected_rows,
|
|
489
|
+
count }`. It reads the connection string from the `DATABASE_URL` environment
|
|
490
|
+
variable, so granting that one variable is the whole authorization — a spec can
|
|
491
|
+
never redirect the connection elsewhere. The connection opens lazily on the
|
|
492
|
+
first query and closes at the end of the run.
|
|
493
|
+
|
|
494
|
+
```
|
|
495
|
+
use db
|
|
496
|
+
|
|
497
|
+
test "an INSERT reports exactly one affected row" {
|
|
498
|
+
when {
|
|
499
|
+
let result = db.query "INSERT INTO ledger (entry) VALUES ('opening balance')"
|
|
500
|
+
}
|
|
501
|
+
then {
|
|
502
|
+
expect result.affected_rows 1
|
|
503
|
+
expect result.count 0
|
|
504
|
+
}
|
|
505
|
+
}
|
|
506
|
+
```
|
|
507
|
+
|
|
508
|
+
```sh
|
|
509
|
+
DATABASE_URL=postgres://localhost/test spec run spec --allow-env=DATABASE_URL
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
### `env` — read a granted variable · `--allow-env`
|
|
513
|
+
|
|
514
|
+
`env.get NAME` reads one environment variable, and only one the caller granted
|
|
515
|
+
by name. It is how a spec names a secret without containing one: the document
|
|
516
|
+
says _which_ variable holds the session token, the environment says what it is,
|
|
517
|
+
and the same spec runs against local, staging, and CI. An unset variable is an
|
|
518
|
+
error unless you give `env.get` a fallback — its optional second argument — which
|
|
519
|
+
covers an absent value, never an absent grant.
|
|
520
|
+
|
|
521
|
+
```
|
|
522
|
+
use env
|
|
523
|
+
use browser
|
|
524
|
+
|
|
525
|
+
test "the dashboard renders for a signed-in session" {
|
|
526
|
+
given {
|
|
527
|
+
let token = env.get "SESSION_COOKIE"
|
|
528
|
+
let jar = { session: token }
|
|
529
|
+
browser.cookie "session" jar.session for "http://localhost:3000/app"
|
|
530
|
+
}
|
|
531
|
+
when {
|
|
532
|
+
browser.open "http://localhost:3000/app"
|
|
533
|
+
}
|
|
534
|
+
then {
|
|
535
|
+
# Without the cookie, this would have redirected to /login.
|
|
536
|
+
expect browser.url "http://localhost:3000/app"
|
|
537
|
+
}
|
|
538
|
+
}
|
|
539
|
+
```
|
|
540
|
+
|
|
541
|
+
```sh
|
|
542
|
+
SESSION_COOKIE=… spec run spec --allow-env=SESSION_COOKIE --allow-net=localhost:3000
|
|
543
|
+
```
|
|
544
|
+
|
|
545
|
+
### `url` — parse a URL · no grant
|
|
546
|
+
|
|
547
|
+
Pure, permissionless URL parsing — no network, no filesystem — so a spec can pull
|
|
548
|
+
a value out of a URL it already holds instead of doing string surgery the language
|
|
549
|
+
deliberately omits. Its typical job is reading the authorization `code` out of the
|
|
550
|
+
redirect URL an OAuth authorize step lands on.
|
|
551
|
+
|
|
552
|
+
- `url.query <url> <name>` — the value of a query-string parameter.
|
|
553
|
+
- `url.fragment <url> <name>` — the value of a parameter after the `#` (the
|
|
554
|
+
implicit/hybrid OAuth response shape).
|
|
555
|
+
- `url.path <url>` — the URL's pathname; `url.host <url>` — its host and port.
|
|
556
|
+
|
|
557
|
+
A missing parameter, a non-string argument, or an unparseable URL is an error —
|
|
558
|
+
`query`/`fragment` never bind a silent null.
|
|
559
|
+
|
|
560
|
+
```
|
|
561
|
+
use url
|
|
562
|
+
|
|
563
|
+
test "the authorization code is read from the redirect URL" {
|
|
564
|
+
when {
|
|
565
|
+
let code = url.query "http://localhost:3000/callback?code=abc123&state=s" "code"
|
|
566
|
+
}
|
|
567
|
+
then {
|
|
568
|
+
expect code "abc123"
|
|
569
|
+
}
|
|
570
|
+
}
|
|
571
|
+
```
|
|
572
|
+
|
|
573
|
+
### `sample` — generate a suite's input · no grant
|
|
574
|
+
|
|
575
|
+
Names, addresses, identifiers, and numbers drawn per test, instead of literals
|
|
576
|
+
typed into the suite. Pure computation over the test's own stream — no network,
|
|
577
|
+
no filesystem, no grant — and every value reproduces: the same test draws the
|
|
578
|
+
same data on every run, so a failure on generated input is replayable.
|
|
579
|
+
|
|
580
|
+
A call target carries at most one dot, so **each module is one tool** returning a
|
|
581
|
+
record of everything that module generates. Bind it, then read fields by path:
|
|
582
|
+
|
|
583
|
+
```
|
|
584
|
+
use sample
|
|
585
|
+
|
|
586
|
+
test "a visitor signs up from somewhere" {
|
|
587
|
+
given {
|
|
588
|
+
let who = sample.person
|
|
589
|
+
let where = sample.location
|
|
590
|
+
}
|
|
591
|
+
when {
|
|
592
|
+
let created = http.post "http://localhost:3000/signup" json {
|
|
593
|
+
email: who.email
|
|
594
|
+
name: who.full_name
|
|
595
|
+
job: who.job_title
|
|
596
|
+
city: where.city
|
|
597
|
+
country: where.country
|
|
598
|
+
}
|
|
599
|
+
}
|
|
600
|
+
then {
|
|
601
|
+
expect created.status 201
|
|
602
|
+
}
|
|
603
|
+
}
|
|
604
|
+
```
|
|
605
|
+
|
|
606
|
+
| Tool | The record holds |
|
|
607
|
+
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
608
|
+
| `sample.person` | `first_name`, `last_name`, `full_name`, `prefix`, `suffix`, `sex`, `gender`, `zodiac_sign`, `job_title`, `bio`, `email`, `username`, `phone` |
|
|
609
|
+
| `sample.internet` | `email`, `username`, `url`, `domain_name`, `password`, `ip`, `ipv4`, `ipv6`, `mac`, `port`, `protocol`, `http_method`, `http_status_code`, `jwt`, `user_agent`, `emoji` |
|
|
610
|
+
| `sample.location` | `country`, `city`, `country_code`, `state`, `county`, `street`, `street_address`, `zip_code`, `postal_address`, `latitude`, `longitude`, `time_zone` |
|
|
611
|
+
| `sample.company` | `name`, `catch_phrase`, `buzz_phrase`, and their parts |
|
|
612
|
+
| `sample.lorem` | `word`, `words`, `sentence`, `paragraph`, `lines`, `slug`, `text` |
|
|
613
|
+
| `sample.date` | `past`, `future`, `recent`, `soon`, `anytime`, `birthdate`, `month`, `weekday`, `time_zone` — dates as ISO timestamps |
|
|
614
|
+
| `sample.string` | `uuid`, `ulid`, `nanoid`, `alpha`, `alphanumeric`, `numeric`, `hexadecimal`, `binary`, `octal`, `symbol` |
|
|
615
|
+
| `sample.number` | `int`, `float`, `hex`, `binary`, `octal`, `roman_numeral`, `big_int` |
|
|
616
|
+
| `sample.color` | `human`, `hex`, `rgb`, `hsl`, `space`, `css_function` |
|
|
617
|
+
| `sample.datatype` | `boolean` |
|
|
618
|
+
| `sample.git` | `branch`, `commit_sha`, `short_sha`, `commit_message`, `commit_date`, `commit_entry` |
|
|
619
|
+
| `sample.hacker` | `abbreviation`, `adjective`, `noun`, `verb`, `ingverb`, `phrase` |
|
|
620
|
+
| `sample.phone` | `number`, `national`, `international`, `imei` |
|
|
621
|
+
| `sample.system` | `file_name`, `file_ext`, `file_type`, `mime_type`, `directory_path`, `file_path`, `network_interface`, `semver`, `cron` |
|
|
622
|
+
|
|
623
|
+
A record's fields agree with each other: a person's address matches their name,
|
|
624
|
+
and a location's postal address names its own city and country.
|
|
625
|
+
|
|
626
|
+
The generators that take an argument keep a tool of their own, alongside two
|
|
627
|
+
shortcuts for the values a suite reaches for most:
|
|
628
|
+
|
|
629
|
+
- `sample.int <min> <max>` — an integer, both bounds included.
|
|
630
|
+
- `sample.float <min> <max>` — a number between the bounds, two decimals.
|
|
631
|
+
- `sample.words <count>` — that many words of placeholder prose.
|
|
632
|
+
- `sample.pick <list>` — one element of a list another tool returned.
|
|
633
|
+
- `sample.email`, `sample.uuid` — the two single values worth their own name.
|
|
634
|
+
|
|
635
|
+
Every tool is an **action**, not an observation: a draw advances the stream, so
|
|
636
|
+
`sample` may not head an `eventually` — polling until a random value matches is
|
|
637
|
+
never what an author meant.
|
|
638
|
+
|
|
639
|
+
A test's data follows its **identity** — the run's seed, the test's file inside
|
|
640
|
+
the suite, and its title — and nothing else. Two runs of a suite generate the
|
|
641
|
+
same data, whatever the concurrency, whatever order the tests ran in, and
|
|
642
|
+
wherever the suite is checked out. Adding or removing a neighboring test does
|
|
643
|
+
not move it either.
|
|
644
|
+
|
|
645
|
+
```sh
|
|
646
|
+
spec run spec --seed=checkout # a different suite-wide seed
|
|
647
|
+
spec run spec --seed=random # draw one, printed so it can be replayed
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
`--seed=random` prints the seed it drew before the run:
|
|
651
|
+
|
|
652
|
+
```
|
|
653
|
+
seed 1007223771 (replay with --seed=1007223771)
|
|
654
|
+
```
|
|
655
|
+
|
|
656
|
+
That is how a suite is shaken for hidden dependence on particular values while
|
|
657
|
+
keeping any failure it turns up reproducible.
|
|
658
|
+
|
|
659
|
+
### `jwt` — read and verify tokens · `--allow-net` (verify only)
|
|
660
|
+
|
|
661
|
+
Read and verify JSON Web Tokens, the heart of specifying an OIDC server.
|
|
662
|
+
|
|
663
|
+
- `jwt.decode <token>` — split a token into `{ header, payload }` with **no**
|
|
664
|
+
signature check; permissionless, for asserting on claims (`decoded.payload.sub`,
|
|
665
|
+
`decoded.header.alg`, …).
|
|
666
|
+
- `jwt.verify <token> <jwks_url>` — fetch the issuer's JWKS, select the key the
|
|
667
|
+
token names by `kid`, verify its **ES256** signature and expiry, and return the
|
|
668
|
+
verified payload — so a spec proves an id_token is genuinely issuer-signed, not
|
|
669
|
+
just well-formed. It reaches the network to read the JWKS, so it needs
|
|
670
|
+
`--allow-net` for that host; a bad signature, an unknown key, an expired token,
|
|
671
|
+
or a non-ES256 algorithm is an error.
|
|
672
|
+
|
|
673
|
+
```
|
|
674
|
+
use jwt
|
|
675
|
+
|
|
676
|
+
test "the id_token is genuinely signed and names the right subject" {
|
|
677
|
+
given {
|
|
678
|
+
let tokens = fixture issued_tokens
|
|
679
|
+
}
|
|
680
|
+
when {
|
|
681
|
+
let claims = jwt.verify tokens.id_token "http://localhost:3000/.well-known/jwks.json"
|
|
682
|
+
}
|
|
683
|
+
then {
|
|
684
|
+
expect claims.iss "https://id.example.com"
|
|
685
|
+
expect claims.aud "the-client-id"
|
|
686
|
+
}
|
|
687
|
+
}
|
|
688
|
+
```
|
|
689
|
+
|
|
690
|
+
```sh
|
|
691
|
+
spec run spec --allow-net=localhost:3000
|
|
692
|
+
```
|
|
693
|
+
|
|
694
|
+
## Permissions
|
|
695
|
+
|
|
696
|
+
Nothing privileged runs without a grant, and any attempt beyond your grants
|
|
697
|
+
fails with the exact flag it needs — so start with no flags and let the denials
|
|
698
|
+
tell you the suite's true footprint.
|
|
699
|
+
|
|
700
|
+
| Flag | Grants |
|
|
701
|
+
| ----------------------------- | ------------------------------------------------------------- |
|
|
702
|
+
| `--allow-run[=name,…]` | Spawn processes (scoped by executable basename) |
|
|
703
|
+
| `--allow-net[=host[:port],…]` | Reach the network (scoped by host, optionally port) |
|
|
704
|
+
| `--allow-env[=VAR,…]` | Read environment variables (scoped by name) |
|
|
705
|
+
| `--allow-host-fs[=dir,…]` | Touch files outside the workspace (path-prefix scoped) |
|
|
706
|
+
| `--allow-plugins[=ns,…]` | Launch project-declared plugins (see below) |
|
|
707
|
+
| `--allow-config` | Apply the permissions the suite's config declares (see below) |
|
|
708
|
+
|
|
709
|
+
A bare flag grants the whole family (`--allow-net`); a scoped flag grants only
|
|
710
|
+
what it lists (`--allow-run=bun,git`); repeated scoped flags union. Some recipes:
|
|
711
|
+
|
|
712
|
+
```sh
|
|
713
|
+
# Pure filesystem suite — the workspace is always writable, so no flags at all:
|
|
714
|
+
spec run spec
|
|
715
|
+
|
|
716
|
+
# One known tool, one local server, and two named env vars:
|
|
717
|
+
spec run spec --allow-run=bun --allow-net=localhost:3000 --allow-env=CI,NODE_ENV
|
|
718
|
+
|
|
719
|
+
# Read shared fixture data from outside the workspace (path prefix):
|
|
720
|
+
spec run spec --allow-host-fs=/opt/fixtures
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
When several tests fail for the same missing grant, they collapse into one
|
|
724
|
+
grouped block — the `(N tests)` count and `Affected tests:` list make it obvious
|
|
725
|
+
what a single flag would unblock:
|
|
726
|
+
|
|
727
|
+
```
|
|
728
|
+
✗ Permission denied: run (3 tests)
|
|
729
|
+
|
|
730
|
+
The spec attempted to reach:
|
|
731
|
+
> cli.run
|
|
732
|
+
|
|
733
|
+
Re-run with an appropriate permission, for example:
|
|
734
|
+
> spec run --allow-run
|
|
735
|
+
|
|
736
|
+
Affected tests:
|
|
737
|
+
- first needs run (spec/denials.spec:3)
|
|
738
|
+
- second needs run (spec/denials.spec:9)
|
|
739
|
+
- third needs run (spec/denials.spec:15)
|
|
740
|
+
```
|
|
741
|
+
|
|
742
|
+
## The `spec/config.jsonc` file
|
|
743
|
+
|
|
744
|
+
Rather than reciting the same `--allow-*` line every run, a suite can carry its
|
|
745
|
+
own configuration in `spec/config.jsonc` (JSONC: comments and trailing commas
|
|
746
|
+
allowed). It has two keys, both optional: `permissions` and `plugins`.
|
|
747
|
+
|
|
748
|
+
```jsonc
|
|
749
|
+
// spec/config.jsonc
|
|
750
|
+
{
|
|
751
|
+
"permissions": {
|
|
752
|
+
// A bare string is a whole-family grant (like a bare --allow-<family>);
|
|
753
|
+
// a [family, ...scopes] tuple is a scoped grant (like --allow-<family>=…).
|
|
754
|
+
"allow": ["run", "plugins", ["env", "DATABASE_URL"]],
|
|
755
|
+
},
|
|
756
|
+
}
|
|
757
|
+
```
|
|
758
|
+
|
|
759
|
+
The families match the flags: `"run"` ≡ `--allow-run`, `["env","DATABASE_URL"]`
|
|
760
|
+
≡ `--allow-env=DATABASE_URL`, `["net","localhost:3000"]` ≡
|
|
761
|
+
`--allow-net=localhost:3000`, and `"plugins"` (or `["plugins","greet"]`) ≡
|
|
762
|
+
`--allow-plugins`. A malformed or unknown entry is a load error naming it — a
|
|
763
|
+
broken declaration is never silently ignored.
|
|
764
|
+
|
|
765
|
+
Crucially, the declaration is **declare + opt-in**, never ambient authority.
|
|
766
|
+
With no opt-in the file grants **nothing**: the suite still fails closed with
|
|
767
|
+
the normal denials. The operator opts in with one flag:
|
|
768
|
+
|
|
769
|
+
```sh
|
|
770
|
+
spec run spec --allow-config
|
|
771
|
+
```
|
|
772
|
+
|
|
773
|
+
Then the effective grants are the config's declared set **unioned** with any
|
|
774
|
+
explicit `--allow-*` flags you also pass — flags only ever add. Because nothing
|
|
775
|
+
in the file takes effect until someone who read it adds `--allow-config`, a
|
|
776
|
+
cloned or untrusted repo can't self-grant. As a convenience, when a denial
|
|
777
|
+
_would_ be covered by the config, the denial adds one line pointing at
|
|
778
|
+
`--allow-config`.
|
|
779
|
+
|
|
780
|
+
### Loading project plugins
|
|
781
|
+
|
|
782
|
+
The `plugins` key maps a namespace to the command that launches its plugin, so
|
|
783
|
+
specs name `greet.hello` and never a path:
|
|
784
|
+
|
|
785
|
+
```jsonc
|
|
786
|
+
// spec/config.jsonc
|
|
787
|
+
{
|
|
788
|
+
"plugins": {
|
|
789
|
+
// A relative "." path resolves against this file's directory.
|
|
790
|
+
"greet": { "command": ["bun", "./greeter.ts"] },
|
|
791
|
+
},
|
|
792
|
+
}
|
|
793
|
+
```
|
|
794
|
+
|
|
795
|
+
Declaring a plugin is **not** permission to run it — launching one executes code
|
|
796
|
+
the project ships, so it's deny-by-default too. `spec run` starts a declared
|
|
797
|
+
plugin only with `--allow-plugins` (all) or `--allow-plugins=greet` (named); a
|
|
798
|
+
suite that imports an unauthorized plugin is refused before any test runs. The
|
|
799
|
+
built-in namespaces (`fs`, `cli`, `http`, `browser`, `db`, `env`, `url`, `jwt`)
|
|
800
|
+
are
|
|
801
|
+
never affected.
|
|
802
|
+
|
|
803
|
+
## Custom and third-party plugins
|
|
804
|
+
|
|
805
|
+
Every capability is just a plugin — one namespace exposing typed tools behind a
|
|
806
|
+
single interface — so your own is a first-class citizen. You can build one
|
|
807
|
+
in-process (for embedders), as an external executable speaking a small
|
|
808
|
+
NDJSON-over-stdio protocol (any language), or install a third party's. The full
|
|
809
|
+
authoring guide — the `Plugin` interface, all three shapes, project loading, and
|
|
810
|
+
the trust model — is [docs/writing-plugins.md](./docs/writing-plugins.md), with
|
|
811
|
+
a runnable showcase under `examples/plugin-loading/`.
|
|
812
|
+
|
|
813
|
+
## Performance
|
|
814
|
+
|
|
815
|
+
The runner itself is cheap — roughly a millisecond per test on top of whatever
|
|
816
|
+
the app under test costs — so wall-time is dominated by your app, not the
|
|
817
|
+
harness. Two levers keep it that way. Use the **compiled binary** (`bun run
|
|
818
|
+
build` → `./bin/spec`) instead of `bun src/cli.ts` to skip the per-launch
|
|
819
|
+
transpile cost, which matters most for suites that shell out to `spec` many
|
|
820
|
+
times. And for suites whose tests spend their time waiting — a browser, an HTTP
|
|
821
|
+
round-trip, a slow process — run them with **`--concurrency=N`** (alias
|
|
822
|
+
`--jobs=N`) to overlap that waiting:
|
|
823
|
+
|
|
824
|
+
```sh
|
|
825
|
+
spec run spec --concurrency=8
|
|
826
|
+
```
|
|
827
|
+
|
|
828
|
+
Concurrency defaults to `1` (strictly sequential — today's behavior). At `N` the
|
|
829
|
+
runner executes up to `N` tests at once but still reports results in **source
|
|
830
|
+
order**, so the output, counts, and exit code are byte-for-byte identical
|
|
831
|
+
regardless of how the schedule shook out; only the wall-time changes. The runner
|
|
832
|
+
isolates each test's **workspace**, not the app under test — so a suite that
|
|
833
|
+
shares one mutable backend (the same database rows, one stateful server) may
|
|
834
|
+
still need `--concurrency=1`.
|
|
835
|
+
|
|
836
|
+
## Beyond the CLI
|
|
837
|
+
|
|
838
|
+
The CLI is the product surface, and its exit codes (0 pass, 1 test failure, 2
|
|
839
|
+
usage/load error) are what you script against. For embedding the runner in
|
|
840
|
+
another program, the package also exports a programmatic `runSuite` and the
|
|
841
|
+
supporting types from `@sdxc/spec`; every fallible export returns a
|
|
842
|
+
[`@sdxc/result`](/packages/result) `Result`, so parse errors, permission denials,
|
|
843
|
+
and tool failures are values you branch on, never thrown exceptions.
|
|
844
|
+
|
|
845
|
+
### Choosing which capabilities exist
|
|
846
|
+
|
|
847
|
+
`runSuite` registers all eight built-in namespaces. Pass `builtins` to register
|
|
848
|
+
only some:
|
|
849
|
+
|
|
850
|
+
```ts
|
|
851
|
+
import { runSuite } from "@sdxc/spec";
|
|
852
|
+
|
|
853
|
+
let run = await runSuite({ root: "spec", grants, builtins: ["http", "url", "jwt"] });
|
|
854
|
+
```
|
|
855
|
+
|
|
856
|
+
This is **not** a permission decision, and the difference matters. A denied
|
|
857
|
+
capability still exists — the denial names the flag that would allow it. A
|
|
858
|
+
namespace left out of `builtins` does not exist: a spec calling `fs.write` fails
|
|
859
|
+
with an unknown name, because there is no flag that would ever lift it. Use
|
|
860
|
+
grants to say "not now", and `builtins` to say "not here".
|
|
861
|
+
|
|
862
|
+
`createBuiltinPlugins(only?)` builds the same list on its own, for callers that
|
|
863
|
+
assemble a plugin set by hand.
|
|
864
|
+
|
|
865
|
+
### Running without a filesystem or a process
|
|
866
|
+
|
|
867
|
+
`runSuite` assumes a Bun or Node process: it reads the suite off a disk, gives
|
|
868
|
+
each test a temp directory, and can spawn `cli`, `browser` and `db`. Underneath
|
|
869
|
+
it is `runTests`, which assumes nothing — the suite, the plugin set, the grants,
|
|
870
|
+
and the workspace factory all arrive as arguments:
|
|
871
|
+
|
|
872
|
+
```ts
|
|
873
|
+
import { isFailure } from "@sdxc/result";
|
|
874
|
+
import {
|
|
875
|
+
createHttpPlugin,
|
|
876
|
+
createJwtPlugin,
|
|
877
|
+
createNoFilesystemWorkspace,
|
|
878
|
+
createUrlPlugin,
|
|
879
|
+
loadSources,
|
|
880
|
+
parseGrants,
|
|
881
|
+
runTests,
|
|
882
|
+
} from "@sdxc/spec/workers";
|
|
883
|
+
|
|
884
|
+
let loaded = loadSources([{ path: "flow.spec", text: source }]);
|
|
885
|
+
if (isFailure(loaded)) return loaded;
|
|
886
|
+
|
|
887
|
+
let grants = parseGrants(["--allow-net=app.example.com"]);
|
|
888
|
+
if (isFailure(grants)) return grants;
|
|
889
|
+
|
|
890
|
+
let outcome = await runTests({
|
|
891
|
+
suite: loaded.data,
|
|
892
|
+
plugins: [createHttpPlugin(), createUrlPlugin(), createJwtPlugin()],
|
|
893
|
+
grants: grants.data,
|
|
894
|
+
createWorkspace: createNoFilesystemWorkspace,
|
|
895
|
+
});
|
|
896
|
+
```
|
|
897
|
+
|
|
898
|
+
`loadSources` is the half of loading that has no filesystem in it: hand it
|
|
899
|
+
`{ path, text }` pairs from wherever the specs live — a database row, an HTTP
|
|
900
|
+
body, a bundled string — and it parses and registers them exactly as
|
|
901
|
+
`loadSuite` does after its directory walk. `createNoFilesystemWorkspace` refuses
|
|
902
|
+
every path, which a run without `fs` and `cli` never asks it to resolve.
|
|
903
|
+
|
|
904
|
+
The `@sdxc/spec/workers` entry point exists because of what a module may
|
|
905
|
+
**import**, not what a run may do: `db` imports Bun's SQL client, and `cli`,
|
|
906
|
+
`browser` and the stdio plugin transport reach for the `Bun` global, so a module
|
|
907
|
+
importing them cannot load in a V8-isolate runtime however carefully the run is
|
|
908
|
+
permissioned. That entry point exports the language core plus the three
|
|
909
|
+
capabilities that are already pure — `http`, `url`, `jwt` — and a test in the
|
|
910
|
+
package walks its import graph to keep it that way. It still needs Node
|
|
911
|
+
compatibility enabled for `node:path` and `node:fs`, which the permission set
|
|
912
|
+
reaches only through a host-filesystem grant.
|
|
913
|
+
|
|
914
|
+
There is no browser capability there, deliberately. Driving a browser without a
|
|
915
|
+
local binary means calling a remote service over HTTP, and which service that is
|
|
916
|
+
belongs to the host, not to this package: implement the same tool surface as a
|
|
917
|
+
[plugin](./docs/writing-plugins.md) and pass it to `runTests` beside the others.
|
|
918
|
+
|
|
919
|
+
## Related packages
|
|
920
|
+
|
|
921
|
+
- [`@sdxc/result`](/packages/result) — the `Result` type every fallible export
|
|
922
|
+
returns; errors are values, never throws.
|
|
923
|
+
- [`@sdxc/duration`](/packages/duration) — parses and validates the duration
|
|
924
|
+
literals (`10s`, `500ms`) the language accepts.
|