@test-lab-ai/cli 0.2.13 → 0.2.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +3 -3
- package/README.md +9 -9
- package/bin/testlab.mjs +88 -24
- package/lib/api.mjs +61 -20
- package/lib/config.mjs +6 -7
- package/lib/examples.mjs +10 -10
- package/lib/import.mjs +5 -5
- package/lib/login.mjs +5 -5
- package/lib/skills.mjs +7 -7
- package/lib/toposort.mjs +1 -1
- package/lib/update-check.mjs +1 -1
- package/package.json +1 -1
- package/skills/test-lab-script/SKILL.md +15 -9
package/AGENTS.md
CHANGED
|
@@ -4,7 +4,7 @@ This guide is for an AI coding agent (Claude Code, Codex, Cursor, etc.) asked to
|
|
|
4
4
|
move a user's existing tests into test-lab.ai. You do the format translation; the
|
|
5
5
|
`testlab` CLI handles auth and upload.
|
|
6
6
|
|
|
7
|
-
**Quickest reference: run `testlab examples`**
|
|
7
|
+
**Quickest reference: run `testlab examples`** - it prints the exact JSON shape
|
|
8
8
|
for every resource (credentials, labels, data fixtures, plans, pre-steps).
|
|
9
9
|
|
|
10
10
|
## Install (zero-dependency)
|
|
@@ -121,7 +121,7 @@ in prompts. A fixture is `{ key, label?, fields: [...] }`; each field is either
|
|
|
121
121
|
```
|
|
122
122
|
|
|
123
123
|
Generators include `internet.email`, `person.firstName`, `person.fullName`,
|
|
124
|
-
`string.uuid`, `number.int`, `company.name`,
|
|
124
|
+
`string.uuid`, `number.int`, `company.name`, ... - run `testlab examples` for the
|
|
125
125
|
full list. Reference: `{{data.newUser.email}}`. Keys: start with a letter,
|
|
126
126
|
letters/digits/underscores, max 50 chars.
|
|
127
127
|
|
|
@@ -146,4 +146,4 @@ testlab import ./tests
|
|
|
146
146
|
dashboard afterward if the user needs them.
|
|
147
147
|
- Prefer calling the CLI over hand-rolling HTTP. If you must call the API
|
|
148
148
|
directly, it is documented at https://test-lab.ai/docs/api/test-plans and uses
|
|
149
|
-
the same `Authorization: Bearer tl_
|
|
149
|
+
the same `Authorization: Bearer tl_...` key.
|
package/README.md
CHANGED
|
@@ -26,8 +26,8 @@ export TESTLAB_API_KEY=tl_xxxxx
|
|
|
26
26
|
# or: testlab login --key tl_xxxxx
|
|
27
27
|
```
|
|
28
28
|
|
|
29
|
-
Create a key at **Settings
|
|
30
|
-
the import target
|
|
29
|
+
Create a key at **Settings -> API Keys**. A key belongs to one account, so it's
|
|
30
|
+
the import target - to import into an organization account, create the key with
|
|
31
31
|
that org selected.
|
|
32
32
|
|
|
33
33
|
## Commands
|
|
@@ -52,8 +52,8 @@ reference and is written so an AI agent can read it and build a valid import.
|
|
|
52
52
|
## Import format
|
|
53
53
|
|
|
54
54
|
`testlab import` reads a JSON file (or a directory of `*.json`). A file can be a
|
|
55
|
-
single plan, an array of plans, or a **bundle** with any of these sections
|
|
56
|
-
created in order: credentials
|
|
55
|
+
single plan, an array of plans, or a **bundle** with any of these sections -
|
|
56
|
+
created in order: credentials -> labels -> fixtures -> plans:
|
|
57
57
|
|
|
58
58
|
```jsonc
|
|
59
59
|
{
|
|
@@ -98,9 +98,9 @@ in the file doesn't matter. Re-running is safe for credentials/labels/fixtures
|
|
|
98
98
|
|
|
99
99
|
## Reference syntax (inside a plan prompt)
|
|
100
100
|
|
|
101
|
-
- `{{credentials.<key>}}`
|
|
102
|
-
- `{{data.<fixture>.<field>}}`
|
|
103
|
-
- `{{run.shortId}}`
|
|
101
|
+
- `{{credentials.<key>}}` - a stored secret (never shown to the AI model).
|
|
102
|
+
- `{{data.<fixture>.<field>}}` - a value from a data fixture (generated test data).
|
|
103
|
+
- `{{run.shortId}}` - a unique per-run id (for unique emails, names, etc.).
|
|
104
104
|
|
|
105
105
|
## Install the test-lab-plan skill (Claude Code, Codex, Cursor)
|
|
106
106
|
|
|
@@ -148,9 +148,9 @@ Import tests you already have:
|
|
|
148
148
|
## For AI agents
|
|
149
149
|
|
|
150
150
|
Run **`testlab examples`** for the complete JSON reference, or read `AGENTS.md`
|
|
151
|
-
(shipped inside this package). The workflow: read the user's existing tests
|
|
151
|
+
(shipped inside this package). The workflow: read the user's existing tests ->
|
|
152
152
|
convert each into the plan/fixture JSON above (explicit URL in the prompt,
|
|
153
|
-
secrets as `{{credentials.<key>}}`, generated data as `{{data.<fixture>.<field>}}`)
|
|
153
|
+
secrets as `{{credentials.<key>}}`, generated data as `{{data.<fixture>.<field>}}`) ->
|
|
154
154
|
`testlab import`.
|
|
155
155
|
|
|
156
156
|
## Under the hood
|
package/bin/testlab.mjs
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
/**
|
|
3
|
-
* testlab
|
|
3
|
+
* testlab - import existing test plans into test-lab.ai (driveable by a human,
|
|
4
4
|
* a CI job, or an AI agent).
|
|
5
5
|
*
|
|
6
6
|
* Commands:
|
|
7
|
-
* testlab login [--key tl_
|
|
7
|
+
* testlab login [--key tl_...] Authenticate (browser flow, or paste/flag a key)
|
|
8
8
|
* testlab whoami Show the authenticated account
|
|
9
9
|
* testlab plans list List the account's test plans
|
|
10
10
|
* testlab plans create -f plan.json Create one plan from a JSON file
|
|
@@ -12,8 +12,8 @@
|
|
|
12
12
|
* testlab credentials set <key> --value <value> Upsert an account credential ({{credentials.<key>}})
|
|
13
13
|
* testlab import <path> [--dry-run] Import a plan file or a directory of *.json
|
|
14
14
|
*
|
|
15
|
-
* Auth: --key
|
|
16
|
-
* API:
|
|
15
|
+
* Auth: --key -> $TESTLAB_API_KEY -> ~/.test-lab/config.json
|
|
16
|
+
* API: https://www.test-lab.ai (fixed)
|
|
17
17
|
*/
|
|
18
18
|
import { parseArgs } from "node:util"
|
|
19
19
|
import fs from "node:fs"
|
|
@@ -32,10 +32,10 @@ function errExit(msg) {
|
|
|
32
32
|
process.exit(1)
|
|
33
33
|
}
|
|
34
34
|
|
|
35
|
-
const HELP = `testlab
|
|
35
|
+
const HELP = `testlab - import test plans (and their data) into test-lab.ai
|
|
36
36
|
|
|
37
37
|
Usage:
|
|
38
|
-
testlab login [--key tl_
|
|
38
|
+
testlab login [--key tl_...] Authenticate (browser, or paste/flag a key)
|
|
39
39
|
testlab whoami Show the authenticated account
|
|
40
40
|
testlab import <path> [--dry-run] Import a file or directory of *.json
|
|
41
41
|
(credentials, labels, fixtures, plans)
|
|
@@ -49,6 +49,8 @@ Usage:
|
|
|
49
49
|
testlab data create -f fixture.json Create a data fixture ({{data.<fixture>.<field>}})
|
|
50
50
|
testlab scripts upload <file> --plan <id> Upload a local Playwright script for a plan
|
|
51
51
|
(skips paid AI generation; --device optional)
|
|
52
|
+
testlab scripts get <id> [--out <file>] Download a plan's active script to fix + re-upload
|
|
53
|
+
(prints to stdout, or writes --out <file>; --device optional)
|
|
52
54
|
testlab examples Print the full JSON reference for every
|
|
53
55
|
resource (designed for AI agents)
|
|
54
56
|
testlab skills install [--agent ...] Install the test-lab skills (test-lab-plan +
|
|
@@ -57,21 +59,20 @@ Usage:
|
|
|
57
59
|
testlab skills update Refresh installed skills (also auto-runs after a CLI upgrade)
|
|
58
60
|
|
|
59
61
|
Options:
|
|
60
|
-
--key <tl_
|
|
62
|
+
--key <tl_...> API key (else $TESTLAB_API_KEY or ~/.test-lab/config.json)
|
|
61
63
|
--stdin Read the value (key/credential) from stdin
|
|
62
64
|
--force (login) re-authenticate even if a stored key still works
|
|
63
65
|
--dry-run (import) validate + print plan order without writing
|
|
64
66
|
--project <id|name> (import/plans) assign plans to a project, or "none" for account-level
|
|
65
67
|
--version, -v Print the installed CLI version
|
|
66
68
|
|
|
67
|
-
Get a key at https://test-lab.ai/admin/settings/api-keys
|
|
69
|
+
Get a key at https://test-lab.ai/admin/settings/api-keys / run \`testlab examples\` for JSON shapes`
|
|
68
70
|
|
|
69
71
|
function parse() {
|
|
70
72
|
return parseArgs({
|
|
71
73
|
allowPositionals: true,
|
|
72
74
|
options: {
|
|
73
75
|
key: { type: "string" },
|
|
74
|
-
"api-url": { type: "string" },
|
|
75
76
|
value: { type: "string" },
|
|
76
77
|
file: { type: "string", short: "f" },
|
|
77
78
|
name: { type: "string" },
|
|
@@ -84,6 +85,7 @@ function parse() {
|
|
|
84
85
|
project: { type: "string" },
|
|
85
86
|
plan: { type: "string" },
|
|
86
87
|
device: { type: "string" },
|
|
88
|
+
out: { type: "string", short: "o" },
|
|
87
89
|
version: { type: "boolean", short: "v" },
|
|
88
90
|
help: { type: "boolean", short: "h" },
|
|
89
91
|
},
|
|
@@ -151,7 +153,6 @@ async function cmdLogin(flags) {
|
|
|
151
153
|
|
|
152
154
|
const cfg = loadConfig()
|
|
153
155
|
cfg.apiKey = apiKey
|
|
154
|
-
cfg.apiUrl = apiUrl
|
|
155
156
|
const path = saveConfig(cfg)
|
|
156
157
|
log(`✓ Authenticated${who?.email ? ` as ${who.email}` : ""}. Credentials saved to ${path}`)
|
|
157
158
|
}
|
|
@@ -159,7 +160,7 @@ async function cmdLogin(flags) {
|
|
|
159
160
|
async function cmdWhoami(flags) {
|
|
160
161
|
const { apiKey, apiUrl } = requireAuth(flags)
|
|
161
162
|
// The two requests are independent (apiFetch never throws), so fire
|
|
162
|
-
// them together
|
|
163
|
+
// them together - whoami is interactive and the serial form doubled
|
|
163
164
|
// its wall-clock. Identity endpoint shipped after the CLI's first
|
|
164
165
|
// release: older servers 404 on /me and we just skip those lines.
|
|
165
166
|
const [r, me] = await Promise.all([
|
|
@@ -284,7 +285,7 @@ async function resolveProjectId(apiUrl, apiKey, flags) {
|
|
|
284
285
|
}
|
|
285
286
|
if (!process.stdin.isTTY) {
|
|
286
287
|
errExit(
|
|
287
|
-
`You have ${projects.length} projects
|
|
288
|
+
`You have ${projects.length} projects - pick one with --project <id|name> (or --project none):\n` +
|
|
288
289
|
projects.map((p) => ` ${p.id} ${p.name}`).join("\n")
|
|
289
290
|
)
|
|
290
291
|
}
|
|
@@ -341,7 +342,7 @@ async function cmdDataList(flags) {
|
|
|
341
342
|
}
|
|
342
343
|
for (const fx of fixtures) {
|
|
343
344
|
const fields = (fx.fields || []).map((f) => f.key).join(", ")
|
|
344
|
-
log(` ${fx.key}${fx.label ? ` (${fx.label})` : ""}${fields ? `
|
|
345
|
+
log(` ${fx.key}${fx.label ? ` (${fx.label})` : ""}${fields ? ` - ${fields}` : ""}`)
|
|
345
346
|
}
|
|
346
347
|
log(`\n${fixtures.length} fixture(s)`)
|
|
347
348
|
}
|
|
@@ -370,10 +371,19 @@ async function cmdDataCreate(flags) {
|
|
|
370
371
|
// paid AI generation. The server validates it (allow-list + intent review),
|
|
371
372
|
// wraps it in the trusted harness, and stores it as the plan's saved script.
|
|
372
373
|
// On rejection the per-line issues are printed so an agent can self-correct.
|
|
374
|
+
// Strict positive-integer plan id from a CLI arg/flag. parseInt is too lenient
|
|
375
|
+
// ("12abc" -> 12, "12.9" -> 12), which would silently fetch/replace the WRONG
|
|
376
|
+
// plan; require all-digits and > 0 so a typo errors instead.
|
|
377
|
+
function parsePlanIdArg(raw) {
|
|
378
|
+
if (typeof raw !== "string" || !/^\d+$/.test(raw.trim())) return NaN
|
|
379
|
+
const n = parseInt(raw.trim(), 10)
|
|
380
|
+
return n > 0 ? n : NaN
|
|
381
|
+
}
|
|
382
|
+
|
|
373
383
|
async function cmdScriptsUpload(flags, args) {
|
|
374
384
|
const file = args[2]
|
|
375
385
|
if (!file) errExit("usage: testlab scripts upload <file.spec.ts> --plan <planId> [--device <device>]")
|
|
376
|
-
const planId =
|
|
386
|
+
const planId = parsePlanIdArg(flags.plan)
|
|
377
387
|
if (!Number.isInteger(planId)) errExit("--plan <planId> is required (the numeric test-plan id; see `testlab plans list`)")
|
|
378
388
|
const { apiKey, apiUrl } = requireAuth(flags)
|
|
379
389
|
|
|
@@ -388,7 +398,9 @@ async function cmdScriptsUpload(flags, args) {
|
|
|
388
398
|
// plan's first configured device. Hardcoding "Desktop Chrome" here would 400
|
|
389
399
|
// on any plan whose device isn't Desktop Chrome (the server validates it).
|
|
390
400
|
const device = flags.device
|
|
391
|
-
|
|
401
|
+
// Idempotent on the server (saved_scripts upsert ON CONFLICT), so a transient
|
|
402
|
+
// 5xx is safe to retry - a successful retry overwrites, it never duplicates.
|
|
403
|
+
const r = await apiFetch(apiUrl, apiKey, "POST", "/api/v1/scripts/upload", { script, planId, device }, { retries: 3 })
|
|
392
404
|
|
|
393
405
|
// 422 = the script was understood but rejected (allow-list or intent review).
|
|
394
406
|
// Print the structured issues so a human or agent can fix and retry.
|
|
@@ -404,16 +416,67 @@ async function cmdScriptsUpload(flags, args) {
|
|
|
404
416
|
log(`✗ Script rejected by security review: ${r.json.reason}`)
|
|
405
417
|
process.exit(1)
|
|
406
418
|
}
|
|
407
|
-
if (!r.ok)
|
|
419
|
+
if (!r.ok) {
|
|
420
|
+
// A 5xx (or network error) survived the retries - almost always a transient
|
|
421
|
+
// server blip, not a rejected script. Tell the user it's safe to re-run:
|
|
422
|
+
// the upload is idempotent, so a retry can't create a duplicate. (A genuinely
|
|
423
|
+
// bad script comes back as 422 and is handled above, never here.)
|
|
424
|
+
if (r.status === 0 || r.status >= 500) {
|
|
425
|
+
errExit(
|
|
426
|
+
`${r.status || "network"}: ${r.json?.error || "upload failed"}\n` +
|
|
427
|
+
` This is usually a transient server error, not a rejected script. ` +
|
|
428
|
+
`Re-run the same command - the upload is idempotent, so it won't create a duplicate.`,
|
|
429
|
+
)
|
|
430
|
+
}
|
|
431
|
+
errExit(`${r.status}: ${r.json?.error || "upload failed"}`)
|
|
432
|
+
}
|
|
408
433
|
|
|
409
434
|
const steps = r.json?.stepCount
|
|
410
435
|
// Show the device the SERVER resolved (it defaults an omitted device to the
|
|
411
436
|
// plan's first), not the CLI-local flag which is undefined when --device is omitted.
|
|
412
437
|
const resolvedDevice = r.json?.device || device || "the plan's default device"
|
|
413
|
-
log(`✓ Uploaded ${file}
|
|
438
|
+
log(`✓ Uploaded ${file} -> plan #${planId} (${steps != null ? `${steps} step${steps === 1 ? "" : "s"}, ` : ""}${resolvedDevice})`)
|
|
414
439
|
log(` This plan now runs your script instead of AI generation.`)
|
|
415
440
|
}
|
|
416
441
|
|
|
442
|
+
async function cmdScriptsGet(flags, args) {
|
|
443
|
+
const planId = parsePlanIdArg(args[2] ?? flags.plan)
|
|
444
|
+
if (!Number.isInteger(planId)) {
|
|
445
|
+
errExit("usage: testlab scripts get <planId> [--device <device>] [--out <file>]")
|
|
446
|
+
}
|
|
447
|
+
const { apiKey, apiUrl } = requireAuth(flags)
|
|
448
|
+
|
|
449
|
+
const qs = new URLSearchParams({ planId: String(planId) })
|
|
450
|
+
if (flags.device) qs.set("device", String(flags.device))
|
|
451
|
+
const r = await apiFetch(apiUrl, apiKey, "GET", `/api/v1/scripts?${qs.toString()}`)
|
|
452
|
+
|
|
453
|
+
// 404 carries a human message (no script / wrong device); surface it verbatim.
|
|
454
|
+
if (!r.ok) errExit(`${r.status}: ${r.json?.error || "failed to fetch script"}`)
|
|
455
|
+
const script = r.json?.script
|
|
456
|
+
if (typeof script !== "string") errExit("server returned no script")
|
|
457
|
+
|
|
458
|
+
const device = r.json?.device || flags.device || "the plan's default device"
|
|
459
|
+
const steps = r.json?.stepCount
|
|
460
|
+
const source = r.json?.source
|
|
461
|
+
const meta = `${steps != null ? `${steps} step${steps === 1 ? "" : "s"}, ` : ""}${device}${source ? `, ${source}` : ""}`
|
|
462
|
+
const withNl = script.endsWith("\n") ? script : script + "\n"
|
|
463
|
+
|
|
464
|
+
if (flags.out) {
|
|
465
|
+
try {
|
|
466
|
+
fs.writeFileSync(flags.out, withNl, "utf8")
|
|
467
|
+
} catch (e) {
|
|
468
|
+
errExit(`could not write ${flags.out}: ${e.message}`)
|
|
469
|
+
}
|
|
470
|
+
log(`✓ Saved plan #${planId} script -> ${flags.out} (${meta})`)
|
|
471
|
+
log(` Edit it, then re-upload: testlab scripts upload ${flags.out} --plan ${planId}${flags.device ? ` --device "${flags.device}"` : ""}`)
|
|
472
|
+
} else {
|
|
473
|
+
// Status line to stderr so a plain `> file` redirect captures only the
|
|
474
|
+
// script (pipeable), while the human still sees what they fetched.
|
|
475
|
+
process.stderr.write(`# plan #${planId} script (${meta})\n`)
|
|
476
|
+
process.stdout.write(withNl)
|
|
477
|
+
}
|
|
478
|
+
}
|
|
479
|
+
|
|
417
480
|
function cmdExamples() {
|
|
418
481
|
log(EXAMPLES_TEXT)
|
|
419
482
|
}
|
|
@@ -445,7 +508,7 @@ async function cmdSkillsInstall(flags, args) {
|
|
|
445
508
|
try {
|
|
446
509
|
const res = await installSkill(name, agent, { global: flags.global })
|
|
447
510
|
installed++
|
|
448
|
-
log(`✓ ${res.name}
|
|
511
|
+
log(`✓ ${res.name} -> ${agent}: ${res.dest}`)
|
|
449
512
|
} catch (e) {
|
|
450
513
|
// When installing for several agents, one failing (e.g. cursor + --global) must not abort the rest.
|
|
451
514
|
if (targets.length > 1) log(` skipped ${agent}: ${e.message}`)
|
|
@@ -463,7 +526,7 @@ function cmdSkillsList() {
|
|
|
463
526
|
}
|
|
464
527
|
|
|
465
528
|
// Install EVERY current skill at every location where any test-lab skill is
|
|
466
|
-
// already present
|
|
529
|
+
// already present - refreshing existing copies AND adding skills shipped since
|
|
467
530
|
// the user last installed, so an update delivers the latest skill SET (not just
|
|
468
531
|
// fresher copies of the old set). `installedSkillTargets()` is the union of
|
|
469
532
|
// (agent, scope) homes; a brand-new skill has no install locations of its own
|
|
@@ -498,7 +561,7 @@ async function cmdSkillsUpdate() {
|
|
|
498
561
|
for (const r of results) {
|
|
499
562
|
const where = `${r.agent}${r.global ? " (global)" : ""}`
|
|
500
563
|
if (!r.ok) log(` ✗ ${r.name} (${where}): ${r.error}`)
|
|
501
|
-
else log(`${r.isNew ? "+ installed" : "✓ refreshed"} ${r.name}
|
|
564
|
+
else log(`${r.isNew ? "+ installed" : "✓ refreshed"} ${r.name} -> ${where}: ${r.dest}`)
|
|
502
565
|
}
|
|
503
566
|
const ok = results.filter((r) => r.ok)
|
|
504
567
|
const added = ok.filter((r) => r.isNew).length
|
|
@@ -509,7 +572,7 @@ async function cmdSkillsUpdate() {
|
|
|
509
572
|
}
|
|
510
573
|
|
|
511
574
|
// After a CLI upgrade, refresh installed skills in place AND add any newly
|
|
512
|
-
// shipped ones
|
|
575
|
+
// shipped ones - runs once per version bump (best effort). Skipped in CI / when
|
|
513
576
|
// NO_UPDATE_NOTIFIER is set.
|
|
514
577
|
async function maybeRefreshSkillsOnUpgrade() {
|
|
515
578
|
if (process.env.CI || process.env.NO_UPDATE_NOTIFIER) return
|
|
@@ -520,7 +583,7 @@ async function maybeRefreshSkillsOnUpgrade() {
|
|
|
520
583
|
try {
|
|
521
584
|
ok = (await refreshInstalledSkills()).filter((r) => r.ok)
|
|
522
585
|
} catch {
|
|
523
|
-
return // best effort
|
|
586
|
+
return // best effort - never break the actual command
|
|
524
587
|
}
|
|
525
588
|
if (ok.length === 0) return
|
|
526
589
|
const added = ok.filter((r) => r.isNew).length
|
|
@@ -528,7 +591,7 @@ async function maybeRefreshSkillsOnUpgrade() {
|
|
|
528
591
|
const bits = []
|
|
529
592
|
if (refreshed) bits.push(`refreshed ${refreshed}`)
|
|
530
593
|
if (added) bits.push(`installed ${added} new`)
|
|
531
|
-
process.stderr.write(`↻ CLI upgraded ${prev}
|
|
594
|
+
process.stderr.write(`↻ CLI upgraded ${prev} -> ${ver}; ${bits.join(", ")} skill location(s).\n`)
|
|
532
595
|
}
|
|
533
596
|
|
|
534
597
|
async function main() {
|
|
@@ -580,7 +643,8 @@ async function main() {
|
|
|
580
643
|
return errExit("usage: testlab data <list|create>")
|
|
581
644
|
case "scripts":
|
|
582
645
|
if (args[1] === "upload") return cmdScriptsUpload(flags, args)
|
|
583
|
-
|
|
646
|
+
if (args[1] === "get") return cmdScriptsGet(flags, args)
|
|
647
|
+
return errExit("usage: testlab scripts <upload <file> --plan <id> | get <id> [--out <file>]>")
|
|
584
648
|
case "examples":
|
|
585
649
|
return cmdExamples()
|
|
586
650
|
case "skills":
|
package/lib/api.mjs
CHANGED
|
@@ -1,31 +1,72 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Tiny fetch wrapper for the test-lab public API. Uses global fetch (Node 18+).
|
|
3
|
-
* Returns { ok, status, json }
|
|
3
|
+
* Returns { ok, status, json } - json is the parsed body (or { raw } on non-JSON).
|
|
4
|
+
*
|
|
5
|
+
* Transient resilience: a 5xx or a network error is retried with linear backoff.
|
|
6
|
+
* Retries are SAFE only for idempotent calls, so they default ON for GET and OFF
|
|
7
|
+
* for writes - a write opts in explicitly via `opts.retries` (used by the
|
|
8
|
+
* idempotent script upload, which the server upserts ON CONFLICT). This is what
|
|
9
|
+
* stops a momentary empty-body 500 (a Worker hiccup or an edge rate-limit that
|
|
10
|
+
* short-circuits before the app's JSON error handler) from surfacing as a hard
|
|
11
|
+
* "500: upload failed" when a one-second retry would have gone through.
|
|
4
12
|
*/
|
|
5
|
-
export async function apiFetch(apiUrl, apiKey, method, pathname, body) {
|
|
13
|
+
export async function apiFetch(apiUrl, apiKey, method, pathname, body, opts = {}) {
|
|
14
|
+
const retries = Number.isInteger(opts.retries) ? opts.retries : method === "GET" ? 2 : 0
|
|
15
|
+
const backoffMs = Number.isInteger(opts.backoffMs) ? opts.backoffMs : 800
|
|
16
|
+
|
|
6
17
|
const headers = {}
|
|
7
18
|
if (apiKey) headers["Authorization"] = `Bearer ${apiKey}`
|
|
8
19
|
if (body !== undefined) headers["Content-Type"] = "application/json"
|
|
9
20
|
|
|
10
|
-
let
|
|
11
|
-
|
|
12
|
-
res = await fetch(`${apiUrl}${pathname}`, {
|
|
13
|
-
method,
|
|
14
|
-
headers,
|
|
15
|
-
body: body !== undefined ? JSON.stringify(body) : undefined,
|
|
16
|
-
})
|
|
17
|
-
} catch (e) {
|
|
18
|
-
return { ok: false, status: 0, json: { error: `network error: ${e.message}` } }
|
|
19
|
-
}
|
|
20
|
-
|
|
21
|
-
const text = await res.text()
|
|
22
|
-
let json = null
|
|
23
|
-
if (text) {
|
|
21
|
+
for (let attempt = 0; ; attempt++) {
|
|
22
|
+
let res
|
|
24
23
|
try {
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
24
|
+
res = await fetch(`${apiUrl}${pathname}`, {
|
|
25
|
+
method,
|
|
26
|
+
headers,
|
|
27
|
+
body: body !== undefined ? JSON.stringify(body) : undefined,
|
|
28
|
+
})
|
|
29
|
+
} catch (e) {
|
|
30
|
+
if (attempt < retries) {
|
|
31
|
+
await sleep(backoffMs * (attempt + 1))
|
|
32
|
+
continue
|
|
33
|
+
}
|
|
34
|
+
return { ok: false, status: 0, json: { error: `network error: ${e.message}` } }
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
// 5xx = transient (Worker error / edge rate-limit). Retry while budget remains.
|
|
38
|
+
if (res.status >= 500 && attempt < retries) {
|
|
39
|
+
await sleep(backoffMs * (attempt + 1))
|
|
40
|
+
continue
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
const text = await res.text()
|
|
44
|
+
let json = null
|
|
45
|
+
if (text) {
|
|
46
|
+
try {
|
|
47
|
+
json = JSON.parse(text)
|
|
48
|
+
} catch {
|
|
49
|
+
json = { raw: text }
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
// A failed response with no usable error body (the empty-body 5xx case) would
|
|
54
|
+
// otherwise reach the caller as `json: null` and print a blank reason.
|
|
55
|
+
// Synthesize an actionable message keyed off the status instead.
|
|
56
|
+
if (!res.ok && (!json || (!json.error && !json.issues && !json.reason))) {
|
|
57
|
+
json = {
|
|
58
|
+
...(json || {}),
|
|
59
|
+
error:
|
|
60
|
+
res.status >= 500
|
|
61
|
+
? `server error (HTTP ${res.status}) with no response body - usually transient`
|
|
62
|
+
: `request failed (HTTP ${res.status})`,
|
|
63
|
+
}
|
|
28
64
|
}
|
|
65
|
+
|
|
66
|
+
return { ok: res.ok, status: res.status, json }
|
|
29
67
|
}
|
|
30
|
-
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
function sleep(ms) {
|
|
71
|
+
return new Promise((resolve) => setTimeout(resolve, ms))
|
|
31
72
|
}
|
package/lib/config.mjs
CHANGED
|
@@ -1,9 +1,8 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Config + auth resolution for the testlab CLI.
|
|
3
3
|
*
|
|
4
|
-
*
|
|
5
|
-
* API
|
|
6
|
-
* API base: --api-url → $TESTLAB_API_URL → config → https://www.test-lab.ai
|
|
4
|
+
* API key: --key flag -> $TESTLAB_API_KEY -> ~/.test-lab/config.json
|
|
5
|
+
* API base: always https://www.test-lab.ai (not configurable)
|
|
7
6
|
*/
|
|
8
7
|
import fs from "node:fs"
|
|
9
8
|
import os from "node:os"
|
|
@@ -36,8 +35,8 @@ export function saveConfig(cfg) {
|
|
|
36
35
|
export function resolveAuth(flags) {
|
|
37
36
|
const cfg = loadConfig()
|
|
38
37
|
const apiKey = flags.key || process.env.TESTLAB_API_KEY || cfg.apiKey || null
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
return { apiKey, apiUrl }
|
|
38
|
+
// API base is fixed to production. It used to be overridable via --api-url /
|
|
39
|
+
// TESTLAB_API_URL / a stored config value; that was removed so an API key is
|
|
40
|
+
// never sent to an arbitrary host.
|
|
41
|
+
return { apiKey, apiUrl: DEFAULT_API_URL }
|
|
43
42
|
}
|
package/lib/examples.mjs
CHANGED
|
@@ -2,9 +2,9 @@
|
|
|
2
2
|
// Written for an AI agent: every resource's exact JSON shape, the command that
|
|
3
3
|
// creates it, the reference syntax, and the combined import-bundle format.
|
|
4
4
|
|
|
5
|
-
export const EXAMPLES_TEXT = `testlab
|
|
5
|
+
export const EXAMPLES_TEXT = `testlab - resource reference (for humans and AI agents)
|
|
6
6
|
|
|
7
|
-
All resources are scoped to the API key's account. Auth: a tl_
|
|
7
|
+
All resources are scoped to the API key's account. Auth: a tl_... key via
|
|
8
8
|
\`testlab login\`, the TESTLAB_API_KEY env var, or --key.
|
|
9
9
|
|
|
10
10
|
═══════════════════════════════════════════════════════════════════════════
|
|
@@ -12,10 +12,10 @@ REFERENCE SYNTAX (use these inside a plan prompt, and in cookie/header values)
|
|
|
12
12
|
═══════════════════════════════════════════════════════════════════════════
|
|
13
13
|
{{credentials.<key>}} a stored secret (never shown to the AI/model)
|
|
14
14
|
{{data.<fixture>.<field>}} a value from a data fixture (generated test data)
|
|
15
|
-
{{run.shortId}} a unique per-run id (for unique emails, names,
|
|
15
|
+
{{run.shortId}} a unique per-run id (for unique emails, names, ...)
|
|
16
16
|
|
|
17
17
|
═══════════════════════════════════════════════════════════════════════════
|
|
18
|
-
1) CREDENTIAL
|
|
18
|
+
1) CREDENTIAL - a secret behind {{credentials.<key>}}
|
|
19
19
|
═══════════════════════════════════════════════════════════════════════════
|
|
20
20
|
Command: testlab credentials set email --value qa@example.com
|
|
21
21
|
JSON (inside an import bundle, under "credentials"):
|
|
@@ -24,14 +24,14 @@ Rules: key starts with a letter; letters/digits/underscores; <=50 chars.
|
|
|
24
24
|
value <= 1000 chars; stored encrypted, never returned.
|
|
25
25
|
|
|
26
26
|
═══════════════════════════════════════════════════════════════════════════
|
|
27
|
-
2) LABEL
|
|
27
|
+
2) LABEL - a tag for grouping plans (auto-created when a plan references it)
|
|
28
28
|
═══════════════════════════════════════════════════════════════════════════
|
|
29
29
|
JSON (under "labels"): "smoke" (just the name)
|
|
30
30
|
Plans can also list labels by name and they're created on the fly:
|
|
31
31
|
"labels": ["smoke", "auth"]
|
|
32
32
|
|
|
33
33
|
═══════════════════════════════════════════════════════════════════════════
|
|
34
|
-
3) DATA FIXTURE
|
|
34
|
+
3) DATA FIXTURE - reusable generated test data behind {{data.<fixture>.<field>}}
|
|
35
35
|
═══════════════════════════════════════════════════════════════════════════
|
|
36
36
|
Command: testlab data create -f fixture.json
|
|
37
37
|
JSON (under "fixtures"):
|
|
@@ -59,7 +59,7 @@ Rules: fixture/field keys start with a letter; letters/digits/underscores;
|
|
|
59
59
|
Reference it in a prompt as {{data.newUser.email}}.
|
|
60
60
|
|
|
61
61
|
═══════════════════════════════════════════════════════════════════════════
|
|
62
|
-
4) TEST PLAN
|
|
62
|
+
4) TEST PLAN - a natural-language test (the URL lives IN the prompt)
|
|
63
63
|
═══════════════════════════════════════════════════════════════════════════
|
|
64
64
|
Command: testlab plans create -f plan.json
|
|
65
65
|
JSON (under "plans"):
|
|
@@ -84,17 +84,17 @@ Rules: name <=200 chars; prompt <=32KB; <=25 labels; <=25 preSteps.
|
|
|
84
84
|
Projects: omit projectId to import account-level. Run "testlab projects list" for ids; "testlab import --project <id|name>" (or --project none) sets it for the whole import, and is auto-picked when you have exactly one.
|
|
85
85
|
|
|
86
86
|
═══════════════════════════════════════════════════════════════════════════
|
|
87
|
-
IMPORT BUNDLE
|
|
87
|
+
IMPORT BUNDLE - create everything at once: testlab import ./bundle.json
|
|
88
88
|
═══════════════════════════════════════════════════════════════════════════
|
|
89
89
|
A file (or a directory of *.json) may contain any of these sections. The CLI
|
|
90
|
-
creates them in order: credentials
|
|
90
|
+
creates them in order: credentials -> labels -> fixtures -> plans, and topo-sorts
|
|
91
91
|
plans so pre-step dependencies (by "ref") are created first.
|
|
92
92
|
{
|
|
93
93
|
"credentials": [ { "key": "password", "value": "hunter2" } ],
|
|
94
94
|
"labels": ["smoke"],
|
|
95
95
|
"fixtures": [ { "key": "newUser", "fields": [ { "key": "email", "mode": "dynamic", "generator": "internet.email" } ] } ],
|
|
96
96
|
"plans": [
|
|
97
|
-
{ "ref": "signup", "name": "Sign up", "prompt": "Go to https://app.example.com/signup, register with {{data.newUser.email}}
|
|
97
|
+
{ "ref": "signup", "name": "Sign up", "prompt": "Go to https://app.example.com/signup, register with {{data.newUser.email}} ..." },
|
|
98
98
|
{ "name": "Onboard", "prompt": "Complete onboarding.", "preSteps": [ { "ref": "signup" } ] }
|
|
99
99
|
]
|
|
100
100
|
}
|
package/lib/import.mjs
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* Import orchestration: load files, then create resources in dependency order
|
|
2
|
+
* Import orchestration: load files, then create resources in dependency order -
|
|
3
3
|
* credentials, labels, data fixtures, then plans (topo-sorted by pre-step deps,
|
|
4
4
|
* wiring intra-batch pre-steps to the ids returned along the way).
|
|
5
5
|
*/
|
|
@@ -67,7 +67,7 @@ export function loadImportFile(target) {
|
|
|
67
67
|
return { credentials, labels, fixtures, plans }
|
|
68
68
|
}
|
|
69
69
|
|
|
70
|
-
/** Resolve a plan's preSteps for the API: intra-batch refs
|
|
70
|
+
/** Resolve a plan's preSteps for the API: intra-batch refs -> concrete ids. */
|
|
71
71
|
export function normalizePreSteps(preSteps, refToId) {
|
|
72
72
|
if (!Array.isArray(preSteps) || preSteps.length === 0) return undefined
|
|
73
73
|
return preSteps.map((ps) => {
|
|
@@ -112,7 +112,7 @@ export async function runImport({
|
|
|
112
112
|
|
|
113
113
|
if (dryRun) {
|
|
114
114
|
log(
|
|
115
|
-
`Dry run
|
|
115
|
+
`Dry run - ${credentials.length} credential(s), ${labels.length} label(s), ${fixtures.length} fixture(s), ${plans.length} plan(s).`
|
|
116
116
|
)
|
|
117
117
|
if (plans.length) {
|
|
118
118
|
log(`Plan creation order:`)
|
|
@@ -137,7 +137,7 @@ export async function runImport({
|
|
|
137
137
|
return r
|
|
138
138
|
}
|
|
139
139
|
|
|
140
|
-
// 1. Credentials (sequential
|
|
140
|
+
// 1. Credentials (sequential - each rewrites the shared encrypted blob).
|
|
141
141
|
for (const c of credentials) {
|
|
142
142
|
if (!c || typeof c.key !== "string") {
|
|
143
143
|
log(` ✗ credential: missing key`)
|
|
@@ -172,7 +172,7 @@ export async function runImport({
|
|
|
172
172
|
})
|
|
173
173
|
}
|
|
174
174
|
|
|
175
|
-
// 4. Plans, in topo order, wiring intra-batch pre-steps by ref
|
|
175
|
+
// 4. Plans, in topo order, wiring intra-batch pre-steps by ref -> created id.
|
|
176
176
|
const refToId = new Map()
|
|
177
177
|
let plansCreated = 0
|
|
178
178
|
let plansFailed = 0
|
package/lib/login.mjs
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* Browser device-login. Reuses test-lab's existing one-time-code
|
|
2
|
+
* Browser device-login. Reuses test-lab's existing one-time-code -> API-key
|
|
3
3
|
* handshake (the same pattern the Chrome extension uses):
|
|
4
4
|
*
|
|
5
5
|
* 1. CLI starts a localhost callback server + opens the browser to
|
|
6
|
-
* <api>/cli/authorize?state
|
|
6
|
+
* <api>/cli/authorize?state=...&port=...
|
|
7
7
|
* 2. The signed-in user approves; the page redirects to
|
|
8
|
-
* http://127.0.0.1:<port>/callback?code
|
|
8
|
+
* http://127.0.0.1:<port>/callback?code=...&state=...
|
|
9
9
|
* 3. CLI verifies `state`, then exchanges the one-time code for a full-scope
|
|
10
|
-
* API key at POST /api/v1/cli/token (direct HTTPS
|
|
10
|
+
* API key at POST /api/v1/cli/token (direct HTTPS - the secret never rides
|
|
11
11
|
* in the browser redirect).
|
|
12
12
|
*
|
|
13
13
|
* If the server doesn't support the flow (older deployment) the caller falls
|
|
@@ -87,7 +87,7 @@ export function browserLogin(apiUrl, { timeoutMs = 600000 } = {}) {
|
|
|
87
87
|
const { port } = server.address()
|
|
88
88
|
const authUrl = `${apiUrl}/cli/authorize?state=${state}&port=${port}`
|
|
89
89
|
console.log(`Opening your browser to authorize the CLI:\n ${authUrl}\n`)
|
|
90
|
-
console.log(`Waiting for you to approve in the browser (up to 10 minutes)
|
|
90
|
+
console.log(`Waiting for you to approve in the browser (up to 10 minutes)...\n`)
|
|
91
91
|
openBrowser(authUrl)
|
|
92
92
|
})
|
|
93
93
|
})
|
package/lib/skills.mjs
CHANGED
|
@@ -10,9 +10,9 @@
|
|
|
10
10
|
* stays as the browsable, open-source view; it is no longer fetched at runtime.
|
|
11
11
|
*
|
|
12
12
|
* Each agent has its own convention (verified against current docs):
|
|
13
|
-
* claude
|
|
14
|
-
* codex
|
|
15
|
-
* cursor
|
|
13
|
+
* claude -> .claude/skills/<name>/SKILL.md (project) | ~/.claude/skills (--global)
|
|
14
|
+
* codex -> .agents/skills/<name>/SKILL.md (project) | ~/.agents/skills (--global)
|
|
15
|
+
* cursor -> .cursor/rules/<name>.mdc (project only - Cursor has no
|
|
16
16
|
* global *file*; user rules live in Settings)
|
|
17
17
|
*
|
|
18
18
|
* claude + codex share the SKILL.md folder format (only the base dir differs);
|
|
@@ -54,8 +54,8 @@ export function detectAgents() {
|
|
|
54
54
|
}
|
|
55
55
|
|
|
56
56
|
/**
|
|
57
|
-
* Where skill `name` is already installed
|
|
58
|
-
* across all agents
|
|
57
|
+
* Where skill `name` is already installed - project (cwd) and global locations
|
|
58
|
+
* across all agents - filtered to the paths that actually exist. Used to
|
|
59
59
|
* refresh installed skills (on `skills update` and after a CLI upgrade).
|
|
60
60
|
*/
|
|
61
61
|
export function installedSkillLocations(name) {
|
|
@@ -74,7 +74,7 @@ export function installedSkillLocations(name) {
|
|
|
74
74
|
* The distinct (agent, scope) locations where AT LEAST ONE test-lab skill is
|
|
75
75
|
* currently installed. `skills update` and the on-upgrade refresh install EVERY
|
|
76
76
|
* current skill at each of these, so a skill added in a later CLI release lands
|
|
77
|
-
* wherever the user already keeps test-lab skills
|
|
77
|
+
* wherever the user already keeps test-lab skills - not just fresher copies of
|
|
78
78
|
* the skills that happened to be installed before.
|
|
79
79
|
*/
|
|
80
80
|
export function installedSkillTargets() {
|
|
@@ -98,7 +98,7 @@ export function agentTarget(agent, { global } = {}) {
|
|
|
98
98
|
return { kind: "skill-dir", base: global ? path.join(home, ".agents", "skills") : path.join(cwd, ".agents", "skills") }
|
|
99
99
|
case "cursor":
|
|
100
100
|
if (global) {
|
|
101
|
-
throw new Error("Cursor has no global rules file
|
|
101
|
+
throw new Error("Cursor has no global rules file - add user rules in Cursor Settings -> Rules, or install per-project (omit --global).")
|
|
102
102
|
}
|
|
103
103
|
return { kind: "cursor-rule", base: path.join(cwd, ".cursor", "rules") }
|
|
104
104
|
default:
|
package/lib/toposort.mjs
CHANGED
|
@@ -55,7 +55,7 @@ export function topoSortPlans(plans) {
|
|
|
55
55
|
}
|
|
56
56
|
}
|
|
57
57
|
|
|
58
|
-
// Kahn's algorithm (iterative
|
|
58
|
+
// Kahn's algorithm (iterative - no recursion depth concerns).
|
|
59
59
|
const queue = []
|
|
60
60
|
for (let i = 0; i < n; i++) if (indegree[i] === 0) queue.push(i)
|
|
61
61
|
const order = []
|
package/lib/update-check.mjs
CHANGED
|
@@ -39,7 +39,7 @@ export function isNewer(a, b) {
|
|
|
39
39
|
/** The notice string for (current, latest), or null if no update / bad input. */
|
|
40
40
|
export function updateNotice(current, latest) {
|
|
41
41
|
if (!current || !latest || !isNewer(latest, current)) return null
|
|
42
|
-
return `\n ⚡ Update available: ${current}
|
|
42
|
+
return `\n ⚡ Update available: ${current} -> ${latest}\n npm i -g ${PKG}@latest\n`
|
|
43
43
|
}
|
|
44
44
|
|
|
45
45
|
function readCache() {
|
package/package.json
CHANGED
|
@@ -32,18 +32,21 @@ This also means the script runs in a **restricted environment**: Playwright and
|
|
|
32
32
|
|
|
33
33
|
Follow these in order.
|
|
34
34
|
|
|
35
|
-
### 1.
|
|
35
|
+
### 1. Design the test as a plan prompt first (with test-lab-plan)
|
|
36
36
|
|
|
37
|
-
|
|
37
|
+
The plan's English **prompt is the spec; your script is one implementation of it.** Design the spec before you write a line of Playwright — that is what keeps the script honest and reviewable, and it is the fallback the platform runs if the script is ever cleared. Skipping this is how you end up with a script glued to a throwaway prompt that describes something else.
|
|
38
38
|
|
|
39
|
-
-
|
|
40
|
-
-
|
|
39
|
+
- **No plan yet?** Use the **test-lab-plan** skill to design it: one user journey, an explicit start URL, and a numbered `Verify that:` block of observable acceptance criteria. Create it (dashboard or `testlab import`), then attach your script.
|
|
40
|
+
- **Plan already exists?** Read its prompt — that *is* your spec. If the prompt is vague, wrong, or describes a different flow, fix it with test-lab-plan **before** writing the script. Never encode behaviour in the script that the prompt doesn't claim.
|
|
41
41
|
|
|
42
|
-
|
|
42
|
+
Keep the prompt a clean user-journey spec: **no setup, auth, or implementation detail in the prose** — logged-in state, injected cookies, environment config, "no login needed", framework internals: none of it belongs there. Setup lives in the plan's pre-step / project environment, not the prompt body (test-lab-plan enforces this). Your script then *implements* that spec: one `test("Step N: …")` per prompt action, and an `expect` for each numbered acceptance criterion.
|
|
43
43
|
|
|
44
|
-
### 2.
|
|
44
|
+
### 2. Confirm the plan is reachable and read the real UI
|
|
45
45
|
|
|
46
|
-
|
|
46
|
+
- `testlab whoami` – confirm the CLI is authenticated. If not, the user runs `testlab login` once or sets `TESTLAB_API_KEY`. If `testlab` is not found, use `npx @test-lab-ai/cli` instead.
|
|
47
|
+
- `testlab plans list` – find the plan id to upload to. The plan must already have a target URL (from its prompt or its project), because the script runs against that origin. Uploading a script replaces AI generation for that plan + device, so from then on the prompt's job is to be the spec you implemented and the fallback — keep the two in sync.
|
|
48
|
+
- **Read the real UI before asserting.** If you are in the target site's repo, read the relevant component / route so your locators and assertions match real DOM text and real success states, not guesses. If you are not in the repo, pull a snapshot with WebFetch or ask the user for the key screens. Anchor every `expect` to something real **and to a criterion in the prompt.**
|
|
49
|
+
- **Fixing a script the plan already runs?** Download the live one with `testlab scripts get <id> [--device] [--out <file>]` (works for uploaded *and* generated scripts) instead of guessing at it, then edit and re-upload. That is the full fetch → fix → `scripts upload` loop, no dashboard needed.
|
|
47
50
|
|
|
48
51
|
### 3. Write the steps against `sharedPage`
|
|
49
52
|
|
|
@@ -140,6 +143,7 @@ The complete lists and the per-rule fixes are in `references/playwright-api.md`.
|
|
|
140
143
|
7. **No Node / browser / network globals** (`process`, `fs`, `fetch`, `window`, `request`, …) and **no `page.evaluate`/`route`**. If you reached for one, rewrite the step to use the page.
|
|
141
144
|
8. **No computed member keys or prototype access** (`x[expr]`, `.constructor`, `.__proto__`).
|
|
142
145
|
9. **The plan id is right** (`testlab plans list`) and the plan has a target URL.
|
|
146
|
+
10. **The script implements the plan's prompt.** Every prompt action maps to a `test("Step N: …")`, and every numbered acceptance criterion maps to an `expect`. The prompt itself reads as a clean user journey — no setup/auth/implementation detail leaked into the prose (that belongs in the plan's pre-step / environment). If the prompt and the script disagree, fix the prompt with test-lab-plan first.
|
|
143
147
|
|
|
144
148
|
If any item fails, fix it before uploading – it will be rejected server-side anyway, and fixing first saves a round-trip.
|
|
145
149
|
|
|
@@ -157,7 +161,9 @@ If any item fails, fix it before uploading – it will be rejected server-side a
|
|
|
157
161
|
|
|
158
162
|
## Relationship to test-lab-plan
|
|
159
163
|
|
|
160
|
-
|
|
161
|
-
|
|
164
|
+
These **compose; they are not either/or.** test-lab-plan designs the spec (the plan's prompt); test-lab-script implements it (the Playwright). Even when you will hand-write the script, start with test-lab-plan so the spec is a clean, reviewable user journey and the script has a concrete contract to satisfy — see Workflow step 1.
|
|
165
|
+
|
|
166
|
+
- **test-lab-plan** – describe a flow in English; the AI generates and maintains the Playwright. Best when you want low effort, or as the spec step before you write your own script.
|
|
167
|
+
- **test-lab-script** (this skill) – you own the Playwright and upload it verbatim to implement that spec. Best when you need exact control, already have a script, or want to save generation credits.
|
|
162
168
|
|
|
163
169
|
An uploaded script can still be refined later with AI from the dashboard (it is tagged as uploaded vs generated). `testlab examples` is the canonical, always-current reference for fixture shapes and resource formats.
|