@test-lab-ai/cli 0.2.13 → 0.2.15

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -4,7 +4,7 @@ This guide is for an AI coding agent (Claude Code, Codex, Cursor, etc.) asked to
4
4
  move a user's existing tests into test-lab.ai. You do the format translation; the
5
5
  `testlab` CLI handles auth and upload.
6
6
 
7
- **Quickest reference: run `testlab examples`** it prints the exact JSON shape
7
+ **Quickest reference: run `testlab examples`** - it prints the exact JSON shape
8
8
  for every resource (credentials, labels, data fixtures, plans, pre-steps).
9
9
 
10
10
  ## Install (zero-dependency)
@@ -121,7 +121,7 @@ in prompts. A fixture is `{ key, label?, fields: [...] }`; each field is either
121
121
  ```
122
122
 
123
123
  Generators include `internet.email`, `person.firstName`, `person.fullName`,
124
- `string.uuid`, `number.int`, `company.name`, run `testlab examples` for the
124
+ `string.uuid`, `number.int`, `company.name`, ... - run `testlab examples` for the
125
125
  full list. Reference: `{{data.newUser.email}}`. Keys: start with a letter,
126
126
  letters/digits/underscores, max 50 chars.
127
127
 
@@ -146,4 +146,4 @@ testlab import ./tests
146
146
  dashboard afterward if the user needs them.
147
147
  - Prefer calling the CLI over hand-rolling HTTP. If you must call the API
148
148
  directly, it is documented at https://test-lab.ai/docs/api/test-plans and uses
149
- the same `Authorization: Bearer tl_…` key.
149
+ the same `Authorization: Bearer tl_...` key.
package/README.md CHANGED
@@ -26,8 +26,8 @@ export TESTLAB_API_KEY=tl_xxxxx
26
26
  # or: testlab login --key tl_xxxxx
27
27
  ```
28
28
 
29
- Create a key at **Settings API Keys**. A key belongs to one account, so it's
30
- the import target to import into an organization account, create the key with
29
+ Create a key at **Settings -> API Keys**. A key belongs to one account, so it's
30
+ the import target - to import into an organization account, create the key with
31
31
  that org selected.
32
32
 
33
33
  ## Commands
@@ -52,8 +52,8 @@ reference and is written so an AI agent can read it and build a valid import.
52
52
  ## Import format
53
53
 
54
54
  `testlab import` reads a JSON file (or a directory of `*.json`). A file can be a
55
- single plan, an array of plans, or a **bundle** with any of these sections
56
- created in order: credentials labels fixtures plans:
55
+ single plan, an array of plans, or a **bundle** with any of these sections -
56
+ created in order: credentials -> labels -> fixtures -> plans:
57
57
 
58
58
  ```jsonc
59
59
  {
@@ -98,9 +98,9 @@ in the file doesn't matter. Re-running is safe for credentials/labels/fixtures
98
98
 
99
99
  ## Reference syntax (inside a plan prompt)
100
100
 
101
- - `{{credentials.<key>}}` a stored secret (never shown to the AI model).
102
- - `{{data.<fixture>.<field>}}` a value from a data fixture (generated test data).
103
- - `{{run.shortId}}` a unique per-run id (for unique emails, names, etc.).
101
+ - `{{credentials.<key>}}` - a stored secret (never shown to the AI model).
102
+ - `{{data.<fixture>.<field>}}` - a value from a data fixture (generated test data).
103
+ - `{{run.shortId}}` - a unique per-run id (for unique emails, names, etc.).
104
104
 
105
105
  ## Install the test-lab-plan skill (Claude Code, Codex, Cursor)
106
106
 
@@ -148,9 +148,9 @@ Import tests you already have:
148
148
  ## For AI agents
149
149
 
150
150
  Run **`testlab examples`** for the complete JSON reference, or read `AGENTS.md`
151
- (shipped inside this package). The workflow: read the user's existing tests
151
+ (shipped inside this package). The workflow: read the user's existing tests ->
152
152
  convert each into the plan/fixture JSON above (explicit URL in the prompt,
153
- secrets as `{{credentials.<key>}}`, generated data as `{{data.<fixture>.<field>}}`)
153
+ secrets as `{{credentials.<key>}}`, generated data as `{{data.<fixture>.<field>}}`) ->
154
154
  `testlab import`.
155
155
 
156
156
  ## Under the hood
package/bin/testlab.mjs CHANGED
@@ -1,10 +1,10 @@
1
1
  #!/usr/bin/env node
2
2
  /**
3
- * testlab import existing test plans into test-lab.ai (driveable by a human,
3
+ * testlab - import existing test plans into test-lab.ai (driveable by a human,
4
4
  * a CI job, or an AI agent).
5
5
  *
6
6
  * Commands:
7
- * testlab login [--key tl_] Authenticate (browser flow, or paste/flag a key)
7
+ * testlab login [--key tl_...] Authenticate (browser flow, or paste/flag a key)
8
8
  * testlab whoami Show the authenticated account
9
9
  * testlab plans list List the account's test plans
10
10
  * testlab plans create -f plan.json Create one plan from a JSON file
@@ -12,8 +12,8 @@
12
12
  * testlab credentials set <key> --value <value> Upsert an account credential ({{credentials.<key>}})
13
13
  * testlab import <path> [--dry-run] Import a plan file or a directory of *.json
14
14
  *
15
- * Auth: --key $TESTLAB_API_KEY ~/.test-lab/config.json
16
- * API: --api-url → $TESTLAB_API_URL → https://www.test-lab.ai
15
+ * Auth: --key -> $TESTLAB_API_KEY -> ~/.test-lab/config.json
16
+ * API: https://www.test-lab.ai (fixed)
17
17
  */
18
18
  import { parseArgs } from "node:util"
19
19
  import fs from "node:fs"
@@ -32,10 +32,10 @@ function errExit(msg) {
32
32
  process.exit(1)
33
33
  }
34
34
 
35
- const HELP = `testlab import test plans (and their data) into test-lab.ai
35
+ const HELP = `testlab - import test plans (and their data) into test-lab.ai
36
36
 
37
37
  Usage:
38
- testlab login [--key tl_] Authenticate (browser, or paste/flag a key)
38
+ testlab login [--key tl_...] Authenticate (browser, or paste/flag a key)
39
39
  testlab whoami Show the authenticated account
40
40
  testlab import <path> [--dry-run] Import a file or directory of *.json
41
41
  (credentials, labels, fixtures, plans)
@@ -49,6 +49,8 @@ Usage:
49
49
  testlab data create -f fixture.json Create a data fixture ({{data.<fixture>.<field>}})
50
50
  testlab scripts upload <file> --plan <id> Upload a local Playwright script for a plan
51
51
  (skips paid AI generation; --device optional)
52
+ testlab scripts get <id> [--out <file>] Download a plan's active script to fix + re-upload
53
+ (prints to stdout, or writes --out <file>; --device optional)
52
54
  testlab examples Print the full JSON reference for every
53
55
  resource (designed for AI agents)
54
56
  testlab skills install [--agent ...] Install the test-lab skills (test-lab-plan +
@@ -57,21 +59,20 @@ Usage:
57
59
  testlab skills update Refresh installed skills (also auto-runs after a CLI upgrade)
58
60
 
59
61
  Options:
60
- --key <tl_…> API key (else $TESTLAB_API_KEY or ~/.test-lab/config.json)
62
+ --key <tl_...> API key (else $TESTLAB_API_KEY or ~/.test-lab/config.json)
61
63
  --stdin Read the value (key/credential) from stdin
62
64
  --force (login) re-authenticate even if a stored key still works
63
65
  --dry-run (import) validate + print plan order without writing
64
66
  --project <id|name> (import/plans) assign plans to a project, or "none" for account-level
65
67
  --version, -v Print the installed CLI version
66
68
 
67
- Get a key at https://test-lab.ai/admin/settings/api-keys · run \`testlab examples\` for JSON shapes`
69
+ Get a key at https://test-lab.ai/admin/settings/api-keys / run \`testlab examples\` for JSON shapes`
68
70
 
69
71
  function parse() {
70
72
  return parseArgs({
71
73
  allowPositionals: true,
72
74
  options: {
73
75
  key: { type: "string" },
74
- "api-url": { type: "string" },
75
76
  value: { type: "string" },
76
77
  file: { type: "string", short: "f" },
77
78
  name: { type: "string" },
@@ -84,6 +85,7 @@ function parse() {
84
85
  project: { type: "string" },
85
86
  plan: { type: "string" },
86
87
  device: { type: "string" },
88
+ out: { type: "string", short: "o" },
87
89
  version: { type: "boolean", short: "v" },
88
90
  help: { type: "boolean", short: "h" },
89
91
  },
@@ -151,7 +153,6 @@ async function cmdLogin(flags) {
151
153
 
152
154
  const cfg = loadConfig()
153
155
  cfg.apiKey = apiKey
154
- cfg.apiUrl = apiUrl
155
156
  const path = saveConfig(cfg)
156
157
  log(`✓ Authenticated${who?.email ? ` as ${who.email}` : ""}. Credentials saved to ${path}`)
157
158
  }
@@ -159,7 +160,7 @@ async function cmdLogin(flags) {
159
160
  async function cmdWhoami(flags) {
160
161
  const { apiKey, apiUrl } = requireAuth(flags)
161
162
  // The two requests are independent (apiFetch never throws), so fire
162
- // them together whoami is interactive and the serial form doubled
163
+ // them together - whoami is interactive and the serial form doubled
163
164
  // its wall-clock. Identity endpoint shipped after the CLI's first
164
165
  // release: older servers 404 on /me and we just skip those lines.
165
166
  const [r, me] = await Promise.all([
@@ -284,7 +285,7 @@ async function resolveProjectId(apiUrl, apiKey, flags) {
284
285
  }
285
286
  if (!process.stdin.isTTY) {
286
287
  errExit(
287
- `You have ${projects.length} projects pick one with --project <id|name> (or --project none):\n` +
288
+ `You have ${projects.length} projects - pick one with --project <id|name> (or --project none):\n` +
288
289
  projects.map((p) => ` ${p.id} ${p.name}`).join("\n")
289
290
  )
290
291
  }
@@ -341,7 +342,7 @@ async function cmdDataList(flags) {
341
342
  }
342
343
  for (const fx of fixtures) {
343
344
  const fields = (fx.fields || []).map((f) => f.key).join(", ")
344
- log(` ${fx.key}${fx.label ? ` (${fx.label})` : ""}${fields ? ` ${fields}` : ""}`)
345
+ log(` ${fx.key}${fx.label ? ` (${fx.label})` : ""}${fields ? ` - ${fields}` : ""}`)
345
346
  }
346
347
  log(`\n${fixtures.length} fixture(s)`)
347
348
  }
@@ -370,10 +371,19 @@ async function cmdDataCreate(flags) {
370
371
  // paid AI generation. The server validates it (allow-list + intent review),
371
372
  // wraps it in the trusted harness, and stores it as the plan's saved script.
372
373
  // On rejection the per-line issues are printed so an agent can self-correct.
374
+ // Strict positive-integer plan id from a CLI arg/flag. parseInt is too lenient
375
+ // ("12abc" -> 12, "12.9" -> 12), which would silently fetch/replace the WRONG
376
+ // plan; require all-digits and > 0 so a typo errors instead.
377
+ function parsePlanIdArg(raw) {
378
+ if (typeof raw !== "string" || !/^\d+$/.test(raw.trim())) return NaN
379
+ const n = parseInt(raw.trim(), 10)
380
+ return n > 0 ? n : NaN
381
+ }
382
+
373
383
  async function cmdScriptsUpload(flags, args) {
374
384
  const file = args[2]
375
385
  if (!file) errExit("usage: testlab scripts upload <file.spec.ts> --plan <planId> [--device <device>]")
376
- const planId = flags.plan ? parseInt(flags.plan, 10) : NaN
386
+ const planId = parsePlanIdArg(flags.plan)
377
387
  if (!Number.isInteger(planId)) errExit("--plan <planId> is required (the numeric test-plan id; see `testlab plans list`)")
378
388
  const { apiKey, apiUrl } = requireAuth(flags)
379
389
 
@@ -388,7 +398,9 @@ async function cmdScriptsUpload(flags, args) {
388
398
  // plan's first configured device. Hardcoding "Desktop Chrome" here would 400
389
399
  // on any plan whose device isn't Desktop Chrome (the server validates it).
390
400
  const device = flags.device
391
- const r = await apiFetch(apiUrl, apiKey, "POST", "/api/v1/scripts/upload", { script, planId, device })
401
+ // Idempotent on the server (saved_scripts upsert ON CONFLICT), so a transient
402
+ // 5xx is safe to retry - a successful retry overwrites, it never duplicates.
403
+ const r = await apiFetch(apiUrl, apiKey, "POST", "/api/v1/scripts/upload", { script, planId, device }, { retries: 3 })
392
404
 
393
405
  // 422 = the script was understood but rejected (allow-list or intent review).
394
406
  // Print the structured issues so a human or agent can fix and retry.
@@ -404,16 +416,67 @@ async function cmdScriptsUpload(flags, args) {
404
416
  log(`✗ Script rejected by security review: ${r.json.reason}`)
405
417
  process.exit(1)
406
418
  }
407
- if (!r.ok) errExit(`${r.status}: ${r.json?.error || "upload failed"}`)
419
+ if (!r.ok) {
420
+ // A 5xx (or network error) survived the retries - almost always a transient
421
+ // server blip, not a rejected script. Tell the user it's safe to re-run:
422
+ // the upload is idempotent, so a retry can't create a duplicate. (A genuinely
423
+ // bad script comes back as 422 and is handled above, never here.)
424
+ if (r.status === 0 || r.status >= 500) {
425
+ errExit(
426
+ `${r.status || "network"}: ${r.json?.error || "upload failed"}\n` +
427
+ ` This is usually a transient server error, not a rejected script. ` +
428
+ `Re-run the same command - the upload is idempotent, so it won't create a duplicate.`,
429
+ )
430
+ }
431
+ errExit(`${r.status}: ${r.json?.error || "upload failed"}`)
432
+ }
408
433
 
409
434
  const steps = r.json?.stepCount
410
435
  // Show the device the SERVER resolved (it defaults an omitted device to the
411
436
  // plan's first), not the CLI-local flag which is undefined when --device is omitted.
412
437
  const resolvedDevice = r.json?.device || device || "the plan's default device"
413
- log(`✓ Uploaded ${file} plan #${planId} (${steps != null ? `${steps} step${steps === 1 ? "" : "s"}, ` : ""}${resolvedDevice})`)
438
+ log(`✓ Uploaded ${file} -> plan #${planId} (${steps != null ? `${steps} step${steps === 1 ? "" : "s"}, ` : ""}${resolvedDevice})`)
414
439
  log(` This plan now runs your script instead of AI generation.`)
415
440
  }
416
441
 
442
+ async function cmdScriptsGet(flags, args) {
443
+ const planId = parsePlanIdArg(args[2] ?? flags.plan)
444
+ if (!Number.isInteger(planId)) {
445
+ errExit("usage: testlab scripts get <planId> [--device <device>] [--out <file>]")
446
+ }
447
+ const { apiKey, apiUrl } = requireAuth(flags)
448
+
449
+ const qs = new URLSearchParams({ planId: String(planId) })
450
+ if (flags.device) qs.set("device", String(flags.device))
451
+ const r = await apiFetch(apiUrl, apiKey, "GET", `/api/v1/scripts?${qs.toString()}`)
452
+
453
+ // 404 carries a human message (no script / wrong device); surface it verbatim.
454
+ if (!r.ok) errExit(`${r.status}: ${r.json?.error || "failed to fetch script"}`)
455
+ const script = r.json?.script
456
+ if (typeof script !== "string") errExit("server returned no script")
457
+
458
+ const device = r.json?.device || flags.device || "the plan's default device"
459
+ const steps = r.json?.stepCount
460
+ const source = r.json?.source
461
+ const meta = `${steps != null ? `${steps} step${steps === 1 ? "" : "s"}, ` : ""}${device}${source ? `, ${source}` : ""}`
462
+ const withNl = script.endsWith("\n") ? script : script + "\n"
463
+
464
+ if (flags.out) {
465
+ try {
466
+ fs.writeFileSync(flags.out, withNl, "utf8")
467
+ } catch (e) {
468
+ errExit(`could not write ${flags.out}: ${e.message}`)
469
+ }
470
+ log(`✓ Saved plan #${planId} script -> ${flags.out} (${meta})`)
471
+ log(` Edit it, then re-upload: testlab scripts upload ${flags.out} --plan ${planId}${flags.device ? ` --device "${flags.device}"` : ""}`)
472
+ } else {
473
+ // Status line to stderr so a plain `> file` redirect captures only the
474
+ // script (pipeable), while the human still sees what they fetched.
475
+ process.stderr.write(`# plan #${planId} script (${meta})\n`)
476
+ process.stdout.write(withNl)
477
+ }
478
+ }
479
+
417
480
  function cmdExamples() {
418
481
  log(EXAMPLES_TEXT)
419
482
  }
@@ -445,7 +508,7 @@ async function cmdSkillsInstall(flags, args) {
445
508
  try {
446
509
  const res = await installSkill(name, agent, { global: flags.global })
447
510
  installed++
448
- log(`✓ ${res.name} ${agent}: ${res.dest}`)
511
+ log(`✓ ${res.name} -> ${agent}: ${res.dest}`)
449
512
  } catch (e) {
450
513
  // When installing for several agents, one failing (e.g. cursor + --global) must not abort the rest.
451
514
  if (targets.length > 1) log(` skipped ${agent}: ${e.message}`)
@@ -463,7 +526,7 @@ function cmdSkillsList() {
463
526
  }
464
527
 
465
528
  // Install EVERY current skill at every location where any test-lab skill is
466
- // already present refreshing existing copies AND adding skills shipped since
529
+ // already present - refreshing existing copies AND adding skills shipped since
467
530
  // the user last installed, so an update delivers the latest skill SET (not just
468
531
  // fresher copies of the old set). `installedSkillTargets()` is the union of
469
532
  // (agent, scope) homes; a brand-new skill has no install locations of its own
@@ -498,7 +561,7 @@ async function cmdSkillsUpdate() {
498
561
  for (const r of results) {
499
562
  const where = `${r.agent}${r.global ? " (global)" : ""}`
500
563
  if (!r.ok) log(` ✗ ${r.name} (${where}): ${r.error}`)
501
- else log(`${r.isNew ? "+ installed" : "✓ refreshed"} ${r.name} ${where}: ${r.dest}`)
564
+ else log(`${r.isNew ? "+ installed" : "✓ refreshed"} ${r.name} -> ${where}: ${r.dest}`)
502
565
  }
503
566
  const ok = results.filter((r) => r.ok)
504
567
  const added = ok.filter((r) => r.isNew).length
@@ -509,7 +572,7 @@ async function cmdSkillsUpdate() {
509
572
  }
510
573
 
511
574
  // After a CLI upgrade, refresh installed skills in place AND add any newly
512
- // shipped ones runs once per version bump (best effort). Skipped in CI / when
575
+ // shipped ones - runs once per version bump (best effort). Skipped in CI / when
513
576
  // NO_UPDATE_NOTIFIER is set.
514
577
  async function maybeRefreshSkillsOnUpgrade() {
515
578
  if (process.env.CI || process.env.NO_UPDATE_NOTIFIER) return
@@ -520,7 +583,7 @@ async function maybeRefreshSkillsOnUpgrade() {
520
583
  try {
521
584
  ok = (await refreshInstalledSkills()).filter((r) => r.ok)
522
585
  } catch {
523
- return // best effort never break the actual command
586
+ return // best effort - never break the actual command
524
587
  }
525
588
  if (ok.length === 0) return
526
589
  const added = ok.filter((r) => r.isNew).length
@@ -528,7 +591,7 @@ async function maybeRefreshSkillsOnUpgrade() {
528
591
  const bits = []
529
592
  if (refreshed) bits.push(`refreshed ${refreshed}`)
530
593
  if (added) bits.push(`installed ${added} new`)
531
- process.stderr.write(`↻ CLI upgraded ${prev} ${ver}; ${bits.join(", ")} skill location(s).\n`)
594
+ process.stderr.write(`↻ CLI upgraded ${prev} -> ${ver}; ${bits.join(", ")} skill location(s).\n`)
532
595
  }
533
596
 
534
597
  async function main() {
@@ -580,7 +643,8 @@ async function main() {
580
643
  return errExit("usage: testlab data <list|create>")
581
644
  case "scripts":
582
645
  if (args[1] === "upload") return cmdScriptsUpload(flags, args)
583
- return errExit("usage: testlab scripts upload <file.spec.ts> --plan <planId> [--device <device>]")
646
+ if (args[1] === "get") return cmdScriptsGet(flags, args)
647
+ return errExit("usage: testlab scripts <upload <file> --plan <id> | get <id> [--out <file>]>")
584
648
  case "examples":
585
649
  return cmdExamples()
586
650
  case "skills":
package/lib/api.mjs CHANGED
@@ -1,31 +1,72 @@
1
1
  /**
2
2
  * Tiny fetch wrapper for the test-lab public API. Uses global fetch (Node 18+).
3
- * Returns { ok, status, json } json is the parsed body (or { raw } on non-JSON).
3
+ * Returns { ok, status, json } - json is the parsed body (or { raw } on non-JSON).
4
+ *
5
+ * Transient resilience: a 5xx or a network error is retried with linear backoff.
6
+ * Retries are SAFE only for idempotent calls, so they default ON for GET and OFF
7
+ * for writes - a write opts in explicitly via `opts.retries` (used by the
8
+ * idempotent script upload, which the server upserts ON CONFLICT). This is what
9
+ * stops a momentary empty-body 500 (a Worker hiccup or an edge rate-limit that
10
+ * short-circuits before the app's JSON error handler) from surfacing as a hard
11
+ * "500: upload failed" when a one-second retry would have gone through.
4
12
  */
5
- export async function apiFetch(apiUrl, apiKey, method, pathname, body) {
13
+ export async function apiFetch(apiUrl, apiKey, method, pathname, body, opts = {}) {
14
+ const retries = Number.isInteger(opts.retries) ? opts.retries : method === "GET" ? 2 : 0
15
+ const backoffMs = Number.isInteger(opts.backoffMs) ? opts.backoffMs : 800
16
+
6
17
  const headers = {}
7
18
  if (apiKey) headers["Authorization"] = `Bearer ${apiKey}`
8
19
  if (body !== undefined) headers["Content-Type"] = "application/json"
9
20
 
10
- let res
11
- try {
12
- res = await fetch(`${apiUrl}${pathname}`, {
13
- method,
14
- headers,
15
- body: body !== undefined ? JSON.stringify(body) : undefined,
16
- })
17
- } catch (e) {
18
- return { ok: false, status: 0, json: { error: `network error: ${e.message}` } }
19
- }
20
-
21
- const text = await res.text()
22
- let json = null
23
- if (text) {
21
+ for (let attempt = 0; ; attempt++) {
22
+ let res
24
23
  try {
25
- json = JSON.parse(text)
26
- } catch {
27
- json = { raw: text }
24
+ res = await fetch(`${apiUrl}${pathname}`, {
25
+ method,
26
+ headers,
27
+ body: body !== undefined ? JSON.stringify(body) : undefined,
28
+ })
29
+ } catch (e) {
30
+ if (attempt < retries) {
31
+ await sleep(backoffMs * (attempt + 1))
32
+ continue
33
+ }
34
+ return { ok: false, status: 0, json: { error: `network error: ${e.message}` } }
35
+ }
36
+
37
+ // 5xx = transient (Worker error / edge rate-limit). Retry while budget remains.
38
+ if (res.status >= 500 && attempt < retries) {
39
+ await sleep(backoffMs * (attempt + 1))
40
+ continue
41
+ }
42
+
43
+ const text = await res.text()
44
+ let json = null
45
+ if (text) {
46
+ try {
47
+ json = JSON.parse(text)
48
+ } catch {
49
+ json = { raw: text }
50
+ }
51
+ }
52
+
53
+ // A failed response with no usable error body (the empty-body 5xx case) would
54
+ // otherwise reach the caller as `json: null` and print a blank reason.
55
+ // Synthesize an actionable message keyed off the status instead.
56
+ if (!res.ok && (!json || (!json.error && !json.issues && !json.reason))) {
57
+ json = {
58
+ ...(json || {}),
59
+ error:
60
+ res.status >= 500
61
+ ? `server error (HTTP ${res.status}) with no response body - usually transient`
62
+ : `request failed (HTTP ${res.status})`,
63
+ }
28
64
  }
65
+
66
+ return { ok: res.ok, status: res.status, json }
29
67
  }
30
- return { ok: res.ok, status: res.status, json }
68
+ }
69
+
70
+ function sleep(ms) {
71
+ return new Promise((resolve) => setTimeout(resolve, ms))
31
72
  }
package/lib/config.mjs CHANGED
@@ -1,9 +1,8 @@
1
1
  /**
2
2
  * Config + auth resolution for the testlab CLI.
3
3
  *
4
- * Resolution order (first hit wins):
5
- * API key: --key flag → $TESTLAB_API_KEY → ~/.test-lab/config.json
6
- * API base: --api-url → $TESTLAB_API_URL → config → https://www.test-lab.ai
4
+ * API key: --key flag -> $TESTLAB_API_KEY -> ~/.test-lab/config.json
5
+ * API base: always https://www.test-lab.ai (not configurable)
7
6
  */
8
7
  import fs from "node:fs"
9
8
  import os from "node:os"
@@ -36,8 +35,8 @@ export function saveConfig(cfg) {
36
35
  export function resolveAuth(flags) {
37
36
  const cfg = loadConfig()
38
37
  const apiKey = flags.key || process.env.TESTLAB_API_KEY || cfg.apiKey || null
39
- const apiUrl = (
40
- flags["api-url"] || process.env.TESTLAB_API_URL || cfg.apiUrl || DEFAULT_API_URL
41
- ).replace(/\/+$/, "")
42
- return { apiKey, apiUrl }
38
+ // API base is fixed to production. It used to be overridable via --api-url /
39
+ // TESTLAB_API_URL / a stored config value; that was removed so an API key is
40
+ // never sent to an arbitrary host.
41
+ return { apiKey, apiUrl: DEFAULT_API_URL }
43
42
  }
package/lib/examples.mjs CHANGED
@@ -2,9 +2,9 @@
2
2
  // Written for an AI agent: every resource's exact JSON shape, the command that
3
3
  // creates it, the reference syntax, and the combined import-bundle format.
4
4
 
5
- export const EXAMPLES_TEXT = `testlab resource reference (for humans and AI agents)
5
+ export const EXAMPLES_TEXT = `testlab - resource reference (for humans and AI agents)
6
6
 
7
- All resources are scoped to the API key's account. Auth: a tl_ key via
7
+ All resources are scoped to the API key's account. Auth: a tl_... key via
8
8
  \`testlab login\`, the TESTLAB_API_KEY env var, or --key.
9
9
 
10
10
  ═══════════════════════════════════════════════════════════════════════════
@@ -12,10 +12,10 @@ REFERENCE SYNTAX (use these inside a plan prompt, and in cookie/header values)
12
12
  ═══════════════════════════════════════════════════════════════════════════
13
13
  {{credentials.<key>}} a stored secret (never shown to the AI/model)
14
14
  {{data.<fixture>.<field>}} a value from a data fixture (generated test data)
15
- {{run.shortId}} a unique per-run id (for unique emails, names, )
15
+ {{run.shortId}} a unique per-run id (for unique emails, names, ...)
16
16
 
17
17
  ═══════════════════════════════════════════════════════════════════════════
18
- 1) CREDENTIAL a secret behind {{credentials.<key>}}
18
+ 1) CREDENTIAL - a secret behind {{credentials.<key>}}
19
19
  ═══════════════════════════════════════════════════════════════════════════
20
20
  Command: testlab credentials set email --value qa@example.com
21
21
  JSON (inside an import bundle, under "credentials"):
@@ -24,14 +24,14 @@ Rules: key starts with a letter; letters/digits/underscores; <=50 chars.
24
24
  value <= 1000 chars; stored encrypted, never returned.
25
25
 
26
26
  ═══════════════════════════════════════════════════════════════════════════
27
- 2) LABEL a tag for grouping plans (auto-created when a plan references it)
27
+ 2) LABEL - a tag for grouping plans (auto-created when a plan references it)
28
28
  ═══════════════════════════════════════════════════════════════════════════
29
29
  JSON (under "labels"): "smoke" (just the name)
30
30
  Plans can also list labels by name and they're created on the fly:
31
31
  "labels": ["smoke", "auth"]
32
32
 
33
33
  ═══════════════════════════════════════════════════════════════════════════
34
- 3) DATA FIXTURE reusable generated test data behind {{data.<fixture>.<field>}}
34
+ 3) DATA FIXTURE - reusable generated test data behind {{data.<fixture>.<field>}}
35
35
  ═══════════════════════════════════════════════════════════════════════════
36
36
  Command: testlab data create -f fixture.json
37
37
  JSON (under "fixtures"):
@@ -59,7 +59,7 @@ Rules: fixture/field keys start with a letter; letters/digits/underscores;
59
59
  Reference it in a prompt as {{data.newUser.email}}.
60
60
 
61
61
  ═══════════════════════════════════════════════════════════════════════════
62
- 4) TEST PLAN a natural-language test (the URL lives IN the prompt)
62
+ 4) TEST PLAN - a natural-language test (the URL lives IN the prompt)
63
63
  ═══════════════════════════════════════════════════════════════════════════
64
64
  Command: testlab plans create -f plan.json
65
65
  JSON (under "plans"):
@@ -84,17 +84,17 @@ Rules: name <=200 chars; prompt <=32KB; <=25 labels; <=25 preSteps.
84
84
  Projects: omit projectId to import account-level. Run "testlab projects list" for ids; "testlab import --project <id|name>" (or --project none) sets it for the whole import, and is auto-picked when you have exactly one.
85
85
 
86
86
  ═══════════════════════════════════════════════════════════════════════════
87
- IMPORT BUNDLE create everything at once: testlab import ./bundle.json
87
+ IMPORT BUNDLE - create everything at once: testlab import ./bundle.json
88
88
  ═══════════════════════════════════════════════════════════════════════════
89
89
  A file (or a directory of *.json) may contain any of these sections. The CLI
90
- creates them in order: credentials labels fixtures plans, and topo-sorts
90
+ creates them in order: credentials -> labels -> fixtures -> plans, and topo-sorts
91
91
  plans so pre-step dependencies (by "ref") are created first.
92
92
  {
93
93
  "credentials": [ { "key": "password", "value": "hunter2" } ],
94
94
  "labels": ["smoke"],
95
95
  "fixtures": [ { "key": "newUser", "fields": [ { "key": "email", "mode": "dynamic", "generator": "internet.email" } ] } ],
96
96
  "plans": [
97
- { "ref": "signup", "name": "Sign up", "prompt": "Go to https://app.example.com/signup, register with {{data.newUser.email}} " },
97
+ { "ref": "signup", "name": "Sign up", "prompt": "Go to https://app.example.com/signup, register with {{data.newUser.email}} ..." },
98
98
  { "name": "Onboard", "prompt": "Complete onboarding.", "preSteps": [ { "ref": "signup" } ] }
99
99
  ]
100
100
  }
package/lib/import.mjs CHANGED
@@ -1,5 +1,5 @@
1
1
  /**
2
- * Import orchestration: load files, then create resources in dependency order
2
+ * Import orchestration: load files, then create resources in dependency order -
3
3
  * credentials, labels, data fixtures, then plans (topo-sorted by pre-step deps,
4
4
  * wiring intra-batch pre-steps to the ids returned along the way).
5
5
  */
@@ -67,7 +67,7 @@ export function loadImportFile(target) {
67
67
  return { credentials, labels, fixtures, plans }
68
68
  }
69
69
 
70
- /** Resolve a plan's preSteps for the API: intra-batch refs concrete ids. */
70
+ /** Resolve a plan's preSteps for the API: intra-batch refs -> concrete ids. */
71
71
  export function normalizePreSteps(preSteps, refToId) {
72
72
  if (!Array.isArray(preSteps) || preSteps.length === 0) return undefined
73
73
  return preSteps.map((ps) => {
@@ -112,7 +112,7 @@ export async function runImport({
112
112
 
113
113
  if (dryRun) {
114
114
  log(
115
- `Dry run ${credentials.length} credential(s), ${labels.length} label(s), ${fixtures.length} fixture(s), ${plans.length} plan(s).`
115
+ `Dry run - ${credentials.length} credential(s), ${labels.length} label(s), ${fixtures.length} fixture(s), ${plans.length} plan(s).`
116
116
  )
117
117
  if (plans.length) {
118
118
  log(`Plan creation order:`)
@@ -137,7 +137,7 @@ export async function runImport({
137
137
  return r
138
138
  }
139
139
 
140
- // 1. Credentials (sequential each rewrites the shared encrypted blob).
140
+ // 1. Credentials (sequential - each rewrites the shared encrypted blob).
141
141
  for (const c of credentials) {
142
142
  if (!c || typeof c.key !== "string") {
143
143
  log(` ✗ credential: missing key`)
@@ -172,7 +172,7 @@ export async function runImport({
172
172
  })
173
173
  }
174
174
 
175
- // 4. Plans, in topo order, wiring intra-batch pre-steps by ref created id.
175
+ // 4. Plans, in topo order, wiring intra-batch pre-steps by ref -> created id.
176
176
  const refToId = new Map()
177
177
  let plansCreated = 0
178
178
  let plansFailed = 0
package/lib/login.mjs CHANGED
@@ -1,13 +1,13 @@
1
1
  /**
2
- * Browser device-login. Reuses test-lab's existing one-time-code API-key
2
+ * Browser device-login. Reuses test-lab's existing one-time-code -> API-key
3
3
  * handshake (the same pattern the Chrome extension uses):
4
4
  *
5
5
  * 1. CLI starts a localhost callback server + opens the browser to
6
- * <api>/cli/authorize?state=…&port=…
6
+ * <api>/cli/authorize?state=...&port=...
7
7
  * 2. The signed-in user approves; the page redirects to
8
- * http://127.0.0.1:<port>/callback?code=…&state=…
8
+ * http://127.0.0.1:<port>/callback?code=...&state=...
9
9
  * 3. CLI verifies `state`, then exchanges the one-time code for a full-scope
10
- * API key at POST /api/v1/cli/token (direct HTTPS the secret never rides
10
+ * API key at POST /api/v1/cli/token (direct HTTPS - the secret never rides
11
11
  * in the browser redirect).
12
12
  *
13
13
  * If the server doesn't support the flow (older deployment) the caller falls
@@ -87,7 +87,7 @@ export function browserLogin(apiUrl, { timeoutMs = 600000 } = {}) {
87
87
  const { port } = server.address()
88
88
  const authUrl = `${apiUrl}/cli/authorize?state=${state}&port=${port}`
89
89
  console.log(`Opening your browser to authorize the CLI:\n ${authUrl}\n`)
90
- console.log(`Waiting for you to approve in the browser (up to 10 minutes)…\n`)
90
+ console.log(`Waiting for you to approve in the browser (up to 10 minutes)...\n`)
91
91
  openBrowser(authUrl)
92
92
  })
93
93
  })
package/lib/skills.mjs CHANGED
@@ -10,9 +10,9 @@
10
10
  * stays as the browsable, open-source view; it is no longer fetched at runtime.
11
11
  *
12
12
  * Each agent has its own convention (verified against current docs):
13
- * claude .claude/skills/<name>/SKILL.md (project) | ~/.claude/skills (--global)
14
- * codex .agents/skills/<name>/SKILL.md (project) | ~/.agents/skills (--global)
15
- * cursor .cursor/rules/<name>.mdc (project only Cursor has no
13
+ * claude -> .claude/skills/<name>/SKILL.md (project) | ~/.claude/skills (--global)
14
+ * codex -> .agents/skills/<name>/SKILL.md (project) | ~/.agents/skills (--global)
15
+ * cursor -> .cursor/rules/<name>.mdc (project only - Cursor has no
16
16
  * global *file*; user rules live in Settings)
17
17
  *
18
18
  * claude + codex share the SKILL.md folder format (only the base dir differs);
@@ -54,8 +54,8 @@ export function detectAgents() {
54
54
  }
55
55
 
56
56
  /**
57
- * Where skill `name` is already installed project (cwd) and global locations
58
- * across all agents filtered to the paths that actually exist. Used to
57
+ * Where skill `name` is already installed - project (cwd) and global locations
58
+ * across all agents - filtered to the paths that actually exist. Used to
59
59
  * refresh installed skills (on `skills update` and after a CLI upgrade).
60
60
  */
61
61
  export function installedSkillLocations(name) {
@@ -74,7 +74,7 @@ export function installedSkillLocations(name) {
74
74
  * The distinct (agent, scope) locations where AT LEAST ONE test-lab skill is
75
75
  * currently installed. `skills update` and the on-upgrade refresh install EVERY
76
76
  * current skill at each of these, so a skill added in a later CLI release lands
77
- * wherever the user already keeps test-lab skills not just fresher copies of
77
+ * wherever the user already keeps test-lab skills - not just fresher copies of
78
78
  * the skills that happened to be installed before.
79
79
  */
80
80
  export function installedSkillTargets() {
@@ -98,7 +98,7 @@ export function agentTarget(agent, { global } = {}) {
98
98
  return { kind: "skill-dir", base: global ? path.join(home, ".agents", "skills") : path.join(cwd, ".agents", "skills") }
99
99
  case "cursor":
100
100
  if (global) {
101
- throw new Error("Cursor has no global rules file add user rules in Cursor Settings Rules, or install per-project (omit --global).")
101
+ throw new Error("Cursor has no global rules file - add user rules in Cursor Settings -> Rules, or install per-project (omit --global).")
102
102
  }
103
103
  return { kind: "cursor-rule", base: path.join(cwd, ".cursor", "rules") }
104
104
  default:
package/lib/toposort.mjs CHANGED
@@ -55,7 +55,7 @@ export function topoSortPlans(plans) {
55
55
  }
56
56
  }
57
57
 
58
- // Kahn's algorithm (iterative no recursion depth concerns).
58
+ // Kahn's algorithm (iterative - no recursion depth concerns).
59
59
  const queue = []
60
60
  for (let i = 0; i < n; i++) if (indegree[i] === 0) queue.push(i)
61
61
  const order = []
@@ -39,7 +39,7 @@ export function isNewer(a, b) {
39
39
  /** The notice string for (current, latest), or null if no update / bad input. */
40
40
  export function updateNotice(current, latest) {
41
41
  if (!current || !latest || !isNewer(latest, current)) return null
42
- return `\n ⚡ Update available: ${current} ${latest}\n npm i -g ${PKG}@latest\n`
42
+ return `\n ⚡ Update available: ${current} -> ${latest}\n npm i -g ${PKG}@latest\n`
43
43
  }
44
44
 
45
45
  function readCache() {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@test-lab-ai/cli",
3
- "version": "0.2.13",
3
+ "version": "0.2.15",
4
4
  "description": "Import existing test plans into test-lab.ai from the command line (or an AI agent).",
5
5
  "type": "module",
6
6
  "bin": {
@@ -32,18 +32,21 @@ This also means the script runs in a **restricted environment**: Playwright and
32
32
 
33
33
  Follow these in order.
34
34
 
35
- ### 1. Confirm the target plan exists and is reachable
35
+ ### 1. Design the test as a plan prompt first (with test-lab-plan)
36
36
 
37
- An upload attaches to an **existing** plan by numeric id. Before writing anything:
37
+ The plan's English **prompt is the spec; your script is one implementation of it.** Design the spec before you write a line of Playwright — that is what keeps the script honest and reviewable, and it is the fallback the platform runs if the script is ever cleared. Skipping this is how you end up with a script glued to a throwaway prompt that describes something else.
38
38
 
39
- - `testlab whoami` confirm the CLI is authenticated. If not, the user runs `testlab login` once or sets `TESTLAB_API_KEY`. If `testlab` is not found, use `npx @test-lab-ai/cli` instead.
40
- - `testlab plans list` find the plan id to upload to. The plan must already have a target URL (from its prompt or its project), because the script runs against that origin.
39
+ - **No plan yet?** Use the **test-lab-plan** skill to design it: one user journey, an explicit start URL, and a numbered `Verify that:` block of observable acceptance criteria. Create it (dashboard or `testlab import`), then attach your script.
40
+ - **Plan already exists?** Read its prompt that *is* your spec. If the prompt is vague, wrong, or describes a different flow, fix it with test-lab-plan **before** writing the script. Never encode behaviour in the script that the prompt doesn't claim.
41
41
 
42
- If there is no suitable plan yet, create one first (in the dashboard, or with the **test-lab-plan** skill + `testlab import`). The plan's English prompt stops driving runs the moment a script is uploaded: your script replaces AI generation for that plan and device.
42
+ Keep the prompt a clean user-journey spec: **no setup, auth, or implementation detail in the prose** — logged-in state, injected cookies, environment config, "no login needed", framework internals: none of it belongs there. Setup lives in the plan's pre-step / project environment, not the prompt body (test-lab-plan enforces this). Your script then *implements* that spec: one `test("Step N: …")` per prompt action, and an `expect` for each numbered acceptance criterion.
43
43
 
44
- ### 2. Know the flow and its checks
44
+ ### 2. Confirm the plan is reachable and read the real UI
45
45
 
46
- Same discipline as a good test plan: one user journey, explicit start, observable assertions. If you are in the target site's repo, read the relevant component / route so your locators and assertions match real DOM text and real success states, not guesses. If you are not in the repo, pull a snapshot with WebFetch or ask the user for the key screens. Anchor every `expect` to something real.
46
+ - `testlab whoami` confirm the CLI is authenticated. If not, the user runs `testlab login` once or sets `TESTLAB_API_KEY`. If `testlab` is not found, use `npx @test-lab-ai/cli` instead.
47
+ - `testlab plans list` – find the plan id to upload to. The plan must already have a target URL (from its prompt or its project), because the script runs against that origin. Uploading a script replaces AI generation for that plan + device, so from then on the prompt's job is to be the spec you implemented and the fallback — keep the two in sync.
48
+ - **Read the real UI before asserting.** If you are in the target site's repo, read the relevant component / route so your locators and assertions match real DOM text and real success states, not guesses. If you are not in the repo, pull a snapshot with WebFetch or ask the user for the key screens. Anchor every `expect` to something real **and to a criterion in the prompt.**
49
+ - **Fixing a script the plan already runs?** Download the live one with `testlab scripts get <id> [--device] [--out <file>]` (works for uploaded *and* generated scripts) instead of guessing at it, then edit and re-upload. That is the full fetch → fix → `scripts upload` loop, no dashboard needed.
47
50
 
48
51
  ### 3. Write the steps against `sharedPage`
49
52
 
@@ -140,6 +143,7 @@ The complete lists and the per-rule fixes are in `references/playwright-api.md`.
140
143
  7. **No Node / browser / network globals** (`process`, `fs`, `fetch`, `window`, `request`, …) and **no `page.evaluate`/`route`**. If you reached for one, rewrite the step to use the page.
141
144
  8. **No computed member keys or prototype access** (`x[expr]`, `.constructor`, `.__proto__`).
142
145
  9. **The plan id is right** (`testlab plans list`) and the plan has a target URL.
146
+ 10. **The script implements the plan's prompt.** Every prompt action maps to a `test("Step N: …")`, and every numbered acceptance criterion maps to an `expect`. The prompt itself reads as a clean user journey — no setup/auth/implementation detail leaked into the prose (that belongs in the plan's pre-step / environment). If the prompt and the script disagree, fix the prompt with test-lab-plan first.
143
147
 
144
148
  If any item fails, fix it before uploading – it will be rejected server-side anyway, and fixing first saves a round-trip.
145
149
 
@@ -157,7 +161,9 @@ If any item fails, fix it before uploading – it will be rejected server-side a
157
161
 
158
162
  ## Relationship to test-lab-plan
159
163
 
160
- - **test-lab-plan** describe a flow in English; the AI generates and maintains the Playwright. Best when you want low effort or don't have a script.
161
- - **test-lab-script** (this skill) – you own the Playwright; upload it verbatim. Best when you already have a script, need exact control, or want to save generation credits.
164
+ These **compose; they are not either/or.** test-lab-plan designs the spec (the plan's prompt); test-lab-script implements it (the Playwright). Even when you will hand-write the script, start with test-lab-plan so the spec is a clean, reviewable user journey and the script has a concrete contract to satisfy — see Workflow step 1.
165
+
166
+ - **test-lab-plan** – describe a flow in English; the AI generates and maintains the Playwright. Best when you want low effort, or as the spec step before you write your own script.
167
+ - **test-lab-script** (this skill) – you own the Playwright and upload it verbatim to implement that spec. Best when you need exact control, already have a script, or want to save generation credits.
162
168
 
163
169
  An uploaded script can still be refined later with AI from the dashboard (it is tagged as uploaded vs generated). `testlab examples` is the canonical, always-current reference for fixture shapes and resource formats.