@seldonqa/cli 0.6.7 → 0.7.0

package/README.md

@@ -1,8 +1,8 @@
 # Seldon QA
 
-**Generate a QA test plan from any codebase, then run AI agents that use your product like a real user — from your terminal or CI.**
+**Write your QA test plan as code, then run AI agents that use your product like a real user — from your terminal or CI.**
 
-`seldon` turns a repository into a **baseline** of prioritized, user-facing test cases, runs those cases against a live URL with autonomous agents, and returns a clean, actionable report for you or your coding agent to act on.
+`seldon` keeps a **baseline** of user-facing test cases in a committed `seldon.yml`. Point it at a live URL and autonomous agents exercise each case — click, type, navigate, verify — then hand back a clean, actionable report for you or your coding agent to act on.
 
 ```bash
 npm install -g @seldonqa/cli
@@ -10,7 +10,7 @@ npm install -g @seldonqa/cli
 
 ## Why
 
-AI is shipping more code than any team can test by hand. Seldon closes the gap: point it at your repo and it proposes the tests that matter; point it at a URL and agents actually exercise them like users — click, type, navigate, verify — then tell you exactly what broke and where.
+AI is shipping more code than any team can test by hand. Seldon closes the gap: describe the journeys that matter once, in plain English, and agents run them against every deploy — telling you exactly what broke, why, and what to do about it.
 
 ## Get started
 
@@ -22,9 +22,10 @@ seldon login seldon_your_api_key       # or: seldon config set-key <key>
 From inside a project (one baseline per project):
 
 ```bash
-seldon init                            # create seldon.yml, preview what will be analyzed
-seldon generate                        # build a QA baseline from this codebase
-seldon cases                           # review / scope test cases
+seldon init                            # scaffold a template seldon.yml
+seldon cases add --name "Login works" \
+  --step "go to /login" --step "sign in" --criteria "lands on the dashboard"
+seldon validate                        # check the file locally (the server re-validates on run)
 seldon run --url https://staging.example.com
 seldon logs --follow                   # watch the agents test, live
 seldon report --md                     # clean, actionable results
@@ -32,56 +33,101 @@ seldon report --md                     # clean, actionable results
 
 ## Highlights
 
-- **Baselines from your code.** `seldon generate` reads the project and produces a prioritized plan of real user journeys — no scripts to write.
-- **Agents that behave like users.** `seldon run` drives your live app and judges each case pass / fail / blocked.
+- **Test plan as code.** Your cases live in a committed `seldon.yml` — reviewed in PRs, versioned with your app. `seldon init` scaffolds it; `seldon cases add` appends more.
+- **Agents that behave like users.** `seldon run` drives your live app and judges each case **PASS / FAIL / BLOCKED / TIMED_OUT**.
+- **Reports built for action.** Every failure carries an `expected` / `observed` pair and a single concrete **recommendation** — `--md` for pull requests, `--json` for coding agents, colorized plain text for the terminal.
 - **Watch it think.** `seldon logs --follow` streams the agent's reasoning, framed by test case and step.
-- **Reports built for action.** `--md` for pull requests, `--json` for coding agents, plain text for the terminal.
-- **Re-evaluate as code changes.** `seldon update` refreshes the plan when the code moves; pinned cases are always preserved.
+- **Interrupt-safe.** A submitted run executes server-side. Press Ctrl-C and the CLI detaches without stopping the run — re-attach any time with `seldon report`.
 - **CI-native gate.** `seldon run --fail-on fail` exits non-zero so a red QA run can block a merge.
 
 ## Configuration — `seldon.yml`
 
-Configuration lives in a single committed file at your project root (config-as-code). With no file, every command still works with sensible defaults.
+Your baseline is a single committed file at the project root — the plain-English cases themselves, not id overrides:
 
 ```yaml
-baseline:
-  run:
-    url: https://staging.example.com   # where runs target by default
+name: web-app                          # optional lineage name; defaults to the git remote owner/repo
+labels: [nightly, team-web]            # optional run tags for profiling; merged with `--label`
+run:
+  url: https://staging.example.com     # default target; override per run with --url
+  username: qa@example.com             # optional login the agent signs in with
+  password: hunter2                    # optional; prefer a managed secret in prod
 testCases:
-  only: []      # run ONLY these case ids (wins over skip)
-  skip: []      # never run these
-  fixed: []     # always keep these when the plan is updated (accepts: [ALL])
+  - name: Login works
+    id: login-works                    # optional stable id; defaults to a slug of the name
+    url: /login                        # optional per-case path (relative) or absolute URL override
+    steps:
+      - go to /login
+      - enter valid credentials and submit
+    criteria: the user is signed in and lands on the dashboard
 ```
 
+## Reading a report
+
+The report lists every case worst-first, then expands each failure:
+
+```
+Seldon QA — Checkout & account flows · 2/6 passed (33%) · ❌2 · 🚫1 · ⌛1
+
+FAIL      BLOCKER  TC-201  Complete payment
+BLOCKED            TC-301  View order history
+PASS               TC-101  Add item to cart
+
+TC-201  Complete payment
+  acceptance:     A valid card places the order and shows a confirmation.
+  expected:       Order is placed and a confirmation screen appears.
+  observed:       The Pay button spins indefinitely; no order is created.
+  recommendation: Wire the Pay button's submit handler to /orders and surface a failure toast.
+  repro:          Add an item, open /checkout, enter a valid card, click Pay
+```
+
+- **Verdicts:** `PASS` · `FAIL` (a real defect, with a `severity`) · `BLOCKED` (couldn't be evaluated — e.g. a missing precondition) · `TIMED_OUT`.
+- **recommendation** appears on `FAIL` / `BLOCKED` cases: the single strongest next step to get the case passing.
+- `--full` expands every failing case (the terminal view caps at 5 by default); `--json` emits the same data as a stable object.
+
 ## In CI
 
 ```bash
 seldon run --url "$PREVIEW_URL" --fail-on fail
 ```
 
-`--fail-on fail` returns a non-zero exit code when any case fails; use `--fail-on any` to also gate on blocked or timed-out cases. With `--json` the report is stable, machine-readable output you can post to a pull request or hand to a coding agent for an automated fix loop.
+`--fail-on fail` returns a non-zero exit code when any case fails; `--fail-on any` also gates on blocked or timed-out cases. Non-interactive shells block until the run finishes; `--no-wait` submits and returns immediately (re-read later with `seldon report`). With `--json` the report is stable, machine-readable output you can post to a pull request or hand a coding agent for an automated fix loop.
 
 ## Commands
 
 | Command | What it does |
 | --- | --- |
-| `seldon init` | Create `seldon.yml` and preview what will be analyzed |
-| `seldon generate` | Build a QA baseline (test plan) from the current codebase |
-| `seldon cases` | Review and scope test cases — only / skip / fixed |
-| `seldon run --url <url>` | Run the baseline against a live URL |
-| `seldon logs --follow` | Stream the agent's live reasoning (thinking only) |
-| `seldon report [--md \| --json]` | Show the latest report |
+| `seldon init` | Scaffold a template `seldon.yml` |
+| `seldon cases add` | Append a test case (`--name` `--step …` `--criteria`) |
+| `seldon cases` | List the cases in the current `seldon.yml` |
+| `seldon validate` | Validate the file locally |
+| `seldon run [--url <url>]` | Ingest `seldon.yml` and run it; `--no-wait` to detach, `--fail-on` to gate, `--label <tag>` to profile |
+| `seldon report [--md \| --json] [--full]` | Show the report for the last run (or `--id <run>`) |
+| `seldon logs --follow` | Stream the agent's live reasoning |
 | `seldon runs` | List QA runs with color-coded status |
 | `seldon cancel --id <run>` | Stop a run mid-flight |
-| `seldon update` | Re-evaluate the baseline after code changes |
-| `seldon review` | PR-style: evaluate your changes and run the affected cases |
-| `seldon list` · `versions` · `promote` | Manage baselines and versions |
+| `seldon list` | List baselines your key can see |
+| `seldon config` · `login` | Manage credentials and settings |
+| `seldon api <operation>` | Call any Blok API operation directly |
 
 Run `seldon --help` or `seldon <command> --help` for the full reference.
 
+## Tagging runs for profiling
+
+Attach freeform labels to a run — persistent ones in `seldon.yml` (`labels: [...]`) and per-invocation ones on the command line — to slice usage later:
+
+```bash
+seldon run --label nightly --label team-web
+```
+
+Labels are merged (file + flags), de-duplicated, and shown in `seldon runs` alongside each run.
+
+## Staying up to date
+
+`seldon --version` prints the installed version. When a newer release is available, the CLI shows a one-line upgrade nudge with the command that matches how you installed it — `npm install -g @seldonqa/cli` or `brew upgrade seldon`. The check is cached and never blocks a command; silence it with `NO_UPDATE_NOTIFIER=1` (it's already quiet in CI and when output is piped).
+
 ## Coding agents (MCP)
 
-Seldon is also available as an MCP server, so coding agents can generate baselines, run them, and read reports directly — enabling a closed test-and-fix loop.
+Seldon is also available as an MCP server, so coding agents can author baselines, run them, and read reports directly — the same report you see, rendered as Markdown — enabling a closed test-and-fix loop. See [`@blok/mcp`](../mcp/README.md).
 
 ## Requirements
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@seldonqa/cli",
-  "version": "0.6.7",
+  "version": "0.7.0",
   "description": "Seldon QA - generate a QA test plan from any codebase and run AI agents against your live app, from your terminal or CI.",
   "type": "module",
   "keywords": [
@@ -32,11 +32,11 @@
     "commander": "^12.1.0"
   },
   "optionalDependencies": {
-    "@seldonqa/cli-darwin-arm64": "0.6.7",
-    "@seldonqa/cli-linux-x64": "0.6.7",
-    "@seldonqa/cli-linux-arm64": "0.6.7",
-    "@seldonqa/cli-darwin-x64": "0.6.7",
-    "@seldonqa/cli-win32-x64": "0.6.7"
+    "@seldonqa/cli-linux-arm64": "0.7.0",
+    "@seldonqa/cli-darwin-arm64": "0.7.0",
+    "@seldonqa/cli-win32-x64": "0.7.0",
+    "@seldonqa/cli-linux-x64": "0.7.0",
+    "@seldonqa/cli-darwin-x64": "0.7.0"
   },
   "devDependencies": {
     "@types/node": "^20.14.0",
@@ -55,6 +55,8 @@
     "typecheck": "tsc --noEmit",
     "lint": "tsc --noEmit",
     "test": "vitest run",
-    "test:live": "BLOK_LIVE=1 vitest run"
+    "test:live": "BLOK_LIVE=1 vitest run",
+    "bench": "vitest run bench",
+    "bench:live": "BLOK_LIVE=1 vitest run bench"
   }
 }

package/postinstall.cjs

@@ -47,6 +47,15 @@ function resolvePackageBin(pkgName, binName) {
 }
 
 function main() {
+  // Only copy the native binary when installed as a dependency (inside node_modules). In the source
+  // monorepo, __dirname is packages/cli — copying there would clobber the committed bin/seldon
+  // launcher placeholder with a ~100MB platform binary and risk it being committed (which shipped a
+  // stale binary inside the published stub). Dev uses `pnpm build`/`tsx`, not this native binary.
+  if (!__dirname.split(path.sep).includes("node_modules")) {
+    console.log("[seldon] Source workspace (not node_modules) — leaving bin/seldon launcher in place.");
+    return;
+  }
+
   const key = getPlatformKey();
   const entry = PLATFORMS[key];
   if (!entry) {