@seldonqa/cli 0.11.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +84 -6
- package/package.json +6 -6
package/README.md
CHANGED
|
@@ -94,13 +94,13 @@ seldon run --url "$PREVIEW_URL" --fail-on fail
|
|
|
94
94
|
|
|
95
95
|
## Local targets — automatic tunnels
|
|
96
96
|
|
|
97
|
-
Point `--url` at a local address — `localhost`, `127.
|
|
97
|
+
Point `--url` at a local address — `localhost`, `*.localhost`, any `127.x.x.x`, `0.0.0.0`, `::1`, or a bare `:PORT` — and Seldon exposes it to the QA agent through a temporary, authenticated **Cloudflare tunnel** for the duration of the run. No deploy, no manual tunnelling:
|
|
98
98
|
|
|
99
99
|
```bash
|
|
100
100
|
seldon run --url http://localhost:3000
|
|
101
101
|
```
|
|
102
102
|
|
|
103
|
-
|
|
103
|
+
First the CLI checks that something is actually listening there. If nothing answers, it stops before doing anything else: `Nothing is answering at http://localhost:3000. Start your app (or fix the port in the URL), then run this again.` Then you'll see a one-line warning that the local address will be exposed and a `[Y/n]` prompt (defaults to yes). The public hostname is protected by Cloudflare Access, so **only this run's agent can reach it**, and the tunnel is **torn down automatically** when the run finishes or you press Ctrl-C. In CI or any non-interactive shell the prompt is skipped automatically; pass **`--auto-approve`** to skip it explicitly:
|
|
104
104
|
|
|
105
105
|
```bash
|
|
106
106
|
seldon run --url localhost:3000 --auto-approve # e.g. in a CI job
|
|
@@ -108,24 +108,102 @@ seldon run --url localhost:3000 --auto-approve # e.g. in a CI job
|
|
|
108
108
|
|
|
109
109
|
Remote `--url`s are used as-is (no tunnel).
|
|
110
110
|
|
|
111
|
-
**
|
|
111
|
+
**Ctrl-C during a tunneled run** deletes the tunnel straight away and exits. The run itself keeps going on the server but can no longer reach your app, so the CLI tells you to cancel it: `Detached — the local tunnel was closed, so run <id> can no longer reach your app. Cancel it with: seldon cancel --id <id>`. Pressed before the run has started, it says `Stopped before the run started — nothing was left running on the server.` If the CLI is killed outright (`kill -9`, or the terminal is closed), it can't clean up: the tunnel stays open until the server reclaims it, and `cloudflared` may keep running (`pkill -f 'cloudflared tunnel run'` stops it).
|
|
112
112
|
|
|
113
|
-
|
|
113
|
+
**`--no-wait` isn't allowed with a local URL.** The tunnel only stays open while `seldon run` is running, so submitting and exiting would leave the agent with nothing to reach. Drop `--no-wait`, or point `--url` at a public URL.
|
|
114
|
+
|
|
115
|
+
**Tunnels are short-lived.** Each tunnel has a hard maximum lifetime (**30 minutes** by default) and is reclaimed by the server once it's exceeded, regardless of the run's state — so the run must finish within that window. When a tunnel comes up the CLI prints the exact cap for your account (`… active for up to 30 min; the run must finish within that window`). Runs that legitimately need longer should target a deployed/public `--url` instead of a local one.
|
|
116
|
+
|
|
117
|
+
If your account has too many tunnels open at once you'll get `Tunnel limit reached — wait a minute and re-run`; if the same local target is already being provisioned you'll get a short retry hint. Neither starts a run — just re-run once things settle. If the retry hint doesn't go away, an earlier tunnel for that address couldn't be fully cleaned up and is still holding it; it's cleared by the server's scheduled tunnel cleanup. Meanwhile the same app is reachable under another address, e.g. `127.0.0.1` instead of `localhost`.
|
|
118
|
+
|
|
119
|
+
**One local target, one run at a time.** Runs against the same local URL in the same workspace share a single tunnel. That includes two terminals, a terminal and a CI job, or two teammates. Whichever run finishes first closes the tunnel, and the other run loses access to your app. Until this is fixed, run them one after another.
|
|
120
|
+
|
|
121
|
+
## Usability studies
|
|
122
|
+
|
|
123
|
+
Alongside QA, a project can keep **usability studies** as code: personas from your workspace attempt a goal on your app, and you get back a summary of where they struggled. Each study is one committed `<name>.usability.yml` file at the project root, next to `seldon.yml`. Every key is required except `version` and `credentials`:
|
|
124
|
+
|
|
125
|
+
```yaml
|
|
126
|
+
version: 1 # optional
|
|
127
|
+
experiment:
|
|
128
|
+
title: Trial signup — first-time visitors
|
|
129
|
+
goal: Sign up for a free trial and invite one teammate to your workspace.
|
|
130
|
+
hypothesis: The pricing step is where hesitation will show up.
|
|
131
|
+
type: Onboarding # Onboarding | Task Completion | Churn | Conversion |
|
|
132
|
+
# Navigation Clarity | Feature Adoption | Trust Signals
|
|
133
|
+
target:
|
|
134
|
+
device_type: Website # Website only for now
|
|
135
|
+
frame_type: Desktop # Desktop | Mobile
|
|
136
|
+
url: https://staging.example.com/signup
|
|
137
|
+
# Optional login (for apps that gate behind sign-in). Committed like a .env value
|
|
138
|
+
# (fine for staging/dev; use a managed secret for production). The agents sign in before testing.
|
|
139
|
+
credentials:
|
|
140
|
+
username: qa@example.com
|
|
141
|
+
password: hunter2
|
|
142
|
+
# To force a logged-out run, use credentials: none.
|
|
143
|
+
personas: # at least one persona name from your workspace
|
|
144
|
+
- Budget-conscious founder
|
|
145
|
+
- Skeptical IT admin
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
seldon init --type usability trial-signup # writes ./trial-signup.usability.yml
|
|
150
|
+
seldon init --type usability trial-signup --dir studies # …or into another directory (created if missing)
|
|
151
|
+
seldon validate --file trial-signup.usability.yml
|
|
152
|
+
seldon run --file trial-signup.usability.yml # waits, then prints a short report (--md / --full / --json)
|
|
153
|
+
seldon report # re-show the last run's report
|
|
154
|
+
seldon personas # the persona names your workspace has (--limit / --offset to page)
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
- **Which kind of file?** A file named `*.usability.yml` is always a usability study, so a study with a YAML error or a misspelled key gets usability errors rather than QA ones. Otherwise the CLI decides from the content: a top-level `experiment:` block is a usability study, `testCases:` is a QA suite. Pass `--type usability` (or `--type qa`) to `run` / `validate` to make that explicit; a contradicting `--type` is an error.
|
|
158
|
+
- **Checked like the server checks it.** Unknown keys are an error at every level, listing the allowed ones, so a typo such as `credential:` can't be dropped silently. `version` must be `1`. `target.url`, and `--url` on `seldon run`, must be a full http(s) URL such as `https://staging.example.com` or `http://localhost:3000` (not `localhost:3000`).
|
|
159
|
+
- **Names, not ids.** Personas and the experiment type are written by name; the server resolves them when the run starts. A typo fails the run before anything is created.
|
|
160
|
+
- **Local URLs** (`localhost`, `127.0.0.1`, …), whether in `target.url` or `--url`, are exposed through the same temporary Cloudflare tunnel as a QA run, after the same `[Y/n]` prompt (skipped with `--auto-approve`, in CI, or when not interactive). The study's starting page keeps its path: `http://localhost:3000/signup` runs as `https://<tunnel-host>/signup`. The same tunnel lifetime applies (30 minutes by default, so the study must finish within it, and `seldon run` waits exactly as long as its tunnel lives), and `--no-wait` isn't allowed. The tunnel closes as soon as all persona runs have finished (`All runs finished — closed the local tunnel.`), before the report is written.
|
|
161
|
+
- **Apps that start a service worker before rendering may load blank through the tunnel.** A dev setup like Mock Service Worker under `vite dev` waits for its worker before rendering, and on backends without the agent's worker fix (blok-main `d7c82e59d`) the worker's own requests are refused by Cloudflare Access, so the page stays blank for the personas even though it works in your own browser. Until your backend has that fix, run studies against a production build (`vite build && vite preview`, or `next build && next start`), or turn the service worker off for the study. With `vite preview`, put your API proxy and allowed hosts under `preview` (`preview.proxy`, `preview.allowedHosts`), not `server`. To check a tunnel yourself, `scripts/tunnel-access-check.sh` in this repo opens one, prints its Cloudflare Access headers so you can browse it, and runs a few curl checks.
|
|
162
|
+
- **Not yet supported for usability runs:** `--fail-on` (there's no pass/fail gate), `--label` / `--name`, and App and StaticAsset studies (`device_type` must be `Website`).
|
|
163
|
+
- **When a study can't start** (for example a persona name that doesn't exist in your workspace), the CLI lists each name that didn't resolve, or each invalid field, on its own line, and nothing is created.
|
|
164
|
+
- **Credentials** work like QA's `run.username` / `run.password`: put a `username` and `password` (always together) under `target.credentials`, and the personas sign in before testing. Any other key is an error. Leave `credentials` out and the server uses a login saved in the app whose site matches the target URL, if there is one; otherwise the personas stay logged out. Use `credentials: none` to make sure they stay logged out even when a saved login matches. A localhost target is reached through the tunnel's address, which no saved login matches, so put the login in the file.
|
|
165
|
+
- **The report prints only when the run left one**, in `seldon run` and in `seldon report`:
|
|
166
|
+
- **With a report:** the CLI prints it. If the run didn't end as `Completed`, the report is labelled partial and the exit code is 1.
|
|
167
|
+
- **Without a report:** the CLI prints why and the app link, doesn't fetch the report, and exits 1.
|
|
168
|
+
- **No report** covers a run whose persona runs all failed, a stalled run, a launch whose setup failed (safe to re-run), and a run whose dispatch was lost. The last two are caught about 30 seconds after launch.
|
|
169
|
+
- **The report is short by default.** It shows the study's question, the one-line answer and the top three findings (severity, title, personas and the recommended fix), then a link to the full report in the app:
|
|
170
|
+
|
|
171
|
+
```
|
|
172
|
+
Seldon — Can new visitors sign up and invite a teammate?
|
|
173
|
+
|
|
174
|
+
Mostly — the invite step confused two of three personas.
|
|
175
|
+
|
|
176
|
+
Top findings (3 of 5):
|
|
177
|
+
HIGH Invite button has no label — Budget-conscious founder, Skeptical IT admin
|
|
178
|
+
Fix: Label the button
|
|
179
|
+
…
|
|
180
|
+
… +2 more findings
|
|
181
|
+
|
|
182
|
+
Full report: https://<base URL>/experiments/<run id>
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
The link is the server's own link to the run in the app: `seldon run` gets it when it launches the study, and `seldon report` gets it from the experiment (or, on a backend that doesn't send it yet, from `.seldon/last-run-app-url`, saved when this project launched that run). Only when neither is available is it built from the CLI's base URL (`--base-url`, else `SELDON_BASE_URL`, else the saved config), which opens the app only when the app and the API share a host. `--md` prints the Markdown summary instead (capped, for pull requests and agents), `--full` prints it with every finding and observation, and `--json` prints the summary object.
|
|
186
|
+
- **`seldon report` on a run that's still going** says there's no report yet.
|
|
187
|
+
- **With `--json`**, nothing is printed to stdout when there's no report, so check the exit code.
|
|
188
|
+
- `seldon logs --follow` streams a study's agent thinking and stops when the run ends.
|
|
114
189
|
|
|
115
190
|
## Commands
|
|
116
191
|
|
|
117
192
|
| Command | What it does |
|
|
118
193
|
| --- | --- |
|
|
119
194
|
| `seldon init` | Scaffold a template `seldon.yml` |
|
|
195
|
+
| `seldon init --type usability [name] [--dir <dir>]` | Scaffold a `<name>.usability.yml` study |
|
|
120
196
|
| `seldon cases add` | Append a test case (`--name` `--step …` `--criteria`) |
|
|
121
197
|
| `seldon cases` | List the cases in the current `seldon.yml` |
|
|
122
|
-
| `seldon validate` | Validate
|
|
198
|
+
| `seldon validate [--file <file>] [--type qa\|usability]` | Validate `seldon.yml` (or any QA suite / usability study) locally |
|
|
199
|
+
| `seldon run --file <name>.usability.yml` | Run a usability study and print a short report with a link to the app |
|
|
123
200
|
| `seldon run [--url <url>]` | Ingest `seldon.yml` and run it; a local `--url` is auto-exposed via a Cloudflare tunnel (`--auto-approve` to skip the prompt); `--no-wait` to detach, `--fail-on` to gate, `--label <tag>` to profile |
|
|
124
|
-
| `seldon report [--md \| --json] [--full]` | Show the report for the last run (or `--id <run>`) |
|
|
201
|
+
| `seldon report [--md \| --json] [--full]` | Show the report for the last run (or `--id <run>`); for a usability run, a short report with a link to the app (`--md` / `--full` for the Markdown summary) |
|
|
125
202
|
| `seldon logs --follow` | Stream the agent's live reasoning |
|
|
126
203
|
| `seldon runs` | List QA runs with color-coded status |
|
|
127
204
|
| `seldon cancel --id <run>` | Stop a run mid-flight |
|
|
128
205
|
| `seldon list` | List baselines your key can see |
|
|
206
|
+
| `seldon personas [--limit <n>] [--offset <n>]` | List the personas your key can see (name and description, 10 at a time), for a study's `personas:` list |
|
|
129
207
|
| `seldon config` · `login` | Manage credentials and settings |
|
|
130
208
|
| `seldon api <operation>` | Call any Blok API operation directly |
|
|
131
209
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@seldonqa/cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.0",
|
|
4
4
|
"description": "Seldon QA - generate a QA test plan from any codebase and run AI agents against your live app, from your terminal or CI.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"keywords": [
|
|
@@ -32,11 +32,11 @@
|
|
|
32
32
|
"commander": "^12.1.0"
|
|
33
33
|
},
|
|
34
34
|
"optionalDependencies": {
|
|
35
|
-
"@seldonqa/cli-darwin-arm64": "0.
|
|
36
|
-
"@seldonqa/cli-
|
|
37
|
-
"@seldonqa/cli-
|
|
38
|
-
"@seldonqa/cli-linux-x64": "0.
|
|
39
|
-
"@seldonqa/cli-
|
|
35
|
+
"@seldonqa/cli-darwin-arm64": "0.13.0",
|
|
36
|
+
"@seldonqa/cli-darwin-x64": "0.13.0",
|
|
37
|
+
"@seldonqa/cli-win32-x64": "0.13.0",
|
|
38
|
+
"@seldonqa/cli-linux-x64": "0.13.0",
|
|
39
|
+
"@seldonqa/cli-linux-arm64": "0.13.0"
|
|
40
40
|
},
|
|
41
41
|
"devDependencies": {
|
|
42
42
|
"@types/node": "^20.14.0",
|