@datadisco/qa 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +172 -10
- package/dist/cli.js +1323 -110
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -9,9 +9,21 @@ simulation ("confirmation testing for agentic teams").
|
|
|
9
9
|
npm i -g @datadisco/qa # or run ad hoc: npx @datadisco/qa <command>
|
|
10
10
|
```
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
### Quick start
|
|
13
|
+
|
|
14
|
+
```sh
|
|
15
|
+
npx @datadisco/qa init
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
`init` signs you in from the browser, connects the current repo to your
|
|
19
|
+
DataDisco workspace, works out how your previews get deployed, and — only when
|
|
20
|
+
your host doesn't emit GitHub deploy events — writes a `report-preview`
|
|
21
|
+
workflow and stores the API token as an Actions secret. Run
|
|
22
|
+
`npx @datadisco/qa doctor` afterwards (or any time) to re-check the setup.
|
|
23
|
+
|
|
24
|
+
Most teams' CI consumes the CLI through the
|
|
13
25
|
[`Data-Disco-Inc/qa-action`](https://github.com/Data-Disco-Inc/qa-action)
|
|
14
|
-
GitHub Action, which wraps this CLI —
|
|
26
|
+
GitHub Action, which wraps this CLI — `init` writes that workflow for you.
|
|
15
27
|
|
|
16
28
|
## Why this exists
|
|
17
29
|
|
|
@@ -23,32 +35,178 @@ App can't see:
|
|
|
23
35
|
|
|
24
36
|
| Command | For teams that… |
|
|
25
37
|
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
38
|
+
| `datadisco-qa init` | want the whole setup — sign-in, repo connection, preview reporting, secret — done from one prompt |
|
|
39
|
+
| `datadisco-qa doctor` | want to know why a run isn't starting: checks the token, connection, preview source, workflow, and secret |
|
|
26
40
|
| `datadisco-qa report-preview --url <url>` | deploy previews without emitting GitHub Deployment events — report the URL from CI so the pending run starts |
|
|
27
41
|
| `datadisco-qa wait --pr <n>` | want a pipeline step to block on the QA verdict instead of relying on branch protection |
|
|
28
42
|
| `datadisco-qa tunnel --port <port>` | want to run personas against a build served on their own machine |
|
|
29
43
|
|
|
30
44
|
## Authentication
|
|
31
45
|
|
|
32
|
-
Every command needs a **workspace API token
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
to the hosted app and is overridable with `--api-url` / `DATADISCO_API_URL`.
|
|
46
|
+
Every command needs a **workspace API token**. In CI, pass it with
|
|
47
|
+
`--api-token` or the `DATADISCO_API_TOKEN` env var (create one under
|
|
48
|
+
_Workspace settings → Integrations → API tokens_):
|
|
36
49
|
|
|
37
50
|
```sh
|
|
38
51
|
export DATADISCO_API_TOKEN=ddqa_xxx
|
|
39
52
|
```
|
|
40
53
|
|
|
54
|
+
On your own machine, `datadisco-qa login` (or the first `init`) signs you in
|
|
55
|
+
from the browser instead: it shows a short pairing code, you confirm it in
|
|
56
|
+
DataDisco, and the resulting token is saved to the CLI config file.
|
|
57
|
+
`init` and `doctor` look for a token in this order: `--api-token`,
|
|
58
|
+
`DATADISCO_API_TOKEN`, then the saved sign-in for the API URL in use.
|
|
59
|
+
`report-preview`, `wait`, and `tunnel` only read the flag and env var.
|
|
60
|
+
|
|
61
|
+
The API base URL defaults to the hosted app and is overridable with
|
|
62
|
+
`--api-url` / `DATADISCO_API_URL`; saved sign-ins are keyed by it, so a
|
|
63
|
+
staging server gets its own entry.
|
|
64
|
+
|
|
65
|
+
### Config file
|
|
66
|
+
|
|
67
|
+
| Location | Contents |
|
|
68
|
+
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
|
|
69
|
+
| `$XDG_CONFIG_HOME/datadisco/qa.json`, else `~/.config/datadisco/qa.json` | `{ "tokens": { "<apiUrl>": { "token", "workspaceName", "workspaceSlug", "savedAt" } } }` |
|
|
70
|
+
|
|
71
|
+
The file is written with mode `0600`. `datadisco-qa logout` removes the entry
|
|
72
|
+
for the current API URL.
|
|
73
|
+
|
|
74
|
+
## Works in any CI
|
|
75
|
+
|
|
76
|
+
`report-preview` and `wait` auto-detect the repository, PR number, and head
|
|
77
|
+
SHA from whatever CI they're running in — GitHub Actions gets full support out
|
|
78
|
+
of the box, and every other CI (or a bare git checkout) still works with zero
|
|
79
|
+
flags in the common case:
|
|
80
|
+
|
|
81
|
+
```sh
|
|
82
|
+
DATADISCO_API_TOKEN=ddqa_xxx npx @datadisco/qa report-preview --url https://pr-482.preview.acme.dev
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Each value is resolved through its own chain of fallbacks, in order, stopping
|
|
86
|
+
at the first one that resolves:
|
|
87
|
+
|
|
88
|
+
| Value | Resolution order |
|
|
89
|
+
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
|
90
|
+
| Repository | `--repo` → `$GITHUB_REPOSITORY` → the git checkout's github.com `origin` remote |
|
|
91
|
+
| Head SHA | `--sha` → the GitHub Actions `pull_request` event payload → `$GITHUB_SHA` → the git checkout's `HEAD` |
|
|
92
|
+
| PR number | `--pr` → GitHub Actions (`GITHUB_REF` / event payload) → CircleCI, Jenkins, Buildkite, Travis CI, or Drone's PR env var → _(unresolved)_ |
|
|
93
|
+
|
|
94
|
+
`--repo`, `--pr`, and `--sha` always override auto-detection when passed.
|
|
95
|
+
|
|
96
|
+
Unlike the repository and head SHA, the PR number is allowed to stay
|
|
97
|
+
unresolved: `report-preview` will still report the preview URL with just the
|
|
98
|
+
repo + head SHA, and the DataDisco server resolves the PR from the commit
|
|
99
|
+
itself. `wait` needs _either_ a PR number or a head SHA to poll by — pass
|
|
100
|
+
`--sha` explicitly if neither auto-detects (e.g. a CI DataDisco doesn't
|
|
101
|
+
recognize yet).
|
|
102
|
+
|
|
103
|
+
| CI | Repo | Head SHA | PR number |
|
|
104
|
+
| ----------------- | ------------------- | ------------------------------------- | ----------------------------------------------------------------- |
|
|
105
|
+
| GitHub Actions | `GITHUB_REPOSITORY` | `pull_request` payload / `GITHUB_SHA` | `GITHUB_REF` / `pull_request` payload |
|
|
106
|
+
| CircleCI | git origin remote | git `HEAD` | `CIRCLE_PULL_REQUEST` / `CIRCLE_PR_NUMBER` |
|
|
107
|
+
| Jenkins | git origin remote | git `HEAD` | `CHANGE_ID` |
|
|
108
|
+
| Buildkite | git origin remote | git `HEAD` | `BUILDKITE_PULL_REQUEST` |
|
|
109
|
+
| Travis CI | git origin remote | git `HEAD` | `TRAVIS_PULL_REQUEST` |
|
|
110
|
+
| Drone | git origin remote | git `HEAD` | `DRONE_PULL_REQUEST` |
|
|
111
|
+
| Bare git checkout | git origin remote | git `HEAD` | not auto-detected — pass `--pr` or rely on server-side resolution |
|
|
112
|
+
|
|
113
|
+
The git `origin` remote fallback only recognizes github.com remotes (SSH or
|
|
114
|
+
HTTPS); for anything else, pass `--repo` explicitly.
|
|
115
|
+
|
|
41
116
|
## Commands
|
|
42
117
|
|
|
118
|
+
### `init`
|
|
119
|
+
|
|
120
|
+
Interactive, idempotent setup for the current repo:
|
|
121
|
+
|
|
122
|
+
1. **Token** — uses `--api-token` / `DATADISCO_API_TOKEN` / the saved
|
|
123
|
+
sign-in, or pairs this machine from the browser and saves the token.
|
|
124
|
+
2. **Repo** — `--repo owner/name` or the git `origin` remote.
|
|
125
|
+
3. **Connection** — looks the repo up in the workspace. If it isn't connected
|
|
126
|
+
yet, prints (and opens) the connect page and waits up to 10 minutes for
|
|
127
|
+
you to finish in the browser.
|
|
128
|
+
4. **Host detection** — purely local: looks for `vercel.json` / `.vercel/`,
|
|
129
|
+
`netlify.toml`, `amplify.yml`, `fly.toml`, `render.yaml`,
|
|
130
|
+
`railway.json` / `railway.toml`, and scans `.github/workflows/*.yml` for
|
|
131
|
+
host names.
|
|
132
|
+
5. **Preview reporting** — Vercel, Netlify, and Amplify's Git integrations
|
|
133
|
+
emit GitHub deploy events, so when one of their marker files is present
|
|
134
|
+
nothing is written (likewise when the connection already uses a URL
|
|
135
|
+
pattern). A workflow that mentions one of them is taken to deploy through
|
|
136
|
+
its CLI, which emits no deploy events. In every other case it writes
|
|
137
|
+
`.github/workflows/datadisco-qa.yml` using `Data-Disco-Inc/qa-action@v1`:
|
|
138
|
+
a `pull_request` recipe with your preview URL pattern
|
|
139
|
+
(`https://pr-{number}.example.dev`, defaulting to the pattern configured in
|
|
140
|
+
DataDisco). Without a pattern nothing is written and `init` exits `1`. An
|
|
141
|
+
existing file is never overwritten without confirmation. It then offers to
|
|
142
|
+
store your sign-in token — a personal token, so CI acts as you — as the
|
|
143
|
+
`DATADISCO_API_TOKEN` Actions secret via `gh secret set` when `gh` is
|
|
144
|
+
installed and signed in, or prints the manual steps. An existing secret is
|
|
145
|
+
kept unless you explicitly agree to replace it (never under `--yes`).
|
|
146
|
+
6. **Summary** — the repo's DataDisco URL and what was written.
|
|
147
|
+
|
|
148
|
+
```sh
|
|
149
|
+
datadisco-qa init
|
|
150
|
+
datadisco-qa init --yes --repo acme/site # non-interactive: never waits, never overwrites
|
|
151
|
+
datadisco-qa init --no-browser # print URLs instead of opening them
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
| Flag | Effect |
|
|
155
|
+
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
156
|
+
| `--yes` | Skip prompts: take the server's pattern (exit `1` when it has none), never overwrite, exit `2` instead of waiting for a connection |
|
|
157
|
+
| `--repo` | Repository, when the git remote isn't the one you mean |
|
|
158
|
+
| `--no-browser` | Don't try to open the pairing / connect pages |
|
|
159
|
+
|
|
160
|
+
**Exit codes:** `0` done, `1` error, `2` stopped waiting for you (re-run once
|
|
161
|
+
the repo is connected).
|
|
162
|
+
|
|
163
|
+
### `doctor`
|
|
164
|
+
|
|
165
|
+
Re-checks the setup and prints one line per check — `✓` pass, `✗` fail with a
|
|
166
|
+
fix hint, `–` skipped:
|
|
167
|
+
|
|
168
|
+
- the API token is present and accepted by the server,
|
|
169
|
+
- the repo is resolved and has a healthy GitHub connection (App installed,
|
|
170
|
+
enabled, `ACTIVE`),
|
|
171
|
+
- the preview source: the configured URL pattern, or whether a preview URL has
|
|
172
|
+
ever arrived via deploy events (warning when it hasn't and the local host
|
|
173
|
+
doesn't emit them),
|
|
174
|
+
- the `datadisco-qa.yml` workflow is present when CI needs to report previews
|
|
175
|
+
(one that only runs on `deployment_status` doesn't count as a preview
|
|
176
|
+
source),
|
|
177
|
+
- the `DATADISCO_API_TOKEN` Actions secret exists (via `gh secret list`;
|
|
178
|
+
skipped without `gh`). Both this and the workflow check are skipped when
|
|
179
|
+
`init` would write neither: the connection uses a URL pattern, the host
|
|
180
|
+
emits deploy events, or previews already arrive without the workflow,
|
|
181
|
+
- the connection's live URL responds (5 s timeout) when one is set,
|
|
182
|
+
- the last run's status, verdict, and time.
|
|
183
|
+
|
|
184
|
+
```sh
|
|
185
|
+
datadisco-qa doctor
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Exits `1` if any check fails, `0` otherwise.
|
|
189
|
+
|
|
190
|
+
### `login` / `logout`
|
|
191
|
+
|
|
192
|
+
```sh
|
|
193
|
+
datadisco-qa login # pair this machine and save the token
|
|
194
|
+
datadisco-qa login --no-browser
|
|
195
|
+
datadisco-qa logout # forget the saved token for the API URL
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
`login` always pairs afresh, replacing any saved token for that API URL.
|
|
199
|
+
|
|
43
200
|
### `report-preview`
|
|
44
201
|
|
|
45
202
|
Report the deployed preview URL for a PR head SHA so its pending QA run leaves
|
|
46
|
-
`PENDING_PREVIEW` and starts.
|
|
47
|
-
|
|
203
|
+
`PENDING_PREVIEW` and starts. `--repo`, `--pr`, and `--sha` are auto-detected —
|
|
204
|
+
see [Works in any CI](#works-in-any-ci) — and the PR number is optional: if it
|
|
205
|
+
can't be auto-detected, the server resolves it from the head SHA itself.
|
|
48
206
|
|
|
49
207
|
```sh
|
|
50
208
|
datadisco-qa report-preview --url https://pr-482.preview.acme.dev
|
|
51
|
-
# explicit outside
|
|
209
|
+
# explicit, e.g. outside any auto-detected CI:
|
|
52
210
|
datadisco-qa report-preview \
|
|
53
211
|
--url https://pr-482.preview.acme.dev \
|
|
54
212
|
--repo acme/site --pr 482 --sha "$GIT_SHA"
|
|
@@ -73,10 +231,14 @@ message, instead of burning the retry window.
|
|
|
73
231
|
### `wait`
|
|
74
232
|
|
|
75
233
|
Block until the PR's QA run reaches a verdict, printing each phase transition.
|
|
76
|
-
Polls every 15s; `--timeout` (minutes, default 30) caps the wait.
|
|
234
|
+
Polls every 15s; `--timeout` (minutes, default 30) caps the wait. `--repo` and
|
|
235
|
+
`--pr` are auto-detected the same way as `report-preview`; if the PR number
|
|
236
|
+
isn't detectable, pass `--sha` to poll by head SHA instead.
|
|
77
237
|
|
|
78
238
|
```sh
|
|
79
239
|
datadisco-qa wait --pr 482
|
|
240
|
+
# or, when the PR number isn't auto-detected:
|
|
241
|
+
datadisco-qa wait --sha "$GIT_SHA"
|
|
80
242
|
```
|
|
81
243
|
|
|
82
244
|
**Exit codes** (so pipelines can branch on the outcome):
|