ccqa 1.12.0 → 1.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -25
- package/dist/bin/ccqa.mjs +390 -358
- package/dist/hub-client/index.d.mts +4 -4
- package/dist/package.json +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -77,7 +77,7 @@ See [Running specs](./docs/running.md) for flags and the report format.
|
|
|
77
77
|
|
|
78
78
|
If the spec sits behind a login that a recording cannot reproduce — an SSO
|
|
79
79
|
redirect, a device-trust gate — record a session by hand once with
|
|
80
|
-
[`ccqa session
|
|
80
|
+
[`ccqa hub session capture`](./docs/sessions.md) and name it in the spec.
|
|
81
81
|
|
|
82
82
|
## How it works
|
|
83
83
|
|
|
@@ -89,7 +89,7 @@ spec.yaml ──► ccqa record ─────► ir.json ────► ccqa
|
|
|
89
89
|
route IR
|
|
90
90
|
|
|
91
91
|
test code ──► ccqa run ────────► report.json ─► ccqa hub push /
|
|
92
|
-
vitest replay / + evidence --
|
|
92
|
+
vitest replay / + evidence --report-to-hub
|
|
93
93
|
runCommand / + artifacts team dashboard,
|
|
94
94
|
live (Claude failure triage,
|
|
95
95
|
drives per step) grading & learning
|
|
@@ -123,7 +123,7 @@ recording would break.
|
|
|
123
123
|
A failing E2E test does not say whose problem it is. ccqa answers that
|
|
124
124
|
question in one vocabulary, from two directions.
|
|
125
125
|
|
|
126
|
-
**When a spec fails**, `ccqa run --
|
|
126
|
+
**When a spec fails**, `ccqa run --on-fail-explain` labels the cause
|
|
127
127
|
— `TEST_DRIFT`, `SPEC_CHANGE`, `PRODUCT_BUG`, or `UNKNOWN` when the evidence
|
|
128
128
|
does not support a call. The label comes with a drift audit of the same spec,
|
|
129
129
|
because "did the test break" and "does the test still describe the product"
|
|
@@ -131,7 +131,7 @@ are the same investigation. `[base]` is what the diff is read against: a git
|
|
|
131
131
|
ref, or `last-green` to have each spec diff against the commit where it last
|
|
132
132
|
passed. With neither, the label rests on the failure alone and says so.
|
|
133
133
|
|
|
134
|
-
**Before anything runs**, `ccqa
|
|
134
|
+
**Before anything runs**, `ccqa audit` asks the second question on its own,
|
|
135
135
|
with no browser: does each spec still describe the code? For a deterministic
|
|
136
136
|
spec that means both artifacts — the spec a human wrote and the test code
|
|
137
137
|
compiled from it — since either can fall out of step. Which one drifted
|
|
@@ -152,7 +152,7 @@ where the shared state lives — there is no second place to put it:
|
|
|
152
152
|
current by `record`/`generate`
|
|
153
153
|
- the variables `${…}` resolve to, and saved browser sessions, fetched at run
|
|
154
154
|
time — so CI holds one secret instead of an environment
|
|
155
|
-
- the deploy log behind `--
|
|
155
|
+
- the deploy log behind `--only-hub-stale`, and the drift ledger
|
|
156
156
|
- a dashboard of runs with per-step screenshots, triage grading, and the
|
|
157
157
|
prompts learned from those grades
|
|
158
158
|
|
|
@@ -186,8 +186,8 @@ All three need two things:
|
|
|
186
186
|
- **A Claude credential.** Replaying a recorded spec uses no model, but the
|
|
187
187
|
change selection, the failure analysis and the audit all do.
|
|
188
188
|
- **A running [hub](#the-hub)**, reached with `CCQA_HUB_URL` and
|
|
189
|
-
`CCQA_HUB_TOKEN`. Only a pre-merge run with no `--profile` and no
|
|
190
|
-
`--
|
|
189
|
+
`CCQA_HUB_TOKEN`. Only a pre-merge run with no `--hub-profile` and no
|
|
190
|
+
`--report-to-hub` can do without one.
|
|
191
191
|
|
|
192
192
|
See [Environment variables](./docs/commands.md#environment-variables) for the
|
|
193
193
|
full list.
|
|
@@ -201,7 +201,7 @@ once, from your machine:
|
|
|
201
201
|
ccqa hub var set APP_URL --value https://app.example --profile staging
|
|
202
202
|
```
|
|
203
203
|
|
|
204
|
-
Pass the same `--profile` and `--project` in every job. That is what makes the
|
|
204
|
+
Pass the same `--hub-profile` and `--project` in every job. That is what makes the
|
|
205
205
|
jobs refer to the same environment.
|
|
206
206
|
|
|
207
207
|
### On a pull request
|
|
@@ -209,23 +209,23 @@ jobs refer to the same environment.
|
|
|
209
209
|
Run the specs the change reaches, and label what broke.
|
|
210
210
|
|
|
211
211
|
```bash
|
|
212
|
-
ccqa run --
|
|
213
|
-
--format github --
|
|
212
|
+
ccqa run --only-affected-by --on-fail-explain --hub-profile staging \
|
|
213
|
+
--report-format github --report-to-hub
|
|
214
214
|
```
|
|
215
215
|
|
|
216
|
-
- `--
|
|
216
|
+
- `--only-affected-by` selects the specs the diff reaches. A spec it cannot clear runs
|
|
217
217
|
anyway.
|
|
218
|
-
- `--
|
|
219
|
-
- `--profile staging` fetches that environment's variables and saved sessions
|
|
218
|
+
- `--on-fail-explain` labels the cause of each failure.
|
|
219
|
+
- `--hub-profile staging` fetches that environment's variables and saved sessions
|
|
220
220
|
from the hub. Without it, a spec's `${…}` references go unresolved.
|
|
221
|
-
- `--format github` annotates the pull request.
|
|
222
|
-
- `--
|
|
221
|
+
- `--report-format github` annotates the pull request.
|
|
222
|
+
- `--report-to-hub` streams results to the hub as the run executes.
|
|
223
223
|
|
|
224
224
|
**Set `fetch-depth: 0` on `actions/checkout`.** Both selection flags read
|
|
225
225
|
their baseline from `GITHUB_BASE_REF` and resolve it against `origin/<base>`,
|
|
226
226
|
which a shallow checkout does not have. Without it the run exits with a usage
|
|
227
227
|
error before the first test. Outside a `pull_request` workflow there is no
|
|
228
|
-
`GITHUB_BASE_REF`, so pass the base yourself: `--
|
|
228
|
+
`GITHUB_BASE_REF`, so pass the base yourself: `--only-affected-by origin/main`.
|
|
229
229
|
|
|
230
230
|
`--dry-run` prints the selection and stops. The selection costs one model call
|
|
231
231
|
either way.
|
|
@@ -242,12 +242,12 @@ ccqa hub deploy record --profile staging --sha "$GITHUB_SHA" --select
|
|
|
242
242
|
Then, in a job of its own, run what that deploy invalidated:
|
|
243
243
|
|
|
244
244
|
```bash
|
|
245
|
-
ccqa run --
|
|
245
|
+
ccqa run --only-hub-stale --hub-profile staging --report-to-hub
|
|
246
246
|
```
|
|
247
247
|
|
|
248
248
|
- `--select` records which specs the deployed range reaches. Without it, every
|
|
249
249
|
spec behind that entry answers `unknown` instead of `notNeeded`.
|
|
250
|
-
- `--
|
|
250
|
+
- `--only-hub-stale` asks the hub, per spec, whether any deploy has touched
|
|
251
251
|
it since that spec last ran.
|
|
252
252
|
|
|
253
253
|
The hub has no checkout and never runs `git`, so it cannot work out what a
|
|
@@ -257,24 +257,24 @@ a hole nothing can fill in afterwards.
|
|
|
257
257
|
|
|
258
258
|
**Expect it to select nothing at first.** A spec with no recorded run is
|
|
259
259
|
`neverRun`; one whose baseline predates the deploy log is `unknown`. Neither
|
|
260
|
-
runs by default. Record a deploy, run every spec once with `--
|
|
260
|
+
runs by default. Record a deploy, run every spec once with `--report-to-hub`,
|
|
261
261
|
and the selection means something from the next deploy on. This job also reads
|
|
262
262
|
the spec inventory from the hub, so `ccqa perspectives` has to have run.
|
|
263
|
-
`--
|
|
263
|
+
`--only-hub-stale-with-unknown` opts the undecided specs in.
|
|
264
264
|
|
|
265
265
|
### On a schedule
|
|
266
266
|
|
|
267
267
|
Audit every spec against the codebase, with no browser and no deploy.
|
|
268
268
|
|
|
269
269
|
```bash
|
|
270
|
-
ccqa
|
|
270
|
+
ccqa audit --report-format github --report-to-hub
|
|
271
271
|
```
|
|
272
272
|
|
|
273
|
-
- `--
|
|
273
|
+
- `--exit-on warn|error` (default `error`) decides whether a verdict fails
|
|
274
274
|
the job.
|
|
275
|
-
- `--
|
|
275
|
+
- `--report-to-hub` records each verdict in the hub's per-spec drift ledger, shown in
|
|
276
276
|
the Perspectives tab. It never changes the exit code.
|
|
277
|
-
- `--
|
|
277
|
+
- `--only-affected-by <ref>` narrows the sweep on a `push` workflow, at the cost
|
|
278
278
|
of one more model call.
|
|
279
279
|
|
|
280
280
|
The pre-merge job already audits the specs that failed. This one covers the
|
|
@@ -299,7 +299,7 @@ deploy job, including a `curl`-only variant for pipelines with no Node.
|
|
|
299
299
|
| Run specs and read the report | [Running specs](./docs/running.md) |
|
|
300
300
|
| Classify failures and grade the calls | [Failure triage](./docs/running.md#failure-triage) |
|
|
301
301
|
| Audit specs against the codebase without running them | [Drift detection](./docs/running.md#drift-detection) |
|
|
302
|
-
| Replay only the specs a change reaches | [Scoping with `--
|
|
302
|
+
| Replay only the specs a change reaches | [Scoping with `--only-affected-by`](./docs/running.md#scoping-with---only-affected-by) |
|
|
303
303
|
| Wire ccqa into GitHub Actions | [CI integration](./docs/running.md#ci-integration) |
|
|
304
304
|
| Run specs live (no codegen), with per-project guidance | [Live specs](./docs/live.md) |
|
|
305
305
|
| Start runs already signed in / skip device-trust gates | [Saved sessions](./docs/sessions.md) |
|