ccqa 1.14.0 → 1.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (6) hide show
  1. package/README.md +125 -233
  2. package/dist/bin/ccqa.mjs +2692 -2196
  3. package/dist/hub-client/index.d.mts +96 -6
  4. package/dist/hub-client/index.mjs +20 -0
  5. package/dist/package.json +1 -1
  6. package/package.json +1 -1
package/README.md CHANGED
@@ -1,40 +1,26 @@
1
1
  # ccqa
2
2
 
3
- **Your Claude subscription already includes a QA engineer.**
3
+ > [!WARNING]
4
+ > ccqa is under active development. Expect breaking changes.
4
5
 
5
- ccqa turns Claude Code into a browser test recorder and runner. You write a
6
- test spec in YAML; Claude drives a real browser **once** to discover the
7
- route; ccqa compiles that recording into ordinary test code your CI replays
8
- with no model in the loop.
6
+ **Your Claude subscription already includes a QA engineer.**
9
7
 
10
- Recording is where the subscription pays off — `claude` on your machine is
11
- enough, no extra API key. CI is where it stops needing one: a recorded spec
12
- replays as plain test code. Only the optional Claude-driven parts —
13
- [failure analysis](#failure-analysis-and-drift), [drift](#failure-analysis-and-drift),
14
- [change selection](#wire-it-into-ci), and `mode: live` specs — need a
15
- credential in CI.
8
+ Write a test spec in YAML. Claude drives a real browser **once** to
9
+ discover the route, and ccqa compiles the recording into plain test code
10
+ your CI replays — no model in the loop, no API key. Claude returns only
11
+ where it pays: auditing specs against the code, explaining failures, and
12
+ driving `mode: live` specs.
16
13
 
17
14
  [日本語版 README](./docs/README.ja.md)
18
15
 
19
- ## Install
16
+ ## Quick start
20
17
 
21
18
  ```bash
22
- pnpm add -D ccqa vitest agent-browser
19
+ pnpm add -D ccqa vitest agent-browser # Node 20+
23
20
  ```
24
21
 
25
- Requires Node.js **20+**.
26
- [agent-browser](https://github.com/vercel-labs/agent-browser) and
27
- [vitest](https://vitest.dev) are peer dependencies of the **default
28
- agent-browser target** — they run its recorded tests. A project that only uses
29
- an external target (`playwright`, `runn`) needs just `ccqa` plus that tool
30
- (e.g. `pnpm add -D ccqa @playwright/test`); ccqa executes it through the
31
- target's `runCommand`.
32
-
33
- ## Quick start
34
-
35
- **1. Write a spec** — by hand, or interactively with
36
- [`ccqa draft`](./docs/draft.md). (`ccqa init` scaffolds the `.ccqa/`
37
- skeleton.)
22
+ Write a spec — `ccqa init` scaffolds the tree,
23
+ [`ccqa draft`](./docs/draft.md) writes one with you:
38
24
 
39
25
  ```yaml
40
26
  # .ccqa/features/tasks/test-cases/create-and-complete/spec.yaml
@@ -50,264 +36,170 @@ steps:
50
36
  expected: Task appears in the task list with status "Open"
51
37
  ```
52
38
 
53
- **2. Tell ccqa what `${APP_URL}` is.** A spec names variables instead of
54
- embedding an environment, so the same spec runs against local and staging. A
55
- `.env` file covers you locally; in CI the values come from a hub
56
- (`ccqa hub var set`) so nothing environment-specific lives in the repo. See
57
- [Profiles and environment variables](./docs/running.md#profiles-and-environment-variables).
58
-
59
- ```bash
60
- echo 'APP_URL=http://localhost:3000' >> .env
61
- ```
62
-
63
- **3. Record once** — Claude drives the browser and generates the test:
64
-
65
- ```bash
66
- ccqa record tasks/create-and-complete
67
- ```
68
-
69
- **4. Run it** — vitest replays the recording; no LLM involved:
39
+ Record once, replay forever:
70
40
 
71
41
  ```bash
72
- ccqa run tasks/create-and-complete
42
+ echo 'APP_URL=http://localhost:3000' >> .env # ${VAR}s stay out of specs
43
+ ccqa record tasks/create-and-complete # Claude drives the browser
44
+ ccqa run tasks/create-and-complete # vitest replays — no LLM
73
45
  ```
74
46
 
75
- A `report.json` (+ step screenshots) is always written to `ccqa-report/`.
76
- See [Running specs](./docs/running.md) for flags and the report format.
47
+ Every run writes `report.json` and step screenshots to `ccqa-report/`.
77
48
 
78
- If the spec sits behind a login that a recording cannot reproduce — an SSO
79
- redirect, a device-trust gate — record a session by hand once with
80
- [`ccqa hub session capture`](./docs/sessions.md) and name it in the spec.
49
+ Some logins cannot be replayed from a recording — an SSO redirect, a
50
+ device-trust prompt. Sign in by hand once with
51
+ [`ccqa hub session capture`](./docs/sessions.md), and specs start from
52
+ that saved session.
81
53
 
82
54
  ## How it works
83
55
 
84
56
  ```
85
- spec.yaml ──► ccqa record ─────► ir.json ────► ccqa generate ──► test code
86
- steps + Claude drives recorded per-target agent-browser
87
- expected the browser and actions as emit / playwright
88
- results discovers the tool-neutral (reuse-first) / runn
89
- route IR
90
-
91
- test code ──► ccqa run ────────► report.json ─► ccqa hub push /
92
- vitest replay / + evidence --report-to-hub
93
- runCommand / + artifacts team dashboard,
94
- live (Claude failure triage,
95
- drives per step) grading & learning
57
+ spec.yaml ──► ccqa record ──► ir.json ──► test code ──► ccqa run
58
+ steps + Claude drives recorded per-target replayed in CI,
59
+ expected the browser actions emit no LLM
96
60
  ```
97
61
 
98
62
  A spec runs in one of two ways:
99
63
 
100
- **Deterministic (the default).** Claude drives the browser once
101
- (`ccqa record`), and the recording is compiled into plain test code. From
102
- then on, CI just replays that code — no LLM at run time, cheapest and most
103
- stable. The `target:` field picks only **what the recording compiles
104
- into**; every target is the same deterministic replay:
64
+ **Deterministic (the default).** The recording compiles into plain test
65
+ code and CI replays it with no model in the loop. `target:` picks only
66
+ what it compiles into:
105
67
 
106
68
  | `target:` | Generated file | Replayed by |
107
69
  |---|---|---|
108
- | `agent-browser` (default) | `test.spec.ts` (vitest + agent-browser) | vitest |
109
- | `playwright` | `test.spec.ts` (plain `@playwright/test`) | your `runCommand` |
110
- | `runn` | `runbook.yaml` (API scenario — compiled from the spec, no recording) | your `runCommand` |
111
-
112
- `runCommand` is the one-line command your repo already uses to run that
113
- tool, declared once in `.ccqa/config.yaml` — e.g.
114
- `pnpm exec playwright test {files}`. See
115
- [Generation targets](./docs/targets.md) for the substitution contract.
70
+ | `agent-browser` (default) | `test.spec.ts` (vitest) | vitest |
71
+ | `playwright` | plain `@playwright/test` spec | your `runCommand` |
72
+ | `runn` | `runbook.yaml` (API scenario, no recording) | your `runCommand` |
116
73
 
117
74
  **Live (`mode: live`).** No codegen: Claude drives every run and judges
118
- each step's `expected` — for fragile, timing-heavy UIs where a fixed
119
- recording would break.
120
-
121
- ## Failure analysis and drift
75
+ each step's `expected` — for UIs a fixed recording would break on.
122
76
 
123
- A failing E2E test does not say whose problem it is. ccqa answers that
124
- question in one vocabulary, from two directions.
77
+ vitest and agent-browser are peer dependencies of the default target; a
78
+ project on an external target alone needs just `ccqa` and that tool.
79
+ `runCommand` and reusing your existing page objects:
80
+ [Generation targets](./docs/targets.md).
125
81
 
126
- **When a spec fails**, `ccqa run --on-fail-explain` labels the cause
127
- — `TEST_DRIFT`, `SPEC_CHANGE`, `PRODUCT_BUG`, or `UNKNOWN` when the evidence
128
- does not support a call. The label comes with a drift audit of the same spec,
129
- because "did the test break" and "does the test still describe the product"
130
- are the same investigation. `[base]` is what the diff is read against: a git
131
- ref, or `last-green` to have each spec diff against the commit where it last
132
- passed. With neither, the label rests on the failure alone and says so.
82
+ ## Audit, then run
133
83
 
134
- **Before anything runs**, `ccqa audit` asks the second question on its own,
135
- with no browser: does each spec still describe the code? For a deterministic
136
- spec that means both artifacts — the spec a human wrote and the test code
137
- compiled from it — since either can fall out of step. Which one drifted
138
- decides the repair, so the audit reports it: stale generated code is
139
- re-recorded, a stale spec needs a human.
84
+ **A spec describes the code your verification environment is running** —
85
+ not your branch, not the tip of main. A deploy moves that code, and some
86
+ specs stop describing it. Those specs are not failing; they say nothing
87
+ true about what runs, so executing them proves nothing.
140
88
 
141
- Every call is gradable on the hub, and the hub learns from your grades. See
142
- [Failure triage](./docs/running.md#failure-triage) and
143
- [Drift detection](./docs/running.md#drift-detection).
144
-
145
- ## The hub
89
+ So ccqa asks the cheap question before the expensive one:
146
90
 
147
- A hub is optional for one person on one machine. For a team, or for CI, it is
148
- where the shared state lives — there is no second place to put it:
149
-
150
- - the coverage inventory of what is tested
151
- ([perspectives](./docs/spec.md#inventory-coverage-with-perspectives)), kept
152
- current by `record`/`generate`
153
- - the variables `${…}` resolve to, and saved browser sessions, fetched at run
154
- time — so CI holds one secret instead of an environment
155
- - the deploy log behind `--only-hub-stale`, and the drift ledger
156
- - a dashboard of runs with per-step screenshots, triage grading, and the
157
- prompts learned from those grades
158
-
159
- ```bash
160
- export CCQA_HUB_TOKEN=$(openssl rand -hex 24)
161
- export CCQA_HUB_ENCRYPTION_KEY=$(openssl rand -hex 32) # required to store
162
- ccqa serve # sessions/variables
91
+ ```
92
+ the code the verification environment is running
93
+ │
94
+ │ a spec describes this
95
+ ▼
96
+ the deployed commit changes
97
+ │
98
+ ▼
99
+ audit the specs that change reaches
100
+ │
101
+ still describes it ───┴─── no longer describes it
102
+ │ │
103
+ ▼ ▼
104
+ run it a person repairs it
105
+ │
106
+ re-audited next round;
107
+ unverified until then
163
108
  ```
164
109
 
165
- The repository root also ships a `Dockerfile` and `docker-compose.yaml` for
166
- container deployment — clone it, or copy them from
167
- [Running the hub in a container](./docs/hub.md#running-the-hub-in-a-container);
168
- they are not part of the npm package.
169
-
170
- See [Hub](./docs/hub.md) for the full setup and
171
- [Hub API](./docs/hub-api.md) to script it over HTTP.
110
+ `ccqa audit` reads each spec against the source — cents per spec, no
111
+ browser — and records every verdict on the **hub**, the small server
112
+ that holds what the team and CI share. Stale generated code is
113
+ re-recorded; a stale spec goes to a human and stays **unverified** —
114
+ neither passing nor failing — until repaired.
172
115
 
173
- ## Wire it into CI
116
+ `ccqa run --only-hub-rerun-needed` asks the hub which specs are worth
117
+ running: cleared by the audit *and* invalidated by a deploy. A drifted spec —
118
+ or one whose last run failed — answers `needsRepair` and is never run. A run
119
+ repairs neither, and it costs dollars to learn that.
174
120
 
175
- Three jobs. They are independent: the pull-request job on its own is a
176
- complete adoption, and the other two can come later.
121
+ Selecting nothing is only an answer when every spec was answered. If any spec
122
+ is waiting on the audit, or the hub could not judge it, the run **exits
123
+ non-zero** rather than report a green run that verified nothing. And while a
124
+ run executes it claims its specs, so a cycle that starts before the last one
125
+ finishes skips what is already running instead of driving the same flow
126
+ twice.
177
127
 
178
- | Job | Trigger | Question it answers |
179
- |---|---|---|
180
- | Pre-merge run | `pull_request` | Does this change break a spec, and whose fault is it? |
181
- | Post-deploy run | after a deploy | Which specs' last result is no longer trustworthy? |
182
- | Drift audit | `schedule` | Do the specs still describe the code? |
128
+ When a clean spec still fails, `--on-fail-explain` labels whose problem
129
+ it is: `TEST_DRIFT`, `SPEC_CHANGE`, `PRODUCT_BUG`, or `UNKNOWN`. You
130
+ grade the calls on the hub, and it learns from your grades.
183
131
 
184
- All three need two things:
132
+ ## In CI
185
133
 
186
- - **A Claude credential.** Replaying a recorded spec uses no model, but the
187
- change selection, the failure analysis and the audit all do.
188
- - **A running [hub](#the-hub)**, reached with `CCQA_HUB_URL` and
189
- `CCQA_HUB_TOKEN`. Only a pre-merge run with no `--hub-profile` and no
190
- `--report-to-hub` can do without one.
191
-
192
- See [Environment variables](./docs/commands.md#environment-variables) for the
193
- full list.
194
-
195
- A **profile** is one deployed environment. It names a bucket of variables and
196
- saved sessions on the hub, and — since two environments sit at different
197
- commits — its own deploy history. Register the variables your specs reference
198
- once, from your machine:
199
-
200
- ```bash
201
- ccqa hub var set APP_URL --value https://app.example --profile staging
202
134
  ```
203
-
204
- Pass the same `--hub-profile` and `--project` in every job. That is what makes the
205
- jobs refer to the same environment.
206
-
207
- ### On a pull request
208
-
209
- Run the specs the change reaches, and label what broke.
210
-
211
- ```bash
212
- ccqa run --only-affected-by --on-fail-explain --hub-profile staging \
213
- --report-format github --report-to-hub
135
+ deploy lands
136
+ ├─ ccqa hub deploy record --select what shipped, which specs it reaches
137
+ ├─ ccqa audit --only-hub-audit-needed --report-to-hub
138
+ │ does each spec still describe it?
139
+ └─ ccqa run --only-hub-rerun-needed --on-fail-explain \
140
+ --hub-profile ci --report-to-hub
214
141
  ```
215
142
 
216
- - `--only-affected-by` selects the specs the diff reaches. A spec it cannot clear runs
217
- anyway.
218
- - `--on-fail-explain` labels the cause of each failure.
219
- - `--hub-profile staging` fetches that environment's variables and saved sessions
220
- from the hub. Without it, a spec's `${…}` references go unresolved.
221
- - `--report-format github` annotates the pull request.
222
- - `--report-to-hub` streams results to the hub as the run executes.
223
-
224
- **Set `fetch-depth: 0` on `actions/checkout`.** Both selection flags read
225
- their baseline from `GITHUB_BASE_REF` and resolve it against `origin/<base>`,
226
- which a shallow checkout does not have. Without it the run exits with a usage
227
- error before the first test. Outside a `pull_request` workflow there is no
228
- `GITHUB_BASE_REF`, so pass the base yourself: `--only-affected-by origin/main`.
143
+ The audit costs cents; a live spec costs dollars. Filtering first leaves
144
+ a run whose failures are worth reading. Record every deploy with
145
+ `--select` — a range recorded without it answers `unanswerable` forever, and
146
+ nothing fills the hole later.
229
147
 
230
- `--dry-run` prints the selection and stops. The selection costs one model call
231
- either way.
232
-
233
- ### On a deploy
148
+ | Job | Trigger | Question it answers |
149
+ |---|---|---|
150
+ | Deploy loop | after a deploy | Which specs did this deploy invalidate? |
151
+ | Pre-merge run | `pull_request` | Does this change break a spec, and whose fault is it? |
152
+ | Full audit | `schedule` | Do all the specs still describe the code? |
234
153
 
235
- Two steps, in two jobs. First, when the deploy succeeds, tell the hub what
236
- shipped:
154
+ The two jobs outside the loop:
237
155
 
238
156
  ```bash
239
- ccqa hub deploy record --profile staging --sha "$GITHUB_SHA" --select
240
- ```
157
+ # pull request — run what the diff reaches, label what broke
158
+ # (checkout with fetch-depth: 0, or the base ref is not there to resolve)
159
+ ccqa run --only-affected-by "origin/$GITHUB_BASE_REF" --on-fail-explain \
160
+ --hub-profile ci --report-format github --report-to-hub
241
161
 
242
- Then, in a job of its own, run what that deploy invalidated:
243
-
244
- ```bash
245
- ccqa run --only-hub-stale --hub-profile staging --report-to-hub
162
+ # schedule — audit everything; no browser, no deploy
163
+ ccqa audit --report-format github --report-to-hub
246
164
  ```
247
165
 
248
- - `--select` records which specs the deployed range reaches. Without it, every
249
- spec behind that entry answers `unknown` instead of `notNeeded`.
250
- - `--only-hub-stale` asks the hub, per spec, whether any deploy has touched
251
- it since that spec last ran.
252
-
253
- The hub has no checkout and never runs `git`, so it cannot work out what a
254
- deploy changed. That is why the selection is submitted with the deploy rather
255
- than reconstructed later — and why a deploy recorded without `--select` leaves
256
- a hole nothing can fill in afterwards.
257
-
258
- **Expect it to select nothing at first.** A spec with no recorded run is
259
- `neverRun`; one whose baseline predates the deploy log is `unknown`. Neither
260
- runs by default. Record a deploy, run every spec once with `--report-to-hub`,
261
- and the selection means something from the next deploy on. This job also reads
262
- the spec inventory from the hub, so `ccqa perspectives` has to have run.
263
- `--only-hub-stale-with-unknown` opts the undecided specs in.
166
+ Runnable workflows and every flag:
167
+ [CI integration](./docs/running.md#ci-integration).
264
168
 
265
- ### On a schedule
169
+ ## The hub
266
170
 
267
- Audit every spec against the codebase, with no browser and no deploy.
171
+ You have met the hub twice now: the audit writes its verdicts there,
172
+ and the run asks it what is worth running. The same server holds the
173
+ rest of what a team shares: the variables and sessions `${…}` resolves
174
+ to (CI keeps one secret), the deploy log behind the selection flags,
175
+ run reports with screenshots, and the prompts learned from your triage
176
+ grades.
268
177
 
269
178
  ```bash
270
- ccqa audit --report-format github --report-to-hub
179
+ export CCQA_HUB_TOKEN=$(openssl rand -hex 24)
180
+ export CCQA_HUB_ENCRYPTION_KEY=$(openssl rand -hex 32)
181
+ ccqa serve
271
182
  ```
272
183
 
273
- - `--exit-on warn|error` (default `error`) decides whether a verdict fails
274
- the job.
275
- - `--report-to-hub` records each verdict in the hub's per-spec drift ledger, shown in
276
- the Perspectives tab. It never changes the exit code.
277
- - `--only-affected-by <ref>` narrows the sweep on a `push` workflow, at the cost
278
- of one more model call.
279
-
280
- The pre-merge job already audits the specs that failed. This one covers the
281
- rest, because a spec can pass and still describe a product that no longer
282
- exists.
283
-
284
- ### Workflows
285
-
286
- [CI integration](./docs/running.md#ci-integration) has runnable workflows for
287
- the pre-merge run and the scheduled audit.
288
- [`ccqa hub deploy record`](./docs/hub.md#ccqa-hub-deploy-record) covers the
289
- deploy job, including a `curl`-only variant for pipelines with no Node.
184
+ Anything that needs the hub names it — `--hub-profile`,
185
+ `--only-hub-rerun-needed`, `--report-to-hub` — and fails rather than
186
+ degrade when it cannot reach one. A **profile** is a named value set — a
187
+ tenant, an account, a role — not an environment: ccqa tracks one
188
+ verification environment
189
+ ([ADR-0013](./docs/adr/0013-one-verification-environment.md)).
290
190
 
291
191
  ## Documentation
292
192
 
293
193
  | I want to… | Read |
294
194
  |---|---|
295
- | Look up a command or an environment variable | [Command reference](./docs/commands.md) |
296
- | Write specs: fields, reusable blocks, file uploads, coverage inventory | [spec.yaml reference](./docs/spec.md) |
297
- | Draft specs interactively with Claude | [Draft](./docs/draft.md) |
298
- | Generate Playwright or runn tests that reuse my existing test code | [Generation targets](./docs/targets.md) |
299
- | Run specs and read the report | [Running specs](./docs/running.md) |
300
- | Classify failures and grade the calls | [Failure triage](./docs/running.md#failure-triage) |
301
- | Audit specs against the codebase without running them | [Drift detection](./docs/running.md#drift-detection) |
302
- | Replay only the specs a change reaches | [Scoping with `--only-affected-by`](./docs/running.md#scoping-with---only-affected-by) |
303
- | Wire ccqa into GitHub Actions | [CI integration](./docs/running.md#ci-integration) |
304
- | Run specs live (no codegen), with per-project guidance | [Live specs](./docs/live.md) |
305
- | Start runs already signed in / skip device-trust gates | [Saved sessions](./docs/sessions.md) |
306
- | See which assertions generated tests use | [Assertions](./docs/assertions.md) |
307
- | Auto-fix failing recorded tests | [Auto-fix](./docs/auto-fix.md) |
308
- | Aggregate results, sessions, and variables on a team server | [Hub](./docs/hub.md) |
309
- | Script the hub over HTTP | [Hub API](./docs/hub-api.md) |
310
- | Understand why ccqa is built this way | [ADR](./docs/adr/README.md) |
195
+ | Write specs — fields, blocks, file uploads | [spec.yaml](./docs/spec.md) |
196
+ | Run specs and read the report | [Running](./docs/running.md) |
197
+ | Wire it into GitHub Actions | [CI integration](./docs/running.md#ci-integration) |
198
+ | Emit Playwright / runn tests | [Targets](./docs/targets.md) |
199
+ | Drive specs live, with per-project guidance | [Live specs](./docs/live.md) |
200
+ | Sign in once and reuse the session | [Sessions](./docs/sessions.md) |
201
+ | Run the team hub / script it over HTTP | [Hub](./docs/hub.md) · [API](./docs/hub-api.md) |
202
+ | Understand why it is built this way | [ADR](./docs/adr/README.md) |
311
203
 
312
204
  ## License
313
205