ccqa 0.13.1 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. package/README.md +94 -140
  2. package/dist/bin/ccqa.mjs +8764 -4715
  3. package/dist/hub-client/index.d.mts +351 -6
  4. package/dist/hub-client/index.mjs +39 -0
  5. package/dist/package.json +1 -1
  6. package/dist/runtime/test-helpers.mjs +1 -1
  7. package/dist/{spawn-ab-Ja8NRRab.mjs → spawn-ab-Cr967J2N.mjs} +2 -1
  8. package/package.json +1 -1
package/README.md CHANGED
@@ -2,10 +2,15 @@
2
2
 
3
3
  **Your Claude subscription already includes a QA engineer.**
4
4
 
5
- ccqa turns Claude Code into a browser test recorder. Write a spec in YAML, declare in the spec whether it should run **deterministic** or **live**, then `ccqa run` does the right thing per spec:
5
+ ccqa turns Claude Code into a browser test recorder and runner:
6
6
 
7
- - **Deterministic** (`mode: deterministic`, default): record once with `ccqa record` — Claude drives the browser and ccqa compiles every action into a `test.spec.ts` you replay in CI under vitest, no LLM at run time. Cheapest and most stable.
8
- - **Live** (`mode: live`): no codegen. `ccqa run` sends each step to Claude every time; Claude drives `agent-browser`, judges pass/fail against the step's `expected`, and saves a before/after screenshot. More flexible for fragile UIs.
7
+ 1. Write a test spec in YAML — plain steps and expected results.
8
+ 2. Claude drives a real browser **once** to discover the route
9
+ (`ccqa record`).
10
+ 3. ccqa compiles the recording into runnable test code for your `target:`
11
+ — vitest replay, plain Playwright, or a runn runbook.
12
+ 4. `ccqa run` replays everything into one report you can push to a
13
+ shared hub.
9
14
 
10
15
  No extra API key. Just `claude`.
11
16
 
@@ -13,15 +18,48 @@ No extra API key. Just `claude`.
13
18
 
14
19
  ## How it works
15
20
 
16
- Each spec picks its own mode; `ccqa run` reads the field and dispatches. One project mixes both, and one report covers both.
17
-
18
- ```mermaid
19
- flowchart LR
20
- A["Write spec\n(spec.yaml + mode:)"] --> B{mode}
21
- B -- deterministic --> C["ccqa record\n(Claude → test.spec.ts)"]
22
- C --> D["ccqa run\n(vitest replay, no LLM)"]
23
- B -- live --> E["ccqa run\n(Claude drives every time,\nper-step pass/fail)"]
24
21
  ```
22
+ spec.yaml ──► ccqa record ─────► ir.json ────► ccqa generate ──► test code
23
+ steps + Claude drives recorded per-target agent-browser
24
+ expected the browser and actions as emit / playwright
25
+ results discovers the tool-neutral (reuse-first) / runn
26
+ route IR
27
+
28
+ test code ──► ccqa run ────────► report.json ─► ccqa hub push /
29
+ vitest replay / + evidence --push-report
30
+ runCommand / + artifacts team dashboard,
31
+ live (Claude failure triage,
32
+ drives per step) grading & learning
33
+ ```
34
+
35
+ A spec runs in one of two ways:
36
+
37
+ **Deterministic (the default).** Claude drives the browser once
38
+ (`ccqa record`), and the recording is compiled into plain test code. From
39
+ then on, CI just replays that code — no LLM at run time, cheapest and most
40
+ stable. The `target:` field picks only **what the recording compiles
41
+ into**; every target is the same deterministic replay:
42
+
43
+ | `target:` | Generated file | Replayed by |
44
+ |---|---|---|
45
+ | `agent-browser` (default) | `test.spec.ts` (vitest + agent-browser) | vitest |
46
+ | `playwright` | `test.spec.ts` (plain `@playwright/test`) | your `runCommand` |
47
+ | `runn` | `runbook.yaml` (API scenario — compiled from the spec, no recording) | your `runCommand` |
48
+
49
+ `runCommand` is the one-line command your repo already uses to run that
50
+ tool, declared once in `.ccqa/config.yaml` — e.g.
51
+ `pnpm exec playwright test {files}`. ccqa substitutes the spec's generated
52
+ test files for `{files}` and a per-spec artifacts directory for
53
+ `{artifactsDir}`; see [Generation targets](./docs/targets.md) for the full
54
+ contract.
55
+
56
+ **Live (`mode: live`).** No codegen: Claude drives every run and judges
57
+ each step's `expected` — for fragile, timing-heavy UIs where a fixed
58
+ recording would break.
59
+
60
+ Either way, every failing spec gets a root-cause call (TEST_DRIFT /
61
+ SPEC_CHANGE / PRODUCT_BUG) you can grade on the hub — and the hub learns
62
+ from your grades.
25
63
 
26
64
  ## Install
27
65
 
@@ -29,16 +67,19 @@ flowchart LR
29
67
  pnpm add -D ccqa vitest agent-browser
30
68
  ```
31
69
 
32
- Requires Node.js **20+**. [agent-browser](https://github.com/vercel-labs/agent-browser) is a peer dependency.
70
+ Requires Node.js **20+**.
71
+ [agent-browser](https://github.com/vercel-labs/agent-browser) and
72
+ [vitest](https://vitest.dev) are peer dependencies.
33
73
 
34
74
  ## Quick start
35
75
 
36
- **1. Write a spec** — by hand, or interactively with [`ccqa draft`](./docs/draft.md). Declare the mode in the spec itself.
76
+ **1. Write a spec** — by hand, or interactively with
77
+ [`ccqa draft`](./docs/draft.md). (`ccqa init` scaffolds the `.ccqa/`
78
+ skeleton.)
37
79
 
38
80
  ```yaml
39
81
  # .ccqa/features/tasks/test-cases/create-and-complete/spec.yaml
40
82
  title: Create a task and mark it complete
41
- mode: deterministic # or: live. Omit for deterministic (the default).
42
83
 
43
84
  steps:
44
85
  - instruction: |
@@ -46,151 +87,64 @@ steps:
46
87
  expected: Redirected to /dashboard, user avatar visible in the header
47
88
 
48
89
  - instruction: |
49
- Click "New Task", fill in the title "Fix login bug", set priority to High, save.
90
+ Click "New Task", fill in the title "Fix login bug", save.
50
91
  expected: Task appears in the task list with status "Open"
51
92
  ```
52
93
 
53
- URLs live inside `instruction` strings — either verbatim or via `${ENV_VAR}` references for environment-specific values.
54
-
55
- **2a. For `mode: deterministic` — record once, then replay**
94
+ **2. Record once** — Claude drives the browser and generates the test:
56
95
 
57
96
  ```bash
58
- ccqa record tasks/create-and-complete # Claude drives the browser; generates test.spec.ts
59
- ccqa run tasks/create-and-complete # vitest replays test.spec.ts; no LLM
97
+ ccqa record tasks/create-and-complete
60
98
  ```
61
99
 
62
- **2b. For `mode: live` — skip codegen, run directly**
100
+ **3. Run it** — vitest replays the recording; no LLM involved:
63
101
 
64
102
  ```bash
65
- ccqa run tasks/create-and-complete # Claude drives the browser every time
103
+ ccqa run tasks/create-and-complete
66
104
  ```
67
105
 
68
- Live specs can start already-signed-in by naming a saved session with `session:` — see [Saved sessions](./docs/sessions.md). Deterministic runs also write step-boundary screenshots to `ccqa-report/evidence/` (disable with `--no-evidence`).
106
+ A `report.json` (+ step screenshots) is always written to `ccqa-report/`.
107
+ See [Running specs](./docs/running.md) for flags, CI recipes, and the
108
+ report format.
69
109
 
70
- In CI, add `--report` to write a `report.json` (+ evidence PNGs) you push to the [ccqa hub](./docs/hub.md) and view in its UI; each failing spec gets a root-cause call (TEST_DRIFT / SPEC_CHANGE / PRODUCT_BUG) you can grade for accuracy. Needs `ANTHROPIC_API_KEY` or a local Claude login. See [Run report](./docs/report.md).
71
-
72
- ```bash
73
- ccqa run tasks/create-and-complete --report --base origin/main
74
- ccqa run --changed --report # only specs whose relatedPaths touch the diff
75
- ```
110
+ **4. Optional: share results on a hub** — `ccqa serve` starts a small
111
+ self-hosted server (or use the bundled `docker-compose.yaml`). Pushing
112
+ reports to it gives your team:
76
113
 
77
- ## Features
78
-
79
- | Feature | Docs |
80
- |---|---|
81
- | Write specs interactively with Claude | [Draft](./docs/draft.md) |
82
- | Run specs live (no codegen), with per-project guidance | [Live specs](./docs/live.md) |
83
- | Restore a saved login to skip device-trust gates | [Saved sessions](./docs/sessions.md) |
84
- | Reuse login and other shared step sequences | [Blocks](./docs/blocks.md) |
85
- | Drive `<input type="file">` without an OS picker | [File upload](./docs/file-upload.md) |
86
- | Assertion helper functions | [Assertions](./docs/assertions.md) |
87
- | Auto-fix failing tests | [Auto-fix](./docs/auto-fix.md) |
88
- | Detect spec/code drift in CI | [Drift](./docs/drift.md) |
89
- | HTML run report with failure root-cause calls | [Run report](./docs/report.md) |
90
- | Inventory existing test coverage | [Perspectives](./docs/perspectives.md) |
91
- | Architecture decision records (why it is built this way) | [ADR](./docs/adr/README.md) |
92
- | Aggregate CI run results, sessions, and variables on a shared server | [Hub](./docs/hub.md) |
93
-
94
- ## Commands
95
-
96
- ```
97
- ccqa init Scaffold .ccqa/prompts/{live,record}.{user,agent}.md templates
98
- ccqa draft [feature/spec] Co-author a test spec with Claude
99
- ccqa perspectives Inventory existing test coverage into .ccqa/perspectives.yaml
100
- ccqa record <feature/spec> (deterministic specs only) Trace browser actions + generate test.spec.ts
101
- ccqa run [feature/spec...] Execute specs. Per spec, the spec.yaml `mode:` field selects deterministic
102
- (vitest replay) or live (Claude drives every time). One run can mix both;
103
- `--report` writes one unified HTML. Pass multiple targets space-separated.
104
- ccqa drift [feature/spec] Standalone spec ↔ codebase static audit (for PR checks)
105
- ccqa serve Start a hub: a control-plane HTTP server that aggregates run results,
106
- sessions, and variables (it does not execute tests)
107
- ccqa hub push Push a finished run's report to a hub
108
- ccqa hub session|var <cmd> Manage sessions/variables stored on a hub
109
- ```
110
-
111
- Key `ccqa run` flags (see `ccqa run --help` for the rest):
112
-
113
- - `--report [dir]` — write the run report (report.json + evidence PNGs) for `ccqa hub push` (default dir: `ccqa-report/`)
114
- - `--profile <name>` — load `.ccqa/profiles/<name>.env` before resolving spec `${VAR}` references, so one spec targets dev/stg/prd. See [Profiles](#profiles---profile).
115
- - `--changed` — restrict execution to specs whose `relatedPaths` intersect `git diff <base>...HEAD`
116
- - `--concurrency <n>` — run up to N specs in parallel **within each mode** (deterministic phase, then live phase — parallelism is per-phase, not across phases). Default `1` (sequential, same behavior as before).
117
- - `--no-failure-analysis` / `--no-drift-audit` — `--no-failure-analysis` skips failure classification (and its drift audit, since drift is the classification's evidence); `--no-drift-audit` keeps classification but skips just the audit.
118
- - `--format <fmt>` — `text` (default), `json` (report.json), `github` (Actions annotations)
119
-
120
- `<feature/spec>` is a 2-segment alias for `.ccqa/features/<feature>/test-cases/<spec>/`; `ccqa run` takes several space-separated targets (a `<feature>/<spec>`, a bare `<feature>`, or none for everything). Claude-driven commands accept `-m/--model`, `--language`, and `--cwd` (for monorepos); they use your local Claude login, and CI commands (`ccqa run --report`, `ccqa drift`) also honor `ANTHROPIC_API_KEY`.
121
-
122
- ## File structure
123
-
124
- ```
125
- .ccqa/
126
- perspectives.yaml # Inventory of existing coverage (canonical)
127
- profiles/ # `--profile <name>` env files (stg.env, prd.env, ...)
128
- prompts/ # `ccqa init` scaffolds these; *.user.md human-maintained, *.agent.md auto-updated
129
- record.user.md record.agent.md live.user.md live.agent.md
130
- blocks/
131
- login/spec.yaml # Reusable block (params + steps)
132
- features/
133
- tasks/
134
- test-cases/
135
- create-and-complete/
136
- spec.yaml # Test definition, with `mode: deterministic | live`
137
- actions.json # (deterministic) recorded actions
138
- test.spec.ts # (deterministic) generated vitest script
139
- runs/<timestamp>/ # (live) one `ccqa run` — run.json/run.md + steps/*.png
140
- ```
141
-
142
- Gitignore the per-run artefacts: `.ccqa/features/*/test-cases/*/runs/` and `ccqa-report*/`.
143
-
144
- ## Profiles (`--profile`)
145
-
146
- Keep environment-specific values out of specs as `${VAR}` references and supply them per environment from a **profile** — a `.env` under `.ccqa/profiles/<name>.env`. `ccqa run`/`record --profile <name>` merges it into the environment before resolving `${VAR}`, so one spec runs anywhere.
114
+ - a dashboard of runs, with per-step screenshots
115
+ - a browsable inventory of what is tested
116
+ ([perspectives](./docs/spec.md#inventory-coverage-with-perspectives)),
117
+ kept fresh automatically by `record`/`generate`
118
+ - triage grading — mark each failure call right or wrong; the hub learns
119
+ from the grades
120
+ - one place for shared sessions, variables, and learned prompts — CI
121
+ needs a single secret
147
122
 
148
123
  ```bash
149
- # .ccqa/profiles/stg.env
150
- APP_BASE_URL=https://<your-app-host>
151
- TEST_USER_EMAIL=<stg-test-account>
152
- TEST_USER_PASSWORD=...
153
- ```
154
- ```bash
155
- ccqa run auth/login --profile stg # same spec, stg values
156
- ```
157
-
158
- - Format is a small `.env` subset (`KEY=value`, `#` comments, `export`, quotes); profile values **override** the inherited environment.
159
- - Name is free-form (`stg`/`prd` by convention); path separators, `..`, and leading dots are rejected, and an unknown name exits 2. Only the name is logged, never values.
160
- - Without `--profile`, ccqa auto-loads `<cwd>/.env` if present; with neither, `${VAR}` resolves against the existing `process.env`.
161
- - **Secrets:** gitignore any profile holding plaintext secrets. To keep secrets off disk entirely, drop `--profile` and run ccqa under your secret manager (e.g. `op run --env-file=... -- ccqa run ...`), which injects resolved values into `process.env`.
162
-
163
- ## Live specs (`mode: live`)
164
-
165
- For specs declared `mode: live` in their spec.yaml, `ccqa run` skips codegen entirely: Claude executes each spec step against `agent-browser` directly, judges whether the step's `expected` outcome holds, and saves a PNG screenshot before and after every step. Use this mode when:
166
-
167
- - you want to validate a spec but don't yet need a replayable, recorded test
168
- - the codegen output for a spec is fragile (heavily timing-dependent UIs, rich-text editors, dynamic selectors)
169
- - you want a visual audit trail of what the page looked like at every step
170
-
171
- Constraints on selectors / `agent-browser` subcommands that apply during `ccqa record` (no `eval`, no `@ref`, no bare-tag positional `find`, no chained agent-browser calls) are relaxed for live specs, since there is no replay contract to honour.
172
-
173
- See [Live specs](./docs/live.md) for usage examples and per-project guidance.
174
-
175
- ### Saved sessions (`session:`)
176
-
177
- Some providers gate every fresh browser with a device-trust check (an "unrecognized device" e-mail code, an MFA prompt) that a human has to clear by hand — impractical to repeat on every run, impossible in CI. `session:` restores a saved, signed-in browser state instead.
178
-
179
- ```yaml
180
- title: Admin can open the settings page
181
- mode: live
182
- session: admin # restore the saved "admin" session before step 1
183
- steps:
184
- - ... # no login steps — the spec starts signed-in
124
+ export CCQA_HUB_TOKEN=$(openssl rand -hex 24)
125
+ ccqa serve # or: docker compose up -d
126
+ ccqa run tasks/create-and-complete --push-report \
127
+ --hub-url http://localhost:8787 --hub-token $CCQA_HUB_TOKEN
185
128
  ```
186
129
 
187
- ccqa does not manage authentication — `session` is purely an optional restore of cookies + localStorage.
188
-
189
- See [Saved sessions](./docs/sessions.md) for how to bootstrap a session and use it in CI.
130
+ See [Hub](./docs/hub.md) for the full setup (encryption, container
131
+ deployment, HTTP API).
190
132
 
191
- ## CI result aggregation (hub)
133
+ ## Documentation
192
134
 
193
- `ccqa serve` starts a hub — a control-plane HTTP server that aggregates CI run results, saved sessions, and variables on a shared server. It does not execute tests: a CI job (or a laptop) runs `ccqa run --report` as usual, fetching whatever sessions/variables/prompts it needs from the hub directly at run time, then `ccqa hub push` uploads the resulting report to the hub. Sessions and variables are registered on the hub ahead of time (`ccqa hub session push`, `ccqa hub var set`, or the bundled UI's Secrets tab), so a CI job needs only one secret: `CCQA_HUB_TOKEN`. One hub manages many projects — secrets are scoped per project/profile, and `--project` defaults to the current directory's name. See [docs/hub.md](./docs/hub.md) for setup and usage.
135
+ | I want to… | Read |
136
+ |---|---|
137
+ | Write specs: fields, reusable blocks, file uploads, coverage inventory | [spec.yaml reference](./docs/spec.md) |
138
+ | Draft specs interactively with Claude | [Draft](./docs/draft.md) |
139
+ | Generate Playwright or runn tests that reuse my existing test code | [Generation targets](./docs/targets.md) |
140
+ | Run specs, read reports, triage failures, detect drift, wire up CI | [Running specs](./docs/running.md) |
141
+ | Run specs live (no codegen), with per-project guidance | [Live specs](./docs/live.md) |
142
+ | Start runs already signed in / skip device-trust gates | [Saved sessions](./docs/sessions.md) |
143
+ | See which assertions generated tests use | [Assertions](./docs/assertions.md) |
144
+ | Auto-fix failing recorded tests | [Auto-fix](./docs/auto-fix.md) |
145
+ | Aggregate results, sessions, and variables on a team server | [Hub](./docs/hub.md) |
146
+ | Script the hub over HTTP | [Hub API](./docs/hub-api.md) |
147
+ | Understand why ccqa is built this way | [ADR](./docs/adr/README.md) |
194
148
 
195
149
  ## License
196
150