ccqa 1.14.0 → 1.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +117 -234
- package/dist/bin/ccqa.mjs +81 -94
- package/dist/hub-client/index.d.mts +6 -1
- package/dist/package.json +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,40 +1,26 @@
|
|
|
1
1
|
# ccqa
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
> [!WARNING]
|
|
4
|
+
> ccqa is under active development. Expect breaking changes.
|
|
4
5
|
|
|
5
|
-
|
|
6
|
-
test spec in YAML; Claude drives a real browser **once** to discover the
|
|
7
|
-
route; ccqa compiles that recording into ordinary test code your CI replays
|
|
8
|
-
with no model in the loop.
|
|
6
|
+
**Your Claude subscription already includes a QA engineer.**
|
|
9
7
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
replays
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
credential in CI.
|
|
8
|
+
Write a test spec in YAML. Claude drives a real browser **once** to
|
|
9
|
+
discover the route, and ccqa compiles the recording into plain test code
|
|
10
|
+
your CI replays — no model in the loop, no API key. Claude returns only
|
|
11
|
+
where it pays: auditing specs against the code, explaining failures, and
|
|
12
|
+
driving `mode: live` specs.
|
|
16
13
|
|
|
17
14
|
[日本語版 README](./docs/README.ja.md)
|
|
18
15
|
|
|
19
|
-
##
|
|
16
|
+
## Quick start
|
|
20
17
|
|
|
21
18
|
```bash
|
|
22
|
-
pnpm add -D ccqa vitest agent-browser
|
|
19
|
+
pnpm add -D ccqa vitest agent-browser # Node 20+
|
|
23
20
|
```
|
|
24
21
|
|
|
25
|
-
|
|
26
|
-
[
|
|
27
|
-
[vitest](https://vitest.dev) are peer dependencies of the **default
|
|
28
|
-
agent-browser target** — they run its recorded tests. A project that only uses
|
|
29
|
-
an external target (`playwright`, `runn`) needs just `ccqa` plus that tool
|
|
30
|
-
(e.g. `pnpm add -D ccqa @playwright/test`); ccqa executes it through the
|
|
31
|
-
target's `runCommand`.
|
|
32
|
-
|
|
33
|
-
## Quick start
|
|
34
|
-
|
|
35
|
-
**1. Write a spec** — by hand, or interactively with
|
|
36
|
-
[`ccqa draft`](./docs/draft.md). (`ccqa init` scaffolds the `.ccqa/`
|
|
37
|
-
skeleton.)
|
|
22
|
+
Write a spec — `ccqa init` scaffolds the tree,
|
|
23
|
+
[`ccqa draft`](./docs/draft.md) writes one with you:
|
|
38
24
|
|
|
39
25
|
```yaml
|
|
40
26
|
# .ccqa/features/tasks/test-cases/create-and-complete/spec.yaml
|
|
@@ -50,264 +36,161 @@ steps:
|
|
|
50
36
|
expected: Task appears in the task list with status "Open"
|
|
51
37
|
```
|
|
52
38
|
|
|
53
|
-
|
|
54
|
-
embedding an environment, so the same spec runs against local and staging. A
|
|
55
|
-
`.env` file covers you locally; in CI the values come from a hub
|
|
56
|
-
(`ccqa hub var set`) so nothing environment-specific lives in the repo. See
|
|
57
|
-
[Profiles and environment variables](./docs/running.md#profiles-and-environment-variables).
|
|
58
|
-
|
|
59
|
-
```bash
|
|
60
|
-
echo 'APP_URL=http://localhost:3000' >> .env
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
**3. Record once** — Claude drives the browser and generates the test:
|
|
64
|
-
|
|
65
|
-
```bash
|
|
66
|
-
ccqa record tasks/create-and-complete
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
**4. Run it** — vitest replays the recording; no LLM involved:
|
|
39
|
+
Record once, replay forever:
|
|
70
40
|
|
|
71
41
|
```bash
|
|
72
|
-
|
|
42
|
+
echo 'APP_URL=http://localhost:3000' >> .env # ${VAR}s stay out of specs
|
|
43
|
+
ccqa record tasks/create-and-complete # Claude drives the browser
|
|
44
|
+
ccqa run tasks/create-and-complete # vitest replays — no LLM
|
|
73
45
|
```
|
|
74
46
|
|
|
75
|
-
|
|
76
|
-
See [Running specs](./docs/running.md) for flags and the report format.
|
|
47
|
+
Every run writes `report.json` and step screenshots to `ccqa-report/`.
|
|
77
48
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
[`ccqa hub session capture`](./docs/sessions.md) and
|
|
49
|
+
Some logins cannot be replayed from a recording — an SSO redirect, a
|
|
50
|
+
device-trust prompt. Sign in by hand once with
|
|
51
|
+
[`ccqa hub session capture`](./docs/sessions.md), and specs start from
|
|
52
|
+
that saved session.
|
|
81
53
|
|
|
82
54
|
## How it works
|
|
83
55
|
|
|
84
56
|
```
|
|
85
|
-
spec.yaml ──► ccqa record
|
|
86
|
-
steps + Claude drives
|
|
87
|
-
expected the browser
|
|
88
|
-
results discovers the tool-neutral (reuse-first) / runn
|
|
89
|
-
route IR
|
|
90
|
-
|
|
91
|
-
test code ──► ccqa run ────────► report.json ─► ccqa hub push /
|
|
92
|
-
vitest replay / + evidence --report-to-hub
|
|
93
|
-
runCommand / + artifacts team dashboard,
|
|
94
|
-
live (Claude failure triage,
|
|
95
|
-
drives per step) grading & learning
|
|
57
|
+
spec.yaml ──► ccqa record ──► ir.json ──► test code ──► ccqa run
|
|
58
|
+
steps + Claude drives recorded per-target replayed in CI,
|
|
59
|
+
expected the browser actions emit no LLM
|
|
96
60
|
```
|
|
97
61
|
|
|
98
62
|
A spec runs in one of two ways:
|
|
99
63
|
|
|
100
|
-
**Deterministic (the default).**
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
stable. The `target:` field picks only **what the recording compiles
|
|
104
|
-
into**; every target is the same deterministic replay:
|
|
64
|
+
**Deterministic (the default).** The recording compiles into plain test
|
|
65
|
+
code and CI replays it with no model in the loop. `target:` picks only
|
|
66
|
+
what it compiles into:
|
|
105
67
|
|
|
106
68
|
| `target:` | Generated file | Replayed by |
|
|
107
69
|
|---|---|---|
|
|
108
|
-
| `agent-browser` (default) | `test.spec.ts` (vitest
|
|
109
|
-
| `playwright` |
|
|
110
|
-
| `runn` | `runbook.yaml` (API scenario
|
|
111
|
-
|
|
112
|
-
`runCommand` is the one-line command your repo already uses to run that
|
|
113
|
-
tool, declared once in `.ccqa/config.yaml` — e.g.
|
|
114
|
-
`pnpm exec playwright test {files}`. See
|
|
115
|
-
[Generation targets](./docs/targets.md) for the substitution contract.
|
|
70
|
+
| `agent-browser` (default) | `test.spec.ts` (vitest) | vitest |
|
|
71
|
+
| `playwright` | plain `@playwright/test` spec | your `runCommand` |
|
|
72
|
+
| `runn` | `runbook.yaml` (API scenario, no recording) | your `runCommand` |
|
|
116
73
|
|
|
117
74
|
**Live (`mode: live`).** No codegen: Claude drives every run and judges
|
|
118
|
-
each step's `expected` — for
|
|
119
|
-
recording would break.
|
|
120
|
-
|
|
121
|
-
## Failure analysis and drift
|
|
75
|
+
each step's `expected` — for UIs a fixed recording would break on.
|
|
122
76
|
|
|
123
|
-
|
|
124
|
-
|
|
77
|
+
vitest and agent-browser are peer dependencies of the default target; a
|
|
78
|
+
project on an external target alone needs just `ccqa` and that tool.
|
|
79
|
+
`runCommand` and reusing your existing page objects:
|
|
80
|
+
[Generation targets](./docs/targets.md).
|
|
125
81
|
|
|
126
|
-
|
|
127
|
-
— `TEST_DRIFT`, `SPEC_CHANGE`, `PRODUCT_BUG`, or `UNKNOWN` when the evidence
|
|
128
|
-
does not support a call. The label comes with a drift audit of the same spec,
|
|
129
|
-
because "did the test break" and "does the test still describe the product"
|
|
130
|
-
are the same investigation. `[base]` is what the diff is read against: a git
|
|
131
|
-
ref, or `last-green` to have each spec diff against the commit where it last
|
|
132
|
-
passed. With neither, the label rests on the failure alone and says so.
|
|
82
|
+
## Audit, then run
|
|
133
83
|
|
|
134
|
-
**
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
decides the repair, so the audit reports it: stale generated code is
|
|
139
|
-
re-recorded, a stale spec needs a human.
|
|
84
|
+
**A spec describes the code your verification environment is running** —
|
|
85
|
+
not your branch, not the tip of main. A deploy moves that code, and some
|
|
86
|
+
specs stop describing it. Those specs are not failing; they say nothing
|
|
87
|
+
true about what runs, so executing them proves nothing.
|
|
140
88
|
|
|
141
|
-
|
|
142
|
-
[Failure triage](./docs/running.md#failure-triage) and
|
|
143
|
-
[Drift detection](./docs/running.md#drift-detection).
|
|
144
|
-
|
|
145
|
-
## The hub
|
|
89
|
+
So ccqa asks the cheap question before the expensive one:
|
|
146
90
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
91
|
+
```
|
|
92
|
+
the code the verification environment is running
|
|
93
|
+
│
|
|
94
|
+
│ a spec describes this
|
|
95
|
+
▼
|
|
96
|
+
the deployed commit changes
|
|
97
|
+
│
|
|
98
|
+
▼
|
|
99
|
+
audit the specs that change reaches
|
|
100
|
+
│
|
|
101
|
+
still describes it ───┴─── no longer describes it
|
|
102
|
+
│ │
|
|
103
|
+
▼ ▼
|
|
104
|
+
run it repair the spec
|
|
105
|
+
│
|
|
106
|
+
re-audited next round;
|
|
107
|
+
unverified until then
|
|
163
108
|
```
|
|
164
109
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
See [Hub](./docs/hub.md) for the full setup and
|
|
171
|
-
[Hub API](./docs/hub-api.md) to script it over HTTP.
|
|
172
|
-
|
|
173
|
-
## Wire it into CI
|
|
174
|
-
|
|
175
|
-
Three jobs. They are independent: the pull-request job on its own is a
|
|
176
|
-
complete adoption, and the other two can come later.
|
|
177
|
-
|
|
178
|
-
| Job | Trigger | Question it answers |
|
|
179
|
-
|---|---|---|
|
|
180
|
-
| Pre-merge run | `pull_request` | Does this change break a spec, and whose fault is it? |
|
|
181
|
-
| Post-deploy run | after a deploy | Which specs' last result is no longer trustworthy? |
|
|
182
|
-
| Drift audit | `schedule` | Do the specs still describe the code? |
|
|
183
|
-
|
|
184
|
-
All three need two things:
|
|
110
|
+
`ccqa audit` reads each spec against the source — cents per spec, no
|
|
111
|
+
browser — and records every verdict on the **hub**, the small server
|
|
112
|
+
that holds what the team and CI share. Stale generated code is
|
|
113
|
+
re-recorded; a stale spec goes to a human and stays **unverified** —
|
|
114
|
+
neither passing nor failing — until repaired.
|
|
185
115
|
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
`CCQA_HUB_TOKEN`. Only a pre-merge run with no `--hub-profile` and no
|
|
190
|
-
`--report-to-hub` can do without one.
|
|
116
|
+
`ccqa run --only-hub-rerun-needed` asks the hub which specs are worth
|
|
117
|
+
running: cleared by the audit *and* invalidated by a deploy. A drifted
|
|
118
|
+
spec answers `blocked` and is never run — a run cannot repair a spec.
|
|
191
119
|
|
|
192
|
-
|
|
193
|
-
|
|
120
|
+
When a clean spec still fails, `--on-fail-explain` labels whose problem
|
|
121
|
+
it is: `TEST_DRIFT`, `SPEC_CHANGE`, `PRODUCT_BUG`, or `UNKNOWN`. You
|
|
122
|
+
grade the calls on the hub, and it learns from your grades.
|
|
194
123
|
|
|
195
|
-
|
|
196
|
-
saved sessions on the hub, and — since two environments sit at different
|
|
197
|
-
commits — its own deploy history. Register the variables your specs reference
|
|
198
|
-
once, from your machine:
|
|
124
|
+
## In CI
|
|
199
125
|
|
|
200
|
-
```bash
|
|
201
|
-
ccqa hub var set APP_URL --value https://app.example --profile staging
|
|
202
126
|
```
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
Run the specs the change reaches, and label what broke.
|
|
210
|
-
|
|
211
|
-
```bash
|
|
212
|
-
ccqa run --only-affected-by --on-fail-explain --hub-profile staging \
|
|
213
|
-
--report-format github --report-to-hub
|
|
127
|
+
deploy lands
|
|
128
|
+
├─ ccqa hub deploy record --select what shipped, which specs it reaches
|
|
129
|
+
├─ ccqa audit --report-to-hub does each spec still describe it?
|
|
130
|
+
└─ ccqa run --only-hub-rerun-needed --on-fail-explain \
|
|
131
|
+
--hub-profile ci --report-to-hub
|
|
214
132
|
```
|
|
215
133
|
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
from the hub. Without it, a spec's `${…}` references go unresolved.
|
|
221
|
-
- `--report-format github` annotates the pull request.
|
|
222
|
-
- `--report-to-hub` streams results to the hub as the run executes.
|
|
223
|
-
|
|
224
|
-
**Set `fetch-depth: 0` on `actions/checkout`.** Both selection flags read
|
|
225
|
-
their baseline from `GITHUB_BASE_REF` and resolve it against `origin/<base>`,
|
|
226
|
-
which a shallow checkout does not have. Without it the run exits with a usage
|
|
227
|
-
error before the first test. Outside a `pull_request` workflow there is no
|
|
228
|
-
`GITHUB_BASE_REF`, so pass the base yourself: `--only-affected-by origin/main`.
|
|
134
|
+
The audit costs cents; a live spec costs dollars. Filtering first leaves
|
|
135
|
+
a run whose failures are worth reading. Record every deploy with
|
|
136
|
+
`--select` — a range recorded without it answers `unknown` forever, and
|
|
137
|
+
nothing fills the hole later.
|
|
229
138
|
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
139
|
+
| Job | Trigger | Question it answers |
|
|
140
|
+
|---|---|---|
|
|
141
|
+
| Deploy loop | after a deploy | Which specs did this deploy invalidate? |
|
|
142
|
+
| Pre-merge run | `pull_request` | Does this change break a spec, and whose fault is it? |
|
|
143
|
+
| Full audit | `schedule` | Do all the specs still describe the code? |
|
|
234
144
|
|
|
235
|
-
|
|
236
|
-
shipped:
|
|
145
|
+
The two jobs outside the loop:
|
|
237
146
|
|
|
238
147
|
```bash
|
|
239
|
-
|
|
240
|
-
|
|
148
|
+
# pull request — run what the diff reaches, label what broke
|
|
149
|
+
# (checkout with fetch-depth: 0, or the base ref is not there to resolve)
|
|
150
|
+
ccqa run --only-affected-by "origin/$GITHUB_BASE_REF" --on-fail-explain \
|
|
151
|
+
--hub-profile ci --report-format github --report-to-hub
|
|
241
152
|
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
```bash
|
|
245
|
-
ccqa run --only-hub-stale --hub-profile staging --report-to-hub
|
|
153
|
+
# schedule — audit everything; no browser, no deploy
|
|
154
|
+
ccqa audit --report-format github --report-to-hub
|
|
246
155
|
```
|
|
247
156
|
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
- `--only-hub-stale` asks the hub, per spec, whether any deploy has touched
|
|
251
|
-
it since that spec last ran.
|
|
252
|
-
|
|
253
|
-
The hub has no checkout and never runs `git`, so it cannot work out what a
|
|
254
|
-
deploy changed. That is why the selection is submitted with the deploy rather
|
|
255
|
-
than reconstructed later — and why a deploy recorded without `--select` leaves
|
|
256
|
-
a hole nothing can fill in afterwards.
|
|
257
|
-
|
|
258
|
-
**Expect it to select nothing at first.** A spec with no recorded run is
|
|
259
|
-
`neverRun`; one whose baseline predates the deploy log is `unknown`. Neither
|
|
260
|
-
runs by default. Record a deploy, run every spec once with `--report-to-hub`,
|
|
261
|
-
and the selection means something from the next deploy on. This job also reads
|
|
262
|
-
the spec inventory from the hub, so `ccqa perspectives` has to have run.
|
|
263
|
-
`--only-hub-stale-with-unknown` opts the undecided specs in.
|
|
157
|
+
Runnable workflows and every flag:
|
|
158
|
+
[CI integration](./docs/running.md#ci-integration).
|
|
264
159
|
|
|
265
|
-
|
|
160
|
+
## The hub
|
|
266
161
|
|
|
267
|
-
|
|
162
|
+
You have met the hub twice now: the audit writes its verdicts there,
|
|
163
|
+
and the run asks it what is worth running. The same server holds the
|
|
164
|
+
rest of what a team shares: the variables and sessions `${…}` resolves
|
|
165
|
+
to (CI keeps one secret), the deploy log behind the selection flags,
|
|
166
|
+
run reports with screenshots, and the prompts learned from your triage
|
|
167
|
+
grades.
|
|
268
168
|
|
|
269
169
|
```bash
|
|
270
|
-
|
|
170
|
+
export CCQA_HUB_TOKEN=$(openssl rand -hex 24)
|
|
171
|
+
export CCQA_HUB_ENCRYPTION_KEY=$(openssl rand -hex 32)
|
|
172
|
+
ccqa serve
|
|
271
173
|
```
|
|
272
174
|
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
The pre-merge job already audits the specs that failed. This one covers the
|
|
281
|
-
rest, because a spec can pass and still describe a product that no longer
|
|
282
|
-
exists.
|
|
283
|
-
|
|
284
|
-
### Workflows
|
|
285
|
-
|
|
286
|
-
[CI integration](./docs/running.md#ci-integration) has runnable workflows for
|
|
287
|
-
the pre-merge run and the scheduled audit.
|
|
288
|
-
[`ccqa hub deploy record`](./docs/hub.md#ccqa-hub-deploy-record) covers the
|
|
289
|
-
deploy job, including a `curl`-only variant for pipelines with no Node.
|
|
175
|
+
Anything that needs the hub names it — `--hub-profile`,
|
|
176
|
+
`--only-hub-rerun-needed`, `--report-to-hub` — and fails rather than
|
|
177
|
+
degrade when it cannot reach one. A **profile** is a named value set — a
|
|
178
|
+
tenant, an account, a role — not an environment: ccqa tracks one
|
|
179
|
+
verification environment
|
|
180
|
+
([ADR-0013](./docs/adr/0013-one-verification-environment.md)).
|
|
290
181
|
|
|
291
182
|
## Documentation
|
|
292
183
|
|
|
293
184
|
| I want to… | Read |
|
|
294
185
|
|---|---|
|
|
295
|
-
|
|
|
296
|
-
|
|
|
297
|
-
|
|
|
298
|
-
|
|
|
299
|
-
|
|
|
300
|
-
|
|
|
301
|
-
|
|
|
302
|
-
|
|
|
303
|
-
| Wire ccqa into GitHub Actions | [CI integration](./docs/running.md#ci-integration) |
|
|
304
|
-
| Run specs live (no codegen), with per-project guidance | [Live specs](./docs/live.md) |
|
|
305
|
-
| Start runs already signed in / skip device-trust gates | [Saved sessions](./docs/sessions.md) |
|
|
306
|
-
| See which assertions generated tests use | [Assertions](./docs/assertions.md) |
|
|
307
|
-
| Auto-fix failing recorded tests | [Auto-fix](./docs/auto-fix.md) |
|
|
308
|
-
| Aggregate results, sessions, and variables on a team server | [Hub](./docs/hub.md) |
|
|
309
|
-
| Script the hub over HTTP | [Hub API](./docs/hub-api.md) |
|
|
310
|
-
| Understand why ccqa is built this way | [ADR](./docs/adr/README.md) |
|
|
186
|
+
| Write specs — fields, blocks, file uploads | [spec.yaml](./docs/spec.md) |
|
|
187
|
+
| Run specs and read the report | [Running](./docs/running.md) |
|
|
188
|
+
| Wire it into GitHub Actions | [CI integration](./docs/running.md#ci-integration) |
|
|
189
|
+
| Emit Playwright / runn tests | [Targets](./docs/targets.md) |
|
|
190
|
+
| Drive specs live, with per-project guidance | [Live specs](./docs/live.md) |
|
|
191
|
+
| Sign in once and reuse the session | [Sessions](./docs/sessions.md) |
|
|
192
|
+
| Run the team hub / script it over HTTP | [Hub](./docs/hub.md) · [API](./docs/hub-api.md) |
|
|
193
|
+
| Understand why it is built this way | [ADR](./docs/adr/README.md) |
|
|
311
194
|
|
|
312
195
|
## License
|
|
313
196
|
|
package/dist/bin/ccqa.mjs
CHANGED
|
@@ -5013,7 +5013,7 @@ function addLanguageOption(command) {
|
|
|
5013
5013
|
* `record`), registered identically so help text and behaviour don't drift.
|
|
5014
5014
|
*/
|
|
5015
5015
|
function addProfileOption(command) {
|
|
5016
|
-
return command.option("--hub-profile <name>", "Load this profile's variables from the hub into the environment before resolving spec ${VAR} references (URLs, credentials), so one spec can
|
|
5016
|
+
return command.option("--hub-profile <name>", "Load this profile's variables from the hub into the environment before resolving spec ${VAR} references (URLs, credentials), so one spec can run as a different tenant, account or role without a copy per set of values. A profile is a value set, not an environment — ccqa tracks one verification environment. Profile values override the inherited environment. Requires --hub-url/--hub-token (or CCQA_HUB_URL/CCQA_HUB_TOKEN).");
|
|
5017
5017
|
}
|
|
5018
5018
|
/**
|
|
5019
5019
|
* Shared `--hub-url` / `--hub-token` flags for commands that optionally talk
|
|
@@ -6702,7 +6702,7 @@ async function tryDeployHeadSha(hubCtx, profile) {
|
|
|
6702
6702
|
*
|
|
6703
6703
|
* This exists because a selection can be wrong in a way that costs money.
|
|
6704
6704
|
* `ccqa select-specs`'s model judgment is not infallible, and both
|
|
6705
|
-
* `--only-affected-by` and `--only-hub-
|
|
6705
|
+
* `--only-affected-by` and `--only-hub-rerun-needed` decide from it, so a human has to
|
|
6706
6706
|
* be able to read the selection back before a live spec spends a Claude
|
|
6707
6707
|
* budget on it.
|
|
6708
6708
|
*
|
|
@@ -6734,53 +6734,9 @@ function formatDryRunLines(agentBrowser, routed) {
|
|
|
6734
6734
|
return tagged.map((t) => ` ${t.key.padEnd(width)} ${t.tag}`);
|
|
6735
6735
|
}
|
|
6736
6736
|
//#endregion
|
|
6737
|
-
//#region src/run/audited-clean.ts
|
|
6738
|
-
/**
|
|
6739
|
-
* Fetch the drift ledger and reduce it to the specs that are safe to run.
|
|
6740
|
-
*
|
|
6741
|
-
* A spec qualifies only when the ledger holds an entry for it *and* that entry
|
|
6742
|
-
* found no drift. A spec that has never been audited does not qualify: the
|
|
6743
|
-
* point of the flag is to spend a run only where a cheap audit already said
|
|
6744
|
-
* the spec still describes the code, and "never looked" is not that.
|
|
6745
|
-
*/
|
|
6746
|
-
async function fetchAuditedLedger(hubCtx) {
|
|
6747
|
-
let ledger;
|
|
6748
|
-
try {
|
|
6749
|
-
ledger = await hubCtx.hub.getDriftLedger(hubCtx.project);
|
|
6750
|
-
} catch (err) {
|
|
6751
|
-
throw new RunUsageError(`--only-hub-audited-clean: could not fetch the drift ledger from the hub: ${errMessage(err)}`);
|
|
6752
|
-
}
|
|
6753
|
-
const clean = /* @__PURE__ */ new Set();
|
|
6754
|
-
const audited = /* @__PURE__ */ new Set();
|
|
6755
|
-
for (const [key, entry] of Object.entries(ledger.specs)) {
|
|
6756
|
-
audited.add(key);
|
|
6757
|
-
if (entry.label === null) clean.add(key);
|
|
6758
|
-
}
|
|
6759
|
-
return {
|
|
6760
|
-
clean,
|
|
6761
|
-
audited
|
|
6762
|
-
};
|
|
6763
|
-
}
|
|
6764
|
-
function selectAuditedClean(specs, ledger) {
|
|
6765
|
-
const selected = [];
|
|
6766
|
-
let unaudited = 0;
|
|
6767
|
-
let drifted = 0;
|
|
6768
|
-
for (const spec of specs) {
|
|
6769
|
-
const key = specKey(spec);
|
|
6770
|
-
if (ledger.clean.has(key)) selected.push(spec);
|
|
6771
|
-
else if (ledger.audited.has(key)) drifted++;
|
|
6772
|
-
else unaudited++;
|
|
6773
|
-
}
|
|
6774
|
-
return {
|
|
6775
|
-
selected,
|
|
6776
|
-
unaudited,
|
|
6777
|
-
drifted
|
|
6778
|
-
};
|
|
6779
|
-
}
|
|
6780
|
-
//#endregion
|
|
6781
6737
|
//#region src/run/rerun-selection.ts
|
|
6782
6738
|
/**
|
|
6783
|
-
* `ccqa run --only-hub-
|
|
6739
|
+
* `ccqa run --only-hub-rerun-needed`: select specs from the hub's re-run verdicts
|
|
6784
6740
|
* instead of from a git diff (ADR-0010). The baseline is not a ref at all —
|
|
6785
6741
|
* it is each spec's own last run, positioned against the deploy log the
|
|
6786
6742
|
* consuming deploy job feeds the hub — so this path does no git work.
|
|
@@ -6792,12 +6748,12 @@ function selectAuditedClean(specs, ledger) {
|
|
|
6792
6748
|
/** First ccqa release whose hub serves `GET /projects/:project/rerun`. */
|
|
6793
6749
|
const RERUN_MIN_HUB_VERSION = "1.9";
|
|
6794
6750
|
/**
|
|
6795
|
-
* The profile `--only-hub-
|
|
6751
|
+
* The profile `--only-hub-rerun-needed` asks about. Mandatory: two environments sit
|
|
6796
6752
|
* at different commits and the deploy log is per-profile, so "needs re-run"
|
|
6797
6753
|
* has no profile-free answer.
|
|
6798
6754
|
*/
|
|
6799
6755
|
function requireRerunProfile(profile) {
|
|
6800
|
-
if (profile === void 0) throw new RunUsageError("--only-hub-
|
|
6756
|
+
if (profile === void 0) throw new RunUsageError("--only-hub-rerun-needed requires --hub-profile <name>: the deploy log it reads is per-profile, so which specs need a re-run has no answer without one");
|
|
6801
6757
|
return profile;
|
|
6802
6758
|
}
|
|
6803
6759
|
/**
|
|
@@ -6810,9 +6766,9 @@ async function fetchRerunReport(hubCtx, profile) {
|
|
|
6810
6766
|
report = await hubCtx.hub.getRerun(hubCtx.project, { profile });
|
|
6811
6767
|
} catch (err) {
|
|
6812
6768
|
if (err instanceof HubApiError && err.status === 404) throw new RunUsageError(explainNotFound(hubCtx, err));
|
|
6813
|
-
throw new RunUsageError(`--only-hub-
|
|
6769
|
+
throw new RunUsageError(`--only-hub-rerun-needed: could not ask the hub which specs need a re-run: ${errMessage(err)}`);
|
|
6814
6770
|
}
|
|
6815
|
-
if (report.deployHead === null) throw new RunUsageError(`--only-hub-
|
|
6771
|
+
if (report.deployHead === null) throw new RunUsageError(`--only-hub-rerun-needed: no deploy has been recorded for profile "${profile}" of project "${hubCtx.project}", so nothing can be compared against. Wire \`ccqa hub deploy record\` into the deploy job, or select with --only-affected-by <ref> instead.`);
|
|
6816
6772
|
return {
|
|
6817
6773
|
...report,
|
|
6818
6774
|
deployHead: report.deployHead
|
|
@@ -6824,11 +6780,12 @@ async function fetchRerunReport(hubCtx, profile) {
|
|
|
6824
6780
|
* means the hub does not serve this route at all.
|
|
6825
6781
|
*/
|
|
6826
6782
|
function explainNotFound(hubCtx, err) {
|
|
6827
|
-
if (err.code === "no_perspectives") return `--only-hub-
|
|
6828
|
-
return `--only-hub-
|
|
6783
|
+
if (err.code === "no_perspectives") return `--only-hub-rerun-needed: project "${hubCtx.project}" has no perspectives document on the hub, so no spec is registered to compare against a deploy. Run \`ccqa perspectives\` first.`;
|
|
6784
|
+
return `--only-hub-rerun-needed: this hub does not serve re-run verdicts — it needs ccqa ${RERUN_MIN_HUB_VERSION} or newer. Upgrade the hub, or select with --only-affected-by <ref> instead.`;
|
|
6829
6785
|
}
|
|
6830
6786
|
/** States the summary line reports, worst-known-first. */
|
|
6831
6787
|
const SUMMARY_ORDER = [
|
|
6788
|
+
"blocked",
|
|
6832
6789
|
"needed",
|
|
6833
6790
|
"unknown",
|
|
6834
6791
|
"neverRun",
|
|
@@ -6846,7 +6803,7 @@ const UNANSWERABLE = new Set([
|
|
|
6846
6803
|
*
|
|
6847
6804
|
* `needed` is always selected. `unknown` and `neverRun` are "the question
|
|
6848
6805
|
* cannot be answered", so they are excluded by default and opted into with
|
|
6849
|
-
* `--only-hub-
|
|
6806
|
+
* `--only-hub-rerun-needed-with-unknown` — fail-open on request, never silently. `notNeeded` and
|
|
6850
6807
|
* `notEvaluated` are never selected.
|
|
6851
6808
|
*/
|
|
6852
6809
|
function selectSpecsNeedingRerun(specs, report, opts) {
|
|
@@ -8029,11 +7986,19 @@ z.record(z.string(), SpecTouchSchema);
|
|
|
8029
7986
|
const RerunStateSchema = z.enum([
|
|
8030
7987
|
"needed",
|
|
8031
7988
|
"notNeeded",
|
|
7989
|
+
"blocked",
|
|
8032
7990
|
"unknown",
|
|
8033
7991
|
"neverRun",
|
|
8034
7992
|
"notEvaluated"
|
|
8035
7993
|
]);
|
|
8036
7994
|
/**
|
|
7995
|
+
* Why a spec is `blocked`. Always carried: the two answers differ in who
|
|
7996
|
+
* repairs them and how long that takes — a stale recording is re-recorded
|
|
7997
|
+
* automatically within minutes, a changed spec waits for a human — so a view
|
|
7998
|
+
* that showed only "blocked" would hide the distinction that matters most.
|
|
7999
|
+
*/
|
|
8000
|
+
const RerunBlockedReasonSchema = z.enum(["testDrift", "specChange"]);
|
|
8001
|
+
/**
|
|
8037
8002
|
* Why a spec is `unknown`. Always carried, so the view can name the missing
|
|
8038
8003
|
* input ("no deploy log for this profile") instead of shrugging. `unknown` is
|
|
8039
8004
|
* never rendered as "not needed".
|
|
@@ -8062,6 +8027,7 @@ const DeployRefSchema = z.object({
|
|
|
8062
8027
|
const SpecRerunSchema = z.object({
|
|
8063
8028
|
state: RerunStateSchema,
|
|
8064
8029
|
reason: RerunUnknownReasonSchema.optional(),
|
|
8030
|
+
blockedReason: RerunBlockedReasonSchema.optional(),
|
|
8065
8031
|
lastRun: SpecLedgerEntrySchema.nullable(),
|
|
8066
8032
|
lastGreen: SpecLedgerEntrySchema.nullable(),
|
|
8067
8033
|
lastRed: SpecLedgerEntrySchema.nullable(),
|
|
@@ -8739,7 +8705,7 @@ const promptRm = new Command("rm").description("Delete a prompt from the hub.").
|
|
|
8739
8705
|
info(`deleted prompt "${name}" from the hub`);
|
|
8740
8706
|
}));
|
|
8741
8707
|
const promptCommand = new Command("prompt").description("Manage prompt assets (per-flow user/agent guidance, triage user guidance, analysis custom prompt) stored on the hub (fetched automatically by `ccqa run` at run time).").addCommand(promptPush).addCommand(promptLs).addCommand(promptRm);
|
|
8742
|
-
const deployRecord = new Command("record").description("Tell the hub what a deploy shipped, so it can answer which specs need a re-run (`ccqa run --only-hub-
|
|
8708
|
+
const deployRecord = new Command("record").description("Tell the hub what a deploy shipped, so it can answer which specs need a re-run (`ccqa run --only-hub-rerun-needed`). Run this from the deploy job, after the deploy succeeds. The changed paths are computed locally with a two-dot `git diff <previous> <sha>`; a job that has only curl and git can POST the same body directly (see docs/hub.md).").requiredOption("--profile <name>", "Environment this deploy landed in (e.g. 'stg'). Required: dev and stg sit at different commits, so the deploy log is per-profile.").requiredOption("--sha <sha>", "Commit that was deployed.").option("--previous <sha>", "Commit this deploy replaced. Defaults to the profile's current deploy-log head on the hub. With neither, there's nothing to diff against: changedPaths is unset and --select is skipped.").option("--ref <ref>", "Ref that was deployed (branch or tag). Recorded for display only.").option("--select", "Also decide which specs this deploy reaches (`ccqa select-specs`) and send the verdict with it. Without it the deploy is a hole in the range: specs behind it report 'unknown' rather than 'not needed'.").option("-m, --model <name>", "Model for --select. Claude alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option(...hubUrlOption).option(...hubTokenOption).option("--project <name>", "Project whose deploy log this entry joins. Defaults to the current directory's name.").option("--cwd <path>", "Directory the git diff and the default --project name are resolved against.").action(withHubErrors(async (opts) => {
|
|
8743
8709
|
const cwd = resolveCwd(opts.cwd);
|
|
8744
8710
|
const project = resolveProject(opts);
|
|
8745
8711
|
const hub = connect(opts);
|
|
@@ -8818,7 +8784,7 @@ function describeSelection(selection, diffAvailable) {
|
|
|
8818
8784
|
const values = Object.values(selection);
|
|
8819
8785
|
return `${values.filter((s) => s.verdict === "needed").length} needed / ${values.filter((s) => s.verdict === "unknown").length} unknown / ${values.length} specs`;
|
|
8820
8786
|
}
|
|
8821
|
-
const deployCommand = new Command("deploy").description("Report deploys to the hub, the input behind `ccqa run --only-hub-
|
|
8787
|
+
const deployCommand = new Command("deploy").description("Report deploys to the hub, the input behind `ccqa run --only-hub-rerun-needed`.").addCommand(deployRecord);
|
|
8822
8788
|
const pushCommand = new Command("push").description("Upload the report directory of a finished `ccqa run --report` to the hub as a run. Run this after `ccqa run` (use `if: always()` in CI so failing runs are pushed too).").option("--report-dir <dir>", `Report directory to push. Default: ${DEFAULT_REPORT_DIR}/`).option("--project <name>", "Logical project name for the run. Defaults to the current directory's name.").option("--branch <name>", "Branch label. Defaults to $GITHUB_HEAD_REF / $GITHUB_REF_NAME / current git branch.").option("--profile <name>", "Profile (environment) the run executed against. Recorded for display; runs are not scoped by profile.").option(...hubUrlOption).option(...hubTokenOption).option("--cwd <path>", "Directory the report dir is resolved against (defaults to the current directory).").action(withHubErrors(async (opts) => {
|
|
8823
8789
|
const cwd = resolveCwd(opts.cwd);
|
|
8824
8790
|
const reportDir = join(cwd, opts.reportDir ?? "ccqa-report");
|
|
@@ -12189,10 +12155,10 @@ function dedupeSpecs(specs) {
|
|
|
12189
12155
|
* maps it to `process.exit(2)`).
|
|
12190
12156
|
*/
|
|
12191
12157
|
async function executeRun(targets, opts) {
|
|
12192
|
-
const filtering = Boolean(opts.onlyAffectedBy || opts.
|
|
12158
|
+
const filtering = Boolean(opts.onlyAffectedBy || opts.onlyHubRerunNeeded);
|
|
12193
12159
|
if (filtering && targets.length > 0) throw new RunUsageError("a --only-* filter and an explicit spec target cannot be combined");
|
|
12194
|
-
const rerunProfile = opts.
|
|
12195
|
-
if (opts.
|
|
12160
|
+
const rerunProfile = opts.onlyHubRerunNeeded === true ? requireRerunProfile(opts.hubProfile) : null;
|
|
12161
|
+
if (opts.onlyHubRerunNeededWithUnknown && rerunProfile === null) warn("--only-hub-rerun-needed-with-unknown is ignored: it only applies to --only-hub-rerun-needed");
|
|
12196
12162
|
const forExecution = opts.dryRun !== true;
|
|
12197
12163
|
const cwd = opts.cwd ?? process.cwd();
|
|
12198
12164
|
const wantsLastGreen = opts.onFailExplain === true && opts.onFailExplainBase === void 0;
|
|
@@ -12250,17 +12216,15 @@ async function executeRun(targets, opts) {
|
|
|
12250
12216
|
}
|
|
12251
12217
|
if (wantsLastGreen && hubCtx == null) throw new RunUsageError("--on-fail-explain needs a hub connection for the per-spec last-green baselines (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN), or an explicit --on-fail-explain-base <ref>");
|
|
12252
12218
|
const ledgerHub = wantsLastGreen ? hubCtx : null;
|
|
12253
|
-
if (rerunProfile !== null && hubCtx == null) throw new RunUsageError("--only-hub-
|
|
12254
|
-
if (opts.onlyHubAuditedClean && hubCtx == null) throw new RunUsageError("--only-hub-audited-clean requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12219
|
+
if (rerunProfile !== null && hubCtx == null) throw new RunUsageError("--only-hub-rerun-needed requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12255
12220
|
if (opts.reportToHub && hubCtx == null) throw new RunUsageError("--report-to-hub requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12256
12221
|
if (opts.learnHubLivePrompt && hubCtx == null) throw new RunUsageError("--learn-hub-live-prompt requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12257
|
-
const [customPrompt, triageUserPrompt, ledgerEntries, rerunReport, fetchedDeployHead
|
|
12222
|
+
const [customPrompt, triageUserPrompt, ledgerEntries, rerunReport, fetchedDeployHead] = await Promise.all([
|
|
12258
12223
|
forExecution ? fetchCustomPrompt(hubCtx) : null,
|
|
12259
12224
|
forExecution ? fetchTriageUserPrompt(hubCtx) : null,
|
|
12260
12225
|
forExecution && ledgerHub ? fetchLastGreenLedger(ledgerHub, opts.hubProfile, cwd) : null,
|
|
12261
12226
|
rerunProfile !== null && hubCtx ? fetchRerunReport(hubCtx, rerunProfile) : null,
|
|
12262
|
-
forExecution && hubCtx && opts.hubProfile && rerunProfile === null ? tryDeployHeadSha(hubCtx, opts.hubProfile) : null
|
|
12263
|
-
opts.onlyHubAuditedClean && hubCtx ? fetchAuditedLedger(hubCtx) : null
|
|
12227
|
+
forExecution && hubCtx && opts.hubProfile && rerunProfile === null ? tryDeployHeadSha(hubCtx, opts.hubProfile) : null
|
|
12264
12228
|
]).catch(asHubReadError);
|
|
12265
12229
|
const deployedSha = rerunReport?.deployHead.sha ?? fetchedDeployHead;
|
|
12266
12230
|
if (ledgerEntries) diffProvider = createDiffProvider({
|
|
@@ -12288,24 +12252,19 @@ async function executeRun(targets, opts) {
|
|
|
12288
12252
|
const before = specs.length;
|
|
12289
12253
|
let unanswerable = 0;
|
|
12290
12254
|
if (rerunReport) {
|
|
12291
|
-
const selection = selectSpecsNeedingRerun(specs, rerunReport, { includeUnknown: opts.
|
|
12255
|
+
const selection = selectSpecsNeedingRerun(specs, rerunReport, { includeUnknown: opts.onlyHubRerunNeededWithUnknown === true });
|
|
12292
12256
|
specs = selection.selected;
|
|
12293
12257
|
unanswerable = selection.excludedUnanswerable;
|
|
12294
12258
|
meta("stale-base", `deploy ${rerunReport.deployHead.sha.slice(0, 12)} (profile ${rerunReport.profile})`);
|
|
12295
12259
|
meta("stale-states", selection.summary);
|
|
12296
12260
|
}
|
|
12297
|
-
if (auditedLedger) {
|
|
12298
|
-
const picked = selectAuditedClean(specs, auditedLedger);
|
|
12299
|
-
specs = picked.selected;
|
|
12300
|
-
meta("audit-states", `${picked.selected.length} clean, ${picked.drifted} drifted, ${picked.unaudited} never audited`);
|
|
12301
|
-
}
|
|
12302
12261
|
if (opts.onlyAffectedBy) specs = (await collectChangedSpecs(specs, {
|
|
12303
12262
|
cwd,
|
|
12304
12263
|
base: opts.onlyAffectedBy,
|
|
12305
12264
|
...opts.model ? { model: opts.model } : {}
|
|
12306
12265
|
})).specs;
|
|
12307
12266
|
meta("selected", `${specs.length} of ${before} spec${before === 1 ? "" : "s"}`);
|
|
12308
|
-
if (specs.length === 0 && unanswerable > 0) hint(`${unanswerable} spec(s) were excluded because the hub could not tell whether they need a re-run; pass --only-hub-
|
|
12267
|
+
if (specs.length === 0 && unanswerable > 0) hint(`${unanswerable} spec(s) were excluded because the hub could not tell whether they need a re-run; pass --only-hub-rerun-needed-with-unknown to run them anyway`);
|
|
12309
12268
|
}
|
|
12310
12269
|
if (specs.length === 0) {
|
|
12311
12270
|
warn("no specs to run");
|
|
@@ -13019,7 +12978,7 @@ function installTeardownSignalHandlers(teardown) {
|
|
|
13019
12978
|
}
|
|
13020
12979
|
//#endregion
|
|
13021
12980
|
//#region src/cli/run.ts
|
|
13022
|
-
const runCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs, on any target. Agent-browser specs replay the recorded test.spec.ts under vitest (default), or, with spec.yaml `mode: live`, have Claude drive agent-browser live per step. External-target specs (playwright, runn) run through the target's configured `runCommand`. A structured report (report.json + evidence) is always written; use --report-to-hub to also stream it to a hub.").optionsGroup("Which specs to run:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Cannot be combined with an explicit spec id.").option("--only-hub-
|
|
12981
|
+
const runCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs, on any target. Agent-browser specs replay the recorded test.spec.ts under vitest (default), or, with spec.yaml `mode: live`, have Claude drive agent-browser live per step. External-target specs (playwright, runn) run through the target's configured `runCommand`. A structured report (report.json + evidence) is always written; use --report-to-hub to also stream it to a hub.").optionsGroup("Which specs to run:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Cannot be combined with an explicit spec id.").option("--only-hub-rerun-needed", "Only specs the hub answers `needed` for: their last result no longer covers what is deployed. Specs the audit rejected answer `blocked` and are never taken — a run cannot repair a spec. No git diff involved. Requires a hub connection and --hub-profile.").option("--only-hub-rerun-needed-with-unknown", "With --only-hub-rerun-needed: also take specs whose re-run need the hub cannot answer ('unknown') and specs that never ran ('neverRun'). Off by default: an unanswerable question is reported, not guessed.").option("--dry-run", "Print the specs this invocation would run, then exit 0 without executing anything and without writing a report. Works with every selection flag.").optionsGroup("How to run them:").option("--concurrency <n>", "Run up to N specs in parallel within each phase (deterministic / external-target / live), never across phases. Default 1 (sequential). Live specs each get an isolated agent-browser session; high values spawn many headed Chrome instances.", parseConcurrency$1, 1).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--live-step-retry <n>", "(live only) Retry each failed step up to N more times before recording failure. This retries a step, not the whole spec — see --on-fail-explain-rerun for that.", (raw) => {
|
|
13023
12982
|
const n = Number(raw);
|
|
13024
12983
|
if (!Number.isFinite(n) || n < 0 || Math.floor(n) !== n) throw new Error(`--live-step-retry must be a non-negative integer, got "${raw}"`);
|
|
13025
12984
|
return n;
|
|
@@ -13042,11 +13001,7 @@ function parseConcurrency$1(raw) {
|
|
|
13042
13001
|
function headerTarget(targets, opts) {
|
|
13043
13002
|
if (targets.length === 1) return targets[0];
|
|
13044
13003
|
if (targets.length > 1) return `${targets.length} targets`;
|
|
13045
|
-
const filters = [
|
|
13046
|
-
opts.onlyAffectedBy ? "affected" : null,
|
|
13047
|
-
opts.onlyHubStale ? "stale" : null,
|
|
13048
|
-
opts.onlyHubAuditedClean ? "audited clean" : null
|
|
13049
|
-
].filter((s) => s !== null);
|
|
13004
|
+
const filters = [opts.onlyAffectedBy ? "affected" : null, opts.onlyHubRerunNeeded ? "needs re-run" : null].filter((s) => s !== null);
|
|
13050
13005
|
return filters.length === 0 ? "(all specs)" : `(${filters.join(" + ")})`;
|
|
13051
13006
|
}
|
|
13052
13007
|
/**
|
|
@@ -15673,7 +15628,7 @@ function driftResultsToReport(results, meta) {
|
|
|
15673
15628
|
//#endregion
|
|
15674
15629
|
//#region src/cli/audit.ts
|
|
15675
15630
|
const DEFAULT_CONCURRENCY = 3;
|
|
15676
|
-
const auditCommand = addLanguageOption(new Command("audit").argument("[feature/spec]", "Optional spec id. If omitted, every spec under .ccqa/features/ is checked.").description("Read each spec against the code it describes and report where the two have drifted. Static: no browser is run, so this is the cheap check to put in front of `ccqa run`.").optionsGroup("Which specs to audit:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Costs one model call; specs it cannot decide are audited rather than skipped.").optionsGroup("How to run it:").option("--concurrency <n>", `Parallel spec checks (default: ${DEFAULT_CONCURRENCY})`).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").optionsGroup("What to do with the results:").option("--report-format <fmt>", "Output format: text | json | github", "text").option("--report-to-hub", "Push the result to a ccqa hub as a run (kind: drift), which is what updates the drift ledger `ccqa run --only-hub-
|
|
15631
|
+
const auditCommand = addLanguageOption(new Command("audit").argument("[feature/spec]", "Optional spec id. If omitted, every spec under .ccqa/features/ is checked.").description("Read each spec against the code it describes and report where the two have drifted. Static: no browser is run, so this is the cheap check to put in front of `ccqa run`.").optionsGroup("Which specs to audit:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Costs one model call; specs it cannot decide are audited rather than skipped.").optionsGroup("How to run it:").option("--concurrency <n>", `Parallel spec checks (default: ${DEFAULT_CONCURRENCY})`).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").optionsGroup("What to do with the results:").option("--report-format <fmt>", "Output format: text | json | github", "text").option("--report-to-hub", "Push the result to a ccqa hub as a run (kind: drift), which is what updates the drift ledger. A spec it finds drifted answers `blocked` to `ccqa run --only-hub-rerun-needed`, and is not run until the drift clears.").option("--exit-on <level>", "Exit non-zero on this severity or higher: warn | error", "error").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory used as both the .ccqa root and the codebase Claude reads. Useful for monorepos. Defaults to process.cwd().").option("--project <name>", "Logical project name for the pushed run. Defaults to the current directory's name.").option(...hubUrlOption).option(...hubTokenOption).option(...hubHeaderOption)).action(withUsageErrors(async (specPath, opts) => {
|
|
15677
15632
|
await withCostTally(() => runAudit(specPath, opts));
|
|
15678
15633
|
}));
|
|
15679
15634
|
async function runAudit(specPath, opts) {
|
|
@@ -17119,7 +17074,7 @@ async function loadSpecTargets(perspectives, project) {
|
|
|
17119
17074
|
//#endregion
|
|
17120
17075
|
//#region src/hub/core/rerun.ts
|
|
17121
17076
|
function computeRerun(input) {
|
|
17122
|
-
const { specs, ledger, log, touchIndex } = input;
|
|
17077
|
+
const { specs, ledger, log, touchIndex, drift } = input;
|
|
17123
17078
|
const notEvaluated = log.entries.length === 0 && Object.keys(ledger.run).length === 0 && Object.keys(ledger.green).length === 0;
|
|
17124
17079
|
const positionBySha = /* @__PURE__ */ new Map();
|
|
17125
17080
|
log.entries.forEach((entry, i) => {
|
|
@@ -17144,7 +17099,12 @@ function computeRerun(input) {
|
|
|
17144
17099
|
lastGreen: ledger.green[spec.key] ?? null,
|
|
17145
17100
|
lastRed: ledger.red[spec.key] ?? null
|
|
17146
17101
|
};
|
|
17147
|
-
|
|
17102
|
+
const blocked = blockedBy(drift, spec.key);
|
|
17103
|
+
out[spec.key] = blocked ? {
|
|
17104
|
+
state: "blocked",
|
|
17105
|
+
blockedReason: blocked,
|
|
17106
|
+
...coords
|
|
17107
|
+
} : notEvaluated ? {
|
|
17148
17108
|
state: "notEvaluated",
|
|
17149
17109
|
...coords
|
|
17150
17110
|
} : {
|
|
@@ -17154,6 +17114,21 @@ function computeRerun(input) {
|
|
|
17154
17114
|
}
|
|
17155
17115
|
return out;
|
|
17156
17116
|
}
|
|
17117
|
+
/**
|
|
17118
|
+
* Why the audit rejects this spec, or null when it does not.
|
|
17119
|
+
*
|
|
17120
|
+
* Only a finding blocks. A spec with no ledger entry was never audited, and a
|
|
17121
|
+
* `UNKNOWN` entry is the audit saying it could not tell — neither is a reason
|
|
17122
|
+
* to withhold a run, and treating them as one would stop every newly written
|
|
17123
|
+
* spec from ever executing.
|
|
17124
|
+
*/
|
|
17125
|
+
function blockedBy(drift, key) {
|
|
17126
|
+
switch (drift.specs[key]?.label) {
|
|
17127
|
+
case "TEST_DRIFT": return "testDrift";
|
|
17128
|
+
case "SPEC_CHANGE": return "specChange";
|
|
17129
|
+
default: return null;
|
|
17130
|
+
}
|
|
17131
|
+
}
|
|
17157
17132
|
function verdict(spec, lastRun, log, positionBySha, touchIndex, range) {
|
|
17158
17133
|
if (!lastRun) return { state: "neverRun" };
|
|
17159
17134
|
if (log.entries.length === 0) return unknown("noDeployLog");
|
|
@@ -17196,21 +17171,23 @@ function unknown(reason) {
|
|
|
17196
17171
|
/**
|
|
17197
17172
|
* GET /api/v1/projects/:project/rerun?profile=
|
|
17198
17173
|
*
|
|
17199
|
-
* Per spec: is
|
|
17174
|
+
* Per spec: is it worth running, and if not, why? Set arithmetic over the spec
|
|
17200
17175
|
* ledger, the profile's deploy log and each deploy's per-spec touch verdicts
|
|
17201
|
-
* recorded by `ccqa select-specs` (ADR-0010, ADR-0011)
|
|
17202
|
-
*
|
|
17203
|
-
*
|
|
17176
|
+
* recorded by `ccqa select-specs` (ADR-0010, ADR-0011), plus the drift ledger —
|
|
17177
|
+
* a spec the audit rejected answers `blocked`, because re-running it cannot
|
|
17178
|
+
* clear what is wrong with it. The spec ledger is read across every branch: a
|
|
17179
|
+
* run exercises the deployed environment whatever branch its code came from.
|
|
17204
17180
|
*/
|
|
17205
17181
|
function createGetRerunHandler(storage) {
|
|
17206
17182
|
return async (ctx) => {
|
|
17207
17183
|
const project = requireSafeSegment(ctx.params.project, "project");
|
|
17208
17184
|
const profile = requireProfileParam(ctx.url);
|
|
17209
|
-
const [specs, ledger, log, touchIndex] = await Promise.all([
|
|
17185
|
+
const [specs, ledger, log, touchIndex, drift] = await Promise.all([
|
|
17210
17186
|
loadSpecTargets(storage.perspectives, project),
|
|
17211
17187
|
storage.ledger.getMerged(project, profile),
|
|
17212
17188
|
storage.deploys.getLog(project, profile),
|
|
17213
|
-
storage.deploys.getTouchIndex(project, profile)
|
|
17189
|
+
storage.deploys.getTouchIndex(project, profile),
|
|
17190
|
+
storage.driftLedger.getMerged(project)
|
|
17214
17191
|
]);
|
|
17215
17192
|
if (specs === null) throw new HttpError(404, "no_perspectives", `no perspectives stored for project "${project}" — push one with \`ccqa perspectives\` before asking which specs need a re-run`);
|
|
17216
17193
|
const head = log.entries[log.entries.length - 1];
|
|
@@ -17226,7 +17203,8 @@ function createGetRerunHandler(storage) {
|
|
|
17226
17203
|
specs,
|
|
17227
17204
|
ledger,
|
|
17228
17205
|
log,
|
|
17229
|
-
touchIndex
|
|
17206
|
+
touchIndex,
|
|
17207
|
+
drift
|
|
17230
17208
|
})
|
|
17231
17209
|
});
|
|
17232
17210
|
};
|
|
@@ -18589,6 +18567,7 @@ const CSS = `
|
|
|
18589
18567
|
eats the next rule, and a backtick ends the template literal this CSS
|
|
18590
18568
|
lives in. */
|
|
18591
18569
|
.sg-drift-found, .sg-needed { background: var(--amber-fill); }
|
|
18570
|
+
.sg-blocked { background: var(--fail); }
|
|
18592
18571
|
.sg-unknown, .sg-drift-unknown { background: var(--info); }
|
|
18593
18572
|
.sg-drift-clean, .sg-notneeded { background: var(--pass); }
|
|
18594
18573
|
.sg-drift-none, .sg-neverrun { background: var(--muted-2); }
|
|
@@ -18797,6 +18776,7 @@ const CLIENT_JS = `
|
|
|
18797
18776
|
"perspectives.result.ci": "CI",
|
|
18798
18777
|
"perspectives.rerun.state.needed": "Re-run needed",
|
|
18799
18778
|
"perspectives.rerun.state.notNeeded": "Not needed",
|
|
18779
|
+
"perspectives.rerun.state.blocked": "Blocked by the audit",
|
|
18800
18780
|
"perspectives.rerun.state.unknown": "Can't tell",
|
|
18801
18781
|
"perspectives.rerun.state.neverRun": "Never run",
|
|
18802
18782
|
"perspectives.rerun.state.notEvaluated": "Not evaluated",
|
|
@@ -18809,6 +18789,8 @@ const CLIENT_JS = `
|
|
|
18809
18789
|
"perspectives.rerun.touchedUnknown": "a deploy since the last run matched this case",
|
|
18810
18790
|
"perspectives.rerun.neverRunHint": "no result recorded for this profile yet",
|
|
18811
18791
|
"perspectives.rerun.notEvaluatedHint": "no run and no deploy has ever been recorded for this profile",
|
|
18792
|
+
"perspectives.rerun.blocked.testDrift": "the generated test no longer matches the code — re-record it",
|
|
18793
|
+
"perspectives.rerun.blocked.specChange": "the spec describes something the code no longer does — a human decides",
|
|
18812
18794
|
"perspectives.rerun.why.noSelectionInRange": "a deploy in range was recorded without a spec selection",
|
|
18813
18795
|
"perspectives.rerun.why.selectionUnknown": "the selector could not tell whether this case was affected",
|
|
18814
18796
|
"perspectives.rerun.why.noDeployLog": "no deploy log for this profile",
|
|
@@ -18946,6 +18928,7 @@ const CLIENT_JS = `
|
|
|
18946
18928
|
"perspectives.result.ci": "CI",
|
|
18947
18929
|
"perspectives.rerun.state.needed": "要再実行",
|
|
18948
18930
|
"perspectives.rerun.state.notNeeded": "不要",
|
|
18931
|
+
"perspectives.rerun.state.blocked": "監査で保留",
|
|
18949
18932
|
"perspectives.rerun.state.unknown": "判定できない",
|
|
18950
18933
|
"perspectives.rerun.state.neverRun": "未実行",
|
|
18951
18934
|
"perspectives.rerun.state.notEvaluated": "未評価",
|
|
@@ -18958,6 +18941,8 @@ const CLIENT_JS = `
|
|
|
18958
18941
|
"perspectives.rerun.touchedUnknown": "前回実行以降のデプロイがこのケースに一致する変更を行っています",
|
|
18959
18942
|
"perspectives.rerun.neverRunHint": "このプロファイルでの実行記録がまだありません",
|
|
18960
18943
|
"perspectives.rerun.notEvaluatedHint": "このプロファイルには実行もデプロイも記録がありません",
|
|
18944
|
+
"perspectives.rerun.blocked.testDrift": "生成されたテストが古くなっています。録り直してください",
|
|
18945
|
+
"perspectives.rerun.blocked.specChange": "spec がコードのやめた動作を書いています。人が判断します",
|
|
18961
18946
|
"perspectives.rerun.why.noSelectionInRange": "対象範囲に判定を伴わないデプロイがあります",
|
|
18962
18947
|
"perspectives.rerun.why.selectionUnknown": "影響の有無を判定できませんでした",
|
|
18963
18948
|
"perspectives.rerun.why.noDeployLog": "このプロファイルのデプロイ記録がありません",
|
|
@@ -20998,6 +20983,7 @@ const CLIENT_JS = `
|
|
|
20998
20983
|
if (!head) return t("perspectives.rerun.noDeployHead");
|
|
20999
20984
|
return t("perspectives.rerun.vsDeploy") + " " + shortSha(head.sha) + " · " + relTime(head.at);
|
|
21000
20985
|
}
|
|
20986
|
+
if (rr.state === "blocked") return rerunReasonText("perspectives.rerun.blocked.", rr.blockedReason || "");
|
|
21001
20987
|
if (rr.state === "unknown") return rerunReasonText("perspectives.rerun.why.", rr.reason || "");
|
|
21002
20988
|
return rerunCannotJudge(rr);
|
|
21003
20989
|
}
|
|
@@ -21010,12 +20996,13 @@ const CLIENT_JS = `
|
|
|
21010
20996
|
// browser to click through.
|
|
21011
20997
|
|
|
21012
20998
|
// Bar segments in drawing order: what to act on first, then what needs no
|
|
21013
|
-
// action, then what was never measured. "
|
|
21014
|
-
//
|
|
21015
|
-
//
|
|
21016
|
-
|
|
20999
|
+
// action, then what was never measured. "blocked" leads because it is the
|
|
21000
|
+
// only state a run cannot clear — someone has to repair the spec. "unknown"
|
|
21001
|
+
// keeps its own place and its own colour: folding it into "notNeeded" would
|
|
21002
|
+
// turn "we cannot say" into "all clear", which is what ADR-0010 forbids.
|
|
21003
|
+
var RERUN_ORDER = ["blocked", "needed", "unknown", "notNeeded", "neverRun", "notEvaluated"];
|
|
21017
21004
|
var RERUN_SEG_CLASS = {
|
|
21018
|
-
needed: "sg-needed", unknown: "sg-unknown", notNeeded: "sg-notneeded",
|
|
21005
|
+
blocked: "sg-blocked", needed: "sg-needed", unknown: "sg-unknown", notNeeded: "sg-notneeded",
|
|
21019
21006
|
neverRun: "sg-neverrun", notEvaluated: "sg-noteval"
|
|
21020
21007
|
};
|
|
21021
21008
|
|
|
@@ -21025,7 +21012,7 @@ const CLIENT_JS = `
|
|
|
21025
21012
|
// state this UI does not know reads as unknown for the same reason: an
|
|
21026
21013
|
// answer we cannot interpret is not evidence that nothing is needed.
|
|
21027
21014
|
function rerunComposition(verdicts) {
|
|
21028
|
-
var counts = { needed: 0, unknown: 0, notNeeded: 0, neverRun: 0, notEvaluated: 0 };
|
|
21015
|
+
var counts = { blocked: 0, needed: 0, unknown: 0, notNeeded: 0, neverRun: 0, notEvaluated: 0 };
|
|
21029
21016
|
verdicts.forEach(function (rr) {
|
|
21030
21017
|
if (!rr || !rr.state) { counts.notEvaluated += 1; return; }
|
|
21031
21018
|
var known = Object.prototype.hasOwnProperty.call(counts, rr.state);
|
|
@@ -21156,7 +21143,7 @@ const CLIENT_JS = `
|
|
|
21156
21143
|
//
|
|
21157
21144
|
// "unknown" keeps its own state rather than folding into the last result:
|
|
21158
21145
|
// it means the hub cannot say whether that result still holds, and
|
|
21159
|
-
// --only-hub-
|
|
21146
|
+
// --only-hub-rerun-needed does not re-run it without --only-hub-rerun-needed-with-unknown. Showing
|
|
21160
21147
|
// it as passed or failed would claim a confidence nothing supports.
|
|
21161
21148
|
function perspRunState(rr) {
|
|
21162
21149
|
if (!rr) return null;
|
|
@@ -187,6 +187,7 @@ declare const RerunReportSchema: z.ZodObject<{
|
|
|
187
187
|
unknown: "unknown";
|
|
188
188
|
needed: "needed";
|
|
189
189
|
notNeeded: "notNeeded";
|
|
190
|
+
blocked: "blocked";
|
|
190
191
|
neverRun: "neverRun";
|
|
191
192
|
notEvaluated: "notEvaluated";
|
|
192
193
|
}>;
|
|
@@ -199,6 +200,10 @@ declare const RerunReportSchema: z.ZodObject<{
|
|
|
199
200
|
deployedShaNotInLog: "deployedShaNotInLog";
|
|
200
201
|
gapInRange: "gapInRange";
|
|
201
202
|
}>>;
|
|
203
|
+
blockedReason: z.ZodOptional<z.ZodEnum<{
|
|
204
|
+
testDrift: "testDrift";
|
|
205
|
+
specChange: "specChange";
|
|
206
|
+
}>>;
|
|
202
207
|
lastRun: z.ZodNullable<z.ZodObject<{
|
|
203
208
|
gitHead: z.ZodString;
|
|
204
209
|
runId: z.ZodString;
|
|
@@ -820,7 +825,7 @@ interface HubClient {
|
|
|
820
825
|
}): Promise<Record<string, LastGreenEntry>>;
|
|
821
826
|
/**
|
|
822
827
|
* Per spec of one project/profile: is its last result still trustworthy?
|
|
823
|
-
* Answers `ccqa run --only-hub-
|
|
828
|
+
* Answers `ccqa run --only-hub-rerun-needed`. 404 when the project has no
|
|
824
829
|
* perspectives document — there is then no spec registered to compare
|
|
825
830
|
* against a deploy, which the caller must report rather than read as
|
|
826
831
|
* "nothing to run".
|
package/dist/package.json
CHANGED