ccqa 0.9.1 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +98 -4
- package/dist/bin/ccqa.mjs +464 -131
- package/dist/package.json +1 -1
- package/dist/runtime/test-helpers.d.mts +8 -1
- package/dist/runtime/test-helpers.mjs +28 -3
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -69,6 +69,8 @@ ccqa run tasks/create-and-complete # vitest replays test.spec.ts; no LLM
|
|
|
69
69
|
ccqa run tasks/create-and-complete # Claude drives the browser every time
|
|
70
70
|
```
|
|
71
71
|
|
|
72
|
+
Live specs can start already-signed-in by pointing `statePath:` at a saved agent-browser state file (cookies + localStorage). Run an interactive login locally once, save the state with `agent-browser state save .ccqa/sessions/<name>.json`, then commit the path (not the file) — see [Pre-authenticated state](#pre-authenticated-state-statepath) below for the local bootstrap and the CI restore pattern.
|
|
73
|
+
|
|
72
74
|
By default deterministic runs write step-boundary screenshots and metadata to `ccqa-report/evidence/<feature>/<spec>/` so a reviewer can confirm a passing spec actually reached the states its `expected` clauses describe. Disable with `--no-evidence`.
|
|
73
75
|
|
|
74
76
|
In CI you can opt in to an HTML run report by passing `--report` — every failing spec gets a drift audit plus a root-cause call (TEST_DRIFT / SPEC_CHANGE / PRODUCT_BUG) using the branch's git diff as context, and the report lets a human grade those calls to measure their accuracy. Requires `ANTHROPIC_API_KEY` or a local Claude login for the analysis part. Opt out with `--no-failure-analysis` (which also implicitly skips the drift audit — the audit is rendered as evidence under the classification, so without the classification the cost has nowhere to land). Use `--no-drift-audit` to keep the classification but skip the audit. See [Run report](./docs/report.md).
|
|
@@ -84,6 +86,7 @@ ccqa run --changed --report # only specs whose relatedPaths t
|
|
|
84
86
|
|---|---|
|
|
85
87
|
| Write specs interactively with Claude | [Draft](./docs/draft.md) |
|
|
86
88
|
| Reuse login and other shared step sequences | [Blocks](./docs/blocks.md) |
|
|
89
|
+
| Drive `<input type="file">` without an OS picker | [File upload](./docs/file-upload.md) |
|
|
87
90
|
| Assertion helper functions | [Assertions](./docs/assertions.md) |
|
|
88
91
|
| Auto-fix failing tests | [Auto-fix](./docs/auto-fix.md) |
|
|
89
92
|
| Detect spec/code drift in CI | [Drift](./docs/drift.md) |
|
|
@@ -98,16 +101,18 @@ ccqa init Scaffold .ccqa/prompts/{live,record}.{user,ag
|
|
|
98
101
|
ccqa draft [feature/spec] Co-author a test spec with Claude
|
|
99
102
|
ccqa perspectives Inventory existing test coverage into .ccqa/perspectives.yaml
|
|
100
103
|
ccqa record <feature/spec> (deterministic specs only) Trace browser actions + generate test.spec.ts
|
|
101
|
-
ccqa run [feature/spec]
|
|
104
|
+
ccqa run [feature/spec...] Execute specs. Per spec, the spec.yaml `mode:` field selects deterministic
|
|
102
105
|
(vitest replay) or live (Claude drives every time). One run can mix both;
|
|
103
|
-
`--report` writes one unified HTML.
|
|
106
|
+
`--report` writes one unified HTML. Pass multiple targets space-separated.
|
|
104
107
|
ccqa drift [feature/spec] Standalone spec ↔ codebase static audit (for PR checks)
|
|
105
108
|
```
|
|
106
109
|
|
|
107
110
|
`ccqa run` flags:
|
|
108
111
|
|
|
109
112
|
- `--report [dir]` — write a self-contained HTML run report (default dir: `ccqa-report/`)
|
|
110
|
-
- `--
|
|
113
|
+
- `--profile <name>` — load `.ccqa/profiles/<name>.env` into the environment before resolving spec `${VAR}` references, so one spec targets dev/stg/prd without per-environment copies. See [Profiles](#profiles---profile).
|
|
114
|
+
- `--changed` — restrict execution to specs whose `relatedPaths` intersect `git diff <base>...HEAD`. Mutually exclusive with explicit spec targets.
|
|
115
|
+
- `--concurrency <n>` — run up to N specs in parallel **within each mode** (deterministic specs run as one phase, live specs as the next; parallelism is within a phase, not across). Default `1` (sequential, identical to before). Above 1, each spec's output is buffered and flushed as a labelled block so parallel logs stay legible. Live specs each launch their own headed Chrome, so high values spawn many browser instances.
|
|
111
116
|
- `--base <ref>` — base ref for the git diff (default: `$GITHUB_BASE_REF`, then `origin/main`)
|
|
112
117
|
- `--no-failure-analysis` — skip the per-failure root-cause classification (also skips the drift audit, since the audit only shows under the classification)
|
|
113
118
|
- `--no-drift-audit` — skip the spec ↔ code drift audit while keeping the classification
|
|
@@ -119,7 +124,7 @@ ccqa drift [feature/spec] Standalone spec ↔ codebase static audit (fo
|
|
|
119
124
|
|
|
120
125
|
All Claude-driven commands accept `-m, --model <name>` (alias `sonnet` | `opus` | `haiku`, or a full model ID). The flag overrides `CCQA_MODEL`; when both are unset, the Claude Code CLI default is used. They also accept `--language <bcp47>` (e.g. `ja`, `en`) to set the language of human-readable output; the default `auto` follows the language of the spec/codebase. `--cwd <path>` works on `record` / `run` / `drift` so you can target a subpackage inside a monorepo from the repo root. Interactive commands authenticate via your local Claude Code login; commands that talk to Claude in CI (`ccqa run --report`, `ccqa drift`) additionally honor `ANTHROPIC_API_KEY`.
|
|
121
126
|
|
|
122
|
-
`<feature/spec>` is a 2-segment alias for the on-disk path `.ccqa/features/<feature>/test-cases/<spec>/`.
|
|
127
|
+
`<feature/spec>` is a 2-segment alias for the on-disk path `.ccqa/features/<feature>/test-cases/<spec>/`. `ccqa run` accepts several targets space-separated (each a `<feature>/<spec>`, a bare `<feature>` for all its specs, or omitted for everything); duplicates are de-duped and `--changed` cannot be combined with explicit targets.
|
|
123
128
|
|
|
124
129
|
## File structure
|
|
125
130
|
|
|
@@ -127,6 +132,9 @@ All Claude-driven commands accept `-m, --model <name>` (alias `sonnet` | `opus`
|
|
|
127
132
|
.ccqa/
|
|
128
133
|
perspectives.yaml # Inventory of existing coverage (machine-readable, canonical)
|
|
129
134
|
perspectives.md # Category index, regenerated from the YAML
|
|
135
|
+
profiles/ # `--profile <name>` env files
|
|
136
|
+
stg.env # URLs + credential refs; commit if it uses secret-manager refs, gitignore if it holds plaintext secrets
|
|
137
|
+
prd.env
|
|
130
138
|
prompts/ # Run `ccqa init` to scaffold these
|
|
131
139
|
record.user.md # Human-maintained guidance appended to `ccqa record` (trace phase)
|
|
132
140
|
record.agent.md # Auto-updated by `ccqa record --update-agent-prompt`
|
|
@@ -155,6 +163,26 @@ All Claude-driven commands accept `-m, --model <name>` (alias `sonnet` | `opus`
|
|
|
155
163
|
|
|
156
164
|
Add `.ccqa/features/*/test-cases/*/runs/` to `.gitignore` — these are per-run artefacts that should not be committed. Likewise `ccqa-report*/`.
|
|
157
165
|
|
|
166
|
+
## Profiles (`--profile`)
|
|
167
|
+
|
|
168
|
+
Keep environment-specific values out of specs as `${VAR}` references and supply them per environment from a **profile** — a `.env` under `.ccqa/profiles/<name>.env`. `ccqa run`/`record --profile <name>` merges it into the environment before resolving `${VAR}`, so one spec runs anywhere.
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
# .ccqa/profiles/stg.env
|
|
172
|
+
APP_BASE_URL=https://<your-app-host>
|
|
173
|
+
TEST_USER_EMAIL=<stg-test-account>
|
|
174
|
+
TEST_USER_PASSWORD=...
|
|
175
|
+
```
|
|
176
|
+
```bash
|
|
177
|
+
ccqa run auth/login --profile stg # same spec, stg values
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
- Name is free-form (`stg`/`prd` are conventions); a path separator / `..` / leading dot is rejected, and a missing profile exits 2. Only the name is logged, never values.
|
|
181
|
+
- Format is a small `.env` subset (`KEY=value`, `#` comments, `export`, quotes). Profile values **override** the inherited environment.
|
|
182
|
+
- Without `--profile`, ccqa auto-loads `<cwd>/.env` if present (like dotenv); with neither, `${VAR}` resolves against the existing `process.env` (e.g. `direnv`).
|
|
183
|
+
|
|
184
|
+
**Secrets:** gitignore any profile that holds plaintext secrets. ccqa only parses `.env` files — it doesn't resolve secret-manager references — so to keep secrets off disk, drop `--profile` and run ccqa under your secret manager instead (e.g. `op run --env-file=.ccqa/profiles/stg.env -- ccqa run ...`), which injects the resolved values into `process.env` for ccqa to read.
|
|
185
|
+
|
|
158
186
|
## Live specs (`mode: live`)
|
|
159
187
|
|
|
160
188
|
For specs declared `mode: live` in their spec.yaml, `ccqa run` skips codegen entirely: Claude executes each spec step against `agent-browser` directly, judges whether the step's `expected` outcome holds, and saves a PNG screenshot before and after every step. Use this mode when:
|
|
@@ -179,6 +207,72 @@ ccqa run --retry 2 tasks/create-and-complete
|
|
|
179
207
|
|
|
180
208
|
Constraints on selectors / `agent-browser` subcommands that apply during `ccqa record` (no `eval`, no `@ref`, no bare-tag positional `find`, no chained agent-browser calls) are **relaxed** for live specs — Claude can use any subcommand and any selector style because there is no replay contract to honour.
|
|
181
209
|
|
|
210
|
+
### Pre-authenticated state (`statePath:`)
|
|
211
|
+
|
|
212
|
+
By default each `ccqa run` of a live spec spins up a fresh `agent-browser` session and starts signed-out. That keeps runs hermetic but forces every device-trust gate (Slack "we don't recognize this browser", Google's unfamiliar-device prompt, MFA challenges, …) to fire on every run.
|
|
213
|
+
|
|
214
|
+
To skip them, save an authenticated browser state to a JSON file once locally and point the spec at it:
|
|
215
|
+
|
|
216
|
+
```yaml
|
|
217
|
+
title: Slack App Home — non-admin access denied
|
|
218
|
+
mode: live
|
|
219
|
+
statePath: .ccqa/sessions/slack-stg.json # cookies + localStorage to restore
|
|
220
|
+
steps:
|
|
221
|
+
- ...
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
ccqa resolves the path against the project root and passes `--state <path>` to every `agent-browser` invocation in the run (including ccqa's own screenshot calls). The file is **read-only** — `--state` loads it but never writes back to it. Re-running locally or in CI does not mutate it.
|
|
225
|
+
|
|
226
|
+
Bootstrap once locally:
|
|
227
|
+
|
|
228
|
+
```bash
|
|
229
|
+
# 1. Log in interactively in a headed browser.
|
|
230
|
+
agent-browser --headed open https://app.slack.com
|
|
231
|
+
# …complete login + device-trust prompts by hand…
|
|
232
|
+
|
|
233
|
+
# 2. Snapshot cookies + localStorage to the path the spec references.
|
|
234
|
+
mkdir -p .ccqa/sessions
|
|
235
|
+
agent-browser state save .ccqa/sessions/slack-stg.json
|
|
236
|
+
agent-browser close
|
|
237
|
+
|
|
238
|
+
# 3. ccqa run reuses the saved state — no login prompt.
|
|
239
|
+
ccqa run slack/app-home-non-admin-access-denied
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
Add `.ccqa/sessions/` to `.gitignore` — these files contain live auth cookies and must never be committed.
|
|
243
|
+
|
|
244
|
+
#### CI: bring the state file with you
|
|
245
|
+
|
|
246
|
+
`statePath:` lives entirely inside `.ccqa/` and never touches `~/`. CI re-uses the state by writing the file into the same path the spec already references:
|
|
247
|
+
|
|
248
|
+
```bash
|
|
249
|
+
# Locally, after the interactive bootstrap above:
|
|
250
|
+
base64 -i .ccqa/sessions/slack-stg.json | pbcopy
|
|
251
|
+
# paste into your CI secret store as CCQA_SLACK_STG_STATE_B64
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
```yaml
|
|
255
|
+
# .github/workflows/ccqa.yml (sketch)
|
|
256
|
+
- name: Restore agent-browser state
|
|
257
|
+
env:
|
|
258
|
+
CCQA_SLACK_STG_STATE_B64: ${{ secrets.CCQA_SLACK_STG_STATE_B64 }}
|
|
259
|
+
run: |
|
|
260
|
+
mkdir -p .ccqa/sessions
|
|
261
|
+
printf '%s' "$CCQA_SLACK_STG_STATE_B64" | base64 -d \
|
|
262
|
+
> .ccqa/sessions/slack-stg.json
|
|
263
|
+
|
|
264
|
+
- name: Run live specs
|
|
265
|
+
env:
|
|
266
|
+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
267
|
+
run: pnpm ccqa run --report
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
Caveats:
|
|
271
|
+
|
|
272
|
+
- **Expiry.** Whatever the upstream service's "remember this device" window is (Slack ≈ 30 days, others vary), the cookies in the state file eventually expire and CI starts failing on the device-trust gate again. Re-bootstrap locally and rotate the secret.
|
|
273
|
+
- **Treat the file as a credential.** It contains live auth cookies. Store it in your CI secret manager (GitHub Actions encrypted secrets, Vault, …) and never commit it.
|
|
274
|
+
- **Deterministic specs ignore `statePath:`.** Today it only affects `mode: live`; vitest-replayed specs always run isolated.
|
|
275
|
+
|
|
182
276
|
### Per-project guidance (`.ccqa/prompts/live.user.md` + `live.agent.md`)
|
|
183
277
|
|
|
184
278
|
ccqa's live-mode system prompt is deliberately product-agnostic. Anything specific to **your** project — staging URLs, login flow quirks, rich-editor types, common access-denied wording — belongs in two sibling files (run `ccqa init` to scaffold both):
|
package/dist/bin/ccqa.mjs
CHANGED
|
@@ -6,12 +6,14 @@ import { accessSync, existsSync, readFileSync, statSync } from "node:fs";
|
|
|
6
6
|
import { fileURLToPath } from "node:url";
|
|
7
7
|
import { access, mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from "node:fs/promises";
|
|
8
8
|
import { homedir, tmpdir } from "node:os";
|
|
9
|
-
import { delimiter, dirname, join, posix, relative, resolve } from "node:path";
|
|
9
|
+
import { delimiter, dirname, isAbsolute, join, posix, relative, resolve } from "node:path";
|
|
10
10
|
import { parse, stringify } from "yaml";
|
|
11
11
|
import { ZodError, z } from "zod";
|
|
12
12
|
import { execFile, spawn, spawnSync } from "node:child_process";
|
|
13
13
|
import { query } from "@anthropic-ai/claude-agent-sdk";
|
|
14
|
+
import { AsyncLocalStorage } from "node:async_hooks";
|
|
14
15
|
import { promisify } from "node:util";
|
|
16
|
+
import { randomUUID } from "node:crypto";
|
|
15
17
|
import { createInterface } from "node:readline";
|
|
16
18
|
import { createInterface as createInterface$1 } from "node:readline/promises";
|
|
17
19
|
//#region src/runtime/env-vars.ts
|
|
@@ -139,6 +141,7 @@ const TestSpecSchema = z.object({
|
|
|
139
141
|
title: z.string().min(1),
|
|
140
142
|
relatedPaths: z.array(z.string().min(1)).optional(),
|
|
141
143
|
mode: SpecModeSchema.optional(),
|
|
144
|
+
statePath: z.string().min(1).optional(),
|
|
142
145
|
steps: z.array(StepSchema).min(1)
|
|
143
146
|
}).strict();
|
|
144
147
|
/** Default mode when `mode:` is absent. */
|
|
@@ -757,6 +760,27 @@ function waitExit(child) {
|
|
|
757
760
|
});
|
|
758
761
|
}
|
|
759
762
|
//#endregion
|
|
763
|
+
//#region src/runtime/pool.ts
|
|
764
|
+
/**
|
|
765
|
+
* Run each item through `fn` with at most `concurrency` running at once.
|
|
766
|
+
* Results preserve input order. A throwing `fn` rejects the whole pool
|
|
767
|
+
* (callers that want per-item isolation should catch inside `fn`).
|
|
768
|
+
*/
|
|
769
|
+
async function runPool(items, concurrency, fn) {
|
|
770
|
+
const results = new Array(items.length);
|
|
771
|
+
let cursor = 0;
|
|
772
|
+
const worker = async () => {
|
|
773
|
+
while (true) {
|
|
774
|
+
const idx = cursor++;
|
|
775
|
+
if (idx >= items.length) return;
|
|
776
|
+
results[idx] = await fn(items[idx], idx);
|
|
777
|
+
}
|
|
778
|
+
};
|
|
779
|
+
const n = Math.max(1, Math.min(concurrency, items.length));
|
|
780
|
+
await Promise.all(Array.from({ length: n }, () => worker()));
|
|
781
|
+
return results;
|
|
782
|
+
}
|
|
783
|
+
//#endregion
|
|
760
784
|
//#region src/claude/extract-json.ts
|
|
761
785
|
/**
|
|
762
786
|
* Pulls a JSON object out of a Claude completion. Accepts either a fenced
|
|
@@ -779,26 +803,70 @@ const STEP_ICONS = {
|
|
|
779
803
|
STEP_SKIPPED: "⊘",
|
|
780
804
|
RUN_COMPLETED: "■"
|
|
781
805
|
};
|
|
806
|
+
/**
|
|
807
|
+
* When a `withBuffer` scope is active, every log line (stdout and stderr) is
|
|
808
|
+
* appended to its buffer instead of being written immediately. Parallel spec
|
|
809
|
+
* runs use this so each spec's narration — including logs emitted deep inside
|
|
810
|
+
* the live executor — flushes as one contiguous block, not interleaved.
|
|
811
|
+
*/
|
|
812
|
+
const bufferStore = new AsyncLocalStorage();
|
|
813
|
+
/** True while inside a `withBuffer` scope: progress lines avoid TTY cursor tricks. */
|
|
814
|
+
function isBuffered() {
|
|
815
|
+
return bufferStore.getStore() !== void 0;
|
|
816
|
+
}
|
|
817
|
+
function emit(text, sink = process.stdout) {
|
|
818
|
+
const store = bufferStore.getStore();
|
|
819
|
+
if (store) {
|
|
820
|
+
store.out.push(text);
|
|
821
|
+
return;
|
|
822
|
+
}
|
|
823
|
+
sink.write(text);
|
|
824
|
+
}
|
|
825
|
+
/**
|
|
826
|
+
* Write raw text to the active `withBuffer` scope, or straight to stdout when
|
|
827
|
+
* none is active. Lets a runner redirect sub-process output (e.g. a child's
|
|
828
|
+
* stdout) into the same buffer as its `log.*` lines so they flush together.
|
|
829
|
+
*/
|
|
830
|
+
function emitRaw(text) {
|
|
831
|
+
emit(text);
|
|
832
|
+
}
|
|
833
|
+
/**
|
|
834
|
+
* Run `fn` with all its log output captured into a buffer, then flush the
|
|
835
|
+
* buffer in one shot under `label`. Used by parallel runners to keep each
|
|
836
|
+
* spec's output legible. Output is flushed even when `fn` throws.
|
|
837
|
+
*
|
|
838
|
+
* When `buffered` is false, `fn` runs with no buffer so its output streams
|
|
839
|
+
* live — this is the sequential (concurrency 1) path, unchanged from before.
|
|
840
|
+
*/
|
|
841
|
+
async function withBuffer(label, buffered, fn) {
|
|
842
|
+
if (!buffered) return fn();
|
|
843
|
+
const store = { out: [] };
|
|
844
|
+
try {
|
|
845
|
+
return await bufferStore.run(store, fn);
|
|
846
|
+
} finally {
|
|
847
|
+
process.stdout.write(`\n──── ${label} ────\n${store.out.join("")}`);
|
|
848
|
+
}
|
|
849
|
+
}
|
|
782
850
|
function header(command, target) {
|
|
783
|
-
|
|
851
|
+
emit(`\nccqa ${command}${target ? ` ${target}` : ""}\n\n`);
|
|
784
852
|
}
|
|
785
853
|
function write(scope, message, sink = process.stdout) {
|
|
786
|
-
|
|
854
|
+
emit(`[${scope}] ${message}\n`, sink);
|
|
787
855
|
}
|
|
788
856
|
function meta(key, value) {
|
|
789
857
|
write("meta", `${key}: ${value}`);
|
|
790
858
|
}
|
|
791
859
|
function blank() {
|
|
792
|
-
|
|
860
|
+
emit("\n");
|
|
793
861
|
}
|
|
794
862
|
function info(message) {
|
|
795
863
|
write("info", message);
|
|
796
864
|
}
|
|
797
865
|
function step(type, stepId, detail) {
|
|
798
|
-
|
|
866
|
+
emit(` ${STEP_ICONS[type]} [${stepId}] ${detail}\n`);
|
|
799
867
|
}
|
|
800
868
|
function bash(command) {
|
|
801
|
-
|
|
869
|
+
emit(` $ ${command.slice(0, 120)}\n`);
|
|
802
870
|
}
|
|
803
871
|
function error(message) {
|
|
804
872
|
write("error", message, process.stderr);
|
|
@@ -807,7 +875,7 @@ function warn(message) {
|
|
|
807
875
|
write("warn", message, process.stderr);
|
|
808
876
|
}
|
|
809
877
|
function hint(message) {
|
|
810
|
-
|
|
878
|
+
emit("\n");
|
|
811
879
|
write("hint", message);
|
|
812
880
|
}
|
|
813
881
|
function fix(message) {
|
|
@@ -832,17 +900,17 @@ const PROGRESS_NONTTY_STRIDE = 5;
|
|
|
832
900
|
let lastProgressNonTtyEmit = -1;
|
|
833
901
|
function progress(current, total, label) {
|
|
834
902
|
const text = `[info] ${current + 1}/${total} ${label}`;
|
|
835
|
-
if (process.stdout.isTTY) {
|
|
903
|
+
if (process.stdout.isTTY && !isBuffered()) {
|
|
836
904
|
process.stdout.write(`\r${text}\x1b[K`);
|
|
837
905
|
return;
|
|
838
906
|
}
|
|
839
907
|
if (current === 0 || current - lastProgressNonTtyEmit >= PROGRESS_NONTTY_STRIDE) {
|
|
840
|
-
|
|
908
|
+
emit(`${text}\n`);
|
|
841
909
|
lastProgressNonTtyEmit = current;
|
|
842
910
|
}
|
|
843
911
|
}
|
|
844
912
|
function progressEnd() {
|
|
845
|
-
if (process.stdout.isTTY) process.stdout.write(`\r\x1b[K`);
|
|
913
|
+
if (process.stdout.isTTY && !isBuffered()) process.stdout.write(`\r\x1b[K`);
|
|
846
914
|
lastProgressNonTtyEmit = -1;
|
|
847
915
|
}
|
|
848
916
|
/**
|
|
@@ -1363,6 +1431,12 @@ function extractAbActionFromBashCommand(cmd) {
|
|
|
1363
1431
|
case "type":
|
|
1364
1432
|
case "select": return `AB_ACTION|${subCmd}|${args[0] ?? ""}|${args[1] ?? ""}|${args[2] ?? ""}`;
|
|
1365
1433
|
case "drag": return `AB_ACTION|drag|${args[0] ?? ""}|${args[1] ?? ""}|${args[2] ?? ""}`;
|
|
1434
|
+
case "upload": {
|
|
1435
|
+
const sel = args[0] ?? "";
|
|
1436
|
+
const files = args.slice(1);
|
|
1437
|
+
if (!sel || files.length === 0) return null;
|
|
1438
|
+
return `AB_ACTION|upload|${sel}|${files.join("|")}`;
|
|
1439
|
+
}
|
|
1366
1440
|
case "snapshot": return null;
|
|
1367
1441
|
case "find": return extractFindAbAction(args);
|
|
1368
1442
|
default: return null;
|
|
@@ -1700,25 +1774,15 @@ const DEFAULT_CONCURRENCY$1 = 3;
|
|
|
1700
1774
|
*/
|
|
1701
1775
|
async function analyzeDrift(input) {
|
|
1702
1776
|
const { targets, cwd, blocks, concurrency = DEFAULT_CONCURRENCY$1, model, language, onSpecStart } = input;
|
|
1703
|
-
|
|
1704
|
-
|
|
1705
|
-
|
|
1706
|
-
|
|
1707
|
-
|
|
1708
|
-
|
|
1709
|
-
|
|
1710
|
-
|
|
1711
|
-
|
|
1712
|
-
cwd,
|
|
1713
|
-
blocks,
|
|
1714
|
-
model,
|
|
1715
|
-
language
|
|
1716
|
-
});
|
|
1717
|
-
}
|
|
1718
|
-
};
|
|
1719
|
-
const pool = Array.from({ length: Math.min(concurrency, targets.length) }, () => worker());
|
|
1720
|
-
await Promise.all(pool);
|
|
1721
|
-
return results;
|
|
1777
|
+
return runPool(targets, concurrency, async (target) => {
|
|
1778
|
+
onSpecStart?.(target);
|
|
1779
|
+
return checkSpec(target, {
|
|
1780
|
+
cwd,
|
|
1781
|
+
blocks,
|
|
1782
|
+
model,
|
|
1783
|
+
language
|
|
1784
|
+
});
|
|
1785
|
+
});
|
|
1722
1786
|
}
|
|
1723
1787
|
async function checkSpec(target, opts) {
|
|
1724
1788
|
const { featureName, specName } = target;
|
|
@@ -4095,6 +4159,123 @@ const CLIENT_JS = `
|
|
|
4095
4159
|
})();
|
|
4096
4160
|
`;
|
|
4097
4161
|
//#endregion
|
|
4162
|
+
//#region src/runtime/profile-env.ts
|
|
4163
|
+
/**
|
|
4164
|
+
* Profile env (Issue #37). A profile is a named `.env` under
|
|
4165
|
+
* `.ccqa/profiles/<name>.env`; its contents merge into `process.env` before any
|
|
4166
|
+
* spec work, so one spec targets dev/stg/prd without per-environment copies.
|
|
4167
|
+
* Spec `${VAR}` references all resolve against `process.env` downstream.
|
|
4168
|
+
*
|
|
4169
|
+
* The `.env` parser is a small hand-rolled subset (no dotenv dependency).
|
|
4170
|
+
*/
|
|
4171
|
+
/**
|
|
4172
|
+
* Parse a `.env` body into a `name → value` map. Subset: blank / `#` lines
|
|
4173
|
+
* skipped, optional leading `export`, split on the first `=`, surrounding
|
|
4174
|
+
* quotes stripped, inline `# comment` dropped. No multi-line / interpolation.
|
|
4175
|
+
*/
|
|
4176
|
+
function parseDotenv(content) {
|
|
4177
|
+
const out = {};
|
|
4178
|
+
for (const rawLine of content.split(/\r?\n/)) {
|
|
4179
|
+
const line = rawLine.trim();
|
|
4180
|
+
if (line === "" || line.startsWith("#")) continue;
|
|
4181
|
+
const withoutExport = line.replace(/^export\s+/, "");
|
|
4182
|
+
const eq = withoutExport.indexOf("=");
|
|
4183
|
+
if (eq === -1) continue;
|
|
4184
|
+
const key = withoutExport.slice(0, eq).trim();
|
|
4185
|
+
if (key === "") continue;
|
|
4186
|
+
out[key] = parseValue(withoutExport.slice(eq + 1).trim());
|
|
4187
|
+
}
|
|
4188
|
+
return out;
|
|
4189
|
+
}
|
|
4190
|
+
function parseValue(raw) {
|
|
4191
|
+
const quote = raw[0];
|
|
4192
|
+
if (quote === "\"" || quote === "'") {
|
|
4193
|
+
const close = raw.indexOf(quote, 1);
|
|
4194
|
+
if (close !== -1 && /^\s*(#.*)?$/.test(raw.slice(close + 1))) return raw.slice(1, close);
|
|
4195
|
+
}
|
|
4196
|
+
const hash = raw.search(/\s#/);
|
|
4197
|
+
return hash === -1 ? raw : raw.slice(0, hash).trimEnd();
|
|
4198
|
+
}
|
|
4199
|
+
var ProfileNotFoundError = class extends Error {
|
|
4200
|
+
profile;
|
|
4201
|
+
path;
|
|
4202
|
+
constructor(profile, path) {
|
|
4203
|
+
super(`profile "${profile}" not found: ${path}`);
|
|
4204
|
+
this.name = "ProfileNotFoundError";
|
|
4205
|
+
this.profile = profile;
|
|
4206
|
+
this.path = path;
|
|
4207
|
+
}
|
|
4208
|
+
};
|
|
4209
|
+
var InvalidProfileNameError = class extends Error {
|
|
4210
|
+
profile;
|
|
4211
|
+
constructor(profile) {
|
|
4212
|
+
super(`invalid profile name "${profile}": expected a bare name like "stg" (no path separators, no leading dot)`);
|
|
4213
|
+
this.name = "InvalidProfileNameError";
|
|
4214
|
+
this.profile = profile;
|
|
4215
|
+
}
|
|
4216
|
+
};
|
|
4217
|
+
/**
|
|
4218
|
+
* A profile name must be a single, non-dot-leading path segment, so
|
|
4219
|
+
* `--profile <name>` can't read a file outside the profiles dir (e.g.
|
|
4220
|
+
* `--profile ../../etc/hosts`). Rejecting separators and a leading dot already
|
|
4221
|
+
* blocks `..` traversal, so an in-name `..` (like `v1..2`) stays allowed.
|
|
4222
|
+
*/
|
|
4223
|
+
function assertValidProfileName(profile) {
|
|
4224
|
+
if (profile === "" || profile.includes("/") || profile.includes("\\") || profile.startsWith(".")) throw new InvalidProfileNameError(profile);
|
|
4225
|
+
}
|
|
4226
|
+
/** Absolute path of the `.env` file backing `<profile>` under `<cwd>/.ccqa/`. */
|
|
4227
|
+
function profilePath(profile, cwd) {
|
|
4228
|
+
assertValidProfileName(profile);
|
|
4229
|
+
return join(cwd, ".ccqa", "profiles", `${profile}.env`);
|
|
4230
|
+
}
|
|
4231
|
+
/** Read + parse a `.env`, or `null` if absent. Other read errors propagate. */
|
|
4232
|
+
async function readDotenv(path) {
|
|
4233
|
+
let content;
|
|
4234
|
+
try {
|
|
4235
|
+
content = await readFile(path, "utf8");
|
|
4236
|
+
} catch (err) {
|
|
4237
|
+
if (err.code === "ENOENT") return null;
|
|
4238
|
+
throw err;
|
|
4239
|
+
}
|
|
4240
|
+
return parseDotenv(content);
|
|
4241
|
+
}
|
|
4242
|
+
/**
|
|
4243
|
+
* Load `.ccqa/profiles/<profile>.env`. A missing file throws — a typo must fail
|
|
4244
|
+
* loudly, not silently resolve every credential to empty.
|
|
4245
|
+
*/
|
|
4246
|
+
async function loadProfileEnv(profile, cwd) {
|
|
4247
|
+
const path = profilePath(profile, cwd);
|
|
4248
|
+
const vars = await readDotenv(path);
|
|
4249
|
+
if (vars === null) throw new ProfileNotFoundError(profile, path);
|
|
4250
|
+
return vars;
|
|
4251
|
+
}
|
|
4252
|
+
/** Absolute path of the default `.env` ccqa loads when `--profile` is absent. */
|
|
4253
|
+
function defaultEnvPath(cwd) {
|
|
4254
|
+
return join(cwd, ".env");
|
|
4255
|
+
}
|
|
4256
|
+
/**
|
|
4257
|
+
* Load `<cwd>/.env`, the default when no `--profile` is given. A missing `.env`
|
|
4258
|
+
* is fine (returns `null`) — the run falls back to the existing `process.env`.
|
|
4259
|
+
*/
|
|
4260
|
+
async function loadDefaultEnv(cwd) {
|
|
4261
|
+
return readDotenv(defaultEnvPath(cwd));
|
|
4262
|
+
}
|
|
4263
|
+
/**
|
|
4264
|
+
* Merge vars into `process.env`. With `override` (the default), the profile
|
|
4265
|
+
* wins over inherited values. Returns the applied names — never values, so
|
|
4266
|
+
* callers log names only and secrets stay out of the log.
|
|
4267
|
+
*/
|
|
4268
|
+
function applyProfileEnv(vars, opts = {}) {
|
|
4269
|
+
const override = opts.override ?? true;
|
|
4270
|
+
const applied = [];
|
|
4271
|
+
for (const [name, value] of Object.entries(vars)) {
|
|
4272
|
+
if (!override && process.env[name] !== void 0) continue;
|
|
4273
|
+
process.env[name] = value;
|
|
4274
|
+
applied.push(name);
|
|
4275
|
+
}
|
|
4276
|
+
return applied;
|
|
4277
|
+
}
|
|
4278
|
+
//#endregion
|
|
4098
4279
|
//#region src/cli/options.ts
|
|
4099
4280
|
/**
|
|
4100
4281
|
* Shared `--language` flag. Every Claude-driven command writes some
|
|
@@ -4105,6 +4286,53 @@ const CLIENT_JS = `
|
|
|
4105
4286
|
function addLanguageOption(command) {
|
|
4106
4287
|
return command.option("--language <bcp47>", "Language for human-readable output (e.g. 'en', 'ja'). Default 'auto' follows the language of the spec/codebase.", DEFAULT_LANGUAGE);
|
|
4107
4288
|
}
|
|
4289
|
+
/**
|
|
4290
|
+
* Shared `--profile <name>` flag for the browser-driving commands (`run`,
|
|
4291
|
+
* `record`), registered identically so help text and behaviour don't drift.
|
|
4292
|
+
*/
|
|
4293
|
+
function addProfileOption(command) {
|
|
4294
|
+
return command.option("--profile <name>", "Load .ccqa/profiles/<name>.env into the environment before resolving spec ${VAR} references (URLs, credentials), so one spec can target dev/stg/prd without per-environment copies. Profile values override the inherited environment.");
|
|
4295
|
+
}
|
|
4296
|
+
/**
|
|
4297
|
+
* Merge the environment for a `run` / `record` invocation into `process.env`
|
|
4298
|
+
* before any spec work. With `--profile <name>`, load that profile (missing /
|
|
4299
|
+
* invalid → exit 2). Without it, auto-load `<cwd>/.env` if present (a missing
|
|
4300
|
+
* `.env` is fine). Checking `!== undefined` rejects `--profile ""` rather than
|
|
4301
|
+
* skipping it.
|
|
4302
|
+
*/
|
|
4303
|
+
async function applyProfileFromOption(profile, cwd) {
|
|
4304
|
+
if (profile !== void 0) await applyNamedProfile(profile, cwd);
|
|
4305
|
+
else await applyDefaultEnv(cwd);
|
|
4306
|
+
}
|
|
4307
|
+
/** "1 var" / "2 vars" — the count summary shared by both load paths' meta line. */
|
|
4308
|
+
function varCount(n) {
|
|
4309
|
+
return `${n} var${n === 1 ? "" : "s"}`;
|
|
4310
|
+
}
|
|
4311
|
+
async function applyNamedProfile(profile, cwd) {
|
|
4312
|
+
try {
|
|
4313
|
+
const applied = applyProfileEnv(await loadProfileEnv(profile, cwd));
|
|
4314
|
+
meta("profile", `${profile} (${varCount(applied.length)})`);
|
|
4315
|
+
if (applied.length === 0) warn(`profile "${profile}" defined no variables — spec $\{VAR} references will resolve to empty`);
|
|
4316
|
+
} catch (err) {
|
|
4317
|
+
if (err instanceof ProfileNotFoundError) {
|
|
4318
|
+
error(err.message);
|
|
4319
|
+
hint(`create ${err.path} with the environment's $\{VAR} values`);
|
|
4320
|
+
} else if (err instanceof InvalidProfileNameError) error(err.message);
|
|
4321
|
+
else error(`failed to load profile "${profile}": ${err instanceof Error ? err.message : String(err)}`);
|
|
4322
|
+
process.exit(2);
|
|
4323
|
+
}
|
|
4324
|
+
}
|
|
4325
|
+
async function applyDefaultEnv(cwd) {
|
|
4326
|
+
let vars;
|
|
4327
|
+
try {
|
|
4328
|
+
vars = await loadDefaultEnv(cwd);
|
|
4329
|
+
} catch (err) {
|
|
4330
|
+
error(`failed to load ${defaultEnvPath(cwd)}: ${err instanceof Error ? err.message : String(err)}`);
|
|
4331
|
+
process.exit(2);
|
|
4332
|
+
}
|
|
4333
|
+
if (vars === null) return;
|
|
4334
|
+
meta("env", `.env (${varCount(applyProfileEnv(vars, { override: false }).length)})`);
|
|
4335
|
+
}
|
|
4108
4336
|
//#endregion
|
|
4109
4337
|
//#region src/cli/resolve-cwd.ts
|
|
4110
4338
|
/**
|
|
@@ -4116,7 +4344,7 @@ function addLanguageOption(command) {
|
|
|
4116
4344
|
*
|
|
4117
4345
|
* It's mostly useful in monorepos where you want to invoke ccqa from the
|
|
4118
4346
|
* repo root but target a subpackage (e.g.
|
|
4119
|
-
* `ccqa run --cwd
|
|
4347
|
+
* `ccqa run --cwd apps/web-app`).
|
|
4120
4348
|
*
|
|
4121
4349
|
* Falls back to `process.cwd()` when the option is not given.
|
|
4122
4350
|
*/
|
|
@@ -4358,6 +4586,12 @@ function stepArtifactPaths(runDir, stepId) {
|
|
|
4358
4586
|
//#endregion
|
|
4359
4587
|
//#region src/claude/agent-browser-invoke.ts
|
|
4360
4588
|
function agentBrowserInvokeBase(input) {
|
|
4589
|
+
const env = {
|
|
4590
|
+
AGENT_BROWSER_SESSION: input.sessionName,
|
|
4591
|
+
CCQA_RUN_ID: input.runId,
|
|
4592
|
+
PATH: pathWithAgentBrowserShim(process.env["PATH"])
|
|
4593
|
+
};
|
|
4594
|
+
if (input.statePath) env["CCQA_AB_STATE"] = input.statePath;
|
|
4361
4595
|
return {
|
|
4362
4596
|
allowedTools: [
|
|
4363
4597
|
"Bash(*)",
|
|
@@ -4365,17 +4599,19 @@ function agentBrowserInvokeBase(input) {
|
|
|
4365
4599
|
"Grep",
|
|
4366
4600
|
"Glob"
|
|
4367
4601
|
],
|
|
4368
|
-
env
|
|
4369
|
-
AGENT_BROWSER_SESSION: input.sessionName,
|
|
4370
|
-
CCQA_RUN_ID: input.runId,
|
|
4371
|
-
PATH: pathWithAgentBrowserShim(process.env["PATH"])
|
|
4372
|
-
}
|
|
4602
|
+
env
|
|
4373
4603
|
};
|
|
4374
4604
|
}
|
|
4375
4605
|
//#endregion
|
|
4376
4606
|
//#region src/prompts/live.ts
|
|
4607
|
+
/**
|
|
4608
|
+
* Unique agent-browser session name. The runId is millisecond-precision wall
|
|
4609
|
+
* clock, so under `--concurrency > 1` two specs can start in the same
|
|
4610
|
+
* millisecond and collide; a random suffix guarantees each spec gets its own
|
|
4611
|
+
* Chrome session and state never bleeds across parallel runs.
|
|
4612
|
+
*/
|
|
4377
4613
|
function generateLiveSessionName() {
|
|
4378
|
-
return `ccqa-live-${buildRunId()}`;
|
|
4614
|
+
return `ccqa-live-${buildRunId()}-${randomUUID().slice(0, 8)}`;
|
|
4379
4615
|
}
|
|
4380
4616
|
/**
|
|
4381
4617
|
* Static prefix of the `ccqa run` (live spec) system prompt. Built once per
|
|
@@ -4405,19 +4641,20 @@ function buildLiveSystemPromptPrefix(input) {
|
|
|
4405
4641
|
const stepsText = input.allSteps.map((s) => `### ${s.id} [${s.source}]
|
|
4406
4642
|
- **Instruction**: ${s.instruction}
|
|
4407
4643
|
- **Expected**: ${s.expected}`).join("\n\n");
|
|
4644
|
+
const stateLine = input.statePath ? `\n\nA pre-recorded auth-state file is provided at \`${input.statePath}\` (also in the env var \`CCQA_AB_STATE\`). **Always also pass \`--state "$CCQA_AB_STATE"\`** to every \`agent-browser\` command — this restores cookies and localStorage from a prior interactive login, so the user is already signed in to the application under test from step 1. The file is loaded read-only; do not run \`agent-browser state save\`.` : "";
|
|
4408
4645
|
return `You are a QA execution agent. You are executing ONE step of a browser-based end-to-end test and judging whether the step's expected outcome was achieved. You are NOT recording a replayable test script — be flexible, explore the DOM as needed, and make a clear pass / fail call at the end.
|
|
4409
4646
|
|
|
4410
4647
|
## Session
|
|
4411
4648
|
|
|
4412
4649
|
SESSION NAME: \`${input.sessionName}\`
|
|
4413
4650
|
|
|
4414
|
-
Always pass \`--session ${input.sessionName}\` to every \`agent-browser\` command. The session persists across steps within this test run, so the browser state from previous steps is already loaded when this turn starts
|
|
4651
|
+
Always pass \`--session ${input.sessionName}\` to every \`agent-browser\` command. The session persists across steps within this test run, so the browser state from previous steps is already loaded when this turn starts.${stateLine}
|
|
4415
4652
|
|
|
4416
4653
|
## Tools
|
|
4417
4654
|
|
|
4418
4655
|
You have:
|
|
4419
4656
|
|
|
4420
|
-
- **Bash** to run \`agent-browser\` (the full surface — \`open\`, \`snapshot\`, \`click\`, \`fill\`, \`press\`, \`wait\`, \`find\`, \`screenshot\`, \`eval\`, \`js\`, \`get\`, etc.). Any selector form is allowed: \`@ref\` (e.g. \`@e14\`), CSS selectors, \`text=...\`, \`[aria-label='...']\`, \`[data-testid='...']\`, bare tags inside \`find first/last/nth\` — whatever works for this single run. There is no replay contract to honour.
|
|
4657
|
+
- **Bash** to run \`agent-browser\` (the full surface — \`open\`, \`snapshot\`, \`click\`, \`fill\`, \`upload\`, \`press\`, \`wait\`, \`find\`, \`screenshot\`, \`eval\`, \`js\`, \`get\`, etc.). Any selector form is allowed: \`@ref\` (e.g. \`@e14\`), CSS selectors, \`text=...\`, \`[aria-label='...']\`, \`[data-testid='...']\`, bare tags inside \`find first/last/nth\` — whatever works for this single run. There is no replay contract to honour. For file inputs (\`<input type="file">\`) do NOT \`click\` the input — use \`agent-browser upload "<selector>" <path>\` so no OS file-picker dialog opens. Fixtures conventionally live under \`.ccqa/fixtures/\`; reference them via \`\${CCQA_FIXTURES_DIR}/<name>\`.
|
|
4421
4658
|
- **Read / Grep / Glob** for inspecting the application source code when you need to find a selector or understand routing. Read-only — do not modify source files.
|
|
4422
4659
|
|
|
4423
4660
|
## Test Specification
|
|
@@ -4525,11 +4762,9 @@ function findLastStepResult(text) {
|
|
|
4525
4762
|
* artifact, not a reason to abort the test step.
|
|
4526
4763
|
*/
|
|
4527
4764
|
function takeScreenshot(sessionName, outPath, options) {
|
|
4528
|
-
const args = [
|
|
4529
|
-
|
|
4530
|
-
|
|
4531
|
-
"screenshot"
|
|
4532
|
-
];
|
|
4765
|
+
const args = ["--session", sessionName];
|
|
4766
|
+
if (options?.statePath) args.push("--state", options.statePath);
|
|
4767
|
+
args.push("screenshot");
|
|
4533
4768
|
if (options?.fullPage) args.push("--full");
|
|
4534
4769
|
args.push(outPath);
|
|
4535
4770
|
const res = spawnAB(args);
|
|
@@ -4562,16 +4797,19 @@ async function runLiveExecutor(input) {
|
|
|
4562
4797
|
const startedAt = /* @__PURE__ */ new Date();
|
|
4563
4798
|
const stepResults = [];
|
|
4564
4799
|
let overallFailed = false;
|
|
4800
|
+
const statePath = input.statePath ?? null;
|
|
4565
4801
|
const promptPrefix = buildLiveSystemPromptPrefix({
|
|
4566
4802
|
title: input.spec.title,
|
|
4567
4803
|
allSteps: input.steps,
|
|
4568
|
-
sessionName: input.sessionName
|
|
4804
|
+
sessionName: input.sessionName,
|
|
4805
|
+
statePath
|
|
4569
4806
|
});
|
|
4570
4807
|
const suffixBlock = input.systemPromptSuffix ? `\n## Project-specific guidance\n\n${input.systemPromptSuffix}\n` : "";
|
|
4571
4808
|
const langDirective = languageDirective(input.language);
|
|
4572
4809
|
const invokeBase = agentBrowserInvokeBase({
|
|
4573
4810
|
sessionName: input.sessionName,
|
|
4574
|
-
runId: input.runId
|
|
4811
|
+
runId: input.runId,
|
|
4812
|
+
statePath
|
|
4575
4813
|
});
|
|
4576
4814
|
const retries = Math.max(0, input.retries ?? 0);
|
|
4577
4815
|
for (let i = 0; i < input.steps.length; i++) {
|
|
@@ -4616,7 +4854,7 @@ async function runLiveExecutor(input) {
|
|
|
4616
4854
|
}
|
|
4617
4855
|
}
|
|
4618
4856
|
async function executeStepAttempt(step, paths, systemPrompt, userPrompt) {
|
|
4619
|
-
const before = takeScreenshot(input.sessionName, paths.beforePng);
|
|
4857
|
+
const before = takeScreenshot(input.sessionName, paths.beforePng, { statePath });
|
|
4620
4858
|
if (!before.ok) warn(`screenshot (before, ${step.id}) failed: ${before.error}`);
|
|
4621
4859
|
const transcriptParts = [];
|
|
4622
4860
|
let isError = false;
|
|
@@ -4648,7 +4886,10 @@ async function runLiveExecutor(input) {
|
|
|
4648
4886
|
transcriptParts.push(`[ccqa] invokeClaudeStreaming threw: ${err instanceof Error ? err.message : String(err)}`);
|
|
4649
4887
|
}
|
|
4650
4888
|
const transcript = transcriptParts.join("\n");
|
|
4651
|
-
const after = takeScreenshot(input.sessionName, paths.afterPng, {
|
|
4889
|
+
const after = takeScreenshot(input.sessionName, paths.afterPng, {
|
|
4890
|
+
fullPage: true,
|
|
4891
|
+
statePath
|
|
4892
|
+
});
|
|
4652
4893
|
if (!after.ok) warn(`screenshot (after, ${step.id}) failed: ${after.error}`);
|
|
4653
4894
|
await writeFile(paths.logTxt, transcript || "(no assistant text captured)", "utf-8");
|
|
4654
4895
|
const { status, reasoning } = judgeStepOutcome({
|
|
@@ -4840,24 +5081,24 @@ async function runLiveSpecs(specs, opts) {
|
|
|
4840
5081
|
await preflightAgentBrowserCommand();
|
|
4841
5082
|
meta("live-specs", specs.length);
|
|
4842
5083
|
const userPromptBundle = await loadLivePromptBundle(cwd);
|
|
4843
|
-
if (userPromptBundle !== null) meta("
|
|
5084
|
+
if (userPromptBundle !== null) meta("prompt", userPromptBundle.loaded.join(" + "));
|
|
4844
5085
|
const userPromptSuffix = userPromptBundle?.text ?? null;
|
|
4845
|
-
const
|
|
4846
|
-
|
|
4847
|
-
const {
|
|
4848
|
-
|
|
4849
|
-
|
|
4850
|
-
|
|
4851
|
-
|
|
4852
|
-
|
|
4853
|
-
|
|
4854
|
-
|
|
4855
|
-
|
|
4856
|
-
|
|
4857
|
-
|
|
4858
|
-
|
|
4859
|
-
})
|
|
4860
|
-
}
|
|
5086
|
+
const concurrency = Math.max(1, opts.concurrency ?? 1);
|
|
5087
|
+
const runs = await runPool(specs, concurrency, (spec, i) => {
|
|
5088
|
+
const label = `${spec.featureName}/${spec.specName}`;
|
|
5089
|
+
return withBuffer(label, concurrency > 1, () => {
|
|
5090
|
+
if (concurrency === 1 && specs.length > 1) {
|
|
5091
|
+
blank();
|
|
5092
|
+
info(`[${i + 1}/${specs.length}] ${label}`);
|
|
5093
|
+
}
|
|
5094
|
+
return runOneSpec({
|
|
5095
|
+
...spec,
|
|
5096
|
+
opts,
|
|
5097
|
+
userPromptSuffix,
|
|
5098
|
+
cwd
|
|
5099
|
+
});
|
|
5100
|
+
});
|
|
5101
|
+
});
|
|
4861
5102
|
const failedCount = runs.filter((r) => r.kind === "error" || r.kind === "run" && r.result.status === "failed").length;
|
|
4862
5103
|
blank();
|
|
4863
5104
|
meta("live-summary", `${runs.length - failedCount} passed / ${failedCount} failed`);
|
|
@@ -4956,6 +5197,23 @@ async function runOneSpec(args) {
|
|
|
4956
5197
|
if (includes.length > 0) meta("blocks", includes.join(", "));
|
|
4957
5198
|
const sessionName = generateLiveSessionName();
|
|
4958
5199
|
meta("session", sessionName);
|
|
5200
|
+
let statePath = null;
|
|
5201
|
+
if (spec.statePath) {
|
|
5202
|
+
statePath = isAbsolute(spec.statePath) ? spec.statePath : resolve(cwd, spec.statePath);
|
|
5203
|
+
try {
|
|
5204
|
+
await access(statePath);
|
|
5205
|
+
} catch {
|
|
5206
|
+
const msg = `spec.statePath points to a missing file: ${statePath}`;
|
|
5207
|
+
error(msg);
|
|
5208
|
+
return {
|
|
5209
|
+
kind: "error",
|
|
5210
|
+
featureName,
|
|
5211
|
+
specName,
|
|
5212
|
+
error: msg
|
|
5213
|
+
};
|
|
5214
|
+
}
|
|
5215
|
+
meta("state", statePath);
|
|
5216
|
+
}
|
|
4959
5217
|
const runId = buildRunId();
|
|
4960
5218
|
const runDir = opts.out ?? join(specDir, "runs", runId);
|
|
4961
5219
|
await mkdir(runDir, { recursive: true });
|
|
@@ -4966,6 +5224,7 @@ async function runOneSpec(args) {
|
|
|
4966
5224
|
runId,
|
|
4967
5225
|
runDir,
|
|
4968
5226
|
sessionName,
|
|
5227
|
+
statePath,
|
|
4969
5228
|
systemPromptSuffix: userPromptSuffix,
|
|
4970
5229
|
model: opts.model,
|
|
4971
5230
|
language: opts.language,
|
|
@@ -5231,28 +5490,57 @@ async function resolveVitestConfig(cwd) {
|
|
|
5231
5490
|
return bundledVitestConfigPath();
|
|
5232
5491
|
}
|
|
5233
5492
|
}
|
|
5234
|
-
const runCommand = addLanguageOption(new Command("run").argument("[
|
|
5493
|
+
const runCommand = addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs. Each spec's execution mode comes from its spec.yaml `mode:` field (default deterministic; set `mode: live` to have Claude drive agent-browser live per step). Deterministic specs replay the recorded test.spec.ts under vitest. Pass --report to write one unified HTML report covering both modes.").option("--report [dir]", `Write a self-contained HTML run report (failure analysis + drift audit by default). Default dir: ${DEFAULT_REPORT_DIR}/`).option("--changed", "Restrict execution to specs whose relatedPaths intersect the git diff against --base (or, in CI, $GITHUB_BASE_REF, else origin/main). Cannot be combined with an explicit spec id.").option("--no-failure-analysis", "Skip the per-failure root-cause classification (TEST_DRIFT / SPEC_CHANGE / PRODUCT_BUG). --report only.").option("--no-drift-audit", "Skip the spec↔code drift audit shown in the report. --report only.").option("--base <ref>", "Base ref the source diff is taken against for failure analysis (default: GITHUB_BASE_REF, then origin/main).").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--format <fmt>", "Additional output format alongside HTML when --report is set: 'text' (default), 'json' (writes report.json), 'github' (GitHub Actions annotations on stdout).", (raw) => {
|
|
5235
5494
|
if (REPORT_FORMATS.includes(raw)) return raw;
|
|
5236
5495
|
throw new Error(`--format must be one of ${REPORT_FORMATS.join(" | ")}`);
|
|
5237
5496
|
}, "text").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--no-evidence", `(deterministic only) Skip step-boundary evidence capture (PNG + meta JSON written to ${DEFAULT_REPORT_DIR}/${EVIDENCE_SUBDIR}/ by default).`).option("--retry <n>", "(live only) Retry each failed step up to N more times before recording failure. Default 0.", (raw) => {
|
|
5238
5497
|
const n = Number(raw);
|
|
5239
5498
|
if (!Number.isFinite(n) || n < 0 || Math.floor(n) !== n) throw new Error(`--retry must be a non-negative integer, got "${raw}"`);
|
|
5240
5499
|
return n;
|
|
5241
|
-
}, 0).option("--out <dir>", "(live only) Override the per-spec artifact directory. Default: <specDir>/runs/<runId>. Ignored when running multiple specs.").option("--update-agent-prompt", "(live only) After the run finishes, ask Claude to refresh .ccqa/prompts/live.agent.md from a summary of the run.")).action(async (
|
|
5242
|
-
await runDispatcher(
|
|
5500
|
+
}, 0).option("--out <dir>", "(live only) Override the per-spec artifact directory. Default: <specDir>/runs/<runId>. Ignored when running multiple specs.").option("--update-agent-prompt", "(live only) After the run finishes, ask Claude to refresh .ccqa/prompts/live.agent.md from a summary of the run.").option("--concurrency <n>", "Run up to N specs in parallel within each mode (deterministic / live). Default 1 (sequential). Live specs each get an isolated agent-browser session; high values spawn many headed Chrome instances.", parseConcurrency$1, 1))).action(async (targets, opts) => {
|
|
5501
|
+
await runDispatcher(targets, opts);
|
|
5243
5502
|
});
|
|
5503
|
+
/** Parse --concurrency: a positive integer. Rejects 0, negatives, non-integers. */
|
|
5504
|
+
function parseConcurrency$1(raw) {
|
|
5505
|
+
const n = Number(raw);
|
|
5506
|
+
if (!Number.isInteger(n) || n < 1) {
|
|
5507
|
+
error(`invalid --concurrency: ${raw} (expected positive integer)`);
|
|
5508
|
+
process.exit(2);
|
|
5509
|
+
}
|
|
5510
|
+
return n;
|
|
5511
|
+
}
|
|
5244
5512
|
function resolveReportDir(report, cwd) {
|
|
5245
5513
|
if (report === void 0 || report === false) return void 0;
|
|
5246
5514
|
return resolve(cwd, typeof report === "string" ? report : DEFAULT_REPORT_DIR);
|
|
5247
5515
|
}
|
|
5248
|
-
|
|
5249
|
-
|
|
5250
|
-
if (
|
|
5516
|
+
/** Header label shown after `ccqa run`: the lone target, a count, or a mode marker. */
|
|
5517
|
+
function headerTarget(targets, opts) {
|
|
5518
|
+
if (targets.length === 1) return targets[0];
|
|
5519
|
+
if (targets.length > 1) return `${targets.length} targets`;
|
|
5520
|
+
return opts.changed ? "(changed)" : "(all specs)";
|
|
5521
|
+
}
|
|
5522
|
+
/** De-dupe by `featureName/specName`, keeping first-seen order. */
|
|
5523
|
+
function dedupeSpecs(specs) {
|
|
5524
|
+
const seen = /* @__PURE__ */ new Set();
|
|
5525
|
+
const out = [];
|
|
5526
|
+
for (const s of specs) {
|
|
5527
|
+
const key = `${s.featureName}/${s.specName}`;
|
|
5528
|
+
if (seen.has(key)) continue;
|
|
5529
|
+
seen.add(key);
|
|
5530
|
+
out.push(s);
|
|
5531
|
+
}
|
|
5532
|
+
return out;
|
|
5533
|
+
}
|
|
5534
|
+
async function runDispatcher(targets, opts) {
|
|
5535
|
+
header("run", headerTarget(targets, opts));
|
|
5536
|
+
if (opts.changed && targets.length > 0) {
|
|
5251
5537
|
error("--changed and an explicit spec target cannot be combined");
|
|
5252
5538
|
process.exit(2);
|
|
5253
5539
|
}
|
|
5254
5540
|
const cwd = resolveCwd(opts.cwd);
|
|
5255
|
-
|
|
5541
|
+
await applyProfileFromOption(opts.profile, cwd);
|
|
5542
|
+
const enumerateAll = () => listAllSpecsWithSpecFile(cwd);
|
|
5543
|
+
let specs = dedupeSpecs((await Promise.all((targets.length ? targets : [void 0]).map((t) => resolveSpecTargets(t, enumerateAll, cwd)))).flat());
|
|
5256
5544
|
if (opts.changed) {
|
|
5257
5545
|
const before = specs.length;
|
|
5258
5546
|
specs = await collectChangedSpecs(specs, {
|
|
@@ -5273,7 +5561,7 @@ async function runDispatcher(target, opts) {
|
|
|
5273
5561
|
if (typeof opts.retry === "number" && opts.retry > 0) warn("--retry is ignored without any 'mode: live' spec");
|
|
5274
5562
|
if (opts.out) warn("--out is ignored without any 'mode: live' spec");
|
|
5275
5563
|
if (opts.updateAgentPrompt) warn("--update-agent-prompt is ignored without any 'mode: live' spec");
|
|
5276
|
-
}
|
|
5564
|
+
} else if (opts.out && liveSpecs.length > 1) warn("--out is ignored when running multiple live specs");
|
|
5277
5565
|
if (detSpecs.length === 0 && opts.evidence === false) warn("--no-evidence is ignored without any 'mode: deterministic' spec");
|
|
5278
5566
|
blank();
|
|
5279
5567
|
const reportDir = resolveReportDir(opts.report, cwd);
|
|
@@ -5282,11 +5570,12 @@ async function runDispatcher(target, opts) {
|
|
|
5282
5570
|
const live = await runLiveSpecs(liveSpecs, {
|
|
5283
5571
|
...opts.model ? { model: opts.model } : {},
|
|
5284
5572
|
...opts.language ? { language: opts.language } : {},
|
|
5285
|
-
...opts.out ? { out: opts.out } : {},
|
|
5573
|
+
...opts.out && liveSpecs.length === 1 ? { out: opts.out } : {},
|
|
5286
5574
|
cwd,
|
|
5287
5575
|
...opts.base ? { base: opts.base } : {},
|
|
5288
5576
|
...reportDir ? { reportDir } : {},
|
|
5289
5577
|
...typeof opts.retry === "number" ? { retry: opts.retry } : {},
|
|
5578
|
+
concurrency: opts.concurrency ?? 1,
|
|
5290
5579
|
...reportDir && opts.driftAudit !== false ? { driftAudit: true } : {},
|
|
5291
5580
|
...reportDir && opts.failureAnalysis === false ? { failureAnalysis: false } : {}
|
|
5292
5581
|
});
|
|
@@ -5345,72 +5634,83 @@ async function runDeterministicSpecs(specs, opts, cwd, reportDirAbs) {
|
|
|
5345
5634
|
exitCode: 0
|
|
5346
5635
|
};
|
|
5347
5636
|
const tmpDir = await mkdtemp(join(tmpdir(), "ccqa-run-"));
|
|
5348
|
-
const summaries = [];
|
|
5349
|
-
let exitCode = 0;
|
|
5350
5637
|
const vitestConfig = await resolveVitestConfig(cwd);
|
|
5351
5638
|
const captureOutput = Boolean(opts.report);
|
|
5352
5639
|
const evidenceRoot = opts.evidence !== false ? join(reportDirAbs, EVIDENCE_SUBDIR) : null;
|
|
5640
|
+
const concurrency = Math.max(1, opts.concurrency ?? 1);
|
|
5641
|
+
const ctx = {
|
|
5642
|
+
cwd,
|
|
5643
|
+
tmpDir,
|
|
5644
|
+
vitestConfig,
|
|
5645
|
+
captureOutput,
|
|
5646
|
+
evidenceRoot
|
|
5647
|
+
};
|
|
5353
5648
|
try {
|
|
5354
|
-
|
|
5355
|
-
const { featureName, specName } = specs[i];
|
|
5356
|
-
const scriptFile = await getTestScript(featureName, specName, cwd);
|
|
5357
|
-
if (!scriptFile) {
|
|
5358
|
-
warn(`${featureName}/${specName}: no test.spec.ts found`);
|
|
5359
|
-
hint("run 'ccqa record <feature>/<spec>' to record it, or set 'mode: live' in spec.yaml");
|
|
5360
|
-
continue;
|
|
5361
|
-
}
|
|
5362
|
-
run(`${featureName}/${specName}`);
|
|
5363
|
-
meta("test", scriptFile);
|
|
5364
|
-
blank();
|
|
5365
|
-
const reportFile = join(tmpDir, `report-${i}.json`);
|
|
5366
|
-
const evidenceDir = evidenceRoot ? join(evidenceRoot, featureName, specName) : null;
|
|
5367
|
-
if (evidenceDir) {
|
|
5368
|
-
await rm(evidenceDir, {
|
|
5369
|
-
recursive: true,
|
|
5370
|
-
force: true
|
|
5371
|
-
});
|
|
5372
|
-
await mkdir(evidenceDir, { recursive: true });
|
|
5373
|
-
}
|
|
5374
|
-
const proc = spawnVitestStreaming([
|
|
5375
|
-
"run",
|
|
5376
|
-
"--config",
|
|
5377
|
-
vitestConfig,
|
|
5378
|
-
scriptFile,
|
|
5379
|
-
"--reporter=json",
|
|
5380
|
-
`--outputFile.json=${reportFile}`
|
|
5381
|
-
], {
|
|
5382
|
-
cwd,
|
|
5383
|
-
env: evidenceDir ? {
|
|
5384
|
-
...process.env,
|
|
5385
|
-
CCQA_EVIDENCE_DIR: evidenceDir
|
|
5386
|
-
} : process.env
|
|
5387
|
-
});
|
|
5388
|
-
const tail = captureOutput ? new TailBuffer(OUTPUT_TAIL_CAP) : null;
|
|
5389
|
-
await Promise.all([streamFiltered(proc.stdout, process.stdout, tail), streamFiltered(proc.stderr, process.stderr, tail)]);
|
|
5390
|
-
const specExitCode = await proc.exited;
|
|
5391
|
-
if (specExitCode !== 0) exitCode = specExitCode;
|
|
5392
|
-
const report = await readReport(reportFile);
|
|
5393
|
-
summaries.push({
|
|
5394
|
-
featureName,
|
|
5395
|
-
specName,
|
|
5396
|
-
scriptFile,
|
|
5397
|
-
report,
|
|
5398
|
-
exitCode: specExitCode,
|
|
5399
|
-
outputTail: tail ? tail.toString() : null,
|
|
5400
|
-
evidenceDir
|
|
5401
|
-
});
|
|
5402
|
-
blank();
|
|
5403
|
-
}
|
|
5649
|
+
const summaries = (await runPool(specs, concurrency, (spec, i) => withBuffer(`${spec.featureName}/${spec.specName}`, concurrency > 1, () => runOneDeterministicSpec(spec, i, ctx)))).filter((s) => s !== null);
|
|
5404
5650
|
printSummary(summaries);
|
|
5651
|
+
return {
|
|
5652
|
+
summaries,
|
|
5653
|
+
exitCode: summaries.reduce((acc, s) => s.exitCode !== 0 ? s.exitCode : acc, 0)
|
|
5654
|
+
};
|
|
5405
5655
|
} finally {
|
|
5406
5656
|
await rm(tmpDir, {
|
|
5407
5657
|
recursive: true,
|
|
5408
5658
|
force: true
|
|
5409
5659
|
});
|
|
5410
5660
|
}
|
|
5661
|
+
}
|
|
5662
|
+
/**
|
|
5663
|
+
* Run one spec under vitest. Returns null when the spec has no recorded
|
|
5664
|
+
* test.spec.ts (skipped). All output goes through the logger, so under a
|
|
5665
|
+
* `log.withBuffer` scope it's captured and flushed as one labelled block.
|
|
5666
|
+
*/
|
|
5667
|
+
async function runOneDeterministicSpec(spec, index, ctx) {
|
|
5668
|
+
const { featureName, specName } = spec;
|
|
5669
|
+
const scriptFile = await getTestScript(featureName, specName, ctx.cwd);
|
|
5670
|
+
if (!scriptFile) {
|
|
5671
|
+
warn(`${featureName}/${specName}: no test.spec.ts found`);
|
|
5672
|
+
hint("run 'ccqa record <feature>/<spec>' to record it, or set 'mode: live' in spec.yaml");
|
|
5673
|
+
return null;
|
|
5674
|
+
}
|
|
5675
|
+
run(`${featureName}/${specName}`);
|
|
5676
|
+
meta("test", scriptFile);
|
|
5677
|
+
blank();
|
|
5678
|
+
const reportFile = join(ctx.tmpDir, `report-${index}.json`);
|
|
5679
|
+
const evidenceDir = ctx.evidenceRoot ? join(ctx.evidenceRoot, featureName, specName) : null;
|
|
5680
|
+
if (evidenceDir) {
|
|
5681
|
+
await rm(evidenceDir, {
|
|
5682
|
+
recursive: true,
|
|
5683
|
+
force: true
|
|
5684
|
+
});
|
|
5685
|
+
await mkdir(evidenceDir, { recursive: true });
|
|
5686
|
+
}
|
|
5687
|
+
const proc = spawnVitestStreaming([
|
|
5688
|
+
"run",
|
|
5689
|
+
"--config",
|
|
5690
|
+
ctx.vitestConfig,
|
|
5691
|
+
scriptFile,
|
|
5692
|
+
"--reporter=json",
|
|
5693
|
+
`--outputFile.json=${reportFile}`
|
|
5694
|
+
], {
|
|
5695
|
+
cwd: ctx.cwd,
|
|
5696
|
+
env: evidenceDir ? {
|
|
5697
|
+
...process.env,
|
|
5698
|
+
CCQA_EVIDENCE_DIR: evidenceDir
|
|
5699
|
+
} : process.env
|
|
5700
|
+
});
|
|
5701
|
+
const sink = { write: emitRaw };
|
|
5702
|
+
const tail = ctx.captureOutput ? new TailBuffer(OUTPUT_TAIL_CAP) : null;
|
|
5703
|
+
await Promise.all([streamFiltered(proc.stdout, sink, tail), streamFiltered(proc.stderr, sink, tail)]);
|
|
5704
|
+
const specExitCode = await proc.exited;
|
|
5705
|
+
blank();
|
|
5411
5706
|
return {
|
|
5412
|
-
|
|
5413
|
-
|
|
5707
|
+
featureName,
|
|
5708
|
+
specName,
|
|
5709
|
+
scriptFile,
|
|
5710
|
+
report: await readReport(reportFile),
|
|
5711
|
+
exitCode: specExitCode,
|
|
5712
|
+
outputTail: tail ? tail.toString() : null,
|
|
5713
|
+
evidenceDir
|
|
5414
5714
|
};
|
|
5415
5715
|
}
|
|
5416
5716
|
function failedSpec(s) {
|
|
@@ -5859,6 +6159,7 @@ agent-browser --session SESSION wait --load networkidle
|
|
|
5859
6159
|
agent-browser --session SESSION get count "<selector>" # element-existence check (returns a number, fast)
|
|
5860
6160
|
agent-browser --session SESSION cookies clear
|
|
5861
6161
|
agent-browser --session SESSION find <locator> <value> <action> [<input>] [--name "<n>"] [--exact]
|
|
6162
|
+
agent-browser --session SESSION upload "<input[type=file] selector>" <file> [<file> ...]
|
|
5862
6163
|
# See "Selector Rules" for the full \`find\` subset.
|
|
5863
6164
|
# IMPORTANT: do NOT use \`wait "<css-selector>"\`. agent-browser ignores --timeout on a
|
|
5864
6165
|
# CSS-selector wait and blocks for ~150s when the selector never matches, killing the run.
|
|
@@ -5934,6 +6235,8 @@ find nth <index> "<ALLOWED-css>" <action>
|
|
|
5934
6235
|
|
|
5935
6236
|
**Verifying cleanup / deletion**: assert the *absence* of the deleted thing, not the surrounding listing screen's text. Use \`wait --fn "!document.body.innerText.includes('<unique-label>')"\` (text disappearance) — never \`wait "<css-selector>" --state hidden\` (blocks the daemon) and never \`wait --text "<navbar label>"\` (passes regardless of the deletion).
|
|
5936
6237
|
|
|
6238
|
+
**File inputs (\`<input type="file">\`) / OS file-picker dialogs**: do NOT \`click\` the input — that opens the OS picker, which agent-browser cannot drive. Use \`upload "<selector>" <path>\` instead. agent-browser sets the input's files directly via the underlying browser API, no native dialog ever opens. Use an ALLOWED selector to identify the input (\`[aria-label='…']\`, \`[data-testid='…']\`, \`[type='file']\` only when it's unique on the page). File paths must be plain shell args — wrap each in \`"\` for safety. Reference fixtures via \`\${CCQA_FIXTURES_DIR}/<name>\` so the same spec works locally and in CI; conventionally fixtures live under \`.ccqa/fixtures/\` and the env var resolves there. Multi-file inputs accept several positionals: \`upload "[aria-label='Attach']" "\${CCQA_FIXTURES_DIR}/a.pdf" "\${CCQA_FIXTURES_DIR}/b.pdf"\`.
|
|
6239
|
+
|
|
5937
6240
|
## Test Specification
|
|
5938
6241
|
|
|
5939
6242
|
Title: ${input.title}
|
|
@@ -6016,6 +6319,7 @@ AB_ACTION|select|<selector>|<value>|<aria label>
|
|
|
6016
6319
|
AB_ACTION|hover|<selector>|<visible label>
|
|
6017
6320
|
AB_ACTION|scroll|<direction>|<pixels>
|
|
6018
6321
|
AB_ACTION|drag|<source selector>|<target selector>|<source label>
|
|
6322
|
+
AB_ACTION|upload|<file-input selector>|<file1>[|<file2>...]
|
|
6019
6323
|
AB_ACTION|wait|<selector or text>|<label>
|
|
6020
6324
|
AB_ACTION|snapshot|<key observation, max 100 chars>
|
|
6021
6325
|
AB_ACTION|assert|<assertType>|<selector or "">|<value or "">|<observation>
|
|
@@ -6332,6 +6636,17 @@ function actionToAbArgs(action, sessionName) {
|
|
|
6332
6636
|
sub(action.selector),
|
|
6333
6637
|
sub(action.target)
|
|
6334
6638
|
];
|
|
6639
|
+
case "upload": {
|
|
6640
|
+
const sel = sub(action.selector);
|
|
6641
|
+
const files = (action.files ?? []).map((f) => sub(f));
|
|
6642
|
+
if (!sel || files.length === 0) return null;
|
|
6643
|
+
return [
|
|
6644
|
+
...base,
|
|
6645
|
+
"upload",
|
|
6646
|
+
sel,
|
|
6647
|
+
...files
|
|
6648
|
+
];
|
|
6649
|
+
}
|
|
6335
6650
|
case "wait": {
|
|
6336
6651
|
const raw = sub(action.selector);
|
|
6337
6652
|
if (!raw) return null;
|
|
@@ -6824,7 +7139,7 @@ async function runTrace(featureName, specName, model, validationMode = "lenient"
|
|
|
6824
7139
|
sessionName
|
|
6825
7140
|
});
|
|
6826
7141
|
const promptBundle = await loadRecordPromptBundle();
|
|
6827
|
-
if (promptBundle !== null) meta("
|
|
7142
|
+
if (promptBundle !== null) meta("prompt", promptBundle.loaded.join(" + "));
|
|
6828
7143
|
const systemPrompt = (promptBundle === null ? baseSystemPrompt : `${baseSystemPrompt}\n## Project-specific guidance\n\n${promptBundle.text}\n`) + languageDirective(language);
|
|
6829
7144
|
const prompt = buildTracePrompt(spec.title);
|
|
6830
7145
|
info("Running agent-browser session...");
|
|
@@ -6970,7 +7285,7 @@ function dedupAndReport(actions) {
|
|
|
6970
7285
|
function isAdjacentDuplicate(a, b) {
|
|
6971
7286
|
if (a.command !== b.command) return false;
|
|
6972
7287
|
if ((a.stepId ?? "") !== (b.stepId ?? "")) return false;
|
|
6973
|
-
return (a.selector ?? "") === (b.selector ?? "") && (a.value ?? "") === (b.value ?? "") && (a.target ?? "") === (b.target ?? "") && (a.label ?? "") === (b.label ?? "") && (a.assertType ?? "") === (b.assertType ?? "") && (a.findLocator ?? "") === (b.findLocator ?? "") && (a.findValue ?? "") === (b.findValue ?? "") && (a.findName ?? "") === (b.findName ?? "") && (a.findIndex ?? -1) === (b.findIndex ?? -1) && (a.findExact ?? false) === (b.findExact ?? false);
|
|
7288
|
+
return (a.selector ?? "") === (b.selector ?? "") && (a.value ?? "") === (b.value ?? "") && (a.target ?? "") === (b.target ?? "") && (a.label ?? "") === (b.label ?? "") && (a.assertType ?? "") === (b.assertType ?? "") && (a.findLocator ?? "") === (b.findLocator ?? "") && (a.findValue ?? "") === (b.findValue ?? "") && (a.findName ?? "") === (b.findName ?? "") && (a.findIndex ?? -1) === (b.findIndex ?? -1) && (a.findExact ?? false) === (b.findExact ?? false) && (a.files ?? []).join("|") === (b.files ?? []).join("|");
|
|
6974
7289
|
}
|
|
6975
7290
|
/**
|
|
6976
7291
|
* Run the post-trace replay validation and emit user-visible drop reports.
|
|
@@ -7192,6 +7507,16 @@ function parseAbAction(line) {
|
|
|
7192
7507
|
target: parts[3],
|
|
7193
7508
|
label: parts[4]
|
|
7194
7509
|
};
|
|
7510
|
+
case "upload": {
|
|
7511
|
+
const selector = parts[2];
|
|
7512
|
+
const files = parts.slice(3).filter((f) => f !== "");
|
|
7513
|
+
if (!selector || files.length === 0) return null;
|
|
7514
|
+
return {
|
|
7515
|
+
command,
|
|
7516
|
+
selector,
|
|
7517
|
+
files
|
|
7518
|
+
};
|
|
7519
|
+
}
|
|
7195
7520
|
case "find_click":
|
|
7196
7521
|
case "find_dblclick":
|
|
7197
7522
|
case "find_hover":
|
|
@@ -7242,6 +7567,7 @@ function actionsToScript(input) {
|
|
|
7242
7567
|
`import { ${[
|
|
7243
7568
|
"ab",
|
|
7244
7569
|
"abWait",
|
|
7570
|
+
"abUpload",
|
|
7245
7571
|
"abAssertTextVisible",
|
|
7246
7572
|
"abAssertVisible",
|
|
7247
7573
|
"abAssertNotVisible",
|
|
@@ -7275,6 +7601,7 @@ const ELEMENT_COMMANDS = new Set([
|
|
|
7275
7601
|
"select",
|
|
7276
7602
|
"hover",
|
|
7277
7603
|
"drag",
|
|
7604
|
+
"upload",
|
|
7278
7605
|
"find_click",
|
|
7279
7606
|
"find_dblclick",
|
|
7280
7607
|
"find_fill",
|
|
@@ -7406,6 +7733,11 @@ function actionToLine(action) {
|
|
|
7406
7733
|
case "hover": return `ab("hover", ${j(action.selector)});`;
|
|
7407
7734
|
case "scroll": return `ab("scroll", ${[action.direction ?? "down", ...action.pixels ? [action.pixels] : []].map(j).join(", ")});`;
|
|
7408
7735
|
case "drag": return `ab("drag", ${j(action.selector)}, ${j(action.target)});`;
|
|
7736
|
+
case "upload": {
|
|
7737
|
+
const files = action.files ?? [];
|
|
7738
|
+
if (!action.selector || files.length === 0) return null;
|
|
7739
|
+
return `abUpload(${[j(action.selector), ...files.map(jExpr)].join(", ")});`;
|
|
7740
|
+
}
|
|
7409
7741
|
case "wait": {
|
|
7410
7742
|
const sel = action.selector;
|
|
7411
7743
|
if (/^\d+$/.test(sel)) return `spawnSync("sleep", [${j(sel)}], { stdio: "inherit" });`;
|
|
@@ -8481,19 +8813,20 @@ function toFixMode(autoFix) {
|
|
|
8481
8813
|
case "interactive": return "interactive";
|
|
8482
8814
|
}
|
|
8483
8815
|
}
|
|
8484
|
-
const recordCommand = addLanguageOption(new Command("record").argument("<feature/spec>", "Spec id in '<feature>/<spec>' form (resolves to .ccqa/features/<feature>/test-cases/<spec>/)").description("Record a deterministic test from a spec: run agent-browser to collect actions (trace), then generate test.spec.ts with auto-fix retries (generate). After recording, `ccqa run <feature/spec>` replays it under vitest (deterministic specs only — live specs do not need recording).").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--validation-mode <mode>", "Post-trace validation behaviour: 'lenient' (default) tags failing actions; 'strict' drops them.", (raw) => {
|
|
8816
|
+
const recordCommand = addProfileOption(addLanguageOption(new Command("record").argument("<feature/spec>", "Spec id in '<feature>/<spec>' form (resolves to .ccqa/features/<feature>/test-cases/<spec>/)").description("Record a deterministic test from a spec: run agent-browser to collect actions (trace), then generate test.spec.ts with auto-fix retries (generate). After recording, `ccqa run <feature/spec>` replays it under vitest (deterministic specs only — live specs do not need recording).").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--validation-mode <mode>", "Post-trace validation behaviour: 'lenient' (default) tags failing actions; 'strict' drops them.", (raw) => {
|
|
8485
8817
|
if (VALIDATION_MODES.includes(raw)) return raw;
|
|
8486
8818
|
throw new Error(`--validation-mode must be one of ${VALIDATION_MODES.join(" | ")}`);
|
|
8487
8819
|
}, "lenient").option("--auto-fix <mode>", "Auto-fix behaviour during script generation: 'interactive' (default, prompt y/N), 'auto' (apply without prompt, for CI), 'skip' (never prompt, only apply high-confidence fixes).", (raw) => {
|
|
8488
8820
|
if (AUTO_FIX_MODES.includes(raw)) return raw;
|
|
8489
8821
|
throw new Error(`--auto-fix must be one of ${AUTO_FIX_MODES.join(" | ")}`);
|
|
8490
|
-
}, "interactive").option("--max-retries <n>", "Maximum number of auto-fix retries", "3").option("--force", "Overwrite an existing test.spec.ts without warning").option("--no-snapshot", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").option("--skip-trace", "Skip the trace step and run codegen against an existing actions.json").option("--skip-codegen", "Run only the trace step (do not generate test.spec.ts)").option("--update-agent-prompt", "After the trace finishes, ask Claude to refresh .ccqa/prompts/record.agent.md from a summary of the run.").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.")).action(async (specPath, opts) => {
|
|
8822
|
+
}, "interactive").option("--max-retries <n>", "Maximum number of auto-fix retries", "3").option("--force", "Overwrite an existing test.spec.ts without warning").option("--no-snapshot", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").option("--skip-trace", "Skip the trace step and run codegen against an existing actions.json").option("--skip-codegen", "Run only the trace step (do not generate test.spec.ts)").option("--update-agent-prompt", "After the trace finishes, ask Claude to refresh .ccqa/prompts/record.agent.md from a summary of the run.").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory."))).action(async (specPath, opts) => {
|
|
8491
8823
|
const { featureName, specName } = parseSpecPath(specPath);
|
|
8492
8824
|
const language = opts.language ?? "auto";
|
|
8493
8825
|
if (opts.skipTrace && opts.skipCodegen) {
|
|
8494
8826
|
error("--skip-trace and --skip-codegen cannot be combined; nothing would run");
|
|
8495
8827
|
process.exit(2);
|
|
8496
8828
|
}
|
|
8829
|
+
await applyProfileFromOption(opts.profile, resolveCwd(opts.cwd));
|
|
8497
8830
|
let traceResult = null;
|
|
8498
8831
|
if (!opts.skipTrace) {
|
|
8499
8832
|
traceResult = await runTrace(featureName, specName, opts.model, opts.validationMode ?? "lenient", language);
|
package/dist/package.json
CHANGED
|
@@ -3,6 +3,13 @@ declare function __setCurrentStep(stepId: string, source: string): void;
|
|
|
3
3
|
declare function ab(...args: string[]): void;
|
|
4
4
|
/** Wait for element/text with an explicit timeout so long-running async ops don't hang. */
|
|
5
5
|
declare function abWait(selector: string, timeoutMs?: number): void;
|
|
6
|
+
/**
|
|
7
|
+
* Upload one or more files to a file input via `agent-browser upload`.
|
|
8
|
+
* Relative paths resolve against the test's cwd. We pre-check existence
|
|
9
|
+
* locally so a typo in a fixture path surfaces as a clear error before
|
|
10
|
+
* agent-browser exits with an opaque non-zero status.
|
|
11
|
+
*/
|
|
12
|
+
declare function abUpload(selector: string, ...files: string[]): void;
|
|
6
13
|
/** Assert stable text is visible on page (via wait --text). */
|
|
7
14
|
declare function abAssertTextVisible(text: string, timeoutMs?: number): void;
|
|
8
15
|
/** Assert element is visible (polls `get count`; never uses the blocking `wait <selector>`). */
|
|
@@ -29,4 +36,4 @@ declare function abAssertUnchecked(selector: string): void;
|
|
|
29
36
|
*/
|
|
30
37
|
declare function abStepEvidence(stepId: string, source: string): void;
|
|
31
38
|
//#endregion
|
|
32
|
-
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abWait };
|
|
39
|
+
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abUpload, abWait };
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { i as FAILURE_STEP_ID, n as spawnAB, r as FAILURE_SOURCE, t as sleepSync } from "../spawn-ab-Ja8NRRab.mjs";
|
|
2
|
-
import { mkdirSync, writeFileSync } from "node:fs";
|
|
3
|
-
import { dirname, join } from "node:path";
|
|
2
|
+
import { existsSync, mkdirSync, writeFileSync } from "node:fs";
|
|
3
|
+
import { dirname, isAbsolute, join, resolve } from "node:path";
|
|
4
4
|
//#region src/runtime/test-helpers.ts
|
|
5
5
|
const POST_OPEN_SETTLE_MS = 600;
|
|
6
6
|
function logStep(action, args) {
|
|
@@ -100,6 +100,31 @@ function abWait(selector, timeoutMs = 3e4) {
|
|
|
100
100
|
stderr: ""
|
|
101
101
|
});
|
|
102
102
|
}
|
|
103
|
+
/**
|
|
104
|
+
* Upload one or more files to a file input via `agent-browser upload`.
|
|
105
|
+
* Relative paths resolve against the test's cwd. We pre-check existence
|
|
106
|
+
* locally so a typo in a fixture path surfaces as a clear error before
|
|
107
|
+
* agent-browser exits with an opaque non-zero status.
|
|
108
|
+
*/
|
|
109
|
+
function abUpload(selector, ...files) {
|
|
110
|
+
logStep("upload", [selector, ...files]);
|
|
111
|
+
const resolved = [];
|
|
112
|
+
for (const file of files) {
|
|
113
|
+
const abs = isAbsolute(file) ? file : resolve(process.cwd(), file);
|
|
114
|
+
if (!existsSync(abs)) fail(`abUpload: file not found (${file} → ${abs})`, {
|
|
115
|
+
status: 1,
|
|
116
|
+
stdout: "",
|
|
117
|
+
stderr: ""
|
|
118
|
+
});
|
|
119
|
+
resolved.push(abs);
|
|
120
|
+
}
|
|
121
|
+
const result = spawnAB([
|
|
122
|
+
"upload",
|
|
123
|
+
selector,
|
|
124
|
+
...resolved
|
|
125
|
+
]);
|
|
126
|
+
if (result.status !== 0) fail(`agent-browser upload failed (exit ${result.status})`, result);
|
|
127
|
+
}
|
|
103
128
|
/** Assert stable text is visible on page (via wait --text). */
|
|
104
129
|
function abAssertTextVisible(text, timeoutMs = 3e4) {
|
|
105
130
|
logStep("assert.text", [text]);
|
|
@@ -289,4 +314,4 @@ function warnEvidence(msg) {
|
|
|
289
314
|
process.stderr.write(`[ccqa] evidence: ${msg}\n`);
|
|
290
315
|
}
|
|
291
316
|
//#endregion
|
|
292
|
-
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abWait };
|
|
317
|
+
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abUpload, abWait };
|