ccqa 0.9.1 → 0.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +98 -4
- package/dist/bin/ccqa.mjs +496 -159
- package/dist/package.json +1 -1
- package/dist/runtime/test-helpers.d.mts +8 -1
- package/dist/runtime/test-helpers.mjs +28 -3
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -69,6 +69,8 @@ ccqa run tasks/create-and-complete # vitest replays test.spec.ts; no LLM
|
|
|
69
69
|
ccqa run tasks/create-and-complete # Claude drives the browser every time
|
|
70
70
|
```
|
|
71
71
|
|
|
72
|
+
Live specs can start already-signed-in by pointing `statePath:` at a saved agent-browser state file (cookies + localStorage). Run an interactive login locally once, save the state with `agent-browser state save .ccqa/sessions/<name>.json`, then commit the path (not the file) — see [Pre-authenticated state](#pre-authenticated-state-statepath) below for the local bootstrap and the CI restore pattern.
|
|
73
|
+
|
|
72
74
|
By default deterministic runs write step-boundary screenshots and metadata to `ccqa-report/evidence/<feature>/<spec>/` so a reviewer can confirm a passing spec actually reached the states its `expected` clauses describe. Disable with `--no-evidence`.
|
|
73
75
|
|
|
74
76
|
In CI you can opt in to an HTML run report by passing `--report` — every failing spec gets a drift audit plus a root-cause call (TEST_DRIFT / SPEC_CHANGE / PRODUCT_BUG) using the branch's git diff as context, and the report lets a human grade those calls to measure their accuracy. Requires `ANTHROPIC_API_KEY` or a local Claude login for the analysis part. Opt out with `--no-failure-analysis` (which also implicitly skips the drift audit — the audit is rendered as evidence under the classification, so without the classification the cost has nowhere to land). Use `--no-drift-audit` to keep the classification but skip the audit. See [Run report](./docs/report.md).
|
|
@@ -84,6 +86,7 @@ ccqa run --changed --report # only specs whose relatedPaths t
|
|
|
84
86
|
|---|---|
|
|
85
87
|
| Write specs interactively with Claude | [Draft](./docs/draft.md) |
|
|
86
88
|
| Reuse login and other shared step sequences | [Blocks](./docs/blocks.md) |
|
|
89
|
+
| Drive `<input type="file">` without an OS picker | [File upload](./docs/file-upload.md) |
|
|
87
90
|
| Assertion helper functions | [Assertions](./docs/assertions.md) |
|
|
88
91
|
| Auto-fix failing tests | [Auto-fix](./docs/auto-fix.md) |
|
|
89
92
|
| Detect spec/code drift in CI | [Drift](./docs/drift.md) |
|
|
@@ -98,16 +101,18 @@ ccqa init Scaffold .ccqa/prompts/{live,record}.{user,ag
|
|
|
98
101
|
ccqa draft [feature/spec] Co-author a test spec with Claude
|
|
99
102
|
ccqa perspectives Inventory existing test coverage into .ccqa/perspectives.yaml
|
|
100
103
|
ccqa record <feature/spec> (deterministic specs only) Trace browser actions + generate test.spec.ts
|
|
101
|
-
ccqa run [feature/spec]
|
|
104
|
+
ccqa run [feature/spec...] Execute specs. Per spec, the spec.yaml `mode:` field selects deterministic
|
|
102
105
|
(vitest replay) or live (Claude drives every time). One run can mix both;
|
|
103
|
-
`--report` writes one unified HTML.
|
|
106
|
+
`--report` writes one unified HTML. Pass multiple targets space-separated.
|
|
104
107
|
ccqa drift [feature/spec] Standalone spec ↔ codebase static audit (for PR checks)
|
|
105
108
|
```
|
|
106
109
|
|
|
107
110
|
`ccqa run` flags:
|
|
108
111
|
|
|
109
112
|
- `--report [dir]` — write a self-contained HTML run report (default dir: `ccqa-report/`)
|
|
110
|
-
- `--
|
|
113
|
+
- `--profile <name>` — load `.ccqa/profiles/<name>.env` into the environment before resolving spec `${VAR}` references, so one spec targets dev/stg/prd without per-environment copies. See [Profiles](#profiles---profile).
|
|
114
|
+
- `--changed` — restrict execution to specs whose `relatedPaths` intersect `git diff <base>...HEAD`. Mutually exclusive with explicit spec targets.
|
|
115
|
+
- `--concurrency <n>` — run up to N specs in parallel **within each mode** (deterministic specs run as one phase, live specs as the next; parallelism is within a phase, not across). Default `1` (sequential, identical to before). Above 1, each spec's output is buffered and flushed as a labelled block so parallel logs stay legible. Live specs each launch their own headed Chrome, so high values spawn many browser instances.
|
|
111
116
|
- `--base <ref>` — base ref for the git diff (default: `$GITHUB_BASE_REF`, then `origin/main`)
|
|
112
117
|
- `--no-failure-analysis` — skip the per-failure root-cause classification (also skips the drift audit, since the audit only shows under the classification)
|
|
113
118
|
- `--no-drift-audit` — skip the spec ↔ code drift audit while keeping the classification
|
|
@@ -119,7 +124,7 @@ ccqa drift [feature/spec] Standalone spec ↔ codebase static audit (fo
|
|
|
119
124
|
|
|
120
125
|
All Claude-driven commands accept `-m, --model <name>` (alias `sonnet` | `opus` | `haiku`, or a full model ID). The flag overrides `CCQA_MODEL`; when both are unset, the Claude Code CLI default is used. They also accept `--language <bcp47>` (e.g. `ja`, `en`) to set the language of human-readable output; the default `auto` follows the language of the spec/codebase. `--cwd <path>` works on `record` / `run` / `drift` so you can target a subpackage inside a monorepo from the repo root. Interactive commands authenticate via your local Claude Code login; commands that talk to Claude in CI (`ccqa run --report`, `ccqa drift`) additionally honor `ANTHROPIC_API_KEY`.
|
|
121
126
|
|
|
122
|
-
`<feature/spec>` is a 2-segment alias for the on-disk path `.ccqa/features/<feature>/test-cases/<spec>/`.
|
|
127
|
+
`<feature/spec>` is a 2-segment alias for the on-disk path `.ccqa/features/<feature>/test-cases/<spec>/`. `ccqa run` accepts several targets space-separated (each a `<feature>/<spec>`, a bare `<feature>` for all its specs, or omitted for everything); duplicates are de-duped and `--changed` cannot be combined with explicit targets.
|
|
123
128
|
|
|
124
129
|
## File structure
|
|
125
130
|
|
|
@@ -127,6 +132,9 @@ All Claude-driven commands accept `-m, --model <name>` (alias `sonnet` | `opus`
|
|
|
127
132
|
.ccqa/
|
|
128
133
|
perspectives.yaml # Inventory of existing coverage (machine-readable, canonical)
|
|
129
134
|
perspectives.md # Category index, regenerated from the YAML
|
|
135
|
+
profiles/ # `--profile <name>` env files
|
|
136
|
+
stg.env # URLs + credential refs; commit if it uses secret-manager refs, gitignore if it holds plaintext secrets
|
|
137
|
+
prd.env
|
|
130
138
|
prompts/ # Run `ccqa init` to scaffold these
|
|
131
139
|
record.user.md # Human-maintained guidance appended to `ccqa record` (trace phase)
|
|
132
140
|
record.agent.md # Auto-updated by `ccqa record --update-agent-prompt`
|
|
@@ -155,6 +163,26 @@ All Claude-driven commands accept `-m, --model <name>` (alias `sonnet` | `opus`
|
|
|
155
163
|
|
|
156
164
|
Add `.ccqa/features/*/test-cases/*/runs/` to `.gitignore` — these are per-run artefacts that should not be committed. Likewise `ccqa-report*/`.
|
|
157
165
|
|
|
166
|
+
## Profiles (`--profile`)
|
|
167
|
+
|
|
168
|
+
Keep environment-specific values out of specs as `${VAR}` references and supply them per environment from a **profile** — a `.env` under `.ccqa/profiles/<name>.env`. `ccqa run`/`record --profile <name>` merges it into the environment before resolving `${VAR}`, so one spec runs anywhere.
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
# .ccqa/profiles/stg.env
|
|
172
|
+
APP_BASE_URL=https://<your-app-host>
|
|
173
|
+
TEST_USER_EMAIL=<stg-test-account>
|
|
174
|
+
TEST_USER_PASSWORD=...
|
|
175
|
+
```
|
|
176
|
+
```bash
|
|
177
|
+
ccqa run auth/login --profile stg # same spec, stg values
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
- Name is free-form (`stg`/`prd` are conventions); a path separator / `..` / leading dot is rejected, and a missing profile exits 2. Only the name is logged, never values.
|
|
181
|
+
- Format is a small `.env` subset (`KEY=value`, `#` comments, `export`, quotes). Profile values **override** the inherited environment.
|
|
182
|
+
- Without `--profile`, ccqa auto-loads `<cwd>/.env` if present (like dotenv); with neither, `${VAR}` resolves against the existing `process.env` (e.g. `direnv`).
|
|
183
|
+
|
|
184
|
+
**Secrets:** gitignore any profile that holds plaintext secrets. ccqa only parses `.env` files — it doesn't resolve secret-manager references — so to keep secrets off disk, drop `--profile` and run ccqa under your secret manager instead (e.g. `op run --env-file=.ccqa/profiles/stg.env -- ccqa run ...`), which injects the resolved values into `process.env` for ccqa to read.
|
|
185
|
+
|
|
158
186
|
## Live specs (`mode: live`)
|
|
159
187
|
|
|
160
188
|
For specs declared `mode: live` in their spec.yaml, `ccqa run` skips codegen entirely: Claude executes each spec step against `agent-browser` directly, judges whether the step's `expected` outcome holds, and saves a PNG screenshot before and after every step. Use this mode when:
|
|
@@ -179,6 +207,72 @@ ccqa run --retry 2 tasks/create-and-complete
|
|
|
179
207
|
|
|
180
208
|
Constraints on selectors / `agent-browser` subcommands that apply during `ccqa record` (no `eval`, no `@ref`, no bare-tag positional `find`, no chained agent-browser calls) are **relaxed** for live specs — Claude can use any subcommand and any selector style because there is no replay contract to honour.
|
|
181
209
|
|
|
210
|
+
### Pre-authenticated state (`statePath:`)
|
|
211
|
+
|
|
212
|
+
By default each `ccqa run` of a live spec spins up a fresh `agent-browser` session and starts signed-out. That keeps runs hermetic but forces every device-trust gate (Slack "we don't recognize this browser", Google's unfamiliar-device prompt, MFA challenges, …) to fire on every run.
|
|
213
|
+
|
|
214
|
+
To skip them, save an authenticated browser state to a JSON file once locally and point the spec at it:
|
|
215
|
+
|
|
216
|
+
```yaml
|
|
217
|
+
title: Slack App Home — non-admin access denied
|
|
218
|
+
mode: live
|
|
219
|
+
statePath: .ccqa/sessions/slack-stg.json # cookies + localStorage to restore
|
|
220
|
+
steps:
|
|
221
|
+
- ...
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
ccqa resolves the path against the project root and passes `--state <path>` to every `agent-browser` invocation in the run (including ccqa's own screenshot calls). The file is **read-only** — `--state` loads it but never writes back to it. Re-running locally or in CI does not mutate it.
|
|
225
|
+
|
|
226
|
+
Bootstrap once locally:
|
|
227
|
+
|
|
228
|
+
```bash
|
|
229
|
+
# 1. Log in interactively in a headed browser.
|
|
230
|
+
agent-browser --headed open https://app.slack.com
|
|
231
|
+
# …complete login + device-trust prompts by hand…
|
|
232
|
+
|
|
233
|
+
# 2. Snapshot cookies + localStorage to the path the spec references.
|
|
234
|
+
mkdir -p .ccqa/sessions
|
|
235
|
+
agent-browser state save .ccqa/sessions/slack-stg.json
|
|
236
|
+
agent-browser close
|
|
237
|
+
|
|
238
|
+
# 3. ccqa run reuses the saved state — no login prompt.
|
|
239
|
+
ccqa run slack/app-home-non-admin-access-denied
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
Add `.ccqa/sessions/` to `.gitignore` — these files contain live auth cookies and must never be committed.
|
|
243
|
+
|
|
244
|
+
#### CI: bring the state file with you
|
|
245
|
+
|
|
246
|
+
`statePath:` lives entirely inside `.ccqa/` and never touches `~/`. CI re-uses the state by writing the file into the same path the spec already references:
|
|
247
|
+
|
|
248
|
+
```bash
|
|
249
|
+
# Locally, after the interactive bootstrap above:
|
|
250
|
+
base64 -i .ccqa/sessions/slack-stg.json | pbcopy
|
|
251
|
+
# paste into your CI secret store as CCQA_SLACK_STG_STATE_B64
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
```yaml
|
|
255
|
+
# .github/workflows/ccqa.yml (sketch)
|
|
256
|
+
- name: Restore agent-browser state
|
|
257
|
+
env:
|
|
258
|
+
CCQA_SLACK_STG_STATE_B64: ${{ secrets.CCQA_SLACK_STG_STATE_B64 }}
|
|
259
|
+
run: |
|
|
260
|
+
mkdir -p .ccqa/sessions
|
|
261
|
+
printf '%s' "$CCQA_SLACK_STG_STATE_B64" | base64 -d \
|
|
262
|
+
> .ccqa/sessions/slack-stg.json
|
|
263
|
+
|
|
264
|
+
- name: Run live specs
|
|
265
|
+
env:
|
|
266
|
+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
267
|
+
run: pnpm ccqa run --report
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
Caveats:
|
|
271
|
+
|
|
272
|
+
- **Expiry.** Whatever the upstream service's "remember this device" window is (Slack ≈ 30 days, others vary), the cookies in the state file eventually expire and CI starts failing on the device-trust gate again. Re-bootstrap locally and rotate the secret.
|
|
273
|
+
- **Treat the file as a credential.** It contains live auth cookies. Store it in your CI secret manager (GitHub Actions encrypted secrets, Vault, …) and never commit it.
|
|
274
|
+
- **Deterministic specs ignore `statePath:`.** Today it only affects `mode: live`; vitest-replayed specs always run isolated.
|
|
275
|
+
|
|
182
276
|
### Per-project guidance (`.ccqa/prompts/live.user.md` + `live.agent.md`)
|
|
183
277
|
|
|
184
278
|
ccqa's live-mode system prompt is deliberately product-agnostic. Anything specific to **your** project — staging URLs, login flow quirks, rich-editor types, common access-denied wording — belongs in two sibling files (run `ccqa init` to scaffold both):
|
package/dist/bin/ccqa.mjs
CHANGED
|
@@ -6,12 +6,14 @@ import { accessSync, existsSync, readFileSync, statSync } from "node:fs";
|
|
|
6
6
|
import { fileURLToPath } from "node:url";
|
|
7
7
|
import { access, mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from "node:fs/promises";
|
|
8
8
|
import { homedir, tmpdir } from "node:os";
|
|
9
|
-
import { delimiter, dirname, join, posix, relative, resolve } from "node:path";
|
|
9
|
+
import { delimiter, dirname, isAbsolute, join, posix, relative, resolve } from "node:path";
|
|
10
10
|
import { parse, stringify } from "yaml";
|
|
11
11
|
import { ZodError, z } from "zod";
|
|
12
12
|
import { execFile, spawn, spawnSync } from "node:child_process";
|
|
13
13
|
import { query } from "@anthropic-ai/claude-agent-sdk";
|
|
14
|
+
import { AsyncLocalStorage } from "node:async_hooks";
|
|
14
15
|
import { promisify } from "node:util";
|
|
16
|
+
import { randomUUID } from "node:crypto";
|
|
15
17
|
import { createInterface } from "node:readline";
|
|
16
18
|
import { createInterface as createInterface$1 } from "node:readline/promises";
|
|
17
19
|
//#region src/runtime/env-vars.ts
|
|
@@ -139,6 +141,7 @@ const TestSpecSchema = z.object({
|
|
|
139
141
|
title: z.string().min(1),
|
|
140
142
|
relatedPaths: z.array(z.string().min(1)).optional(),
|
|
141
143
|
mode: SpecModeSchema.optional(),
|
|
144
|
+
statePath: z.string().min(1).optional(),
|
|
142
145
|
steps: z.array(StepSchema).min(1)
|
|
143
146
|
}).strict();
|
|
144
147
|
/** Default mode when `mode:` is absent. */
|
|
@@ -757,6 +760,55 @@ function waitExit(child) {
|
|
|
757
760
|
});
|
|
758
761
|
}
|
|
759
762
|
//#endregion
|
|
763
|
+
//#region src/runtime/live-artifacts.ts
|
|
764
|
+
/**
|
|
765
|
+
* Build a sortable run id from the current wall-clock time. ISO8601 with
|
|
766
|
+
* `:` / `.` replaced so it's filename-safe. Caller is expected to mkdir the
|
|
767
|
+
* directory once and pass `runDir = <baseDir>/<runId>` to the path helpers
|
|
768
|
+
* below.
|
|
769
|
+
*/
|
|
770
|
+
function buildRunId() {
|
|
771
|
+
return (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-");
|
|
772
|
+
}
|
|
773
|
+
/**
|
|
774
|
+
* Per-step artifact paths under a run directory. `<runDir>/steps/<stepId>.*`.
|
|
775
|
+
* Three files per step:
|
|
776
|
+
* - <stepId>.before.png : screenshot taken BEFORE Claude executes the step.
|
|
777
|
+
* - <stepId>.after.png : screenshot taken AFTER Claude executes the step.
|
|
778
|
+
* - <stepId>.log.txt : full assistant transcript for the step (judgement
|
|
779
|
+
* reasoning, any STEP_RESULT lines, raw tool output
|
|
780
|
+
* summaries the model chose to keep).
|
|
781
|
+
*/
|
|
782
|
+
function stepArtifactPaths(runDir, stepId) {
|
|
783
|
+
const dir = join(runDir, "steps");
|
|
784
|
+
return {
|
|
785
|
+
beforePng: join(dir, `${stepId}.before.png`),
|
|
786
|
+
afterPng: join(dir, `${stepId}.after.png`),
|
|
787
|
+
logTxt: join(dir, `${stepId}.log.txt`)
|
|
788
|
+
};
|
|
789
|
+
}
|
|
790
|
+
//#endregion
|
|
791
|
+
//#region src/runtime/pool.ts
|
|
792
|
+
/**
|
|
793
|
+
* Run each item through `fn` with at most `concurrency` running at once.
|
|
794
|
+
* Results preserve input order. A throwing `fn` rejects the whole pool
|
|
795
|
+
* (callers that want per-item isolation should catch inside `fn`).
|
|
796
|
+
*/
|
|
797
|
+
async function runPool(items, concurrency, fn) {
|
|
798
|
+
const results = new Array(items.length);
|
|
799
|
+
let cursor = 0;
|
|
800
|
+
const worker = async () => {
|
|
801
|
+
while (true) {
|
|
802
|
+
const idx = cursor++;
|
|
803
|
+
if (idx >= items.length) return;
|
|
804
|
+
results[idx] = await fn(items[idx], idx);
|
|
805
|
+
}
|
|
806
|
+
};
|
|
807
|
+
const n = Math.max(1, Math.min(concurrency, items.length));
|
|
808
|
+
await Promise.all(Array.from({ length: n }, () => worker()));
|
|
809
|
+
return results;
|
|
810
|
+
}
|
|
811
|
+
//#endregion
|
|
760
812
|
//#region src/claude/extract-json.ts
|
|
761
813
|
/**
|
|
762
814
|
* Pulls a JSON object out of a Claude completion. Accepts either a fenced
|
|
@@ -779,26 +831,70 @@ const STEP_ICONS = {
|
|
|
779
831
|
STEP_SKIPPED: "⊘",
|
|
780
832
|
RUN_COMPLETED: "■"
|
|
781
833
|
};
|
|
834
|
+
/**
|
|
835
|
+
* When a `withBuffer` scope is active, every log line (stdout and stderr) is
|
|
836
|
+
* appended to its buffer instead of being written immediately. Parallel spec
|
|
837
|
+
* runs use this so each spec's narration — including logs emitted deep inside
|
|
838
|
+
* the live executor — flushes as one contiguous block, not interleaved.
|
|
839
|
+
*/
|
|
840
|
+
const bufferStore = new AsyncLocalStorage();
|
|
841
|
+
/** True while inside a `withBuffer` scope: progress lines avoid TTY cursor tricks. */
|
|
842
|
+
function isBuffered() {
|
|
843
|
+
return bufferStore.getStore() !== void 0;
|
|
844
|
+
}
|
|
845
|
+
function emit(text, sink = process.stdout) {
|
|
846
|
+
const store = bufferStore.getStore();
|
|
847
|
+
if (store) {
|
|
848
|
+
store.out.push(text);
|
|
849
|
+
return;
|
|
850
|
+
}
|
|
851
|
+
sink.write(text);
|
|
852
|
+
}
|
|
853
|
+
/**
|
|
854
|
+
* Write raw text to the active `withBuffer` scope, or straight to stdout when
|
|
855
|
+
* none is active. Lets a runner redirect sub-process output (e.g. a child's
|
|
856
|
+
* stdout) into the same buffer as its `log.*` lines so they flush together.
|
|
857
|
+
*/
|
|
858
|
+
function emitRaw(text) {
|
|
859
|
+
emit(text);
|
|
860
|
+
}
|
|
861
|
+
/**
|
|
862
|
+
* Run `fn` with all its log output captured into a buffer, then flush the
|
|
863
|
+
* buffer in one shot under `label`. Used by parallel runners to keep each
|
|
864
|
+
* spec's output legible. Output is flushed even when `fn` throws.
|
|
865
|
+
*
|
|
866
|
+
* When `buffered` is false, `fn` runs with no buffer so its output streams
|
|
867
|
+
* live — this is the sequential (concurrency 1) path, unchanged from before.
|
|
868
|
+
*/
|
|
869
|
+
async function withBuffer(label, buffered, fn) {
|
|
870
|
+
if (!buffered) return fn();
|
|
871
|
+
const store = { out: [] };
|
|
872
|
+
try {
|
|
873
|
+
return await bufferStore.run(store, fn);
|
|
874
|
+
} finally {
|
|
875
|
+
process.stdout.write(`\n──── ${label} ────\n${store.out.join("")}`);
|
|
876
|
+
}
|
|
877
|
+
}
|
|
782
878
|
function header(command, target) {
|
|
783
|
-
|
|
879
|
+
emit(`\nccqa ${command}${target ? ` ${target}` : ""}\n\n`);
|
|
784
880
|
}
|
|
785
881
|
function write(scope, message, sink = process.stdout) {
|
|
786
|
-
|
|
882
|
+
emit(`[${scope}] ${message}\n`, sink);
|
|
787
883
|
}
|
|
788
884
|
function meta(key, value) {
|
|
789
885
|
write("meta", `${key}: ${value}`);
|
|
790
886
|
}
|
|
791
887
|
function blank() {
|
|
792
|
-
|
|
888
|
+
emit("\n");
|
|
793
889
|
}
|
|
794
890
|
function info(message) {
|
|
795
891
|
write("info", message);
|
|
796
892
|
}
|
|
797
893
|
function step(type, stepId, detail) {
|
|
798
|
-
|
|
894
|
+
emit(` ${STEP_ICONS[type]} [${stepId}] ${detail}\n`);
|
|
799
895
|
}
|
|
800
896
|
function bash(command) {
|
|
801
|
-
|
|
897
|
+
emit(` $ ${command.slice(0, 120)}\n`);
|
|
802
898
|
}
|
|
803
899
|
function error(message) {
|
|
804
900
|
write("error", message, process.stderr);
|
|
@@ -807,7 +903,7 @@ function warn(message) {
|
|
|
807
903
|
write("warn", message, process.stderr);
|
|
808
904
|
}
|
|
809
905
|
function hint(message) {
|
|
810
|
-
|
|
906
|
+
emit("\n");
|
|
811
907
|
write("hint", message);
|
|
812
908
|
}
|
|
813
909
|
function fix(message) {
|
|
@@ -832,17 +928,17 @@ const PROGRESS_NONTTY_STRIDE = 5;
|
|
|
832
928
|
let lastProgressNonTtyEmit = -1;
|
|
833
929
|
function progress(current, total, label) {
|
|
834
930
|
const text = `[info] ${current + 1}/${total} ${label}`;
|
|
835
|
-
if (process.stdout.isTTY) {
|
|
931
|
+
if (process.stdout.isTTY && !isBuffered()) {
|
|
836
932
|
process.stdout.write(`\r${text}\x1b[K`);
|
|
837
933
|
return;
|
|
838
934
|
}
|
|
839
935
|
if (current === 0 || current - lastProgressNonTtyEmit >= PROGRESS_NONTTY_STRIDE) {
|
|
840
|
-
|
|
936
|
+
emit(`${text}\n`);
|
|
841
937
|
lastProgressNonTtyEmit = current;
|
|
842
938
|
}
|
|
843
939
|
}
|
|
844
940
|
function progressEnd() {
|
|
845
|
-
if (process.stdout.isTTY) process.stdout.write(`\r\x1b[K`);
|
|
941
|
+
if (process.stdout.isTTY && !isBuffered()) process.stdout.write(`\r\x1b[K`);
|
|
846
942
|
lastProgressNonTtyEmit = -1;
|
|
847
943
|
}
|
|
848
944
|
/**
|
|
@@ -1363,6 +1459,12 @@ function extractAbActionFromBashCommand(cmd) {
|
|
|
1363
1459
|
case "type":
|
|
1364
1460
|
case "select": return `AB_ACTION|${subCmd}|${args[0] ?? ""}|${args[1] ?? ""}|${args[2] ?? ""}`;
|
|
1365
1461
|
case "drag": return `AB_ACTION|drag|${args[0] ?? ""}|${args[1] ?? ""}|${args[2] ?? ""}`;
|
|
1462
|
+
case "upload": {
|
|
1463
|
+
const sel = args[0] ?? "";
|
|
1464
|
+
const files = args.slice(1);
|
|
1465
|
+
if (!sel || files.length === 0) return null;
|
|
1466
|
+
return `AB_ACTION|upload|${sel}|${files.join("|")}`;
|
|
1467
|
+
}
|
|
1366
1468
|
case "snapshot": return null;
|
|
1367
1469
|
case "find": return extractFindAbAction(args);
|
|
1368
1470
|
default: return null;
|
|
@@ -1700,25 +1802,15 @@ const DEFAULT_CONCURRENCY$1 = 3;
|
|
|
1700
1802
|
*/
|
|
1701
1803
|
async function analyzeDrift(input) {
|
|
1702
1804
|
const { targets, cwd, blocks, concurrency = DEFAULT_CONCURRENCY$1, model, language, onSpecStart } = input;
|
|
1703
|
-
|
|
1704
|
-
|
|
1705
|
-
|
|
1706
|
-
|
|
1707
|
-
|
|
1708
|
-
|
|
1709
|
-
|
|
1710
|
-
|
|
1711
|
-
|
|
1712
|
-
cwd,
|
|
1713
|
-
blocks,
|
|
1714
|
-
model,
|
|
1715
|
-
language
|
|
1716
|
-
});
|
|
1717
|
-
}
|
|
1718
|
-
};
|
|
1719
|
-
const pool = Array.from({ length: Math.min(concurrency, targets.length) }, () => worker());
|
|
1720
|
-
await Promise.all(pool);
|
|
1721
|
-
return results;
|
|
1805
|
+
return runPool(targets, concurrency, async (target) => {
|
|
1806
|
+
onSpecStart?.(target);
|
|
1807
|
+
return checkSpec(target, {
|
|
1808
|
+
cwd,
|
|
1809
|
+
blocks,
|
|
1810
|
+
model,
|
|
1811
|
+
language
|
|
1812
|
+
});
|
|
1813
|
+
});
|
|
1722
1814
|
}
|
|
1723
1815
|
async function checkSpec(target, opts) {
|
|
1724
1816
|
const { featureName, specName } = target;
|
|
@@ -4095,6 +4187,123 @@ const CLIENT_JS = `
|
|
|
4095
4187
|
})();
|
|
4096
4188
|
`;
|
|
4097
4189
|
//#endregion
|
|
4190
|
+
//#region src/runtime/profile-env.ts
|
|
4191
|
+
/**
|
|
4192
|
+
* Profile env (Issue #37). A profile is a named `.env` under
|
|
4193
|
+
* `.ccqa/profiles/<name>.env`; its contents merge into `process.env` before any
|
|
4194
|
+
* spec work, so one spec targets dev/stg/prd without per-environment copies.
|
|
4195
|
+
* Spec `${VAR}` references all resolve against `process.env` downstream.
|
|
4196
|
+
*
|
|
4197
|
+
* The `.env` parser is a small hand-rolled subset (no dotenv dependency).
|
|
4198
|
+
*/
|
|
4199
|
+
/**
|
|
4200
|
+
* Parse a `.env` body into a `name → value` map. Subset: blank / `#` lines
|
|
4201
|
+
* skipped, optional leading `export`, split on the first `=`, surrounding
|
|
4202
|
+
* quotes stripped, inline `# comment` dropped. No multi-line / interpolation.
|
|
4203
|
+
*/
|
|
4204
|
+
function parseDotenv(content) {
|
|
4205
|
+
const out = {};
|
|
4206
|
+
for (const rawLine of content.split(/\r?\n/)) {
|
|
4207
|
+
const line = rawLine.trim();
|
|
4208
|
+
if (line === "" || line.startsWith("#")) continue;
|
|
4209
|
+
const withoutExport = line.replace(/^export\s+/, "");
|
|
4210
|
+
const eq = withoutExport.indexOf("=");
|
|
4211
|
+
if (eq === -1) continue;
|
|
4212
|
+
const key = withoutExport.slice(0, eq).trim();
|
|
4213
|
+
if (key === "") continue;
|
|
4214
|
+
out[key] = parseValue(withoutExport.slice(eq + 1).trim());
|
|
4215
|
+
}
|
|
4216
|
+
return out;
|
|
4217
|
+
}
|
|
4218
|
+
function parseValue(raw) {
|
|
4219
|
+
const quote = raw[0];
|
|
4220
|
+
if (quote === "\"" || quote === "'") {
|
|
4221
|
+
const close = raw.indexOf(quote, 1);
|
|
4222
|
+
if (close !== -1 && /^\s*(#.*)?$/.test(raw.slice(close + 1))) return raw.slice(1, close);
|
|
4223
|
+
}
|
|
4224
|
+
const hash = raw.search(/\s#/);
|
|
4225
|
+
return hash === -1 ? raw : raw.slice(0, hash).trimEnd();
|
|
4226
|
+
}
|
|
4227
|
+
var ProfileNotFoundError = class extends Error {
|
|
4228
|
+
profile;
|
|
4229
|
+
path;
|
|
4230
|
+
constructor(profile, path) {
|
|
4231
|
+
super(`profile "${profile}" not found: ${path}`);
|
|
4232
|
+
this.name = "ProfileNotFoundError";
|
|
4233
|
+
this.profile = profile;
|
|
4234
|
+
this.path = path;
|
|
4235
|
+
}
|
|
4236
|
+
};
|
|
4237
|
+
var InvalidProfileNameError = class extends Error {
|
|
4238
|
+
profile;
|
|
4239
|
+
constructor(profile) {
|
|
4240
|
+
super(`invalid profile name "${profile}": expected a bare name like "stg" (no path separators, no leading dot)`);
|
|
4241
|
+
this.name = "InvalidProfileNameError";
|
|
4242
|
+
this.profile = profile;
|
|
4243
|
+
}
|
|
4244
|
+
};
|
|
4245
|
+
/**
|
|
4246
|
+
* A profile name must be a single, non-dot-leading path segment, so
|
|
4247
|
+
* `--profile <name>` can't read a file outside the profiles dir (e.g.
|
|
4248
|
+
* `--profile ../../etc/hosts`). Rejecting separators and a leading dot already
|
|
4249
|
+
* blocks `..` traversal, so an in-name `..` (like `v1..2`) stays allowed.
|
|
4250
|
+
*/
|
|
4251
|
+
function assertValidProfileName(profile) {
|
|
4252
|
+
if (profile === "" || profile.includes("/") || profile.includes("\\") || profile.startsWith(".")) throw new InvalidProfileNameError(profile);
|
|
4253
|
+
}
|
|
4254
|
+
/** Absolute path of the `.env` file backing `<profile>` under `<cwd>/.ccqa/`. */
|
|
4255
|
+
function profilePath(profile, cwd) {
|
|
4256
|
+
assertValidProfileName(profile);
|
|
4257
|
+
return join(cwd, ".ccqa", "profiles", `${profile}.env`);
|
|
4258
|
+
}
|
|
4259
|
+
/** Read + parse a `.env`, or `null` if absent. Other read errors propagate. */
|
|
4260
|
+
async function readDotenv(path) {
|
|
4261
|
+
let content;
|
|
4262
|
+
try {
|
|
4263
|
+
content = await readFile(path, "utf8");
|
|
4264
|
+
} catch (err) {
|
|
4265
|
+
if (err.code === "ENOENT") return null;
|
|
4266
|
+
throw err;
|
|
4267
|
+
}
|
|
4268
|
+
return parseDotenv(content);
|
|
4269
|
+
}
|
|
4270
|
+
/**
|
|
4271
|
+
* Load `.ccqa/profiles/<profile>.env`. A missing file throws — a typo must fail
|
|
4272
|
+
* loudly, not silently resolve every credential to empty.
|
|
4273
|
+
*/
|
|
4274
|
+
async function loadProfileEnv(profile, cwd) {
|
|
4275
|
+
const path = profilePath(profile, cwd);
|
|
4276
|
+
const vars = await readDotenv(path);
|
|
4277
|
+
if (vars === null) throw new ProfileNotFoundError(profile, path);
|
|
4278
|
+
return vars;
|
|
4279
|
+
}
|
|
4280
|
+
/** Absolute path of the default `.env` ccqa loads when `--profile` is absent. */
|
|
4281
|
+
function defaultEnvPath(cwd) {
|
|
4282
|
+
return join(cwd, ".env");
|
|
4283
|
+
}
|
|
4284
|
+
/**
|
|
4285
|
+
* Load `<cwd>/.env`, the default when no `--profile` is given. A missing `.env`
|
|
4286
|
+
* is fine (returns `null`) — the run falls back to the existing `process.env`.
|
|
4287
|
+
*/
|
|
4288
|
+
async function loadDefaultEnv(cwd) {
|
|
4289
|
+
return readDotenv(defaultEnvPath(cwd));
|
|
4290
|
+
}
|
|
4291
|
+
/**
|
|
4292
|
+
* Merge vars into `process.env`. With `override` (the default), the profile
|
|
4293
|
+
* wins over inherited values. Returns the applied names — never values, so
|
|
4294
|
+
* callers log names only and secrets stay out of the log.
|
|
4295
|
+
*/
|
|
4296
|
+
function applyProfileEnv(vars, opts = {}) {
|
|
4297
|
+
const override = opts.override ?? true;
|
|
4298
|
+
const applied = [];
|
|
4299
|
+
for (const [name, value] of Object.entries(vars)) {
|
|
4300
|
+
if (!override && process.env[name] !== void 0) continue;
|
|
4301
|
+
process.env[name] = value;
|
|
4302
|
+
applied.push(name);
|
|
4303
|
+
}
|
|
4304
|
+
return applied;
|
|
4305
|
+
}
|
|
4306
|
+
//#endregion
|
|
4098
4307
|
//#region src/cli/options.ts
|
|
4099
4308
|
/**
|
|
4100
4309
|
* Shared `--language` flag. Every Claude-driven command writes some
|
|
@@ -4105,6 +4314,53 @@ const CLIENT_JS = `
|
|
|
4105
4314
|
function addLanguageOption(command) {
|
|
4106
4315
|
return command.option("--language <bcp47>", "Language for human-readable output (e.g. 'en', 'ja'). Default 'auto' follows the language of the spec/codebase.", DEFAULT_LANGUAGE);
|
|
4107
4316
|
}
|
|
4317
|
+
/**
|
|
4318
|
+
* Shared `--profile <name>` flag for the browser-driving commands (`run`,
|
|
4319
|
+
* `record`), registered identically so help text and behaviour don't drift.
|
|
4320
|
+
*/
|
|
4321
|
+
function addProfileOption(command) {
|
|
4322
|
+
return command.option("--profile <name>", "Load .ccqa/profiles/<name>.env into the environment before resolving spec ${VAR} references (URLs, credentials), so one spec can target dev/stg/prd without per-environment copies. Profile values override the inherited environment.");
|
|
4323
|
+
}
|
|
4324
|
+
/**
|
|
4325
|
+
* Merge the environment for a `run` / `record` invocation into `process.env`
|
|
4326
|
+
* before any spec work. With `--profile <name>`, load that profile (missing /
|
|
4327
|
+
* invalid → exit 2). Without it, auto-load `<cwd>/.env` if present (a missing
|
|
4328
|
+
* `.env` is fine). Checking `!== undefined` rejects `--profile ""` rather than
|
|
4329
|
+
* skipping it.
|
|
4330
|
+
*/
|
|
4331
|
+
async function applyProfileFromOption(profile, cwd) {
|
|
4332
|
+
if (profile !== void 0) await applyNamedProfile(profile, cwd);
|
|
4333
|
+
else await applyDefaultEnv(cwd);
|
|
4334
|
+
}
|
|
4335
|
+
/** "1 var" / "2 vars" — the count summary shared by both load paths' meta line. */
|
|
4336
|
+
function varCount(n) {
|
|
4337
|
+
return `${n} var${n === 1 ? "" : "s"}`;
|
|
4338
|
+
}
|
|
4339
|
+
async function applyNamedProfile(profile, cwd) {
|
|
4340
|
+
try {
|
|
4341
|
+
const applied = applyProfileEnv(await loadProfileEnv(profile, cwd));
|
|
4342
|
+
meta("profile", `${profile} (${varCount(applied.length)})`);
|
|
4343
|
+
if (applied.length === 0) warn(`profile "${profile}" defined no variables — spec $\{VAR} references will resolve to empty`);
|
|
4344
|
+
} catch (err) {
|
|
4345
|
+
if (err instanceof ProfileNotFoundError) {
|
|
4346
|
+
error(err.message);
|
|
4347
|
+
hint(`create ${err.path} with the environment's $\{VAR} values`);
|
|
4348
|
+
} else if (err instanceof InvalidProfileNameError) error(err.message);
|
|
4349
|
+
else error(`failed to load profile "${profile}": ${err instanceof Error ? err.message : String(err)}`);
|
|
4350
|
+
process.exit(2);
|
|
4351
|
+
}
|
|
4352
|
+
}
|
|
4353
|
+
async function applyDefaultEnv(cwd) {
|
|
4354
|
+
let vars;
|
|
4355
|
+
try {
|
|
4356
|
+
vars = await loadDefaultEnv(cwd);
|
|
4357
|
+
} catch (err) {
|
|
4358
|
+
error(`failed to load ${defaultEnvPath(cwd)}: ${err instanceof Error ? err.message : String(err)}`);
|
|
4359
|
+
process.exit(2);
|
|
4360
|
+
}
|
|
4361
|
+
if (vars === null) return;
|
|
4362
|
+
meta("env", `.env (${varCount(applyProfileEnv(vars, { override: false }).length)})`);
|
|
4363
|
+
}
|
|
4108
4364
|
//#endregion
|
|
4109
4365
|
//#region src/cli/resolve-cwd.ts
|
|
4110
4366
|
/**
|
|
@@ -4116,7 +4372,7 @@ function addLanguageOption(command) {
|
|
|
4116
4372
|
*
|
|
4117
4373
|
* It's mostly useful in monorepos where you want to invoke ccqa from the
|
|
4118
4374
|
* repo root but target a subpackage (e.g.
|
|
4119
|
-
* `ccqa run --cwd
|
|
4375
|
+
* `ccqa run --cwd apps/web-app`).
|
|
4120
4376
|
*
|
|
4121
4377
|
* Falls back to `process.cwd()` when the option is not given.
|
|
4122
4378
|
*/
|
|
@@ -4328,36 +4584,14 @@ function oneLine$1(s) {
|
|
|
4328
4584
|
return s.replace(/\s+/g, " ").trim();
|
|
4329
4585
|
}
|
|
4330
4586
|
//#endregion
|
|
4331
|
-
//#region src/runtime/live-artifacts.ts
|
|
4332
|
-
/**
|
|
4333
|
-
* Build a sortable run id from the current wall-clock time. ISO8601 with
|
|
4334
|
-
* `:` / `.` replaced so it's filename-safe. Caller is expected to mkdir the
|
|
4335
|
-
* directory once and pass `runDir = <baseDir>/<runId>` to the path helpers
|
|
4336
|
-
* below.
|
|
4337
|
-
*/
|
|
4338
|
-
function buildRunId() {
|
|
4339
|
-
return (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-");
|
|
4340
|
-
}
|
|
4341
|
-
/**
|
|
4342
|
-
* Per-step artifact paths under a run directory. `<runDir>/steps/<stepId>.*`.
|
|
4343
|
-
* Three files per step:
|
|
4344
|
-
* - <stepId>.before.png : screenshot taken BEFORE Claude executes the step.
|
|
4345
|
-
* - <stepId>.after.png : screenshot taken AFTER Claude executes the step.
|
|
4346
|
-
* - <stepId>.log.txt : full assistant transcript for the step (judgement
|
|
4347
|
-
* reasoning, any STEP_RESULT lines, raw tool output
|
|
4348
|
-
* summaries the model chose to keep).
|
|
4349
|
-
*/
|
|
4350
|
-
function stepArtifactPaths(runDir, stepId) {
|
|
4351
|
-
const dir = join(runDir, "steps");
|
|
4352
|
-
return {
|
|
4353
|
-
beforePng: join(dir, `${stepId}.before.png`),
|
|
4354
|
-
afterPng: join(dir, `${stepId}.after.png`),
|
|
4355
|
-
logTxt: join(dir, `${stepId}.log.txt`)
|
|
4356
|
-
};
|
|
4357
|
-
}
|
|
4358
|
-
//#endregion
|
|
4359
4587
|
//#region src/claude/agent-browser-invoke.ts
|
|
4360
4588
|
function agentBrowserInvokeBase(input) {
|
|
4589
|
+
const env = {
|
|
4590
|
+
AGENT_BROWSER_SESSION: input.sessionName,
|
|
4591
|
+
CCQA_RUN_ID: input.runId,
|
|
4592
|
+
PATH: pathWithAgentBrowserShim(process.env["PATH"])
|
|
4593
|
+
};
|
|
4594
|
+
if (input.statePath) env["CCQA_AB_STATE"] = input.statePath;
|
|
4361
4595
|
return {
|
|
4362
4596
|
allowedTools: [
|
|
4363
4597
|
"Bash(*)",
|
|
@@ -4365,17 +4599,19 @@ function agentBrowserInvokeBase(input) {
|
|
|
4365
4599
|
"Grep",
|
|
4366
4600
|
"Glob"
|
|
4367
4601
|
],
|
|
4368
|
-
env
|
|
4369
|
-
AGENT_BROWSER_SESSION: input.sessionName,
|
|
4370
|
-
CCQA_RUN_ID: input.runId,
|
|
4371
|
-
PATH: pathWithAgentBrowserShim(process.env["PATH"])
|
|
4372
|
-
}
|
|
4602
|
+
env
|
|
4373
4603
|
};
|
|
4374
4604
|
}
|
|
4375
4605
|
//#endregion
|
|
4376
4606
|
//#region src/prompts/live.ts
|
|
4607
|
+
/**
|
|
4608
|
+
* Unique agent-browser session name. The runId is millisecond-precision wall
|
|
4609
|
+
* clock, so under `--concurrency > 1` two specs can start in the same
|
|
4610
|
+
* millisecond and collide; a random suffix guarantees each spec gets its own
|
|
4611
|
+
* Chrome session and state never bleeds across parallel runs.
|
|
4612
|
+
*/
|
|
4377
4613
|
function generateLiveSessionName() {
|
|
4378
|
-
return `ccqa-live-${buildRunId()}`;
|
|
4614
|
+
return `ccqa-live-${buildRunId()}-${randomUUID().slice(0, 8)}`;
|
|
4379
4615
|
}
|
|
4380
4616
|
/**
|
|
4381
4617
|
* Static prefix of the `ccqa run` (live spec) system prompt. Built once per
|
|
@@ -4405,19 +4641,20 @@ function buildLiveSystemPromptPrefix(input) {
|
|
|
4405
4641
|
const stepsText = input.allSteps.map((s) => `### ${s.id} [${s.source}]
|
|
4406
4642
|
- **Instruction**: ${s.instruction}
|
|
4407
4643
|
- **Expected**: ${s.expected}`).join("\n\n");
|
|
4644
|
+
const stateLine = input.statePath ? `\n\nA pre-recorded auth-state file is provided at \`${input.statePath}\` (also in the env var \`CCQA_AB_STATE\`). **Always also pass \`--state "$CCQA_AB_STATE"\`** to every \`agent-browser\` command — this restores cookies and localStorage from a prior interactive login, so the user is already signed in to the application under test from step 1. The file is loaded read-only; do not run \`agent-browser state save\`.` : "";
|
|
4408
4645
|
return `You are a QA execution agent. You are executing ONE step of a browser-based end-to-end test and judging whether the step's expected outcome was achieved. You are NOT recording a replayable test script — be flexible, explore the DOM as needed, and make a clear pass / fail call at the end.
|
|
4409
4646
|
|
|
4410
4647
|
## Session
|
|
4411
4648
|
|
|
4412
4649
|
SESSION NAME: \`${input.sessionName}\`
|
|
4413
4650
|
|
|
4414
|
-
Always pass \`--session ${input.sessionName}\` to every \`agent-browser\` command. The session persists across steps within this test run, so the browser state from previous steps is already loaded when this turn starts
|
|
4651
|
+
Always pass \`--session ${input.sessionName}\` to every \`agent-browser\` command. The session persists across steps within this test run, so the browser state from previous steps is already loaded when this turn starts.${stateLine}
|
|
4415
4652
|
|
|
4416
4653
|
## Tools
|
|
4417
4654
|
|
|
4418
4655
|
You have:
|
|
4419
4656
|
|
|
4420
|
-
- **Bash** to run \`agent-browser\` (the full surface — \`open\`, \`snapshot\`, \`click\`, \`fill\`, \`press\`, \`wait\`, \`find\`, \`screenshot\`, \`eval\`, \`js\`, \`get\`, etc.). Any selector form is allowed: \`@ref\` (e.g. \`@e14\`), CSS selectors, \`text=...\`, \`[aria-label='...']\`, \`[data-testid='...']\`, bare tags inside \`find first/last/nth\` — whatever works for this single run. There is no replay contract to honour.
|
|
4657
|
+
- **Bash** to run \`agent-browser\` (the full surface — \`open\`, \`snapshot\`, \`click\`, \`fill\`, \`upload\`, \`press\`, \`wait\`, \`find\`, \`screenshot\`, \`eval\`, \`js\`, \`get\`, etc.). Any selector form is allowed: \`@ref\` (e.g. \`@e14\`), CSS selectors, \`text=...\`, \`[aria-label='...']\`, \`[data-testid='...']\`, bare tags inside \`find first/last/nth\` — whatever works for this single run. There is no replay contract to honour. For file inputs (\`<input type="file">\`) do NOT \`click\` the input — use \`agent-browser upload "<selector>" <path>\` so no OS file-picker dialog opens. Fixtures conventionally live under \`.ccqa/fixtures/\`; reference them via \`\${CCQA_FIXTURES_DIR}/<name>\`.
|
|
4421
4658
|
- **Read / Grep / Glob** for inspecting the application source code when you need to find a selector or understand routing. Read-only — do not modify source files.
|
|
4422
4659
|
|
|
4423
4660
|
## Test Specification
|
|
@@ -4525,11 +4762,9 @@ function findLastStepResult(text) {
|
|
|
4525
4762
|
* artifact, not a reason to abort the test step.
|
|
4526
4763
|
*/
|
|
4527
4764
|
function takeScreenshot(sessionName, outPath, options) {
|
|
4528
|
-
const args = [
|
|
4529
|
-
|
|
4530
|
-
|
|
4531
|
-
"screenshot"
|
|
4532
|
-
];
|
|
4765
|
+
const args = ["--session", sessionName];
|
|
4766
|
+
if (options?.statePath) args.push("--state", options.statePath);
|
|
4767
|
+
args.push("screenshot");
|
|
4533
4768
|
if (options?.fullPage) args.push("--full");
|
|
4534
4769
|
args.push(outPath);
|
|
4535
4770
|
const res = spawnAB(args);
|
|
@@ -4562,16 +4797,19 @@ async function runLiveExecutor(input) {
|
|
|
4562
4797
|
const startedAt = /* @__PURE__ */ new Date();
|
|
4563
4798
|
const stepResults = [];
|
|
4564
4799
|
let overallFailed = false;
|
|
4800
|
+
const statePath = input.statePath ?? null;
|
|
4565
4801
|
const promptPrefix = buildLiveSystemPromptPrefix({
|
|
4566
4802
|
title: input.spec.title,
|
|
4567
4803
|
allSteps: input.steps,
|
|
4568
|
-
sessionName: input.sessionName
|
|
4804
|
+
sessionName: input.sessionName,
|
|
4805
|
+
statePath
|
|
4569
4806
|
});
|
|
4570
4807
|
const suffixBlock = input.systemPromptSuffix ? `\n## Project-specific guidance\n\n${input.systemPromptSuffix}\n` : "";
|
|
4571
4808
|
const langDirective = languageDirective(input.language);
|
|
4572
4809
|
const invokeBase = agentBrowserInvokeBase({
|
|
4573
4810
|
sessionName: input.sessionName,
|
|
4574
|
-
runId: input.runId
|
|
4811
|
+
runId: input.runId,
|
|
4812
|
+
statePath
|
|
4575
4813
|
});
|
|
4576
4814
|
const retries = Math.max(0, input.retries ?? 0);
|
|
4577
4815
|
for (let i = 0; i < input.steps.length; i++) {
|
|
@@ -4616,7 +4854,7 @@ async function runLiveExecutor(input) {
|
|
|
4616
4854
|
}
|
|
4617
4855
|
}
|
|
4618
4856
|
async function executeStepAttempt(step, paths, systemPrompt, userPrompt) {
|
|
4619
|
-
const before = takeScreenshot(input.sessionName, paths.beforePng);
|
|
4857
|
+
const before = takeScreenshot(input.sessionName, paths.beforePng, { statePath });
|
|
4620
4858
|
if (!before.ok) warn(`screenshot (before, ${step.id}) failed: ${before.error}`);
|
|
4621
4859
|
const transcriptParts = [];
|
|
4622
4860
|
let isError = false;
|
|
@@ -4648,7 +4886,10 @@ async function runLiveExecutor(input) {
|
|
|
4648
4886
|
transcriptParts.push(`[ccqa] invokeClaudeStreaming threw: ${err instanceof Error ? err.message : String(err)}`);
|
|
4649
4887
|
}
|
|
4650
4888
|
const transcript = transcriptParts.join("\n");
|
|
4651
|
-
const after = takeScreenshot(input.sessionName, paths.afterPng, {
|
|
4889
|
+
const after = takeScreenshot(input.sessionName, paths.afterPng, {
|
|
4890
|
+
fullPage: true,
|
|
4891
|
+
statePath
|
|
4892
|
+
});
|
|
4652
4893
|
if (!after.ok) warn(`screenshot (after, ${step.id}) failed: ${after.error}`);
|
|
4653
4894
|
await writeFile(paths.logTxt, transcript || "(no assistant text captured)", "utf-8");
|
|
4654
4895
|
const { status, reasoning } = judgeStepOutcome({
|
|
@@ -4840,24 +5081,24 @@ async function runLiveSpecs(specs, opts) {
|
|
|
4840
5081
|
await preflightAgentBrowserCommand();
|
|
4841
5082
|
meta("live-specs", specs.length);
|
|
4842
5083
|
const userPromptBundle = await loadLivePromptBundle(cwd);
|
|
4843
|
-
if (userPromptBundle !== null) meta("
|
|
5084
|
+
if (userPromptBundle !== null) meta("prompt", userPromptBundle.loaded.join(" + "));
|
|
4844
5085
|
const userPromptSuffix = userPromptBundle?.text ?? null;
|
|
4845
|
-
const
|
|
4846
|
-
|
|
4847
|
-
const {
|
|
4848
|
-
|
|
4849
|
-
|
|
4850
|
-
|
|
4851
|
-
|
|
4852
|
-
|
|
4853
|
-
|
|
4854
|
-
|
|
4855
|
-
|
|
4856
|
-
|
|
4857
|
-
|
|
4858
|
-
|
|
4859
|
-
})
|
|
4860
|
-
}
|
|
5086
|
+
const concurrency = Math.max(1, opts.concurrency ?? 1);
|
|
5087
|
+
const runs = await runPool(specs, concurrency, (spec, i) => {
|
|
5088
|
+
const label = `${spec.featureName}/${spec.specName}`;
|
|
5089
|
+
return withBuffer(label, concurrency > 1, () => {
|
|
5090
|
+
if (concurrency === 1 && specs.length > 1) {
|
|
5091
|
+
blank();
|
|
5092
|
+
info(`[${i + 1}/${specs.length}] ${label}`);
|
|
5093
|
+
}
|
|
5094
|
+
return runOneSpec({
|
|
5095
|
+
...spec,
|
|
5096
|
+
opts,
|
|
5097
|
+
userPromptSuffix,
|
|
5098
|
+
cwd
|
|
5099
|
+
});
|
|
5100
|
+
});
|
|
5101
|
+
});
|
|
4861
5102
|
const failedCount = runs.filter((r) => r.kind === "error" || r.kind === "run" && r.result.status === "failed").length;
|
|
4862
5103
|
blank();
|
|
4863
5104
|
meta("live-summary", `${runs.length - failedCount} passed / ${failedCount} failed`);
|
|
@@ -4956,6 +5197,23 @@ async function runOneSpec(args) {
|
|
|
4956
5197
|
if (includes.length > 0) meta("blocks", includes.join(", "));
|
|
4957
5198
|
const sessionName = generateLiveSessionName();
|
|
4958
5199
|
meta("session", sessionName);
|
|
5200
|
+
let statePath = null;
|
|
5201
|
+
if (spec.statePath) {
|
|
5202
|
+
statePath = isAbsolute(spec.statePath) ? spec.statePath : resolve(cwd, spec.statePath);
|
|
5203
|
+
try {
|
|
5204
|
+
await access(statePath);
|
|
5205
|
+
} catch {
|
|
5206
|
+
const msg = `spec.statePath points to a missing file: ${statePath}`;
|
|
5207
|
+
error(msg);
|
|
5208
|
+
return {
|
|
5209
|
+
kind: "error",
|
|
5210
|
+
featureName,
|
|
5211
|
+
specName,
|
|
5212
|
+
error: msg
|
|
5213
|
+
};
|
|
5214
|
+
}
|
|
5215
|
+
meta("state", statePath);
|
|
5216
|
+
}
|
|
4959
5217
|
const runId = buildRunId();
|
|
4960
5218
|
const runDir = opts.out ?? join(specDir, "runs", runId);
|
|
4961
5219
|
await mkdir(runDir, { recursive: true });
|
|
@@ -4966,6 +5224,7 @@ async function runOneSpec(args) {
|
|
|
4966
5224
|
runId,
|
|
4967
5225
|
runDir,
|
|
4968
5226
|
sessionName,
|
|
5227
|
+
statePath,
|
|
4969
5228
|
systemPromptSuffix: userPromptSuffix,
|
|
4970
5229
|
model: opts.model,
|
|
4971
5230
|
language: opts.language,
|
|
@@ -5231,28 +5490,57 @@ async function resolveVitestConfig(cwd) {
|
|
|
5231
5490
|
return bundledVitestConfigPath();
|
|
5232
5491
|
}
|
|
5233
5492
|
}
|
|
5234
|
-
const runCommand = addLanguageOption(new Command("run").argument("[
|
|
5493
|
+
const runCommand = addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs. Each spec's execution mode comes from its spec.yaml `mode:` field (default deterministic; set `mode: live` to have Claude drive agent-browser live per step). Deterministic specs replay the recorded test.spec.ts under vitest. Pass --report to write one unified HTML report covering both modes.").option("--report [dir]", `Write a self-contained HTML run report (failure analysis + drift audit by default). Default dir: ${DEFAULT_REPORT_DIR}/`).option("--changed", "Restrict execution to specs whose relatedPaths intersect the git diff against --base (or, in CI, $GITHUB_BASE_REF, else origin/main). Cannot be combined with an explicit spec id.").option("--no-failure-analysis", "Skip the per-failure root-cause classification (TEST_DRIFT / SPEC_CHANGE / PRODUCT_BUG). --report only.").option("--no-drift-audit", "Skip the spec↔code drift audit shown in the report. --report only.").option("--base <ref>", "Base ref the source diff is taken against for failure analysis (default: GITHUB_BASE_REF, then origin/main).").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--format <fmt>", "Additional output format alongside HTML when --report is set: 'text' (default), 'json' (writes report.json), 'github' (GitHub Actions annotations on stdout).", (raw) => {
|
|
5235
5494
|
if (REPORT_FORMATS.includes(raw)) return raw;
|
|
5236
5495
|
throw new Error(`--format must be one of ${REPORT_FORMATS.join(" | ")}`);
|
|
5237
5496
|
}, "text").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--no-evidence", `(deterministic only) Skip step-boundary evidence capture (PNG + meta JSON written to ${DEFAULT_REPORT_DIR}/${EVIDENCE_SUBDIR}/ by default).`).option("--retry <n>", "(live only) Retry each failed step up to N more times before recording failure. Default 0.", (raw) => {
|
|
5238
5497
|
const n = Number(raw);
|
|
5239
5498
|
if (!Number.isFinite(n) || n < 0 || Math.floor(n) !== n) throw new Error(`--retry must be a non-negative integer, got "${raw}"`);
|
|
5240
5499
|
return n;
|
|
5241
|
-
}, 0).option("--out <dir>", "(live only) Override the per-spec artifact directory. Default: <specDir>/runs/<runId>. Ignored when running multiple specs.").option("--update-agent-prompt", "(live only) After the run finishes, ask Claude to refresh .ccqa/prompts/live.agent.md from a summary of the run.")).action(async (
|
|
5242
|
-
await runDispatcher(
|
|
5500
|
+
}, 0).option("--out <dir>", "(live only) Override the per-spec artifact directory. Default: <specDir>/runs/<runId>. Ignored when running multiple specs.").option("--update-agent-prompt", "(live only) After the run finishes, ask Claude to refresh .ccqa/prompts/live.agent.md from a summary of the run.").option("--concurrency <n>", "Run up to N specs in parallel within each mode (deterministic / live). Default 1 (sequential). Live specs each get an isolated agent-browser session; high values spawn many headed Chrome instances.", parseConcurrency$1, 1))).action(async (targets, opts) => {
|
|
5501
|
+
await runDispatcher(targets, opts);
|
|
5243
5502
|
});
|
|
5503
|
+
/** Parse --concurrency: a positive integer. Rejects 0, negatives, non-integers. */
|
|
5504
|
+
function parseConcurrency$1(raw) {
|
|
5505
|
+
const n = Number(raw);
|
|
5506
|
+
if (!Number.isInteger(n) || n < 1) {
|
|
5507
|
+
error(`invalid --concurrency: ${raw} (expected positive integer)`);
|
|
5508
|
+
process.exit(2);
|
|
5509
|
+
}
|
|
5510
|
+
return n;
|
|
5511
|
+
}
|
|
5244
5512
|
function resolveReportDir(report, cwd) {
|
|
5245
5513
|
if (report === void 0 || report === false) return void 0;
|
|
5246
5514
|
return resolve(cwd, typeof report === "string" ? report : DEFAULT_REPORT_DIR);
|
|
5247
5515
|
}
|
|
5248
|
-
|
|
5249
|
-
|
|
5250
|
-
if (
|
|
5516
|
+
/** Header label shown after `ccqa run`: the lone target, a count, or a mode marker. */
|
|
5517
|
+
function headerTarget(targets, opts) {
|
|
5518
|
+
if (targets.length === 1) return targets[0];
|
|
5519
|
+
if (targets.length > 1) return `${targets.length} targets`;
|
|
5520
|
+
return opts.changed ? "(changed)" : "(all specs)";
|
|
5521
|
+
}
|
|
5522
|
+
/** De-dupe by `featureName/specName`, keeping first-seen order. */
|
|
5523
|
+
function dedupeSpecs(specs) {
|
|
5524
|
+
const seen = /* @__PURE__ */ new Set();
|
|
5525
|
+
const out = [];
|
|
5526
|
+
for (const s of specs) {
|
|
5527
|
+
const key = `${s.featureName}/${s.specName}`;
|
|
5528
|
+
if (seen.has(key)) continue;
|
|
5529
|
+
seen.add(key);
|
|
5530
|
+
out.push(s);
|
|
5531
|
+
}
|
|
5532
|
+
return out;
|
|
5533
|
+
}
|
|
5534
|
+
async function runDispatcher(targets, opts) {
|
|
5535
|
+
header("run", headerTarget(targets, opts));
|
|
5536
|
+
if (opts.changed && targets.length > 0) {
|
|
5251
5537
|
error("--changed and an explicit spec target cannot be combined");
|
|
5252
5538
|
process.exit(2);
|
|
5253
5539
|
}
|
|
5254
5540
|
const cwd = resolveCwd(opts.cwd);
|
|
5255
|
-
|
|
5541
|
+
await applyProfileFromOption(opts.profile, cwd);
|
|
5542
|
+
const enumerateAll = () => listAllSpecsWithSpecFile(cwd);
|
|
5543
|
+
let specs = dedupeSpecs((await Promise.all((targets.length ? targets : [void 0]).map((t) => resolveSpecTargets(t, enumerateAll, cwd)))).flat());
|
|
5256
5544
|
if (opts.changed) {
|
|
5257
5545
|
const before = specs.length;
|
|
5258
5546
|
specs = await collectChangedSpecs(specs, {
|
|
@@ -5273,7 +5561,7 @@ async function runDispatcher(target, opts) {
|
|
|
5273
5561
|
if (typeof opts.retry === "number" && opts.retry > 0) warn("--retry is ignored without any 'mode: live' spec");
|
|
5274
5562
|
if (opts.out) warn("--out is ignored without any 'mode: live' spec");
|
|
5275
5563
|
if (opts.updateAgentPrompt) warn("--update-agent-prompt is ignored without any 'mode: live' spec");
|
|
5276
|
-
}
|
|
5564
|
+
} else if (opts.out && liveSpecs.length > 1) warn("--out is ignored when running multiple live specs");
|
|
5277
5565
|
if (detSpecs.length === 0 && opts.evidence === false) warn("--no-evidence is ignored without any 'mode: deterministic' spec");
|
|
5278
5566
|
blank();
|
|
5279
5567
|
const reportDir = resolveReportDir(opts.report, cwd);
|
|
@@ -5282,11 +5570,12 @@ async function runDispatcher(target, opts) {
|
|
|
5282
5570
|
const live = await runLiveSpecs(liveSpecs, {
|
|
5283
5571
|
...opts.model ? { model: opts.model } : {},
|
|
5284
5572
|
...opts.language ? { language: opts.language } : {},
|
|
5285
|
-
...opts.out ? { out: opts.out } : {},
|
|
5573
|
+
...opts.out && liveSpecs.length === 1 ? { out: opts.out } : {},
|
|
5286
5574
|
cwd,
|
|
5287
5575
|
...opts.base ? { base: opts.base } : {},
|
|
5288
5576
|
...reportDir ? { reportDir } : {},
|
|
5289
5577
|
...typeof opts.retry === "number" ? { retry: opts.retry } : {},
|
|
5578
|
+
concurrency: opts.concurrency ?? 1,
|
|
5290
5579
|
...reportDir && opts.driftAudit !== false ? { driftAudit: true } : {},
|
|
5291
5580
|
...reportDir && opts.failureAnalysis === false ? { failureAnalysis: false } : {}
|
|
5292
5581
|
});
|
|
@@ -5345,72 +5634,87 @@ async function runDeterministicSpecs(specs, opts, cwd, reportDirAbs) {
|
|
|
5345
5634
|
exitCode: 0
|
|
5346
5635
|
};
|
|
5347
5636
|
const tmpDir = await mkdtemp(join(tmpdir(), "ccqa-run-"));
|
|
5348
|
-
const summaries = [];
|
|
5349
|
-
let exitCode = 0;
|
|
5350
5637
|
const vitestConfig = await resolveVitestConfig(cwd);
|
|
5351
5638
|
const captureOutput = Boolean(opts.report);
|
|
5352
5639
|
const evidenceRoot = opts.evidence !== false ? join(reportDirAbs, EVIDENCE_SUBDIR) : null;
|
|
5640
|
+
const concurrency = Math.max(1, opts.concurrency ?? 1);
|
|
5641
|
+
const ctx = {
|
|
5642
|
+
cwd,
|
|
5643
|
+
tmpDir,
|
|
5644
|
+
vitestConfig,
|
|
5645
|
+
captureOutput,
|
|
5646
|
+
evidenceRoot
|
|
5647
|
+
};
|
|
5353
5648
|
try {
|
|
5354
|
-
|
|
5355
|
-
const { featureName, specName } = specs[i];
|
|
5356
|
-
const scriptFile = await getTestScript(featureName, specName, cwd);
|
|
5357
|
-
if (!scriptFile) {
|
|
5358
|
-
warn(`${featureName}/${specName}: no test.spec.ts found`);
|
|
5359
|
-
hint("run 'ccqa record <feature>/<spec>' to record it, or set 'mode: live' in spec.yaml");
|
|
5360
|
-
continue;
|
|
5361
|
-
}
|
|
5362
|
-
run(`${featureName}/${specName}`);
|
|
5363
|
-
meta("test", scriptFile);
|
|
5364
|
-
blank();
|
|
5365
|
-
const reportFile = join(tmpDir, `report-${i}.json`);
|
|
5366
|
-
const evidenceDir = evidenceRoot ? join(evidenceRoot, featureName, specName) : null;
|
|
5367
|
-
if (evidenceDir) {
|
|
5368
|
-
await rm(evidenceDir, {
|
|
5369
|
-
recursive: true,
|
|
5370
|
-
force: true
|
|
5371
|
-
});
|
|
5372
|
-
await mkdir(evidenceDir, { recursive: true });
|
|
5373
|
-
}
|
|
5374
|
-
const proc = spawnVitestStreaming([
|
|
5375
|
-
"run",
|
|
5376
|
-
"--config",
|
|
5377
|
-
vitestConfig,
|
|
5378
|
-
scriptFile,
|
|
5379
|
-
"--reporter=json",
|
|
5380
|
-
`--outputFile.json=${reportFile}`
|
|
5381
|
-
], {
|
|
5382
|
-
cwd,
|
|
5383
|
-
env: evidenceDir ? {
|
|
5384
|
-
...process.env,
|
|
5385
|
-
CCQA_EVIDENCE_DIR: evidenceDir
|
|
5386
|
-
} : process.env
|
|
5387
|
-
});
|
|
5388
|
-
const tail = captureOutput ? new TailBuffer(OUTPUT_TAIL_CAP) : null;
|
|
5389
|
-
await Promise.all([streamFiltered(proc.stdout, process.stdout, tail), streamFiltered(proc.stderr, process.stderr, tail)]);
|
|
5390
|
-
const specExitCode = await proc.exited;
|
|
5391
|
-
if (specExitCode !== 0) exitCode = specExitCode;
|
|
5392
|
-
const report = await readReport(reportFile);
|
|
5393
|
-
summaries.push({
|
|
5394
|
-
featureName,
|
|
5395
|
-
specName,
|
|
5396
|
-
scriptFile,
|
|
5397
|
-
report,
|
|
5398
|
-
exitCode: specExitCode,
|
|
5399
|
-
outputTail: tail ? tail.toString() : null,
|
|
5400
|
-
evidenceDir
|
|
5401
|
-
});
|
|
5402
|
-
blank();
|
|
5403
|
-
}
|
|
5649
|
+
const summaries = (await runPool(specs, concurrency, (spec, i) => withBuffer(`${spec.featureName}/${spec.specName}`, concurrency > 1, () => runOneDeterministicSpec(spec, i, ctx)))).filter((s) => s !== null);
|
|
5404
5650
|
printSummary(summaries);
|
|
5651
|
+
return {
|
|
5652
|
+
summaries,
|
|
5653
|
+
exitCode: summaries.reduce((acc, s) => s.exitCode !== 0 ? s.exitCode : acc, 0)
|
|
5654
|
+
};
|
|
5405
5655
|
} finally {
|
|
5406
5656
|
await rm(tmpDir, {
|
|
5407
5657
|
recursive: true,
|
|
5408
5658
|
force: true
|
|
5409
5659
|
});
|
|
5410
5660
|
}
|
|
5661
|
+
}
|
|
5662
|
+
/**
|
|
5663
|
+
* Run one spec under vitest. Returns null when the spec has no recorded
|
|
5664
|
+
* test.spec.ts (skipped). All output goes through the logger, so under a
|
|
5665
|
+
* `log.withBuffer` scope it's captured and flushed as one labelled block.
|
|
5666
|
+
*/
|
|
5667
|
+
async function runOneDeterministicSpec(spec, index, ctx) {
|
|
5668
|
+
const { featureName, specName } = spec;
|
|
5669
|
+
const scriptFile = await getTestScript(featureName, specName, ctx.cwd);
|
|
5670
|
+
if (!scriptFile) {
|
|
5671
|
+
warn(`${featureName}/${specName}: no test.spec.ts found`);
|
|
5672
|
+
hint("run 'ccqa record <feature>/<spec>' to record it, or set 'mode: live' in spec.yaml");
|
|
5673
|
+
return null;
|
|
5674
|
+
}
|
|
5675
|
+
run(`${featureName}/${specName}`);
|
|
5676
|
+
meta("test", scriptFile);
|
|
5677
|
+
const runId = buildRunId();
|
|
5678
|
+
meta("runId", runId);
|
|
5679
|
+
blank();
|
|
5680
|
+
const reportFile = join(ctx.tmpDir, `report-${index}.json`);
|
|
5681
|
+
const evidenceDir = ctx.evidenceRoot ? join(ctx.evidenceRoot, featureName, specName) : null;
|
|
5682
|
+
if (evidenceDir) {
|
|
5683
|
+
await rm(evidenceDir, {
|
|
5684
|
+
recursive: true,
|
|
5685
|
+
force: true
|
|
5686
|
+
});
|
|
5687
|
+
await mkdir(evidenceDir, { recursive: true });
|
|
5688
|
+
}
|
|
5689
|
+
const specEnv = {
|
|
5690
|
+
...process.env,
|
|
5691
|
+
CCQA_RUN_ID: runId
|
|
5692
|
+
};
|
|
5693
|
+
if (evidenceDir) specEnv.CCQA_EVIDENCE_DIR = evidenceDir;
|
|
5694
|
+
const proc = spawnVitestStreaming([
|
|
5695
|
+
"run",
|
|
5696
|
+
"--config",
|
|
5697
|
+
ctx.vitestConfig,
|
|
5698
|
+
scriptFile,
|
|
5699
|
+
"--reporter=json",
|
|
5700
|
+
`--outputFile.json=${reportFile}`
|
|
5701
|
+
], {
|
|
5702
|
+
cwd: ctx.cwd,
|
|
5703
|
+
env: specEnv
|
|
5704
|
+
});
|
|
5705
|
+
const sink = { write: emitRaw };
|
|
5706
|
+
const tail = ctx.captureOutput ? new TailBuffer(OUTPUT_TAIL_CAP) : null;
|
|
5707
|
+
await Promise.all([streamFiltered(proc.stdout, sink, tail), streamFiltered(proc.stderr, sink, tail)]);
|
|
5708
|
+
const specExitCode = await proc.exited;
|
|
5709
|
+
blank();
|
|
5411
5710
|
return {
|
|
5412
|
-
|
|
5413
|
-
|
|
5711
|
+
featureName,
|
|
5712
|
+
specName,
|
|
5713
|
+
scriptFile,
|
|
5714
|
+
report: await readReport(reportFile),
|
|
5715
|
+
exitCode: specExitCode,
|
|
5716
|
+
outputTail: tail ? tail.toString() : null,
|
|
5717
|
+
evidenceDir
|
|
5414
5718
|
};
|
|
5415
5719
|
}
|
|
5416
5720
|
function failedSpec(s) {
|
|
@@ -5859,6 +6163,7 @@ agent-browser --session SESSION wait --load networkidle
|
|
|
5859
6163
|
agent-browser --session SESSION get count "<selector>" # element-existence check (returns a number, fast)
|
|
5860
6164
|
agent-browser --session SESSION cookies clear
|
|
5861
6165
|
agent-browser --session SESSION find <locator> <value> <action> [<input>] [--name "<n>"] [--exact]
|
|
6166
|
+
agent-browser --session SESSION upload "<input[type=file] selector>" <file> [<file> ...]
|
|
5862
6167
|
# See "Selector Rules" for the full \`find\` subset.
|
|
5863
6168
|
# IMPORTANT: do NOT use \`wait "<css-selector>"\`. agent-browser ignores --timeout on a
|
|
5864
6169
|
# CSS-selector wait and blocks for ~150s when the selector never matches, killing the run.
|
|
@@ -5934,6 +6239,8 @@ find nth <index> "<ALLOWED-css>" <action>
|
|
|
5934
6239
|
|
|
5935
6240
|
**Verifying cleanup / deletion**: assert the *absence* of the deleted thing, not the surrounding listing screen's text. Use \`wait --fn "!document.body.innerText.includes('<unique-label>')"\` (text disappearance) — never \`wait "<css-selector>" --state hidden\` (blocks the daemon) and never \`wait --text "<navbar label>"\` (passes regardless of the deletion).
|
|
5936
6241
|
|
|
6242
|
+
**File inputs (\`<input type="file">\`) / OS file-picker dialogs**: do NOT \`click\` the input — that opens the OS picker, which agent-browser cannot drive. Use \`upload "<selector>" <path>\` instead. agent-browser sets the input's files directly via the underlying browser API, no native dialog ever opens. Use an ALLOWED selector to identify the input (\`[aria-label='…']\`, \`[data-testid='…']\`, \`[type='file']\` only when it's unique on the page). File paths must be plain shell args — wrap each in \`"\` for safety. Reference fixtures via \`\${CCQA_FIXTURES_DIR}/<name>\` so the same spec works locally and in CI; conventionally fixtures live under \`.ccqa/fixtures/\` and the env var resolves there. Multi-file inputs accept several positionals: \`upload "[aria-label='Attach']" "\${CCQA_FIXTURES_DIR}/a.pdf" "\${CCQA_FIXTURES_DIR}/b.pdf"\`.
|
|
6243
|
+
|
|
5937
6244
|
## Test Specification
|
|
5938
6245
|
|
|
5939
6246
|
Title: ${input.title}
|
|
@@ -6016,6 +6323,7 @@ AB_ACTION|select|<selector>|<value>|<aria label>
|
|
|
6016
6323
|
AB_ACTION|hover|<selector>|<visible label>
|
|
6017
6324
|
AB_ACTION|scroll|<direction>|<pixels>
|
|
6018
6325
|
AB_ACTION|drag|<source selector>|<target selector>|<source label>
|
|
6326
|
+
AB_ACTION|upload|<file-input selector>|<file1>[|<file2>...]
|
|
6019
6327
|
AB_ACTION|wait|<selector or text>|<label>
|
|
6020
6328
|
AB_ACTION|snapshot|<key observation, max 100 chars>
|
|
6021
6329
|
AB_ACTION|assert|<assertType>|<selector or "">|<value or "">|<observation>
|
|
@@ -6332,6 +6640,17 @@ function actionToAbArgs(action, sessionName) {
|
|
|
6332
6640
|
sub(action.selector),
|
|
6333
6641
|
sub(action.target)
|
|
6334
6642
|
];
|
|
6643
|
+
case "upload": {
|
|
6644
|
+
const sel = sub(action.selector);
|
|
6645
|
+
const files = (action.files ?? []).map((f) => sub(f));
|
|
6646
|
+
if (!sel || files.length === 0) return null;
|
|
6647
|
+
return [
|
|
6648
|
+
...base,
|
|
6649
|
+
"upload",
|
|
6650
|
+
sel,
|
|
6651
|
+
...files
|
|
6652
|
+
];
|
|
6653
|
+
}
|
|
6335
6654
|
case "wait": {
|
|
6336
6655
|
const raw = sub(action.selector);
|
|
6337
6656
|
if (!raw) return null;
|
|
@@ -6824,7 +7143,7 @@ async function runTrace(featureName, specName, model, validationMode = "lenient"
|
|
|
6824
7143
|
sessionName
|
|
6825
7144
|
});
|
|
6826
7145
|
const promptBundle = await loadRecordPromptBundle();
|
|
6827
|
-
if (promptBundle !== null) meta("
|
|
7146
|
+
if (promptBundle !== null) meta("prompt", promptBundle.loaded.join(" + "));
|
|
6828
7147
|
const systemPrompt = (promptBundle === null ? baseSystemPrompt : `${baseSystemPrompt}\n## Project-specific guidance\n\n${promptBundle.text}\n`) + languageDirective(language);
|
|
6829
7148
|
const prompt = buildTracePrompt(spec.title);
|
|
6830
7149
|
info("Running agent-browser session...");
|
|
@@ -6970,7 +7289,7 @@ function dedupAndReport(actions) {
|
|
|
6970
7289
|
function isAdjacentDuplicate(a, b) {
|
|
6971
7290
|
if (a.command !== b.command) return false;
|
|
6972
7291
|
if ((a.stepId ?? "") !== (b.stepId ?? "")) return false;
|
|
6973
|
-
return (a.selector ?? "") === (b.selector ?? "") && (a.value ?? "") === (b.value ?? "") && (a.target ?? "") === (b.target ?? "") && (a.label ?? "") === (b.label ?? "") && (a.assertType ?? "") === (b.assertType ?? "") && (a.findLocator ?? "") === (b.findLocator ?? "") && (a.findValue ?? "") === (b.findValue ?? "") && (a.findName ?? "") === (b.findName ?? "") && (a.findIndex ?? -1) === (b.findIndex ?? -1) && (a.findExact ?? false) === (b.findExact ?? false);
|
|
7292
|
+
return (a.selector ?? "") === (b.selector ?? "") && (a.value ?? "") === (b.value ?? "") && (a.target ?? "") === (b.target ?? "") && (a.label ?? "") === (b.label ?? "") && (a.assertType ?? "") === (b.assertType ?? "") && (a.findLocator ?? "") === (b.findLocator ?? "") && (a.findValue ?? "") === (b.findValue ?? "") && (a.findName ?? "") === (b.findName ?? "") && (a.findIndex ?? -1) === (b.findIndex ?? -1) && (a.findExact ?? false) === (b.findExact ?? false) && (a.files ?? []).join("|") === (b.files ?? []).join("|");
|
|
6974
7293
|
}
|
|
6975
7294
|
/**
|
|
6976
7295
|
* Run the post-trace replay validation and emit user-visible drop reports.
|
|
@@ -7192,6 +7511,16 @@ function parseAbAction(line) {
|
|
|
7192
7511
|
target: parts[3],
|
|
7193
7512
|
label: parts[4]
|
|
7194
7513
|
};
|
|
7514
|
+
case "upload": {
|
|
7515
|
+
const selector = parts[2];
|
|
7516
|
+
const files = parts.slice(3).filter((f) => f !== "");
|
|
7517
|
+
if (!selector || files.length === 0) return null;
|
|
7518
|
+
return {
|
|
7519
|
+
command,
|
|
7520
|
+
selector,
|
|
7521
|
+
files
|
|
7522
|
+
};
|
|
7523
|
+
}
|
|
7195
7524
|
case "find_click":
|
|
7196
7525
|
case "find_dblclick":
|
|
7197
7526
|
case "find_hover":
|
|
@@ -7242,6 +7571,7 @@ function actionsToScript(input) {
|
|
|
7242
7571
|
`import { ${[
|
|
7243
7572
|
"ab",
|
|
7244
7573
|
"abWait",
|
|
7574
|
+
"abUpload",
|
|
7245
7575
|
"abAssertTextVisible",
|
|
7246
7576
|
"abAssertVisible",
|
|
7247
7577
|
"abAssertNotVisible",
|
|
@@ -7275,6 +7605,7 @@ const ELEMENT_COMMANDS = new Set([
|
|
|
7275
7605
|
"select",
|
|
7276
7606
|
"hover",
|
|
7277
7607
|
"drag",
|
|
7608
|
+
"upload",
|
|
7278
7609
|
"find_click",
|
|
7279
7610
|
"find_dblclick",
|
|
7280
7611
|
"find_fill",
|
|
@@ -7406,6 +7737,11 @@ function actionToLine(action) {
|
|
|
7406
7737
|
case "hover": return `ab("hover", ${j(action.selector)});`;
|
|
7407
7738
|
case "scroll": return `ab("scroll", ${[action.direction ?? "down", ...action.pixels ? [action.pixels] : []].map(j).join(", ")});`;
|
|
7408
7739
|
case "drag": return `ab("drag", ${j(action.selector)}, ${j(action.target)});`;
|
|
7740
|
+
case "upload": {
|
|
7741
|
+
const files = action.files ?? [];
|
|
7742
|
+
if (!action.selector || files.length === 0) return null;
|
|
7743
|
+
return `abUpload(${[j(action.selector), ...files.map(jExpr)].join(", ")});`;
|
|
7744
|
+
}
|
|
7409
7745
|
case "wait": {
|
|
7410
7746
|
const sel = action.selector;
|
|
7411
7747
|
if (/^\d+$/.test(sel)) return `spawnSync("sleep", [${j(sel)}], { stdio: "inherit" });`;
|
|
@@ -8481,19 +8817,20 @@ function toFixMode(autoFix) {
|
|
|
8481
8817
|
case "interactive": return "interactive";
|
|
8482
8818
|
}
|
|
8483
8819
|
}
|
|
8484
|
-
const recordCommand = addLanguageOption(new Command("record").argument("<feature/spec>", "Spec id in '<feature>/<spec>' form (resolves to .ccqa/features/<feature>/test-cases/<spec>/)").description("Record a deterministic test from a spec: run agent-browser to collect actions (trace), then generate test.spec.ts with auto-fix retries (generate). After recording, `ccqa run <feature/spec>` replays it under vitest (deterministic specs only — live specs do not need recording).").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--validation-mode <mode>", "Post-trace validation behaviour: 'lenient' (default) tags failing actions; 'strict' drops them.", (raw) => {
|
|
8820
|
+
const recordCommand = addProfileOption(addLanguageOption(new Command("record").argument("<feature/spec>", "Spec id in '<feature>/<spec>' form (resolves to .ccqa/features/<feature>/test-cases/<spec>/)").description("Record a deterministic test from a spec: run agent-browser to collect actions (trace), then generate test.spec.ts with auto-fix retries (generate). After recording, `ccqa run <feature/spec>` replays it under vitest (deterministic specs only — live specs do not need recording).").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--validation-mode <mode>", "Post-trace validation behaviour: 'lenient' (default) tags failing actions; 'strict' drops them.", (raw) => {
|
|
8485
8821
|
if (VALIDATION_MODES.includes(raw)) return raw;
|
|
8486
8822
|
throw new Error(`--validation-mode must be one of ${VALIDATION_MODES.join(" | ")}`);
|
|
8487
8823
|
}, "lenient").option("--auto-fix <mode>", "Auto-fix behaviour during script generation: 'interactive' (default, prompt y/N), 'auto' (apply without prompt, for CI), 'skip' (never prompt, only apply high-confidence fixes).", (raw) => {
|
|
8488
8824
|
if (AUTO_FIX_MODES.includes(raw)) return raw;
|
|
8489
8825
|
throw new Error(`--auto-fix must be one of ${AUTO_FIX_MODES.join(" | ")}`);
|
|
8490
|
-
}, "interactive").option("--max-retries <n>", "Maximum number of auto-fix retries", "3").option("--force", "Overwrite an existing test.spec.ts without warning").option("--no-snapshot", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").option("--skip-trace", "Skip the trace step and run codegen against an existing actions.json").option("--skip-codegen", "Run only the trace step (do not generate test.spec.ts)").option("--update-agent-prompt", "After the trace finishes, ask Claude to refresh .ccqa/prompts/record.agent.md from a summary of the run.").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.")).action(async (specPath, opts) => {
|
|
8826
|
+
}, "interactive").option("--max-retries <n>", "Maximum number of auto-fix retries", "3").option("--force", "Overwrite an existing test.spec.ts without warning").option("--no-snapshot", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").option("--skip-trace", "Skip the trace step and run codegen against an existing actions.json").option("--skip-codegen", "Run only the trace step (do not generate test.spec.ts)").option("--update-agent-prompt", "After the trace finishes, ask Claude to refresh .ccqa/prompts/record.agent.md from a summary of the run.").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory."))).action(async (specPath, opts) => {
|
|
8491
8827
|
const { featureName, specName } = parseSpecPath(specPath);
|
|
8492
8828
|
const language = opts.language ?? "auto";
|
|
8493
8829
|
if (opts.skipTrace && opts.skipCodegen) {
|
|
8494
8830
|
error("--skip-trace and --skip-codegen cannot be combined; nothing would run");
|
|
8495
8831
|
process.exit(2);
|
|
8496
8832
|
}
|
|
8833
|
+
await applyProfileFromOption(opts.profile, resolveCwd(opts.cwd));
|
|
8497
8834
|
let traceResult = null;
|
|
8498
8835
|
if (!opts.skipTrace) {
|
|
8499
8836
|
traceResult = await runTrace(featureName, specName, opts.model, opts.validationMode ?? "lenient", language);
|
package/dist/package.json
CHANGED
|
@@ -3,6 +3,13 @@ declare function __setCurrentStep(stepId: string, source: string): void;
|
|
|
3
3
|
declare function ab(...args: string[]): void;
|
|
4
4
|
/** Wait for element/text with an explicit timeout so long-running async ops don't hang. */
|
|
5
5
|
declare function abWait(selector: string, timeoutMs?: number): void;
|
|
6
|
+
/**
|
|
7
|
+
* Upload one or more files to a file input via `agent-browser upload`.
|
|
8
|
+
* Relative paths resolve against the test's cwd. We pre-check existence
|
|
9
|
+
* locally so a typo in a fixture path surfaces as a clear error before
|
|
10
|
+
* agent-browser exits with an opaque non-zero status.
|
|
11
|
+
*/
|
|
12
|
+
declare function abUpload(selector: string, ...files: string[]): void;
|
|
6
13
|
/** Assert stable text is visible on page (via wait --text). */
|
|
7
14
|
declare function abAssertTextVisible(text: string, timeoutMs?: number): void;
|
|
8
15
|
/** Assert element is visible (polls `get count`; never uses the blocking `wait <selector>`). */
|
|
@@ -29,4 +36,4 @@ declare function abAssertUnchecked(selector: string): void;
|
|
|
29
36
|
*/
|
|
30
37
|
declare function abStepEvidence(stepId: string, source: string): void;
|
|
31
38
|
//#endregion
|
|
32
|
-
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abWait };
|
|
39
|
+
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abUpload, abWait };
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { i as FAILURE_STEP_ID, n as spawnAB, r as FAILURE_SOURCE, t as sleepSync } from "../spawn-ab-Ja8NRRab.mjs";
|
|
2
|
-
import { mkdirSync, writeFileSync } from "node:fs";
|
|
3
|
-
import { dirname, join } from "node:path";
|
|
2
|
+
import { existsSync, mkdirSync, writeFileSync } from "node:fs";
|
|
3
|
+
import { dirname, isAbsolute, join, resolve } from "node:path";
|
|
4
4
|
//#region src/runtime/test-helpers.ts
|
|
5
5
|
const POST_OPEN_SETTLE_MS = 600;
|
|
6
6
|
function logStep(action, args) {
|
|
@@ -100,6 +100,31 @@ function abWait(selector, timeoutMs = 3e4) {
|
|
|
100
100
|
stderr: ""
|
|
101
101
|
});
|
|
102
102
|
}
|
|
103
|
+
/**
|
|
104
|
+
* Upload one or more files to a file input via `agent-browser upload`.
|
|
105
|
+
* Relative paths resolve against the test's cwd. We pre-check existence
|
|
106
|
+
* locally so a typo in a fixture path surfaces as a clear error before
|
|
107
|
+
* agent-browser exits with an opaque non-zero status.
|
|
108
|
+
*/
|
|
109
|
+
function abUpload(selector, ...files) {
|
|
110
|
+
logStep("upload", [selector, ...files]);
|
|
111
|
+
const resolved = [];
|
|
112
|
+
for (const file of files) {
|
|
113
|
+
const abs = isAbsolute(file) ? file : resolve(process.cwd(), file);
|
|
114
|
+
if (!existsSync(abs)) fail(`abUpload: file not found (${file} → ${abs})`, {
|
|
115
|
+
status: 1,
|
|
116
|
+
stdout: "",
|
|
117
|
+
stderr: ""
|
|
118
|
+
});
|
|
119
|
+
resolved.push(abs);
|
|
120
|
+
}
|
|
121
|
+
const result = spawnAB([
|
|
122
|
+
"upload",
|
|
123
|
+
selector,
|
|
124
|
+
...resolved
|
|
125
|
+
]);
|
|
126
|
+
if (result.status !== 0) fail(`agent-browser upload failed (exit ${result.status})`, result);
|
|
127
|
+
}
|
|
103
128
|
/** Assert stable text is visible on page (via wait --text). */
|
|
104
129
|
function abAssertTextVisible(text, timeoutMs = 3e4) {
|
|
105
130
|
logStep("assert.text", [text]);
|
|
@@ -289,4 +314,4 @@ function warnEvidence(msg) {
|
|
|
289
314
|
process.stderr.write(`[ccqa] evidence: ${msg}\n`);
|
|
290
315
|
}
|
|
291
316
|
//#endregion
|
|
292
|
-
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abWait };
|
|
317
|
+
export { __setCurrentStep, ab, abAssertChecked, abAssertDisabled, abAssertEnabled, abAssertNotVisible, abAssertTextVisible, abAssertUnchecked, abAssertUrl, abAssertVisible, abStepEvidence, abUpload, abWait };
|