ccqa 1.13.0 → 1.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -8
- package/dist/bin/ccqa.mjs +131 -122
- package/dist/hub-client/index.d.mts +3 -3
- package/dist/package.json +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -152,7 +152,7 @@ where the shared state lives — there is no second place to put it:
|
|
|
152
152
|
current by `record`/`generate`
|
|
153
153
|
- the variables `${…}` resolve to, and saved browser sessions, fetched at run
|
|
154
154
|
time — so CI holds one secret instead of an environment
|
|
155
|
-
- the deploy log behind `--only-stale`, and the drift ledger
|
|
155
|
+
- the deploy log behind `--only-hub-stale`, and the drift ledger
|
|
156
156
|
- a dashboard of runs with per-step screenshots, triage grading, and the
|
|
157
157
|
prompts learned from those grades
|
|
158
158
|
|
|
@@ -186,7 +186,7 @@ All three need two things:
|
|
|
186
186
|
- **A Claude credential.** Replaying a recorded spec uses no model, but the
|
|
187
187
|
change selection, the failure analysis and the audit all do.
|
|
188
188
|
- **A running [hub](#the-hub)**, reached with `CCQA_HUB_URL` and
|
|
189
|
-
`CCQA_HUB_TOKEN`. Only a pre-merge run with no `--profile` and no
|
|
189
|
+
`CCQA_HUB_TOKEN`. Only a pre-merge run with no `--hub-profile` and no
|
|
190
190
|
`--report-to-hub` can do without one.
|
|
191
191
|
|
|
192
192
|
See [Environment variables](./docs/commands.md#environment-variables) for the
|
|
@@ -201,7 +201,7 @@ once, from your machine:
|
|
|
201
201
|
ccqa hub var set APP_URL --value https://app.example --profile staging
|
|
202
202
|
```
|
|
203
203
|
|
|
204
|
-
Pass the same `--profile` and `--project` in every job. That is what makes the
|
|
204
|
+
Pass the same `--hub-profile` and `--project` in every job. That is what makes the
|
|
205
205
|
jobs refer to the same environment.
|
|
206
206
|
|
|
207
207
|
### On a pull request
|
|
@@ -209,14 +209,14 @@ jobs refer to the same environment.
|
|
|
209
209
|
Run the specs the change reaches, and label what broke.
|
|
210
210
|
|
|
211
211
|
```bash
|
|
212
|
-
ccqa run --only-affected-by --on-fail-explain --profile staging \
|
|
212
|
+
ccqa run --only-affected-by --on-fail-explain --hub-profile staging \
|
|
213
213
|
--report-format github --report-to-hub
|
|
214
214
|
```
|
|
215
215
|
|
|
216
216
|
- `--only-affected-by` selects the specs the diff reaches. A spec it cannot clear runs
|
|
217
217
|
anyway.
|
|
218
218
|
- `--on-fail-explain` labels the cause of each failure.
|
|
219
|
-
- `--profile staging` fetches that environment's variables and saved sessions
|
|
219
|
+
- `--hub-profile staging` fetches that environment's variables and saved sessions
|
|
220
220
|
from the hub. Without it, a spec's `${…}` references go unresolved.
|
|
221
221
|
- `--report-format github` annotates the pull request.
|
|
222
222
|
- `--report-to-hub` streams results to the hub as the run executes.
|
|
@@ -242,12 +242,12 @@ ccqa hub deploy record --profile staging --sha "$GITHUB_SHA" --select
|
|
|
242
242
|
Then, in a job of its own, run what that deploy invalidated:
|
|
243
243
|
|
|
244
244
|
```bash
|
|
245
|
-
ccqa run --only-stale --profile staging --report-to-hub
|
|
245
|
+
ccqa run --only-hub-stale --hub-profile staging --report-to-hub
|
|
246
246
|
```
|
|
247
247
|
|
|
248
248
|
- `--select` records which specs the deployed range reaches. Without it, every
|
|
249
249
|
spec behind that entry answers `unknown` instead of `notNeeded`.
|
|
250
|
-
- `--only-stale` asks the hub, per spec, whether any deploy has touched
|
|
250
|
+
- `--only-hub-stale` asks the hub, per spec, whether any deploy has touched
|
|
251
251
|
it since that spec last ran.
|
|
252
252
|
|
|
253
253
|
The hub has no checkout and never runs `git`, so it cannot work out what a
|
|
@@ -260,7 +260,7 @@ a hole nothing can fill in afterwards.
|
|
|
260
260
|
runs by default. Record a deploy, run every spec once with `--report-to-hub`,
|
|
261
261
|
and the selection means something from the next deploy on. This job also reads
|
|
262
262
|
the spec inventory from the hub, so `ccqa perspectives` has to have run.
|
|
263
|
-
`--only-stale-with-unknown` opts the undecided specs in.
|
|
263
|
+
`--only-hub-stale-with-unknown` opts the undecided specs in.
|
|
264
264
|
|
|
265
265
|
### On a schedule
|
|
266
266
|
|
package/dist/bin/ccqa.mjs
CHANGED
|
@@ -477,7 +477,7 @@ function getBlocksDir(cwd) {
|
|
|
477
477
|
/**
|
|
478
478
|
* Inverse of `getBlockDir`. Given a file path that appears in a git diff,
|
|
479
479
|
* return the block name if the path points at the block's spec.yaml, else
|
|
480
|
-
* null. Used by `
|
|
480
|
+
* null. Used by `audit --only-affected-by` to invalidate specs whose included blocks
|
|
481
481
|
* were edited. (v0.4 inlines blocks into every spec's own trace, so the
|
|
482
482
|
* block directory holds only spec.yaml — no per-block recording lives
|
|
483
483
|
* here anymore.)
|
|
@@ -524,24 +524,22 @@ const USER_PROMPT_MAX_BYTES = 32768;
|
|
|
524
524
|
* Load the prompt bundle from the hub for one guidance kind ("record" /
|
|
525
525
|
* "live" / an LLM-generation target such as "playwright" or "runn").
|
|
526
526
|
* Best-effort: no hub client, a fetch failure, or both prompts absent all
|
|
527
|
-
*
|
|
527
|
+
* A prompt that was never stored resolves to null. A hub that cannot be
|
|
528
|
+
* reached throws: running with silently different guidance than the project
|
|
529
|
+
* configured is worse than stopping.
|
|
528
530
|
*/
|
|
529
531
|
async function loadPromptBundleFromHub(ctx, kind) {
|
|
530
532
|
if (!ctx) return null;
|
|
531
533
|
const userName = `${kind}.user`;
|
|
532
534
|
const agentName = `${kind}.agent`;
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
});
|
|
542
|
-
} catch {
|
|
543
|
-
return null;
|
|
544
|
-
}
|
|
535
|
+
const [userText, agentText] = await Promise.all([ctx.hub.getPrompt(ctx.project, userName).then(normalizePromptText), ctx.hub.getPrompt(ctx.project, agentName).then(normalizePromptText)]);
|
|
536
|
+
return assemblePromptBundle({
|
|
537
|
+
text: userText,
|
|
538
|
+
label: userName
|
|
539
|
+
}, {
|
|
540
|
+
text: agentText,
|
|
541
|
+
label: agentName
|
|
542
|
+
});
|
|
545
543
|
}
|
|
546
544
|
/**
|
|
547
545
|
* Shared concatenation logic behind `loadPromptBundleFromHub`: section
|
|
@@ -557,7 +555,7 @@ function assemblePromptBundle(user, agent) {
|
|
|
557
555
|
loaded.push(user.label);
|
|
558
556
|
}
|
|
559
557
|
if (agent.text !== null) {
|
|
560
|
-
sections.push(`### Agent learnings (auto-updated by ccqa --
|
|
558
|
+
sections.push(`### Agent learnings (auto-updated by ccqa's --learn-*-prompt flags)\n\n${agent.text}`);
|
|
561
559
|
loaded.push(agent.label);
|
|
562
560
|
}
|
|
563
561
|
let text = sections.join("\n\n");
|
|
@@ -2475,7 +2473,7 @@ const FailureAnalysisSchema = z.object({
|
|
|
2475
2473
|
});
|
|
2476
2474
|
/**
|
|
2477
2475
|
* What a drift audit may conclude, in the same vocabulary `ccqa run
|
|
2478
|
-
* --
|
|
2476
|
+
* --on-fail-explain` uses for a failure. One question, one answer, the same
|
|
2479
2477
|
* words whether it was reached by running the spec or by reading the code — so
|
|
2480
2478
|
* a reader never translates between two taxonomies, and the hub renders,
|
|
2481
2479
|
* grades and learns from both through one path.
|
|
@@ -3312,7 +3310,7 @@ function relativeToCwd(path, cwd) {
|
|
|
3312
3310
|
/** The model's reply: a diagnosis, or `null` for "the spec still matches the code". */
|
|
3313
3311
|
const DriftReplySchema = z.object({ drift: DriftDiagnosisSchema.nullable() });
|
|
3314
3312
|
/**
|
|
3315
|
-
* How a label reads against `--
|
|
3313
|
+
* How a label reads against `--exit-on`. The threshold asks "would a
|
|
3316
3314
|
* deterministic replay fail today", which is what the label already answers:
|
|
3317
3315
|
* both findings mean the spec no longer describes the code, while `UNKNOWN`
|
|
3318
3316
|
* means the audit could not tell and should not fail a build on its own.
|
|
@@ -3971,14 +3969,10 @@ ${customPrompt.guidance}
|
|
|
3971
3969
|
*/
|
|
3972
3970
|
async function fetchCustomPrompt(ctx) {
|
|
3973
3971
|
if (!ctx) return null;
|
|
3974
|
-
|
|
3975
|
-
|
|
3976
|
-
|
|
3977
|
-
|
|
3978
|
-
return parsed.success ? parsed.data : null;
|
|
3979
|
-
} catch {
|
|
3980
|
-
return null;
|
|
3981
|
-
}
|
|
3972
|
+
const raw = await ctx.hub.getPrompt(ctx.project, "analysis-custom-prompt");
|
|
3973
|
+
if (raw === null) return null;
|
|
3974
|
+
const parsed = AnalysisCustomPromptSchema.safeParse(JSON.parse(raw));
|
|
3975
|
+
return parsed.success ? parsed.data : null;
|
|
3982
3976
|
}
|
|
3983
3977
|
/**
|
|
3984
3978
|
* Render the human-maintained `triage.user` guidance as a prompt section, or
|
|
@@ -4006,12 +4000,8 @@ ${trimmed}
|
|
|
4006
4000
|
*/
|
|
4007
4001
|
async function fetchTriageUserPrompt(ctx) {
|
|
4008
4002
|
if (!ctx) return null;
|
|
4009
|
-
|
|
4010
|
-
|
|
4011
|
-
return trimmed ? trimmed : null;
|
|
4012
|
-
} catch {
|
|
4013
|
-
return null;
|
|
4014
|
-
}
|
|
4003
|
+
const trimmed = (await ctx.hub.getPrompt(ctx.project, "triage.user"))?.trim();
|
|
4004
|
+
return trimmed ? trimmed : null;
|
|
4015
4005
|
}
|
|
4016
4006
|
/**
|
|
4017
4007
|
* Short, stable content hash for a `triage.user` prompt. The Markdown body
|
|
@@ -4812,12 +4802,12 @@ async function readDotenv(path) {
|
|
|
4812
4802
|
}
|
|
4813
4803
|
return parseDotenv(content);
|
|
4814
4804
|
}
|
|
4815
|
-
/** Absolute path of the default `.env` ccqa loads when `--profile` is absent. */
|
|
4805
|
+
/** Absolute path of the default `.env` ccqa loads when `--hub-profile` is absent. */
|
|
4816
4806
|
function defaultEnvPath(cwd) {
|
|
4817
4807
|
return join(cwd, ".env");
|
|
4818
4808
|
}
|
|
4819
4809
|
/**
|
|
4820
|
-
* Load `<cwd>/.env`, the default when no `--profile` is given. A missing `.env`
|
|
4810
|
+
* Load `<cwd>/.env`, the default when no `--hub-profile` is given. A missing `.env`
|
|
4821
4811
|
* is fine (returns `null`) — the run falls back to the existing `process.env`.
|
|
4822
4812
|
*/
|
|
4823
4813
|
async function loadDefaultEnv(cwd) {
|
|
@@ -5023,7 +5013,7 @@ function addLanguageOption(command) {
|
|
|
5023
5013
|
* `record`), registered identically so help text and behaviour don't drift.
|
|
5024
5014
|
*/
|
|
5025
5015
|
function addProfileOption(command) {
|
|
5026
|
-
return command.option("--profile <name>", "Load this profile's variables from the hub into the environment before resolving spec ${VAR} references (URLs, credentials), so one spec can target dev/stg/prd without per-environment copies. Profile values override the inherited environment. Requires --hub-url/--hub-token (or CCQA_HUB_URL/CCQA_HUB_TOKEN).");
|
|
5016
|
+
return command.option("--hub-profile <name>", "Load this profile's variables from the hub into the environment before resolving spec ${VAR} references (URLs, credentials), so one spec can target dev/stg/prd without per-environment copies. Profile values override the inherited environment. Requires --hub-url/--hub-token (or CCQA_HUB_URL/CCQA_HUB_TOKEN).");
|
|
5027
5017
|
}
|
|
5028
5018
|
/**
|
|
5029
5019
|
* Shared `--hub-url` / `--hub-token` flags for commands that optionally talk
|
|
@@ -5901,7 +5891,7 @@ async function runVerificationLoop(p, ref, state) {
|
|
|
5901
5891
|
const runCommand = p.ctx.targetConfig.runCommand;
|
|
5902
5892
|
if (!runCommand) return true;
|
|
5903
5893
|
const maxRetries = p.ctx.fix.mode === "non-interactive" ? 0 : p.ctx.fix.maxRetries;
|
|
5904
|
-
if (!p.ctx.fix.useSnapshot) warn(`--no-
|
|
5894
|
+
if (!p.ctx.fix.useSnapshot) warn(`--no-session-pin has no effect on the ${p.target} target — it captures no browser snapshot; the fix loop uses the command's output instead`);
|
|
5905
5895
|
for (let attempt = 0;; attempt++) {
|
|
5906
5896
|
const testFiles = [...state.entries()].filter(([, f]) => f.kind === "test").map(([rel]) => rel);
|
|
5907
5897
|
const artifactsDir = await mkdtemp(join(tmpdir(), "ccqa-verify-artifacts-"));
|
|
@@ -6089,8 +6079,8 @@ const C$1 = {
|
|
|
6089
6079
|
* script, so it keeps its own caller in `cli/run-live.ts`; only the
|
|
6090
6080
|
* `ANALYSIS_DISABLED` string is shared with it.
|
|
6091
6081
|
*/
|
|
6092
|
-
/** `analysisSkipped` for a failed row when `--
|
|
6093
|
-
const ANALYSIS_DISABLED = "skipped: --
|
|
6082
|
+
/** `analysisSkipped` for a failed row when `--on-fail-explain` was not requested. */
|
|
6083
|
+
const ANALYSIS_DISABLED = "skipped: --on-fail-explain not enabled";
|
|
6094
6084
|
/**
|
|
6095
6085
|
* Create one analysis pass. The returned object is stateful on purpose: the
|
|
6096
6086
|
* "source diff unavailable" notice and the summary block's header are printed
|
|
@@ -6610,7 +6600,7 @@ async function detectDefaultBranch(cwd) {
|
|
|
6610
6600
|
/**
|
|
6611
6601
|
* Fetch the last-green ledger for this run — one hub round trip, logged as
|
|
6612
6602
|
* the run's analysis-base meta line. Fails fast (RunUsageError) when the hub
|
|
6613
|
-
* can't serve it: `--
|
|
6603
|
+
* can't serve it: `--on-fail-explain` explicitly opted into
|
|
6614
6604
|
* hub-backed baselines, so a broken hub connection is a usage error, never a
|
|
6615
6605
|
* silent no-baseline run.
|
|
6616
6606
|
*/
|
|
@@ -6633,7 +6623,7 @@ async function fetchLastGreenLedger(hubCtx, profile, cwd) {
|
|
|
6633
6623
|
return entries;
|
|
6634
6624
|
}
|
|
6635
6625
|
/**
|
|
6636
|
-
* Per-spec baseline resolver for `--
|
|
6626
|
+
* Per-spec baseline resolver for `--on-fail-explain` without an explicit base. A spec
|
|
6637
6627
|
* missing from the ledger (never green on a pushed run yet) or whose
|
|
6638
6628
|
* baseline commit isn't in this checkout resolves to a skip — the run
|
|
6639
6629
|
* continues; only that spec's classification is withheld, with the reason
|
|
@@ -6687,7 +6677,7 @@ async function deployHeadSha(hub, project, profile) {
|
|
|
6687
6677
|
*
|
|
6688
6678
|
* Captured before any spec executes and asserted on both push paths
|
|
6689
6679
|
* (`?deployedSha=` on `POST /runs` via `ccqa hub push`, and on `POST
|
|
6690
|
-
* /runs/open` for `--
|
|
6680
|
+
* /runs/open` for `--report-to-hub`). Left to itself the hub reads its own
|
|
6691
6681
|
* deploy-log head when the call lands — after the whole run for a single-shot
|
|
6692
6682
|
* push, after the deterministic phase for an incremental one — so a deploy
|
|
6693
6683
|
* landing in that window would be recorded as the run's baseline and
|
|
@@ -6712,7 +6702,7 @@ async function tryDeployHeadSha(hubCtx, profile) {
|
|
|
6712
6702
|
*
|
|
6713
6703
|
* This exists because a selection can be wrong in a way that costs money.
|
|
6714
6704
|
* `ccqa select-specs`'s model judgment is not infallible, and both
|
|
6715
|
-
* `--
|
|
6705
|
+
* `--only-affected-by` and `--only-hub-stale` decide from it, so a human has to
|
|
6716
6706
|
* be able to read the selection back before a live spec spends a Claude
|
|
6717
6707
|
* budget on it.
|
|
6718
6708
|
*
|
|
@@ -6758,7 +6748,7 @@ async function fetchAuditedLedger(hubCtx) {
|
|
|
6758
6748
|
try {
|
|
6759
6749
|
ledger = await hubCtx.hub.getDriftLedger(hubCtx.project);
|
|
6760
6750
|
} catch (err) {
|
|
6761
|
-
throw new RunUsageError(`--only-audited-clean: could not fetch the drift ledger from the hub: ${errMessage(err)}`);
|
|
6751
|
+
throw new RunUsageError(`--only-hub-audited-clean: could not fetch the drift ledger from the hub: ${errMessage(err)}`);
|
|
6762
6752
|
}
|
|
6763
6753
|
const clean = /* @__PURE__ */ new Set();
|
|
6764
6754
|
const audited = /* @__PURE__ */ new Set();
|
|
@@ -6790,7 +6780,7 @@ function selectAuditedClean(specs, ledger) {
|
|
|
6790
6780
|
//#endregion
|
|
6791
6781
|
//#region src/run/rerun-selection.ts
|
|
6792
6782
|
/**
|
|
6793
|
-
* `ccqa run --
|
|
6783
|
+
* `ccqa run --only-hub-stale`: select specs from the hub's re-run verdicts
|
|
6794
6784
|
* instead of from a git diff (ADR-0010). The baseline is not a ref at all —
|
|
6795
6785
|
* it is each spec's own last run, positioned against the deploy log the
|
|
6796
6786
|
* consuming deploy job feeds the hub — so this path does no git work.
|
|
@@ -6802,12 +6792,12 @@ function selectAuditedClean(specs, ledger) {
|
|
|
6802
6792
|
/** First ccqa release whose hub serves `GET /projects/:project/rerun`. */
|
|
6803
6793
|
const RERUN_MIN_HUB_VERSION = "1.9";
|
|
6804
6794
|
/**
|
|
6805
|
-
* The profile `--
|
|
6795
|
+
* The profile `--only-hub-stale` asks about. Mandatory: two environments sit
|
|
6806
6796
|
* at different commits and the deploy log is per-profile, so "needs re-run"
|
|
6807
6797
|
* has no profile-free answer.
|
|
6808
6798
|
*/
|
|
6809
6799
|
function requireRerunProfile(profile) {
|
|
6810
|
-
if (profile === void 0) throw new RunUsageError("--only-stale requires --profile <name>: the deploy log it reads is per-profile, so which specs need a re-run has no answer without one");
|
|
6800
|
+
if (profile === void 0) throw new RunUsageError("--only-hub-stale requires --hub-profile <name>: the deploy log it reads is per-profile, so which specs need a re-run has no answer without one");
|
|
6811
6801
|
return profile;
|
|
6812
6802
|
}
|
|
6813
6803
|
/**
|
|
@@ -6820,9 +6810,9 @@ async function fetchRerunReport(hubCtx, profile) {
|
|
|
6820
6810
|
report = await hubCtx.hub.getRerun(hubCtx.project, { profile });
|
|
6821
6811
|
} catch (err) {
|
|
6822
6812
|
if (err instanceof HubApiError && err.status === 404) throw new RunUsageError(explainNotFound(hubCtx, err));
|
|
6823
|
-
throw new RunUsageError(`--only-stale: could not ask the hub which specs need a re-run: ${errMessage(err)}`);
|
|
6813
|
+
throw new RunUsageError(`--only-hub-stale: could not ask the hub which specs need a re-run: ${errMessage(err)}`);
|
|
6824
6814
|
}
|
|
6825
|
-
if (report.deployHead === null) throw new RunUsageError(`--only-stale: no deploy has been recorded for profile "${profile}" of project "${hubCtx.project}", so nothing can be compared against. Wire \`ccqa hub deploy record\` into the deploy job, or
|
|
6815
|
+
if (report.deployHead === null) throw new RunUsageError(`--only-hub-stale: no deploy has been recorded for profile "${profile}" of project "${hubCtx.project}", so nothing can be compared against. Wire \`ccqa hub deploy record\` into the deploy job, or select with --only-affected-by <ref> instead.`);
|
|
6826
6816
|
return {
|
|
6827
6817
|
...report,
|
|
6828
6818
|
deployHead: report.deployHead
|
|
@@ -6834,8 +6824,8 @@ async function fetchRerunReport(hubCtx, profile) {
|
|
|
6834
6824
|
* means the hub does not serve this route at all.
|
|
6835
6825
|
*/
|
|
6836
6826
|
function explainNotFound(hubCtx, err) {
|
|
6837
|
-
if (err.code === "no_perspectives") return `--only-stale: project "${hubCtx.project}" has no perspectives document on the hub, so no spec is registered to compare against a deploy. Run \`ccqa perspectives\` first.`;
|
|
6838
|
-
return `--only-stale: this hub does not serve re-run verdicts — it needs ccqa ${RERUN_MIN_HUB_VERSION} or newer. Upgrade the hub, or
|
|
6827
|
+
if (err.code === "no_perspectives") return `--only-hub-stale: project "${hubCtx.project}" has no perspectives document on the hub, so no spec is registered to compare against a deploy. Run \`ccqa perspectives\` first.`;
|
|
6828
|
+
return `--only-hub-stale: this hub does not serve re-run verdicts — it needs ccqa ${RERUN_MIN_HUB_VERSION} or newer. Upgrade the hub, or select with --only-affected-by <ref> instead.`;
|
|
6839
6829
|
}
|
|
6840
6830
|
/** States the summary line reports, worst-known-first. */
|
|
6841
6831
|
const SUMMARY_ORDER = [
|
|
@@ -6856,7 +6846,7 @@ const UNANSWERABLE = new Set([
|
|
|
6856
6846
|
*
|
|
6857
6847
|
* `needed` is always selected. `unknown` and `neverRun` are "the question
|
|
6858
6848
|
* cannot be answered", so they are excluded by default and opted into with
|
|
6859
|
-
* `--
|
|
6849
|
+
* `--only-hub-stale-with-unknown` — fail-open on request, never silently. `notNeeded` and
|
|
6860
6850
|
* `notEvaluated` are never selected.
|
|
6861
6851
|
*/
|
|
6862
6852
|
function selectSpecsNeedingRerun(specs, report, opts) {
|
|
@@ -7634,7 +7624,7 @@ function safeOriginPath(href) {
|
|
|
7634
7624
|
*
|
|
7635
7625
|
* Two kinds share one namespace:
|
|
7636
7626
|
* - "guidance": the record/live prompt bundle — `.user.md` (human-maintained)
|
|
7637
|
-
* and `.agent.md` (auto-rewritten by `ccqa run --
|
|
7627
|
+
* and `.agent.md` (auto-rewritten by `ccqa run --learn-hub-live-prompt`) —
|
|
7638
7628
|
* plus `triage.user`, the human-maintained guidance injected into the
|
|
7639
7629
|
* failure-analysis (triage) prompt.
|
|
7640
7630
|
* - "custom-prompt": `analysis-custom-prompt` — Claude-written calibration guidance
|
|
@@ -8103,7 +8093,7 @@ z.object({
|
|
|
8103
8093
|
lastRed: z.record(z.string(), SpecLedgerEntrySchema).default({})
|
|
8104
8094
|
});
|
|
8105
8095
|
/**
|
|
8106
|
-
* One spec's last drift audit, as recorded by `ccqa
|
|
8096
|
+
* One spec's last drift audit, as recorded by `ccqa audit --report-to-hub`. Unlike the
|
|
8107
8097
|
* spec ledger above, this carries no profile: drift asks whether a spec still
|
|
8108
8098
|
* describes the code, which has nothing to do with which environment is
|
|
8109
8099
|
* running it (ADR-0010 draws the same line for "needs re-run").
|
|
@@ -8313,7 +8303,7 @@ function isCcqaPath(path) {
|
|
|
8313
8303
|
}
|
|
8314
8304
|
/**
|
|
8315
8305
|
* A malformed reply costs the whole selection, so it is worth one more call
|
|
8316
|
-
* before giving up. `ccqa
|
|
8306
|
+
* before giving up. `ccqa audit` retries per spec for the same reason; this
|
|
8317
8307
|
* call carries every undecided spec at once, so the blast radius is larger,
|
|
8318
8308
|
* not smaller. Observed in practice: three runs over one commit produced a
|
|
8319
8309
|
* parse failure, a clean answer, and a different clean answer.
|
|
@@ -8510,7 +8500,7 @@ const sessionCaptureCommand = new Command("capture").description("Open a headed
|
|
|
8510
8500
|
error(err.message);
|
|
8511
8501
|
process.exit(2);
|
|
8512
8502
|
}
|
|
8513
|
-
header("session
|
|
8503
|
+
header("session capture", name);
|
|
8514
8504
|
meta("project", project);
|
|
8515
8505
|
meta("profile", opts.profile ?? "default");
|
|
8516
8506
|
blank();
|
|
@@ -8631,7 +8621,7 @@ const sessionPush = new Command("push").description("Upload a locally-saved brow
|
|
|
8631
8621
|
state = await loadStorageState(path);
|
|
8632
8622
|
} catch (err) {
|
|
8633
8623
|
error(`could not read session "${name}" at ${path}: ${err instanceof Error ? err.message : String(err)}`);
|
|
8634
|
-
hint(`create it first with: ccqa session
|
|
8624
|
+
hint(`create it first with: ccqa hub session capture ${name}${opts.profile ? ` --profile ${opts.profile}` : ""}`);
|
|
8635
8625
|
process.exit(2);
|
|
8636
8626
|
}
|
|
8637
8627
|
await connect(opts).putSession(project, profile, name, state);
|
|
@@ -8718,12 +8708,12 @@ const promptPush = new Command("push").description("Upload a locally-generated p
|
|
|
8718
8708
|
body = await readFile(path, "utf8");
|
|
8719
8709
|
} catch (err) {
|
|
8720
8710
|
error(`could not read prompt "${name}" at ${path}: ${err instanceof Error ? err.message : String(err)}`);
|
|
8721
|
-
hint("nothing to push; generate it first (e.g. ccqa run --
|
|
8711
|
+
hint("nothing to push; generate it first (e.g. ccqa run --learn-hub-live-prompt)");
|
|
8722
8712
|
process.exit(2);
|
|
8723
8713
|
}
|
|
8724
8714
|
if (body.trim().length === 0) {
|
|
8725
8715
|
error(`prompt "${name}" at ${path} is empty`);
|
|
8726
|
-
hint("nothing to push; generate it first (e.g. ccqa run --
|
|
8716
|
+
hint("nothing to push; generate it first (e.g. ccqa run --learn-hub-live-prompt)");
|
|
8727
8717
|
process.exit(2);
|
|
8728
8718
|
}
|
|
8729
8719
|
await connect(opts).putPrompt(project, name, body);
|
|
@@ -8749,7 +8739,7 @@ const promptRm = new Command("rm").description("Delete a prompt from the hub.").
|
|
|
8749
8739
|
info(`deleted prompt "${name}" from the hub`);
|
|
8750
8740
|
}));
|
|
8751
8741
|
const promptCommand = new Command("prompt").description("Manage prompt assets (per-flow user/agent guidance, triage user guidance, analysis custom prompt) stored on the hub (fetched automatically by `ccqa run` at run time).").addCommand(promptPush).addCommand(promptLs).addCommand(promptRm);
|
|
8752
|
-
const deployRecord = new Command("record").description("Tell the hub what a deploy shipped, so it can answer which specs need a re-run (`ccqa run --
|
|
8742
|
+
const deployRecord = new Command("record").description("Tell the hub what a deploy shipped, so it can answer which specs need a re-run (`ccqa run --only-hub-stale`). Run this from the deploy job, after the deploy succeeds. The changed paths are computed locally with a two-dot `git diff <previous> <sha>`; a job that has only curl and git can POST the same body directly (see docs/hub.md).").requiredOption("--profile <name>", "Environment this deploy landed in (e.g. 'stg'). Required: dev and stg sit at different commits, so the deploy log is per-profile.").requiredOption("--sha <sha>", "Commit that was deployed.").option("--previous <sha>", "Commit this deploy replaced. Defaults to the profile's current deploy-log head on the hub. With neither, there's nothing to diff against: changedPaths is unset and --select is skipped.").option("--ref <ref>", "Ref that was deployed (branch or tag). Recorded for display only.").option("--select", "Also decide which specs this deploy reaches (`ccqa select-specs`) and send the verdict with it. Without it the deploy is a hole in the range: specs behind it report 'unknown' rather than 'not needed'.").option("-m, --model <name>", "Model for --select. Claude alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option(...hubUrlOption).option(...hubTokenOption).option("--project <name>", "Project whose deploy log this entry joins. Defaults to the current directory's name.").option("--cwd <path>", "Directory the git diff and the default --project name are resolved against.").action(withHubErrors(async (opts) => {
|
|
8753
8743
|
const cwd = resolveCwd(opts.cwd);
|
|
8754
8744
|
const project = resolveProject(opts);
|
|
8755
8745
|
const hub = connect(opts);
|
|
@@ -8828,7 +8818,7 @@ function describeSelection(selection, diffAvailable) {
|
|
|
8828
8818
|
const values = Object.values(selection);
|
|
8829
8819
|
return `${values.filter((s) => s.verdict === "needed").length} needed / ${values.filter((s) => s.verdict === "unknown").length} unknown / ${values.length} specs`;
|
|
8830
8820
|
}
|
|
8831
|
-
const deployCommand = new Command("deploy").description("Report deploys to the hub, the input behind `ccqa run --
|
|
8821
|
+
const deployCommand = new Command("deploy").description("Report deploys to the hub, the input behind `ccqa run --only-hub-stale`.").addCommand(deployRecord);
|
|
8832
8822
|
const pushCommand = new Command("push").description("Upload the report directory of a finished `ccqa run --report` to the hub as a run. Run this after `ccqa run` (use `if: always()` in CI so failing runs are pushed too).").option("--report-dir <dir>", `Report directory to push. Default: ${DEFAULT_REPORT_DIR}/`).option("--project <name>", "Logical project name for the run. Defaults to the current directory's name.").option("--branch <name>", "Branch label. Defaults to $GITHUB_HEAD_REF / $GITHUB_REF_NAME / current git branch.").option("--profile <name>", "Profile (environment) the run executed against. Recorded for display; runs are not scoped by profile.").option(...hubUrlOption).option(...hubTokenOption).option("--cwd <path>", "Directory the report dir is resolved against (defaults to the current directory).").action(withHubErrors(async (opts) => {
|
|
8833
8823
|
const cwd = resolveCwd(opts.cwd);
|
|
8834
8824
|
const reportDir = join(cwd, opts.reportDir ?? "ccqa-report");
|
|
@@ -8971,7 +8961,7 @@ function generateLiveSessionName() {
|
|
|
8971
8961
|
* Project-specific guidance ("the admin tenant is foo.example", "session
|
|
8972
8962
|
* times out at X minutes", …) is appended from
|
|
8973
8963
|
* `.ccqa/prompts/live.user.md` (human-maintained) and
|
|
8974
|
-
* `.ccqa/prompts/live.agent.md` (updated by `ccqa run --
|
|
8964
|
+
* `.ccqa/prompts/live.agent.md` (updated by `ccqa run --learn-hub-live-prompt`)
|
|
8975
8965
|
* by the caller, so ccqa stays clean of downstream-product context.
|
|
8976
8966
|
*
|
|
8977
8967
|
* Constraint posture: `ccqa record` (trace) enforces a strict selector
|
|
@@ -9726,7 +9716,7 @@ const verifiedSessions = /* @__PURE__ */ new Set();
|
|
|
9726
9716
|
* each named session from the hub (`.ccqa/sessions/*.json` is no longer
|
|
9727
9717
|
* read here). Every name must load as a valid agent-browser state (the spec
|
|
9728
9718
|
* assumes it starts signed-in); a missing/malformed session fails with a
|
|
9729
|
-
* `ccqa session
|
|
9719
|
+
* `ccqa hub session capture` hint instead of running unauthenticated.
|
|
9730
9720
|
*
|
|
9731
9721
|
* If a session carries an embedded verify URL (bootstrap saved it), the
|
|
9732
9722
|
* restore is health-checked before the run starts, so an expired/unusable
|
|
@@ -9769,7 +9759,7 @@ async function resolveSessionState(names, hubCtx, profile, verify = verifySessio
|
|
|
9769
9759
|
if (!check.restored) return {
|
|
9770
9760
|
ok: false,
|
|
9771
9761
|
error: `session '${name}' did not restore to a signed-in page — ${check.reason}`,
|
|
9772
|
-
hint: `re-bootstrap it: ccqa session
|
|
9762
|
+
hint: `re-bootstrap it: ccqa hub session capture ${name}${profileFlag}`
|
|
9773
9763
|
};
|
|
9774
9764
|
verifiedSessions.add(memoKey);
|
|
9775
9765
|
}
|
|
@@ -9779,7 +9769,7 @@ async function resolveSessionState(names, hubCtx, profile, verify = verifySessio
|
|
|
9779
9769
|
if (broken.length > 0) return {
|
|
9780
9770
|
ok: false,
|
|
9781
9771
|
error: `session not usable on the hub: ${broken.join(", ")}`,
|
|
9782
|
-
hint: `create it with: ${broken.map((name) => `ccqa session
|
|
9772
|
+
hint: `create it with: ${broken.map((name) => `ccqa hub session capture ${name}${profileFlag}`).join(" · ")}`
|
|
9783
9773
|
};
|
|
9784
9774
|
const statePath = await writeMergedTempState(mergeStorageStates(loaded));
|
|
9785
9775
|
return {
|
|
@@ -11652,7 +11642,7 @@ async function groupSpecsByTarget(specs, config, cwd, resolve = resolveTarget) {
|
|
|
11652
11642
|
* then each external target group through its runner. Every row is upserted
|
|
11653
11643
|
* into the incremental report the moment it exists — the runner reports each
|
|
11654
11644
|
* spec through `onSpecComplete` as it finishes — so an interrupt keeps what
|
|
11655
|
-
* already ran and `--
|
|
11645
|
+
* already ran and `--report-to-hub` streams spec by spec. Rows are also
|
|
11656
11646
|
* returned for the tail phase (failure analysis) and the final batch write. A
|
|
11657
11647
|
* crashing runner marks its own specs failed instead of aborting the run.
|
|
11658
11648
|
*/
|
|
@@ -11788,7 +11778,7 @@ function createIncrementalReport(reportDir, envelope, sink) {
|
|
|
11788
11778
|
//#endregion
|
|
11789
11779
|
//#region src/prompts/agent-update.ts
|
|
11790
11780
|
/**
|
|
11791
|
-
* Build the prompts used by `--
|
|
11781
|
+
* Build the prompts used by the `--learn-*-prompt` flags to refresh
|
|
11792
11782
|
* `.ccqa/prompts/<kind>.agent.md` after a run:
|
|
11793
11783
|
* - `ccqa run` (live) → `live.agent`
|
|
11794
11784
|
* - `ccqa record` (trace) → `record.agent`
|
|
@@ -12044,10 +12034,7 @@ async function updateAgentPrompt(args) {
|
|
|
12044
12034
|
warn(`${flag} skipped (${auth.reason})`);
|
|
12045
12035
|
return;
|
|
12046
12036
|
}
|
|
12047
|
-
if (!hubContext) {
|
|
12048
|
-
warn(`${flag} skipped (hub connection required; pass --hub-url/--hub-token or set CCQA_HUB_URL/CCQA_HUB_TOKEN)`);
|
|
12049
|
-
return;
|
|
12050
|
-
}
|
|
12037
|
+
if (!hubContext) throw new Error(`${flag} requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)`);
|
|
12051
12038
|
const { hub, project } = hubContext;
|
|
12052
12039
|
const promptName = `${kind}.agent`;
|
|
12053
12040
|
try {
|
|
@@ -12169,6 +12156,19 @@ async function resolveVitestConfig(cwd) {
|
|
|
12169
12156
|
function resolveReportDir(reportDir, cwd) {
|
|
12170
12157
|
return resolve(cwd, reportDir ?? "ccqa-report");
|
|
12171
12158
|
}
|
|
12159
|
+
/**
|
|
12160
|
+
* Turn a hub transport failure into a usage error, as a `.catch` so the
|
|
12161
|
+
* tuple types of the `Promise.all` it guards survive.
|
|
12162
|
+
*
|
|
12163
|
+
* Without it the raw `fetch failed` escapes as an unhandled rejection: a stack
|
|
12164
|
+
* trace and exit 1, where the user needs "the hub is unreachable" and exit 2.
|
|
12165
|
+
* Errors the callers already shaped pass through — they say more than this
|
|
12166
|
+
* wrapper could.
|
|
12167
|
+
*/
|
|
12168
|
+
function asHubReadError(err) {
|
|
12169
|
+
if (err instanceof RunUsageError) throw err;
|
|
12170
|
+
throw new RunUsageError(`could not read from the hub: ${errMessage(err)}`);
|
|
12171
|
+
}
|
|
12172
12172
|
/** De-dupe by `featureName/specName`, keeping first-seen order. */
|
|
12173
12173
|
function dedupeSpecs(specs) {
|
|
12174
12174
|
const seen = /* @__PURE__ */ new Set();
|
|
@@ -12189,10 +12189,10 @@ function dedupeSpecs(specs) {
|
|
|
12189
12189
|
* maps it to `process.exit(2)`).
|
|
12190
12190
|
*/
|
|
12191
12191
|
async function executeRun(targets, opts) {
|
|
12192
|
-
const filtering = Boolean(opts.onlyAffectedBy || opts.
|
|
12192
|
+
const filtering = Boolean(opts.onlyAffectedBy || opts.onlyHubStale || opts.onlyHubAuditedClean);
|
|
12193
12193
|
if (filtering && targets.length > 0) throw new RunUsageError("a --only-* filter and an explicit spec target cannot be combined");
|
|
12194
|
-
const rerunProfile = opts.
|
|
12195
|
-
if (opts.
|
|
12194
|
+
const rerunProfile = opts.onlyHubStale === true ? requireRerunProfile(opts.hubProfile) : null;
|
|
12195
|
+
if (opts.onlyHubStaleWithUnknown && rerunProfile === null) warn("--only-hub-stale-with-unknown is ignored: it only applies to --only-hub-stale");
|
|
12196
12196
|
const forExecution = opts.dryRun !== true;
|
|
12197
12197
|
const cwd = opts.cwd ?? process.cwd();
|
|
12198
12198
|
const wantsLastGreen = opts.onFailExplain === true && opts.onFailExplainBase === void 0;
|
|
@@ -12217,8 +12217,8 @@ async function executeRun(targets, opts) {
|
|
|
12217
12217
|
meta("analysis-base", `${fixedBase.ref} (${fixedBase.sha.slice(0, 12)}, ${fixedBase.source})`);
|
|
12218
12218
|
}
|
|
12219
12219
|
if (forExecution) try {
|
|
12220
|
-
if (opts.
|
|
12221
|
-
profile: opts.
|
|
12220
|
+
if (opts.hubProfile !== void 0) await resolveProfileEnv({
|
|
12221
|
+
profile: opts.hubProfile,
|
|
12222
12222
|
project: resolveProjectOrThrow(opts.project, cwd),
|
|
12223
12223
|
cwd,
|
|
12224
12224
|
hubUrl: opts.hubUrl,
|
|
@@ -12234,7 +12234,7 @@ async function executeRun(targets, opts) {
|
|
|
12234
12234
|
if (err instanceof RunUsageError) throw err;
|
|
12235
12235
|
if (err instanceof ProjectNameError) throw new RunUsageError(err.message);
|
|
12236
12236
|
if (err instanceof HubConnectionError || err instanceof HubApiError) throw new RunUsageError(err.message);
|
|
12237
|
-
throw new RunUsageError(`failed to load profile "${opts.
|
|
12237
|
+
throw new RunUsageError(`failed to load profile "${opts.hubProfile}": ${errMessage(err)}`);
|
|
12238
12238
|
}
|
|
12239
12239
|
let hubCtx = null;
|
|
12240
12240
|
try {
|
|
@@ -12250,16 +12250,18 @@ async function executeRun(targets, opts) {
|
|
|
12250
12250
|
}
|
|
12251
12251
|
if (wantsLastGreen && hubCtx == null) throw new RunUsageError("--on-fail-explain needs a hub connection for the per-spec last-green baselines (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN), or an explicit --on-fail-explain-base <ref>");
|
|
12252
12252
|
const ledgerHub = wantsLastGreen ? hubCtx : null;
|
|
12253
|
-
if (rerunProfile !== null && hubCtx == null) throw new RunUsageError("--only-stale requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12254
|
-
if (opts.
|
|
12253
|
+
if (rerunProfile !== null && hubCtx == null) throw new RunUsageError("--only-hub-stale requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12254
|
+
if (opts.onlyHubAuditedClean && hubCtx == null) throw new RunUsageError("--only-hub-audited-clean requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12255
|
+
if (opts.reportToHub && hubCtx == null) throw new RunUsageError("--report-to-hub requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12256
|
+
if (opts.learnHubLivePrompt && hubCtx == null) throw new RunUsageError("--learn-hub-live-prompt requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
12255
12257
|
const [customPrompt, triageUserPrompt, ledgerEntries, rerunReport, fetchedDeployHead, auditedLedger] = await Promise.all([
|
|
12256
12258
|
forExecution ? fetchCustomPrompt(hubCtx) : null,
|
|
12257
12259
|
forExecution ? fetchTriageUserPrompt(hubCtx) : null,
|
|
12258
|
-
forExecution && ledgerHub ? fetchLastGreenLedger(ledgerHub, opts.
|
|
12260
|
+
forExecution && ledgerHub ? fetchLastGreenLedger(ledgerHub, opts.hubProfile, cwd) : null,
|
|
12259
12261
|
rerunProfile !== null && hubCtx ? fetchRerunReport(hubCtx, rerunProfile) : null,
|
|
12260
|
-
forExecution && hubCtx && opts.
|
|
12261
|
-
opts.
|
|
12262
|
-
]);
|
|
12262
|
+
forExecution && hubCtx && opts.hubProfile && rerunProfile === null ? tryDeployHeadSha(hubCtx, opts.hubProfile) : null,
|
|
12263
|
+
opts.onlyHubAuditedClean && hubCtx ? fetchAuditedLedger(hubCtx) : null
|
|
12264
|
+
]).catch(asHubReadError);
|
|
12263
12265
|
const deployedSha = rerunReport?.deployHead.sha ?? fetchedDeployHead;
|
|
12264
12266
|
if (ledgerEntries) diffProvider = createDiffProvider({
|
|
12265
12267
|
resolveBase: createLastGreenResolver(ledgerEntries, cwd),
|
|
@@ -12286,7 +12288,7 @@ async function executeRun(targets, opts) {
|
|
|
12286
12288
|
const before = specs.length;
|
|
12287
12289
|
let unanswerable = 0;
|
|
12288
12290
|
if (rerunReport) {
|
|
12289
|
-
const selection = selectSpecsNeedingRerun(specs, rerunReport, { includeUnknown: opts.
|
|
12291
|
+
const selection = selectSpecsNeedingRerun(specs, rerunReport, { includeUnknown: opts.onlyHubStaleWithUnknown === true });
|
|
12290
12292
|
specs = selection.selected;
|
|
12291
12293
|
unanswerable = selection.excludedUnanswerable;
|
|
12292
12294
|
meta("stale-base", `deploy ${rerunReport.deployHead.sha.slice(0, 12)} (profile ${rerunReport.profile})`);
|
|
@@ -12303,7 +12305,7 @@ async function executeRun(targets, opts) {
|
|
|
12303
12305
|
...opts.model ? { model: opts.model } : {}
|
|
12304
12306
|
})).specs;
|
|
12305
12307
|
meta("selected", `${specs.length} of ${before} spec${before === 1 ? "" : "s"}`);
|
|
12306
|
-
if (specs.length === 0 && unanswerable > 0) hint(`${unanswerable} spec(s) were excluded because the hub could not tell whether they need a re-run; pass --only-stale-with-unknown to run them anyway`);
|
|
12308
|
+
if (specs.length === 0 && unanswerable > 0) hint(`${unanswerable} spec(s) were excluded because the hub could not tell whether they need a re-run; pass --only-hub-stale-with-unknown to run them anyway`);
|
|
12307
12309
|
}
|
|
12308
12310
|
if (specs.length === 0) {
|
|
12309
12311
|
warn("no specs to run");
|
|
@@ -12328,7 +12330,7 @@ async function executeRun(targets, opts) {
|
|
|
12328
12330
|
const why = "it only applies to agent-browser 'mode: live' specs, and this run has none";
|
|
12329
12331
|
if (typeof opts.liveStepRetry === "number" && opts.liveStepRetry > 0) warn(`--live-step-retry is ignored: ${why}`);
|
|
12330
12332
|
if (opts.liveArtifactsDir) warn(`--live-artifacts-dir is ignored: ${why}`);
|
|
12331
|
-
if (opts.
|
|
12333
|
+
if (opts.learnHubLivePrompt) warn(`--learn-live-prompt is ignored: ${why}`);
|
|
12332
12334
|
} else if (opts.liveArtifactsDir && liveSpecs.length > 1) warn("--out is ignored when running multiple live specs");
|
|
12333
12335
|
if (detSpecs.length === 0 && opts.replaySkipEvidence === true) warn("--no-evidence is ignored: it only applies to agent-browser 'mode: deterministic' specs, and this run has none");
|
|
12334
12336
|
blank();
|
|
@@ -12343,7 +12345,6 @@ async function executeRun(targets, opts) {
|
|
|
12343
12345
|
};
|
|
12344
12346
|
}
|
|
12345
12347
|
const det = await runDeterministicSpecs(detSpecs, opts, cwd, reportDir);
|
|
12346
|
-
if (opts.reportToHub && hubCtx == null) warn("--push-report requires --hub-url/--hub-token (or CCQA_HUB_URL/CCQA_HUB_TOKEN); skipping push");
|
|
12347
12348
|
let hubRunId = null;
|
|
12348
12349
|
let hubSink;
|
|
12349
12350
|
if (hubCtx != null && opts.reportToHub) try {
|
|
@@ -12353,7 +12354,7 @@ async function executeRun(targets, opts) {
|
|
|
12353
12354
|
const opened = await hubCtx.hub.openRun({
|
|
12354
12355
|
project: hubCtx.project,
|
|
12355
12356
|
...branch ? { branch } : {},
|
|
12356
|
-
...opts.
|
|
12357
|
+
...opts.hubProfile ? { profile: opts.hubProfile } : {},
|
|
12357
12358
|
...git.head ? { gitHead: git.head } : {},
|
|
12358
12359
|
...deployedSha ? { deployedSha } : {},
|
|
12359
12360
|
...ciRunId ? { ciRunId } : {},
|
|
@@ -12375,7 +12376,7 @@ async function executeRun(targets, opts) {
|
|
|
12375
12376
|
}
|
|
12376
12377
|
} };
|
|
12377
12378
|
} catch (err) {
|
|
12378
|
-
|
|
12379
|
+
throw new RunUsageError(`--report-to-hub: could not open a run on the hub (${errMessage(err)})`);
|
|
12379
12380
|
}
|
|
12380
12381
|
const incrementalReport = createIncrementalReport(reportDir, buildReportEnvelope({
|
|
12381
12382
|
git,
|
|
@@ -12414,7 +12415,7 @@ async function executeRun(targets, opts) {
|
|
|
12414
12415
|
reportDir,
|
|
12415
12416
|
...typeof opts.liveStepRetry === "number" ? { retry: opts.liveStepRetry } : {},
|
|
12416
12417
|
concurrency: opts.concurrency ?? 1,
|
|
12417
|
-
...opts.
|
|
12418
|
+
...opts.hubProfile ? { profile: opts.hubProfile } : {},
|
|
12418
12419
|
diffProvider,
|
|
12419
12420
|
hubContext: hubCtx,
|
|
12420
12421
|
customPrompt,
|
|
@@ -12476,7 +12477,7 @@ async function executeRun(targets, opts) {
|
|
|
12476
12477
|
}
|
|
12477
12478
|
}
|
|
12478
12479
|
}
|
|
12479
|
-
if (opts.
|
|
12480
|
+
if (opts.learnHubLivePrompt && liveSpecs.length > 0) {
|
|
12480
12481
|
blank();
|
|
12481
12482
|
await updateAgentPrompt({
|
|
12482
12483
|
kind: "live",
|
|
@@ -12789,7 +12790,7 @@ const PATCH_FILES_RAW_BUDGET = 20 * 1024 * 1024;
|
|
|
12789
12790
|
/**
|
|
12790
12791
|
* Add one row's file assets to `acc` as `{ reportDir-relative posix path →
|
|
12791
12792
|
* base64 }`. Every kind of screenshot a row can carry has to be collected here,
|
|
12792
|
-
* or `--
|
|
12793
|
+
* or `--report-to-hub` — the way CI publishes — silently ships a report whose
|
|
12793
12794
|
* images 404 on the hub: a live row's per-step PNGs
|
|
12794
12795
|
* (`liveRun.steps[].beforePng/afterPng`), a script-driven row's step evidence
|
|
12795
12796
|
* (`evidence[].pngPath` / `beforePngPath`, written by agent-browser replays and
|
|
@@ -13018,14 +13019,14 @@ function installTeardownSignalHandlers(teardown) {
|
|
|
13018
13019
|
}
|
|
13019
13020
|
//#endregion
|
|
13020
13021
|
//#region src/cli/run.ts
|
|
13021
|
-
const runCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs, on any target. Agent-browser specs replay the recorded test.spec.ts under vitest (default), or, with spec.yaml `mode: live`, have Claude drive agent-browser live per step. External-target specs (playwright, runn) run through the target's configured `runCommand`. A structured report (report.json + evidence) is always written; use --report-to-hub to also stream it to a hub.").optionsGroup("Which specs to run:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Cannot be combined with an explicit spec id.").option("--only-stale", "Only specs
|
|
13022
|
+
const runCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs, on any target. Agent-browser specs replay the recorded test.spec.ts under vitest (default), or, with spec.yaml `mode: live`, have Claude drive agent-browser live per step. External-target specs (playwright, runn) run through the target's configured `runCommand`. A structured report (report.json + evidence) is always written; use --report-to-hub to also stream it to a hub.").optionsGroup("Which specs to run:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Cannot be combined with an explicit spec id.").option("--only-hub-stale", "Only specs the hub says are no longer covered by their last result — each spec's own last run compared against the hub's deploy log. No git diff involved. Requires a hub connection and --hub-profile.").option("--only-hub-stale-with-unknown", "With --only-hub-stale: also take specs whose re-run need the hub cannot answer ('unknown') and specs that never ran ('neverRun'). Off by default: an unanswerable question is reported, not guessed.").option("--only-hub-audited-clean", "Only specs the hub's drift ledger records as audited with no drift. A spec that has never been audited is not taken: this flag spends a run where a cheap audit already cleared the spec, and \"never looked\" is not that. Requires a hub connection.").option("--dry-run", "Print the specs this invocation would run, then exit 0 without executing anything and without writing a report. Works with every selection flag.").optionsGroup("How to run them:").option("--concurrency <n>", "Run up to N specs in parallel within each phase (deterministic / external-target / live), never across phases. Default 1 (sequential). Live specs each get an isolated agent-browser session; high values spawn many headed Chrome instances.", parseConcurrency$1, 1).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--live-step-retry <n>", "(live only) Retry each failed step up to N more times before recording failure. This retries a step, not the whole spec — see --on-fail-explain-rerun for that.", (raw) => {
|
|
13022
13023
|
const n = Number(raw);
|
|
13023
13024
|
if (!Number.isFinite(n) || n < 0 || Math.floor(n) !== n) throw new Error(`--live-step-retry must be a non-negative integer, got "${raw}"`);
|
|
13024
13025
|
return n;
|
|
13025
13026
|
}, 0).option("--live-artifacts-dir <dir>", "(live only) Override the per-spec artifact directory. Default: <specDir>/runs/<runId>. Ignored when running multiple specs.").option("--replay-skip-evidence", `(deterministic replay only) Skip step-boundary evidence capture (PNG + meta JSON written to ${DEFAULT_REPORT_DIR}/${EVIDENCE_SUBDIR}/ by default).`).optionsGroup("What to do about failures:").option("--on-fail-explain", "Classify each failure against the source diff since the commit where that spec last passed (per-spec baselines from the hub). Off by default — no Claude calls without it.").option("--on-fail-explain-base <ref>", "With --on-fail-explain: diff against <ref> instead of each spec's last green. Use when there is no hub to hold the baselines.").optionsGroup("What to do with the results:").option("--report-dir <dir>", `Directory for the structured run results (report.json + evidence PNGs), which are always written. Default: ${DEFAULT_REPORT_DIR}/.`).option("--report-format <fmt>", "Additional output format alongside HTML: 'text' (default), 'json' (writes report.json), 'github' (GitHub Actions annotations on stdout).", (raw) => {
|
|
13026
13027
|
if (REPORT_FORMATS.includes(raw)) return raw;
|
|
13027
13028
|
throw new Error(`--report-format must be one of ${REPORT_FORMATS.join(" | ")}`);
|
|
13028
|
-
}, "text").option("--report-to-hub", "Incrementally push the run report to the hub as the run progresses (open → patch per spec → finalize). Requires --hub-url/--hub-token (or CCQA_HUB_URL/CCQA_HUB_TOKEN). Without it, hub credentials are used only to fetch variables/sessions/prompts, not to push.").optionsGroup("Learning:").option("--learn-live-prompt", "(live only) After the run finishes, ask Claude to refresh the \"live.agent\" prompt on the hub from a summary of the run. Requires a hub connection.").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--project <name>", "Project name for the hub. Defaults to the current directory's name.")))).action(async (targets, opts) => {
|
|
13029
|
+
}, "text").option("--report-to-hub", "Incrementally push the run report to the hub as the run progresses (open → patch per spec → finalize). Requires --hub-url/--hub-token (or CCQA_HUB_URL/CCQA_HUB_TOKEN). Without it, hub credentials are used only to fetch variables/sessions/prompts, not to push.").optionsGroup("Learning:").option("--learn-hub-live-prompt", "(live only) After the run finishes, ask Claude to refresh the \"live.agent\" prompt on the hub from a summary of the run. Requires a hub connection.").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--project <name>", "Project name for the hub. Defaults to the current directory's name.")))).action(async (targets, opts) => {
|
|
13029
13030
|
await runCliAction(targets, opts);
|
|
13030
13031
|
});
|
|
13031
13032
|
/** Parse --concurrency: a positive integer. Rejects 0, negatives, non-integers. */
|
|
@@ -13043,8 +13044,8 @@ function headerTarget(targets, opts) {
|
|
|
13043
13044
|
if (targets.length > 1) return `${targets.length} targets`;
|
|
13044
13045
|
const filters = [
|
|
13045
13046
|
opts.onlyAffectedBy ? "affected" : null,
|
|
13046
|
-
opts.
|
|
13047
|
-
opts.
|
|
13047
|
+
opts.onlyHubStale ? "stale" : null,
|
|
13048
|
+
opts.onlyHubAuditedClean ? "audited clean" : null
|
|
13048
13049
|
].filter((s) => s !== null);
|
|
13049
13050
|
return filters.length === 0 ? "(all specs)" : `(${filters.join(" + ")})`;
|
|
13050
13051
|
}
|
|
@@ -15142,21 +15143,21 @@ async function runGenerateLocked(featureName, specName, opts, cwd) {
|
|
|
15142
15143
|
hint(`run 'ccqa run ${featureName}/${specName}' to execute the test`);
|
|
15143
15144
|
}
|
|
15144
15145
|
/**
|
|
15145
|
-
* `ccqa generate --
|
|
15146
|
+
* `ccqa generate --learn-hub-codegen-prompt`: refresh the target's learned
|
|
15146
15147
|
* `<target>.agent` playbook from this generation. Only targets that declare a
|
|
15147
15148
|
* `guidanceKind` (the LLM-generating ones: playwright, runn) have such a
|
|
15148
15149
|
* prompt — agent-browser's codegen is mechanical, so point at `ccqa record
|
|
15149
|
-
* --
|
|
15150
|
+
* --learn-hub-trace-prompt` for its tracer instead.
|
|
15150
15151
|
*/
|
|
15151
15152
|
async function runGenerateAgentPromptUpdate(target, featureName, specName, result, opts, cwd) {
|
|
15152
15153
|
if (target.guidanceKind === void 0) {
|
|
15153
|
-
warn(`--
|
|
15154
|
+
warn(`--learn-hub-codegen-prompt has no effect on the "${target.id}" target — it has no learned generation prompt (only LLM-generating targets like playwright/runn do)`);
|
|
15154
15155
|
return;
|
|
15155
15156
|
}
|
|
15156
15157
|
blank();
|
|
15157
15158
|
await updateAgentPrompt({
|
|
15158
15159
|
kind: target.guidanceKind,
|
|
15159
|
-
flag: "--learn-codegen-prompt",
|
|
15160
|
+
flag: "--learn-hub-codegen-prompt",
|
|
15160
15161
|
runSummary: buildGenerateRunSummary(target.id, featureName, specName, result, cwd),
|
|
15161
15162
|
hubContext: opts.hubContext ?? null,
|
|
15162
15163
|
...opts.model ? { model: opts.model } : {},
|
|
@@ -15182,7 +15183,7 @@ async function confirmOverwrite(path) {
|
|
|
15182
15183
|
rl.close();
|
|
15183
15184
|
}
|
|
15184
15185
|
}
|
|
15185
|
-
const generateCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("generate").argument("<feature/spec>", "Spec id in '<feature>/<spec>' form (resolves to .ccqa/features/<feature>/test-cases/<spec>/)").description("Generate test code from a spec via its target plugin. Recording-backed targets compile the existing ir.json (run `ccqa record` first); spec-input targets generate directly from the spec.").optionsGroup("How to generate:").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--target <id>", "Generate through this target instead of the spec's own — e.g. emit a Playwright spec from an agent-browser recording. The spec's `target:` stays the default for `ccqa run`.").option("--auto-fix <mode>", "Auto-fix behaviour during script generation: 'interactive' (default, prompt y/N; declines on non-TTY), 'auto' (apply without prompt, for CI), 'skip' (agent-browser: apply only high-confidence fixes; external targets like playwright/runn: no fix pass at all).", parseAutoFixFlag, "interactive").option("--auto-fix-max-retries <n>", "Maximum number of auto-fix retries", "3").option("--no-session-pin", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").optionsGroup("What to do with the result:").option("--overwrite", "Replace previously generated test code without warning").optionsGroup("Learning:").option("--learn-codegen-prompt", "After generation, ask Claude to refresh the target's \"<target>.agent\" learning prompt on the hub from a summary of the run. LLM-generating targets (playwright, runn) only; requires a hub connection.").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--project <name>", "Project name for the hub. Defaults to the current directory's name.")))).action(withUsageErrors(async (specPath, opts) => {
|
|
15186
|
+
const generateCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("generate").argument("<feature/spec>", "Spec id in '<feature>/<spec>' form (resolves to .ccqa/features/<feature>/test-cases/<spec>/)").description("Generate test code from a spec via its target plugin. Recording-backed targets compile the existing ir.json (run `ccqa record` first); spec-input targets generate directly from the spec.").optionsGroup("How to generate:").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--target <id>", "Generate through this target instead of the spec's own — e.g. emit a Playwright spec from an agent-browser recording. The spec's `target:` stays the default for `ccqa run`.").option("--auto-fix <mode>", "Auto-fix behaviour during script generation: 'interactive' (default, prompt y/N; declines on non-TTY), 'auto' (apply without prompt, for CI), 'skip' (agent-browser: apply only high-confidence fixes; external targets like playwright/runn: no fix pass at all).", parseAutoFixFlag, "interactive").option("--auto-fix-max-retries <n>", "Maximum number of auto-fix retries", "3").option("--no-session-pin", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").optionsGroup("What to do with the result:").option("--overwrite", "Replace previously generated test code without warning").optionsGroup("Learning:").option("--learn-hub-codegen-prompt", "After generation, ask Claude to refresh the target's \"<target>.agent\" learning prompt on the hub from a summary of the run. LLM-generating targets (playwright, runn) only; requires a hub connection.").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--project <name>", "Project name for the hub. Defaults to the current directory's name.")))).action(withUsageErrors(async (specPath, opts) => {
|
|
15186
15187
|
const { featureName, specName } = parseSpecPath(specPath);
|
|
15187
15188
|
const language = opts.language ?? "auto";
|
|
15188
15189
|
const cwd = resolveCwd(opts.cwd);
|
|
@@ -15191,9 +15192,9 @@ const generateCommand = addHubOptions(addProfileOption(addLanguageOption(new Com
|
|
|
15191
15192
|
hubToken: opts.hubToken,
|
|
15192
15193
|
hubHeader: opts.hubHeader
|
|
15193
15194
|
});
|
|
15194
|
-
const project = opts.
|
|
15195
|
-
if (opts.
|
|
15196
|
-
profile: opts.
|
|
15195
|
+
const project = opts.hubProfile !== void 0 || hubClient !== null ? resolveProject(opts) : void 0;
|
|
15196
|
+
if (opts.hubProfile !== void 0) await applyProfileFromOption({
|
|
15197
|
+
profile: opts.hubProfile,
|
|
15197
15198
|
project,
|
|
15198
15199
|
cwd,
|
|
15199
15200
|
hubUrl: opts.hubUrl,
|
|
@@ -15205,6 +15206,10 @@ const generateCommand = addHubOptions(addProfileOption(addLanguageOption(new Com
|
|
|
15205
15206
|
project: "",
|
|
15206
15207
|
cwd
|
|
15207
15208
|
});
|
|
15209
|
+
if (opts.learnHubCodegenPrompt && hubClient === null) {
|
|
15210
|
+
error("--learn-hub-codegen-prompt requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
15211
|
+
process.exit(2);
|
|
15212
|
+
}
|
|
15208
15213
|
const hubContext = hubClient && project ? {
|
|
15209
15214
|
hub: hubClient,
|
|
15210
15215
|
project
|
|
@@ -15220,7 +15225,7 @@ const generateCommand = addHubOptions(addProfileOption(addLanguageOption(new Com
|
|
|
15220
15225
|
targetOverride: opts.target,
|
|
15221
15226
|
cwd,
|
|
15222
15227
|
hubContext,
|
|
15223
|
-
updateAgentPrompt: opts.
|
|
15228
|
+
updateAgentPrompt: opts.learnHubCodegenPrompt ?? false
|
|
15224
15229
|
});
|
|
15225
15230
|
} catch (e) {
|
|
15226
15231
|
if (e instanceof SpecLockedError) {
|
|
@@ -15259,7 +15264,7 @@ const VALIDATION_MODES = ["lenient", "strict"];
|
|
|
15259
15264
|
const recordCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("record").argument("<feature/spec>", "Spec id in '<feature>/<spec>' form (resolves to .ccqa/features/<feature>/test-cases/<spec>/)").description("Record a test from a spec: run agent-browser to collect actions (trace), then compile them into runnable code via the spec's target (generate) — a vitest test.spec.ts for agent-browser, a @playwright/test spec for the playwright target. Recording-backed targets only; spec-input targets like runn have no trace step (use `ccqa generate`), and agent-browser live specs need no recording.").optionsGroup("How to record:").option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--trace-validation <mode>", "What to do with actions that fail post-trace validation: 'lenient' (default) tags them; 'strict' drops them.", (raw) => {
|
|
15260
15265
|
if (VALIDATION_MODES.includes(raw)) return raw;
|
|
15261
15266
|
throw new Error(`--trace-validation must be one of ${VALIDATION_MODES.join(" | ")}`);
|
|
15262
|
-
}, "lenient").option("--auto-fix <mode>", "Auto-fix behaviour during script generation: 'interactive' (default, prompt y/N; declines on non-TTY), 'auto' (apply without prompt, for CI), 'skip' (agent-browser: apply only high-confidence fixes; external targets like playwright/runn: no fix pass at all).", parseAutoFixFlag, "interactive").option("--auto-fix-max-retries <n>", "Maximum number of auto-fix retries", "3").option("--trace-only", "Stop after the trace step; do not generate test code").option("--no-session-pin", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").optionsGroup("What to do with the result:").option("--overwrite", "Replace an existing test.spec.ts without warning").optionsGroup("Learning:").option("--learn-trace-prompt", "After the trace finishes, ask Claude to refresh the \"record.agent\" prompt on the hub from a summary of the run. Requires a hub connection.").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--project <name>", "Project name for the hub. Defaults to the current directory's name.")))).action(withUsageErrors(async (specPath, opts) => {
|
|
15267
|
+
}, "lenient").option("--auto-fix <mode>", "Auto-fix behaviour during script generation: 'interactive' (default, prompt y/N; declines on non-TTY), 'auto' (apply without prompt, for CI), 'skip' (agent-browser: apply only high-confidence fixes; external targets like playwright/runn: no fix pass at all).", parseAutoFixFlag, "interactive").option("--auto-fix-max-retries <n>", "Maximum number of auto-fix retries", "3").option("--trace-only", "Stop after the trace step; do not generate test code").option("--no-session-pin", "Don't pin AGENT_BROWSER_SESSION / capture page snapshots after a failure (debug toggle)").optionsGroup("What to do with the result:").option("--overwrite", "Replace an existing test.spec.ts without warning").optionsGroup("Learning:").option("--learn-hub-trace-prompt", "After the trace finishes, ask Claude to refresh the \"record.agent\" prompt on the hub from a summary of the run. Requires a hub connection.").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory containing the .ccqa/ tree (monorepo support). Defaults to the current directory.").option("--project <name>", "Project name for the hub. Defaults to the current directory's name.")))).action(withUsageErrors(async (specPath, opts) => {
|
|
15263
15268
|
await withCostTally(async () => {
|
|
15264
15269
|
try {
|
|
15265
15270
|
await runRecord(specPath, opts);
|
|
@@ -15279,9 +15284,9 @@ async function runRecord(specPath, opts) {
|
|
|
15279
15284
|
error(`target "${target.id}" does not use a browser recording — run 'ccqa generate ${featureName}/${specName}' instead`);
|
|
15280
15285
|
process.exit(2);
|
|
15281
15286
|
}
|
|
15282
|
-
const project = opts.
|
|
15283
|
-
if (opts.
|
|
15284
|
-
profile: opts.
|
|
15287
|
+
const project = opts.hubProfile !== void 0 ? resolveProject(opts) : void 0;
|
|
15288
|
+
if (opts.hubProfile !== void 0) await applyProfileFromOption({
|
|
15289
|
+
profile: opts.hubProfile,
|
|
15285
15290
|
project,
|
|
15286
15291
|
cwd: cwdForProfile,
|
|
15287
15292
|
hubUrl: opts.hubUrl,
|
|
@@ -15303,6 +15308,10 @@ async function runRecord(specPath, opts) {
|
|
|
15303
15308
|
hub: hubClientForTrace,
|
|
15304
15309
|
project: hubProject
|
|
15305
15310
|
} : null;
|
|
15311
|
+
if (opts.learnHubTracePrompt && hubContext === null) {
|
|
15312
|
+
error("--learn-hub-trace-prompt requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
15313
|
+
process.exit(2);
|
|
15314
|
+
}
|
|
15306
15315
|
const releaseLock = await acquireSpecLock(featureName, specName, "record", cwdForProfile).catch((e) => {
|
|
15307
15316
|
if (e instanceof SpecLockedError) {
|
|
15308
15317
|
error(e.message);
|
|
@@ -15330,11 +15339,11 @@ async function runRecord(specPath, opts) {
|
|
|
15330
15339
|
} finally {
|
|
15331
15340
|
await releaseLock();
|
|
15332
15341
|
}
|
|
15333
|
-
if (opts.
|
|
15342
|
+
if (opts.learnHubTracePrompt && traceResult !== null) {
|
|
15334
15343
|
blank();
|
|
15335
15344
|
await updateAgentPrompt({
|
|
15336
15345
|
kind: "record",
|
|
15337
|
-
flag: "--learn-trace-prompt",
|
|
15346
|
+
flag: "--learn-hub-trace-prompt",
|
|
15338
15347
|
runSummary: buildRecordRunSummary(featureName, specName, traceResult),
|
|
15339
15348
|
hubContext,
|
|
15340
15349
|
...opts.model ? { model: opts.model } : {},
|
|
@@ -15615,7 +15624,7 @@ function specStatus(result, threshold) {
|
|
|
15615
15624
|
return "passed";
|
|
15616
15625
|
}
|
|
15617
15626
|
/**
|
|
15618
|
-
* Adapts `ccqa
|
|
15627
|
+
* Adapts `ccqa audit` results into the shared RunReportData shape so they can
|
|
15619
15628
|
* be pushed to the hub (`ccqa audit --report-to-hub`) and rendered by the same report
|
|
15620
15629
|
* UI as `ccqa run`/`ccqa live`. Browser-execution fields (testCounts,
|
|
15621
15630
|
* evidence, liveRun, ...) don't apply to a drift audit and are always null —
|
|
@@ -15664,7 +15673,7 @@ function driftResultsToReport(results, meta) {
|
|
|
15664
15673
|
//#endregion
|
|
15665
15674
|
//#region src/cli/audit.ts
|
|
15666
15675
|
const DEFAULT_CONCURRENCY = 3;
|
|
15667
|
-
const auditCommand = addLanguageOption(new Command("audit").argument("[feature/spec]", "Optional spec id. If omitted, every spec under .ccqa/features/ is checked.").description("Read each spec against the code it describes and report where the two have drifted. Static: no browser is run, so this is the cheap check to put in front of `ccqa run`.").optionsGroup("Which specs to audit:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Costs one model call; specs it cannot decide are audited rather than skipped.").optionsGroup("How to run it:").option("--concurrency <n>", `Parallel spec checks (default: ${DEFAULT_CONCURRENCY})`).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").optionsGroup("What to do with the results:").option("--report-format <fmt>", "Output format: text | json | github", "text").option("--report-to-hub", "Push the result to a ccqa hub as a run (kind: drift), which is what updates the drift ledger `ccqa run --only-audited-clean` reads.").option("--exit-on <level>", "Exit non-zero on this severity or higher: warn | error", "error").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory used as both the .ccqa root and the codebase Claude reads. Useful for monorepos. Defaults to process.cwd().").option("--project <name>", "Logical project name for the pushed run. Defaults to the current directory's name.").option(...hubUrlOption).option(...hubTokenOption).option(...hubHeaderOption)).action(withUsageErrors(async (specPath, opts) => {
|
|
15676
|
+
const auditCommand = addLanguageOption(new Command("audit").argument("[feature/spec]", "Optional spec id. If omitted, every spec under .ccqa/features/ is checked.").description("Read each spec against the code it describes and report where the two have drifted. Static: no browser is run, so this is the cheap check to put in front of `ccqa run`.").optionsGroup("Which specs to audit:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` judges reached by the diff against <ref> (e.g. origin/main). In pull_request CI, pass $GITHUB_BASE_REF. Costs one model call; specs it cannot decide are audited rather than skipped.").optionsGroup("How to run it:").option("--concurrency <n>", `Parallel spec checks (default: ${DEFAULT_CONCURRENCY})`).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").optionsGroup("What to do with the results:").option("--report-format <fmt>", "Output format: text | json | github", "text").option("--report-to-hub", "Push the result to a ccqa hub as a run (kind: drift), which is what updates the drift ledger `ccqa run --only-hub-audited-clean` reads.").option("--exit-on <level>", "Exit non-zero on this severity or higher: warn | error", "error").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory used as both the .ccqa root and the codebase Claude reads. Useful for monorepos. Defaults to process.cwd().").option("--project <name>", "Logical project name for the pushed run. Defaults to the current directory's name.").option(...hubUrlOption).option(...hubTokenOption).option(...hubHeaderOption)).action(withUsageErrors(async (specPath, opts) => {
|
|
15668
15677
|
await withCostTally(() => runAudit(specPath, opts));
|
|
15669
15678
|
}));
|
|
15670
15679
|
async function runAudit(specPath, opts) {
|
|
@@ -15723,9 +15732,9 @@ async function runAudit(specPath, opts) {
|
|
|
15723
15732
|
}
|
|
15724
15733
|
/**
|
|
15725
15734
|
* Push a finished drift audit to a ccqa hub as a `kind: "drift"` run, so it
|
|
15726
|
-
* shows up alongside `ccqa run` runs in the hub UI.
|
|
15727
|
-
*
|
|
15728
|
-
*
|
|
15735
|
+
* shows up alongside `ccqa run` runs in the hub UI. A missing hub connection
|
|
15736
|
+
* is a usage error, not a silent skip — a CI job that asked to publish and
|
|
15737
|
+
* did not must say so.
|
|
15729
15738
|
*
|
|
15730
15739
|
* `resolveHub` is injectable so tests can supply a fake `HubClient` without
|
|
15731
15740
|
* a real hub connection; it defaults to the real flag/env resolution.
|
|
@@ -15734,8 +15743,8 @@ async function pushDriftResults(args, resolveHub = resolveHubClient) {
|
|
|
15734
15743
|
const { results, threshold, cwd, opts, format, baseRef } = args;
|
|
15735
15744
|
const hub = resolveHub(opts);
|
|
15736
15745
|
if (!hub) {
|
|
15737
|
-
|
|
15738
|
-
|
|
15746
|
+
error("--report-to-hub requires a hub connection (--hub-url/--hub-token or CCQA_HUB_URL/CCQA_HUB_TOKEN)");
|
|
15747
|
+
process.exit(2);
|
|
15739
15748
|
}
|
|
15740
15749
|
try {
|
|
15741
15750
|
const project = resolveProject({
|
|
@@ -15812,7 +15821,7 @@ function parseFormat$1(raw) {
|
|
|
15812
15821
|
function parseSeverity(raw) {
|
|
15813
15822
|
const v = raw ?? "error";
|
|
15814
15823
|
if (v === "warn" || v === "error") return v;
|
|
15815
|
-
error(`invalid --
|
|
15824
|
+
error(`invalid --exit-on: ${v} (expected warn|error)`);
|
|
15816
15825
|
process.exit(2);
|
|
15817
15826
|
}
|
|
15818
15827
|
function parseConcurrency(raw) {
|
|
@@ -16282,7 +16291,7 @@ function countSpecs(results) {
|
|
|
16282
16291
|
* or `red` (the last outcome), which are orthogonal axes.
|
|
16283
16292
|
*
|
|
16284
16293
|
* Best-effort — a ledger failure must not fail the push; the ledger is an
|
|
16285
|
-
* accelerator for `--
|
|
16294
|
+
* accelerator for `--on-fail-explain` baselines and re-run selection, not
|
|
16286
16295
|
* part of the run record. Runs without a branch or gitHead can't be placed in
|
|
16287
16296
|
* the ledger and are skipped.
|
|
16288
16297
|
*
|
|
@@ -16926,7 +16935,7 @@ function createGetLastGreenHandler(storage) {
|
|
|
16926
16935
|
/**
|
|
16927
16936
|
* GET /api/v1/projects/:project/drift
|
|
16928
16937
|
*
|
|
16929
|
-
* Every spec's last `ccqa
|
|
16938
|
+
* Every spec's last `ccqa audit --report-to-hub` result, keyed by "feature/spec". No
|
|
16930
16939
|
* `?profile=` — drift asks whether a spec still describes the code, which
|
|
16931
16940
|
* has nothing to do with which environment is running it, unlike the
|
|
16932
16941
|
* `/rerun` and `/last-green` endpoints. Merged across every branch (newest
|
|
@@ -18028,7 +18037,7 @@ const HTML_BODY = `
|
|
|
18028
18037
|
<div style="font-weight:600" data-i18n="session.help.title">How to get this JSON</div>
|
|
18029
18038
|
<ol class="help-steps">
|
|
18030
18039
|
<li><span class="step-n">1</span><div class="step-b"><span data-i18n="session.help.step1">Run this in your terminal and log in by hand when the browser opens:</span>
|
|
18031
|
-
<div class="cmd"><code id="session-help-cmd">ccqa session
|
|
18040
|
+
<div class="cmd"><code id="session-help-cmd">ccqa hub session capture <name></code><button type="button" class="copy" id="session-help-copy"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><rect x="9" y="9" width="11" height="11" rx="2"/><path d="M5 15V5a2 2 0 0 1 2-2h10"/></svg><span data-i18n="common.copy">Copy</span></button></div>
|
|
18032
18041
|
</div></li>
|
|
18033
18042
|
<li><span class="step-n">2</span><div class="step-b"><span data-i18n="session.help.step2">Open the saved file and paste its contents below:</span>
|
|
18034
18043
|
<div style="margin-top:5px"><span class="path">.ccqa/sessions/<profile>/<name>.json</span></div>
|
|
@@ -20154,7 +20163,7 @@ const CLIENT_JS = `
|
|
|
20154
20163
|
}
|
|
20155
20164
|
|
|
20156
20165
|
// A drift-kind row's diagnosis lives in analysis regardless of status
|
|
20157
|
-
// (an UNKNOWN-labelled finding below the --
|
|
20166
|
+
// (an UNKNOWN-labelled finding below the --exit-on threshold still
|
|
20158
20167
|
// "passes" but has something to show); a normal run only ever classifies
|
|
20159
20168
|
// a failed spec.
|
|
20160
20169
|
var hasAnalysis = isDrift ? !!r.analysis : r.status === "failed" && r.analysis;
|
|
@@ -21147,7 +21156,7 @@ const CLIENT_JS = `
|
|
|
21147
21156
|
//
|
|
21148
21157
|
// "unknown" keeps its own state rather than folding into the last result:
|
|
21149
21158
|
// it means the hub cannot say whether that result still holds, and
|
|
21150
|
-
// --
|
|
21159
|
+
// --only-hub-stale does not re-run it without --only-hub-stale-with-unknown. Showing
|
|
21151
21160
|
// it as passed or failed would claim a confidence nothing supports.
|
|
21152
21161
|
function perspRunState(rr) {
|
|
21153
21162
|
if (!rr) return null;
|
|
@@ -151,7 +151,7 @@ type PutActualCauseRequest = z.infer<typeof PutActualCauseRequestSchema>;
|
|
|
151
151
|
* updates the ledger whenever a `kind: "run"` run reaches a terminal state:
|
|
152
152
|
* every spec that passed gets its entry advanced to that run's `gitHead`
|
|
153
153
|
* (newest `at` wins, so out-of-order finalizes can't move a baseline
|
|
154
|
-
* backwards). `ccqa run --
|
|
154
|
+
* backwards). `ccqa run --on-fail-explain` reads it to diff each
|
|
155
155
|
* failing spec against the commit where that spec was last green.
|
|
156
156
|
*/
|
|
157
157
|
declare const LastGreenEntrySchema: z.ZodObject<{
|
|
@@ -297,7 +297,7 @@ type DriftLedgerResponse = z.infer<typeof DriftLedgerResponseSchema>;
|
|
|
297
297
|
*
|
|
298
298
|
* Two kinds share one namespace:
|
|
299
299
|
* - "guidance": the record/live prompt bundle — `.user.md` (human-maintained)
|
|
300
|
-
* and `.agent.md` (auto-rewritten by `ccqa run --
|
|
300
|
+
* and `.agent.md` (auto-rewritten by `ccqa run --learn-hub-live-prompt`) —
|
|
301
301
|
* plus `triage.user`, the human-maintained guidance injected into the
|
|
302
302
|
* failure-analysis (triage) prompt.
|
|
303
303
|
* - "custom-prompt": `analysis-custom-prompt` — Claude-written calibration guidance
|
|
@@ -820,7 +820,7 @@ interface HubClient {
|
|
|
820
820
|
}): Promise<Record<string, LastGreenEntry>>;
|
|
821
821
|
/**
|
|
822
822
|
* Per spec of one project/profile: is its last result still trustworthy?
|
|
823
|
-
* Answers `ccqa run --
|
|
823
|
+
* Answers `ccqa run --only-hub-stale`. 404 when the project has no
|
|
824
824
|
* perspectives document — there is then no spec registered to compare
|
|
825
825
|
* against a deploy, which the caller must report rather than read as
|
|
826
826
|
* "nothing to run".
|
package/dist/package.json
CHANGED