ccqa 1.38.0 → 1.40.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -4
- package/dist/bin/ccqa.mjs +514 -502
- package/dist/hub-client/index.d.mts +56 -1
- package/dist/hub-client/index.mjs +6 -0
- package/dist/package.json +1 -1
- package/package.json +1 -1
package/dist/bin/ccqa.mjs
CHANGED
|
@@ -4130,10 +4130,11 @@ const execFileP = promisify(execFile);
|
|
|
4130
4130
|
* `.ccqa/`. Changes outside `cwd` are kept under their repo-root path and
|
|
4131
4131
|
* flagged `outsideCwd` (see `ChangedFile`) rather than dropped.
|
|
4132
4132
|
*
|
|
4133
|
-
* `detectRenames` defaults on.
|
|
4134
|
-
* `
|
|
4135
|
-
* only the destination, so a file's
|
|
4136
|
-
*
|
|
4133
|
+
* `detectRenames` defaults on. Callers whose diff is held against past facts
|
|
4134
|
+
* — `ccqa hub deploy record` and spec selection — turn it off: with rename
|
|
4135
|
+
* detection, a rename is one entry naming only the destination, so a file's
|
|
4136
|
+
* old path would silently drop out — off, it appears as a delete plus an add
|
|
4137
|
+
* and both paths are kept.
|
|
4137
4138
|
*/
|
|
4138
4139
|
async function getChangedFilesBetween(base, head, cwd, options = {}) {
|
|
4139
4140
|
return diffNameStatus(`${base}..${head}`, cwd, options.detectRenames ?? true);
|
|
@@ -7340,7 +7341,9 @@ async function closeMeasurement(collector, ref, coverageDir) {
|
|
|
7340
7341
|
* Every failure here is otherwise silent and identical to success: a root that
|
|
7341
7342
|
* does not exist, or does not contain the project, sends every relative source
|
|
7342
7343
|
* outside it, and the run reports a smaller file set with no error at all —
|
|
7343
|
-
* the answer this measurement exists to prevent.
|
|
7344
|
+
* the answer this measurement exists to prevent. Spec selection
|
|
7345
|
+
* (src/select/analyze.ts) resolves the same key through this function too, so
|
|
7346
|
+
* both sides of an intersection agree on what the root means.
|
|
7344
7347
|
*/
|
|
7345
7348
|
async function resolveRoot(cwd, declared) {
|
|
7346
7349
|
if (declared === void 0) return void 0;
|
|
@@ -9390,8 +9393,8 @@ z.object({
|
|
|
9390
9393
|
*
|
|
9391
9394
|
* This is the whole re-run computation, not an accelerator for it. Each spec's
|
|
9392
9395
|
* baseline sits at a different position in the log, so the question is always
|
|
9393
|
-
* "since *this* spec last ran, was it ever needed" —
|
|
9394
|
-
* against the
|
|
9396
|
+
* "since *this* spec last ran, was it ever needed" — one integer comparison
|
|
9397
|
+
* against the `needed` position kept here. Re-deriving it per read is impossible
|
|
9395
9398
|
* anyway: the selections were made by a model against each deploy's diff, and
|
|
9396
9399
|
* neither the diff nor the model is available at read time.
|
|
9397
9400
|
*
|
|
@@ -10835,159 +10838,170 @@ function capDeployPaths(paths) {
|
|
|
10835
10838
|
return paths.slice(0, MAX_SENT_CHANGED_PATHS);
|
|
10836
10839
|
}
|
|
10837
10840
|
//#endregion
|
|
10838
|
-
//#region src/
|
|
10841
|
+
//#region src/config/project-config.ts
|
|
10839
10842
|
/**
|
|
10840
|
-
*
|
|
10843
|
+
* Loader for the consumer project's `.ccqa/config.yaml` — per-target
|
|
10844
|
+
* generation settings (default target, output dirs, reusable code resources,
|
|
10845
|
+
* generation conventions).
|
|
10841
10846
|
*
|
|
10842
|
-
*
|
|
10843
|
-
*
|
|
10844
|
-
*
|
|
10845
|
-
* a claim that has to be earned, and `unknown` is always available instead.
|
|
10847
|
+
* This module only validates and holds the config. `path` / `guides` /
|
|
10848
|
+
* `examples` entries may be glob patterns; they are kept verbatim here and
|
|
10849
|
+
* expanded by the generation engine, which owns size limits and warnings.
|
|
10846
10850
|
*/
|
|
10847
|
-
function buildSelectSystemPrompt() {
|
|
10848
|
-
return `You decide which end-to-end test specs have to be re-run after a set of source changes.
|
|
10849
|
-
|
|
10850
|
-
You are given the files that changed between two commits, and an inventory of every test spec: what each one does, step by step. Return one verdict per spec.
|
|
10851
|
-
|
|
10852
|
-
## The three verdicts
|
|
10853
|
-
|
|
10854
|
-
- **needed** — at least one changed file plausibly affects what this spec verifies. Name the file(s) in \`touchedBy\`.
|
|
10855
|
-
- **notNeeded** — you have accounted for the changed files and none of them reach what this spec does.
|
|
10856
|
-
- **unknown** — you cannot tell.
|
|
10857
|
-
|
|
10858
|
-
## Why \`notNeeded\` is the only answer that can hurt
|
|
10859
|
-
|
|
10860
|
-
Re-running a spec that did not need it costs a few minutes of CI. NOT re-running a spec that needed it lets a regression reach users with the suite still green — the failure mode this tool exists to prevent.
|
|
10861
|
-
|
|
10862
|
-
So the two answers are not symmetric:
|
|
10863
|
-
|
|
10864
|
-
- \`needed\` and \`unknown\` are both safe. The caller runs them.
|
|
10865
|
-
- \`notNeeded\` is a **positive claim**. Make it only when you have actually looked at what the spec exercises and at what changed, and can say the two do not meet.
|
|
10866
|
-
|
|
10867
|
-
When you cannot make that claim — a file whose purpose you cannot infer, a change whose blast radius you cannot bound, a spec whose steps are missing or unclear — answer \`unknown\`. That is a correct answer, not a failure. Guessing \`notNeeded\` is the one thing that causes damage.
|
|
10868
|
-
|
|
10869
|
-
**Do not over-correct.** Marking every spec \`needed\` throws away the entire point: the caller ends up running the full suite on every change. When a change is confined to an area that a spec demonstrably never touches, say \`notNeeded\` and say why. Both a reflexive "run everything" and a careless "skip it" are wrong; judge each spec on the evidence.
|
|
10870
|
-
|
|
10871
|
-
## How to judge
|
|
10872
|
-
|
|
10873
|
-
1. Read the changed paths. Most are self-describing — a path naming a screen, a component, a handler, or a route tells you what it belongs to.
|
|
10874
|
-
2. For a path whose purpose is not obvious from its name, use \`Read\` or \`Grep\` to find out what it does before judging. Prefer this over guessing. Stay focused: this is a routing decision, not a code review.
|
|
10875
|
-
3. Match against what each spec actually does — the screens its steps open, the controls they drive, the strings they assert on.
|
|
10876
|
-
4. Remember indirect reach: shared layout, navigation, authentication, permission checks, and data-access code are touched by specs that never mention them. A change to a sign-in path can break every spec that has to sign in first.
|
|
10877
|
-
|
|
10878
|
-
## What does not affect any spec
|
|
10879
|
-
|
|
10880
|
-
Treat these as irrelevant unless something specific says otherwise: documentation and Markdown, changes to test files of the product's own unit-test suite, lockfile-only churn, formatting-only changes, and code paths that only run in a build or tooling context.
|
|
10881
|
-
|
|
10882
|
-
## Output (STRICT)
|
|
10883
|
-
|
|
10884
|
-
Output ONE fenced \`\`\`json block, and nothing else outside it.
|
|
10885
|
-
|
|
10886
|
-
\`\`\`json
|
|
10887
|
-
{
|
|
10888
|
-
"specs": [
|
|
10889
|
-
{
|
|
10890
|
-
"spec": "<feature>/<spec>",
|
|
10891
|
-
"verdict": "needed" | "notNeeded" | "unknown",
|
|
10892
|
-
"reason": "<one sentence>",
|
|
10893
|
-
"touchedBy": ["<changed path>"]
|
|
10894
|
-
}
|
|
10895
|
-
]
|
|
10896
|
-
}
|
|
10897
|
-
\`\`\`
|
|
10898
|
-
|
|
10899
|
-
Rules for the output:
|
|
10900
|
-
|
|
10901
|
-
- Include **every** spec from the inventory, exactly once, using the key as it was given to you.
|
|
10902
|
-
- \`touchedBy\` is required for \`needed\` and must name paths from the changed-file list. Omit it otherwise.
|
|
10903
|
-
- \`reason\` for \`notNeeded\` must say what you checked, not just "unrelated".
|
|
10904
|
-
- \`reason\` for \`unknown\` must name what you could not determine.
|
|
10905
|
-
`;
|
|
10906
|
-
}
|
|
10907
|
-
/** Cap on how many changed paths are spelled out before the list is summarised. */
|
|
10908
|
-
const MAX_LISTED_PATHS = 400;
|
|
10909
|
-
function buildSelectPrompt(input) {
|
|
10910
|
-
return `## Changed files (${input.base} → ${input.head})
|
|
10911
|
-
|
|
10912
|
-
${formatChangedFiles(input.changed)}
|
|
10913
|
-
|
|
10914
|
-
## Test spec inventory
|
|
10915
|
-
|
|
10916
|
-
${input.specs.map(formatSpec).join("\n\n")}
|
|
10917
|
-
|
|
10918
|
-
## Task
|
|
10919
|
-
|
|
10920
|
-
Return one verdict for each of the ${input.specs.length} specs above. Clear a spec only when you can account for the changes; otherwise answer \`needed\` or \`unknown\`.
|
|
10921
|
-
`;
|
|
10922
|
-
}
|
|
10923
|
-
function formatChangedFiles(changed) {
|
|
10924
|
-
if (changed.length === 0) return "(none)";
|
|
10925
|
-
const listed = changed.slice(0, MAX_LISTED_PATHS);
|
|
10926
|
-
const lines = listed.map((f) => `- ${f.status.padEnd(8)} ${f.path}${f.outsideCwd ? " (outside the tested package)" : ""}`);
|
|
10927
|
-
if (changed.length > listed.length) lines.push(`- ... and ${changed.length - listed.length} more paths, not listed. You have NOT seen the full change set: do not answer \`notNeeded\` for a spec unless the listed paths alone rule it out.`);
|
|
10928
|
-
return lines.join("\n");
|
|
10929
|
-
}
|
|
10930
|
-
function formatSpec(spec) {
|
|
10931
|
-
const steps = spec.steps.map((s, i) => ` ${i + 1}. ${s}`).join("\n");
|
|
10932
|
-
return `### ${specKey(spec)}\n${spec.title}\n${steps}`;
|
|
10933
|
-
}
|
|
10934
|
-
//#endregion
|
|
10935
|
-
//#region src/select/types.ts
|
|
10936
10851
|
/**
|
|
10937
|
-
*
|
|
10938
|
-
*
|
|
10939
|
-
* `
|
|
10852
|
+
* An existing code asset the generated tests should reuse (import), in one of
|
|
10853
|
+
* two forms — exactly one of:
|
|
10854
|
+
* - `path`: code inside the consumer repo (literal path or glob pattern);
|
|
10855
|
+
* - `package`: an installed npm package (imported by name).
|
|
10856
|
+
* `description` tells the generator what the asset contains.
|
|
10940
10857
|
*/
|
|
10941
|
-
const
|
|
10942
|
-
|
|
10943
|
-
|
|
10944
|
-
|
|
10945
|
-
|
|
10946
|
-
|
|
10947
|
-
|
|
10948
|
-
|
|
10949
|
-
|
|
10950
|
-
|
|
10951
|
-
|
|
10952
|
-
|
|
10953
|
-
|
|
10954
|
-
|
|
10955
|
-
|
|
10956
|
-
|
|
10957
|
-
|
|
10958
|
-
|
|
10959
|
-
}
|
|
10858
|
+
const ResourceRefSchema = z.union([z.object({
|
|
10859
|
+
path: z.string().min(1),
|
|
10860
|
+
description: z.string().optional()
|
|
10861
|
+
}).strict(), z.object({
|
|
10862
|
+
package: z.string().min(1),
|
|
10863
|
+
description: z.string().optional()
|
|
10864
|
+
}).strict()], { error: "a resource must have exactly one of `path` (code in this repo) or `package` (installed npm package), plus an optional `description`" });
|
|
10865
|
+
/**
|
|
10866
|
+
* How generated code should be written, as guide inputs to the prompt (never
|
|
10867
|
+
* imported as code): `guides` are convention documents, `examples` are
|
|
10868
|
+
* existing tests whose style to imitate. Entries may be glob patterns.
|
|
10869
|
+
*/
|
|
10870
|
+
const ConventionsSchema = z.object({
|
|
10871
|
+
guides: z.array(z.string().min(1)).default([]),
|
|
10872
|
+
examples: z.array(z.string().min(1)).default([])
|
|
10873
|
+
}).strict();
|
|
10874
|
+
/**
|
|
10875
|
+
* Per-target settings. `outDir` (where generated tests are written) and
|
|
10876
|
+
* `runCommand` (how to execute them; `{files}` expands to the generated
|
|
10877
|
+
* paths, `{artifactsDir}` to the spec's report artifacts dir — see
|
|
10878
|
+
* src/targets/run-artifacts.ts) are optional at this layer because not every
|
|
10879
|
+
* target needs them — e.g. agent-browser stores its output in the spec
|
|
10880
|
+
* directory. A target that requires either must validate its presence itself.
|
|
10881
|
+
*/
|
|
10882
|
+
const TargetConfigSchema = z.object({
|
|
10883
|
+
outDir: z.string().min(1).optional(),
|
|
10884
|
+
runCommand: z.string().min(1).optional(),
|
|
10885
|
+
resources: z.array(ResourceRefSchema).default([]),
|
|
10886
|
+
conventions: ConventionsSchema.default({
|
|
10887
|
+
guides: [],
|
|
10888
|
+
examples: []
|
|
10889
|
+
})
|
|
10890
|
+
}).strict();
|
|
10960
10891
|
/**
|
|
10961
|
-
*
|
|
10962
|
-
* `touchedBy` is cross-checked against the actual diff.
|
|
10892
|
+
* Specs that must not run at the same time, grouped by the thing they share.
|
|
10963
10893
|
*
|
|
10964
|
-
*
|
|
10965
|
-
* the
|
|
10966
|
-
*
|
|
10967
|
-
*
|
|
10968
|
-
*
|
|
10894
|
+
* The key names the shared thing (a chat channel, a seeded account, a tenant);
|
|
10895
|
+
* the list names the specs that write to it. `ccqa run` never runs two members
|
|
10896
|
+
* of one group concurrently, and specs sharing no group still run in parallel.
|
|
10897
|
+
*
|
|
10898
|
+
* Kept here rather than on each spec so there is one place to read the whole
|
|
10899
|
+
* picture, and so a mistyped member is a spec key that does not resolve —
|
|
10900
|
+
* caught — rather than a resource name that silently matches nothing.
|
|
10969
10901
|
*/
|
|
10970
|
-
const
|
|
10971
|
-
|
|
10972
|
-
|
|
10973
|
-
|
|
10974
|
-
|
|
10975
|
-
|
|
10976
|
-
|
|
10977
|
-
|
|
10902
|
+
const SerialGroupsSchema = z.record(z.string().regex(/^[a-z0-9][a-z0-9._-]*$/i, "serial group name must be a slug (letters, digits, '.', '_', '-')"), z.array(z.string().min(1)).min(1));
|
|
10903
|
+
/**
|
|
10904
|
+
* Which specs act as which external identity, for the flows whose requests
|
|
10905
|
+
* cannot carry a spec id at all.
|
|
10906
|
+
*
|
|
10907
|
+
* A chat platform's webhook is sent by the platform, not the browser, so no
|
|
10908
|
+
* cookie rides along and everything the flow reaches would be unattributed.
|
|
10909
|
+
* What the request does carry is who caused it, and if only one spec is allowed
|
|
10910
|
+
* to act as that identity at a time, "who" plus "when" is enough.
|
|
10911
|
+
*
|
|
10912
|
+
* ```yaml
|
|
10913
|
+
* coverage:
|
|
10914
|
+
* actors:
|
|
10915
|
+
* slack: # the preset's tag prefix
|
|
10916
|
+
* ${TEST_USER_ID}: [chat/create-item, chat/resolve-item]
|
|
10917
|
+
* ```
|
|
10918
|
+
*
|
|
10919
|
+
* The provider name is the prefix the matching preset stamps, and the key is an
|
|
10920
|
+
* identity expression the run's variables resolve. Only the unexpanded text is
|
|
10921
|
+
* ever displayed or used as a lock key, so the identity itself stays out of
|
|
10922
|
+
* reports and the hub.
|
|
10923
|
+
*/
|
|
10924
|
+
const CoverageActorsSchema = z.record(z.string().regex(/^[a-z0-9][a-z0-9._-]*$/i, "actor provider must be a slug (letters, digits, '.', '_', '-')"), z.record(z.string().min(1), z.array(z.string().min(1)).min(1)));
|
|
10925
|
+
/**
|
|
10926
|
+
* Settings for `ccqa run --coverage`, which measures what each spec actually
|
|
10927
|
+
* reached in the application under test.
|
|
10928
|
+
*/
|
|
10929
|
+
const CoverageConfigSchema = z.object({
|
|
10930
|
+
instrumentedOrigins: z.array(z.string().min(1)).min(1),
|
|
10931
|
+
sink: z.string().min(1).default("http://127.0.0.1:4757"),
|
|
10932
|
+
projectRoot: z.string().min(1).optional(),
|
|
10933
|
+
include: z.array(z.string().min(1)).optional(),
|
|
10934
|
+
actors: CoverageActorsSchema.default({})
|
|
10935
|
+
}).strict();
|
|
10936
|
+
/**
|
|
10937
|
+
* Top-level `.ccqa/config.yaml` schema. `defaultTarget` is used by specs
|
|
10938
|
+
* with no `target:` of their own. Both defaults make a missing config file
|
|
10939
|
+
* equivalent to "agent-browser only, no extra settings".
|
|
10940
|
+
*/
|
|
10941
|
+
const ProjectConfigSchema = z.object({
|
|
10942
|
+
defaultTarget: TargetIdSchema.default(AGENT_BROWSER_TARGET),
|
|
10943
|
+
targets: z.record(TargetIdSchema, TargetConfigSchema).default({}),
|
|
10944
|
+
serialGroups: SerialGroupsSchema.default({}),
|
|
10945
|
+
coverage: CoverageConfigSchema.optional()
|
|
10946
|
+
}).strict();
|
|
10947
|
+
/** Config file location, relative to the project root (`--cwd`). */
|
|
10948
|
+
const PROJECT_CONFIG_PATH = ".ccqa/config.yaml";
|
|
10949
|
+
/**
|
|
10950
|
+
* Load `<cwd>/.ccqa/config.yaml`. A missing file yields the defaults (an
|
|
10951
|
+
* empty file too); a present but broken file is an error — never silently
|
|
10952
|
+
* fall back when the user wrote a config.
|
|
10953
|
+
*/
|
|
10954
|
+
async function loadProjectConfig(cwd) {
|
|
10955
|
+
let content;
|
|
10956
|
+
try {
|
|
10957
|
+
content = await readFile(join(cwd, PROJECT_CONFIG_PATH), "utf8");
|
|
10958
|
+
} catch (e) {
|
|
10959
|
+
if (e.code === "ENOENT") return ProjectConfigSchema.parse({});
|
|
10960
|
+
throw e;
|
|
10961
|
+
}
|
|
10962
|
+
return parseProjectConfig(content);
|
|
10963
|
+
}
|
|
10964
|
+
/** Parse config YAML. Schema rejections are rewritten with actionable messages. */
|
|
10965
|
+
function parseProjectConfig(content, source = PROJECT_CONFIG_PATH) {
|
|
10966
|
+
let raw;
|
|
10967
|
+
try {
|
|
10968
|
+
raw = parse(content);
|
|
10969
|
+
} catch (e) {
|
|
10970
|
+
throw new Error(`Failed to parse YAML (${source}): ${e.message}`);
|
|
10971
|
+
}
|
|
10972
|
+
try {
|
|
10973
|
+
return ProjectConfigSchema.parse(raw ?? {});
|
|
10974
|
+
} catch (e) {
|
|
10975
|
+
throw enrichZodError(e, source);
|
|
10976
|
+
}
|
|
10977
|
+
}
|
|
10978
|
+
/** Flatten a ZodError into one `Invalid <source>:` message, path per line. */
|
|
10979
|
+
function enrichZodError(error, source) {
|
|
10980
|
+
if (!(error instanceof ZodError)) return error;
|
|
10981
|
+
const lines = [`Invalid ${source}:`];
|
|
10982
|
+
for (const issue of error.issues) {
|
|
10983
|
+
const path = issue.path.join(".") || "(root)";
|
|
10984
|
+
const message = issue.code === "invalid_key" && issue.issues[0] ? issue.issues[0].message : issue.message;
|
|
10985
|
+
lines.push(` - ${path}: ${message}`);
|
|
10986
|
+
}
|
|
10987
|
+
return new Error(lines.join("\n"));
|
|
10988
|
+
}
|
|
10978
10989
|
//#endregion
|
|
10979
10990
|
//#region src/select/analyze.ts
|
|
10980
10991
|
/**
|
|
10981
10992
|
* Decide which specs a change set reaches.
|
|
10982
10993
|
*
|
|
10983
|
-
* Two passes, in this order and for this reason: what
|
|
10984
|
-
*
|
|
10985
|
-
*
|
|
10986
|
-
* spec must re-run —
|
|
10987
|
-
*
|
|
10994
|
+
* Two passes, in this order and for this reason: what a change to ccqa's own
|
|
10995
|
+
* tree settles is settled first, and only the remainder is held against
|
|
10996
|
+
* measured reach. A change to a spec's own file, or to a block it includes,
|
|
10997
|
+
* means that spec must re-run — reach cannot see the test's own definition,
|
|
10998
|
+
* so no measurement is consulted for it. Everything else intersects the diff
|
|
10999
|
+
* with the files the spec's last measured run actually reached (ADR-0024);
|
|
11000
|
+
* a spec with no measurement stays `unknown`, because an unmeasured edge is
|
|
11001
|
+
* not an unreached one.
|
|
10988
11002
|
*/
|
|
10989
11003
|
async function selectSpecs(input) {
|
|
10990
|
-
const { changed, specs, cwd, base, head,
|
|
11004
|
+
const { changed, specs, cwd, base, head, edges } = input;
|
|
10991
11005
|
const { productChanges, mechanicallyNeeded } = partitionChanges(changed, specs);
|
|
10992
11006
|
const byInventoryKey = new Map(specs.map((s) => [specKey(s), s]));
|
|
10993
11007
|
const byKey = /* @__PURE__ */ new Map();
|
|
@@ -11011,13 +11025,11 @@ async function selectSpecs(input) {
|
|
|
11011
11025
|
source: "mechanical",
|
|
11012
11026
|
reason: "no file outside .ccqa/ changed in this range"
|
|
11013
11027
|
});
|
|
11014
|
-
else if (undecided.length > 0) for (const selection of await
|
|
11028
|
+
else if (undecided.length > 0) for (const selection of await judgeWithCoverage({
|
|
11029
|
+
pending: undecided,
|
|
11015
11030
|
productChanges,
|
|
11016
|
-
undecided,
|
|
11017
11031
|
cwd,
|
|
11018
|
-
|
|
11019
|
-
head,
|
|
11020
|
-
model
|
|
11032
|
+
edges
|
|
11021
11033
|
})) byKey.set(specKey(selection), selection);
|
|
11022
11034
|
return {
|
|
11023
11035
|
base,
|
|
@@ -11027,12 +11039,12 @@ async function selectSpecs(input) {
|
|
|
11027
11039
|
};
|
|
11028
11040
|
}
|
|
11029
11041
|
/**
|
|
11030
|
-
* Split the diff into the part
|
|
11031
|
-
* decides itself.
|
|
11042
|
+
* Split the diff into the part measured reach has to answer for and the part
|
|
11043
|
+
* that decides itself.
|
|
11032
11044
|
*
|
|
11033
11045
|
* `.ccqa/` paths are ccqa's own: a spec directory names the spec it belongs
|
|
11034
11046
|
* to, a block names the specs that include it. Product paths carry no such
|
|
11035
|
-
* mapping —
|
|
11047
|
+
* mapping — those are what the coverage intersection exists to answer.
|
|
11036
11048
|
*/
|
|
11037
11049
|
function partitionChanges(changed, specs) {
|
|
11038
11050
|
const productChanges = [];
|
|
@@ -11079,118 +11091,214 @@ function isCcqaPath(path) {
|
|
|
11079
11091
|
return /(?:^|\/)\.ccqa\//.test(path);
|
|
11080
11092
|
}
|
|
11081
11093
|
/**
|
|
11082
|
-
*
|
|
11083
|
-
*
|
|
11084
|
-
*
|
|
11085
|
-
*
|
|
11086
|
-
*
|
|
11087
|
-
|
|
11088
|
-
|
|
11089
|
-
|
|
11090
|
-
*
|
|
11091
|
-
*
|
|
11092
|
-
*
|
|
11093
|
-
*
|
|
11094
|
-
|
|
11095
|
-
|
|
11096
|
-
|
|
11097
|
-
|
|
11098
|
-
|
|
11099
|
-
const
|
|
11100
|
-
|
|
11101
|
-
|
|
11102
|
-
|
|
11103
|
-
const
|
|
11104
|
-
|
|
11105
|
-
|
|
11106
|
-
|
|
11107
|
-
base,
|
|
11108
|
-
head
|
|
11109
|
-
}),
|
|
11110
|
-
systemPrompt: buildSelectSystemPrompt(),
|
|
11111
|
-
allowedTools: [
|
|
11112
|
-
"Read",
|
|
11113
|
-
"Grep",
|
|
11114
|
-
"Glob"
|
|
11115
|
-
],
|
|
11116
|
-
silenceBashLog: true,
|
|
11117
|
-
cwd,
|
|
11118
|
-
...model ? { model } : {}
|
|
11119
|
-
}, (_msg) => {});
|
|
11120
|
-
if (isError) {
|
|
11121
|
-
lastError = "the selection model returned an error";
|
|
11122
|
-
continue;
|
|
11123
|
-
}
|
|
11124
|
-
const json = extractJsonBlock(result);
|
|
11125
|
-
if (!json) {
|
|
11126
|
-
lastError = "the selection model returned no JSON block";
|
|
11127
|
-
continue;
|
|
11128
|
-
}
|
|
11129
|
-
try {
|
|
11130
|
-
parsed = JSON.parse(json);
|
|
11131
|
-
lastError = "";
|
|
11132
|
-
break;
|
|
11133
|
-
} catch (e) {
|
|
11134
|
-
lastError = `the selection model's JSON did not parse: ${e.message}`;
|
|
11135
|
-
}
|
|
11136
|
-
}
|
|
11137
|
-
if (lastError) return abandonSelection(undecided, `${lastError} (${MAX_ATTEMPTS} attempts)`);
|
|
11138
|
-
const changedPaths = new Set(productChanges.map((f) => f.path));
|
|
11139
|
-
const byUndecidedKey = new Map(undecided.map((s) => [specKey(s), s]));
|
|
11140
|
-
const answers = /* @__PURE__ */ new Map();
|
|
11141
|
-
for (const raw of readSpecArray(parsed)) {
|
|
11142
|
-
const spec = byUndecidedKey.get(raw.spec);
|
|
11143
|
-
if (!spec) continue;
|
|
11144
|
-
answers.set(raw.spec, {
|
|
11094
|
+
* Hold each undecided spec's last measured reach against the diff.
|
|
11095
|
+
*
|
|
11096
|
+
* Three outcomes, and only the middle one is a positive claim: no
|
|
11097
|
+
* measurement means `unknown` (an unmeasured edge is not an unreached one —
|
|
11098
|
+
* the absence of evidence runs the spec); a non-empty intersection means
|
|
11099
|
+
* `needed`, with the intersecting paths as the reason; an empty one means
|
|
11100
|
+
* `notNeeded` — the measurement accounts for everything the spec reached,
|
|
11101
|
+
* and the diff missed all of it. Changes outside the measured root fall out
|
|
11102
|
+
* of the comparison entirely: the root is the declared boundary of what
|
|
11103
|
+
* measurement governs, so what lies beyond it clears specs quietly — one
|
|
11104
|
+
* warning names the dropped paths, because a root configured too narrow
|
|
11105
|
+
* looks exactly like this and hides real reach (see docs/coverage.md).
|
|
11106
|
+
*/
|
|
11107
|
+
async function judgeWithCoverage(input) {
|
|
11108
|
+
const { pending, productChanges, cwd, edges } = input;
|
|
11109
|
+
const noMeasurement = "no measurement to consult: the hub holds no measured reach for this spec";
|
|
11110
|
+
if (edges.size === 0) return pending.map((s) => unknownSelection(s, noMeasurement));
|
|
11111
|
+
const measuredChanges = rerootChangesForCoverage(productChanges, await resolveCoverageRoots(productChanges, cwd));
|
|
11112
|
+
const dropped = productChanges.length - measuredChanges.length;
|
|
11113
|
+
if (dropped > 0) warn(`select-specs: ${dropped} of ${productChanges.length} changed files fall outside coverage.projectRoot and cannot be compared against measured reach`);
|
|
11114
|
+
return pending.map((spec) => {
|
|
11115
|
+
const edge = edges.get(specKey(spec));
|
|
11116
|
+
if (!edge) return unknownSelection(spec, noMeasurement);
|
|
11117
|
+
const touchedBy = measuredChanges.filter((c) => edge.files.has(c.measured)).map((c) => c.original);
|
|
11118
|
+
if (touchedBy.length > 0) return {
|
|
11145
11119
|
featureName: spec.featureName,
|
|
11146
11120
|
specName: spec.specName,
|
|
11147
|
-
verdict:
|
|
11148
|
-
source: "
|
|
11149
|
-
reason:
|
|
11150
|
-
|
|
11151
|
-
}
|
|
11152
|
-
|
|
11153
|
-
|
|
11154
|
-
|
|
11155
|
-
|
|
11121
|
+
verdict: "needed",
|
|
11122
|
+
source: "coverage",
|
|
11123
|
+
reason: "the change touches files this spec's last measured run reached",
|
|
11124
|
+
touchedBy
|
|
11125
|
+
};
|
|
11126
|
+
return {
|
|
11127
|
+
featureName: spec.featureName,
|
|
11128
|
+
specName: spec.specName,
|
|
11129
|
+
verdict: "notNeeded",
|
|
11130
|
+
source: "coverage",
|
|
11131
|
+
reason: "the spec's last measured run reached none of the changed files"
|
|
11132
|
+
};
|
|
11133
|
+
});
|
|
11134
|
+
}
|
|
11135
|
+
function unknownSelection(spec, reason) {
|
|
11136
|
+
return {
|
|
11137
|
+
featureName: spec.featureName,
|
|
11138
|
+
specName: spec.specName,
|
|
11139
|
+
verdict: "unknown",
|
|
11140
|
+
source: "coverage",
|
|
11141
|
+
reason
|
|
11142
|
+
};
|
|
11156
11143
|
}
|
|
11157
11144
|
/**
|
|
11158
|
-
*
|
|
11159
|
-
*
|
|
11160
|
-
*
|
|
11161
|
-
* `
|
|
11145
|
+
* Re-root diff paths to the measurement's own base. The two sides must speak
|
|
11146
|
+
* the same paths or every intersection silently misses (ADR-0024): the diff
|
|
11147
|
+
* is cwd-relative (repo-root relative for `outsideCwd` entries) while
|
|
11148
|
+
* measured files are `coverage.projectRoot`-relative. A file resolving
|
|
11149
|
+
* outside the coverage root is dropped — the measurement drops those files
|
|
11150
|
+
* too, so it could never intersect an edge.
|
|
11162
11151
|
*/
|
|
11163
|
-
function
|
|
11164
|
-
const reply = SelectModelReplySchema.safeParse(parsed);
|
|
11165
|
-
if (!reply.success) return [];
|
|
11152
|
+
function rerootChangesForCoverage(changed, roots) {
|
|
11166
11153
|
const out = [];
|
|
11167
|
-
for (const
|
|
11168
|
-
const
|
|
11169
|
-
if (
|
|
11154
|
+
for (const file of changed) {
|
|
11155
|
+
const base = file.outsideCwd ? roots.repoRoot : roots.cwd;
|
|
11156
|
+
if (base === null) continue;
|
|
11157
|
+
const measured = relative(roots.coverageRoot, resolve(base, file.path)).replaceAll("\\", "/");
|
|
11158
|
+
if (measured.startsWith("..")) continue;
|
|
11159
|
+
out.push({
|
|
11160
|
+
original: file.path,
|
|
11161
|
+
measured
|
|
11162
|
+
});
|
|
11170
11163
|
}
|
|
11171
11164
|
return out;
|
|
11172
11165
|
}
|
|
11173
11166
|
/**
|
|
11174
|
-
* The
|
|
11175
|
-
*
|
|
11167
|
+
* The roots `rerootChangesForCoverage` needs: `coverage.projectRoot` from
|
|
11168
|
+
* `.ccqa/config.yaml` (defaults to cwd), and the git repo root — resolved
|
|
11169
|
+
* only when an `outsideCwd` entry exists to anchor.
|
|
11176
11170
|
*
|
|
11177
|
-
*
|
|
11178
|
-
*
|
|
11179
|
-
*
|
|
11180
|
-
*
|
|
11171
|
+
* The projectRoot goes through the measurement's own `resolveRoot` — env refs
|
|
11172
|
+
* expanded, the directory verified to exist and contain cwd — and a config it
|
|
11173
|
+
* rejects fails here too. Resolving it any other way would silently re-root
|
|
11174
|
+
* every path somewhere the measurement never stored files under.
|
|
11181
11175
|
*/
|
|
11182
|
-
function
|
|
11183
|
-
|
|
11184
|
-
|
|
11176
|
+
async function resolveCoverageRoots(changed, cwd) {
|
|
11177
|
+
const coverageRoot = await resolveRoot(cwd, (await loadProjectConfig(cwd)).coverage?.projectRoot) ?? resolve(cwd);
|
|
11178
|
+
let repoRoot = null;
|
|
11179
|
+
if (changed.some((f) => f.outsideCwd)) try {
|
|
11180
|
+
const { stdout } = await execFileP("git", ["rev-parse", "--show-toplevel"], { cwd });
|
|
11181
|
+
repoRoot = stdout.trim();
|
|
11182
|
+
} catch {}
|
|
11183
|
+
return {
|
|
11184
|
+
cwd: resolve(cwd),
|
|
11185
|
+
repoRoot,
|
|
11186
|
+
coverageRoot
|
|
11187
|
+
};
|
|
11185
11188
|
}
|
|
11186
|
-
|
|
11187
|
-
|
|
11188
|
-
|
|
11189
|
-
|
|
11190
|
-
|
|
11191
|
-
|
|
11192
|
-
|
|
11193
|
-
|
|
11189
|
+
//#endregion
|
|
11190
|
+
//#region src/select/coverage-edges.ts
|
|
11191
|
+
/**
|
|
11192
|
+
* How many recent hub runs are probed for report-row coverage. Bounded
|
|
11193
|
+
* because every probe downloads a whole report.json; past this many runs a
|
|
11194
|
+
* measurement is old enough that treating it as absent — `unknown`, so the
|
|
11195
|
+
* spec runs — is the safer answer anyway. The stream side carries its own
|
|
11196
|
+
* bound: the hub lists at most its newest twenty measured runs.
|
|
11197
|
+
*/
|
|
11198
|
+
const MAX_REPORT_RUNS = 20;
|
|
11199
|
+
/**
|
|
11200
|
+
* How old a measurement may be and still decide a spec. The same fourteen
|
|
11201
|
+
* days the stream store retains events for (`COVERAGE_RETENTION_DAYS`): past
|
|
11202
|
+
* it an edge is too stale to clear a spec with confidence, so it is not
|
|
11203
|
+
* adopted and the spec degrades to `unknown` — which runs.
|
|
11204
|
+
*/
|
|
11205
|
+
const EDGE_MAX_AGE_MS = 336 * 60 * 60 * 1e3;
|
|
11206
|
+
/**
|
|
11207
|
+
* Read every spec's most recent measured reach from the hub.
|
|
11208
|
+
*
|
|
11209
|
+
* Never throws: a hub that cannot be read yields an empty map (warned), which
|
|
11210
|
+
* the selection degrades to `unknown` across the board — the caller runs
|
|
11211
|
+
* those specs, so an unreadable hub costs runs, never a skipped regression.
|
|
11212
|
+
*/
|
|
11213
|
+
async function loadCoverageEdges(input) {
|
|
11214
|
+
const edges = /* @__PURE__ */ new Map();
|
|
11215
|
+
const freshAfter = Date.now() - EDGE_MAX_AGE_MS;
|
|
11216
|
+
const merge = (key, edge) => {
|
|
11217
|
+
if (edge.measuredAt < freshAfter) return;
|
|
11218
|
+
const existing = edges.get(key);
|
|
11219
|
+
if (!existing || edge.measuredAt > existing.measuredAt) edges.set(key, edge);
|
|
11220
|
+
};
|
|
11221
|
+
const results = await Promise.allSettled([collectStreamEdges(input, merge), collectReportEdges(input, merge)]);
|
|
11222
|
+
let skipped = 0;
|
|
11223
|
+
for (const result of results) if (result.status === "rejected") warn(`select-specs: could not read coverage measurements from the hub (${errMessage(result.reason)})`);
|
|
11224
|
+
else skipped += result.value;
|
|
11225
|
+
if (skipped > 0) warn(`select-specs: ${skipped} measured run(s) on the hub could not be read; their reach is treated as absent`);
|
|
11226
|
+
return edges;
|
|
11227
|
+
}
|
|
11228
|
+
/**
|
|
11229
|
+
* Edges from the coverage event stream. The plain read answers for the most
|
|
11230
|
+
* recently measured run and lists every run the stream retains; each older
|
|
11231
|
+
* run is then resolved individually. Returns how many runs could not be read.
|
|
11232
|
+
*/
|
|
11233
|
+
async function collectStreamEdges(input, merge) {
|
|
11234
|
+
const { hub, project } = input;
|
|
11235
|
+
const latest = await hub.getCoverage(project);
|
|
11236
|
+
ingestResolved(latest.resolved, merge);
|
|
11237
|
+
const older = latest.runIds.filter((runId) => runId !== latest.resolved?.runId);
|
|
11238
|
+
return (await Promise.allSettled(older.map(async (runId) => ingestResolved((await hub.getCoverage(project, { runId })).resolved, merge)))).filter((r) => r.status === "rejected").length;
|
|
11239
|
+
}
|
|
11240
|
+
function ingestResolved(resolved, merge) {
|
|
11241
|
+
if (!resolved) return;
|
|
11242
|
+
for (const spec of resolved.specs) {
|
|
11243
|
+
const key = stripRunIdPrefix(spec.specId, resolved.runId);
|
|
11244
|
+
if (key === null) continue;
|
|
11245
|
+
if (spec.files.length === 0) continue;
|
|
11246
|
+
merge(key, {
|
|
11247
|
+
files: new Set(spec.files),
|
|
11248
|
+
measuredAt: resolved.asOf
|
|
11249
|
+
});
|
|
11250
|
+
}
|
|
11251
|
+
}
|
|
11252
|
+
/**
|
|
11253
|
+
* A stream specId is `<runId>.<feature>/<spec>` (src/coverage/session.ts).
|
|
11254
|
+
* The runId itself may contain `.`, so the known prefix is stripped by
|
|
11255
|
+
* length, never by splitting on the dot.
|
|
11256
|
+
*/
|
|
11257
|
+
function stripRunIdPrefix(specId, runId) {
|
|
11258
|
+
return specId.startsWith(`${runId}.`) ? specId.slice(runId.length + 1) : null;
|
|
11259
|
+
}
|
|
11260
|
+
/**
|
|
11261
|
+
* The one slice of report.json this consumer reads. Parsed with its own
|
|
11262
|
+
* narrow schema rather than the full report schema so a report from another
|
|
11263
|
+
* ccqa version still yields its edges as long as this shape holds.
|
|
11264
|
+
*/
|
|
11265
|
+
const ReportCoverageRowsSchema = z.object({ results: z.array(z.object({
|
|
11266
|
+
feature: z.string(),
|
|
11267
|
+
spec: z.string(),
|
|
11268
|
+
coverage: z.object({ files: z.array(z.string()) }).optional()
|
|
11269
|
+
})) });
|
|
11270
|
+
/**
|
|
11271
|
+
* Edges from pushed run reports, newest first. Only `kind: run` runs are
|
|
11272
|
+
* probed — audits and recordings execute no specs, so they carry no reach —
|
|
11273
|
+
* and a still-`running` run is skipped: its rows are still arriving, so its
|
|
11274
|
+
* measurement is not settled. Returns how many reports could not be read.
|
|
11275
|
+
*/
|
|
11276
|
+
async function collectReportEdges(input, merge) {
|
|
11277
|
+
const { hub, project } = input;
|
|
11278
|
+
const eligible = (await hub.listRuns({
|
|
11279
|
+
project,
|
|
11280
|
+
kind: "run",
|
|
11281
|
+
limit: MAX_REPORT_RUNS
|
|
11282
|
+
})).flatMap((run) => {
|
|
11283
|
+
if (run.status === "running") return [];
|
|
11284
|
+
const measuredAt = Date.parse(run.createdAt);
|
|
11285
|
+
if (Number.isNaN(measuredAt)) return [];
|
|
11286
|
+
return [{
|
|
11287
|
+
id: run.id,
|
|
11288
|
+
measuredAt
|
|
11289
|
+
}];
|
|
11290
|
+
});
|
|
11291
|
+
return (await Promise.allSettled(eligible.map(async ({ id, measuredAt }) => {
|
|
11292
|
+
const parsed = ReportCoverageRowsSchema.safeParse(await hub.getReport(id));
|
|
11293
|
+
if (!parsed.success) return;
|
|
11294
|
+
for (const row of parsed.data.results) {
|
|
11295
|
+
if (!row.coverage || row.coverage.files.length === 0) continue;
|
|
11296
|
+
merge(`${row.feature}/${row.spec}`, {
|
|
11297
|
+
files: new Set(row.coverage.files),
|
|
11298
|
+
measuredAt
|
|
11299
|
+
});
|
|
11300
|
+
}
|
|
11301
|
+
}))).filter((r) => r.status === "rejected").length;
|
|
11194
11302
|
}
|
|
11195
11303
|
//#endregion
|
|
11196
11304
|
//#region src/select/inventory.ts
|
|
@@ -11243,6 +11351,68 @@ function describeStep(step) {
|
|
|
11243
11351
|
function oneLine$1(text) {
|
|
11244
11352
|
return text.trim().replace(/\s+/g, " ");
|
|
11245
11353
|
}
|
|
11354
|
+
function emptyDeployLog() {
|
|
11355
|
+
return {
|
|
11356
|
+
nextIndex: 0,
|
|
11357
|
+
entries: []
|
|
11358
|
+
};
|
|
11359
|
+
}
|
|
11360
|
+
/** Append `input` to `current`; the appended entry is always the last of `entries`. */
|
|
11361
|
+
function appendDeploy(current, input) {
|
|
11362
|
+
const log = current ?? emptyDeployLog();
|
|
11363
|
+
const head = log.entries[log.entries.length - 1];
|
|
11364
|
+
const gapBefore = head ? head.sha !== input.previousSha : log.nextIndex > 0;
|
|
11365
|
+
const entries = [...log.entries, {
|
|
11366
|
+
...input,
|
|
11367
|
+
index: log.nextIndex,
|
|
11368
|
+
changedPaths: input.changedPaths === null ? null : input.changedPaths.slice(0, 500),
|
|
11369
|
+
gapBefore
|
|
11370
|
+
}];
|
|
11371
|
+
if (entries.length > 200) {
|
|
11372
|
+
entries.splice(0, entries.length - 200);
|
|
11373
|
+
entries[0] = {
|
|
11374
|
+
...entries[0],
|
|
11375
|
+
gapBefore: true
|
|
11376
|
+
};
|
|
11377
|
+
}
|
|
11378
|
+
return {
|
|
11379
|
+
nextIndex: log.nextIndex + 1,
|
|
11380
|
+
entries
|
|
11381
|
+
};
|
|
11382
|
+
}
|
|
11383
|
+
/**
|
|
11384
|
+
* Fold one deploy's selection into the touch index.
|
|
11385
|
+
*
|
|
11386
|
+
* Only `needed` moves a position a verdict reads. An `unknown` records the
|
|
11387
|
+
* newest undecided position too, but that one is record-only (ADR-0023) —
|
|
11388
|
+
* freshness reads an undecided judgment as "did not reach". A `notNeeded`
|
|
11389
|
+
* writes neither — it is the absence of a marker at this position, which is
|
|
11390
|
+
* exactly what a later baseline comparison reads it as.
|
|
11391
|
+
*
|
|
11392
|
+
* Positions only ever advance. Deploys are folded in log order, so a spec
|
|
11393
|
+
* needed at #7 and cleared at #9 keeps `needed.index: 7`: a baseline at #5
|
|
11394
|
+
* must still see that #7 touched it.
|
|
11395
|
+
*/
|
|
11396
|
+
function foldTouchIndex(current, entry, selection) {
|
|
11397
|
+
const out = { ...current };
|
|
11398
|
+
for (const [key, decision] of Object.entries(selection)) {
|
|
11399
|
+
const previous = out[key] ?? {};
|
|
11400
|
+
if (decision.verdict === "needed") out[key] = {
|
|
11401
|
+
...previous,
|
|
11402
|
+
needed: {
|
|
11403
|
+
index: entry.index,
|
|
11404
|
+
sha: entry.sha,
|
|
11405
|
+
at: entry.at,
|
|
11406
|
+
matchedPaths: (decision.touchedBy ?? []).slice(0, 10)
|
|
11407
|
+
}
|
|
11408
|
+
};
|
|
11409
|
+
else if (decision.verdict === "unknown") out[key] = {
|
|
11410
|
+
...previous,
|
|
11411
|
+
undecidedIndex: entry.index
|
|
11412
|
+
};
|
|
11413
|
+
}
|
|
11414
|
+
return out;
|
|
11415
|
+
}
|
|
11246
11416
|
//#endregion
|
|
11247
11417
|
//#region src/cli/session.ts
|
|
11248
11418
|
const AB = resolveAgentBrowserBin$1();
|
|
@@ -11530,8 +11700,8 @@ const promptRm = new Command("rm").description("Delete a prompt from the hub.").
|
|
|
11530
11700
|
info(`deleted prompt "${name}" from the hub`);
|
|
11531
11701
|
}));
|
|
11532
11702
|
const promptCommand = new Command("prompt").description("Manage prompt assets (per-flow user/agent guidance, triage/audit user guidance, learned calibration prompts) stored on the hub (fetched automatically by `ccqa run` / `ccqa audit` at run time).").addCommand(promptPush).addCommand(promptLs).addCommand(promptRm);
|
|
11533
|
-
const deployRecord = new Command("record").description("Tell the hub what a deploy shipped, so it can answer which specs need a re-run (`ccqa run --only-hub-rerun-needed`). Run this from the deploy job, after the deploy succeeds. The changed paths are computed locally with a two-dot `git diff <previous> <sha>`; a job that has only curl and git can POST the same body directly (see docs/hub.md).").requiredOption("--profile <name>", "Environment this deploy landed in (e.g. 'stg'). Required: dev and stg sit at different commits, so the deploy log is per-profile.").requiredOption("--sha <sha>", "Commit that was deployed.").option("--previous <sha>", "Commit this deploy replaced. Omit it and the hub's current log head is used — the normal case, recording no discontinuity. Pass a sha that differs from the head and the hub records one (gapBefore) in the chain: use this for a first record with a real baseline, or to re-anchor a head that no longer matches reality. With no head and nothing passed, there's nothing to diff against: changedPaths is unset and the spec selection is skipped.").option("--ref <ref>", "Ref that was deployed (branch or tag). Recorded for display only.").option("--no-select-specs", "Record the deploy without deciding which specs it reaches. The entry then becomes a hole in the range — every spec behind it is assumed reached rather than being cleared, and nothing can fill it in later, since the hub has no checkout to diff.
|
|
11534
|
-
await
|
|
11703
|
+
const deployRecord = new Command("record").description("Tell the hub what a deploy shipped, so it can answer which specs need a re-run (`ccqa run --only-hub-rerun-needed`). Run this from the deploy job, after the deploy succeeds. The changed paths are computed locally with a two-dot `git diff <previous> <sha>`; a job that has only curl and git can POST the same body directly (see docs/hub.md).").requiredOption("--profile <name>", "Environment this deploy landed in (e.g. 'stg'). Required: dev and stg sit at different commits, so the deploy log is per-profile.").requiredOption("--sha <sha>", "Commit that was deployed.").option("--previous <sha>", "Commit this deploy replaced. Omit it and the hub's current log head is used — the normal case, recording no discontinuity. Pass a sha that differs from the head and the hub records one (gapBefore) in the chain: use this for a first record with a real baseline, or to re-anchor a head that no longer matches reality. With no head and nothing passed, there's nothing to diff against: changedPaths is unset and the spec selection is skipped.").option("--ref <ref>", "Ref that was deployed (branch or tag). Recorded for display only.").option("--no-select-specs", "Record the deploy without deciding which specs it reaches. The entry then becomes a hole in the range — every spec behind it is assumed reached rather than being cleared, and nothing can fill it in later, since the hub has no checkout to diff. The decision intersects the diff with measured coverage from the hub and calls no model, so there is rarely a reason to skip it.").option(...hubUrlOption).option(...hubTokenOption).option("--project <name>", "Project whose deploy log this entry joins. Defaults to the current directory's name.").option("--cwd <path>", "Directory the git diff and the default --project name are resolved against.").action(withHubErrors(async (opts) => {
|
|
11704
|
+
await runDeployRecord(opts);
|
|
11535
11705
|
}));
|
|
11536
11706
|
async function runDeployRecord(opts) {
|
|
11537
11707
|
const cwd = resolveCwd(opts.cwd);
|
|
@@ -11541,7 +11711,7 @@ async function runDeployRecord(opts) {
|
|
|
11541
11711
|
const runUrl = githubRunUrl();
|
|
11542
11712
|
const diff = previous === null ? null : await diffOrNull(previous, opts.sha, cwd);
|
|
11543
11713
|
const changedPaths = diff ? capDeployPaths(diff.map((f) => f.path)) : null;
|
|
11544
|
-
const selection = opts.selectSpecs !== false && previous !== null && diff !== null ? await selectionForDeploy(diff, previous, opts.sha, cwd
|
|
11714
|
+
const selection = opts.selectSpecs !== false && previous !== null && diff !== null ? await selectionForDeploy(hub, project, diff, previous, opts.sha, cwd) : void 0;
|
|
11545
11715
|
const entry = await hub.recordDeploy(project, opts.profile, {
|
|
11546
11716
|
sha: opts.sha,
|
|
11547
11717
|
previousSha: previous,
|
|
@@ -11569,11 +11739,13 @@ async function runDeployRecord(opts) {
|
|
|
11569
11739
|
* Takes the diff `deployRecord` already fetched for `changedPaths`, rather
|
|
11570
11740
|
* than diffing again — the decision needs the diff and the spec tree, and the
|
|
11571
11741
|
* hub has neither, but there's no reason to ask git for the same range twice.
|
|
11742
|
+
* The hub does hold the coverage measurements the verdicts rest on, so those
|
|
11743
|
+
* are read back through the same connection the entry is posted over.
|
|
11572
11744
|
* `undefined` on failure rather than a half-answer: the deploy is then
|
|
11573
11745
|
* recorded without a selection, and specs behind it read `unknown` instead of
|
|
11574
11746
|
* being cleared by a selection that isn't there.
|
|
11575
11747
|
*/
|
|
11576
|
-
async function selectionForDeploy(changed, previous, sha, cwd
|
|
11748
|
+
async function selectionForDeploy(hub, project, changed, previous, sha, cwd) {
|
|
11577
11749
|
try {
|
|
11578
11750
|
const specs = await loadSpecInventory(cwd);
|
|
11579
11751
|
if (specs.length === 0) return void 0;
|
|
@@ -11583,12 +11755,15 @@ async function selectionForDeploy(changed, previous, sha, cwd, model) {
|
|
|
11583
11755
|
cwd,
|
|
11584
11756
|
base: previous,
|
|
11585
11757
|
head: sha,
|
|
11586
|
-
|
|
11758
|
+
edges: await loadCoverageEdges({
|
|
11759
|
+
hub,
|
|
11760
|
+
project
|
|
11761
|
+
})
|
|
11587
11762
|
});
|
|
11588
11763
|
return Object.fromEntries(report.specs.map((s) => [specKey(s), {
|
|
11589
11764
|
verdict: s.verdict,
|
|
11590
11765
|
reason: s.reason,
|
|
11591
|
-
...s.touchedBy?.length ? { touchedBy: s.touchedBy } : {}
|
|
11766
|
+
...s.touchedBy?.length ? { touchedBy: s.touchedBy.slice(0, 10) } : {}
|
|
11592
11767
|
}]));
|
|
11593
11768
|
} catch (err) {
|
|
11594
11769
|
warn(`could not decide which specs this deploy reaches (${errMessage(err)}); recording the deploy without a selection`);
|
|
@@ -12851,155 +13026,6 @@ function oneLine(s) {
|
|
|
12851
13026
|
return s.replace(/\s+/g, " ").trim();
|
|
12852
13027
|
}
|
|
12853
13028
|
//#endregion
|
|
12854
|
-
//#region src/config/project-config.ts
|
|
12855
|
-
/**
|
|
12856
|
-
* Loader for the consumer project's `.ccqa/config.yaml` — per-target
|
|
12857
|
-
* generation settings (default target, output dirs, reusable code resources,
|
|
12858
|
-
* generation conventions).
|
|
12859
|
-
*
|
|
12860
|
-
* This module only validates and holds the config. `path` / `guides` /
|
|
12861
|
-
* `examples` entries may be glob patterns; they are kept verbatim here and
|
|
12862
|
-
* expanded by the generation engine, which owns size limits and warnings.
|
|
12863
|
-
*/
|
|
12864
|
-
/**
|
|
12865
|
-
* An existing code asset the generated tests should reuse (import), in one of
|
|
12866
|
-
* two forms — exactly one of:
|
|
12867
|
-
* - `path`: code inside the consumer repo (literal path or glob pattern);
|
|
12868
|
-
* - `package`: an installed npm package (imported by name).
|
|
12869
|
-
* `description` tells the generator what the asset contains.
|
|
12870
|
-
*/
|
|
12871
|
-
const ResourceRefSchema = z.union([z.object({
|
|
12872
|
-
path: z.string().min(1),
|
|
12873
|
-
description: z.string().optional()
|
|
12874
|
-
}).strict(), z.object({
|
|
12875
|
-
package: z.string().min(1),
|
|
12876
|
-
description: z.string().optional()
|
|
12877
|
-
}).strict()], { error: "a resource must have exactly one of `path` (code in this repo) or `package` (installed npm package), plus an optional `description`" });
|
|
12878
|
-
/**
|
|
12879
|
-
* How generated code should be written, as guide inputs to the prompt (never
|
|
12880
|
-
* imported as code): `guides` are convention documents, `examples` are
|
|
12881
|
-
* existing tests whose style to imitate. Entries may be glob patterns.
|
|
12882
|
-
*/
|
|
12883
|
-
const ConventionsSchema = z.object({
|
|
12884
|
-
guides: z.array(z.string().min(1)).default([]),
|
|
12885
|
-
examples: z.array(z.string().min(1)).default([])
|
|
12886
|
-
}).strict();
|
|
12887
|
-
/**
|
|
12888
|
-
* Per-target settings. `outDir` (where generated tests are written) and
|
|
12889
|
-
* `runCommand` (how to execute them; `{files}` expands to the generated
|
|
12890
|
-
* paths, `{artifactsDir}` to the spec's report artifacts dir — see
|
|
12891
|
-
* src/targets/run-artifacts.ts) are optional at this layer because not every
|
|
12892
|
-
* target needs them — e.g. agent-browser stores its output in the spec
|
|
12893
|
-
* directory. A target that requires either must validate its presence itself.
|
|
12894
|
-
*/
|
|
12895
|
-
const TargetConfigSchema = z.object({
|
|
12896
|
-
outDir: z.string().min(1).optional(),
|
|
12897
|
-
runCommand: z.string().min(1).optional(),
|
|
12898
|
-
resources: z.array(ResourceRefSchema).default([]),
|
|
12899
|
-
conventions: ConventionsSchema.default({
|
|
12900
|
-
guides: [],
|
|
12901
|
-
examples: []
|
|
12902
|
-
})
|
|
12903
|
-
}).strict();
|
|
12904
|
-
/**
|
|
12905
|
-
* Specs that must not run at the same time, grouped by the thing they share.
|
|
12906
|
-
*
|
|
12907
|
-
* The key names the shared thing (a chat channel, a seeded account, a tenant);
|
|
12908
|
-
* the list names the specs that write to it. `ccqa run` never runs two members
|
|
12909
|
-
* of one group concurrently, and specs sharing no group still run in parallel.
|
|
12910
|
-
*
|
|
12911
|
-
* Kept here rather than on each spec so there is one place to read the whole
|
|
12912
|
-
* picture, and so a mistyped member is a spec key that does not resolve —
|
|
12913
|
-
* caught — rather than a resource name that silently matches nothing.
|
|
12914
|
-
*/
|
|
12915
|
-
const SerialGroupsSchema = z.record(z.string().regex(/^[a-z0-9][a-z0-9._-]*$/i, "serial group name must be a slug (letters, digits, '.', '_', '-')"), z.array(z.string().min(1)).min(1));
|
|
12916
|
-
/**
|
|
12917
|
-
* Which specs act as which external identity, for the flows whose requests
|
|
12918
|
-
* cannot carry a spec id at all.
|
|
12919
|
-
*
|
|
12920
|
-
* A chat platform's webhook is sent by the platform, not the browser, so no
|
|
12921
|
-
* cookie rides along and everything the flow reaches would be unattributed.
|
|
12922
|
-
* What the request does carry is who caused it, and if only one spec is allowed
|
|
12923
|
-
* to act as that identity at a time, "who" plus "when" is enough.
|
|
12924
|
-
*
|
|
12925
|
-
* ```yaml
|
|
12926
|
-
* coverage:
|
|
12927
|
-
* actors:
|
|
12928
|
-
* slack: # the preset's tag prefix
|
|
12929
|
-
* ${TEST_USER_ID}: [chat/create-item, chat/resolve-item]
|
|
12930
|
-
* ```
|
|
12931
|
-
*
|
|
12932
|
-
* The provider name is the prefix the matching preset stamps, and the key is an
|
|
12933
|
-
* identity expression the run's variables resolve. Only the unexpanded text is
|
|
12934
|
-
* ever displayed or used as a lock key, so the identity itself stays out of
|
|
12935
|
-
* reports and the hub.
|
|
12936
|
-
*/
|
|
12937
|
-
const CoverageActorsSchema = z.record(z.string().regex(/^[a-z0-9][a-z0-9._-]*$/i, "actor provider must be a slug (letters, digits, '.', '_', '-')"), z.record(z.string().min(1), z.array(z.string().min(1)).min(1)));
|
|
12938
|
-
/**
|
|
12939
|
-
* Settings for `ccqa run --coverage`, which measures what each spec actually
|
|
12940
|
-
* reached in the application under test.
|
|
12941
|
-
*/
|
|
12942
|
-
const CoverageConfigSchema = z.object({
|
|
12943
|
-
instrumentedOrigins: z.array(z.string().min(1)).min(1),
|
|
12944
|
-
sink: z.string().min(1).default("http://127.0.0.1:4757"),
|
|
12945
|
-
projectRoot: z.string().min(1).optional(),
|
|
12946
|
-
include: z.array(z.string().min(1)).optional(),
|
|
12947
|
-
actors: CoverageActorsSchema.default({})
|
|
12948
|
-
}).strict();
|
|
12949
|
-
/**
|
|
12950
|
-
* Top-level `.ccqa/config.yaml` schema. `defaultTarget` is used by specs
|
|
12951
|
-
* with no `target:` of their own. Both defaults make a missing config file
|
|
12952
|
-
* equivalent to "agent-browser only, no extra settings".
|
|
12953
|
-
*/
|
|
12954
|
-
const ProjectConfigSchema = z.object({
|
|
12955
|
-
defaultTarget: TargetIdSchema.default(AGENT_BROWSER_TARGET),
|
|
12956
|
-
targets: z.record(TargetIdSchema, TargetConfigSchema).default({}),
|
|
12957
|
-
serialGroups: SerialGroupsSchema.default({}),
|
|
12958
|
-
coverage: CoverageConfigSchema.optional()
|
|
12959
|
-
}).strict();
|
|
12960
|
-
/** Config file location, relative to the project root (`--cwd`). */
|
|
12961
|
-
const PROJECT_CONFIG_PATH = ".ccqa/config.yaml";
|
|
12962
|
-
/**
|
|
12963
|
-
* Load `<cwd>/.ccqa/config.yaml`. A missing file yields the defaults (an
|
|
12964
|
-
* empty file too); a present but broken file is an error — never silently
|
|
12965
|
-
* fall back when the user wrote a config.
|
|
12966
|
-
*/
|
|
12967
|
-
async function loadProjectConfig(cwd) {
|
|
12968
|
-
let content;
|
|
12969
|
-
try {
|
|
12970
|
-
content = await readFile(join(cwd, PROJECT_CONFIG_PATH), "utf8");
|
|
12971
|
-
} catch (e) {
|
|
12972
|
-
if (e.code === "ENOENT") return ProjectConfigSchema.parse({});
|
|
12973
|
-
throw e;
|
|
12974
|
-
}
|
|
12975
|
-
return parseProjectConfig(content);
|
|
12976
|
-
}
|
|
12977
|
-
/** Parse config YAML. Schema rejections are rewritten with actionable messages. */
|
|
12978
|
-
function parseProjectConfig(content, source = PROJECT_CONFIG_PATH) {
|
|
12979
|
-
let raw;
|
|
12980
|
-
try {
|
|
12981
|
-
raw = parse(content);
|
|
12982
|
-
} catch (e) {
|
|
12983
|
-
throw new Error(`Failed to parse YAML (${source}): ${e.message}`);
|
|
12984
|
-
}
|
|
12985
|
-
try {
|
|
12986
|
-
return ProjectConfigSchema.parse(raw ?? {});
|
|
12987
|
-
} catch (e) {
|
|
12988
|
-
throw enrichZodError(e, source);
|
|
12989
|
-
}
|
|
12990
|
-
}
|
|
12991
|
-
/** Flatten a ZodError into one `Invalid <source>:` message, path per line. */
|
|
12992
|
-
function enrichZodError(error, source) {
|
|
12993
|
-
if (!(error instanceof ZodError)) return error;
|
|
12994
|
-
const lines = [`Invalid ${source}:`];
|
|
12995
|
-
for (const issue of error.issues) {
|
|
12996
|
-
const path = issue.path.join(".") || "(root)";
|
|
12997
|
-
const message = issue.code === "invalid_key" && issue.issues[0] ? issue.issues[0].message : issue.message;
|
|
12998
|
-
lines.push(` - ${path}: ${message}`);
|
|
12999
|
-
}
|
|
13000
|
-
return new Error(lines.join("\n"));
|
|
13001
|
-
}
|
|
13002
|
-
//#endregion
|
|
13003
13029
|
//#region src/coverage/inbox.ts
|
|
13004
13030
|
/**
|
|
13005
13031
|
* The run's side of the hub coverage inbox (ADR-0022). Under
|
|
@@ -15289,28 +15315,58 @@ function stripCodeFences(text) {
|
|
|
15289
15315
|
return m && m[1] !== void 0 ? m[1] : text;
|
|
15290
15316
|
}
|
|
15291
15317
|
//#endregion
|
|
15318
|
+
//#region src/select/types.ts
|
|
15319
|
+
/**
|
|
15320
|
+
* How the verdict was reached. Kept because the two sources have different
|
|
15321
|
+
* trust: `mechanical` is set arithmetic on paths and cannot be wrong;
|
|
15322
|
+
* `coverage` intersects the diff with the spec's last measured reach
|
|
15323
|
+
* (ADR-0024), which can only be wrong through staleness — and staleness
|
|
15324
|
+
* degrades to `unknown`, never to a guess.
|
|
15325
|
+
*/
|
|
15326
|
+
const SelectSourceSchema = z.enum(["mechanical", "coverage"]);
|
|
15327
|
+
const SpecSelectionSchema = z.object({
|
|
15328
|
+
featureName: z.string().min(1),
|
|
15329
|
+
specName: z.string().min(1),
|
|
15330
|
+
verdict: SelectVerdictSchema,
|
|
15331
|
+
source: SelectSourceSchema,
|
|
15332
|
+
reason: z.string(),
|
|
15333
|
+
touchedBy: z.array(z.string()).optional()
|
|
15334
|
+
});
|
|
15335
|
+
z.object({
|
|
15336
|
+
base: z.string(),
|
|
15337
|
+
head: z.string(),
|
|
15338
|
+
changedFiles: z.number().int().nonnegative(),
|
|
15339
|
+
specs: z.array(SpecSelectionSchema)
|
|
15340
|
+
});
|
|
15341
|
+
/** Specs the caller should actually run: everything not positively cleared. */
|
|
15342
|
+
function specsToRun(report) {
|
|
15343
|
+
return report.specs.filter((s) => s.verdict !== "notNeeded");
|
|
15344
|
+
}
|
|
15345
|
+
//#endregion
|
|
15292
15346
|
//#region src/cli/changed-specs.ts
|
|
15293
15347
|
/**
|
|
15294
15348
|
* Filter specs to those a range of commits reaches. Powers `ccqa run
|
|
15295
15349
|
* --only-affected-by <ref>`; `ccqa audit` uses the same call.
|
|
15296
15350
|
*
|
|
15297
|
-
* The decision is made by `ccqa select-specs`, which
|
|
15298
|
-
*
|
|
15299
|
-
*
|
|
15351
|
+
* The decision is made by `ccqa select-specs`, which intersects the diff with
|
|
15352
|
+
* each spec's last measured reach from the hub (ADR-0024). Deterministic and
|
|
15353
|
+
* free — no model call — and wrong in only one direction: a spec without a
|
|
15354
|
+
* measurement comes back `unknown` and runs.
|
|
15300
15355
|
*
|
|
15301
15356
|
* Specs come back `needed`, `notNeeded` or `unknown`; everything but
|
|
15302
|
-
* `notNeeded` runs. `unknown` is the selector saying it
|
|
15303
|
-
* the safe reading of that is to run the spec.
|
|
15357
|
+
* `notNeeded` runs. `unknown` is the selector saying it has no measurement to
|
|
15358
|
+
* consult, and the safe reading of that is to run the spec.
|
|
15304
15359
|
*/
|
|
15305
15360
|
async function collectChangedSpecs(specs, opts) {
|
|
15306
|
-
const { cwd, base,
|
|
15307
|
-
const
|
|
15361
|
+
const { cwd, base, hub, quiet, flagName } = opts;
|
|
15362
|
+
const flag = flagName ?? "--only-affected-by";
|
|
15363
|
+
const resolved = await resolveAnalysisBase(base, flag, cwd);
|
|
15308
15364
|
const meta$1 = (key, value) => {
|
|
15309
15365
|
if (!quiet) meta(key, value);
|
|
15310
15366
|
};
|
|
15311
15367
|
let changed;
|
|
15312
15368
|
try {
|
|
15313
|
-
changed = await getChangedFilesBetween(resolved.sha, "HEAD", cwd);
|
|
15369
|
+
changed = await getChangedFilesBetween(resolved.sha, "HEAD", cwd, { detectRenames: false });
|
|
15314
15370
|
} catch (e) {
|
|
15315
15371
|
throw new RunUsageError(`failed to run 'git diff' against ${resolved.ref}: ${e.message}`);
|
|
15316
15372
|
}
|
|
@@ -15326,13 +15382,16 @@ async function collectChangedSpecs(specs, opts) {
|
|
|
15326
15382
|
} catch (e) {
|
|
15327
15383
|
throw new RunUsageError(e.message);
|
|
15328
15384
|
}
|
|
15385
|
+
let edges = /* @__PURE__ */ new Map();
|
|
15386
|
+
if (hub) edges = await loadCoverageEdges(hub);
|
|
15387
|
+
else warn(`${flag}: no hub connection, so coverage measurements cannot be consulted — undecided specs will run`);
|
|
15329
15388
|
const report = await selectSpecs({
|
|
15330
15389
|
changed,
|
|
15331
15390
|
specs: inventory,
|
|
15332
15391
|
cwd,
|
|
15333
15392
|
base: resolved.sha,
|
|
15334
15393
|
head: "HEAD",
|
|
15335
|
-
|
|
15394
|
+
edges
|
|
15336
15395
|
});
|
|
15337
15396
|
const toRun = new Set(specsToRun(report).map(specKey));
|
|
15338
15397
|
const undecided = report.specs.filter((s) => s.verdict === "unknown").length;
|
|
@@ -15588,7 +15647,7 @@ async function executeRun(targets, opts) {
|
|
|
15588
15647
|
if (opts.onlyAffectedBy) specs = (await collectChangedSpecs(specs, {
|
|
15589
15648
|
cwd,
|
|
15590
15649
|
base: opts.onlyAffectedBy,
|
|
15591
|
-
|
|
15650
|
+
hub: hubCtx
|
|
15592
15651
|
})).specs;
|
|
15593
15652
|
meta("selected", `${specs.length} of ${before} spec${before === 1 ? "" : "s"}`);
|
|
15594
15653
|
if (specs.length === 0 && inProgress > 0) {
|
|
@@ -16501,7 +16560,7 @@ function installTeardownSignalHandlers(teardown, onSignal) {
|
|
|
16501
16560
|
}
|
|
16502
16561
|
//#endregion
|
|
16503
16562
|
//#region src/cli/run.ts
|
|
16504
|
-
const runCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs, on any target. Agent-browser specs replay the recorded test.spec.ts under vitest (default), or, with spec.yaml `mode: live`, have Claude drive agent-browser live per step. External-target specs (playwright, runn) run through the target's configured `runCommand`. A structured report (report.json + evidence) is always written; use --report-to-hub to also stream it to a hub.").optionsGroup("Which specs to run:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs`
|
|
16563
|
+
const runCommand = addHubOptions(addProfileOption(addLanguageOption(new Command("run").argument("[targets...]", "Specs to run, space-separated: each '<feature>/<spec>', '<feature>', or omit for all. Duplicates are de-duped.").description("Run specs, on any target. Agent-browser specs replay the recorded test.spec.ts under vitest (default), or, with spec.yaml `mode: live`, have Claude drive agent-browser live per step. External-target specs (playwright, runn) run through the target's configured `runCommand`. A structured report (report.json + evidence) is always written; use --report-to-hub to also stream it to a hub.").optionsGroup("Which specs to run:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` decides the diff against <ref> reaches (e.g. origin/main), by intersecting it with each spec's measured coverage from the hub. In pull_request CI, pass $GITHUB_BASE_REF. Cannot be combined with an explicit spec id.").option("--only-hub-rerun-needed", "Only specs the hub answers `rerunNeeded` for: the audit cleared them, and their last result does not cover what is deployed — including every spec the deploy log cannot place, which is assumed reached rather than skipped. A spec whose audit has not caught up answers `inProgress`, and one the audit rejected or whose last run failed answers `needsRepair`; neither is taken, because running them races the audit or repairs nothing. No git diff involved. Requires a hub connection and --hub-profile.").option("--dry-run", "Print the specs this invocation would run, then exit 0 without executing anything and without writing a report. Works with every selection flag.").optionsGroup("How to run them:").option("--concurrency <n>", "Run up to N specs in parallel within each phase (deterministic / external-target / live), never across phases. Default 1 (sequential). Specs in the same `serialGroups` entry of .ccqa/config.yaml still take turns. Live specs each get an isolated agent-browser session; high values spawn many headed Chrome instances.", parseConcurrency$1, 1).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").option("--live-step-retry <n>", "(live only) Retry each failed step up to N more times before recording failure. This retries a step, not the whole spec — see --on-fail-explain-rerun for that.", (raw) => {
|
|
16505
16564
|
const n = Number(raw);
|
|
16506
16565
|
if (!Number.isFinite(n) || n < 0 || Math.floor(n) !== n) throw new Error(`--live-step-retry must be a non-negative integer, got "${raw}"`);
|
|
16507
16566
|
return n;
|
|
@@ -19041,8 +19100,7 @@ function freshness(baselineSha, key, range) {
|
|
|
19041
19100
|
const baselinePos = positionBySha.get(baselineSha);
|
|
19042
19101
|
if (baselinePos === void 0) return unanswerable("deployedShaNotInLog");
|
|
19043
19102
|
const baselineIndex = log.entries[baselinePos].index;
|
|
19044
|
-
const
|
|
19045
|
-
const touched = touch?.needed;
|
|
19103
|
+
const touched = touchIndex[key]?.needed;
|
|
19046
19104
|
if (touched && touched.index > baselineIndex) {
|
|
19047
19105
|
const entry = range.entryByIndex.get(touched.index);
|
|
19048
19106
|
return {
|
|
@@ -19053,7 +19111,6 @@ function freshness(baselineSha, key, range) {
|
|
|
19053
19111
|
}
|
|
19054
19112
|
if (range.gapFromPos[baselinePos + 1]) return unanswerable("gapInRange");
|
|
19055
19113
|
if (range.noSelectionFromPos[baselinePos + 1]) return unanswerable("noSelectionInRange");
|
|
19056
|
-
if (touch?.undecidedIndex !== void 0 && touch.undecidedIndex > baselineIndex) return unanswerable("selectionUnknown");
|
|
19057
19114
|
return { kind: "current" };
|
|
19058
19115
|
}
|
|
19059
19116
|
function deployRef(entry) {
|
|
@@ -19220,7 +19277,7 @@ function selectSpecsNeedingAudit(targets, report, stillDrifted = /* @__PURE__ */
|
|
|
19220
19277
|
//#endregion
|
|
19221
19278
|
//#region src/cli/audit.ts
|
|
19222
19279
|
const DEFAULT_CONCURRENCY = 3;
|
|
19223
|
-
const auditCommand = addProfileOption(addLanguageOption(new Command("audit").argument("[feature/spec]", "Optional spec id. If omitted, every spec under .ccqa/features/ is checked.").description("Read each spec against the code it describes and report where the two have drifted. Static: no browser is run, so this is the cheap check to put in front of `ccqa run`.").optionsGroup("Which specs to audit:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs`
|
|
19280
|
+
const auditCommand = addProfileOption(addLanguageOption(new Command("audit").argument("[feature/spec]", "Optional spec id. If omitted, every spec under .ccqa/features/ is checked.").description("Read each spec against the code it describes and report where the two have drifted. Static: no browser is run, so this is the cheap check to put in front of `ccqa run`.").optionsGroup("Which specs to audit:").option("--only-affected-by <ref>", "Only specs `ccqa select-specs` decides the diff against <ref> reaches (e.g. origin/main), by intersecting it with each spec's measured coverage from the hub. In pull_request CI, pass $GITHUB_BASE_REF. Specs without a measurement are audited rather than skipped.").option("--only-hub-audit-needed", "Only specs the hub says a deploy has landed on since the audit last read them. A spec that was never audited is always included, one the hub cannot answer for is audited rather than skipped, and one whose drift entry is still open is always re-audited — a merged fix changes only the spec tree, which no deploy answer covers. No git diff involved. Requires a hub connection and --hub-profile.").optionsGroup("How to run it:").option("--concurrency <n>", `Parallel spec checks (default: ${DEFAULT_CONCURRENCY})`).option("-m, --model <name>", "Claude model alias ('sonnet'|'opus'|'haiku') or full ID. Overrides CCQA_MODEL.").optionsGroup("What to do with the results:").option("--report-format <fmt>", "Output format: text | json | github", "text").option("--report-to-hub", "Push the result to a ccqa hub as a run (kind: drift), which is what updates the drift ledger. A spec it finds drifted answers `needsRepair` to `ccqa run --only-hub-rerun-needed`, and is not run until a person repairs it.").option("--exit-on <level>", "Exit non-zero on this severity or higher: warn | error", "error").optionsGroup("Environment and connection:").option("--cwd <path>", "Working directory used as both the .ccqa root and the codebase Claude reads. Useful for monorepos. Defaults to process.cwd().").option("--project <name>", "Logical project name for the pushed run. Defaults to the current directory's name.").option(...hubUrlOption).option(...hubTokenOption).option(...hubHeaderOption))).action(withUsageErrors(async (specPath, opts) => {
|
|
19224
19281
|
await withCostReporting("audit", () => runAudit(specPath, opts));
|
|
19225
19282
|
}));
|
|
19226
19283
|
async function runAudit(specPath, opts) {
|
|
@@ -19289,7 +19346,7 @@ async function runAudit(specPath, opts) {
|
|
|
19289
19346
|
cwd,
|
|
19290
19347
|
base: opts.onlyAffectedBy,
|
|
19291
19348
|
quiet: format !== "text",
|
|
19292
|
-
|
|
19349
|
+
hub: resolveAuditHubContext(opts, cwd)
|
|
19293
19350
|
});
|
|
19294
19351
|
targets = selection.specs;
|
|
19295
19352
|
baseRef = selection.base.ref;
|
|
@@ -20062,26 +20119,40 @@ function parseSummaries(json) {
|
|
|
20062
20119
|
//#endregion
|
|
20063
20120
|
//#region src/cli/select-specs.ts
|
|
20064
20121
|
const DEFAULT_HEAD = "HEAD";
|
|
20065
|
-
const selectSpecsCommand = new Command("select-specs").description("Decide which specs a range of commits reaches.
|
|
20066
|
-
await withCostReporting("select-specs", () => runSelectSpecs(opts));
|
|
20067
|
-
});
|
|
20122
|
+
const selectSpecsCommand = new Command("select-specs").description("Decide which specs a range of commits reaches. Intersects the diff with each spec's last measured reach from the hub (`ccqa run --coverage`) and returns one verdict per spec: needed | notNeeded | unknown. Requires a hub connection; a spec with no measurement is unknown, which runs.").requiredOption("--base <ref>", "Commit the range starts at — typically what is currently deployed, or the previous commit on the branch.").option("--head <ref>", `Commit the range ends at (default: ${DEFAULT_HEAD})`).option("--cwd <path>", "Working directory used as the .ccqa root. Changes outside it are reported but never attributed to a spec. Defaults to process.cwd().").option("--project <name>", "Project whose coverage measurements are read from the hub. Defaults to the current directory's name.").option(...hubUrlOption).option(...hubTokenOption).option(...hubHeaderOption).option("--format <fmt>", "Output format: text | json", "text").action(runSelectSpecs);
|
|
20068
20123
|
async function runSelectSpecs(opts) {
|
|
20069
20124
|
const format = parseFormat(opts.format);
|
|
20070
20125
|
const cwd = resolveCwd(opts.cwd);
|
|
20071
20126
|
const head = opts.head ?? DEFAULT_HEAD;
|
|
20072
|
-
const
|
|
20073
|
-
|
|
20074
|
-
specs
|
|
20075
|
-
|
|
20076
|
-
|
|
20077
|
-
|
|
20078
|
-
|
|
20079
|
-
|
|
20080
|
-
|
|
20081
|
-
|
|
20082
|
-
|
|
20083
|
-
|
|
20084
|
-
|
|
20127
|
+
const hub = resolveHubClient(opts);
|
|
20128
|
+
if (!hub) {
|
|
20129
|
+
error(needsHubConnection("select-specs"));
|
|
20130
|
+
process.exit(2);
|
|
20131
|
+
}
|
|
20132
|
+
const project = resolveProject({
|
|
20133
|
+
project: opts.project,
|
|
20134
|
+
cwd: opts.cwd
|
|
20135
|
+
});
|
|
20136
|
+
const [specsResult, changedResult, edges] = await Promise.all([
|
|
20137
|
+
loadSpecInventory(cwd).then((specs) => ({
|
|
20138
|
+
ok: true,
|
|
20139
|
+
specs
|
|
20140
|
+
}), (e) => ({
|
|
20141
|
+
ok: false,
|
|
20142
|
+
error: e
|
|
20143
|
+
})),
|
|
20144
|
+
getChangedFilesBetween(opts.base, head, cwd, { detectRenames: false }).then((changed) => ({
|
|
20145
|
+
ok: true,
|
|
20146
|
+
changed
|
|
20147
|
+
}), (e) => ({
|
|
20148
|
+
ok: false,
|
|
20149
|
+
error: e
|
|
20150
|
+
})),
|
|
20151
|
+
loadCoverageEdges({
|
|
20152
|
+
hub,
|
|
20153
|
+
project
|
|
20154
|
+
})
|
|
20155
|
+
]);
|
|
20085
20156
|
if (!specsResult.ok) {
|
|
20086
20157
|
error(specsResult.error.message);
|
|
20087
20158
|
process.exit(1);
|
|
@@ -20099,8 +20170,10 @@ async function runSelectSpecs(opts) {
|
|
|
20099
20170
|
if (format === "text") {
|
|
20100
20171
|
header("select-specs", `${opts.base} → ${head}`);
|
|
20101
20172
|
if (opts.cwd) meta("cwd", cwd);
|
|
20173
|
+
meta("project", project);
|
|
20102
20174
|
meta("changed-files", changed.length);
|
|
20103
20175
|
meta("specs", specs.length);
|
|
20176
|
+
meta("measured-specs", edges.size);
|
|
20104
20177
|
}
|
|
20105
20178
|
const report = await selectSpecs({
|
|
20106
20179
|
changed,
|
|
@@ -20108,7 +20181,7 @@ async function runSelectSpecs(opts) {
|
|
|
20108
20181
|
cwd,
|
|
20109
20182
|
base: opts.base,
|
|
20110
20183
|
head,
|
|
20111
|
-
|
|
20184
|
+
edges
|
|
20112
20185
|
});
|
|
20113
20186
|
process.stdout.write(format === "json" ? `${JSON.stringify(report, null, 2)}\n` : renderText(report));
|
|
20114
20187
|
process.exit(0);
|
|
@@ -21268,67 +21341,6 @@ function createGetDriftLedgerHandler(storage) {
|
|
|
21268
21341
|
});
|
|
21269
21342
|
};
|
|
21270
21343
|
}
|
|
21271
|
-
function emptyDeployLog() {
|
|
21272
|
-
return {
|
|
21273
|
-
nextIndex: 0,
|
|
21274
|
-
entries: []
|
|
21275
|
-
};
|
|
21276
|
-
}
|
|
21277
|
-
/** Append `input` to `current`; the appended entry is always the last of `entries`. */
|
|
21278
|
-
function appendDeploy(current, input) {
|
|
21279
|
-
const log = current ?? emptyDeployLog();
|
|
21280
|
-
const head = log.entries[log.entries.length - 1];
|
|
21281
|
-
const gapBefore = head ? head.sha !== input.previousSha : log.nextIndex > 0;
|
|
21282
|
-
const entries = [...log.entries, {
|
|
21283
|
-
...input,
|
|
21284
|
-
index: log.nextIndex,
|
|
21285
|
-
changedPaths: input.changedPaths === null ? null : input.changedPaths.slice(0, 500),
|
|
21286
|
-
gapBefore
|
|
21287
|
-
}];
|
|
21288
|
-
if (entries.length > 200) {
|
|
21289
|
-
entries.splice(0, entries.length - 200);
|
|
21290
|
-
entries[0] = {
|
|
21291
|
-
...entries[0],
|
|
21292
|
-
gapBefore: true
|
|
21293
|
-
};
|
|
21294
|
-
}
|
|
21295
|
-
return {
|
|
21296
|
-
nextIndex: log.nextIndex + 1,
|
|
21297
|
-
entries
|
|
21298
|
-
};
|
|
21299
|
-
}
|
|
21300
|
-
/**
|
|
21301
|
-
* Fold one deploy's selection into the touch index.
|
|
21302
|
-
*
|
|
21303
|
-
* Only the two positions that decide a verdict move: the newest deploy that
|
|
21304
|
-
* needed a spec, and the newest that could not decide it. A `notNeeded` writes
|
|
21305
|
-
* neither — it is the absence of a marker at this position, which is exactly
|
|
21306
|
-
* what a later baseline comparison reads it as.
|
|
21307
|
-
*
|
|
21308
|
-
* Positions only ever advance. Deploys are folded in log order, so a spec
|
|
21309
|
-
* needed at #7 and cleared at #9 keeps `needed.index: 7`: a baseline at #5
|
|
21310
|
-
* must still see that #7 touched it.
|
|
21311
|
-
*/
|
|
21312
|
-
function foldTouchIndex(current, entry, selection) {
|
|
21313
|
-
const out = { ...current };
|
|
21314
|
-
for (const [key, decision] of Object.entries(selection)) {
|
|
21315
|
-
const previous = out[key] ?? {};
|
|
21316
|
-
if (decision.verdict === "needed") out[key] = {
|
|
21317
|
-
...previous,
|
|
21318
|
-
needed: {
|
|
21319
|
-
index: entry.index,
|
|
21320
|
-
sha: entry.sha,
|
|
21321
|
-
at: entry.at,
|
|
21322
|
-
matchedPaths: (decision.touchedBy ?? []).slice(0, 10)
|
|
21323
|
-
}
|
|
21324
|
-
};
|
|
21325
|
-
else if (decision.verdict === "unknown") out[key] = {
|
|
21326
|
-
...previous,
|
|
21327
|
-
undecidedIndex: entry.index
|
|
21328
|
-
};
|
|
21329
|
-
}
|
|
21330
|
-
return out;
|
|
21331
|
-
}
|
|
21332
21344
|
//#endregion
|
|
21333
21345
|
//#region src/hub/api/handlers/deploys.ts
|
|
21334
21346
|
/** `changedPaths` for a wide refactor can run to tens of thousands of entries. */
|
|
@@ -21366,7 +21378,7 @@ function createRecordDeployHandler(storage) {
|
|
|
21366
21378
|
}
|
|
21367
21379
|
/**
|
|
21368
21380
|
* Record what this deploy's selection decided, so a later read can answer each
|
|
21369
|
-
* spec's own range
|
|
21381
|
+
* spec's own range by comparing positions.
|
|
21370
21382
|
*
|
|
21371
21383
|
* It does not fail the request: the log is the record of what shipped and has
|
|
21372
21384
|
* to land even if the fold cannot. What a lost fold must not do is *look* like
|