@sun-asterisk/sungen 3.1.0 → 3.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/commands/challenge.d.ts.map +1 -1
- package/dist/cli/commands/challenge.js +9 -2
- package/dist/cli/commands/challenge.js.map +1 -1
- package/dist/cli/commands/delivery.d.ts.map +1 -1
- package/dist/cli/commands/delivery.js +3 -2
- package/dist/cli/commands/delivery.js.map +1 -1
- package/dist/cli/commands/generate.d.ts.map +1 -1
- package/dist/cli/commands/generate.js +8 -0
- package/dist/cli/commands/generate.js.map +1 -1
- package/dist/exporters/csv-exporter.d.ts.map +1 -1
- package/dist/exporters/csv-exporter.js +92 -76
- package/dist/exporters/csv-exporter.js.map +1 -1
- package/dist/exporters/spec-parser.d.ts.map +1 -1
- package/dist/exporters/spec-parser.js +3 -1
- package/dist/exporters/spec-parser.js.map +1 -1
- package/dist/generators/test-generator/adapters/adapter-interface.d.ts +1 -0
- package/dist/generators/test-generator/adapters/adapter-interface.d.ts.map +1 -1
- package/dist/generators/test-generator/adapters/playwright/templates/scenario.hbs +19 -1
- package/dist/generators/test-generator/code-generator.d.ts +8 -0
- package/dist/generators/test-generator/code-generator.d.ts.map +1 -1
- package/dist/generators/test-generator/code-generator.js +107 -3
- package/dist/generators/test-generator/code-generator.js.map +1 -1
- package/dist/generators/test-generator/patterns/database-patterns.d.ts +2 -1
- package/dist/generators/test-generator/patterns/database-patterns.d.ts.map +1 -1
- package/dist/generators/test-generator/patterns/database-patterns.js +2 -1
- package/dist/generators/test-generator/patterns/database-patterns.js.map +1 -1
- package/dist/generators/test-generator/patterns/expect-patterns.d.ts +3 -0
- package/dist/generators/test-generator/patterns/expect-patterns.d.ts.map +1 -0
- package/dist/generators/test-generator/patterns/expect-patterns.js +54 -0
- package/dist/generators/test-generator/patterns/expect-patterns.js.map +1 -0
- package/dist/generators/test-generator/patterns/index.d.ts.map +1 -1
- package/dist/generators/test-generator/patterns/index.js +2 -0
- package/dist/generators/test-generator/patterns/index.js.map +1 -1
- package/dist/generators/test-generator/step-mapper.d.ts +6 -0
- package/dist/generators/test-generator/step-mapper.d.ts.map +1 -1
- package/dist/generators/test-generator/step-mapper.js +8 -0
- package/dist/generators/test-generator/step-mapper.js.map +1 -1
- package/dist/generators/test-generator/template-engine.d.ts +3 -0
- package/dist/generators/test-generator/template-engine.d.ts.map +1 -1
- package/dist/generators/test-generator/template-engine.js.map +1 -1
- package/dist/generators/test-generator/utils/runtime-data-transformer.d.ts +1 -1
- package/dist/generators/test-generator/utils/runtime-data-transformer.d.ts.map +1 -1
- package/dist/generators/test-generator/utils/runtime-data-transformer.js +5 -5
- package/dist/generators/test-generator/utils/runtime-data-transformer.js.map +1 -1
- package/dist/harness/challenge.d.ts +1 -0
- package/dist/harness/challenge.d.ts.map +1 -1
- package/dist/harness/challenge.js +49 -2
- package/dist/harness/challenge.js.map +1 -1
- package/dist/harness/data-driven-lint.d.ts +7 -0
- package/dist/harness/data-driven-lint.d.ts.map +1 -0
- package/dist/harness/data-driven-lint.js +153 -0
- package/dist/harness/data-driven-lint.js.map +1 -0
- package/dist/harness/parse.d.ts +2 -0
- package/dist/harness/parse.d.ts.map +1 -1
- package/dist/harness/parse.js +16 -0
- package/dist/harness/parse.js.map +1 -1
- package/dist/harness/query-catalog.d.ts +48 -0
- package/dist/harness/query-catalog.d.ts.map +1 -0
- package/dist/harness/query-catalog.js +0 -0
- package/dist/harness/query-catalog.js.map +1 -0
- package/dist/harness/script-check.d.ts.map +1 -1
- package/dist/harness/script-check.js +7 -4
- package/dist/harness/script-check.js.map +1 -1
- package/dist/orchestrator/templates/ai-instructions/claude-agent-challenge.md +3 -2
- package/dist/orchestrator/templates/ai-instructions/claude-skill-gherkin-syntax.md +40 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +19 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-review.md +1 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +6 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-gherkin-syntax.md +40 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +19 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-review.md +1 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +6 -0
- package/dist/orchestrator/templates/specs-db.d.ts +8 -0
- package/dist/orchestrator/templates/specs-db.d.ts.map +1 -1
- package/dist/orchestrator/templates/specs-db.js +22 -0
- package/dist/orchestrator/templates/specs-db.js.map +1 -1
- package/dist/orchestrator/templates/specs-db.ts +22 -0
- package/dist/orchestrator/templates/specs-test-data.ts +76 -15
- package/package.json +1 -1
- package/src/cli/commands/challenge.ts +6 -2
- package/src/cli/commands/delivery.ts +3 -2
- package/src/cli/commands/generate.ts +8 -0
- package/src/exporters/csv-exporter.ts +22 -6
- package/src/exporters/spec-parser.ts +3 -1
- package/src/generators/test-generator/adapters/adapter-interface.ts +1 -0
- package/src/generators/test-generator/adapters/playwright/templates/scenario.hbs +19 -1
- package/src/generators/test-generator/code-generator.ts +105 -3
- package/src/generators/test-generator/patterns/database-patterns.ts +2 -1
- package/src/generators/test-generator/patterns/expect-patterns.ts +49 -0
- package/src/generators/test-generator/patterns/index.ts +2 -0
- package/src/generators/test-generator/step-mapper.ts +9 -0
- package/src/generators/test-generator/template-engine.ts +3 -0
- package/src/generators/test-generator/utils/runtime-data-transformer.ts +5 -5
- package/src/harness/challenge.ts +47 -2
- package/src/harness/data-driven-lint.ts +119 -0
- package/src/harness/parse.ts +12 -0
- package/src/harness/query-catalog.ts +0 -0
- package/src/harness/script-check.ts +8 -5
- package/src/orchestrator/templates/ai-instructions/claude-agent-challenge.md +3 -2
- package/src/orchestrator/templates/ai-instructions/claude-skill-gherkin-syntax.md +40 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +19 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-review.md +1 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +6 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-gherkin-syntax.md +40 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +19 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-review.md +1 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +6 -0
- package/src/orchestrator/templates/specs-db.ts +22 -0
- package/src/orchestrator/templates/specs-test-data.ts +76 -15
|
@@ -20,7 +20,9 @@ function extractTestBlock(content: string, startIdx: number): {
|
|
|
20
20
|
// test('title', { tag: [...] }, async ({ page }) => {
|
|
21
21
|
// Backreference \1 lets the inner title contain the opposite quote type
|
|
22
22
|
// (e.g. test('Footer "X" link', ...) — common when scenarios cite UI labels).
|
|
23
|
-
|
|
23
|
+
// Quote char may be ' " or ` — the backtick form is a data-driven (@cases) title
|
|
24
|
+
// like `VP-… — ${__row.__label}`.
|
|
25
|
+
const testRegex = /test\s*\(\s*(['"`])((?:(?!\1).)+)\1\s*,\s*(?:\{[^}]*\}\s*,\s*)?async\s*\([^)]*\)\s*=>\s*\{/g;
|
|
24
26
|
testRegex.lastIndex = startIdx;
|
|
25
27
|
const match = testRegex.exec(content);
|
|
26
28
|
if (!match) return null;
|
|
@@ -36,6 +36,7 @@ export interface ScenarioData {
|
|
|
36
36
|
authRole?: string; // Auth role for storage state
|
|
37
37
|
isParallel?: boolean; // @parallel: use fresh page from fixture
|
|
38
38
|
tags?: string; // Pass-through tags for Playwright { tag: [...] }, e.g. "'@smoke', '@high'"
|
|
39
|
+
casesDataset?: string; // @cases:<dataset> — emit one test() per row of the runtime dataset
|
|
39
40
|
}
|
|
40
41
|
|
|
41
42
|
export interface StepTemplateData {
|
|
@@ -1,3 +1,20 @@
|
|
|
1
|
+
{{#if casesDataset}}
|
|
2
|
+
for (const __row of testData.cases('{{casesDataset}}')) {
|
|
3
|
+
{{#if tags}}
|
|
4
|
+
test(`{{scenarioName}} — ${__row.__label}`, { tag: [{{{tags}}}] }, async ({{#if isParallel}}{ page }{{/if}}) => {
|
|
5
|
+
{{else}}
|
|
6
|
+
test(`{{scenarioName}} — ${__row.__label}`, async ({{#if isParallel}}{ page }{{/if}}) => {
|
|
7
|
+
{{/if}}
|
|
8
|
+
const rowData = testData.withRow(__row);
|
|
9
|
+
{{#each steps}}
|
|
10
|
+
{{#if comment}}
|
|
11
|
+
// {{comment}}
|
|
12
|
+
{{/if}}
|
|
13
|
+
{{code}}
|
|
14
|
+
{{/each}}
|
|
15
|
+
});
|
|
16
|
+
}
|
|
17
|
+
{{else}}
|
|
1
18
|
{{#if isParallel}}
|
|
2
19
|
{{#if tags}}
|
|
3
20
|
test('{{scenarioName}}', { tag: [{{{tags}}}] }, async ({ page }) => {
|
|
@@ -24,4 +41,5 @@
|
|
|
24
41
|
{{code}}
|
|
25
42
|
{{/each}}
|
|
26
43
|
});
|
|
27
|
-
{{/if}}
|
|
44
|
+
{{/if}}
|
|
45
|
+
{{/if}}
|
|
@@ -5,6 +5,7 @@ import { StepMapper } from './step-mapper';
|
|
|
5
5
|
import { TestGeneratorAdapter, adapterRegistry } from './adapters';
|
|
6
6
|
import { transformToRuntimeData } from './utils/runtime-data-transformer';
|
|
7
7
|
import { isDbStep } from './patterns/database-patterns';
|
|
8
|
+
import { resolveQuery, compileQuery } from '../../harness/query-catalog';
|
|
8
9
|
|
|
9
10
|
/**
|
|
10
11
|
* Filter base scenario steps for @extend: only keep Given→When steps.
|
|
@@ -74,7 +75,7 @@ function extractCleanupFlags(tags: string[]): { overlay?: boolean; forms?: boole
|
|
|
74
75
|
const FUNCTIONAL_TAG_PREFIXES = [
|
|
75
76
|
'@parallel', '@cleanup:', '@auth:', '@manual', '@no-auth',
|
|
76
77
|
'@steps:', '@extend:', '@screenshot:', '@beforeAll', '@afterEach', '@afterAll',
|
|
77
|
-
'@flow',
|
|
78
|
+
'@flow', '@cases:',
|
|
78
79
|
];
|
|
79
80
|
|
|
80
81
|
function extractPassThroughTags(scenarioTags: string[], featureTags: string[]): string | undefined {
|
|
@@ -170,6 +171,8 @@ export class CodeGenerator {
|
|
|
170
171
|
private adapter: TestGeneratorAdapter;
|
|
171
172
|
private screenName?: string;
|
|
172
173
|
private options: any;
|
|
174
|
+
// Screen/flow name for the CURRENT feature, in catalog-resolution form (`flows/<x>` for flows).
|
|
175
|
+
private queryScreenName: string = '';
|
|
173
176
|
// Steps registry built per feature during generateTestCode(); used by countSteps()
|
|
174
177
|
private stepsRegistry = new Map<string, ParsedScenario>();
|
|
175
178
|
|
|
@@ -302,7 +305,59 @@ export class CodeGenerator {
|
|
|
302
305
|
const steps: ParsedStep[] = [];
|
|
303
306
|
if (feature.background?.steps) steps.push(...feature.background.steps);
|
|
304
307
|
for (const sc of feature.scenarios || []) if (sc.steps) steps.push(...sc.steps);
|
|
305
|
-
|
|
308
|
+
if (steps.some((s) => s && typeof s.text === 'string' && isDbStep(s.text))) return true;
|
|
309
|
+
// A scenario may carry only a @query:<name> post-condition tag (no DB step in its body).
|
|
310
|
+
return (feature.scenarios || []).some((sc) => (sc.tags || []).some((t) => t.startsWith('@query:')));
|
|
311
|
+
}
|
|
312
|
+
|
|
313
|
+
/**
|
|
314
|
+
* Build the precondition bind steps for a scenario's `@query:<name>[(p={{v}},…)]` tags.
|
|
315
|
+
* Each runs the catalog query and binds the result array to a `{{name}}` variable.
|
|
316
|
+
*/
|
|
317
|
+
private buildQueryBinds(scenario: ParsedScenario): Array<{ comment?: string; code: string }> {
|
|
318
|
+
const out: Array<{ comment?: string; code: string }> = [];
|
|
319
|
+
const TAG = /^@query:([A-Za-z_][A-Za-z0-9_]*)(?:\((.*)\))?$/;
|
|
320
|
+
for (const tag of scenario.tags || []) {
|
|
321
|
+
const m = tag.match(TAG);
|
|
322
|
+
if (!m) continue;
|
|
323
|
+
const name = m[1];
|
|
324
|
+
const overrides = this.parseQueryOverrides(m[2]);
|
|
325
|
+
const entry = resolveQuery(name, this.queryScreenName); // throws (fail-fast) if missing/ambiguous
|
|
326
|
+
const { sql, paramNames } = compileQuery(entry);
|
|
327
|
+
const paramExprs = paramNames.map((p) =>
|
|
328
|
+
p in overrides ? overrides[p] : `testData.get(${JSON.stringify(p)})`,
|
|
329
|
+
);
|
|
330
|
+
const label = JSON.stringify(entry.description ? `query "${name}" — ${entry.description}` : `query "${name}"`);
|
|
331
|
+
const ds = entry.datasource ? JSON.stringify(entry.datasource) : 'undefined';
|
|
332
|
+
out.push({
|
|
333
|
+
comment: `@query:${name} → bind {{${name}}} from ${entry.datasource || 'default datasource'}`,
|
|
334
|
+
code: this.indentCode(
|
|
335
|
+
`testData.bind(${JSON.stringify(name)}, await db.fetchQuery(${label}, ${JSON.stringify(sql)}, [${paramExprs.join(', ')}], ${ds}));`,
|
|
336
|
+
4,
|
|
337
|
+
),
|
|
338
|
+
});
|
|
339
|
+
}
|
|
340
|
+
return out;
|
|
341
|
+
}
|
|
342
|
+
|
|
343
|
+
/** Parse `@query:name(a={{x}},b="lit",c=3)` overrides → { a: "testData.get('x')", … } JS exprs. */
|
|
344
|
+
private parseQueryOverrides(raw?: string): Record<string, string> {
|
|
345
|
+
const out: Record<string, string> = {};
|
|
346
|
+
if (!raw) return out;
|
|
347
|
+
for (const part of raw.split(',')) {
|
|
348
|
+
const eq = part.indexOf('=');
|
|
349
|
+
if (eq < 0) continue;
|
|
350
|
+
const key = part.slice(0, eq).trim();
|
|
351
|
+
const val = part.slice(eq + 1).trim();
|
|
352
|
+
if (!key) continue;
|
|
353
|
+
const v = val.match(/^\{\{\s*([^}]+?)\s*\}\}$/);
|
|
354
|
+
const q = val.match(/^["'](.*)["']$/);
|
|
355
|
+
if (v) out[key] = `testData.get(${JSON.stringify(v[1])})`;
|
|
356
|
+
else if (q) out[key] = JSON.stringify(q[1]);
|
|
357
|
+
else if (/^-?\d+(?:\.\d+)?$/.test(val)) out[key] = val;
|
|
358
|
+
else out[key] = JSON.stringify(val);
|
|
359
|
+
}
|
|
360
|
+
return out;
|
|
306
361
|
}
|
|
307
362
|
|
|
308
363
|
/** Copy the Data Driver runtime helper into specs/db.ts (idempotent). */
|
|
@@ -391,6 +446,8 @@ export class CodeGenerator {
|
|
|
391
446
|
}
|
|
392
447
|
this.stepMapper.setScreenContext(effectiveScreenName);
|
|
393
448
|
}
|
|
449
|
+
// Catalog-resolution screen name for @query binds (flows are prefixed `flows/`).
|
|
450
|
+
this.queryScreenName = isFlowFeature ? `flows/${effectiveScreenName}` : (effectiveScreenName || '');
|
|
394
451
|
|
|
395
452
|
// Reset flow mode per feature to prevent state leak in --all mode
|
|
396
453
|
this.stepMapper.setFlowMode(isFlowFeature);
|
|
@@ -637,6 +694,35 @@ export class CodeGenerator {
|
|
|
637
694
|
// Set scenario context for path variable resolution (full merged list)
|
|
638
695
|
this.stepMapper.setScenarioContext(stepsToMap);
|
|
639
696
|
|
|
697
|
+
// Data-driven (@cases): the scenario's {{col}} refs are dataset *row columns* that
|
|
698
|
+
// exist only at runtime — register them as captured so they resolve to a runtime
|
|
699
|
+
// get() (→ rowData.get) instead of compile-time YAML lookup.
|
|
700
|
+
const casesTag = scenario.tags.find((t) => t.startsWith('@cases:'));
|
|
701
|
+
const casesDataset = casesTag ? casesTag.slice('@cases:'.length).trim() : undefined;
|
|
702
|
+
if (casesDataset) {
|
|
703
|
+
const refs = new Set<string>();
|
|
704
|
+
for (const st of stepsToMap) {
|
|
705
|
+
for (const m of (st.text || '').matchAll(/\{\{\s*([\w.]+)\s*\}\}/g)) refs.add(m[1]);
|
|
706
|
+
}
|
|
707
|
+
for (const r of refs) this.stepMapper.registerCaptured(r);
|
|
708
|
+
}
|
|
709
|
+
|
|
710
|
+
// @query-bound vars: `{{name.col}}` / `{{name[2].col}}` / `{{name.count}}` exist only at
|
|
711
|
+
// runtime (the bind fetches them) — register as captured so they resolve to a runtime get()
|
|
712
|
+
// instead of a compile-time YAML lookup that would fail.
|
|
713
|
+
const queryNames = (scenario.tags || [])
|
|
714
|
+
.map((t) => t.match(/^@query:([A-Za-z_][A-Za-z0-9_]*)/))
|
|
715
|
+
.filter((m): m is RegExpMatchArray => !!m)
|
|
716
|
+
.map((m) => m[1]);
|
|
717
|
+
if (queryNames.length) {
|
|
718
|
+
for (const st of stepsToMap) {
|
|
719
|
+
for (const mt of (st.text || '').matchAll(/\{\{\s*([^}]+?)\s*\}\}/g)) {
|
|
720
|
+
const head = mt[1].split(/[.[]/)[0];
|
|
721
|
+
if (queryNames.includes(head)) this.stepMapper.registerCaptured(mt[1]);
|
|
722
|
+
}
|
|
723
|
+
}
|
|
724
|
+
}
|
|
725
|
+
|
|
640
726
|
const steps: Array<{ comment?: string; code: string }> = [];
|
|
641
727
|
|
|
642
728
|
if (scenario.extendsName && this.stepsRegistry.has(scenario.extendsName)) {
|
|
@@ -666,17 +752,33 @@ export class CodeGenerator {
|
|
|
666
752
|
}
|
|
667
753
|
}
|
|
668
754
|
|
|
755
|
+
// @query:<name>[(p={{v}},…)] tags → run the catalog query as a PRECONDITION and bind its
|
|
756
|
+
// result to a `{{name}}` variable (scenario asserts on it with `expect …` + path access).
|
|
757
|
+
// Prepended so the binds run before the scenario's own steps.
|
|
758
|
+
const queryBinds = this.buildQueryBinds(scenario);
|
|
759
|
+
if (queryBinds.length) steps.unshift(...queryBinds);
|
|
760
|
+
|
|
669
761
|
// Extract pass-through tags (feature + scenario, excluding functional tags)
|
|
670
762
|
const tags = extractPassThroughTags(scenario.tags, featureTags);
|
|
671
763
|
|
|
672
764
|
// Use adapter to render scenario
|
|
673
|
-
|
|
765
|
+
const rendered = this.adapter.renderScenario({
|
|
674
766
|
scenarioName: scenario.name,
|
|
675
767
|
steps,
|
|
676
768
|
authRole,
|
|
677
769
|
isParallel,
|
|
678
770
|
tags,
|
|
771
|
+
casesDataset,
|
|
679
772
|
});
|
|
773
|
+
|
|
774
|
+
// Data-driven (@cases): the per-row test() binds a row-scoped view (`rowData`).
|
|
775
|
+
// Pre-transform THIS scenario's runtime-data markers to read from `rowData`, so the
|
|
776
|
+
// global `testData` transform that runs next on the rest of the file leaves it alone.
|
|
777
|
+
// The loop header's `testData.cases()/withRow()` are literal code (no markers) → untouched.
|
|
778
|
+
if (casesDataset && this.options.runtimeData) {
|
|
779
|
+
return transformToRuntimeData(rendered, 'rowData');
|
|
780
|
+
}
|
|
781
|
+
return rendered;
|
|
680
782
|
}
|
|
681
783
|
|
|
682
784
|
/**
|
|
@@ -21,7 +21,8 @@ const reRow = new RegExp(`see\\s+${TABLE}\\s+row\\s+where\\b`, 'i');
|
|
|
21
21
|
const reNoRow = new RegExp(`see\\s+${TABLE}\\s+no\\s+row\\s+where\\b`, 'i');
|
|
22
22
|
const reCount = new RegExp(`see\\s+${TABLE}.*\\bcount\\s+is\\b`, 'i');
|
|
23
23
|
|
|
24
|
-
/** True when a step is a DB-verification step (used to wire the `db` import).
|
|
24
|
+
/** True when a step is a declarative DB-verification step (used to wire the `db` import).
|
|
25
|
+
* Named queries are invoked via the `@query:` annotation, not a step — see code-generator. */
|
|
25
26
|
export function isDbStep(text: string): boolean {
|
|
26
27
|
return reNoRow.test(text) || reRow.test(text) || reCount.test(text);
|
|
27
28
|
}
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
import { ParsedStep } from '../../gherkin-parser';
|
|
2
|
+
import { StepPattern, PatternContext } from './types';
|
|
3
|
+
import { MappedStep } from '../step-mapper';
|
|
4
|
+
|
|
5
|
+
/**
|
|
6
|
+
* Data-vs-data assertions — compare two runtime values directly (no UI element).
|
|
7
|
+
* The primary use is asserting on an `@query`-bound result via path access, e.g.
|
|
8
|
+
*
|
|
9
|
+
* Then expect {{products.count}} is {{expected}}
|
|
10
|
+
* Then expect {{products.count}} is at least {{one}}
|
|
11
|
+
* Then expect {{active_user.status}} is "active"
|
|
12
|
+
* Then expect {{order.total}} is not {{zero}}
|
|
13
|
+
*
|
|
14
|
+
* Both sides are `{{var|path}}` | "literal" | 'literal' | number. Generic — works for any
|
|
15
|
+
* test-data values, not only DB results.
|
|
16
|
+
*/
|
|
17
|
+
const VALUE = String.raw`\{\{[^}]+\}\}|"[^"]*"|'[^']*'|-?\d+(?:\.\d+)?`;
|
|
18
|
+
const reExpect = new RegExp(`^\\s*(?:User\\s+)?expect\\s+(${VALUE})\\s+is\\s+(not\\s+|at\\s+least\\s+|at\\s+most\\s+)?(${VALUE})\\s*$`, 'i');
|
|
19
|
+
|
|
20
|
+
/** Render a value token (`{{var}}` | "literal" | 'literal' | number) as a JS expression. */
|
|
21
|
+
function valueExpr(token: string): string {
|
|
22
|
+
const t = token.trim();
|
|
23
|
+
const v = t.match(/^\{\{\s*([^}]+?)\s*\}\}$/);
|
|
24
|
+
if (v) return `testData.get(${JSON.stringify(v[1])})`;
|
|
25
|
+
const q = t.match(/^["'](.*)["']$/);
|
|
26
|
+
if (q) return JSON.stringify(q[1]);
|
|
27
|
+
if (/^-?\d+(?:\.\d+)?$/.test(t)) return t;
|
|
28
|
+
return JSON.stringify(t);
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
export const expectPatterns: StepPattern[] = [
|
|
32
|
+
{
|
|
33
|
+
name: 'expect-data',
|
|
34
|
+
priority: 62, // above generic see-assertions; sibling of the DB assertions
|
|
35
|
+
matcher: (step: ParsedStep) => reExpect.test(step.text),
|
|
36
|
+
generator: (step: ParsedStep, _ctx: PatternContext): MappedStep => {
|
|
37
|
+
const m = step.text.match(reExpect)!;
|
|
38
|
+
const a = valueExpr(m[1]);
|
|
39
|
+
const op = (m[2] || '').trim().toLowerCase();
|
|
40
|
+
const b = valueExpr(m[3]);
|
|
41
|
+
let code: string;
|
|
42
|
+
if (op === 'at least') code = `expect(Number(${a})).toBeGreaterThanOrEqual(Number(${b}));`;
|
|
43
|
+
else if (op === 'at most') code = `expect(Number(${a})).toBeLessThanOrEqual(Number(${b}));`;
|
|
44
|
+
else if (op === 'not') code = `expect(String(${a})).not.toBe(String(${b}));`;
|
|
45
|
+
else code = `expect(String(${a})).toBe(String(${b}));`;
|
|
46
|
+
return { code, comment: `Expect ${m[1]} ${op ? op + ' ' : 'is '}${m[3]}` };
|
|
47
|
+
},
|
|
48
|
+
},
|
|
49
|
+
];
|
|
@@ -12,6 +12,7 @@ import { scopePatterns } from './scope-patterns';
|
|
|
12
12
|
import { tablePatterns } from './table-patterns';
|
|
13
13
|
import { capturePatterns } from './capture-patterns';
|
|
14
14
|
import { databasePatterns } from './database-patterns';
|
|
15
|
+
import { expectPatterns } from './expect-patterns';
|
|
15
16
|
|
|
16
17
|
/**
|
|
17
18
|
* Pattern Registry - manages all step patterns
|
|
@@ -38,6 +39,7 @@ export class PatternRegistry {
|
|
|
38
39
|
this.patterns.push(...tablePatterns);
|
|
39
40
|
this.patterns.push(...capturePatterns);
|
|
40
41
|
this.patterns.push(...databasePatterns);
|
|
42
|
+
this.patterns.push(...expectPatterns);
|
|
41
43
|
|
|
42
44
|
// Sort by priority (higher first)
|
|
43
45
|
this.patterns.sort((a, b) => (b.priority || 0) - (a.priority || 0));
|
|
@@ -94,6 +94,15 @@ export class StepMapper {
|
|
|
94
94
|
this.templateEngine.resetBaseContext();
|
|
95
95
|
}
|
|
96
96
|
|
|
97
|
+
/**
|
|
98
|
+
* Register a runtime data variable so `{{name}}` resolves to `testData.get('name')`
|
|
99
|
+
* (skipping compile-time YAML validation). Used for @cases row columns, which exist
|
|
100
|
+
* only at runtime in the dataset rows. Scenario-scoped (cleared by setScenarioContext).
|
|
101
|
+
*/
|
|
102
|
+
registerCaptured(name: string): void {
|
|
103
|
+
this.dataResolver.registerCaptured(name);
|
|
104
|
+
}
|
|
105
|
+
|
|
97
106
|
/**
|
|
98
107
|
* Map a Gherkin step to Playwright code
|
|
99
108
|
* Uses pattern registry first, falls back to AI if enabled
|
|
@@ -284,6 +284,9 @@ export class TemplateEngine {
|
|
|
284
284
|
scenarioName: string;
|
|
285
285
|
steps: Array<{ comment?: string; code: string }>;
|
|
286
286
|
authRole?: string;
|
|
287
|
+
isParallel?: boolean;
|
|
288
|
+
tags?: string;
|
|
289
|
+
casesDataset?: string;
|
|
287
290
|
}): string {
|
|
288
291
|
return this.render('scenario', data);
|
|
289
292
|
}
|
|
@@ -4,7 +4,7 @@ const MARKER_PATTERN = /__SUNGEN_TD_([A-Za-z0-9_]+)__/;
|
|
|
4
4
|
* Replace __SUNGEN_TD_ markers with testData.get() calls in generated code.
|
|
5
5
|
* Three passes: comments, string literals, then regex literals.
|
|
6
6
|
*/
|
|
7
|
-
export function transformToRuntimeData(code: string): string {
|
|
7
|
+
export function transformToRuntimeData(code: string, accessor: string = 'testData'): string {
|
|
8
8
|
// Pass 0: Comments — replace markers in // comments with decoded key name
|
|
9
9
|
// Prevents Pass 2 from misinterpreting // comment markers as regex delimiters
|
|
10
10
|
code = code.replace(
|
|
@@ -20,9 +20,9 @@ export function transformToRuntimeData(code: string): string {
|
|
|
20
20
|
(_, _quote, prefix, enc, suffix) => {
|
|
21
21
|
const key = decodeKey(enc);
|
|
22
22
|
if (!prefix && !suffix) {
|
|
23
|
-
return
|
|
23
|
+
return `${accessor}.get('${key}')`;
|
|
24
24
|
}
|
|
25
|
-
return `\`${prefix}\${
|
|
25
|
+
return `\`${prefix}\${${accessor}.get('${key}')}${suffix}\``;
|
|
26
26
|
}
|
|
27
27
|
);
|
|
28
28
|
|
|
@@ -32,7 +32,7 @@ export function transformToRuntimeData(code: string): string {
|
|
|
32
32
|
/\/((?:[^/\\\n]|\\.)*?)__SUNGEN_TD_([A-Za-z0-9_]+)__((?:[^/\\\n]|\\.)*?)\/([gimsuy]*)/g,
|
|
33
33
|
(_, prefix, enc, suffix, flags) => {
|
|
34
34
|
const key = decodeKey(enc);
|
|
35
|
-
const ref =
|
|
35
|
+
const ref = `${accessor}.get('${key}')`;
|
|
36
36
|
const flagStr = flags ? `, '${flags}'` : '';
|
|
37
37
|
if (!prefix && !suffix) return `new RegExp(${ref}${flagStr})`;
|
|
38
38
|
return `new RegExp(\`${prefix}\${${ref}}${suffix}\`${flagStr})`;
|
|
@@ -44,7 +44,7 @@ export function transformToRuntimeData(code: string): string {
|
|
|
44
44
|
// table/list count templates). testData.get() returns a string, so coerce with Number().
|
|
45
45
|
code = code.replace(
|
|
46
46
|
/__SUNGEN_TD_([A-Za-z0-9_]+)__/g,
|
|
47
|
-
(_, enc) => `Number(
|
|
47
|
+
(_, enc) => `Number(${accessor}.get('${decodeKey(enc)}'))`
|
|
48
48
|
);
|
|
49
49
|
|
|
50
50
|
return code;
|
package/src/harness/challenge.ts
CHANGED
|
@@ -34,6 +34,8 @@ export interface ChallengeReport {
|
|
|
34
34
|
shallowThemes: string[];
|
|
35
35
|
// Depth critic
|
|
36
36
|
collectionClaimSingular: ChallengeFinding[];
|
|
37
|
+
// Data-driven critic — @cases-worthy gaps (spec-independent)
|
|
38
|
+
dataDriven: ChallengeFinding[];
|
|
37
39
|
// Novelty critic (deterministic prompts → AI agent fills candidates)
|
|
38
40
|
noveltyPrompts: string[];
|
|
39
41
|
// Roll-up
|
|
@@ -47,6 +49,10 @@ const PLURAL_NOUN = /\b(cards|items|products|rows|results|prices|entries|records
|
|
|
47
49
|
const QUANTIFIER = /\b(all|every|each)\b/i;
|
|
48
50
|
const DISPLAY_VERB = /\b(displays?|shows?|lists?|grid|contains?)\b/i;
|
|
49
51
|
|
|
52
|
+
// Title lexicon implying a CLASS of inputs (→ a data-driven @cases candidate). Used by
|
|
53
|
+
// Detector A. Spec-independent — reads the title, not the spec.
|
|
54
|
+
const INPUT_CLASS_LEXICON = /\b(invalid|formats?|boundary|range|min|max|too (?:long|short)|special char|each|various|classes)\b/i;
|
|
55
|
+
|
|
50
56
|
/** Risk lenses the Novelty critic prompts the AI to explore (beyond the catalog). */
|
|
51
57
|
const NOVELTY_LENSES = [
|
|
52
58
|
'double-submit / rapid repeat of the primary action (duplicate side-effects?)',
|
|
@@ -88,17 +94,49 @@ export function buildChallenge(screenDir: string, screenName: string): Challenge
|
|
|
88
94
|
}
|
|
89
95
|
}
|
|
90
96
|
|
|
91
|
-
// 3.
|
|
97
|
+
// 3. Data-driven critic — surface @cases-worthy gaps (spec-independent).
|
|
98
|
+
const dataDriven: ChallengeFinding[] = [];
|
|
99
|
+
const clustered = new Set<string>();
|
|
100
|
+
// Detector B (high precision): ≥2 non-@cases scenarios sharing the SAME step skeleton
|
|
101
|
+
// (stepSkeleton normalizes {{vars}}/quoted values) → they differ only by data → collapse.
|
|
102
|
+
const bySkeleton = new Map<string, ScenarioInfo[]>();
|
|
103
|
+
for (const s of scenarios) {
|
|
104
|
+
if (s.manual || s.casesDataset || !s.stepSkeleton) continue;
|
|
105
|
+
(bySkeleton.get(s.stepSkeleton) ?? bySkeleton.set(s.stepSkeleton, []).get(s.stepSkeleton)!).push(s);
|
|
106
|
+
}
|
|
107
|
+
for (const group of bySkeleton.values()) {
|
|
108
|
+
if (group.length < 2) continue;
|
|
109
|
+
group.forEach((s) => clustered.add(s.name));
|
|
110
|
+
dataDriven.push({
|
|
111
|
+
scenario: group.map((s) => s.name).join(' | '),
|
|
112
|
+
issue: `${group.length} scenarios share the same steps and differ only by input value (data-variants).`,
|
|
113
|
+
suggestion: `Collapse into ONE \`@cases:<dataset>\` — ${group.length} rows in test-data, each \`{{col}}\` a column. See Gherkin → Advanced → Data-driven.`,
|
|
114
|
+
});
|
|
115
|
+
}
|
|
116
|
+
// Detector A (advisory): a lone scenario whose title implies a class of inputs → suggest @cases.
|
|
117
|
+
for (const s of scenarios) {
|
|
118
|
+
if (s.manual || s.casesDataset || clustered.has(s.name)) continue;
|
|
119
|
+
if (INPUT_CLASS_LEXICON.test(s.name)) {
|
|
120
|
+
dataDriven.push({
|
|
121
|
+
scenario: s.name,
|
|
122
|
+
issue: 'Title reads like a CLASS of inputs (validation/boundary/error) but tests a single value.',
|
|
123
|
+
suggestion: 'Consider `@cases` to cover the EP/boundary classes (one row per valid/invalid class), not just one value.',
|
|
124
|
+
});
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
// 4. Novelty critic — deterministic prompts; the AI agent expands these into candidates.
|
|
92
129
|
const noveltyPrompts = NOVELTY_LENSES.map((l) => `Find 1 non-obvious, valuable scenario via: ${l}`);
|
|
93
130
|
|
|
94
131
|
// Roll-up — exploration readiness signals (not a fake score).
|
|
95
132
|
const explorationReadiness: string[] = [];
|
|
133
|
+
if (dataDriven.length) explorationReadiness.push(`${dataDriven.length} data-driven gap(s) — scenarios that should be one \`@cases\` (collapse data-variants / cover EP-boundary classes).`);
|
|
96
134
|
if (collectionClaimSingular.length) explorationReadiness.push(`${collectionClaimSingular.length} title↔assertion gap(s) — deterministic depth critic flagged these; an AI Business-Depth critic should confirm + fix.`);
|
|
97
135
|
if (overCovered.length) explorationReadiness.push(`${overCovered.length} possibly over-covered area(s) — rebalance toward correctness.`);
|
|
98
136
|
if (shallowThemes.length) explorationReadiness.push(`Shallow themes: ${shallowThemes.join(', ')}.`);
|
|
99
137
|
explorationReadiness.push('Novelty candidates are NOT generated deterministically — run the `sungen-challenge` agent (Claude) or its inline criteria (Copilot) to propose them, then QA accept/reject (≤20% of official, no auto-merge).');
|
|
100
138
|
|
|
101
|
-
return { screen: screenName, overCovered, shallowThemes, collectionClaimSingular, noveltyPrompts, explorationReadiness };
|
|
139
|
+
return { screen: screenName, overCovered, shallowThemes, collectionClaimSingular, dataDriven, noveltyPrompts, explorationReadiness };
|
|
102
140
|
}
|
|
103
141
|
|
|
104
142
|
/** Render the Challenge Report as Markdown (advisory — not part of the official suite). */
|
|
@@ -114,6 +152,13 @@ export function renderChallengeMarkdown(r: ChallengeReport): string {
|
|
|
114
152
|
} else lines.push('_none_');
|
|
115
153
|
lines.push('');
|
|
116
154
|
|
|
155
|
+
lines.push('## Data-driven — scenarios that should be `@cases` (one test case, many inputs)');
|
|
156
|
+
if (r.dataDriven.length) {
|
|
157
|
+
lines.push('| Scenario(s) | Issue | Suggested |', '|---|---|---|');
|
|
158
|
+
for (const f of r.dataDriven) lines.push(`| ${f.scenario} | ${f.issue} | ${f.suggestion} |`);
|
|
159
|
+
} else lines.push('_none_');
|
|
160
|
+
lines.push('');
|
|
161
|
+
|
|
117
162
|
lines.push('## Coverage — possibly over-covered / shallow');
|
|
118
163
|
if (r.overCovered.length) for (const o of r.overCovered) lines.push(`- **${o.bucket}** — ${o.note}`);
|
|
119
164
|
if (r.shallowThemes.length) lines.push(`- Shallow themes: ${r.shallowThemes.join(', ')}`);
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Deterministic lint for the data-driven features (`@cases` + `@query`).
|
|
3
|
+
*
|
|
4
|
+
* Advisory (warn, never block) — surfaced after `sungen generate`. Catches the silent
|
|
5
|
+
* footguns: a `{{var}}` in a @cases scenario that is neither a row column nor a top-level
|
|
6
|
+
* key, a name collision that shadows a top-level value, inconsistent dataset rows, and a
|
|
7
|
+
* @query whose param has no value / whose name doesn't resolve.
|
|
8
|
+
*/
|
|
9
|
+
import * as fs from 'fs';
|
|
10
|
+
import * as path from 'path';
|
|
11
|
+
import { parse as parseYaml } from 'yaml';
|
|
12
|
+
import { GherkinParser, ParsedScenario } from '../generators/gherkin-parser';
|
|
13
|
+
import { resolveQuery, lintCatalog } from './query-catalog';
|
|
14
|
+
|
|
15
|
+
export interface DataDrivenWarning {
|
|
16
|
+
scenario?: string;
|
|
17
|
+
message: string;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
const ROW_META = new Set(['case', 'name', 'label', '__label']);
|
|
21
|
+
|
|
22
|
+
/** Collect the inner text of every `{{…}}` reference in a scenario's steps. */
|
|
23
|
+
function collectRefs(sc: ParsedScenario): string[] {
|
|
24
|
+
const refs = new Set<string>();
|
|
25
|
+
for (const st of sc.steps || []) {
|
|
26
|
+
for (const m of (st.text || '').matchAll(/\{\{\s*([^}]+?)\s*\}\}/g)) refs.add(m[1]);
|
|
27
|
+
}
|
|
28
|
+
return [...refs];
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
/** Keys overridden in a `@query:name(a=…,b=…)` annotation. */
|
|
32
|
+
function overrideKeys(raw?: string): Set<string> {
|
|
33
|
+
const out = new Set<string>();
|
|
34
|
+
if (!raw) return out;
|
|
35
|
+
for (const part of raw.split(',')) {
|
|
36
|
+
const eq = part.indexOf('=');
|
|
37
|
+
if (eq > 0) out.add(part.slice(0, eq).trim());
|
|
38
|
+
}
|
|
39
|
+
return out;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
/** Lint the @cases/@query usage of one screen/flow directory. Returns advisory warnings. */
|
|
43
|
+
export function lintDataDriven(screenDir: string, cwd: string = process.cwd()): DataDrivenWarning[] {
|
|
44
|
+
const base = path.basename(screenDir);
|
|
45
|
+
const isFlow = path.basename(path.dirname(screenDir)) === 'flows';
|
|
46
|
+
const screenName = isFlow ? `flows/${base}` : base;
|
|
47
|
+
const featurePath = path.join(screenDir, 'features', `${base}.feature`);
|
|
48
|
+
if (!fs.existsSync(featurePath)) return [];
|
|
49
|
+
|
|
50
|
+
let scenarios: ParsedScenario[];
|
|
51
|
+
try {
|
|
52
|
+
scenarios = new GherkinParser().parseFeatureFile(featurePath).scenarios || [];
|
|
53
|
+
} catch {
|
|
54
|
+
return [];
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
const tdPath = path.join(screenDir, 'test-data', `${base}.yaml`);
|
|
58
|
+
const td: Record<string, any> = tdPath && fs.existsSync(tdPath) ? parseYaml(fs.readFileSync(tdPath, 'utf8')) || {} : {};
|
|
59
|
+
const topKeys = new Set(Object.keys(td));
|
|
60
|
+
const warns: DataDrivenWarning[] = [];
|
|
61
|
+
|
|
62
|
+
for (const sc of scenarios) {
|
|
63
|
+
const tags: string[] = sc.tags || [];
|
|
64
|
+
const refs = collectRefs(sc);
|
|
65
|
+
|
|
66
|
+
// --- @cases -----------------------------------------------------------
|
|
67
|
+
const casesTag = tags.find((t) => t.startsWith('@cases:'));
|
|
68
|
+
if (casesTag) {
|
|
69
|
+
const ds = casesTag.slice('@cases:'.length).trim();
|
|
70
|
+
const rows = td[ds];
|
|
71
|
+
if (!Array.isArray(rows)) {
|
|
72
|
+
warns.push({ scenario: sc.name, message: `@cases:${ds} → dataset "${ds}" is missing or not a list in test-data.` });
|
|
73
|
+
} else {
|
|
74
|
+
const colSets = rows.map((r) => new Set(Object.keys(r || {}).filter((k) => !ROW_META.has(k))));
|
|
75
|
+
const allCols = new Set<string>();
|
|
76
|
+
colSets.forEach((s) => s.forEach((c) => allCols.add(c)));
|
|
77
|
+
for (const c of allCols) {
|
|
78
|
+
const missing = colSets.filter((s) => !s.has(c)).length;
|
|
79
|
+
if (missing) warns.push({ scenario: sc.name, message: `@cases:${ds} → column "${c}" is missing in ${missing}/${rows.length} row(s) — rows are inconsistent.` });
|
|
80
|
+
if (topKeys.has(c)) warns.push({ scenario: sc.name, message: `@cases:${ds} → "${c}" is both a dataset column and a top-level test-data key — the row value shadows the top-level one.` });
|
|
81
|
+
}
|
|
82
|
+
for (const r of refs) {
|
|
83
|
+
const head = r.split(/[.[]/)[0];
|
|
84
|
+
if (!allCols.has(head) && !topKeys.has(head)) {
|
|
85
|
+
warns.push({ scenario: sc.name, message: `@cases:${ds} → {{${r}}} is neither a dataset column nor a top-level test-data key.` });
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
// --- @query -----------------------------------------------------------
|
|
92
|
+
for (const t of tags) {
|
|
93
|
+
const m = t.match(/^@query:([A-Za-z_][A-Za-z0-9_]*)(?:\((.*)\))?$/);
|
|
94
|
+
if (!m) continue;
|
|
95
|
+
const name = m[1];
|
|
96
|
+
const overrides = overrideKeys(m[2]);
|
|
97
|
+
let entry;
|
|
98
|
+
try {
|
|
99
|
+
entry = resolveQuery(name, screenName, cwd);
|
|
100
|
+
} catch (e: any) {
|
|
101
|
+
warns.push({ scenario: sc.name, message: e?.message || `@query:${name} → cannot resolve query.` });
|
|
102
|
+
continue;
|
|
103
|
+
}
|
|
104
|
+
for (const p of entry.params || []) {
|
|
105
|
+
if (!overrides.has(p) && !topKeys.has(p)) {
|
|
106
|
+
warns.push({ scenario: sc.name, message: `@query:${name} → param "${p}" has no value: not in the annotation and not a top-level test-data key.` });
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
// Catalog-level lint (SELECT-only, params declared/used, datasource present).
|
|
113
|
+
try {
|
|
114
|
+
for (const e of lintCatalog(screenName, null, cwd).errors) warns.push({ message: e });
|
|
115
|
+
} catch {
|
|
116
|
+
/* no catalog → nothing to lint */
|
|
117
|
+
}
|
|
118
|
+
return warns;
|
|
119
|
+
}
|
package/src/harness/parse.ts
CHANGED
|
@@ -30,6 +30,8 @@ export interface ScenarioInfo {
|
|
|
30
30
|
haystack: string; // lowercase name + steps text (for keyword coverage)
|
|
31
31
|
stepsText: string; // lowercase steps ONLY (name excluded) — for claim-proof
|
|
32
32
|
vpId?: string; // raw leading ID token of the title (project's scheme: VP0-001, MS-HP-001, VP-LIST-001)
|
|
33
|
+
casesDataset?: string; // @cases:<dataset> — data-driven; one scenario expands to N row-tests
|
|
34
|
+
queryRefs?: string[]; // named queries referenced by this scenario (inline `query [name]` + @query: tags)
|
|
33
35
|
}
|
|
34
36
|
|
|
35
37
|
/** Format-tolerant: is this token an ID (project's scheme), not a prose word?
|
|
@@ -98,6 +100,14 @@ const PRIORITY_TAGS: Record<string, Priority> = { '@high': 'high', '@normal': 'n
|
|
|
98
100
|
function classifyScenario(sc: ParsedScenario): ScenarioInfo {
|
|
99
101
|
const tags = sc.tags || [];
|
|
100
102
|
const manual = tags.includes('@manual');
|
|
103
|
+
const casesTag = tags.find((t) => t.startsWith('@cases:'));
|
|
104
|
+
const casesDataset = casesTag ? casesTag.slice('@cases:'.length).trim() : undefined;
|
|
105
|
+
// Named-query references: @query:<name> tags + inline `query [name]` step refs.
|
|
106
|
+
const queryRefs = new Set<string>();
|
|
107
|
+
for (const t of tags) if (t.startsWith('@query:')) { const n = t.slice('@query:'.length).trim(); if (n) queryRefs.add(n); }
|
|
108
|
+
for (const step of (sc.steps as ParsedStep[]) || []) {
|
|
109
|
+
for (const m of (step.text || '').matchAll(/\bquery\s+\[([A-Za-z_][A-Za-z0-9_]*)\]/gi)) queryRefs.add(m[1]);
|
|
110
|
+
}
|
|
101
111
|
let priority: Priority = 'unknown';
|
|
102
112
|
for (const t of tags) if (PRIORITY_TAGS[t]) priority = PRIORITY_TAGS[t];
|
|
103
113
|
|
|
@@ -152,6 +162,8 @@ function classifyScenario(sc: ParsedScenario): ScenarioInfo {
|
|
|
152
162
|
haystack: textParts.join(' ').toLowerCase(),
|
|
153
163
|
stepsText: stepTextParts.join(' ').toLowerCase(),
|
|
154
164
|
vpId,
|
|
165
|
+
casesDataset,
|
|
166
|
+
queryRefs: queryRefs.size ? [...queryRefs] : undefined,
|
|
155
167
|
};
|
|
156
168
|
}
|
|
157
169
|
|
|
Binary file
|
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
import * as fs from 'fs';
|
|
16
16
|
import * as path from 'path';
|
|
17
17
|
import * as os from 'os';
|
|
18
|
-
import { loadScenarios } from './parse';
|
|
18
|
+
import { loadScenarios, ScenarioInfo } from './parse';
|
|
19
19
|
|
|
20
20
|
export interface ScriptCheckResult {
|
|
21
21
|
screen: string;
|
|
@@ -146,10 +146,13 @@ export async function runScriptCheck(screenDir: string, screenName: string, flow
|
|
|
146
146
|
}
|
|
147
147
|
|
|
148
148
|
// A. Structural 1:1
|
|
149
|
+
// A @cases scenario emits ONE source test() inside a per-row loop, titled
|
|
150
|
+
// `<name> — ${__row.__label}` — match that literal title, not the bare name.
|
|
151
|
+
const expectedTitle = (s: ScenarioInfo) => (s.casesDataset ? `${s.name} — ${'${'}__row.__label}` : s.name);
|
|
149
152
|
const specTitleSet = new Set(specTitles);
|
|
150
|
-
const
|
|
151
|
-
const missingInSpec = automated.filter((s) => !specTitleSet.has(s
|
|
152
|
-
const extraInSpec = specTitles.filter((t) => !
|
|
153
|
+
const expectedSet = new Set(automated.map(expectedTitle));
|
|
154
|
+
const missingInSpec = automated.filter((s) => !specTitleSet.has(expectedTitle(s))).map((s) => s.name);
|
|
155
|
+
const extraInSpec = specTitles.filter((t) => !expectedSet.has(t));
|
|
153
156
|
const countMatch = committedSpec ? automated.length === specTitles.length : false;
|
|
154
157
|
if (committedSpec && !countMatch) {
|
|
155
158
|
findings.push(`Count mismatch: ${automated.length} automated scenarios vs ${specTitles.length} test() blocks.`);
|
|
@@ -193,7 +196,7 @@ export async function runScriptCheck(screenDir: string, screenName: string, flow
|
|
|
193
196
|
|
|
194
197
|
// C. Anti-bypass / faithfulness
|
|
195
198
|
const { assertionlessTests, hollowSteps } = committedSpec
|
|
196
|
-
? analyzeFaithfulness(specSrc,
|
|
199
|
+
? analyzeFaithfulness(specSrc, expectedSet)
|
|
197
200
|
: { assertionlessTests: [], hollowSteps: [] };
|
|
198
201
|
for (const t of assertionlessTests) {
|
|
199
202
|
findings.push(`BYPASS: test "${t}" has 0 assertions (action-only — proves nothing). The testcase is not really automated.`);
|
|
@@ -15,11 +15,12 @@ Run `sungen challenge --screen <name>` (Bash) and read its report (`.sungen/repo
|
|
|
15
15
|
- `.sungen/reports/<name>-audit.json` — what the gate already measured.
|
|
16
16
|
- Blind-spot patterns — run `sungen blindspot list --prompt` (Bash) and check the suite against each known pattern.
|
|
17
17
|
|
|
18
|
-
##
|
|
18
|
+
## Four critics
|
|
19
19
|
|
|
20
20
|
1. **Coverage critic** — viewpoints that are missing or covered only shallowly; areas over-covered with low value (e.g. many subscription edge cases while cart correctness is thin). Recommend rebalancing, not just adding.
|
|
21
21
|
2. **Business-Depth critic** — scenarios whose **title claims more than the steps prove** (a set/collection asserted by one element; "correct X" asserted by mere visibility). For each, give the exact deep step to add. Confirm or dismiss the deterministic flags from `sungen challenge`.
|
|
22
|
-
3. **
|
|
22
|
+
3. **Data-driven critic** — surface `@cases`-worthy gaps, *spec-independent*: (a) **confirm the deterministic collapse suggestions** (`sungen challenge` flags ≥2 scenarios with the same step skeleton → propose the one `@cases:<dataset>` with the rows + a `case` label each); (b) for any action reaching a backend/logic (login, search, create, an API/error path), propose the **corner/error matrix** as `@cases` rows — *invalid · empty · boundary · injection · duplicate · not-found · unauthorized · malformed · rate-limit* — picking the family that fits the screen. These are the cases a spec/viewpoint usually under-specifies.
|
|
23
|
+
4. **Novelty critic** — 3–5 **non-obvious, valuable** scenarios outside the existing pattern, via risk lenses (double-submit, partial-load, boundary/unusual data, concurrency/back-button, historical incidents). Each must map to a risk or viewpoint and explain why it isn't a duplicate.
|
|
23
24
|
|
|
24
25
|
## Guardrails (hard)
|
|
25
26
|
- **Read-only.** Never edit the feature or any file. You return findings; the QA/orchestrator decides.
|