@sun-asterisk/sungen 3.2.20 → 3.2.21-beta.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/commands/audit.d.ts.map +1 -1
- package/dist/cli/commands/audit.js +8 -0
- package/dist/cli/commands/audit.js.map +1 -1
- package/dist/cli/commands/delivery.d.ts.map +1 -1
- package/dist/cli/commands/delivery.js +9 -0
- package/dist/cli/commands/delivery.js.map +1 -1
- package/dist/cli/commands/trace.d.ts.map +1 -1
- package/dist/cli/commands/trace.js +9 -0
- package/dist/cli/commands/trace.js.map +1 -1
- package/dist/exporters/api-testcase-formatter.d.ts +8 -0
- package/dist/exporters/api-testcase-formatter.d.ts.map +1 -1
- package/dist/exporters/api-testcase-formatter.js +44 -22
- package/dist/exporters/api-testcase-formatter.js.map +1 -1
- package/dist/exporters/matrix/build.d.ts +4 -2
- package/dist/exporters/matrix/build.d.ts.map +1 -1
- package/dist/exporters/matrix/build.js +25 -2
- package/dist/exporters/matrix/build.js.map +1 -1
- package/dist/exporters/matrix/export.d.ts +6 -0
- package/dist/exporters/matrix/export.d.ts.map +1 -1
- package/dist/exporters/matrix/export.js +20 -1
- package/dist/exporters/matrix/export.js.map +1 -1
- package/dist/exporters/matrix/gates.js +7 -1
- package/dist/exporters/matrix/gates.js.map +1 -1
- package/dist/exporters/matrix/render-xlsx.d.ts.map +1 -1
- package/dist/exporters/matrix/render-xlsx.js +15 -0
- package/dist/exporters/matrix/render-xlsx.js.map +1 -1
- package/dist/exporters/matrix/types.d.ts +7 -0
- package/dist/exporters/matrix/types.d.ts.map +1 -1
- package/dist/exporters/matrix/types.js.map +1 -1
- package/dist/exporters/playwright-report-parser.d.ts.map +1 -1
- package/dist/exporters/playwright-report-parser.js +1 -0
- package/dist/exporters/playwright-report-parser.js.map +1 -1
- package/dist/exporters/types.d.ts +2 -0
- package/dist/exporters/types.d.ts.map +1 -1
- package/dist/generators/test-generator/diagnostics.d.ts +6 -1
- package/dist/generators/test-generator/diagnostics.d.ts.map +1 -1
- package/dist/generators/test-generator/diagnostics.js +5 -0
- package/dist/generators/test-generator/diagnostics.js.map +1 -1
- package/dist/generators/test-generator/patterns/index.d.ts.map +1 -1
- package/dist/generators/test-generator/patterns/index.js +27 -19
- package/dist/generators/test-generator/patterns/index.js.map +1 -1
- package/dist/generators/test-generator/step-mapper.d.ts.map +1 -1
- package/dist/generators/test-generator/step-mapper.js +22 -0
- package/dist/generators/test-generator/step-mapper.js.map +1 -1
- package/dist/harness/audit.d.ts +2 -0
- package/dist/harness/audit.d.ts.map +1 -1
- package/dist/harness/audit.js +79 -9
- package/dist/harness/audit.js.map +1 -1
- package/dist/harness/flow-contract.d.ts +71 -0
- package/dist/harness/flow-contract.d.ts.map +1 -0
- package/dist/harness/flow-contract.js +235 -0
- package/dist/harness/flow-contract.js.map +1 -0
- package/dist/harness/flow-plan.d.ts +3 -0
- package/dist/harness/flow-plan.d.ts.map +1 -1
- package/dist/harness/flow-plan.js +6 -2
- package/dist/harness/flow-plan.js.map +1 -1
- package/dist/harness/parse.d.ts +5 -0
- package/dist/harness/parse.d.ts.map +1 -1
- package/dist/harness/parse.js +29 -1
- package/dist/harness/parse.js.map +1 -1
- package/dist/harness/perf.d.ts +40 -0
- package/dist/harness/perf.d.ts.map +1 -0
- package/dist/harness/perf.js +136 -0
- package/dist/harness/perf.js.map +1 -0
- package/dist/harness/script-check.d.ts.map +1 -1
- package/dist/harness/script-check.js +17 -4
- package/dist/harness/script-check.js.map +1 -1
- package/dist/harness/sensors.d.ts.map +1 -1
- package/dist/harness/sensors.js +13 -1
- package/dist/harness/sensors.js.map +1 -1
- package/dist/harness/spec-coverage.d.ts.map +1 -1
- package/dist/harness/spec-coverage.js +29 -5
- package/dist/harness/spec-coverage.js.map +1 -1
- package/dist/orchestrator/templates/ai-src/commands/add-flow.md +41 -3
- package/dist/orchestrator/templates/ai-src/commands/create-test.md +10 -0
- package/dist/orchestrator/templates/ai-src/skills/sungen-tc-generation/SKILL.md +35 -16
- package/dist/orchestrator/templates/qa-context.md +14 -1
- package/package.json +3 -3
- package/src/cli/commands/audit.ts +8 -0
- package/src/cli/commands/delivery.ts +8 -0
- package/src/cli/commands/trace.ts +9 -0
- package/src/exporters/api-testcase-formatter.ts +46 -20
- package/src/exporters/matrix/build.ts +30 -4
- package/src/exporters/matrix/export.ts +28 -1
- package/src/exporters/matrix/gates.ts +7 -1
- package/src/exporters/matrix/render-xlsx.ts +15 -0
- package/src/exporters/matrix/types.ts +7 -0
- package/src/exporters/playwright-report-parser.ts +2 -0
- package/src/exporters/types.ts +2 -0
- package/src/generators/test-generator/diagnostics.ts +6 -1
- package/src/generators/test-generator/patterns/index.ts +30 -24
- package/src/generators/test-generator/step-mapper.ts +22 -0
- package/src/harness/audit.ts +82 -10
- package/src/harness/flow-contract.ts +229 -0
- package/src/harness/flow-plan.ts +10 -3
- package/src/harness/parse.ts +31 -1
- package/src/harness/perf.ts +112 -0
- package/src/harness/script-check.ts +18 -4
- package/src/harness/sensors.ts +13 -1
- package/src/harness/spec-coverage.ts +26 -4
- package/src/orchestrator/templates/ai-src/commands/add-flow.md +41 -3
- package/src/orchestrator/templates/ai-src/commands/create-test.md +10 -0
- package/src/orchestrator/templates/ai-src/skills/sungen-tc-generation/SKILL.md +35 -16
- package/src/orchestrator/templates/qa-context.md +14 -1
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Performance budgets — config + percentile math + the per-unit verdict. (#569)
|
|
3
|
+
*
|
|
4
|
+
* A flow's regression value includes "still fast enough": after a lib/framework
|
|
5
|
+
* upgrade the main journeys must not only pass but hold their response-time
|
|
6
|
+
* budget. Sungen had no perf concept at all — the Playwright JSON parser even
|
|
7
|
+
* dropped the `duration` field Playwright already emits on every result.
|
|
8
|
+
*
|
|
9
|
+
* Scope discipline:
|
|
10
|
+
* - This is config + measurement + report over runs sungen already makes.
|
|
11
|
+
* Real load tests stay @manual:M8 → a dedicated tool.
|
|
12
|
+
* - The AUDIT never reads it: the quality score is documented as a pure
|
|
13
|
+
* function of the design artifacts ("reads no test-results, live page, or
|
|
14
|
+
* clock"). Perf reports where runs are already read — `sungen delivery`
|
|
15
|
+
* and the dashboard. Advisory: a blown budget never fails the design gate.
|
|
16
|
+
*
|
|
17
|
+
* Config: qa/perf.yaml
|
|
18
|
+
* percentile: p75 # default p75 — "≥75% of runs meet the budget"
|
|
19
|
+
* defaults:
|
|
20
|
+
* scenario_ms: 30000 # whole-scenario wall clock (Playwright duration)
|
|
21
|
+
* page_load_ms: 3000 # Phase B — needs per-transition runtime timing
|
|
22
|
+
* transition_ms: 2000 # Phase B
|
|
23
|
+
* units:
|
|
24
|
+
* place-order: { scenario_ms: 20000 }
|
|
25
|
+
*/
|
|
26
|
+
import * as fs from 'fs';
|
|
27
|
+
import * as path from 'path';
|
|
28
|
+
import { parse as parseYaml } from 'yaml';
|
|
29
|
+
import { readTextFile } from './read-text';
|
|
30
|
+
|
|
31
|
+
export interface PerfConfig {
|
|
32
|
+
/** 0..100 — e.g. 75 for p75. */
|
|
33
|
+
percentile: number;
|
|
34
|
+
defaults: Record<string, number>;
|
|
35
|
+
units: Record<string, Record<string, number>>;
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
export interface PerfVerdict {
|
|
39
|
+
unit: string;
|
|
40
|
+
metric: string; // 'scenario_ms' today; page_load_ms/transition_ms in Phase B
|
|
41
|
+
percentile: number; // 75
|
|
42
|
+
budgetMs: number;
|
|
43
|
+
measuredMs: number; // the pXX of the observed durations
|
|
44
|
+
samples: number;
|
|
45
|
+
pass: boolean;
|
|
46
|
+
/** Titles of the slowest offenders (only when failing), for the report. */
|
|
47
|
+
slowest: Array<{ title: string; ms: number }>;
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
export function perfConfigPath(projectRoot: string): string {
|
|
51
|
+
return path.join(projectRoot, 'qa', 'perf.yaml');
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
/** Absent file → null (perf reporting is opt-in; nothing changes until configured). */
|
|
55
|
+
export function loadPerfConfig(projectRoot: string): PerfConfig | null {
|
|
56
|
+
const p = perfConfigPath(projectRoot);
|
|
57
|
+
if (!fs.existsSync(p)) return null;
|
|
58
|
+
let raw: Record<string, unknown>;
|
|
59
|
+
try { raw = parseYaml(readTextFile(p)) as Record<string, unknown>; } catch { return null; }
|
|
60
|
+
if (!raw || typeof raw !== 'object') return null;
|
|
61
|
+
const pctRaw = String(raw.percentile ?? 'p75').toLowerCase().replace(/^p/, '');
|
|
62
|
+
const percentile = Math.min(100, Math.max(1, Number(pctRaw) || 75));
|
|
63
|
+
const num = (o: unknown): Record<string, number> => {
|
|
64
|
+
const out: Record<string, number> = {};
|
|
65
|
+
if (o && typeof o === 'object') {
|
|
66
|
+
for (const [k, v] of Object.entries(o as Record<string, unknown>)) {
|
|
67
|
+
const n = Number(v);
|
|
68
|
+
if (Number.isFinite(n) && n > 0) out[k] = n;
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
return out;
|
|
72
|
+
};
|
|
73
|
+
const units: Record<string, Record<string, number>> = {};
|
|
74
|
+
if (raw.units && typeof raw.units === 'object') {
|
|
75
|
+
for (const [u, o] of Object.entries(raw.units as Record<string, unknown>)) units[u] = num(o);
|
|
76
|
+
}
|
|
77
|
+
return { percentile, defaults: num(raw.defaults), units };
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
/**
|
|
81
|
+
* Nearest-rank percentile (ceil), the standard "≥pXX of samples meet the budget"
|
|
82
|
+
* reading: p75 of [a…] is the value at ceil(0.75·n) in the sorted list. One
|
|
83
|
+
* sample → that sample. Deterministic, no interpolation.
|
|
84
|
+
*/
|
|
85
|
+
export function percentileOf(p: number, values: number[]): number {
|
|
86
|
+
if (values.length === 0) return 0;
|
|
87
|
+
const sorted = [...values].sort((a, b) => a - b);
|
|
88
|
+
const rank = Math.min(sorted.length, Math.max(1, Math.ceil((p / 100) * sorted.length)));
|
|
89
|
+
return sorted[rank - 1];
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
/** Budget for a metric on a unit: per-unit override, else defaults, else none. */
|
|
93
|
+
export function budgetFor(config: PerfConfig, unit: string, metric: string): number | undefined {
|
|
94
|
+
return config.units[unit]?.[metric] ?? config.defaults[metric];
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
/**
|
|
98
|
+
* The scenario_ms verdict for one unit's run. `durations` = per-test wall-clock ms
|
|
99
|
+
* (a @cases scenario contributes one sample per row-test — each is a real run).
|
|
100
|
+
*/
|
|
101
|
+
export function perfVerdict(
|
|
102
|
+
config: PerfConfig,
|
|
103
|
+
unit: string,
|
|
104
|
+
samples: Array<{ title: string; ms: number }>,
|
|
105
|
+
): PerfVerdict | null {
|
|
106
|
+
const budgetMs = budgetFor(config, unit, 'scenario_ms');
|
|
107
|
+
if (budgetMs === undefined || samples.length === 0) return null;
|
|
108
|
+
const measuredMs = percentileOf(config.percentile, samples.map((s) => s.ms));
|
|
109
|
+
const pass = measuredMs <= budgetMs;
|
|
110
|
+
const slowest = pass ? [] : [...samples].sort((a, b) => b.ms - a.ms).slice(0, 3);
|
|
111
|
+
return { unit, metric: 'scenario_ms', percentile: config.percentile, budgetMs, measuredMs, samples: samples.length, pass, slowest };
|
|
112
|
+
}
|
|
@@ -43,7 +43,10 @@ export function extractTestBlocks(specSrc: string): { title: string; body: strin
|
|
|
43
43
|
const lines = specSrc.split('\n');
|
|
44
44
|
const blocks: { title: string; body: string[] }[] = [];
|
|
45
45
|
for (let i = 0; i < lines.length; i++) {
|
|
46
|
-
|
|
46
|
+
// Same escape-aware title match as extractTestTitles — with `[^'"`]+` a
|
|
47
|
+
// quote-bearing title extracted truncated (or not at all), missed the
|
|
48
|
+
// automatedTitles lookup, and silently dropped out of the BYPASS scan.
|
|
49
|
+
const m = lines[i].match(/\btest(?:\.(?:only|skip|fixme))?\(\s*(['"`])((?:\\.|(?!\1).)+)\1/);
|
|
47
50
|
if (!m) continue;
|
|
48
51
|
let depth = 0, started = false;
|
|
49
52
|
const body: string[] = [];
|
|
@@ -52,7 +55,7 @@ export function extractTestBlocks(specSrc: string): { title: string; body: strin
|
|
|
52
55
|
body.push(lines[j]);
|
|
53
56
|
if (started && depth <= 0) break;
|
|
54
57
|
}
|
|
55
|
-
blocks.push({ title: m[2].trim(), body });
|
|
58
|
+
blocks.push({ title: unescapeJsLiteral(m[2]).trim(), body });
|
|
56
59
|
}
|
|
57
60
|
return blocks;
|
|
58
61
|
}
|
|
@@ -104,13 +107,24 @@ function extractTestTitles(specSrc: string): string[] {
|
|
|
104
107
|
// Count real test cases only. Web/Playwright specs use test(...) (+ .only/.skip/.fixme);
|
|
105
108
|
// mobile/Appium specs use mocha it(...) (#429 — matching only test() reported EVERY mobile
|
|
106
109
|
// scenario as MISSING). Exclude describe / hooks (not test cases).
|
|
110
|
+
// The content match is escape-aware and excludes ONLY the actual delimiter (same
|
|
111
|
+
// pattern as spec-parser.ts): the old `[^'"`]+` class truncated a title at its
|
|
112
|
+
// escaped same-quote (`lottery\'s` → EXTRA `lottery\` + the real title MISSING)
|
|
113
|
+
// and dropped a title containing an other-quote (`the "x" case`) entirely — so
|
|
114
|
+
// in-sync specs of quote-bearing scenarios failed the CI gate (field report 3.2.19).
|
|
107
115
|
const titles: string[] = [];
|
|
108
|
-
const re = /\b(?:test|it)(?:\.(?:only|skip|fixme))?\(\s*(['"`])(
|
|
116
|
+
const re = /\b(?:test|it)(?:\.(?:only|skip|fixme))?\(\s*(['"`])((?:\\.|(?!\1).)+)\1/g;
|
|
109
117
|
let m: RegExpExecArray | null;
|
|
110
|
-
while ((m = re.exec(specSrc))) titles.push(m[2].trim());
|
|
118
|
+
while ((m = re.exec(specSrc))) titles.push(unescapeJsLiteral(m[2]).trim());
|
|
111
119
|
return titles;
|
|
112
120
|
}
|
|
113
121
|
|
|
122
|
+
/** Undo the compiler's escapeQuotes escaping (\' \\ \n \r \t) so extracted titles
|
|
123
|
+
* compare against the raw scenario names from the .feature. */
|
|
124
|
+
function unescapeJsLiteral(s: string): string {
|
|
125
|
+
return s.replace(/\\(.)/g, (_m, c: string) => (c === 'n' ? '\n' : c === 'r' ? '\r' : c === 't' ? '\t' : c));
|
|
126
|
+
}
|
|
127
|
+
|
|
114
128
|
function normalize(src: string): string {
|
|
115
129
|
return src
|
|
116
130
|
.split('\n')
|
package/src/harness/sensors.ts
CHANGED
|
@@ -33,10 +33,22 @@ const BUCKET_ORDER: Array<[string, string[]]> = [
|
|
|
33
33
|
];
|
|
34
34
|
const BUCKETS: Record<string, string[]> = Object.fromEntries(BUCKET_ORDER);
|
|
35
35
|
|
|
36
|
+
// Flow journey-phase categories (FL-HP-001, FL-ER-002 …). Matched on exact SEGMENTS,
|
|
37
|
+
// never by containment — 'SHOP'.includes('HP') is true, which is exactly the kind of
|
|
38
|
+
// false hit substring matching would produce for two-letter phase tokens. (#569)
|
|
39
|
+
const PHASE_BUCKETS: Record<string, string> = {
|
|
40
|
+
HP: 'business-core', // happy path = the business goal itself
|
|
41
|
+
ER: 'validation-security', // error recovery (validation must not trap the journey)
|
|
42
|
+
EH: 'validation-security', // guards & leakage (direct access, back, refresh)
|
|
43
|
+
};
|
|
44
|
+
|
|
36
45
|
/** Classify a VP category into a balance bucket by keyword containment + precedence (H1). */
|
|
37
46
|
export function bucketForCategory(category: string | undefined): string {
|
|
38
47
|
const cat = (category || '').toUpperCase();
|
|
39
48
|
if (!cat) return 'other';
|
|
49
|
+
for (const seg of cat.split('-')) {
|
|
50
|
+
if (PHASE_BUCKETS[seg]) return PHASE_BUCKETS[seg];
|
|
51
|
+
}
|
|
40
52
|
for (const [bucket, kws] of BUCKET_ORDER) {
|
|
41
53
|
if (kws.some((k) => cat.includes(k))) return bucket;
|
|
42
54
|
}
|
|
@@ -351,7 +363,7 @@ export function flowRegressionDepth(scenarios: ScenarioInfo[]): FlowDepthResult
|
|
|
351
363
|
// 1. Count/quantity proof — a row count or item quantity, not just presence of a row.
|
|
352
364
|
const countProof = any(/\b(quantity|qty|two (?:rows|lines|cart)|row count|count column|number of items|one[_ ]row|two[_ ]rows|qty[_ ])/i);
|
|
353
365
|
// 2. Teardown — removes the item and verifies the empty/zero state (the inverse operation).
|
|
354
|
-
const teardown = any(/\b(remove|delete|clear)
|
|
366
|
+
const teardown = any(/\b(remove|delete|clear)(?:s|d|ed|ing)?\b/i) && any(/\b(empty|emptied|no items|zero|removed|cleared|0 items)\b/i);
|
|
355
367
|
// 3. Multi-source — the cart is fed from >1 source (the main list AND a recommended/related rail).
|
|
356
368
|
const multiSource = any(/\b(recommended|related|you may also|suggest)\b/i) && addsToCart;
|
|
357
369
|
|
|
@@ -66,21 +66,43 @@ export function parseSpecClauses(specPath: string): { frs: FrClause[]; valRows:
|
|
|
66
66
|
if (!fs.existsSync(specPath)) return { frs: [], valRows: [] };
|
|
67
67
|
const lines = readTextFile(specPath).split('\n');
|
|
68
68
|
|
|
69
|
+
// Requirement ids follow the PROJECT's scheme, not ours (#572): `**FR-1**:` is one
|
|
70
|
+
// convention among many — a real spec declared ~30 MUST clauses as `` `REQ-SRCH-001`: ``
|
|
71
|
+
// and the FR-locked pattern returned zero, so the MUST-coverage gate never ran and the
|
|
72
|
+
// specFR axis was excluded "for lack of evidence" that was sitting right there. Same
|
|
73
|
+
// silent-failure class as the CRLF parsers, same id-scheme-tolerance lesson as delivery.
|
|
74
|
+
//
|
|
75
|
+
// A declaration is: line-leading (optionally bulleted / bold / backticked) `<ID>:` where
|
|
76
|
+
// the id ends in a number. Table rows are EXCLUDED — traceability tables cite requirement
|
|
77
|
+
// ids without declaring them — and so are prefixes that are never requirements
|
|
78
|
+
// (test cases, viewpoints, known-defect records, data-factory checks, flows, delivery items).
|
|
79
|
+
const NON_REQUIREMENT_PREFIX = /^(TC|VP|KD|CHK|FL|DI)-/i;
|
|
69
80
|
const frs: FrClause[] = [];
|
|
81
|
+
const seen = new Set<string>();
|
|
70
82
|
for (const line of lines) {
|
|
71
|
-
|
|
72
|
-
|
|
83
|
+
if (/^\s*\|/.test(line)) continue; // table row = citation, not declaration
|
|
84
|
+
const m = line.match(/^\s*(?:[-*+]\s+)?[*_`]*([A-Z][A-Z0-9]*(?:-[A-Z0-9]+)*-\d+[a-zA-Z]?)[*_`]*\s*:\s*(.+)$/);
|
|
85
|
+
if (!m || NON_REQUIREMENT_PREFIX.test(m[1])) continue;
|
|
86
|
+
const id = m[1].toUpperCase();
|
|
87
|
+
if (seen.has(id)) continue; // first declaration wins
|
|
88
|
+
seen.add(id);
|
|
89
|
+
frs.push({ id, text: m[2].replace(/\*\*/g, '').trim(), modality: modalityOf(m[2]) });
|
|
73
90
|
}
|
|
74
91
|
|
|
75
92
|
// Validation Rules table: a row carries a Constraint, a Trigger cell, and (often) a code.
|
|
93
|
+
// A "Trigger" column alone is NOT enough to claim the table (#578): a screen-STATES table
|
|
94
|
+
// ("State ID | Trigger | URL/heading oracle | …") uses Trigger for the user action that
|
|
95
|
+
// enters the state, and reading it as validation rows invented three gate-relevant
|
|
96
|
+
// TRIGGER-UNCOVERED gaps on a real spec. The table must also name a Constraint/Rule/
|
|
97
|
+
// Validation column — the thing a validation row is ABOUT.
|
|
76
98
|
const valRows: ValRow[] = [];
|
|
77
99
|
let cTrigger = -1, cConstraint = -1, cCode = -1, inTable = false;
|
|
78
100
|
for (const raw of lines) {
|
|
79
101
|
const line = raw.trim();
|
|
80
|
-
if (line.startsWith('|') && /\btrigger\b/i.test(line) && cTrigger < 0) {
|
|
102
|
+
if (line.startsWith('|') && /\btrigger\b/i.test(line) && /\b(constraint|rule|validation)\b/i.test(line) && cTrigger < 0) {
|
|
81
103
|
const cells = line.split('|').map((c) => c.trim());
|
|
82
104
|
cTrigger = cells.findIndex((c) => /^trigger$/i.test(c));
|
|
83
|
-
cConstraint = cells.findIndex((c) => /constraint/i.test(c));
|
|
105
|
+
cConstraint = cells.findIndex((c) => /constraint|rule|validation/i.test(c));
|
|
84
106
|
cCode = cells.findIndex((c) => /code/i.test(c));
|
|
85
107
|
inTable = cTrigger >= 0;
|
|
86
108
|
continue;
|
|
@@ -86,15 +86,52 @@ qa/flows/${input:flow}/
|
|
|
86
86
|
└── ui/ # Screenshots, mockups
|
|
87
87
|
```
|
|
88
88
|
|
|
89
|
-
### 1a.
|
|
89
|
+
### 1a. Define the flow's BOUNDARY, then its screens
|
|
90
90
|
|
|
91
|
-
|
|
91
|
+
A flow is the **smallest complete business action chain**: one clear trigger ending in ONE
|
|
92
|
+
observable, valuable outcome. Before asking for screens, walk this checklist with the user —
|
|
93
|
+
if 1, 3 or 8 fails, propose SPLITTING into separate flows:
|
|
94
|
+
|
|
95
|
+
1. Exactly **one business goal**? (cart correctness + category filtering = two flows)
|
|
96
|
+
2. A clear **trigger** and precondition?
|
|
97
|
+
3. **One observable final outcome**? (a final assertion you can write in one sentence)
|
|
98
|
+
4. Is that outcome **valuable to the actor**? (an order placed, a password reset — not "a page rendered")
|
|
99
|
+
5. Is **every step necessary** for that outcome?
|
|
100
|
+
6. Are all steps at the **same business abstraction**?
|
|
101
|
+
7. Are optional/error branches **phases of this goal** (ER/EH), not new goals?
|
|
102
|
+
8. Does **no segment** form an independently valuable flow on its own?
|
|
103
|
+
9. Can you write **a single clear final assertion**?
|
|
104
|
+
10. Can you name it "**Verb + outcome**"? (`place-order`, `reset-password` — not `cart-and-filter`)
|
|
105
|
+
|
|
106
|
+
Then ask: "Which screens does this flow visit, in order? (e.g., login → dashboard → award-form → confirmation)"
|
|
92
107
|
|
|
93
108
|
Record the screen list — you will need it for:
|
|
94
109
|
- Filling `spec.md` (Step 3)
|
|
95
110
|
- Suggesting `[Screen:Element]` namespace prefixes
|
|
96
111
|
- Capturing visuals per screen (Step 2)
|
|
97
112
|
|
|
113
|
+
### 1b. Author the Flow Contract (`requirements/flow-contract.yaml`)
|
|
114
|
+
|
|
115
|
+
Write the answers down as the flow's contract — `sungen audit` scores the flow **against it**
|
|
116
|
+
(the `flowCoverage` axis: HP/ER/EH journey phases; `FLOW-OUTCOME-UNPROVEN` when no automated
|
|
117
|
+
scenario asserts data on the outcome screen; `FLOW-SCOPE-CREEP` when scenarios never touch it):
|
|
118
|
+
|
|
119
|
+
```yaml
|
|
120
|
+
goal: "Place an order for a product added from home" # Verb + outcome
|
|
121
|
+
actor: user
|
|
122
|
+
trigger: "Add a product to the cart from the home featured list"
|
|
123
|
+
precondition: "A registered account; an empty cart"
|
|
124
|
+
outcome:
|
|
125
|
+
screen: checkout # the [Screen:...] namespace carrying the final proof
|
|
126
|
+
assertion: "The confirmation shows the order number and the paid total"
|
|
127
|
+
value: "The customer has paid; the shop has a new order"
|
|
128
|
+
phases: [HP, ER, EH] # journey phases (default); add UI only if the flow owns UI states
|
|
129
|
+
stateful: cart # the mutated collection, if any — enables regression-depth dims
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
**A filled contract is an INPUT to generation — never an output.** Like `test-viewpoint.md`,
|
|
133
|
+
generation must not rewrite it to match what was generated; disagree → propose the diff and ask.
|
|
134
|
+
|
|
98
135
|
### 2. Capture visual source
|
|
99
136
|
|
|
100
137
|
**Mobile path** (`platform: mobile`):
|
|
@@ -187,7 +224,8 @@ If user picks `/sungen:create-test`, **you MUST use the Skill tool** to invoke i
|
|
|
187
224
|
- Test data namespaced by phase: `login.email`, `submission.nominee`
|
|
188
225
|
- `@flow` tag required at feature level
|
|
189
226
|
- `Background:` should only contain the starting navigation — the URL path (web) or the `--reach` nav recipe (mobile)
|
|
190
|
-
- Each scenario = one phase of the journey
|
|
227
|
+
- Each scenario = one phase of the journey; ids are `FL-<PHASE>-NNN` (`HP`/`ER`/`EH`, optional `UI`)
|
|
228
|
+
- One flow = ONE business goal with ONE observable outcome (`requirements/flow-contract.yaml`) — a segment with its own value is its own flow
|
|
191
229
|
{{#cap parallel-subagents}}
|
|
192
230
|
- Mobile flows are tagged `@platform:mobile` and run via `/sungen:run-test <flow>` (WebdriverIO, not Playwright)
|
|
193
231
|
{{/cap}}
|
|
@@ -6,6 +6,16 @@ order: 20
|
|
|
6
6
|
claude-tools: "Read, Grep, Bash, Glob, Write, AskUserQuestion, Skill, mcp__playwright__browser_navigate, mcp__playwright__browser_snapshot, mcp__playwright__browser_take_screenshot"
|
|
7
7
|
copilot-tools: "[vscode, execute, read, agent, edit, search, web, browser, todo, 'playwright/*']"
|
|
8
8
|
codex-trigger: "Run when the user asks to CREATE, generate, write, or author test cases / a .feature file for a screen or flow. Step 2 (after add-screen/add-flow, before run-test). Do NOT use for executing, running, or compiling existing tests."
|
|
9
|
+
---
|
|
10
|
+
## ⛔ HARD RULE — the run's LAST action is the next-step hand-back
|
|
11
|
+
|
|
12
|
+
A create-test run is NOT finished when the files are written or the audit prints. The final
|
|
13
|
+
action of EVERY run — success, partial, or aborted — is the next-step hand-back
|
|
14
|
+
({{#cap parallel-subagents}}an `AskUserQuestion` offering the next actions{{/cap}}{{^cap parallel-subagents}}a numbered list of next-action choices{{/cap}};
|
|
15
|
+
see "Finish — always hand the next step back" at the end of this file). Ending with a prose
|
|
16
|
+
summary and no choices is a broken run: the operator is left guessing. This holds no matter
|
|
17
|
+
how long the generation/repair loop ran.
|
|
18
|
+
|
|
9
19
|
---
|
|
10
20
|
{{#cap parallel-subagents}}
|
|
11
21
|
## ⛔ HARD RULE — No Figma MCP when PAT data exists
|
|
@@ -605,22 +605,40 @@ error:
|
|
|
605
605
|
|
|
606
606
|
> **Auto-detect**: if path is `qa/flows/<name>/` → use this section. Skip Steps 1–4 above.
|
|
607
607
|
|
|
608
|
+
**Read `requirements/flow-contract.yaml` FIRST — it is the flow's boundary and the yardstick
|
|
609
|
+
`sungen audit` scores the flow against** (`flowCoverage` axis = journey phases HP/ER/EH automated;
|
|
610
|
+
`FLOW-OUTCOME-UNPROVEN`; `FLOW-SCOPE-CREEP`). No contract yet → author it with the user via the
|
|
611
|
+
boundary checklist in `add-flow` (one business goal · clear trigger · ONE observable outcome
|
|
612
|
+
valuable to the actor · name = "Verb + outcome"), THEN generate. **A filled contract is an INPUT —
|
|
613
|
+
never rewrite it to match your output** (same rule as `test-viewpoint.md`).
|
|
614
|
+
|
|
608
615
|
| Aspect | Screen | Flow |
|
|
609
616
|
|---|---|---|
|
|
610
|
-
| Section focus | UI patterns per section | Journey phases
|
|
617
|
+
| Section focus | UI patterns per section | Journey phases toward ONE declared outcome |
|
|
611
618
|
| Selector format | `[Element]` | `[Screen:Element]` (namespaced) |
|
|
612
619
|
| Test data keys | `{{variable}}` | `{{phase.variable}}` |
|
|
613
620
|
| Feature tag | `@auto` / `@smoke` etc. | `@flow` (required) |
|
|
614
|
-
|
|
|
621
|
+
| Scenario ids | `VP-<CATEGORY>-NNN` | `FL-<PHASE>-NNN` — phases: `HP` (happy path), `ER` (error recovery), `EH` (guards), `UI` (journey UI states, optional) |
|
|
615
622
|
|
|
616
|
-
**Scenarios to generate:**
|
|
623
|
+
**Scenarios to generate — every phase demanded by the contract, automated:**
|
|
617
624
|
|
|
618
|
-
|
|
|
619
|
-
|
|
620
|
-
|
|
|
621
|
-
|
|
|
622
|
-
|
|
|
623
|
-
| Cross-screen
|
|
625
|
+
| Phase | What to test | Scoring |
|
|
626
|
+
|---|---|---|
|
|
627
|
+
| `FL-HP` happy path | The complete journey ending in the contract's `outcome.assertion` — an AUTOMATED **data** assertion on `outcome.screen` (an order number, a summed total — not just "page visible"). This scenario is WHY the flow exists: it is the regression proof after a lib/framework upgrade. | uncovered → `flowCoverage` drops + `FLOW-OUTCOME-UNPROVEN` |
|
|
628
|
+
| `FL-ER` error recovery | Invalid input mid-flow → error shown → fix → the journey still completes. Validation must not trap the journey. | uncovered → `flowCoverage` drops |
|
|
629
|
+
| `FL-EH` guards | Direct URL access without the precondition · browser back · refresh · expired context — each ends in a safe observable state. | uncovered → `flowCoverage` drops |
|
|
630
|
+
| Cross-screen handoff | After every screen transition, assert the CARRIED state on the new screen (the added product's name in the cart, the email echoed on the sent screen). | blind tails cap `businessDepth` (`FLOW-HANDOFF-SHALLOW`) |
|
|
631
|
+
| Stateful regression (when `stateful:` declared) | Count/quantity proof · teardown (remove → empty) · multi-source add. | missing dims cap `businessDepth` (`FLOW-DEPTH`) |
|
|
632
|
+
|
|
633
|
+
**Boundary discipline while generating:** every scenario must serve the contract's goal. A scenario
|
|
634
|
+
that never touches `outcome.screen` and is not a guard (`EH`) or error-recovery (`ER`) belongs in a
|
|
635
|
+
DIFFERENT flow — propose the split instead of writing it here (`FLOW-SCOPE-CREEP` will flag it).
|
|
636
|
+
Auth persistence across transitions is part of `EH` unless the project declares it its own phase.
|
|
637
|
+
|
|
638
|
+
**Manual in flows**: always `@manual:Mx` with the reason code — bare `@manual` is flagged
|
|
639
|
+
(`MANUAL-CODE-MISSING`) because the capability planner cannot route it. Typical flow deferrals:
|
|
640
|
+
inbox/mail oracle → `M5`, network-request count → `M3`, context expiry control → `M7`. A
|
|
641
|
+
cross-screen scenario inside the flow's own goal is NOT manual — automate it here.
|
|
624
642
|
|
|
625
643
|
```gherkin
|
|
626
644
|
@flow @auth:user
|
|
@@ -630,19 +648,20 @@ Feature: Award Submission Flow
|
|
|
630
648
|
Given User is on [Login] page
|
|
631
649
|
|
|
632
650
|
@high
|
|
633
|
-
Scenario:
|
|
651
|
+
Scenario: FL-HP-001 A signed-in user's nomination is submitted and confirmed
|
|
634
652
|
When User fill [Login:Email] field with {{login.email}}
|
|
635
653
|
And User fill [Login:Password] field with {{login.password}}
|
|
636
654
|
And User click [Login:Submit] button
|
|
637
655
|
Then User see [Dashboard] page
|
|
638
|
-
|
|
639
|
-
@high
|
|
640
|
-
Scenario: User submits nomination
|
|
641
656
|
When User click [Dashboard:Awards] link
|
|
642
|
-
|
|
643
|
-
When User fill [Awards:Nominee] field with {{submission.nominee}}
|
|
657
|
+
And User fill [Awards:Nominee] field with {{submission.nominee}}
|
|
644
658
|
And User click [Awards:Submit] button
|
|
645
|
-
Then User see {{success_message}}
|
|
659
|
+
Then User see [Awards:Success Message] text with {{success_message}}
|
|
660
|
+
|
|
661
|
+
@high
|
|
662
|
+
Scenario: FL-EH-001 Direct access to the award form without login redirects to login
|
|
663
|
+
When User go to [Awards] page
|
|
664
|
+
Then User see [Login] page
|
|
646
665
|
```
|
|
647
666
|
|
|
648
667
|
```yaml
|
|
@@ -45,8 +45,21 @@ Example:
|
|
|
45
45
|
|
|
46
46
|
## Testing Strategy
|
|
47
47
|
|
|
48
|
+
Machine-readable intent — `sungen audit` reads these keys (Intent Profile). Values here are
|
|
49
|
+
live even when the surrounding text changes; an invalid value silently falls back to the default.
|
|
50
|
+
|
|
51
|
+
focus: functional
|
|
52
|
+
<!-- focus: functional | e-commerce | security | smoke — drives the audit's depth threshold -->
|
|
53
|
+
|
|
54
|
+
risk_tier: normal
|
|
55
|
+
<!-- risk_tier: high | normal | low -->
|
|
56
|
+
|
|
57
|
+
To silence driver suggestions in audit findings, add a line: capability_suggestions with value off.
|
|
58
|
+
|
|
48
59
|
**Focus areas** — what to cover thoroughly:
|
|
49
|
-
<!--
|
|
60
|
+
<!-- Prose for humans; the parseable value is the `focus:` key above.
|
|
61
|
+
Response-time budgets are NOT a focus value — declare them in qa/perf.yaml
|
|
62
|
+
(percentile + scenario_ms budgets; reported by `sungen delivery`). -->
|
|
50
63
|
<!-- Example: functional, security -->
|
|
51
64
|
|
|
52
65
|
**Mandatory coverage:**
|