@sun-asterisk/sungen 3.2.21 → 3.2.22-beta.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. package/dist/cli/commands/audit.d.ts.map +1 -1
  2. package/dist/cli/commands/audit.js +8 -0
  3. package/dist/cli/commands/audit.js.map +1 -1
  4. package/dist/cli/commands/delivery.d.ts +3 -0
  5. package/dist/cli/commands/delivery.d.ts.map +1 -1
  6. package/dist/cli/commands/delivery.js +15 -0
  7. package/dist/cli/commands/delivery.js.map +1 -1
  8. package/dist/cli/commands/inspect.d.ts +41 -0
  9. package/dist/cli/commands/inspect.d.ts.map +1 -0
  10. package/dist/cli/commands/inspect.js +134 -0
  11. package/dist/cli/commands/inspect.js.map +1 -0
  12. package/dist/cli/commands/trace.d.ts.map +1 -1
  13. package/dist/cli/commands/trace.js +9 -0
  14. package/dist/cli/commands/trace.js.map +1 -1
  15. package/dist/cli/index.js +2 -0
  16. package/dist/cli/index.js.map +1 -1
  17. package/dist/exporters/matrix/build.d.ts +10 -0
  18. package/dist/exporters/matrix/build.d.ts.map +1 -1
  19. package/dist/exporters/matrix/build.js +38 -0
  20. package/dist/exporters/matrix/build.js.map +1 -1
  21. package/dist/exporters/matrix/export.d.ts.map +1 -1
  22. package/dist/exporters/matrix/export.js +11 -0
  23. package/dist/exporters/matrix/export.js.map +1 -1
  24. package/dist/exporters/matrix/render-xlsx.d.ts.map +1 -1
  25. package/dist/exporters/matrix/render-xlsx.js +44 -1
  26. package/dist/exporters/matrix/render-xlsx.js.map +1 -1
  27. package/dist/exporters/matrix/types.d.ts +10 -0
  28. package/dist/exporters/matrix/types.d.ts.map +1 -1
  29. package/dist/exporters/matrix/types.js.map +1 -1
  30. package/dist/exporters/playwright-report-parser.d.ts.map +1 -1
  31. package/dist/exporters/playwright-report-parser.js +1 -0
  32. package/dist/exporters/playwright-report-parser.js.map +1 -1
  33. package/dist/exporters/types.d.ts +2 -0
  34. package/dist/exporters/types.d.ts.map +1 -1
  35. package/dist/generators/test-generator/diagnostics.d.ts +6 -1
  36. package/dist/generators/test-generator/diagnostics.d.ts.map +1 -1
  37. package/dist/generators/test-generator/diagnostics.js +5 -0
  38. package/dist/generators/test-generator/diagnostics.js.map +1 -1
  39. package/dist/generators/test-generator/patterns/index.d.ts.map +1 -1
  40. package/dist/generators/test-generator/patterns/index.js +27 -19
  41. package/dist/generators/test-generator/patterns/index.js.map +1 -1
  42. package/dist/generators/test-generator/step-mapper.d.ts.map +1 -1
  43. package/dist/generators/test-generator/step-mapper.js +22 -0
  44. package/dist/generators/test-generator/step-mapper.js.map +1 -1
  45. package/dist/harness/audit.d.ts +2 -0
  46. package/dist/harness/audit.d.ts.map +1 -1
  47. package/dist/harness/audit.js +101 -10
  48. package/dist/harness/audit.js.map +1 -1
  49. package/dist/harness/flow-contract.d.ts +87 -0
  50. package/dist/harness/flow-contract.d.ts.map +1 -0
  51. package/dist/harness/flow-contract.js +259 -0
  52. package/dist/harness/flow-contract.js.map +1 -0
  53. package/dist/harness/flow-plan.d.ts +3 -0
  54. package/dist/harness/flow-plan.d.ts.map +1 -1
  55. package/dist/harness/flow-plan.js +6 -2
  56. package/dist/harness/flow-plan.js.map +1 -1
  57. package/dist/harness/parse.d.ts +5 -0
  58. package/dist/harness/parse.d.ts.map +1 -1
  59. package/dist/harness/parse.js +29 -1
  60. package/dist/harness/parse.js.map +1 -1
  61. package/dist/harness/perf.d.ts +40 -0
  62. package/dist/harness/perf.d.ts.map +1 -0
  63. package/dist/harness/perf.js +136 -0
  64. package/dist/harness/perf.js.map +1 -0
  65. package/dist/harness/sensors.d.ts.map +1 -1
  66. package/dist/harness/sensors.js +13 -1
  67. package/dist/harness/sensors.js.map +1 -1
  68. package/dist/harness/spec-coverage.d.ts +8 -0
  69. package/dist/harness/spec-coverage.d.ts.map +1 -1
  70. package/dist/harness/spec-coverage.js +60 -6
  71. package/dist/harness/spec-coverage.js.map +1 -1
  72. package/dist/orchestrator/templates/ai-src/commands/add-flow.md +51 -3
  73. package/dist/orchestrator/templates/ai-src/commands/create-test.md +10 -0
  74. package/dist/orchestrator/templates/ai-src/commands/run-test.md +23 -0
  75. package/dist/orchestrator/templates/ai-src/skills/sungen-tc-generation/SKILL.md +60 -16
  76. package/dist/orchestrator/templates/qa-context.md +14 -1
  77. package/package.json +3 -3
  78. package/src/cli/commands/audit.ts +8 -0
  79. package/src/cli/commands/delivery.ts +14 -2
  80. package/src/cli/commands/inspect.ts +128 -0
  81. package/src/cli/commands/trace.ts +9 -0
  82. package/src/cli/index.ts +2 -0
  83. package/src/exporters/matrix/build.ts +40 -0
  84. package/src/exporters/matrix/export.ts +11 -0
  85. package/src/exporters/matrix/render-xlsx.ts +45 -1
  86. package/src/exporters/matrix/types.ts +10 -0
  87. package/src/exporters/playwright-report-parser.ts +2 -0
  88. package/src/exporters/types.ts +2 -0
  89. package/src/generators/test-generator/diagnostics.ts +6 -1
  90. package/src/generators/test-generator/patterns/index.ts +30 -24
  91. package/src/generators/test-generator/step-mapper.ts +22 -0
  92. package/src/harness/audit.ts +104 -11
  93. package/src/harness/flow-contract.ts +261 -0
  94. package/src/harness/flow-plan.ts +10 -3
  95. package/src/harness/parse.ts +31 -1
  96. package/src/harness/perf.ts +112 -0
  97. package/src/harness/sensors.ts +13 -1
  98. package/src/harness/spec-coverage.ts +55 -5
  99. package/src/orchestrator/templates/ai-src/commands/add-flow.md +51 -3
  100. package/src/orchestrator/templates/ai-src/commands/create-test.md +10 -0
  101. package/src/orchestrator/templates/ai-src/commands/run-test.md +23 -0
  102. package/src/orchestrator/templates/ai-src/skills/sungen-tc-generation/SKILL.md +60 -16
  103. package/src/orchestrator/templates/qa-context.md +14 -1
@@ -38,6 +38,8 @@ export interface ScenarioInfo {
38
38
  requiresCaps?: string[]; // @requires:<cap> — automation-ready but needs an opt-in driver (TQ-11)
39
39
  deferredToFlow?: boolean; // @deferred:flow — owned by a flow, not automated on this screen (H6)
40
40
  ownedByFlow?: string; // @owned-by:<flow> — the flow that owns this deferred scenario (H6)
41
+ /** Ordered steps with their resolved bucket (And/But inherit) — flow handoff analysis (#569). */
42
+ steps?: Array<{ bucket: 'given' | 'when' | 'then'; text: string }>;
41
43
  }
42
44
 
43
45
  /** Format-tolerant: is this token an ID (project's scheme), not a prose word?
@@ -91,6 +93,29 @@ export function parseViewpointOverview(filePath: string): ViewpointEntry[] {
91
93
  }
92
94
  }
93
95
 
96
+ // 1b) Flow-style declarations (#569). Flow viewpoint files commonly declare per-item
97
+ // ids at the END of a bullet ("… → **FL-HP-001**") under phase section headers
98
+ // ("## FL-HP — Happy Path"). Neither matched the table/group passes, so every flow
99
+ // audit collapsed to taxonomy=0% / traceability n-a — scenarios correctly tagged
100
+ // FL-HP-001 were reported as unmapped. Both forms are additive here.
101
+ for (const raw of lines) {
102
+ const line = raw.trim();
103
+ const section = line.match(/^##\s+([A-Z]{2,}(?:-[A-Z0-9]{2,})*)\s+[—–-]\s*(.*)$/);
104
+ if (section && isViewpointId(section[1] + '-0')) {
105
+ const id = section[1].toUpperCase();
106
+ if (!entries.has(id)) entries.set(id, { id, priority: 'Unknown', reason: section[2] ?? '' });
107
+ }
108
+ if (/^[-*+]\s/.test(line)) {
109
+ const arrow = line.match(/(?:→|->)\s*\*{0,2}([A-Z][A-Z0-9]*(?:-[A-Z0-9]+)*-\d+[a-zA-Z]?)\*{0,2}\s*$/);
110
+ if (arrow) {
111
+ const id = arrow[1].toUpperCase();
112
+ if (!entries.has(id)) {
113
+ entries.set(id, { id, priority: 'Unknown', reason: line.replace(/\s*(?:→|->).*$/, '').replace(/^[-*+]\s+/, '') });
114
+ }
115
+ }
116
+ }
117
+ }
118
+
94
119
  // 2) Viewpoint Grouping: ### Required / ### Recommended / ### Optional → bullet list
95
120
  let group: ViewpointEntry['group'] | undefined;
96
121
  for (const raw of lines) {
@@ -144,7 +169,9 @@ function classifyScenario(sc: ParsedScenario): ScenarioInfo {
144
169
  // Category is everything between `VP-` and the final `-<sequence>` — INCLUDING hyphens, so
145
170
  // compound categories (VP-LIST-DISPLAY-01, VP-ADD-TO-CART-03, VP-PRODUCT-DISCOVERY-02) parse,
146
171
  // not just single-word ones. A single-word category (VP-CART-001) still works. (H1)
147
- const codeMatch = sc.name.match(/\bVP-([A-Z]+(?:-[A-Z]+)*)-\d+/i);
172
+ // Flows use journey-phase ids (FL-HP-001 / FL-ER-002) — the VP- anchor rejected them, so every
173
+ // flow scenario had NO category and the whole suite bucketed `other` (taxonomy=0%, #569).
174
+ const codeMatch = sc.name.match(/\b(?:VP|FL)-([A-Z]+(?:-[A-Z]+)*)-\d+/i);
148
175
  const vpCode = codeMatch ? codeMatch[0].toUpperCase() : undefined;
149
176
  const category = codeMatch ? codeMatch[1].toUpperCase() : undefined;
150
177
  // Project-scheme ID: the leading token of the title (VP0-001 / MS-HP-001 / VP-LIST-001).
@@ -158,12 +185,14 @@ function classifyScenario(sc: ParsedScenario): ScenarioInfo {
158
185
  const skeletonParts: string[] = [];
159
186
  const textParts: string[] = [sc.name];
160
187
  const stepTextParts: string[] = [];
188
+ const orderedSteps: Array<{ bucket: 'given' | 'when' | 'then'; text: string }> = [];
161
189
 
162
190
  for (const step of sc.steps as ParsedStep[]) {
163
191
  const kw = step.keyword.trim();
164
192
  if (kw === 'Given' || kw === 'When' || kw === 'Then') last = kw;
165
193
  textParts.push(step.text);
166
194
  stepTextParts.push(step.text);
195
+ orderedSteps.push({ bucket: (kw === 'And' || kw === 'But' ? last : kw).toLowerCase() as 'given' | 'when' | 'then', text: step.text });
167
196
  // normalized skeleton: keep [refs] (distinct targets = distinct tests),
168
197
  // but neutralize {{vars}} and quoted values so EP/data families collapse.
169
198
  const skel = step.text
@@ -199,6 +228,7 @@ function classifyScenario(sc: ParsedScenario): ScenarioInfo {
199
228
  stepSkeleton: skeletonParts.join(' | '),
200
229
  haystack: textParts.join(' ').toLowerCase(),
201
230
  stepsText: stepTextParts.join(' ').toLowerCase(),
231
+ steps: orderedSteps,
202
232
  vpId,
203
233
  casesDataset,
204
234
  queryRefs: queryRefs.size ? [...queryRefs] : undefined,
@@ -0,0 +1,112 @@
1
+ /**
2
+ * Performance budgets — config + percentile math + the per-unit verdict. (#569)
3
+ *
4
+ * A flow's regression value includes "still fast enough": after a lib/framework
5
+ * upgrade the main journeys must not only pass but hold their response-time
6
+ * budget. Sungen had no perf concept at all — the Playwright JSON parser even
7
+ * dropped the `duration` field Playwright already emits on every result.
8
+ *
9
+ * Scope discipline:
10
+ * - This is config + measurement + report over runs sungen already makes.
11
+ * Real load tests stay @manual:M8 → a dedicated tool.
12
+ * - The AUDIT never reads it: the quality score is documented as a pure
13
+ * function of the design artifacts ("reads no test-results, live page, or
14
+ * clock"). Perf reports where runs are already read — `sungen delivery`
15
+ * and the dashboard. Advisory: a blown budget never fails the design gate.
16
+ *
17
+ * Config: qa/perf.yaml
18
+ * percentile: p75 # default p75 — "≥75% of runs meet the budget"
19
+ * defaults:
20
+ * scenario_ms: 30000 # whole-scenario wall clock (Playwright duration)
21
+ * page_load_ms: 3000 # Phase B — needs per-transition runtime timing
22
+ * transition_ms: 2000 # Phase B
23
+ * units:
24
+ * place-order: { scenario_ms: 20000 }
25
+ */
26
+ import * as fs from 'fs';
27
+ import * as path from 'path';
28
+ import { parse as parseYaml } from 'yaml';
29
+ import { readTextFile } from './read-text';
30
+
31
+ export interface PerfConfig {
32
+ /** 0..100 — e.g. 75 for p75. */
33
+ percentile: number;
34
+ defaults: Record<string, number>;
35
+ units: Record<string, Record<string, number>>;
36
+ }
37
+
38
+ export interface PerfVerdict {
39
+ unit: string;
40
+ metric: string; // 'scenario_ms' today; page_load_ms/transition_ms in Phase B
41
+ percentile: number; // 75
42
+ budgetMs: number;
43
+ measuredMs: number; // the pXX of the observed durations
44
+ samples: number;
45
+ pass: boolean;
46
+ /** Titles of the slowest offenders (only when failing), for the report. */
47
+ slowest: Array<{ title: string; ms: number }>;
48
+ }
49
+
50
+ export function perfConfigPath(projectRoot: string): string {
51
+ return path.join(projectRoot, 'qa', 'perf.yaml');
52
+ }
53
+
54
+ /** Absent file → null (perf reporting is opt-in; nothing changes until configured). */
55
+ export function loadPerfConfig(projectRoot: string): PerfConfig | null {
56
+ const p = perfConfigPath(projectRoot);
57
+ if (!fs.existsSync(p)) return null;
58
+ let raw: Record<string, unknown>;
59
+ try { raw = parseYaml(readTextFile(p)) as Record<string, unknown>; } catch { return null; }
60
+ if (!raw || typeof raw !== 'object') return null;
61
+ const pctRaw = String(raw.percentile ?? 'p75').toLowerCase().replace(/^p/, '');
62
+ const percentile = Math.min(100, Math.max(1, Number(pctRaw) || 75));
63
+ const num = (o: unknown): Record<string, number> => {
64
+ const out: Record<string, number> = {};
65
+ if (o && typeof o === 'object') {
66
+ for (const [k, v] of Object.entries(o as Record<string, unknown>)) {
67
+ const n = Number(v);
68
+ if (Number.isFinite(n) && n > 0) out[k] = n;
69
+ }
70
+ }
71
+ return out;
72
+ };
73
+ const units: Record<string, Record<string, number>> = {};
74
+ if (raw.units && typeof raw.units === 'object') {
75
+ for (const [u, o] of Object.entries(raw.units as Record<string, unknown>)) units[u] = num(o);
76
+ }
77
+ return { percentile, defaults: num(raw.defaults), units };
78
+ }
79
+
80
+ /**
81
+ * Nearest-rank percentile (ceil), the standard "≥pXX of samples meet the budget"
82
+ * reading: p75 of [a…] is the value at ceil(0.75·n) in the sorted list. One
83
+ * sample → that sample. Deterministic, no interpolation.
84
+ */
85
+ export function percentileOf(p: number, values: number[]): number {
86
+ if (values.length === 0) return 0;
87
+ const sorted = [...values].sort((a, b) => a - b);
88
+ const rank = Math.min(sorted.length, Math.max(1, Math.ceil((p / 100) * sorted.length)));
89
+ return sorted[rank - 1];
90
+ }
91
+
92
+ /** Budget for a metric on a unit: per-unit override, else defaults, else none. */
93
+ export function budgetFor(config: PerfConfig, unit: string, metric: string): number | undefined {
94
+ return config.units[unit]?.[metric] ?? config.defaults[metric];
95
+ }
96
+
97
+ /**
98
+ * The scenario_ms verdict for one unit's run. `durations` = per-test wall-clock ms
99
+ * (a @cases scenario contributes one sample per row-test — each is a real run).
100
+ */
101
+ export function perfVerdict(
102
+ config: PerfConfig,
103
+ unit: string,
104
+ samples: Array<{ title: string; ms: number }>,
105
+ ): PerfVerdict | null {
106
+ const budgetMs = budgetFor(config, unit, 'scenario_ms');
107
+ if (budgetMs === undefined || samples.length === 0) return null;
108
+ const measuredMs = percentileOf(config.percentile, samples.map((s) => s.ms));
109
+ const pass = measuredMs <= budgetMs;
110
+ const slowest = pass ? [] : [...samples].sort((a, b) => b.ms - a.ms).slice(0, 3);
111
+ return { unit, metric: 'scenario_ms', percentile: config.percentile, budgetMs, measuredMs, samples: samples.length, pass, slowest };
112
+ }
@@ -33,10 +33,22 @@ const BUCKET_ORDER: Array<[string, string[]]> = [
33
33
  ];
34
34
  const BUCKETS: Record<string, string[]> = Object.fromEntries(BUCKET_ORDER);
35
35
 
36
+ // Flow journey-phase categories (FL-HP-001, FL-ER-002 …). Matched on exact SEGMENTS,
37
+ // never by containment — 'SHOP'.includes('HP') is true, which is exactly the kind of
38
+ // false hit substring matching would produce for two-letter phase tokens. (#569)
39
+ const PHASE_BUCKETS: Record<string, string> = {
40
+ HP: 'business-core', // happy path = the business goal itself
41
+ ER: 'validation-security', // error recovery (validation must not trap the journey)
42
+ EH: 'validation-security', // guards & leakage (direct access, back, refresh)
43
+ };
44
+
36
45
  /** Classify a VP category into a balance bucket by keyword containment + precedence (H1). */
37
46
  export function bucketForCategory(category: string | undefined): string {
38
47
  const cat = (category || '').toUpperCase();
39
48
  if (!cat) return 'other';
49
+ for (const seg of cat.split('-')) {
50
+ if (PHASE_BUCKETS[seg]) return PHASE_BUCKETS[seg];
51
+ }
40
52
  for (const [bucket, kws] of BUCKET_ORDER) {
41
53
  if (kws.some((k) => cat.includes(k))) return bucket;
42
54
  }
@@ -351,7 +363,7 @@ export function flowRegressionDepth(scenarios: ScenarioInfo[]): FlowDepthResult
351
363
  // 1. Count/quantity proof — a row count or item quantity, not just presence of a row.
352
364
  const countProof = any(/\b(quantity|qty|two (?:rows|lines|cart)|row count|count column|number of items|one[_ ]row|two[_ ]rows|qty[_ ])/i);
353
365
  // 2. Teardown — removes the item and verifies the empty/zero state (the inverse operation).
354
- const teardown = any(/\b(remove|delete|clear)\b/i) && any(/\b(empty|no items|zero|removed|0 items)\b/i);
366
+ const teardown = any(/\b(remove|delete|clear)(?:s|d|ed|ing)?\b/i) && any(/\b(empty|emptied|no items|zero|removed|cleared|0 items)\b/i);
355
367
  // 3. Multi-source — the cart is fed from >1 source (the main list AND a recommended/related rail).
356
368
  const multiSource = any(/\b(recommended|related|you may also|suggest)\b/i) && addsToCart;
357
369
 
@@ -66,21 +66,43 @@ export function parseSpecClauses(specPath: string): { frs: FrClause[]; valRows:
66
66
  if (!fs.existsSync(specPath)) return { frs: [], valRows: [] };
67
67
  const lines = readTextFile(specPath).split('\n');
68
68
 
69
+ // Requirement ids follow the PROJECT's scheme, not ours (#572): `**FR-1**:` is one
70
+ // convention among many — a real spec declared ~30 MUST clauses as `` `REQ-SRCH-001`: ``
71
+ // and the FR-locked pattern returned zero, so the MUST-coverage gate never ran and the
72
+ // specFR axis was excluded "for lack of evidence" that was sitting right there. Same
73
+ // silent-failure class as the CRLF parsers, same id-scheme-tolerance lesson as delivery.
74
+ //
75
+ // A declaration is: line-leading (optionally bulleted / bold / backticked) `<ID>:` where
76
+ // the id ends in a number. Table rows are EXCLUDED — traceability tables cite requirement
77
+ // ids without declaring them — and so are prefixes that are never requirements
78
+ // (test cases, viewpoints, known-defect records, data-factory checks, flows, delivery items).
79
+ const NON_REQUIREMENT_PREFIX = /^(TC|VP|KD|CHK|FL|DI)-/i;
69
80
  const frs: FrClause[] = [];
81
+ const seen = new Set<string>();
70
82
  for (const line of lines) {
71
- const m = line.match(/\*\*FR-(\d+)\*\*\s*:\s*(.+)$/);
72
- if (m) frs.push({ id: `FR-${m[1]}`, text: m[2].replace(/\*\*/g, '').trim(), modality: modalityOf(m[2]) });
83
+ if (/^\s*\|/.test(line)) continue; // table row = citation, not declaration
84
+ const m = line.match(/^\s*(?:[-*+]\s+)?[*_`]*([A-Z][A-Z0-9]*(?:-[A-Z0-9]+)*-\d+[a-zA-Z]?)[*_`]*\s*:\s*(.+)$/);
85
+ if (!m || NON_REQUIREMENT_PREFIX.test(m[1])) continue;
86
+ const id = m[1].toUpperCase();
87
+ if (seen.has(id)) continue; // first declaration wins
88
+ seen.add(id);
89
+ frs.push({ id, text: m[2].replace(/\*\*/g, '').trim(), modality: modalityOf(m[2]) });
73
90
  }
74
91
 
75
92
  // Validation Rules table: a row carries a Constraint, a Trigger cell, and (often) a code.
93
+ // A "Trigger" column alone is NOT enough to claim the table (#578): a screen-STATES table
94
+ // ("State ID | Trigger | URL/heading oracle | …") uses Trigger for the user action that
95
+ // enters the state, and reading it as validation rows invented three gate-relevant
96
+ // TRIGGER-UNCOVERED gaps on a real spec. The table must also name a Constraint/Rule/
97
+ // Validation column — the thing a validation row is ABOUT.
76
98
  const valRows: ValRow[] = [];
77
99
  let cTrigger = -1, cConstraint = -1, cCode = -1, inTable = false;
78
100
  for (const raw of lines) {
79
101
  const line = raw.trim();
80
- if (line.startsWith('|') && /\btrigger\b/i.test(line) && cTrigger < 0) {
102
+ if (line.startsWith('|') && /\btrigger\b/i.test(line) && /\b(constraint|rule|validation)\b/i.test(line) && cTrigger < 0) {
81
103
  const cells = line.split('|').map((c) => c.trim());
82
104
  cTrigger = cells.findIndex((c) => /^trigger$/i.test(c));
83
- cConstraint = cells.findIndex((c) => /constraint/i.test(c));
105
+ cConstraint = cells.findIndex((c) => /constraint|rule|validation/i.test(c));
84
106
  cCode = cells.findIndex((c) => /code/i.test(c));
85
107
  inTable = cTrigger >= 0;
86
108
  continue;
@@ -104,12 +126,40 @@ function scenarioBlocks(featureText: string): string[] {
104
126
  return featureText.split(/\n\s*\n/).filter((b) => /\bScenario:/.test(b)).map((b) => b.toLowerCase());
105
127
  }
106
128
 
129
+ /**
130
+ * Requirement ids CITED anywhere in the feature (tags, comments, prose) — with the
131
+ * compressed notations authors naturally write expanded (#578 follow-up, 3rd recurrence):
132
+ * `REQ-CART-001..005` (range), `REQ-QTY-001/003` (enumeration). A plain substring check
133
+ * missed exactly the ids inside the shorthand, so a deferral comment that HONESTLY listed
134
+ * its flow-owned requirements still left them reported as uncovered.
135
+ */
136
+ export function citedIds(featureText: string): Set<string> {
137
+ const t = featureText.toUpperCase();
138
+ const out = new Set<string>();
139
+ for (const m of t.matchAll(/\b([A-Z][A-Z0-9]*(?:-[A-Z0-9]+)*-\d+[A-Z]?)\b/g)) out.add(m[1]);
140
+ // Range: PREFIX-001..005 (also – — ~ as the dash). Width follows the FIRST number.
141
+ for (const m of t.matchAll(/\b([A-Z][A-Z0-9]*(?:-[A-Z0-9]+)*-)(\d+)\s*(?:\.\.|–|—|~)\s*(\d+)/g)) {
142
+ const w = m[2].length;
143
+ for (let i = parseInt(m[2], 10); i <= parseInt(m[3], 10) && i - parseInt(m[2], 10) < 200; i++) {
144
+ out.add(m[1] + String(i).padStart(w, '0'));
145
+ }
146
+ }
147
+ // Enumeration: PREFIX-001/003/007 — every slash part is an id under the same prefix.
148
+ for (const m of t.matchAll(/\b([A-Z][A-Z0-9]*(?:-[A-Z0-9]+)*-)(\d+)((?:\/\d+)+)/g)) {
149
+ const w = m[2].length;
150
+ out.add(m[1] + m[2]);
151
+ for (const part of m[3].split('/').filter(Boolean)) out.add(m[1] + part.padStart(w, '0'));
152
+ }
153
+ return out;
154
+ }
155
+
107
156
  export function specCoverage(specPath: string, scenarios: ScenarioInfo[], featureText: string): SpecCoverageResult {
108
157
  const { frs, valRows } = parseSpecClauses(specPath);
109
158
  if (!fs.existsSync(specPath) || (frs.length === 0 && valRows.length === 0)) {
110
159
  return { hasSpec: fs.existsSync(specPath), frTotal: 0, frCovered: 0, uncoveredMust: [], inferredOnly: [], triggerGaps: [], verdict: 'pass' };
111
160
  }
112
161
  const featLower = featureText.toLowerCase();
162
+ const cites = citedIds(featureText);
113
163
 
114
164
  // FR coverage: explicit @spec:FR / literal FR-id citation, else keyword fallback.
115
165
  const uncoveredMust: { id: string; text: string }[] = [];
@@ -117,7 +167,7 @@ export function specCoverage(specPath: string, scenarios: ScenarioInfo[], featur
117
167
  let frCovered = 0;
118
168
  for (const fr of frs) {
119
169
  const idLower = fr.id.toLowerCase();
120
- const cited = featLower.includes(idLower);
170
+ const cited = featLower.includes(idLower) || cites.has(fr.id.toUpperCase());
121
171
  const words = [...new Set((fr.text.toLowerCase().match(/[a-z][a-z-]{4,}/g) || []))]
122
172
  .filter((w) => !/must|should|system|screen|users?|value|input|field/.test(w));
123
173
  const kwHit = words.length > 0 && scenarios.some((s) => words.filter((w) => s.haystack.includes(w)).length >= Math.min(2, words.length));
@@ -86,15 +86,62 @@ qa/flows/${input:flow}/
86
86
  └── ui/ # Screenshots, mockups
87
87
  ```
88
88
 
89
- ### 1a. Identify the screens in the flow
89
+ ### 1a. Define the flow's BOUNDARY, then its screens
90
90
 
91
- Ask the user: "Which screens does this flow visit, in order? (e.g., login dashboard → award-form → confirmation)"
91
+ > QA teams often call this level **System Test** same thing: one fully-integrated business
92
+ > journey verified against the spec. Use whichever name the team knows; the boundary rules
93
+ > below are the ISTQB system-test design rules.
94
+
95
+
96
+ A flow is the **smallest complete business action chain**: one clear trigger ending in ONE
97
+ observable, valuable outcome. Before asking for screens, walk this checklist with the user —
98
+ if 1, 3 or 8 fails, propose SPLITTING into separate flows:
99
+
100
+ 1. Exactly **one business goal**? (cart correctness + category filtering = two flows)
101
+ 2. A clear **trigger** and precondition?
102
+ 3. **One observable final outcome**? (a final assertion you can write in one sentence)
103
+ 4. Is that outcome **valuable to the actor**? (an order placed, a password reset — not "a page rendered")
104
+ 5. Is **every step necessary** for that outcome?
105
+ 6. Are all steps at the **same business abstraction**?
106
+ 7. Are optional/error branches **phases of this goal** (ER/EH), not new goals?
107
+ 8. Does **no segment** form an independently valuable flow on its own?
108
+ 9. Can you write **a single clear final assertion**?
109
+ 10. Can you name it "**Verb + outcome**"? (`place-order`, `reset-password` — not `cart-and-filter`)
110
+
111
+ Then ask: "Which screens does this flow visit, in order? (e.g., login → dashboard → award-form → confirmation)"
92
112
 
93
113
  Record the screen list — you will need it for:
94
114
  - Filling `spec.md` (Step 3)
95
115
  - Suggesting `[Screen:Element]` namespace prefixes
96
116
  - Capturing visuals per screen (Step 2)
97
117
 
118
+ ### 1b. Author the Flow Contract (`requirements/flow-contract.yaml`)
119
+
120
+ Write the answers down as the flow's contract — `sungen audit` scores the flow **against it**
121
+ (the `flowCoverage` axis: HP/ER/EH journey phases; `FLOW-OUTCOME-UNPROVEN` when no automated
122
+ scenario asserts data on the outcome screen; `FLOW-SCOPE-CREEP` when scenarios never touch it):
123
+
124
+ ```yaml
125
+ goal: "Place an order for a product added from home" # Verb + outcome
126
+ actor: user
127
+ trigger: "Add a product to the cart from the home featured list"
128
+ precondition: "A registered account; an empty cart"
129
+ outcome:
130
+ screen: checkout # the [Screen:...] namespace carrying the final proof
131
+ assertion: "The confirmation shows the order number and the paid total"
132
+ value: "The customer has paid; the shop has a new order"
133
+ phases: [HP, ER, EH] # journey phases (default); add UI only if the flow owns UI states
134
+ stateful: cart # the mutated collection, if any — enables regression-depth dims
135
+ golden: true # optional — release-critical: Final Inspection expects @golden scenarios here
136
+ external: # optional — legs owned by another team/vendor (System INTEGRATION Testing)
137
+ - name: payment-gateway
138
+ owner: vendor-x
139
+ screens: [payment] # the flow namespaces that leg passes through
140
+ ```
141
+
142
+ **A filled contract is an INPUT to generation — never an output.** Like `test-viewpoint.md`,
143
+ generation must not rewrite it to match what was generated; disagree → propose the diff and ask.
144
+
98
145
  ### 2. Capture visual source
99
146
 
100
147
  **Mobile path** (`platform: mobile`):
@@ -187,7 +234,8 @@ If user picks `/sungen:create-test`, **you MUST use the Skill tool** to invoke i
187
234
  - Test data namespaced by phase: `login.email`, `submission.nominee`
188
235
  - `@flow` tag required at feature level
189
236
  - `Background:` should only contain the starting navigation — the URL path (web) or the `--reach` nav recipe (mobile)
190
- - Each scenario = one phase of the journey
237
+ - Each scenario = one phase of the journey; ids are `FL-<PHASE>-NNN` (`HP`/`ER`/`EH`, optional `UI`)
238
+ - One flow = ONE business goal with ONE observable outcome (`requirements/flow-contract.yaml`) — a segment with its own value is its own flow
191
239
  {{#cap parallel-subagents}}
192
240
  - Mobile flows are tagged `@platform:mobile` and run via `/sungen:run-test <flow>` (WebdriverIO, not Playwright)
193
241
  {{/cap}}
@@ -6,6 +6,16 @@ order: 20
6
6
  claude-tools: "Read, Grep, Bash, Glob, Write, AskUserQuestion, Skill, mcp__playwright__browser_navigate, mcp__playwright__browser_snapshot, mcp__playwright__browser_take_screenshot"
7
7
  copilot-tools: "[vscode, execute, read, agent, edit, search, web, browser, todo, 'playwright/*']"
8
8
  codex-trigger: "Run when the user asks to CREATE, generate, write, or author test cases / a .feature file for a screen or flow. Step 2 (after add-screen/add-flow, before run-test). Do NOT use for executing, running, or compiling existing tests."
9
+ ---
10
+ ## ⛔ HARD RULE — the run's LAST action is the next-step hand-back
11
+
12
+ A create-test run is NOT finished when the files are written or the audit prints. The final
13
+ action of EVERY run — success, partial, or aborted — is the next-step hand-back
14
+ ({{#cap parallel-subagents}}an `AskUserQuestion` offering the next actions{{/cap}}{{^cap parallel-subagents}}a numbered list of next-action choices{{/cap}};
15
+ see "Finish — always hand the next step back" at the end of this file). Ending with a prose
16
+ summary and no choices is a broken run: the operator is left guessing. This holds no matter
17
+ how long the generation/repair loop ran.
18
+
9
19
  ---
10
20
  {{#cap parallel-subagents}}
11
21
  ## ⛔ HARD RULE — No Figma MCP when PAT data exists
@@ -7,6 +7,29 @@ claude-tools: "Read, Grep, Bash, Glob, Edit, Write, AskUserQuestion, mcp__playwr
7
7
  copilot-tools: "[read, execute, edit, vscode/askQuestions, playwright/*, appium/*]"
8
8
  codex-trigger: "Run when the user asks to RUN, execute, or compile tests, generate selectors.yaml, or run tests. Step 4. Do NOT use for authoring/creating new test cases."
9
9
  ---
10
+ ## ⛔ HARD RULE — the transition INTO this run is a tool call, never prose
11
+
12
+ A QA field report: the user picked "Run test" from create-test's hand-back, and the session
13
+ answered with a BRIEFING — missing selector, expected reds, "run under Node 22" — written as
14
+ "Before you do…", then stopped. Correct facts, wrong role: those are conditions YOU handle
15
+ inside the run, not reasons to stop and hand the work back.
16
+
17
+ - The first thing this command produces is a **tool call** (platform detection, preflight,
18
+ compile — whatever comes first). Never open with a plan and end the turn.
19
+ - A **missing selector** goes to the selector-generation/fix step — that is what this command
20
+ is FOR.
21
+ - **Expected-red scenarios** (@known-defect asserting a live defect) stay red; note them in
22
+ the results summary, never "fix" them and never stop for them.
23
+ - **Runtime selection is yours**: check `node -v` first. If the major version is ≥ 23 and
24
+ Playwright browser runs are known-broken on it, select Node 22 yourself when available —
25
+ `export PATH="$HOME/.nvm/versions/node/$(ls $HOME/.nvm/versions/node | grep '^v22' | tail -1)/bin:$PATH"`
26
+ — and say so in one line. Only if NO compatible Node exists do you stop, with the exact
27
+ install command as the hand-back.
28
+ - Ending this run follows the same law as create-test: the LAST action is the next-step
29
+ hand-back (the AskUserQuestion in "After showing results"), no matter how the run went.
30
+
31
+ ---
32
+
10
33
  ## Role
11
34
 
12
35
  You are a **Senior Developer**.
@@ -603,24 +603,67 @@ error:
603
603
 
604
604
  ## Flow Test Generation
605
605
 
606
+ > **Terminology**: many QA teams call this level **System Test** (ISTQB: verify the fully
607
+ > integrated system against its specified requirements, designed around business
608
+ > processes/use cases). In sungen a *flow* is that system-test suite run through the real UI;
609
+ > its automation then serves as the regression suite on later runs. The names are
610
+ > interchangeable in conversation; the artifacts stay `qa/flows/<name>/`.
611
+
612
+ **Representative, not exhaustive (ISTQB system-test design rule).** A flow case takes ONE
613
+ representative input per branch — one valid value for the main flow, one error value that
614
+ triggers the exception branch and proves the journey recovers. The full equivalence/boundary
615
+ matrix belongs to the OWNING SCREEN (as `@cases` there), never re-run inside the flow: if the
616
+ screen already proved "the field rejects negatives", the flow only proves "the journey survives
617
+ one rejected input". `sungen audit` reports `FLOW-EXHAUSTIVE-INPUTS` when ≥3 same-shape flow
618
+ scenarios differ only in data. **Mocked dependencies**: a flow scenario using `@mock` is
619
+ delivered as *Pass (Mocked)* — verified handling logic, not the real dependency; the delivery's
620
+ External Dependency Tracking table lists it until re-verified real.
621
+
622
+ **Release selection & external legs**: when the contract declares `golden: true`, tag the
623
+ happy-path scenario(s) that prove the outcome with **`@golden`** — that is the set
624
+ `sungen inspect` (Final Inspection, Go/No-Go) runs before a release; `GOLDEN-MISSING` fires
625
+ until it exists, and a @golden scenario that passes only against mocks BLOCKS the release.
626
+ When the contract declares `external:` legs (another team/vendor's system), scenarios touching
627
+ those namespaces are the **System INTEGRATION** group — keep their oracles about the boundary
628
+ (contract/format/handoff), and expect `SIT-MOCK-ONLY` until they are re-verified real.
629
+
630
+
606
631
  > **Auto-detect**: if path is `qa/flows/<name>/` → use this section. Skip Steps 1–4 above.
607
632
 
633
+ **Read `requirements/flow-contract.yaml` FIRST — it is the flow's boundary and the yardstick
634
+ `sungen audit` scores the flow against** (`flowCoverage` axis = journey phases HP/ER/EH automated;
635
+ `FLOW-OUTCOME-UNPROVEN`; `FLOW-SCOPE-CREEP`). No contract yet → author it with the user via the
636
+ boundary checklist in `add-flow` (one business goal · clear trigger · ONE observable outcome
637
+ valuable to the actor · name = "Verb + outcome"), THEN generate. **A filled contract is an INPUT —
638
+ never rewrite it to match your output** (same rule as `test-viewpoint.md`).
639
+
608
640
  | Aspect | Screen | Flow |
609
641
  |---|---|---|
610
- | Section focus | UI patterns per section | Journey phases across screens |
642
+ | Section focus | UI patterns per section | Journey phases toward ONE declared outcome |
611
643
  | Selector format | `[Element]` | `[Screen:Element]` (namespaced) |
612
644
  | Test data keys | `{{variable}}` | `{{phase.variable}}` |
613
645
  | Feature tag | `@auto` / `@smoke` etc. | `@flow` (required) |
614
- | Viewpoints | VP-UI/VAL/LOGIC/SEC per section | VP-LOGIC (transitions), VP-SEC (auth persistence), VP-VAL (cross-screen data) |
646
+ | Scenario ids | `VP-<CATEGORY>-NNN` | `FL-<PHASE>-NNN` phases: `HP` (happy path), `ER` (error recovery), `EH` (guards), `UI` (journey UI states, optional) |
615
647
 
616
- **Scenarios to generate:**
648
+ **Scenarios to generate — every phase demanded by the contract, automated:**
617
649
 
618
- | Category | What to test |
619
- |---|---|
620
- | Happy path | Complete flow end-to-end with valid data |
621
- | Auth persistence | Auth state maintained across screen transitions |
622
- | Error recovery | Invalid input mid-flow fixcontinue |
623
- | Cross-screen data | Data entered on screen A visible on screen B |
650
+ | Phase | What to test | Scoring |
651
+ |---|---|---|
652
+ | `FL-HP` happy path | The complete journey ending in the contract's `outcome.assertion` — an AUTOMATED **data** assertion on `outcome.screen` (an order number, a summed total — not just "page visible"). This scenario is WHY the flow exists: it is the regression proof after a lib/framework upgrade. | uncovered → `flowCoverage` drops + `FLOW-OUTCOME-UNPROVEN` |
653
+ | `FL-ER` error recovery | Invalid input mid-flow error shown → fix → the journey still completes. Validation must not trap the journey. | uncovered → `flowCoverage` drops |
654
+ | `FL-EH` guards | Direct URL access without the precondition · browser back · refresh · expired context — each ends in a safe observable state. | uncovered `flowCoverage` drops |
655
+ | Cross-screen handoff | After every screen transition, assert the CARRIED state on the new screen (the added product's name in the cart, the email echoed on the sent screen). | blind tails cap `businessDepth` (`FLOW-HANDOFF-SHALLOW`) |
656
+ | Stateful regression (when `stateful:` declared) | Count/quantity proof · teardown (remove → empty) · multi-source add. | missing dims cap `businessDepth` (`FLOW-DEPTH`) |
657
+
658
+ **Boundary discipline while generating:** every scenario must serve the contract's goal. A scenario
659
+ that never touches `outcome.screen` and is not a guard (`EH`) or error-recovery (`ER`) belongs in a
660
+ DIFFERENT flow — propose the split instead of writing it here (`FLOW-SCOPE-CREEP` will flag it).
661
+ Auth persistence across transitions is part of `EH` unless the project declares it its own phase.
662
+
663
+ **Manual in flows**: always `@manual:Mx` with the reason code — bare `@manual` is flagged
664
+ (`MANUAL-CODE-MISSING`) because the capability planner cannot route it. Typical flow deferrals:
665
+ inbox/mail oracle → `M5`, network-request count → `M3`, context expiry control → `M7`. A
666
+ cross-screen scenario inside the flow's own goal is NOT manual — automate it here.
624
667
 
625
668
  ```gherkin
626
669
  @flow @auth:user
@@ -630,19 +673,20 @@ Feature: Award Submission Flow
630
673
  Given User is on [Login] page
631
674
 
632
675
  @high
633
- Scenario: User logs in successfully
676
+ Scenario: FL-HP-001 A signed-in user's nomination is submitted and confirmed
634
677
  When User fill [Login:Email] field with {{login.email}}
635
678
  And User fill [Login:Password] field with {{login.password}}
636
679
  And User click [Login:Submit] button
637
680
  Then User see [Dashboard] page
638
-
639
- @high
640
- Scenario: User submits nomination
641
681
  When User click [Dashboard:Awards] link
642
- Then User see [Awards] page
643
- When User fill [Awards:Nominee] field with {{submission.nominee}}
682
+ And User fill [Awards:Nominee] field with {{submission.nominee}}
644
683
  And User click [Awards:Submit] button
645
- Then User see {{success_message}} message
684
+ Then User see [Awards:Success Message] text with {{success_message}}
685
+
686
+ @high
687
+ Scenario: FL-EH-001 Direct access to the award form without login redirects to login
688
+ When User go to [Awards] page
689
+ Then User see [Login] page
646
690
  ```
647
691
 
648
692
  ```yaml
@@ -45,8 +45,21 @@ Example:
45
45
 
46
46
  ## Testing Strategy
47
47
 
48
+ Machine-readable intent — `sungen audit` reads these keys (Intent Profile). Values here are
49
+ live even when the surrounding text changes; an invalid value silently falls back to the default.
50
+
51
+ focus: functional
52
+ <!-- focus: functional | e-commerce | security | smoke — drives the audit's depth threshold -->
53
+
54
+ risk_tier: normal
55
+ <!-- risk_tier: high | normal | low -->
56
+
57
+ To silence driver suggestions in audit findings, add a line: capability_suggestions with value off.
58
+
48
59
  **Focus areas** — what to cover thoroughly:
49
- <!-- List from: functional, security, ui, accessibility, performance -->
60
+ <!-- Prose for humans; the parseable value is the `focus:` key above.
61
+ Response-time budgets are NOT a focus value — declare them in qa/perf.yaml
62
+ (percentile + scenario_ms budgets; reported by `sungen delivery`). -->
50
63
  <!-- Example: functional, security -->
51
64
 
52
65
  **Mandatory coverage:**