@sun-asterisk/sungen 3.0.0 → 3.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/commands/audit.d.ts.map +1 -1
- package/dist/cli/commands/audit.js +24 -0
- package/dist/cli/commands/audit.js.map +1 -1
- package/dist/cli/commands/delivery.d.ts.map +1 -1
- package/dist/cli/commands/delivery.js +30 -14
- package/dist/cli/commands/delivery.js.map +1 -1
- package/dist/cli/commands/eval.d.ts +3 -0
- package/dist/cli/commands/eval.d.ts.map +1 -0
- package/dist/cli/commands/eval.js +37 -0
- package/dist/cli/commands/eval.js.map +1 -0
- package/dist/cli/commands/ingest.d.ts +3 -0
- package/dist/cli/commands/ingest.d.ts.map +1 -0
- package/dist/cli/commands/ingest.js +179 -0
- package/dist/cli/commands/ingest.js.map +1 -0
- package/dist/cli/index.js +4 -0
- package/dist/cli/index.js.map +1 -1
- package/dist/dashboard/templates/index.html +108 -194
- package/dist/harness/audit.d.ts +16 -0
- package/dist/harness/audit.d.ts.map +1 -1
- package/dist/harness/audit.js +68 -4
- package/dist/harness/audit.js.map +1 -1
- package/dist/harness/capability-plan.d.ts +6 -0
- package/dist/harness/capability-plan.d.ts.map +1 -1
- package/dist/harness/capability-plan.js +13 -0
- package/dist/harness/capability-plan.js.map +1 -1
- package/dist/harness/eval/skill-lint.d.ts +16 -0
- package/dist/harness/eval/skill-lint.d.ts.map +1 -0
- package/dist/harness/eval/skill-lint.js +129 -0
- package/dist/harness/eval/skill-lint.js.map +1 -0
- package/dist/harness/parse.d.ts +6 -0
- package/dist/harness/parse.d.ts.map +1 -1
- package/dist/harness/parse.js +18 -3
- package/dist/harness/parse.js.map +1 -1
- package/dist/harness/quality-gates.d.ts +29 -0
- package/dist/harness/quality-gates.d.ts.map +1 -0
- package/dist/harness/quality-gates.js +183 -0
- package/dist/harness/quality-gates.js.map +1 -0
- package/dist/harness/sensors.d.ts.map +1 -1
- package/dist/harness/sensors.js +85 -6
- package/dist/harness/sensors.js.map +1 -1
- package/dist/harness/spec-coverage.d.ts +37 -0
- package/dist/harness/spec-coverage.d.ts.map +1 -0
- package/dist/harness/spec-coverage.js +159 -0
- package/dist/harness/spec-coverage.js.map +1 -0
- package/dist/harness/viewpoint-ledger.d.ts +23 -0
- package/dist/harness/viewpoint-ledger.d.ts.map +1 -0
- package/dist/harness/viewpoint-ledger.js +118 -0
- package/dist/harness/viewpoint-ledger.js.map +1 -0
- package/dist/ingest/baseline-audit.d.ts +38 -0
- package/dist/ingest/baseline-audit.d.ts.map +1 -0
- package/dist/ingest/baseline-audit.js +85 -0
- package/dist/ingest/baseline-audit.js.map +1 -0
- package/dist/ingest/gsheet-fetch.d.ts +9 -0
- package/dist/ingest/gsheet-fetch.d.ts.map +1 -0
- package/dist/ingest/gsheet-fetch.js +180 -0
- package/dist/ingest/gsheet-fetch.js.map +1 -0
- package/dist/ingest/index.d.ts +6 -0
- package/dist/ingest/index.d.ts.map +1 -0
- package/dist/ingest/index.js +22 -0
- package/dist/ingest/index.js.map +1 -0
- package/dist/ingest/legacy-parser.d.ts +39 -0
- package/dist/ingest/legacy-parser.d.ts.map +1 -0
- package/dist/ingest/legacy-parser.js +218 -0
- package/dist/ingest/legacy-parser.js.map +1 -0
- package/dist/ingest/reconcile.d.ts +30 -0
- package/dist/ingest/reconcile.d.ts.map +1 -0
- package/dist/ingest/reconcile.js +65 -0
- package/dist/ingest/reconcile.js.map +1 -0
- package/dist/ingest/to-gherkin.d.ts +33 -0
- package/dist/ingest/to-gherkin.d.ts.map +1 -0
- package/dist/ingest/to-gherkin.js +93 -0
- package/dist/ingest/to-gherkin.js.map +1 -0
- package/dist/orchestrator/ai-rules-updater.d.ts.map +1 -1
- package/dist/orchestrator/ai-rules-updater.js +2 -0
- package/dist/orchestrator/ai-rules-updater.js.map +1 -1
- package/dist/orchestrator/templates/ai-instructions/claude-agent-reviewer.md +1 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-delivery.md +10 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-ingest-legacy.md +79 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +25 -1
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-delivery.md +10 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-ingest-legacy.md +79 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +44 -7
- package/package.json +3 -3
- package/src/cli/commands/audit.ts +19 -0
- package/src/cli/commands/delivery.ts +31 -15
- package/src/cli/commands/eval.ts +28 -0
- package/src/cli/commands/ingest.ts +141 -0
- package/src/cli/index.ts +4 -0
- package/src/dashboard/templates/index.html +108 -194
- package/src/harness/audit.ts +81 -4
- package/src/harness/capability-plan.ts +11 -0
- package/src/harness/eval/skill-lint.ts +87 -0
- package/src/harness/parse.ts +19 -3
- package/src/harness/quality-gates.ts +152 -0
- package/src/harness/sensors.ts +84 -7
- package/src/harness/spec-coverage.ts +139 -0
- package/src/harness/viewpoint-ledger.ts +80 -0
- package/src/ingest/baseline-audit.ts +100 -0
- package/src/ingest/gsheet-fetch.ts +152 -0
- package/src/ingest/index.ts +5 -0
- package/src/ingest/legacy-parser.ts +184 -0
- package/src/ingest/reconcile.ts +80 -0
- package/src/ingest/to-gherkin.ts +108 -0
- package/src/orchestrator/ai-rules-updater.ts +2 -0
- package/src/orchestrator/templates/ai-instructions/claude-agent-reviewer.md +1 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-delivery.md +10 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-ingest-legacy.md +79 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +25 -1
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-delivery.md +10 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-ingest-legacy.md +79 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +44 -7
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Viewpoint Reconciliation (P-C) — the link between a legacy suite and the Harness.
|
|
3
|
+
*
|
|
4
|
+
* A 1:1 convert reproduces the legacy suite's blind spots. Reconciliation runs the
|
|
5
|
+
* harness viewpoint-gate over the legacy-derived scenarios to answer: which catalog-
|
|
6
|
+
* expected viewpoints does the legacy suite ALREADY cover, and which are MISSING
|
|
7
|
+
* (blind spots the old QA never wrote)? It emits a draft viewpoint overview that
|
|
8
|
+
* seeds `/sungen:create-test` + the harness coverage-gate, so the generated suite is
|
|
9
|
+
* raised to catalog quality instead of merely mirroring the legacy.
|
|
10
|
+
*/
|
|
11
|
+
import { LegacyInventory } from './legacy-parser';
|
|
12
|
+
import { legacyToScenarioInfo } from './baseline-audit';
|
|
13
|
+
import { loadCatalog, viewpointGate } from '../harness/sensors';
|
|
14
|
+
|
|
15
|
+
export interface Reconciliation {
|
|
16
|
+
pageType: string | null;
|
|
17
|
+
coverageRatio: number;
|
|
18
|
+
themesCovered: number;
|
|
19
|
+
themesTotal: number;
|
|
20
|
+
blindSpots: { theme: string; status: 'missing' | 'shallow' }[]; // catalog themes the legacy suite lacks/under-covers
|
|
21
|
+
universalGaps: string[];
|
|
22
|
+
legacyViewpoints: { vpGroup: string; count: number }[]; // what the legacy suite implies
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
function vpGroupOf(category: string): string {
|
|
26
|
+
const c = (category || '').toLowerCase();
|
|
27
|
+
if (/search|filter|điều kiện|条件/.test(c)) return 'SEARCH';
|
|
28
|
+
if (/input|validation|nhập|入力/.test(c)) return 'VAL';
|
|
29
|
+
if (/display|hiển thị|表示|data/.test(c)) return 'DISPLAY';
|
|
30
|
+
if (/layout|responsive|tab order|ui|giao diện/.test(c)) return 'UI';
|
|
31
|
+
if (/flow|navigation|遷移|chuyển|event/.test(c)) return 'NAV';
|
|
32
|
+
if (/paging|sort|phân trang/.test(c)) return 'LIST';
|
|
33
|
+
return (c.replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '').toUpperCase().slice(0, 8)) || 'GEN';
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
export function reconcileViewpoints(inv: LegacyInventory): Reconciliation {
|
|
37
|
+
const scenarios = inv.testcases.map(legacyToScenarioInfo);
|
|
38
|
+
const catalog = loadCatalog();
|
|
39
|
+
const gate = viewpointGate(scenarios, [], catalog);
|
|
40
|
+
|
|
41
|
+
const groups: Record<string, number> = {};
|
|
42
|
+
for (const tc of inv.testcases) {
|
|
43
|
+
const g = vpGroupOf(tc.category);
|
|
44
|
+
groups[g] = (groups[g] || 0) + 1;
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
return {
|
|
48
|
+
pageType: gate.pageType,
|
|
49
|
+
coverageRatio: gate.coverageRatio,
|
|
50
|
+
themesCovered: gate.themesCovered,
|
|
51
|
+
themesTotal: gate.themesTotal,
|
|
52
|
+
blindSpots: gate.gaps.map((g) => ({ theme: g.theme, status: g.status })),
|
|
53
|
+
universalGaps: gate.universalGaps,
|
|
54
|
+
legacyViewpoints: Object.entries(groups).map(([vpGroup, count]) => ({ vpGroup, count }))
|
|
55
|
+
.sort((a, b) => b.count - a.count),
|
|
56
|
+
};
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
/** A draft test-viewpoint.md: legacy-derived viewpoints + catalog blind-spots to fill. */
|
|
60
|
+
export function renderViewpointOverview(featureName: string, r: Reconciliation): string {
|
|
61
|
+
const L: string[] = [];
|
|
62
|
+
L.push(`# Test Viewpoint — ${featureName} (drafted from legacy + reconciled with the catalog)`, '');
|
|
63
|
+
L.push('> Auto-drafted by `sungen ingest`. Rows from the legacy suite are listed first;');
|
|
64
|
+
L.push('> rows marked **⚠ BLIND-SPOT** are catalog-expected viewpoints the legacy suite does');
|
|
65
|
+
L.push('> NOT cover — add scenarios for them via `/sungen:create-test`. Refine before use.', '');
|
|
66
|
+
L.push(`Detected page-type: \`${r.pageType ?? 'unknown'}\` · legacy covers ${r.themesCovered}/${r.themesTotal} catalog themes (${(r.coverageRatio * 100).toFixed(0)}%)`, '');
|
|
67
|
+
L.push('## Priority Viewpoints', '', '| VP | Priority | Reason |', '|---|---|---|');
|
|
68
|
+
let n = 1;
|
|
69
|
+
for (const v of r.legacyViewpoints) {
|
|
70
|
+
L.push(`| VP-${v.vpGroup}-${String(n).padStart(3, '0')} | Medium | legacy suite: ${v.count} testcase(s) |`);
|
|
71
|
+
n++;
|
|
72
|
+
}
|
|
73
|
+
for (const b of r.blindSpots) {
|
|
74
|
+
L.push(`| VP-GAP-${String(n).padStart(3, '0')} | High | ⚠ BLIND-SPOT — catalog expects "${b.theme}" (${b.status}); legacy has none |`);
|
|
75
|
+
n++;
|
|
76
|
+
}
|
|
77
|
+
if (r.universalGaps.length)
|
|
78
|
+
L.push('', `> Universal reminders not covered: ${r.universalGaps.join(', ')}`);
|
|
79
|
+
return L.join('\n') + '\n';
|
|
80
|
+
}
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Legacy inventory → Gherkin DRAFT + traceability + gap report (P-B).
|
|
3
|
+
*
|
|
4
|
+
* Deterministic SCAFFOLDING only: it preserves every legacy testcase as a scenario
|
|
5
|
+
* with its source (TC id + verbatim Steps/Expected as comments), the right priority +
|
|
6
|
+
* @manual tags (from the capability classifier), and a Given seeded from the
|
|
7
|
+
* precondition. The free-text → real `[Reference] type` step mapping is a SEMANTIC step
|
|
8
|
+
* done by the AI layer (`/sungen:create-test`) on top of this draft — the draft makes
|
|
9
|
+
* that refinement traceable and correctly scoped, it is not final runnable Gherkin.
|
|
10
|
+
*/
|
|
11
|
+
import { LegacyInventory, LegacyTestcase } from './legacy-parser';
|
|
12
|
+
import { classifyReason, MANUAL_REASONS } from '../harness/capability-plan';
|
|
13
|
+
|
|
14
|
+
export interface TraceEntry {
|
|
15
|
+
tcId: string;
|
|
16
|
+
scenario: string;
|
|
17
|
+
vpGroup: string;
|
|
18
|
+
mode: 'ui' | 'cross-screen' | 'manual-capability' | 'manual-keep';
|
|
19
|
+
reasonCode: string; // '' for ui
|
|
20
|
+
drivers: string[];
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
export interface ConvertResult {
|
|
24
|
+
feature: string; // the draft .feature text
|
|
25
|
+
trace: TraceEntry[];
|
|
26
|
+
gap: {
|
|
27
|
+
total: number; ui: number; crossScreen: number;
|
|
28
|
+
manualCapability: number; manualKeep: number; noExpected: number;
|
|
29
|
+
};
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
// Map a legacy category phrase → a VP group code (best-effort; falls back to a slug).
|
|
33
|
+
function vpGroupOf(category: string): string {
|
|
34
|
+
const c = (category || '').toLowerCase();
|
|
35
|
+
if (/search|filter|điều kiện|条件/.test(c)) return 'SEARCH';
|
|
36
|
+
if (/input|validation|nhập|入力/.test(c)) return 'VAL';
|
|
37
|
+
if (/display|hiển thị|表示|data/.test(c)) return 'DISPLAY';
|
|
38
|
+
if (/layout|responsive|tab order|ui|giao diện/.test(c)) return 'UI';
|
|
39
|
+
if (/flow|navigation|遷移|chuyển|event/.test(c)) return 'NAV';
|
|
40
|
+
if (/paging|sort|phân trang/.test(c)) return 'LIST';
|
|
41
|
+
const slug = c.replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '').toUpperCase().slice(0, 8);
|
|
42
|
+
return slug || 'GEN';
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
function oneLine(s: string | undefined, max = 60): string {
|
|
46
|
+
return (s || '').replace(/\s+/g, ' ').trim().slice(0, max);
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
function classify(tc: LegacyTestcase): { mode: TraceEntry['mode']; code: string; drivers: string[] } {
|
|
50
|
+
const code = classifyReason(`${tc.precondition || ''} ${tc.steps} ${tc.expected} ${tc.testData || ''}`);
|
|
51
|
+
if (!code) return { mode: 'ui', code: '', drivers: [] };
|
|
52
|
+
if (code === 'XS') return { mode: 'cross-screen', code, drivers: [] };
|
|
53
|
+
const def = MANUAL_REASONS[code];
|
|
54
|
+
if (def?.cls === 'keep') return { mode: 'manual-keep', code, drivers: [] };
|
|
55
|
+
return { mode: 'manual-capability', code, drivers: def?.drivers || [] };
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
export function inventoryToGherkin(inv: LegacyInventory, featureName: string): ConvertResult {
|
|
59
|
+
const tcs = inv.testcases;
|
|
60
|
+
const trace: TraceEntry[] = [];
|
|
61
|
+
const gap = { total: tcs.length, ui: 0, crossScreen: 0, manualCapability: 0, manualKeep: 0, noExpected: 0 };
|
|
62
|
+
const seq: Record<string, number> = {};
|
|
63
|
+
|
|
64
|
+
const lines: string[] = [];
|
|
65
|
+
lines.push('@legacy');
|
|
66
|
+
lines.push(`Feature: ${featureName} (drafted from legacy testcases: ${inv.source.file})`);
|
|
67
|
+
lines.push('');
|
|
68
|
+
lines.push(' # Auto-drafted by `sungen ingest --emit-gherkin` from a legacy manual testcase suite.');
|
|
69
|
+
lines.push(' # Each scenario carries its source TC id (@legacy:<id>) + the original');
|
|
70
|
+
lines.push(' # precondition/steps/expected as comments. REFINE the When/Then into real');
|
|
71
|
+
lines.push(' # [Reference] type steps with /sungen:create-test — the free-text → step');
|
|
72
|
+
lines.push(' # mapping is an AI step; this draft only fixes structure + traceability.');
|
|
73
|
+
lines.push('');
|
|
74
|
+
|
|
75
|
+
for (const tc of tcs) {
|
|
76
|
+
const vp = vpGroupOf(tc.category);
|
|
77
|
+
seq[vp] = (seq[vp] || 0) + 1;
|
|
78
|
+
const code = `VP-${vp}-${String(seq[vp]).padStart(3, '0')}`;
|
|
79
|
+
const desc = oneLine(tc.subCategory || tc.steps) || tc.id;
|
|
80
|
+
const title = `${code} ${desc}`;
|
|
81
|
+
const { mode, code: reason, drivers } = classify(tc);
|
|
82
|
+
if (!tc.expected) gap.noExpected++;
|
|
83
|
+
if (mode === 'ui') gap.ui++;
|
|
84
|
+
else if (mode === 'cross-screen') gap.crossScreen++;
|
|
85
|
+
else if (mode === 'manual-capability') gap.manualCapability++;
|
|
86
|
+
else gap.manualKeep++;
|
|
87
|
+
|
|
88
|
+
const tags = [`@${tc.priority === 'unknown' ? 'normal' : tc.priority}`];
|
|
89
|
+
if (mode !== 'ui') tags.push('@manual');
|
|
90
|
+
tags.push(`@legacy:${tc.id}`);
|
|
91
|
+
|
|
92
|
+
lines.push(` ${tags.join(' ')}`);
|
|
93
|
+
lines.push(` Scenario: ${title}`);
|
|
94
|
+
if (mode === 'manual-capability') lines.push(` # @manual:${reason} — needs ${drivers.join('/') || 'capability'} (driver candidate)`);
|
|
95
|
+
else if (mode === 'manual-keep') lines.push(` # @manual:${reason} — judgment floor, keep manual`);
|
|
96
|
+
else if (mode === 'cross-screen') lines.push(` # cross-screen → model as a flow (/sungen:add-flow)`);
|
|
97
|
+
if (tc.precondition) lines.push(` # precondition: ${oneLine(tc.precondition, 120)}`);
|
|
98
|
+
lines.push(` # steps: ${oneLine(tc.steps, 160)}`);
|
|
99
|
+
lines.push(` # expected: ${oneLine(tc.expected, 160)}`);
|
|
100
|
+
if (tc.precondition) lines.push(` Given User is on [${featureName}] page`);
|
|
101
|
+
lines.push(' # TODO(create-test): map the steps above to real When/Then [Reference] steps');
|
|
102
|
+
lines.push('');
|
|
103
|
+
|
|
104
|
+
trace.push({ tcId: tc.id, scenario: title, vpGroup: vp, mode, reasonCode: reason, drivers });
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
return { feature: lines.join('\n'), trace, gap };
|
|
108
|
+
}
|
|
@@ -47,6 +47,7 @@ export const AI_RULES_FILE_MAPPING: [string, string][] = [
|
|
|
47
47
|
['claude-skill-selector-fix.md', '.claude/skills/sungen-selector-fix/SKILL.md'],
|
|
48
48
|
['claude-skill-tc-review.md', '.claude/skills/sungen-tc-review/SKILL.md'],
|
|
49
49
|
['claude-skill-harness-audit.md', '.claude/skills/sungen-harness-audit/SKILL.md'],
|
|
50
|
+
['claude-skill-ingest-legacy.md', '.claude/skills/sungen-ingest-legacy/SKILL.md'],
|
|
50
51
|
['claude-skill-viewpoint.md', '.claude/skills/sungen-viewpoint/SKILL.md'],
|
|
51
52
|
['claude-skill-viewpoint-group-a-data-entry.md', '.claude/skills/sungen-viewpoint/group-a-data-entry.md'],
|
|
52
53
|
['claude-skill-viewpoint-group-b-data-ops.md', '.claude/skills/sungen-viewpoint/group-b-data-ops.md'],
|
|
@@ -78,6 +79,7 @@ export const AI_RULES_FILE_MAPPING: [string, string][] = [
|
|
|
78
79
|
['github-skill-sungen-selector-fix.md', '.github/skills/sungen-selector-fix/SKILL.md'],
|
|
79
80
|
['github-skill-sungen-tc-review.md', '.github/skills/sungen-tc-review/SKILL.md'],
|
|
80
81
|
['github-skill-sungen-harness-audit.md', '.github/skills/sungen-harness-audit/SKILL.md'],
|
|
82
|
+
['github-skill-sungen-ingest-legacy.md', '.github/skills/sungen-ingest-legacy/SKILL.md'],
|
|
81
83
|
['github-skill-sungen-viewpoint.md', '.github/skills/sungen-viewpoint/SKILL.md'],
|
|
82
84
|
['github-skill-sungen-viewpoint-group-a-data-entry.md', '.github/skills/sungen-viewpoint/group-a-data-entry.md'],
|
|
83
85
|
['github-skill-sungen-viewpoint-group-b-data-ops.md', '.github/skills/sungen-viewpoint/group-b-data-ops.md'],
|
|
@@ -14,6 +14,7 @@ You are an **independent Senior QA Reviewer**. You did **not** write these tests
|
|
|
14
14
|
|
|
15
15
|
## What to judge (semantic — the gate misses these)
|
|
16
16
|
1. **Title ↔ steps proof.** For every scenario, do the **steps actually prove the title/viewpoint**? Flag "title claims X but steps only assert Y". (e.g. title "adds the selected product, not a random one" but Then only `see [Added] modal`.)
|
|
17
|
+
- **Negative / "does-not-happen" claims** (any language — "does not", "no", "prevents", "không", "chưa"): the proof must be a step whose result **differs** between the claim holding and not holding. Ask: *would this `Then` still pass if the bad thing happened?* If yes, it proves nothing. The classic trap: title "browser back does **not** re-submit" with `Then see [sent] page` — that page is identical whether or not the request re-fired. Demand a **contrast/count** proof (record count unchanged, state hidden/empty, error shown) or a justified `@manual` with a setup→action→assert-absence oracle. This generalises to every side-effect (re-charge, duplicate order, resend OTP, data leak), not just re-submit.
|
|
17
18
|
2. **Observable Then.** Is each `Then` an **observable outcome**, not a restated action or a tautology (e.g. `Then User see [Carousel] section` after clicking next — proves nothing changed)?
|
|
18
19
|
3. **Business-critical depth.** For cart / product-detail / filter / list viewpoints, do steps assert **DATA** (name, price, quantity, all-items-belong) — not just page/modal visibility? Recommend the concrete deep step: `User remember [X] text as {{v}}` + `... with {{v}}`, or `User see all [X] contain {{v}}`.
|
|
19
20
|
4. **@manual justification.** Is each `@manual` genuinely unautomatable (cross-screen/external/visual) — or a cop-out to dodge the gate? Cross-screen → should be a flow.
|
|
@@ -78,6 +78,16 @@ The CLI reads the **per-target result file first** (co-located with `.spec.ts`),
|
|
|
78
78
|
|
|
79
79
|
---
|
|
80
80
|
|
|
81
|
+
## XLSX sheets — Auto / Manual split
|
|
82
|
+
|
|
83
|
+
The `.xlsx` is split into two sheets so QA manages the sets separately:
|
|
84
|
+
- **`Auto`** — automatable test cases (`Auto` + `Not compiled`).
|
|
85
|
+
- **`Manual`** — `@manual` test cases (always present, header-only when there are none).
|
|
86
|
+
|
|
87
|
+
Multi-locale (no `SUNGEN_ENV`): one **`<LOCALE> Auto`** sheet per locale + a single shared **`Manual`** sheet (manual TCs are locale-invariant). The **CSV stays one file with every row** — the `Testcase type` column distinguishes Auto vs Manual.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
81
91
|
## Excluded from CSV
|
|
82
92
|
|
|
83
93
|
- `@steps:<name>` **base** scenarios — these are setup-only, inlined into `@extend:...` scenarios at compile time
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: sungen-ingest-legacy
|
|
3
|
+
description: 'Import a legacy manual testcase suite from Google Sheets (multi-tab) or a local file into Sungen — fetch via MCP, then sungen ingest. Use when the user wants to convert/evaluate an existing manual testcase spreadsheet.'
|
|
4
|
+
user-invocable: true
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# sungen-ingest-legacy
|
|
8
|
+
|
|
9
|
+
Bring an existing **manual testcase workbook** into Sungen for evaluation + conversion. The
|
|
10
|
+
fetch (Google login + pick file) is done here via MCP; the parsing/audit is deterministic
|
|
11
|
+
(`sungen ingest`). **Security:** the workbook is the user's project data — read it on
|
|
12
|
+
consent, keep the output in their project, never upload or commit the content.
|
|
13
|
+
|
|
14
|
+
## Flow
|
|
15
|
+
|
|
16
|
+
1. **Locate the source.**
|
|
17
|
+
- **Google Sheets (recommended):** use the Google Drive MCP. Authenticate if needed, then
|
|
18
|
+
`search_files` / list to find the workbook; confirm the file with the user.
|
|
19
|
+
- **Local:** if the user points to a `.xlsx`/`.csv`, skip to step 4.
|
|
20
|
+
|
|
21
|
+
2. **List the tabs.** Read the workbook's sheet/tab names (Drive MCP file metadata or a values
|
|
22
|
+
read). A legacy workbook usually has **many** tabs — some are testcases, some are
|
|
23
|
+
viewpoint/UI matrices.
|
|
24
|
+
|
|
25
|
+
3. **Assemble a JSON sheet-bundle.** For each tab, read its cell values (a 2-D array of
|
|
26
|
+
strings) and write a local bundle in the user project (e.g.
|
|
27
|
+
`qa/screens/<screen>/requirements/legacy/bundle.json`):
|
|
28
|
+
```json
|
|
29
|
+
{ "source": "<workbook name>", "sheets": [ { "name": "<tab>", "rows": [["TC ID","Page",…],["TC-01",…]] } ] }
|
|
30
|
+
```
|
|
31
|
+
Record only the **source link** in `requirements/legacy/source.yaml` — never the content.
|
|
32
|
+
|
|
33
|
+
4. **Classify the tabs.** Run:
|
|
34
|
+
```bash
|
|
35
|
+
sungen ingest --legacy <bundle.json|file.xlsx|file.csv> --list-sheets
|
|
36
|
+
```
|
|
37
|
+
It prints each tab + detected type (`testcase` / `viewpoint-matrix` / `ui-checklist`).
|
|
38
|
+
|
|
39
|
+
5. **Confirm which tabs to ingest.** Use `AskUserQuestion` to let the user pick the
|
|
40
|
+
**testcase** tabs (matrix/UI tabs feed the viewpoint layer later, not the inventory).
|
|
41
|
+
|
|
42
|
+
6. **Ingest + reconcile.**
|
|
43
|
+
```bash
|
|
44
|
+
sungen ingest --legacy <source> --screen <screen> --sheets "<Tab A>,<Tab B>" --emit-gherkin
|
|
45
|
+
```
|
|
46
|
+
Produces: `inventory.json` (+ baseline audit), `*.legacy-draft.feature` + `legacy-trace.json`
|
|
47
|
+
(parity: `@legacy:<id>` per scenario), and `test-viewpoint.draft.md` with **blind-spots**
|
|
48
|
+
(catalog-expected viewpoints the legacy suite lacks).
|
|
49
|
+
|
|
50
|
+
7. **Hand off to quality.** Tell the user the next step is `/sungen:create-test <screen>` —
|
|
51
|
+
it discovers + refines the draft into real `[Reference]` steps and fills the blind-spots;
|
|
52
|
+
then `sungen audit <screen>` gates quality. A 1:1 convert is NOT the deliverable; the
|
|
53
|
+
harness raises the legacy floor to catalog quality.
|
|
54
|
+
|
|
55
|
+
## Governance block (important)
|
|
56
|
+
|
|
57
|
+
Many orgs mark confidential files as **"ineligible for generative AI contexts"** — the
|
|
58
|
+
Google Drive MCP will then **refuse** to read the file (metadata + download both error).
|
|
59
|
+
This is the org's DLP policy, not a bug, and it is the *expected* outcome for a
|
|
60
|
+
confidential testcase suite. When you hit it, **do not retry** — fall back:
|
|
61
|
+
|
|
62
|
+
> "This sheet is restricted by your org's data policy, so I can't read it through the
|
|
63
|
+
> AI connector. Two ways to proceed, both running as **you**, not AI:
|
|
64
|
+
> (1) `sungen ingest --gsheet <url>` — fetches under your own Google identity
|
|
65
|
+
> (read-only; Viewer/Commenter is enough). It offers to install `googleapis`
|
|
66
|
+
> and to open the Google login in your browser (pick your account), then
|
|
67
|
+
> retries automatically. Needs the gcloud SDK for the browser login.
|
|
68
|
+
> (2) Export it manually (**File → Download → Microsoft Excel `.xlsx`**) and I'll run
|
|
69
|
+
> `sungen ingest --legacy <file>.xlsx`."
|
|
70
|
+
|
|
71
|
+
The local-file path is deterministic and **never sends the content through AI** — the
|
|
72
|
+
correct, governance-compliant channel for confidential data. The MCP auto-pick is only
|
|
73
|
+
for files the org does *not* restrict.
|
|
74
|
+
|
|
75
|
+
## Notes
|
|
76
|
+
- Multiple local CSVs (one per tab) also work: `--legacy tab1.csv tab2.csv …`.
|
|
77
|
+
- Re-run only re-fetches when the user asks; otherwise reuse the saved bundle.
|
|
78
|
+
- Do not invent testcases. Only ingest what the workbook contains; the *augmentation*
|
|
79
|
+
(blind-spots) happens in `/sungen:create-test`, flagged for human review.
|
|
@@ -105,6 +105,9 @@ Auto-detected by `create-test` before invoking this skill:
|
|
|
105
105
|
2. Each row / bullet / item = 1 viewpoint → add to `Viewpoint items` in Coverage Map.
|
|
106
106
|
3. Do NOT pre-classify into buckets before scanning — classify only when
|
|
107
107
|
writing the scenario.
|
|
108
|
+
4. **If it declares viewpoint IDs** (e.g. `VP0`, `VP1`…`VP12`, `MS-HP-001`), capture each
|
|
109
|
+
item WITH its ID and **reuse that ID as the scenario code** — do not invent a generic
|
|
110
|
+
`VP-<CAT>` scheme (the harness Taxonomy-match gate FAILs on mismatch).
|
|
108
111
|
- `qa/context.md` — project-wide context set by the QA lead. Read ONCE before building the Coverage Map; apply to every screen. Extraction rules:
|
|
109
112
|
- **Roles** → for each role in the table: add to the `@auth:X` tag pool; generate a VP-SEC blocked-access scenario for every role boundary relevant to this screen.
|
|
110
113
|
- **Testing strategy → Focus areas** → if `security` listed: VP-SEC is mandatory Tier 1 for every free-text input regardless of spec risk level; if `ui` not listed: all VP-UI scenarios move to Tier 2 minimum.
|
|
@@ -260,6 +263,27 @@ Security: [S1 – admin only]
|
|
|
260
263
|
|
|
261
264
|
**Balance:** cover all the above (deep) BEFORE expanding subscription / UI-presence / extra validation edge cases. Do not over-invest in subscription while cart/detail/filter correctness are shallow.
|
|
262
265
|
|
|
266
|
+
#### Harness gates — satisfy on the FIRST pass (don't make the repair loop fix them)
|
|
267
|
+
|
|
268
|
+
`sungen audit` enforces these. Generate compliant output up front:
|
|
269
|
+
|
|
270
|
+
1. **Taxonomy-match** (`VP-TAXONOMY-MISMATCH`, gate-FAIL) — when `test-viewpoint.md` declares its own viewpoint IDs (e.g. `VP0`, `VP1`, … `VP12`, `MS-HP-001`, `MS-EH-001`), **reuse those IDs verbatim as the scenario codes**. Do NOT invent a generic `VP-UI / VP-LOGIC / VP-VAL` scheme — that breaks the coverage matrix. Only fall back to `VP-<CATEGORY>-<NNN>` when the viewpoint file declares no IDs.
|
|
271
|
+
2. **Spec-coverage triggers** (`TRIGGER-UNCOVERED`, gate-FAIL) — the Validation-Rules table lists a **trigger** per constraint (e.g. `blur, submit`). Generate one scenario **per (constraint × trigger)** — a `format` rule validating *on blur AND on submit* needs BOTH a blur scenario (`press Tab`) and a submit scenario (`click [Submit]` / `press Enter`). Never collapse the trigger × input matrix to one representative case.
|
|
272
|
+
3. **Claim-Proof** (`CLAIM-UNPROVEN`) — a title claiming `all`/`only`/`every`/`single`/`correct`/`same`/`changes`/`hidden`/`cleared`/`restored`/`independent`/`sanitized`/`announces` MUST have the matching assertion (`see all …`, count, `remember`+compare, `is hidden`, return-and-assert-empty, etc.). If the title promises it, the steps must prove it.
|
|
273
|
+
- **Negative / absence claims** (`does not` / `no` / `never` / `prevents` / `không` / `chưa` — any language; `no-side-effect/no-duplicate`, `negative-claim/absence`): the `Then` must **differ** between the claim holding and not holding. A terminal `see [X] page` that looks identical whether or not the bad thing happened proves nothing. For a side-effect that should NOT repeat (re-submit on back, re-charge, duplicate order, resend OTP), assert the **count is unchanged** (`User see [Records] table with {{one}}` / `row with {{count}}`); if it's not UI-observable, mark `@manual` with a request-count oracle (shape below). This is general — it covers any side-effect, not a fixed verb list.
|
|
274
|
+
4. **Downstream-scope** (`DOWNSTREAM-SCOPE-MISSING`) — when the spec's Navigation Flow / success target is **another screen** (e.g. a confirmation/sent page), don't stop at a terminal `see [X] page`. Either cover that screen's content/guards (if its viewpoint items are in scope — they often have their own `MS-*` IDs), or scaffold it (`sungen add --screen <name>`) and note the handoff. Do not silently drop the downstream surface.
|
|
275
|
+
5. **Manual-oracle** (`MANUAL-STEPS-INSUFFICIENT`) — every `@manual` scenario needs **setup · action · observable expected · oracle/tool**, not a one-line note. Use this comment shape:
|
|
276
|
+
```gherkin
|
|
277
|
+
@high @manual
|
|
278
|
+
Scenario: VP-… <claim>
|
|
279
|
+
# MANUAL: <why it can't be automated — needs network capture / inbox / screen-reader / multi-tab>
|
|
280
|
+
# Tester verifies:
|
|
281
|
+
# 1. <setup> e.g. seed a registered email; throttle the network
|
|
282
|
+
# 2. <action> e.g. click [Submit] with the request in flight
|
|
283
|
+
# 3. <observable> e.g. only ONE POST is dispatched
|
|
284
|
+
# 4. Oracle: <tool> e.g. DevTools Network panel / mail-catcher / NVDA
|
|
285
|
+
```
|
|
286
|
+
|
|
263
287
|
#### Tier 1 guard — minimum before writing scenarios
|
|
264
288
|
|
|
265
289
|
| Spec section | Minimum requirement | Tag |
|
|
@@ -376,7 +400,7 @@ Add cleanup tags per the `sungen-gherkin-syntax` Cleanup table. Key rules:
|
|
|
376
400
|
**Files:** `qa/screens/<screen>/features/<screen>.feature` + `qa/screens/<screen>/test-data/<screen>.yaml`
|
|
377
401
|
|
|
378
402
|
Use step patterns and element types from `sungen-gherkin-syntax`.
|
|
379
|
-
**Naming**: `VP-<CATEGORY>-<NNN>`. Scenario name must use the **same element type** as the steps.
|
|
403
|
+
**Naming**: reuse the **project's `test-viewpoint.md` IDs** when it declares them (e.g. `VP0`, `MS-HP-001`); otherwise `VP-<CATEGORY>-<NNN>`. Scenario name must use the **same element type** as the steps.
|
|
380
404
|
|
|
381
405
|
**Test data** — grouped by section, loaded at runtime:
|
|
382
406
|
|
|
@@ -78,6 +78,16 @@ The CLI reads the **per-target result file first** (co-located with `.spec.ts`),
|
|
|
78
78
|
|
|
79
79
|
---
|
|
80
80
|
|
|
81
|
+
## XLSX sheets — Auto / Manual split
|
|
82
|
+
|
|
83
|
+
The `.xlsx` is split into two sheets so QA manages the sets separately:
|
|
84
|
+
- **`Auto`** — automatable test cases (`Auto` + `Not compiled`).
|
|
85
|
+
- **`Manual`** — `@manual` test cases (always present, header-only when there are none).
|
|
86
|
+
|
|
87
|
+
Multi-locale (no `SUNGEN_ENV`): one **`<LOCALE> Auto`** sheet per locale + a single shared **`Manual`** sheet (manual TCs are locale-invariant). The **CSV stays one file with every row** — the `Testcase type` column distinguishes Auto vs Manual.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
81
91
|
## Excluded from CSV
|
|
82
92
|
|
|
83
93
|
- `@steps:<name>` **base** scenarios — these are setup-only, inlined into `@extend:...` scenarios at compile time
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: sungen-ingest-legacy
|
|
3
|
+
description: 'Import a legacy manual testcase suite from Google Sheets (multi-tab) or a local file into Sungen — fetch via MCP, then sungen ingest. Use when the user wants to convert/evaluate an existing manual testcase spreadsheet.'
|
|
4
|
+
user-invocable: true
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# sungen-ingest-legacy
|
|
8
|
+
|
|
9
|
+
Bring an existing **manual testcase workbook** into Sungen for evaluation + conversion. The
|
|
10
|
+
fetch (Google login + pick file) is done here via MCP; the parsing/audit is deterministic
|
|
11
|
+
(`sungen ingest`). **Security:** the workbook is the user's project data — read it on
|
|
12
|
+
consent, keep the output in their project, never upload or commit the content.
|
|
13
|
+
|
|
14
|
+
## Flow
|
|
15
|
+
|
|
16
|
+
1. **Locate the source.**
|
|
17
|
+
- **Google Sheets (recommended):** use the Google Drive MCP. Authenticate if needed, then
|
|
18
|
+
`search_files` / list to find the workbook; confirm the file with the user.
|
|
19
|
+
- **Local:** if the user points to a `.xlsx`/`.csv`, skip to step 4.
|
|
20
|
+
|
|
21
|
+
2. **List the tabs.** Read the workbook's sheet/tab names (Drive MCP file metadata or a values
|
|
22
|
+
read). A legacy workbook usually has **many** tabs — some are testcases, some are
|
|
23
|
+
viewpoint/UI matrices.
|
|
24
|
+
|
|
25
|
+
3. **Assemble a JSON sheet-bundle.** For each tab, read its cell values (a 2-D array of
|
|
26
|
+
strings) and write a local bundle in the user project (e.g.
|
|
27
|
+
`qa/screens/<screen>/requirements/legacy/bundle.json`):
|
|
28
|
+
```json
|
|
29
|
+
{ "source": "<workbook name>", "sheets": [ { "name": "<tab>", "rows": [["TC ID","Page",…],["TC-01",…]] } ] }
|
|
30
|
+
```
|
|
31
|
+
Record only the **source link** in `requirements/legacy/source.yaml` — never the content.
|
|
32
|
+
|
|
33
|
+
4. **Classify the tabs.** Run:
|
|
34
|
+
```bash
|
|
35
|
+
sungen ingest --legacy <bundle.json|file.xlsx|file.csv> --list-sheets
|
|
36
|
+
```
|
|
37
|
+
It prints each tab + detected type (`testcase` / `viewpoint-matrix` / `ui-checklist`).
|
|
38
|
+
|
|
39
|
+
5. **Confirm which tabs to ingest.** Use `AskUserQuestion` to let the user pick the
|
|
40
|
+
**testcase** tabs (matrix/UI tabs feed the viewpoint layer later, not the inventory).
|
|
41
|
+
|
|
42
|
+
6. **Ingest + reconcile.**
|
|
43
|
+
```bash
|
|
44
|
+
sungen ingest --legacy <source> --screen <screen> --sheets "<Tab A>,<Tab B>" --emit-gherkin
|
|
45
|
+
```
|
|
46
|
+
Produces: `inventory.json` (+ baseline audit), `*.legacy-draft.feature` + `legacy-trace.json`
|
|
47
|
+
(parity: `@legacy:<id>` per scenario), and `test-viewpoint.draft.md` with **blind-spots**
|
|
48
|
+
(catalog-expected viewpoints the legacy suite lacks).
|
|
49
|
+
|
|
50
|
+
7. **Hand off to quality.** Tell the user the next step is `/sungen:create-test <screen>` —
|
|
51
|
+
it discovers + refines the draft into real `[Reference]` steps and fills the blind-spots;
|
|
52
|
+
then `sungen audit <screen>` gates quality. A 1:1 convert is NOT the deliverable; the
|
|
53
|
+
harness raises the legacy floor to catalog quality.
|
|
54
|
+
|
|
55
|
+
## Governance block (important)
|
|
56
|
+
|
|
57
|
+
Many orgs mark confidential files as **"ineligible for generative AI contexts"** — the
|
|
58
|
+
Google Drive MCP will then **refuse** to read the file (metadata + download both error).
|
|
59
|
+
This is the org's DLP policy, not a bug, and it is the *expected* outcome for a
|
|
60
|
+
confidential testcase suite. When you hit it, **do not retry** — fall back:
|
|
61
|
+
|
|
62
|
+
> "This sheet is restricted by your org's data policy, so I can't read it through the
|
|
63
|
+
> AI connector. Two ways to proceed, both running as **you**, not AI:
|
|
64
|
+
> (1) `sungen ingest --gsheet <url>` — fetches under your own Google identity
|
|
65
|
+
> (read-only; Viewer/Commenter is enough). It offers to install `googleapis`
|
|
66
|
+
> and to open the Google login in your browser (pick your account), then
|
|
67
|
+
> retries automatically. Needs the gcloud SDK for the browser login.
|
|
68
|
+
> (2) Export it manually (**File → Download → Microsoft Excel `.xlsx`**) and I'll run
|
|
69
|
+
> `sungen ingest --legacy <file>.xlsx`."
|
|
70
|
+
|
|
71
|
+
The local-file path is deterministic and **never sends the content through AI** — the
|
|
72
|
+
correct, governance-compliant channel for confidential data. The MCP auto-pick is only
|
|
73
|
+
for files the org does *not* restrict.
|
|
74
|
+
|
|
75
|
+
## Notes
|
|
76
|
+
- Multiple local CSVs (one per tab) also work: `--legacy tab1.csv tab2.csv …`.
|
|
77
|
+
- Re-run only re-fetches when the user asks; otherwise reuse the saved bundle.
|
|
78
|
+
- Do not invent testcases. Only ingest what the workbook contains; the *augmentation*
|
|
79
|
+
(blind-spots) happens in `/sungen:create-test`, flagged for human review.
|
|
@@ -105,6 +105,17 @@ Auto-detected by `create-test` before invoking this skill:
|
|
|
105
105
|
2. Each row / bullet / item = 1 viewpoint → add to `Viewpoint items` in Coverage Map.
|
|
106
106
|
3. Do NOT pre-classify into buckets before scanning — classify only when
|
|
107
107
|
writing the scenario.
|
|
108
|
+
4. **If it declares viewpoint IDs** (e.g. `VP0`, `VP1`…`VP12`, `MS-HP-001`), capture each
|
|
109
|
+
item WITH its ID and **reuse that ID as the scenario code** — do not invent a generic
|
|
110
|
+
`VP-<CAT>` scheme (the harness Taxonomy-match gate FAILs on mismatch).
|
|
111
|
+
- `qa/context.md` — project-wide context set by the QA lead. Read ONCE before building the Coverage Map; apply to every screen. Extraction rules:
|
|
112
|
+
- **Roles** → for each role in the table: add to the `@auth:X` tag pool; generate a VP-SEC blocked-access scenario for every role boundary relevant to this screen.
|
|
113
|
+
- **Testing strategy → Focus areas** → if `security` listed: VP-SEC is mandatory Tier 1 for every free-text input regardless of spec risk level; if `ui` not listed: all VP-UI scenarios move to Tier 2 minimum.
|
|
114
|
+
- **Testing strategy → Mandatory coverage** → each line is a hard override applied to this screen regardless of spec risk; document in `Context constraints` of the Coverage Map.
|
|
115
|
+
- **Testing strategy → Deprioritize/skip** → record in `Context constraints`; suppress those VP categories from Tier 1 generation.
|
|
116
|
+
- **Global business rules** → add each to the `Business rules` section tagged `[G]` (e.g. `[G1 – soft-delete only]`); treat as `HIGH` risk unless stated otherwise.
|
|
117
|
+
- **Error patterns** → use as fallback only when `spec.md` does not give exact error text; never override spec-specified messages.
|
|
118
|
+
- If `qa/context.md` is absent: proceed without it — no impact on the generation flow.
|
|
108
119
|
|
|
109
120
|
**Single screen focus**: one URL = one screen. Modals on same page = part of this screen.
|
|
110
121
|
This means: do not test other screens' UI layout or navigation. It does NOT mean skip documenting business outcomes that your screen's actions cause on other surfaces. Those cross-surface outcomes must appear in the Coverage Map and be covered by at least `@manual` scenarios.
|
|
@@ -129,6 +140,11 @@ Read `spec.md` fully, then extract into a Coverage Map **before writing any scen
|
|
|
129
140
|
**Risk tags:** HIGH = complex business rules, cascading fields, multi-step state changes, auth/integration. LOW = display-only, static labels, read-only fields.
|
|
130
141
|
|
|
131
142
|
```
|
|
143
|
+
Context constraints: [populated from qa/context.md before writing any scenario]
|
|
144
|
+
roles: [list roles, e.g. admin / manager / staff]
|
|
145
|
+
strategy: [active overrides, e.g. "VP-SEC mandatory T1", "VP-UI → T2 only"]
|
|
146
|
+
global rules: [G1 – ...] → also appear in Business rules below tagged [G]
|
|
147
|
+
→ leave empty if qa/context.md is absent or has no entries applicable to this screen
|
|
132
148
|
User journeys: [J1 – ...], [J2 – ...]
|
|
133
149
|
Validation rules: [V1 – field → "exact error text"], [V2 – ...]
|
|
134
150
|
Business rules: [B1 HIGH – ...], [B2 LOW – ...]
|
|
@@ -221,7 +237,7 @@ Security: [S1 – admin only]
|
|
|
221
237
|
| **auth** | valid-login · invalid-credential · access-control |
|
|
222
238
|
|
|
223
239
|
**Required assertion shapes (use these, not bare visibility):**
|
|
224
|
-
- Card info: assert at **card level** (image+name+price together), e.g. `User see all [Product Card] contain {{...}}` — not `see [Section]
|
|
240
|
+
- Card info: assert at **card level** (image+name+price together), e.g. `User see all [Product Card] contain {{...}}` — not `see [Section]` (section-level passes even if one card lacks price).
|
|
225
241
|
- Cross-screen consistency (detail/cart): **capture then compare** —
|
|
226
242
|
```gherkin
|
|
227
243
|
When User remember [Product Name] text as {{selected_product_name}}
|
|
@@ -239,13 +255,34 @@ Security: [S1 – admin only]
|
|
|
239
255
|
- **If the spec lacks the concrete value** a deep assertion needs (exact message, price, count): still write the deep shape with a `{{var}}` placeholder and leave a `# SPEC-GAP: <field> value not in spec` comment — do **not** downgrade to `see [X] section`. A visible gap is better than a silent shallow pass.
|
|
240
256
|
- **Blind-Spot Memory:** before finishing, run `sungen blindspot list --prompt` (Bash) and make sure the suite satisfies each recorded pattern (e.g. "for any Add/Create action: check success + resulting data state + duplicate/double-submit"). These are gaps QA hit before — don't repeat them.
|
|
241
257
|
|
|
242
|
-
**First-pass anti-patterns (exactly what the gate/reviewer reject — avoid them):**
|
|
243
|
-
- Title↔steps mismatch
|
|
244
|
-
- Tautology `Then`: `click [Next Slide]` → `see [Carousel] section` (proves nothing).
|
|
245
|
-
- Business-critical scenario ending at `see [Added] modal` / `see [Cart] page` with no data assertion.
|
|
258
|
+
**First-pass anti-patterns (these are exactly what the gate/reviewer reject — avoid them):**
|
|
259
|
+
- Title↔steps mismatch: e.g. a "no-result state" scenario that clicks a query which **returns** products. Steps must create the condition the title claims.
|
|
260
|
+
- Tautology `Then`: `click [Next Slide]` → `see [Carousel] section` (always visible, proves nothing). Assert the change (new slide title differs).
|
|
261
|
+
- Business-critical scenario ending at `see [Added] modal` / `see [Cart] page` / `see [Category Products] page` with no data assertion.
|
|
246
262
|
- Brand filter covered only as navigation (must assert products belong to the brand).
|
|
247
263
|
|
|
248
|
-
**Balance:** cover all the above (deep) BEFORE expanding subscription / UI-presence / extra validation edge cases.
|
|
264
|
+
**Balance:** cover all the above (deep) BEFORE expanding subscription / UI-presence / extra validation edge cases. Do not over-invest in subscription while cart/detail/filter correctness are shallow.
|
|
265
|
+
|
|
266
|
+
#### Harness gates — satisfy on the FIRST pass (don't make the repair loop fix them)
|
|
267
|
+
|
|
268
|
+
`sungen audit` enforces these. Generate compliant output up front:
|
|
269
|
+
|
|
270
|
+
1. **Taxonomy-match** (`VP-TAXONOMY-MISMATCH`, gate-FAIL) — when `test-viewpoint.md` declares its own viewpoint IDs (e.g. `VP0`, `VP1`, … `VP12`, `MS-HP-001`, `MS-EH-001`), **reuse those IDs verbatim as the scenario codes**. Do NOT invent a generic `VP-UI / VP-LOGIC / VP-VAL` scheme — that breaks the coverage matrix. Only fall back to `VP-<CATEGORY>-<NNN>` when the viewpoint file declares no IDs.
|
|
271
|
+
2. **Spec-coverage triggers** (`TRIGGER-UNCOVERED`, gate-FAIL) — the Validation-Rules table lists a **trigger** per constraint (e.g. `blur, submit`). Generate one scenario **per (constraint × trigger)** — a `format` rule validating *on blur AND on submit* needs BOTH a blur scenario (`press Tab`) and a submit scenario (`click [Submit]` / `press Enter`). Never collapse the trigger × input matrix to one representative case.
|
|
272
|
+
3. **Claim-Proof** (`CLAIM-UNPROVEN`) — a title claiming `all`/`only`/`every`/`single`/`correct`/`same`/`changes`/`hidden`/`cleared`/`restored`/`independent`/`sanitized`/`announces` MUST have the matching assertion (`see all …`, count, `remember`+compare, `is hidden`, return-and-assert-empty, etc.). If the title promises it, the steps must prove it.
|
|
273
|
+
- **Negative / absence claims** (`does not` / `no` / `never` / `prevents` / `không` / `chưa` — any language; `no-side-effect/no-duplicate`, `negative-claim/absence`): the `Then` must **differ** between the claim holding and not holding. A terminal `see [X] page` that looks identical whether or not the bad thing happened proves nothing. For a side-effect that should NOT repeat (re-submit on back, re-charge, duplicate order, resend OTP), assert the **count is unchanged** (`User see [Records] table with {{one}}` / `row with {{count}}`); if it's not UI-observable, mark `@manual` with a request-count oracle (shape below). This is general — it covers any side-effect, not a fixed verb list.
|
|
274
|
+
4. **Downstream-scope** (`DOWNSTREAM-SCOPE-MISSING`) — when the spec's Navigation Flow / success target is **another screen** (e.g. a confirmation/sent page), don't stop at a terminal `see [X] page`. Either cover that screen's content/guards (if its viewpoint items are in scope — they often have their own `MS-*` IDs), or scaffold it (`sungen add --screen <name>`) and note the handoff. Do not silently drop the downstream surface.
|
|
275
|
+
5. **Manual-oracle** (`MANUAL-STEPS-INSUFFICIENT`) — every `@manual` scenario needs **setup · action · observable expected · oracle/tool**, not a one-line note. Use this comment shape:
|
|
276
|
+
```gherkin
|
|
277
|
+
@high @manual
|
|
278
|
+
Scenario: VP-… <claim>
|
|
279
|
+
# MANUAL: <why it can't be automated — needs network capture / inbox / screen-reader / multi-tab>
|
|
280
|
+
# Tester verifies:
|
|
281
|
+
# 1. <setup> e.g. seed a registered email; throttle the network
|
|
282
|
+
# 2. <action> e.g. click [Submit] with the request in flight
|
|
283
|
+
# 3. <observable> e.g. only ONE POST is dispatched
|
|
284
|
+
# 4. Oracle: <tool> e.g. DevTools Network panel / mail-catcher / NVDA
|
|
285
|
+
```
|
|
249
286
|
|
|
250
287
|
#### Tier 1 guard — minimum before writing scenarios
|
|
251
288
|
|
|
@@ -363,7 +400,7 @@ Add cleanup tags per the `sungen-gherkin-syntax` Cleanup table. Key rules:
|
|
|
363
400
|
**Files:** `qa/screens/<screen>/features/<screen>.feature` + `qa/screens/<screen>/test-data/<screen>.yaml`
|
|
364
401
|
|
|
365
402
|
Use step patterns and element types from `sungen-gherkin-syntax`.
|
|
366
|
-
**Naming**: `VP-<CATEGORY>-<NNN>`. Scenario name must use the **same element type** as the steps.
|
|
403
|
+
**Naming**: reuse the **project's `test-viewpoint.md` IDs** when it declares them (e.g. `VP0`, `MS-HP-001`); otherwise `VP-<CATEGORY>-<NNN>`. Scenario name must use the **same element type** as the steps.
|
|
367
404
|
|
|
368
405
|
**Test data** — grouped by section, loaded at runtime:
|
|
369
406
|
|