@sun-asterisk/sungen 3.2.5 → 3.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/commands/delivery.d.ts.map +1 -1
- package/dist/cli/commands/delivery.js +7 -46
- package/dist/cli/commands/delivery.js.map +1 -1
- package/dist/dashboard/snapshot-builder.d.ts.map +1 -1
- package/dist/dashboard/snapshot-builder.js +28 -6
- package/dist/dashboard/snapshot-builder.js.map +1 -1
- package/dist/dashboard/templates/index.html +20 -20
- package/dist/dashboard/types.d.ts +18 -0
- package/dist/dashboard/types.d.ts.map +1 -1
- package/dist/exporters/api-catalog-loader.d.ts +18 -0
- package/dist/exporters/api-catalog-loader.d.ts.map +1 -0
- package/dist/exporters/api-catalog-loader.js +77 -0
- package/dist/exporters/api-catalog-loader.js.map +1 -0
- package/dist/exporters/csv-exporter.d.ts.map +1 -1
- package/dist/exporters/csv-exporter.js +20 -26
- package/dist/exporters/csv-exporter.js.map +1 -1
- package/dist/exporters/feature-parser.d.ts +6 -0
- package/dist/exporters/feature-parser.d.ts.map +1 -1
- package/dist/exporters/feature-parser.js +9 -3
- package/dist/exporters/feature-parser.js.map +1 -1
- package/dist/exporters/json-exporter.d.ts +10 -1
- package/dist/exporters/json-exporter.d.ts.map +1 -1
- package/dist/exporters/json-exporter.js +92 -50
- package/dist/exporters/json-exporter.js.map +1 -1
- package/dist/exporters/playwright-report-parser.d.ts +14 -0
- package/dist/exporters/playwright-report-parser.d.ts.map +1 -1
- package/dist/exporters/playwright-report-parser.js +30 -0
- package/dist/exporters/playwright-report-parser.js.map +1 -1
- package/dist/exporters/result-variants.d.ts +29 -0
- package/dist/exporters/result-variants.d.ts.map +1 -0
- package/dist/exporters/result-variants.js +33 -0
- package/dist/exporters/result-variants.js.map +1 -0
- package/dist/exporters/xlsx-exporter.d.ts +18 -17
- package/dist/exporters/xlsx-exporter.d.ts.map +1 -1
- package/dist/exporters/xlsx-exporter.js +25 -563
- package/dist/exporters/xlsx-exporter.js.map +1 -1
- package/dist/exporters/xlsx-report-builder.d.ts +196 -0
- package/dist/exporters/xlsx-report-builder.d.ts.map +1 -0
- package/dist/exporters/xlsx-report-builder.js +653 -0
- package/dist/exporters/xlsx-report-builder.js.map +1 -0
- package/dist/generators/test-generator/adapters/appium/templates/scenario.hbs +1 -1
- package/dist/generators/test-generator/adapters/playwright/templates/scenario.hbs +6 -6
- package/dist/generators/test-generator/adapters/playwright/templates/steps/actions/expand-action.hbs +30 -8
- package/dist/generators/test-generator/adapters/playwright/templates/steps/assertions/selected-assertion.hbs +16 -1
- package/dist/generators/test-generator/patterns/expect-patterns.d.ts.map +1 -1
- package/dist/generators/test-generator/patterns/expect-patterns.js +23 -0
- package/dist/generators/test-generator/patterns/expect-patterns.js.map +1 -1
- package/dist/generators/test-generator/utils/selector-resolver.d.ts +7 -0
- package/dist/generators/test-generator/utils/selector-resolver.d.ts.map +1 -1
- package/dist/generators/test-generator/utils/selector-resolver.js +13 -2
- package/dist/generators/test-generator/utils/selector-resolver.js.map +1 -1
- package/dist/orchestrator/project-initializer.d.ts +9 -0
- package/dist/orchestrator/project-initializer.d.ts.map +1 -1
- package/dist/orchestrator/project-initializer.js +66 -9
- package/dist/orchestrator/project-initializer.js.map +1 -1
- package/dist/orchestrator/templates/ai-src/commands/add-api.md +76 -0
- package/dist/orchestrator/templates/ai-src/commands/create-test.md +2 -2
- package/dist/orchestrator/templates/ai-src/config/codex.md +3 -2
- package/dist/orchestrator/templates/ai-src/skills/sungen-api-coverage-model/SKILL.md +73 -0
- package/dist/orchestrator/templates/ai-src/skills/sungen-api-design/SKILL.md +42 -8
- package/dist/orchestrator/templates/ai-src/skills/sungen-gherkin-syntax/SKILL.md +21 -0
- package/dist/orchestrator/templates/ai-src/skills/sungen-selector-fix/SKILL.md +2 -0
- package/dist/orchestrator/templates/ai-src/skills/sungen-selector-keys/SKILL.md +33 -0
- package/dist/orchestrator/templates/ai-src/skills/sungen-tc-review/SKILL.md +6 -0
- package/dist/orchestrator/templates/specs-api.d.ts +26 -0
- package/dist/orchestrator/templates/specs-api.d.ts.map +1 -1
- package/dist/orchestrator/templates/specs-api.js +131 -3
- package/dist/orchestrator/templates/specs-api.js.map +1 -1
- package/dist/orchestrator/templates/specs-api.ts +118 -5
- package/dist/orchestrator/templates/specs-test-data.ts +20 -1
- package/package.json +3 -3
- package/src/cli/commands/delivery.ts +8 -42
- package/src/dashboard/snapshot-builder.ts +31 -7
- package/src/dashboard/templates/index.html +20 -20
- package/src/dashboard/types.ts +19 -0
- package/src/exporters/api-catalog-loader.ts +44 -0
- package/src/exporters/csv-exporter.ts +19 -24
- package/src/exporters/feature-parser.ts +8 -2
- package/src/exporters/json-exporter.ts +102 -51
- package/src/exporters/playwright-report-parser.ts +27 -0
- package/src/exporters/result-variants.ts +47 -0
- package/src/exporters/xlsx-exporter.ts +34 -623
- package/src/exporters/xlsx-report-builder.ts +806 -0
- package/src/generators/test-generator/adapters/appium/templates/scenario.hbs +1 -1
- package/src/generators/test-generator/adapters/playwright/templates/scenario.hbs +6 -6
- package/src/generators/test-generator/adapters/playwright/templates/steps/actions/expand-action.hbs +30 -8
- package/src/generators/test-generator/adapters/playwright/templates/steps/assertions/selected-assertion.hbs +16 -1
- package/src/generators/test-generator/patterns/expect-patterns.ts +23 -0
- package/src/generators/test-generator/utils/selector-resolver.ts +30 -2
- package/src/orchestrator/project-initializer.ts +67 -9
- package/src/orchestrator/templates/ai-src/commands/add-api.md +76 -0
- package/src/orchestrator/templates/ai-src/commands/create-test.md +2 -2
- package/src/orchestrator/templates/ai-src/config/codex.md +3 -2
- package/src/orchestrator/templates/ai-src/skills/sungen-api-coverage-model/SKILL.md +73 -0
- package/src/orchestrator/templates/ai-src/skills/sungen-api-design/SKILL.md +42 -8
- package/src/orchestrator/templates/ai-src/skills/sungen-gherkin-syntax/SKILL.md +21 -0
- package/src/orchestrator/templates/ai-src/skills/sungen-selector-fix/SKILL.md +2 -0
- package/src/orchestrator/templates/ai-src/skills/sungen-selector-keys/SKILL.md +33 -0
- package/src/orchestrator/templates/ai-src/skills/sungen-tc-review/SKILL.md +6 -0
- package/src/orchestrator/templates/specs-api.ts +118 -5
- package/src/orchestrator/templates/specs-test-data.ts +20 -1
- package/dist/cli/types.d.ts +0 -9
- package/dist/cli/types.d.ts.map +0 -1
- package/dist/cli/types.js +0 -7
- package/dist/cli/types.js.map +0 -1
- package/dist/generators/gherkin-parser/selector-extractor.d.ts +0 -37
- package/dist/generators/gherkin-parser/selector-extractor.d.ts.map +0 -1
- package/dist/generators/gherkin-parser/selector-extractor.js +0 -108
- package/dist/generators/gherkin-parser/selector-extractor.js.map +0 -1
- package/dist/generators/test-generator/types.d.ts +0 -25
- package/dist/generators/test-generator/types.d.ts.map +0 -1
- package/dist/generators/test-generator/types.js +0 -7
- package/dist/generators/test-generator/types.js.map +0 -1
- package/dist/generators/types.d.ts +0 -119
- package/dist/generators/types.d.ts.map +0 -1
- package/dist/generators/types.js +0 -48
- package/dist/generators/types.js.map +0 -1
- package/dist/orchestrator/figma/node-path-collapser.d.ts +0 -16
- package/dist/orchestrator/figma/node-path-collapser.d.ts.map +0 -1
- package/dist/orchestrator/figma/node-path-collapser.js +0 -37
- package/dist/orchestrator/figma/node-path-collapser.js.map +0 -1
- package/src/cli/types.ts +0 -9
- package/src/generators/gherkin-parser/selector-extractor.ts +0 -142
- package/src/generators/test-generator/types.ts +0 -26
- package/src/generators/types.ts +0 -227
- package/src/orchestrator/figma/node-path-collapser.ts +0 -38
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
{{#if casesDataset}}
|
|
2
2
|
for (const __row of testData.cases('{{casesDataset}}')) {
|
|
3
3
|
{{#if tags}}
|
|
4
|
-
test(
|
|
4
|
+
test('{{escapeQuotes scenarioName}} — ' + __row.__label, { tag: [{{{tags}}}] }, async ({{#if isParallel}}{ page }{{/if}}) => {
|
|
5
5
|
{{else}}
|
|
6
|
-
test(
|
|
6
|
+
test('{{escapeQuotes scenarioName}} — ' + __row.__label, async ({{#if isParallel}}{ page }{{/if}}) => {
|
|
7
7
|
{{/if}}
|
|
8
8
|
const rowData = testData.withRow(__row);
|
|
9
9
|
{{#each steps}}
|
|
@@ -17,9 +17,9 @@
|
|
|
17
17
|
{{else}}
|
|
18
18
|
{{#if isParallel}}
|
|
19
19
|
{{#if tags}}
|
|
20
|
-
test('{{scenarioName}}', { tag: [{{{tags}}}] }, async ({ page }) => {
|
|
20
|
+
test('{{escapeQuotes scenarioName}}', { tag: [{{{tags}}}] }, async ({ page }) => {
|
|
21
21
|
{{else}}
|
|
22
|
-
test('{{scenarioName}}', async ({ page }) => {
|
|
22
|
+
test('{{escapeQuotes scenarioName}}', async ({ page }) => {
|
|
23
23
|
{{/if}}
|
|
24
24
|
{{#each steps}}
|
|
25
25
|
{{#if comment}}
|
|
@@ -30,9 +30,9 @@
|
|
|
30
30
|
});
|
|
31
31
|
{{else}}
|
|
32
32
|
{{#if tags}}
|
|
33
|
-
test('{{scenarioName}}', { tag: [{{{tags}}}] }, async () => {
|
|
33
|
+
test('{{escapeQuotes scenarioName}}', { tag: [{{{tags}}}] }, async () => {
|
|
34
34
|
{{else}}
|
|
35
|
-
test('{{scenarioName}}', async () => {
|
|
35
|
+
test('{{escapeQuotes scenarioName}}', async () => {
|
|
36
36
|
{{/if}}
|
|
37
37
|
{{#each steps}}
|
|
38
38
|
{{#if comment}}
|
package/src/generators/test-generator/adapters/playwright/templates/steps/actions/expand-action.hbs
CHANGED
|
@@ -1,11 +1,33 @@
|
|
|
1
|
+
{ const el = {{> locator}};
|
|
2
|
+
{{#if expanded.state}}
|
|
3
|
+
{{!-- Toggle-only: component exposes no observable expanded state → click without asserting. --}}
|
|
4
|
+
await el.click();
|
|
5
|
+
{{else if expanded.class}}
|
|
6
|
+
{{!-- State signalled by a class on a descendant (e.g. an accordion chevron icon). --}}
|
|
1
7
|
{{#if (eq direction 'expand')}}
|
|
2
|
-
if (await {{
|
|
3
|
-
|
|
4
|
-
}
|
|
5
|
-
await
|
|
8
|
+
if (!(await el.locator('.{{expanded.class}}').count())) { await el.click(); }
|
|
9
|
+
await expect(el.locator('.{{expanded.class}}')).toBeVisible();
|
|
10
|
+
{{else}}
|
|
11
|
+
if (await el.locator('.{{expanded.class}}').count()) { await el.click(); }
|
|
12
|
+
await expect(el.locator('.{{expanded.class}}')).toHaveCount(0);
|
|
13
|
+
{{/if}}
|
|
14
|
+
{{else if expanded.attribute}}
|
|
15
|
+
{{!-- Explicit boolean attribute ('true'/'false'), custom or aria-expanded. --}}
|
|
16
|
+
{{#if (eq direction 'expand')}}
|
|
17
|
+
if (await el.getAttribute('{{expanded.attribute}}') !== 'true') { await el.click(); }
|
|
18
|
+
await expect(el).toHaveAttribute('{{expanded.attribute}}', 'true');
|
|
19
|
+
{{else}}
|
|
20
|
+
if (await el.getAttribute('{{expanded.attribute}}') !== 'false') { await el.click(); }
|
|
21
|
+
await expect(el).toHaveAttribute('{{expanded.attribute}}', 'false');
|
|
22
|
+
{{/if}}
|
|
23
|
+
{{else}}
|
|
24
|
+
{{!-- Default (zero-config): drive by aria-expanded, but assert it ONLY when the element exposes it. --}}
|
|
25
|
+
{{#if (eq direction 'expand')}}
|
|
26
|
+
if (await el.getAttribute('aria-expanded') !== 'true') { await el.click(); }
|
|
27
|
+
if (await el.getAttribute('aria-expanded') !== null) { await expect(el).toHaveAttribute('aria-expanded', 'true'); }
|
|
6
28
|
{{else}}
|
|
7
|
-
if (await
|
|
8
|
-
|
|
29
|
+
if (await el.getAttribute('aria-expanded') !== 'false') { await el.click(); }
|
|
30
|
+
if (await el.getAttribute('aria-expanded') !== null) { await expect(el).toHaveAttribute('aria-expanded', 'false'); }
|
|
31
|
+
{{/if}}
|
|
32
|
+
{{/if}}
|
|
9
33
|
}
|
|
10
|
-
await expect({{> locator}}).toHaveAttribute('aria-expanded', 'false');
|
|
11
|
-
{{/if}}
|
|
@@ -1 +1,16 @@
|
|
|
1
|
-
|
|
1
|
+
{{#if (or (eq role 'checkbox') (eq role 'radio') (eq role 'switch'))}}
|
|
2
|
+
{{!-- Form control: "selected" is the checked state, never aria-selected. A native checkbox/radio
|
|
3
|
+
exposes no aria-selected attribute — its state lives in the `checked` property. --}}
|
|
4
|
+
await expect({{> locator}}).toBeChecked();
|
|
5
|
+
{{else}}
|
|
6
|
+
{{!-- Other/unknown role (tab, option, or a text/testid/css selector): probe at runtime. An ARIA
|
|
7
|
+
single-select widget exposes aria-selected → assert it; anything without it falls back to the
|
|
8
|
+
checked state. Never hardcode aria-selected — it breaks on native form controls. --}}
|
|
9
|
+
{ const el = {{> locator}};
|
|
10
|
+
if ((await el.getAttribute('aria-selected')) !== null) {
|
|
11
|
+
await expect(el).toHaveAttribute('aria-selected', 'true');
|
|
12
|
+
} else {
|
|
13
|
+
await expect(el).toBeChecked();
|
|
14
|
+
}
|
|
15
|
+
}
|
|
16
|
+
{{/if}}
|
|
@@ -17,6 +17,14 @@ import { MappedStep } from '../step-mapper';
|
|
|
17
17
|
const VALUE = String.raw`\{\{[^}]+\}\}|"[^"]*"|'[^']*'|-?\d+(?:\.\d+)?`;
|
|
18
18
|
const reExpect = new RegExp(`^\\s*(?:User\\s+)?expect\\s+(${VALUE})\\s+is\\s+(not\\s+|at\\s+least\\s+|at\\s+most\\s+)?(${VALUE})\\s*$`, 'i');
|
|
19
19
|
|
|
20
|
+
/** `expect {{name.body}} matches schema [Ref]` — validate a bound API response body against a named
|
|
21
|
+
* JSON Schema (from qa/api/schemas.yaml, populated by `sungen api import`). Requires the API Driver
|
|
22
|
+
* (the scenario carries `@api:<name>`, so the `api` runtime helper is imported).
|
|
23
|
+
* Two regexes: the MATCHER sees ref contents masked (`{{~}}`, `[~]` — see maskRefsForMatching); the
|
|
24
|
+
* GENERATOR gets the ORIGINAL step text, so it extracts the real `[Ref]`. */
|
|
25
|
+
const reMatchesSchemaMasked = /^\s*(?:User\s+)?expect\s+\{\{~\}\}\s+matches\s+schema\s+\[~\]\s*$/i;
|
|
26
|
+
const reMatchesSchema = new RegExp(`^\\s*(?:User\\s+)?expect\\s+(\\{\\{[^}]+\\}\\})\\s+matches\\s+schema\\s+\\[([A-Za-z0-9_.-]+)\\]\\s*$`, 'i');
|
|
27
|
+
|
|
20
28
|
/** Render a value token (`{{var}}` | "literal" | 'literal' | number) as a JS expression. */
|
|
21
29
|
function valueExpr(token: string): string {
|
|
22
30
|
const t = token.trim();
|
|
@@ -29,6 +37,21 @@ function valueExpr(token: string): string {
|
|
|
29
37
|
}
|
|
30
38
|
|
|
31
39
|
export const expectPatterns: StepPattern[] = [
|
|
40
|
+
{
|
|
41
|
+
name: 'expect-matches-schema',
|
|
42
|
+
priority: 63, // above expect-data; its regex needs `matches schema`, so no overlap
|
|
43
|
+
matcher: (step: ParsedStep) => reMatchesSchemaMasked.test(step.text),
|
|
44
|
+
generator: (step: ParsedStep, _ctx: PatternContext): MappedStep => {
|
|
45
|
+
// The masked matcher is looser than this extraction regex, so a malformed step (empty var/ref,
|
|
46
|
+
// odd chars) can claim the pattern yet not re-match here — emit a clear TODO instead of crashing.
|
|
47
|
+
const m = step.text.match(reMatchesSchema);
|
|
48
|
+
if (!m) return { code: `// TODO: malformed 'matches schema' step (expected: expect {{name.body}} matches schema [Ref]): ${step.text}`, comment: 'Unrecognized matches-schema step' };
|
|
49
|
+
const value = valueExpr(m[1]);
|
|
50
|
+
const ref = m[2];
|
|
51
|
+
const code = `{ const __schema = api.assertSchema(${value}, ${JSON.stringify(ref)}); expect(__schema.ok, \`matches schema [${ref}]: \${__schema.errors}\`).toBe(true); }`;
|
|
52
|
+
return { code, comment: `Expect ${m[1]} matches schema [${ref}]` };
|
|
53
|
+
},
|
|
54
|
+
},
|
|
32
55
|
{
|
|
33
56
|
name: 'expect-data',
|
|
34
57
|
priority: 62, // above generic see-assertions; sibling of the DB assertions
|
|
@@ -39,6 +39,17 @@ interface SelectorEntry {
|
|
|
39
39
|
// === v2: Assertion Helpers ===
|
|
40
40
|
attribute?: string; // Attribute to check (for "has" assertions)
|
|
41
41
|
pattern?: string; // Regex for attribute value
|
|
42
|
+
|
|
43
|
+
// === Expand/collapse state signal ===
|
|
44
|
+
// How the open/closed state of an expandable element is observed. Many accordions expose no
|
|
45
|
+
// `aria-expanded` (state shows only via an icon class swap), so hardcoding it makes the assertion
|
|
46
|
+
// time out. When omitted, codegen drives by aria-expanded but asserts it ONLY if present — a safe
|
|
47
|
+
// zero-config fallback. Provide one of these to verify a non-aria component explicitly.
|
|
48
|
+
expanded?: {
|
|
49
|
+
class?: string; // class present on a descendant when open (e.g. 'mdi-chevron-up')
|
|
50
|
+
attribute?: string; // boolean attribute ('true'/'false') on the toggle element
|
|
51
|
+
state?: 'none'; // no observable state → toggle-click only, skip the assertion
|
|
52
|
+
};
|
|
42
53
|
}
|
|
43
54
|
|
|
44
55
|
// New selector file structure: flat key-value pairs
|
|
@@ -64,6 +75,12 @@ export interface ResolvedSelector {
|
|
|
64
75
|
strategy: 'testid' | 'id' | 'placeholder' | 'label' | 'text' | 'css' | 'role' | 'locator'
|
|
65
76
|
| 'accessibility-id' | 'xpath' | 'android-uiautomator' | 'ios-predicate';
|
|
66
77
|
value: string;
|
|
78
|
+
// Original YAML `type:` of the matched entry (before it collapses into a `strategy`). Callers
|
|
79
|
+
// that assign meaning to the entry kind — e.g. page assertions, whose `value` is a URL ONLY when
|
|
80
|
+
// the entry is a `page` selector — must verify this instead of trusting a key match. A key can
|
|
81
|
+
// collide across types (one `読取結果一覧` entry, type `role`, reused for a page assertion), so a
|
|
82
|
+
// key-only lookup can hand back an entry whose `value` ('button') is meaningless as a URL.
|
|
83
|
+
selectorType?: SelectorType;
|
|
67
84
|
// Mobile per-platform variant (`android:`/`ios:` in selectors.yaml). When set, the appium codegen
|
|
68
85
|
// emits a runtime `driver.isIOS ? '<ios>' : '<android>'` so one @platform:mobile spec resolves the
|
|
69
86
|
// right native selector per OS. Each side carries its own strategy+value (issue #392).
|
|
@@ -87,6 +104,7 @@ export interface ResolvedSelector {
|
|
|
87
104
|
pagination?: Record<string, string>;
|
|
88
105
|
attribute?: string; // Attribute to check
|
|
89
106
|
pattern?: string; // Regex for attribute value
|
|
107
|
+
expanded?: { class?: string; attribute?: string; state?: 'none' }; // Expand/collapse state signal
|
|
90
108
|
}
|
|
91
109
|
|
|
92
110
|
/**
|
|
@@ -460,8 +478,14 @@ export class SelectorResolver {
|
|
|
460
478
|
const type = entry.type || 'placeholder';
|
|
461
479
|
// Check if value exists in entry (even if empty string), use it; otherwise use originalLabel
|
|
462
480
|
const rawValue = entry.value !== undefined && entry.value !== null ? entry.value : originalLabel;
|
|
463
|
-
//
|
|
464
|
-
|
|
481
|
+
// Name resolution has three cases:
|
|
482
|
+
// - key absent (undefined) → fall back to originalLabel (the [Reference] text)
|
|
483
|
+
// - explicit null / empty string → disable the name filter entirely (rawName = '')
|
|
484
|
+
// - a real string → use it verbatim
|
|
485
|
+
// Explicit `name: null` in YAML is a deliberate opt-out (e.g. a role selector that must NOT
|
|
486
|
+
// filter by accessible name); it must not be conflated with "unset". The role template omits
|
|
487
|
+
// `{ name }` when this resolves to a falsy value.
|
|
488
|
+
const rawName = entry.name === undefined ? originalLabel : (entry.name ?? '');
|
|
465
489
|
const value = SelectorResolver.resolveTemplateVars(rawValue);
|
|
466
490
|
const name = SelectorResolver.resolveTemplateVars(rawName);
|
|
467
491
|
const nth = entry.nth !== undefined && entry.nth !== null ? entry.nth : undefined;
|
|
@@ -480,10 +504,14 @@ export class SelectorResolver {
|
|
|
480
504
|
if (entry.pagination) v2Fields.pagination = entry.pagination;
|
|
481
505
|
if (entry.attribute) v2Fields.attribute = entry.attribute;
|
|
482
506
|
if (entry.pattern) v2Fields.pattern = entry.pattern;
|
|
507
|
+
if (entry.expanded) v2Fields.expanded = entry.expanded;
|
|
483
508
|
|
|
484
509
|
// Helper to attach v2 fields and inputMethod
|
|
485
510
|
const withExtras = (resolved: ResolvedSelector): ResolvedSelector => {
|
|
486
511
|
if (inputMethod) resolved.inputMethod = inputMethod;
|
|
512
|
+
// Preserve the entry's original YAML type so callers can verify the entry kind (a key match
|
|
513
|
+
// alone does not guarantee the entry means what the step expects — see ResolvedSelector.selectorType).
|
|
514
|
+
resolved.selectorType = type;
|
|
487
515
|
return { ...resolved, ...v2Fields };
|
|
488
516
|
};
|
|
489
517
|
|
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
*/
|
|
5
5
|
|
|
6
6
|
import * as fs from 'fs';
|
|
7
|
+
import * as os from 'os';
|
|
7
8
|
import * as path from 'path';
|
|
8
9
|
import { execSync } from 'child_process';
|
|
9
10
|
import { buildOutputs } from './ai-skills/build';
|
|
@@ -47,18 +48,53 @@ export function mergeCodexMcpToml(existing: string, mobileOnly: boolean): string
|
|
|
47
48
|
const serverKey = mobileOnly ? 'appium-mcp' : 'playwright';
|
|
48
49
|
// Already configured (by the user or a previous run) → preserve, do nothing.
|
|
49
50
|
if (new RegExp(String.raw`^\s*\[mcp_servers\.(?:"?${serverKey}"?)\]`, 'm').test(existing)) return null;
|
|
51
|
+
// startup_timeout_sec: Codex's default is 10s, which `npx <pkg>@latest` blows past on a cold
|
|
52
|
+
// cache (it downloads the package first) — the server then "fails to start" and Codex drops it.
|
|
53
|
+
// default_tools_approval_mode: Codex gates every MCP tool call behind an approval elicitation;
|
|
54
|
+
// non-interactive runs (`codex exec`, full-auto) auto-CANCEL it — "user cancelled MCP tool call"
|
|
55
|
+
// — so run-test could never drive the browser headlessly. Auto-approve is scoped to this server
|
|
56
|
+
// only and the whole file already requires project trust.
|
|
57
|
+
const serverOpts = 'startup_timeout_sec = 120\ndefault_tools_approval_mode = "approve"\n';
|
|
50
58
|
const block = mobileOnly
|
|
51
|
-
?
|
|
52
|
-
:
|
|
59
|
+
? `[mcp_servers.appium-mcp]\ncommand = "npx"\nargs = ["-y", "appium-mcp"]\n${serverOpts}`
|
|
60
|
+
: `[mcp_servers.playwright]\ncommand = "npx"\nargs = ["@playwright/mcp@latest"]\n${serverOpts}`;
|
|
53
61
|
if (existing.trim() === '') return CODEX_MCP_HEADER + block;
|
|
54
62
|
return existing + (existing.endsWith('\n') ? '\n' : '\n\n') + block;
|
|
55
63
|
}
|
|
56
64
|
|
|
65
|
+
/**
|
|
66
|
+
* Whether the user's global Codex config (`~/.codex/config.toml`) marks `projectDir` trusted.
|
|
67
|
+
* Codex loads a project `.codex/config.toml` ONLY for trusted projects and skips it SILENTLY
|
|
68
|
+
* otherwise — the #1 "Codex doesn't see the Playwright MCP server" failure — so init checks and
|
|
69
|
+
* warns instead of leaving the user with a config that never loads. Pure string scan (no toml dep).
|
|
70
|
+
*/
|
|
71
|
+
export function codexTrustsProject(configText: string, projectDir: string): boolean {
|
|
72
|
+
let currentProject: string | null = null;
|
|
73
|
+
const trusted: string[] = [];
|
|
74
|
+
for (const line of configText.split('\n')) {
|
|
75
|
+
const header = line.match(/^\s*\[(.+)\]\s*$/);
|
|
76
|
+
if (header) {
|
|
77
|
+
const m = header[1].trim().match(/^projects\.(?:"(.+)"|'(.+)'|(.+))$/);
|
|
78
|
+
currentProject = m ? (m[1] ?? m[2] ?? m[3]) : null;
|
|
79
|
+
continue;
|
|
80
|
+
}
|
|
81
|
+
if (currentProject && /^\s*trust_level\s*=\s*"trusted"\s*$/.test(line)) trusted.push(currentProject);
|
|
82
|
+
}
|
|
83
|
+
// Codex resolves the project by walking UP from cwd (git root / trusted ancestor), so a dir
|
|
84
|
+
// nested inside a trusted project is itself trusted — verified live: `codex mcp list` inside
|
|
85
|
+
// examples/<new>/ loads its .codex/config.toml with only the repo root in the trust list.
|
|
86
|
+
return trusted.some(
|
|
87
|
+
(dir) => projectDir === dir || projectDir.startsWith(dir.endsWith(path.sep) ? dir : dir + path.sep),
|
|
88
|
+
);
|
|
89
|
+
}
|
|
90
|
+
|
|
57
91
|
export class ProjectInitializer {
|
|
58
92
|
private baseCwd: string;
|
|
59
93
|
private cwd: string;
|
|
60
94
|
private createdItems: string[] = [];
|
|
61
95
|
private skippedItems: string[] = [];
|
|
96
|
+
/** set by createCodexMCPConfig when the project isn't Codex-trusted (config would never load) */
|
|
97
|
+
private codexTrustWarning = false;
|
|
62
98
|
private agents: AdapterKey[];
|
|
63
99
|
private force: boolean;
|
|
64
100
|
private mobile: MobileInitOptions;
|
|
@@ -360,12 +396,23 @@ export class ProjectInitializer {
|
|
|
360
396
|
const existed = fs.existsSync(filePath);
|
|
361
397
|
const existing = existed ? fs.readFileSync(filePath, 'utf-8') : '';
|
|
362
398
|
const merged = mergeCodexMcpToml(existing, this.mobileOnly);
|
|
363
|
-
if (merged
|
|
399
|
+
if (merged !== null) {
|
|
400
|
+
fs.writeFileSync(filePath, merged, 'utf-8');
|
|
401
|
+
this.createdItems.push(existed ? '.codex/config.toml (updated)' : '.codex/config.toml');
|
|
402
|
+
} else {
|
|
364
403
|
this.skippedItems.push('.codex/config.toml');
|
|
365
|
-
return;
|
|
366
404
|
}
|
|
367
|
-
|
|
368
|
-
|
|
405
|
+
|
|
406
|
+
// Trust gate: without a trusted entry in the GLOBAL Codex config, the project config above
|
|
407
|
+
// is silently ignored and run-test never gets the MCP browser. Warn in the summary.
|
|
408
|
+
const codexHome = process.env.CODEX_HOME || path.join(os.homedir(), '.codex');
|
|
409
|
+
let trusted = false;
|
|
410
|
+
try {
|
|
411
|
+
trusted = codexTrustsProject(fs.readFileSync(path.join(codexHome, 'config.toml'), 'utf-8'), this.cwd);
|
|
412
|
+
} catch {
|
|
413
|
+
// no global config yet → definitely untrusted
|
|
414
|
+
}
|
|
415
|
+
this.codexTrustWarning = !trusted;
|
|
369
416
|
}
|
|
370
417
|
|
|
371
418
|
/**
|
|
@@ -436,6 +483,16 @@ export class ProjectInitializer {
|
|
|
436
483
|
});
|
|
437
484
|
}
|
|
438
485
|
|
|
486
|
+
if (this.codexTrustWarning) {
|
|
487
|
+
console.log('\n⚠ Codex: this project is not trusted yet, so .codex/config.toml (the MCP browser');
|
|
488
|
+
console.log(' config for run-test) will be SILENTLY IGNORED by Codex. To fix, either:');
|
|
489
|
+
console.log(' • open the project with `codex` once and approve the trust prompt, or');
|
|
490
|
+
console.log(' • add to ~/.codex/config.toml:');
|
|
491
|
+
console.log(` [projects."${this.cwd}"]`);
|
|
492
|
+
console.log(' trust_level = "trusted"');
|
|
493
|
+
console.log(' Then verify with `codex mcp list` — it should show the sungen MCP server.');
|
|
494
|
+
}
|
|
495
|
+
|
|
439
496
|
console.log('\nNext steps:');
|
|
440
497
|
let stepIndex = 1;
|
|
441
498
|
if (projectName) {
|
|
@@ -597,10 +654,11 @@ export class ProjectInitializer {
|
|
|
597
654
|
// Ensure standard scripts exist in package.json
|
|
598
655
|
this.ensurePackageScripts(packageJsonPath);
|
|
599
656
|
|
|
600
|
-
// Mobile-only drops @playwright/test (no web runner needed).
|
|
657
|
+
// Mobile-only drops @playwright/test (no web runner needed). `ajv`/`ajv-formats` back the API
|
|
658
|
+
// Driver's `expect {{name.body}} matches schema [Ref]` (loaded lazily; harmless if unused).
|
|
601
659
|
const requiredDeps = this.mobileOnly
|
|
602
|
-
? ['@types/node', 'yaml']
|
|
603
|
-
: ['@playwright/test', '@types/node', 'yaml'];
|
|
660
|
+
? ['@types/node', 'yaml', 'ajv', 'ajv-formats']
|
|
661
|
+
: ['@playwright/test', '@types/node', 'yaml', 'ajv', 'ajv-formats'];
|
|
604
662
|
|
|
605
663
|
let missingDeps: string[] = requiredDeps;
|
|
606
664
|
try {
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: add-api
|
|
3
|
+
description: "Add a new Sungen API area or flow — scaffolds the datasource + qa/api/<area>/ directories and can import an OpenAPI/Swagger or CSV spec so the endpoint catalog (+ field metadata, auth, schemas) is populated for you"
|
|
4
|
+
argument-hint: "[area-name] [--base-url <url>] [--flow <name>] [--import <openapi|csv|sheet-url>] [--datasource <name>]"
|
|
5
|
+
order: 10
|
|
6
|
+
claude-tools: "Read, Grep, Bash, Glob, Edit, Write, AskUserQuestion"
|
|
7
|
+
copilot-tools: "[vscode, execute, read, agent, edit, search, todo]"
|
|
8
|
+
codex-trigger: "Run when the user asks to add or scaffold a NEW API area/resource or an API flow to test (REST/HTTP endpoints, not a UI screen). Step 1 of the API workflow. Do NOT use for UI screens, flows across screens, creating test cases, or running tests."
|
|
9
|
+
---
|
|
10
|
+
{{^cap parallel-subagents}}
|
|
11
|
+
**Input**: an API area name (a resource/tag, e.g. `orders`, `users`) and the base URL:
|
|
12
|
+
`/sungen-add-api orders --base-url https://api.example.com`.
|
|
13
|
+
Import a spec so the catalog is filled in: `/sungen-add-api orders --base-url https://api.example.com --import ./openapi.yaml`.
|
|
14
|
+
Add an API flow (ordered CRUD/auth journey) instead of a single area: `/sungen-add-api user-lifecycle --flow`.
|
|
15
|
+
{{/cap}}
|
|
16
|
+
You are scaffolding an **API-first** unit for test generation — an area (one resource) or a flow (a cross-endpoint journey). There is **no visual capture and no selectors**: the contract is the named-endpoint catalog (`api/apis.yaml`), referenced by `@api:<name>`. Follow the `sungen-api-design` skill after scaffolding.
|
|
17
|
+
|
|
18
|
+
## Parameters
|
|
19
|
+
|
|
20
|
+
{{#cap parallel-subagents}}
|
|
21
|
+
Parse from `$ARGUMENTS`:
|
|
22
|
+
- **name** — area or flow name (e.g. `orders`, `users`, `user-lifecycle`)
|
|
23
|
+
- **`--base-url <url>`** — the API base URL (stored in `.env.qa`, referenced as `${<DS>_URL}`). Needed the first time (project not yet API-initialized); skippable if `qa/datasources.yaml` already has a `kind: api` datasource.
|
|
24
|
+
- **`--flow <name>`** — scaffold an ordered `@api` chain (auth → CRUD) under `qa/api/flows/<name>/` instead of a single area.
|
|
25
|
+
- **`--import <source>`** — an OpenAPI/Swagger file/URL or a CSV / published Google-Sheet URL. When given, the catalog + `fields:` metadata + inferred auth headers + response `schemas.yaml` are generated from it.
|
|
26
|
+
- **`--datasource <name>`** — datasource name (default `app_api`).
|
|
27
|
+
{{/cap}}
|
|
28
|
+
{{^cap parallel-subagents}}
|
|
29
|
+
- **name** — ${input:name:API area or flow name (e.g. orders, users)}
|
|
30
|
+
- **`--base-url <url>`** — API base URL (first-time init; skip if already API-initialized)
|
|
31
|
+
- **`--flow <name>`** — scaffold an ordered @api chain instead of a single area
|
|
32
|
+
- **`--import <source>`** — OpenAPI/Swagger or CSV/Sheet URL → auto-populate the catalog
|
|
33
|
+
- **`--datasource <name>`** — datasource name (default app_api)
|
|
34
|
+
{{/cap}}
|
|
35
|
+
|
|
36
|
+
If **name** is missing, ask: "What is the API area (resource) name? (e.g., `orders`, `users`)".
|
|
37
|
+
If **`--base-url`** is missing AND `qa/datasources.yaml` has no `kind: api` datasource, ask for it (it's needed to run the tests later; you can proceed with scaffolding and add it after).
|
|
38
|
+
|
|
39
|
+
## Steps
|
|
40
|
+
|
|
41
|
+
### 1. Initialize the API project (first area only)
|
|
42
|
+
|
|
43
|
+
Check `qa/datasources.yaml` for a `kind: api` datasource. If none, run:
|
|
44
|
+
```bash
|
|
45
|
+
sungen api init --base-url <base-url> --datasource <datasource>
|
|
46
|
+
```
|
|
47
|
+
This creates the `kind: api` datasource, `qa/api/`, and the `.env.qa` URL key. Skip if already initialized (never clobbers).
|
|
48
|
+
|
|
49
|
+
### 2. Scaffold the area (or flow)
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
sungen api add --area <name> # a single resource → qa/api/<name>/
|
|
53
|
+
# or
|
|
54
|
+
sungen api add --flow <name> # an ordered CRUD/auth journey → qa/api/flows/<name>/
|
|
55
|
+
```
|
|
56
|
+
This creates `features/`, `api/apis.yaml`, `test-data/`, and `requirements/spec.md`.
|
|
57
|
+
|
|
58
|
+
### 3. Populate the catalog (recommended)
|
|
59
|
+
|
|
60
|
+
If `--import <source>` was given, run it — importing a spec is the single biggest quality lever (the catalog gets accurate paths, query/body params, `fields:` constraints, inferred auth headers, and response `schemas.yaml`, so `/sungen:create-test` can enumerate a complete case set instead of guessing):
|
|
61
|
+
```bash
|
|
62
|
+
sungen api import <source> --datasource <datasource>
|
|
63
|
+
```
|
|
64
|
+
No spec? Hand-author `qa/api/<name>/api/apis.yaml` (or fill `requirements/spec.md` and let `/sungen:create-test` draft it). Review the generated catalog + wire the datasource `base_url`/auth in `datasources.yaml` + `.env.qa`.
|
|
65
|
+
|
|
66
|
+
### 4. Converge (next actions)
|
|
67
|
+
|
|
68
|
+
Use `AskUserQuestion` to offer the next step — do NOT just print text:
|
|
69
|
+
- **`/sungen:create-test <name>`** (Recommended) — generate `.feature` + `test-data` (auto-detects the API unit; enumerates the coverage model from `fields:`).
|
|
70
|
+
- **`/sungen:add-api <another> --base-url …`** — scaffold another area.
|
|
71
|
+
- Review `qa/api/<name>/api/apis.yaml` + wire the datasource, then create tests.
|
|
72
|
+
|
|
73
|
+
## Rules
|
|
74
|
+
- **No visual capture, no selectors** — API units are catalog-driven; don't run Playwright/Figma capture.
|
|
75
|
+
- **Never clobber** — `init` merges datasources; `add` refuses an existing area; `import` needs `--merge` to combine.
|
|
76
|
+
- **Non-prod default** — a `production` datasource is refused at runtime unless `SUNGEN_ALLOW_PROD=1`.
|
|
@@ -62,10 +62,10 @@ Parse **name** from `$ARGUMENTS`. If missing, ask the user.
|
|
|
62
62
|
## API unit mode (driver-api)
|
|
63
63
|
|
|
64
64
|
{{#cap parallel-subagents}}
|
|
65
|
-
If the unit is **api-first** (`qa/api/<name>/` or `qa/api/flows/<name>/`), the design loop differs — **no visual capture, no selectors**; the contract is the named-endpoint catalog. **Follow the `sungen-api-design`
|
|
65
|
+
If the unit is **api-first** (`qa/api/<name>/` or `qa/api/flows/<name>/`), the design loop differs — **no visual capture, no selectors**; the contract is the named-endpoint catalog. **Follow the `sungen-api-design` + `sungen-api-coverage-model` skills end-to-end** instead of the screen/flow steps below: `sungen context --area <name>` (discover endpoints + `fields:`) → **enumerate the Tier-1 case list per endpoint from the coverage model** (contract + not-found/required-matrix + auth + idempotency, expanded mechanically from `fields:`) → generate `@api`/`@cases`/flow/`@concurrent`/`@query` scenarios with **strict assertions** (prove the effect, never status-only) → **`sungen audit --area <name>` gate + the `sungen-reviewer` sub-agent + repair loop to businessDepth ≥ 0.7** → record + trace. Then jump to the "Converge" next-step options (recommend `/sungen:run-test <name>`). The capture / viewpoint-group / selector steps do **not** apply.
|
|
66
66
|
{{/cap}}
|
|
67
67
|
{{^cap parallel-subagents}}
|
|
68
|
-
If the unit is **api-first** (`qa/api/<name>/` or `qa/api/flows/<name>/`), the design loop differs — **no visual capture, no selectors**; the contract is the named-endpoint catalog. **Follow the `sungen-api-design`
|
|
68
|
+
If the unit is **api-first** (`qa/api/<name>/` or `qa/api/flows/<name>/`), the design loop differs — **no visual capture, no selectors**; the contract is the named-endpoint catalog. **Follow the `sungen-api-design` + `sungen-api-coverage-model` skills end-to-end** instead of the screen/flow steps: `sungen context --area <name>` (discover endpoints + `fields:`) → **enumerate the Tier-1 case list per endpoint from the coverage model** (contract + not-found/required-matrix + auth + idempotency, expanded mechanically from `fields:`) → generate `@api`/`@cases`/flow/`@concurrent`/`@query` scenarios with **strict assertions** (prove the effect, never status-only) → **`sungen audit --area <name>` gate + reviewer + repair loop to businessDepth ≥ 0.7** → record + trace. Then recommend `/sungen-run-test <name>`. The capture / viewpoint-group / selector steps do **not** apply.
|
|
69
69
|
{{/cap}}
|
|
70
70
|
|
|
71
71
|
## Steps
|
|
@@ -2,7 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
This project uses **sungen** (Gherkin → Playwright test compiler). Its workflow and
|
|
4
4
|
reference skills are installed under `.agents/skills/sungen-*`. Codex auto-discovers them;
|
|
5
|
-
|
|
6
|
-
(e.g. "create a test for the login screen") and
|
|
5
|
+
invoke a workflow step with `$sungen-create-test <args>` (typing `$` opens the skill
|
|
6
|
+
autocomplete), or just describe the task (e.g. "create a test for the login screen") and
|
|
7
|
+
the matching skill triggers by description. `/skills` lists the installed skills.
|
|
7
8
|
|
|
8
9
|
Workflow order: add-screen → create-test → review → run-test → delivery.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: sungen-api-coverage-model
|
|
3
|
+
description: The per-endpoint API coverage model — the systematic case set to enumerate for each endpoint (read / mutating / flow), keyed to the catalog's fields: metadata, tiered so a first pass is complete (not "happy path + 1 error"). Load this in API unit mode alongside sungen-api-design; it turns endpoints × field constraints into a concrete case list, mechanically, instead of guessing.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# API coverage model (what to test per endpoint)
|
|
7
|
+
|
|
8
|
+
The gap this closes: a bare `@api` generation defaults to **one happy path + maybe one error** per endpoint — 3 of 8 needed cases. This skill is the **enumeration rule**: for each endpoint, walk the checklist below and emit **every applicable case**. Cases are derived **mechanically** from the catalog `fields:` block (`required` / `type` / `format` / `min`/`max` / `minLength`/`maxLength` / `pattern` / enum `values`) surfaced by `sungen context --area <name>` — do NOT guess the matrix from prose when `fields:` exists.
|
|
9
|
+
|
|
10
|
+
**Tiers** — always emit **Tier 1** (the complete-first-pass minimum). Emit **Tier 2** when the user asks for full coverage or the spec flags the concern as business-critical. Don't stop at Tier 1's *count* if an endpoint has more required/constrained fields — the matrix scales with the fields.
|
|
11
|
+
|
|
12
|
+
## Read endpoint (GET / HEAD)
|
|
13
|
+
|
|
14
|
+
| # | Case | Tier | Route | From |
|
|
15
|
+
|---|------|------|-------|------|
|
|
16
|
+
| R1 | **Contract** — 2xx + response body asserted (≥1 server-derived field value; when a `schemaRef` exists, add `expect {{name.body}} matches schema [Ref]` for the full shape) | 1 | `@api` | always |
|
|
17
|
+
| R2 | **Not found** — by-id read with a missing id → 404 (or the spec's code) | 1 | `@api` | path has an id param |
|
|
18
|
+
| R3 | **Unauthorized** — no / invalid token → 401 | 1 | `@api` + `@cases` | endpoint requires auth |
|
|
19
|
+
| R4 | **Empty state** — query that returns 0 rows → 2xx + empty array / count 0 | 2 | `@api` | list endpoint |
|
|
20
|
+
| R5 | **Forbidden** — valid token, wrong role → 403 | 2 | `@api` | role-scoped |
|
|
21
|
+
| R6 | **Per query param validation** — for each `fields:` query param: required→omit→400; type mismatch→400; enum→invalid value→400; `min/max`→BVA rows | 2 | `@cases` | each constrained query field |
|
|
22
|
+
| R7 | **Pagination / filter / sort** — offset/limit boundaries, sort direction, filter match | 2 | `@api`/`@cases` | paginated list |
|
|
23
|
+
|
|
24
|
+
## Mutating endpoint (POST / PUT / PATCH / DELETE)
|
|
25
|
+
|
|
26
|
+
| # | Case | Tier | Route | From |
|
|
27
|
+
|---|------|------|-------|------|
|
|
28
|
+
| M1 | **Happy path** — 2xx + response body asserted (id/echoed fields) **and prove the effect** (`@query` DB read-back, or a follow-up GET) | 1 | `@api` (+`@query`) | always |
|
|
29
|
+
| M2 | **Required-field matrix** — one row per `required` field omitted → 400 + error message | 1 | `@cases` | each `required` field |
|
|
30
|
+
| M3 | **Unauthorized** — no / invalid token → 401 | 1 | `@api`/`@cases` | endpoint requires auth |
|
|
31
|
+
| M4 | **Idempotency / duplicate** — `@concurrent:N` + `@query` oracle → exactly one effect (or 409 on the 2nd) | 1 | `@concurrent`+`@query` | create / non-idempotent |
|
|
32
|
+
| M5 | **Type + boundary matrix** — per constrained field: wrong type; `min-1/min/max/max+1`; `minLength/maxLength` edges | 2 | `@cases` | each constrained field |
|
|
33
|
+
| M6 | **Format / enum matrix** — per `format`/`pattern`/enum field: an invalid value → 400 | 2 | `@cases` | each format/enum field |
|
|
34
|
+
| M7 | **Forbidden** — valid token, wrong role → 403 | 2 | `@api`/`@cases` | role-scoped |
|
|
35
|
+
| M8 | **Semantic (422)** — fields individually valid but violate a business rule (e.g. end < start) | 2 | `@cases` | cross-field rule |
|
|
36
|
+
| M9 | **Payload size / content-type** — oversized body → 413/400; missing/wrong `Content-Type` → 400 | 2 | `@cases` | as spec dictates |
|
|
37
|
+
|
|
38
|
+
## Flow (multi-endpoint journey — qa/api/flows/<flow>)
|
|
39
|
+
|
|
40
|
+
| # | Case | Tier |
|
|
41
|
+
|---|------|------|
|
|
42
|
+
| F1 | **CRUD chain** — create → read-back → update → read-back → delete → 404 (each step asserts + self-cleans) | 1 |
|
|
43
|
+
| F2 | **Auth flow** — login → authed call 2xx → logout → same call 401 | 1 |
|
|
44
|
+
| F3 | **Cross-endpoint invariant** — create in A, verify in B (data consistency) | 2 |
|
|
45
|
+
|
|
46
|
+
## Deriving the matrix from `fields:`
|
|
47
|
+
|
|
48
|
+
For an endpoint with `fields:` metadata, expand mechanically:
|
|
49
|
+
|
|
50
|
+
- `required: true` → **one M2/R6 missing-field row** (omit it, expect 400 + message).
|
|
51
|
+
- `type` (integer/number/boolean) → a **wrong-type row** (send a string).
|
|
52
|
+
- `min`/`max` (or `minLength`/`maxLength`) → **BVA rows**: `min-1` (invalid), `min` (valid edge), `max` (valid edge), `max+1` (invalid).
|
|
53
|
+
- `values` (enum) → a **not-in-enum row**.
|
|
54
|
+
- `format`/`pattern` → an **ill-formed row** (bad email, wrong pattern).
|
|
55
|
+
|
|
56
|
+
Collapse these into compact `@cases` datasets — one dataset per endpoint, one row per (field, technique), each row carrying its `expect_status`. Don't write one scenario per row.
|
|
57
|
+
|
|
58
|
+
## Assertion depth (tie every case to a proof)
|
|
59
|
+
|
|
60
|
+
Never emit a status-only assertion for a success case. Three tiers — see also `sungen-tc-review` (API assertion rubric):
|
|
61
|
+
|
|
62
|
+
- **Shallow ❌** — `expect {{x.status}} is 201` alone. Proves the server didn't 500; not that anything happened.
|
|
63
|
+
- **Medium ⚠️** — status + a response body field (`{{x.body.id}}`). OK for a read contract.
|
|
64
|
+
- **Strict ✅ (required for mutating success + Tier-1 reads)** — status + body field **value that isn't just the input echoed back** + a **proof of effect**: a `@query` DB read-back, a follow-up GET, `matches schema [Ref]` (full-shape contract, when a `schemaRef` exists), or a `@concurrent` `ok_count` invariant.
|
|
65
|
+
|
|
66
|
+
> `expect {{name.body}} matches schema [Ref]` validates the whole response body against the named JSON Schema in `qa/api/schemas.yaml` (populated by `sungen api import`). Use it in addition to — not instead of — a couple of concrete field-value asserts on the critical fields.
|
|
67
|
+
|
|
68
|
+
Anti-pattern to avoid: re-asserting the value you just sent (`expect {{x.body.username}} is {{username}}` on the thing you created proves little) — assert a **server-derived** field (generated id, timestamp, computed status) or read it back independently.
|
|
69
|
+
|
|
70
|
+
## Output discipline
|
|
71
|
+
- Enumerate the **full Tier-1 list per endpoint first**, then write incrementally (≈10–15 scenarios per Write call) — never the whole suite in one response (output-token cap).
|
|
72
|
+
- Name scenarios `VP-API-NNN <what it proves>`; keep the endpoint + technique legible.
|
|
73
|
+
- Enumerate the matrix **mechanically from `fields:`** — that is the coverage guarantee this phase; don't rely on a gate to backfill it. (A future audit gate will additionally flag a `required`/constrained field with no matching negative case — `VIEWPOINT-API-REQUIRED-MATRIX` / `-BOUNDARY` / `-FORMAT` — but until it ships, completeness comes from the enumeration above, not from repair.)
|
|
@@ -10,20 +10,30 @@ Use this when the unit is **api-first** — `qa/api/<area>/` or `qa/api/flows/<f
|
|
|
10
10
|
## The loop (mirror of /sungen:design, API-native)
|
|
11
11
|
|
|
12
12
|
### 1. Discover (no capture)
|
|
13
|
-
Run `sungen context --area <name>` — it reads the catalog and prints the **endpoints**
|
|
13
|
+
Run `sungen context --area <name>` — it reads the catalog and prints the **endpoints**, their **`fields:` metadata** (required/type/format/range/enum), any **`schemaRef`**, and the **generation units** (one `matrix` unit per endpoint, an `async` unit per mutating endpoint, a `flow` unit for an api flow). Read `qa/api/<name>/requirements/spec.md` if present. No `apis.yaml` yet? → `sungen api import <openapi|csv>` or `sungen api add --area <name>` first.
|
|
14
14
|
|
|
15
|
-
### 2.
|
|
16
|
-
|
|
15
|
+
### 2. Enumerate the coverage model (don't guess the matrix)
|
|
16
|
+
**Load the `sungen-api-coverage-model` skill** and, for each endpoint, walk its checklist to produce the **concrete Tier-1 case list** (a read endpoint → contract + not-found + unauthorized; a mutating endpoint → happy+side-effect + required-field matrix + unauthorized + idempotency). Expand the validation matrix **mechanically from `fields:`** — one `@cases` row per (required/constrained field × technique), each with its `expect_status`. This is the difference between 3-of-8 and a complete first pass.
|
|
17
17
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
|
21
|
-
|
|
18
|
+
**Mapping Contract** — endpoint profile → cases → route:
|
|
19
|
+
|
|
20
|
+
| Profile | Viewpoint | Cases (Tier 1) | Technique | Route |
|
|
21
|
+
|---|---|---|---|---|
|
|
22
|
+
| GET/HEAD | contract | 1 | happy + body-field assert (+`matches schema [Ref]` if `schemaRef`) | `@api` |
|
|
23
|
+
| GET/HEAD | not-found | 1 (by-id) | missing id → 404 | `@api` |
|
|
24
|
+
| GET/HEAD | auth | 1 | no/invalid token → 401 | `@api`/`@cases` |
|
|
25
|
+
| GET/HEAD | param validation | N (Tier 2) | EP/BVA per query field | `@cases` |
|
|
26
|
+
| POST/PUT/PATCH/DELETE | contract + side-effect | 1 | happy + `@query` read-back | `@api`+`@query` |
|
|
27
|
+
| POST/PUT/PATCH/DELETE | required matrix | N = #required fields | one row per omitted field → 400 | `@cases` |
|
|
28
|
+
| POST/PUT/PATCH/DELETE | auth | 1 | no/invalid token → 401 | `@api`/`@cases` |
|
|
29
|
+
| POST/PUT/PATCH/DELETE | idempotency | 1 | `@concurrent:N` + `@query` | `@concurrent`+`@query` |
|
|
30
|
+
| POST/PUT/PATCH/DELETE | type/boundary/format | N (Tier 2) | per constrained field | `@cases` |
|
|
22
31
|
|
|
23
32
|
Bands: **~70%** success+failure matrix · **~20%** flows (auth/CRUD chains) · **~10%** async/idempotency.
|
|
24
33
|
|
|
25
34
|
### 3. Generate (incremental — never the whole suite in one Write)
|
|
26
|
-
|
|
35
|
+
Follow the coverage model's **assertion depth tiers** — a mutating success case must **prove the effect** (body value + `@query`/read-back), never status-only, and must not merely re-assert the input it just sent.
|
|
36
|
+
- **Contract**: `@api:<name>` + `expect {{name.status}} is …` **and a body assertion** on a server-derived field (`{{name.body.<path>}}`); when the entry has a `schemaRef`, add `expect {{name.body}} matches schema [<Ref>]` for the full-shape contract (validates against `qa/api/schemas.yaml`).
|
|
27
37
|
- **Error matrix**: `@api:<name>(p={{p}}) @cases:<dataset>` — one scenario, a dataset of `input → expected status`.
|
|
28
38
|
- **Flow**: ordered `@api` tags threading a prior response (`token={{login.body.token}}` → the catalog `Bearer :token` header; `id={{create.body.id}}` → a path param). Self-clean (delete what you create).
|
|
29
39
|
- **Idempotency**: `@api:<name> @concurrent:N` + `expect {{name.ok_count}} is 1`, cross-checked with `@query` (the DB is the oracle).
|
|
@@ -37,8 +47,13 @@ Run `sungen audit --area <name>`; read `gateStatus` + `findings`. Then the **sem
|
|
|
37
47
|
| `VIEWPOINT-API-ERROR` | a mutating endpoint has no failure scenario → add a `@cases` error matrix (or an explicit 4xx) |
|
|
38
48
|
| `VIEWPOINT-API-IDEMPOTENCY` | a mutating endpoint has no race check → add `@concurrent:N` + a `@query` DB cross-check |
|
|
39
49
|
| `VIEWPOINT-API-MANUAL-AUTOMATABLE` | a `@manual` scenario whose endpoint resolves is automatable → drop `@manual`, use `@api` (+ `@cases`); reserve `@manual` for genuine judgment cases |
|
|
50
|
+
| `VIEWPOINT-API-REQUIRED-MATRIX` | a `required` field (from `fields:`) has no missing-field case → add a `@cases` row omitting it, asserting 4xx |
|
|
51
|
+
| `VIEWPOINT-API-BOUNDARY` | a field with a `min/max/minLength/maxLength` constraint has no boundary case → add BVA `@cases` rows (`min-1/min/max/max+1`) |
|
|
52
|
+
| `VIEWPOINT-API-FORMAT` | a `format`/`pattern`/enum field has no invalid-value case → add a `@cases` row with an ill-formed value → 4xx |
|
|
40
53
|
| **`DEPTH-FAIL`** (businessDepth < 0.7) | a **mutating success** scenario asserts only `status` → make it **prove the effect**: assert a response **body** field, a **`@query`** side-effect, or a **`@concurrent` `ok_count`** invariant. (An error/`@cases` scenario proving the status is correct — it is *not* depth-required.) |
|
|
41
54
|
|
|
55
|
+
The `VIEWPOINT-API-REQUIRED-MATRIX`/`-BOUNDARY`/`-FORMAT` rows are the repair actions for those findings **once the field-driven gate sensors ship** (they will fire only when the endpoint has `fields:`, so a bare catalog raises none — no false demands). Until then, completeness comes from enumerating the matrix mechanically from `fields:` at generation time (see `sungen-api-coverage-model`), not from these gate findings.
|
|
56
|
+
|
|
42
57
|
Stop when the gate PASSes + businessDepth ≥ 0.7, or the budget is exhausted → report residual gaps honestly (mark genuinely-unautomatable cases `@manual` with an oracle). Never fake a pass.
|
|
43
58
|
|
|
44
59
|
### 5. Record + converge
|
|
@@ -54,6 +69,25 @@ Stop when the gate PASSes + businessDepth ≥ 0.7, or the budget is exhausted
|
|
|
54
69
|
|
|
55
70
|
A flow (`create → login → delete`) is a **Functional integration** test, **not** non-functional — don't file it under "Non-Functional". Reserve non-functional for perf/security/concurrency.
|
|
56
71
|
|
|
72
|
+
## File upload (real files)
|
|
73
|
+
`body:` carries only scalar fields — to send a **real file**, declare it in the catalog entry; the runtime reads the fixture from disk. Two forms, by how the server accepts the file:
|
|
74
|
+
- **`files:`** → `multipart/form-data`, the file is a form part: `files: { file: { path: ":avatar", mimeType: image/png } }` (+ optional text `body:` fields alongside). Shorthand `files: { file: ":avatar" }`.
|
|
75
|
+
- **`bodyFile:`** → raw binary body (the whole body IS the file's bytes, e.g. `application/octet-stream`): `bodyFile: { path: ":image", mimeType: application/octet-stream }`.
|
|
76
|
+
|
|
77
|
+
Fixture path resolves cwd-relative/absolute first, else `qa/fixtures/<path>` (drop sample files there, reference by name from `test-data`). An empty resolved file param omits the part → use for missing-file `@cases` error rows. `files:`/`bodyFile:` are mutually exclusive. **Automate the upload success case with `@api`** — don't defer it to `@manual`.
|
|
78
|
+
|
|
79
|
+
## Per-endpoint knobs & auth patterns
|
|
80
|
+
- **Timeout** — a slow endpoint can override the datasource default (15s) with `timeout_ms: 30000` on its catalog entry (else the datasource `timeout_ms` applies).
|
|
81
|
+
- **OAuth2 (pure API, no browser)** — no new grammar; it's `@api` chaining. Call the token endpoint, bind the token, thread it into later calls' `Bearer :token` header:
|
|
82
|
+
```gherkin
|
|
83
|
+
@api:get_token(client_id={{client_id}},client_secret={{client_secret}}) @api:list_orders(token={{get_token.body.access_token}})
|
|
84
|
+
Scenario: VP-API-FLOW client-credentials token authorizes the call
|
|
85
|
+
Then expect {{get_token.status}} is 200
|
|
86
|
+
And expect {{list_orders.status}} is 200
|
|
87
|
+
```
|
|
88
|
+
The token endpoint is a normal catalog entry (`POST /oauth/token`, form/JSON body); `list_orders` carries `headers: { authorization: "Bearer :token" }`. For a browser-authorized session instead, use `@hybrid` + `@auth:<role>`.
|
|
89
|
+
- **Pagination** — use query params (`?page=:page&limit=:limit`, imported into `params`) + assert the page shape with plain field access (`{{list.body.next_page_token}}`, `{{list.body.items.length}}`); no special primitive needed.
|
|
90
|
+
|
|
57
91
|
## Rules
|
|
58
92
|
- **No HTTP, no selectors** — only `.feature` + the reviewed `apis.yaml` + `test-data`.
|
|
59
93
|
- **Non-prod default** — a `production` datasource is refused unless `SUNGEN_ALLOW_PROD=1`.
|