@sun-asterisk/sungen 3.1.0 → 3.1.2-beta.100
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -428
- package/dist/capabilities/builtins.d.ts +31 -0
- package/dist/capabilities/builtins.d.ts.map +1 -0
- package/dist/capabilities/builtins.js +84 -0
- package/dist/capabilities/builtins.js.map +1 -0
- package/dist/capabilities/context-router.d.ts +34 -0
- package/dist/capabilities/context-router.d.ts.map +1 -0
- package/dist/capabilities/context-router.js +49 -0
- package/dist/capabilities/context-router.js.map +1 -0
- package/dist/capabilities/context.d.ts +51 -0
- package/dist/capabilities/context.d.ts.map +1 -0
- package/dist/capabilities/context.js +17 -0
- package/dist/capabilities/context.js.map +1 -0
- package/dist/capabilities/discover.d.ts +2 -0
- package/dist/capabilities/discover.d.ts.map +1 -0
- package/dist/capabilities/discover.js +48 -0
- package/dist/capabilities/discover.js.map +1 -0
- package/dist/capabilities/registry.d.ts +90 -0
- package/dist/capabilities/registry.d.ts.map +1 -0
- package/dist/capabilities/registry.js +43 -0
- package/dist/capabilities/registry.js.map +1 -0
- package/dist/capabilities/sensor.d.ts +49 -0
- package/dist/capabilities/sensor.d.ts.map +1 -0
- package/dist/capabilities/sensor.js +3 -0
- package/dist/capabilities/sensor.js.map +1 -0
- package/dist/cli/commands/challenge.d.ts.map +1 -1
- package/dist/cli/commands/challenge.js +9 -2
- package/dist/cli/commands/challenge.js.map +1 -1
- package/dist/cli/commands/delivery.d.ts.map +1 -1
- package/dist/cli/commands/delivery.js +3 -2
- package/dist/cli/commands/delivery.js.map +1 -1
- package/dist/cli/commands/generate.d.ts.map +1 -1
- package/dist/cli/commands/generate.js +12 -0
- package/dist/cli/commands/generate.js.map +1 -1
- package/dist/cli/index.js +10 -1
- package/dist/cli/index.js.map +1 -1
- package/dist/exporters/csv-exporter.d.ts.map +1 -1
- package/dist/exporters/csv-exporter.js +92 -76
- package/dist/exporters/csv-exporter.js.map +1 -1
- package/dist/exporters/spec-parser.d.ts.map +1 -1
- package/dist/exporters/spec-parser.js +6 -1
- package/dist/exporters/spec-parser.js.map +1 -1
- package/dist/generators/test-generator/adapters/adapter-interface.d.ts +2 -0
- package/dist/generators/test-generator/adapters/adapter-interface.d.ts.map +1 -1
- package/dist/generators/test-generator/adapters/playwright/playwright-adapter.d.ts +1 -0
- package/dist/generators/test-generator/adapters/playwright/playwright-adapter.d.ts.map +1 -1
- package/dist/generators/test-generator/adapters/playwright/playwright-adapter.js.map +1 -1
- package/dist/generators/test-generator/adapters/playwright/templates/imports.hbs +3 -0
- package/dist/generators/test-generator/adapters/playwright/templates/scenario.hbs +19 -1
- package/dist/generators/test-generator/code-generator.d.ts +21 -4
- package/dist/generators/test-generator/code-generator.d.ts.map +1 -1
- package/dist/generators/test-generator/code-generator.js +118 -57
- package/dist/generators/test-generator/code-generator.js.map +1 -1
- package/dist/generators/test-generator/patterns/expect-patterns.d.ts +3 -0
- package/dist/generators/test-generator/patterns/expect-patterns.d.ts.map +1 -0
- package/dist/generators/test-generator/patterns/expect-patterns.js +54 -0
- package/dist/generators/test-generator/patterns/expect-patterns.js.map +1 -0
- package/dist/generators/test-generator/patterns/index.d.ts +0 -10
- package/dist/generators/test-generator/patterns/index.d.ts.map +1 -1
- package/dist/generators/test-generator/patterns/index.js +10 -45
- package/dist/generators/test-generator/patterns/index.js.map +1 -1
- package/dist/generators/test-generator/step-mapper.d.ts +6 -0
- package/dist/generators/test-generator/step-mapper.d.ts.map +1 -1
- package/dist/generators/test-generator/step-mapper.js +8 -0
- package/dist/generators/test-generator/step-mapper.js.map +1 -1
- package/dist/generators/test-generator/template-engine.d.ts +4 -0
- package/dist/generators/test-generator/template-engine.d.ts.map +1 -1
- package/dist/generators/test-generator/template-engine.js +1 -1
- package/dist/generators/test-generator/template-engine.js.map +1 -1
- package/dist/generators/test-generator/utils/runtime-data-transformer.d.ts +1 -1
- package/dist/generators/test-generator/utils/runtime-data-transformer.d.ts.map +1 -1
- package/dist/generators/test-generator/utils/runtime-data-transformer.js +5 -5
- package/dist/generators/test-generator/utils/runtime-data-transformer.js.map +1 -1
- package/dist/harness/annotation-overrides.d.ts +9 -0
- package/dist/harness/annotation-overrides.d.ts.map +1 -0
- package/dist/harness/annotation-overrides.js +36 -0
- package/dist/harness/annotation-overrides.js.map +1 -0
- package/dist/harness/audit.d.ts.map +1 -1
- package/dist/harness/audit.js +35 -7
- package/dist/harness/audit.js.map +1 -1
- package/dist/harness/catalog/drivers.yaml +35 -12
- package/dist/harness/challenge.d.ts +1 -0
- package/dist/harness/challenge.d.ts.map +1 -1
- package/dist/harness/challenge.js +49 -2
- package/dist/harness/challenge.js.map +1 -1
- package/dist/harness/data-driven-lint.d.ts +7 -0
- package/dist/harness/data-driven-lint.d.ts.map +1 -0
- package/dist/harness/data-driven-lint.js +153 -0
- package/dist/harness/data-driven-lint.js.map +1 -0
- package/dist/harness/parse.d.ts +3 -0
- package/dist/harness/parse.d.ts.map +1 -1
- package/dist/harness/parse.js +25 -0
- package/dist/harness/parse.js.map +1 -1
- package/dist/harness/query-catalog.d.ts +48 -0
- package/dist/harness/query-catalog.d.ts.map +1 -0
- package/dist/harness/query-catalog.js +0 -0
- package/dist/harness/query-catalog.js.map +1 -0
- package/dist/harness/script-check.d.ts.map +1 -1
- package/dist/harness/script-check.js +7 -4
- package/dist/harness/script-check.js.map +1 -1
- package/dist/index.d.ts +20 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +32 -0
- package/dist/index.js.map +1 -0
- package/dist/orchestrator/templates/ai-instructions/claude-agent-challenge.md +3 -2
- package/dist/orchestrator/templates/ai-instructions/claude-cmd-create-test.md +4 -3
- package/dist/orchestrator/templates/ai-instructions/claude-skill-gherkin-syntax.md +41 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +22 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-review.md +1 -0
- package/dist/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +6 -0
- package/dist/orchestrator/templates/ai-instructions/copilot-cmd-create-test.md +4 -3
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-gherkin-syntax.md +41 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +22 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-review.md +1 -0
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +6 -0
- package/dist/orchestrator/templates/specs-api.d.ts +19 -0
- package/dist/orchestrator/templates/specs-api.d.ts.map +1 -0
- package/dist/orchestrator/templates/specs-api.js +128 -0
- package/dist/orchestrator/templates/specs-api.js.map +1 -0
- package/dist/orchestrator/templates/specs-api.ts +101 -0
- package/dist/orchestrator/templates/specs-db.d.ts +8 -0
- package/dist/orchestrator/templates/specs-db.d.ts.map +1 -1
- package/dist/orchestrator/templates/specs-db.js +22 -0
- package/dist/orchestrator/templates/specs-db.js.map +1 -1
- package/dist/orchestrator/templates/specs-db.ts +22 -0
- package/dist/orchestrator/templates/specs-test-data.ts +76 -15
- package/package.json +7 -30
- package/src/capabilities/builtins.ts +85 -0
- package/src/capabilities/context-router.ts +66 -0
- package/src/capabilities/context.ts +46 -0
- package/src/capabilities/discover.ts +42 -0
- package/src/capabilities/registry.ts +111 -0
- package/src/capabilities/sensor.ts +47 -0
- package/src/cli/commands/challenge.ts +6 -2
- package/src/cli/commands/delivery.ts +3 -2
- package/src/cli/commands/generate.ts +12 -0
- package/src/cli/index.ts +10 -1
- package/src/exporters/csv-exporter.ts +22 -6
- package/src/exporters/spec-parser.ts +6 -1
- package/src/generators/test-generator/adapters/adapter-interface.ts +2 -1
- package/src/generators/test-generator/adapters/playwright/playwright-adapter.ts +1 -1
- package/src/generators/test-generator/adapters/playwright/templates/imports.hbs +3 -0
- package/src/generators/test-generator/adapters/playwright/templates/scenario.hbs +19 -1
- package/src/generators/test-generator/code-generator.ts +114 -59
- package/src/generators/test-generator/patterns/expect-patterns.ts +49 -0
- package/src/generators/test-generator/patterns/index.ts +9 -33
- package/src/generators/test-generator/step-mapper.ts +9 -0
- package/src/generators/test-generator/template-engine.ts +5 -2
- package/src/generators/test-generator/utils/runtime-data-transformer.ts +5 -5
- package/src/harness/annotation-overrides.ts +25 -0
- package/src/harness/audit.ts +37 -8
- package/src/harness/catalog/drivers.yaml +35 -12
- package/src/harness/challenge.ts +47 -2
- package/src/harness/data-driven-lint.ts +119 -0
- package/src/harness/parse.ts +17 -0
- package/src/harness/query-catalog.ts +0 -0
- package/src/harness/script-check.ts +8 -5
- package/src/index.ts +30 -0
- package/src/orchestrator/templates/ai-instructions/claude-agent-challenge.md +3 -2
- package/src/orchestrator/templates/ai-instructions/claude-cmd-create-test.md +4 -3
- package/src/orchestrator/templates/ai-instructions/claude-skill-gherkin-syntax.md +41 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +22 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-review.md +1 -0
- package/src/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +6 -0
- package/src/orchestrator/templates/ai-instructions/copilot-cmd-create-test.md +4 -3
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-gherkin-syntax.md +41 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +22 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-review.md +1 -0
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +6 -0
- package/src/orchestrator/templates/specs-api.ts +101 -0
- package/src/orchestrator/templates/specs-db.ts +22 -0
- package/src/orchestrator/templates/specs-test-data.ts +76 -15
- package/dist/generators/test-generator/patterns/assertion-patterns.d.ts +0 -7
- package/dist/generators/test-generator/patterns/assertion-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/assertion-patterns.js +0 -626
- package/dist/generators/test-generator/patterns/assertion-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/capture-patterns.d.ts +0 -21
- package/dist/generators/test-generator/patterns/capture-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/capture-patterns.js +0 -87
- package/dist/generators/test-generator/patterns/capture-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/database-patterns.d.ts +0 -5
- package/dist/generators/test-generator/patterns/database-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/database-patterns.js +0 -94
- package/dist/generators/test-generator/patterns/database-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/form-patterns.d.ts +0 -6
- package/dist/generators/test-generator/patterns/form-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/form-patterns.js +0 -160
- package/dist/generators/test-generator/patterns/form-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/interaction-patterns.d.ts +0 -6
- package/dist/generators/test-generator/patterns/interaction-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/interaction-patterns.js +0 -433
- package/dist/generators/test-generator/patterns/interaction-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/keyboard-patterns.d.ts +0 -7
- package/dist/generators/test-generator/patterns/keyboard-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/keyboard-patterns.js +0 -47
- package/dist/generators/test-generator/patterns/keyboard-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/navigation-patterns.d.ts +0 -6
- package/dist/generators/test-generator/patterns/navigation-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/navigation-patterns.js +0 -125
- package/dist/generators/test-generator/patterns/navigation-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/scope-patterns.d.ts +0 -7
- package/dist/generators/test-generator/patterns/scope-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/scope-patterns.js +0 -36
- package/dist/generators/test-generator/patterns/scope-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/scroll-patterns.d.ts +0 -7
- package/dist/generators/test-generator/patterns/scroll-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/scroll-patterns.js +0 -25
- package/dist/generators/test-generator/patterns/scroll-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/setup-patterns.d.ts +0 -6
- package/dist/generators/test-generator/patterns/setup-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/setup-patterns.js +0 -72
- package/dist/generators/test-generator/patterns/setup-patterns.js.map +0 -1
- package/dist/generators/test-generator/patterns/table-patterns.d.ts +0 -19
- package/dist/generators/test-generator/patterns/table-patterns.d.ts.map +0 -1
- package/dist/generators/test-generator/patterns/table-patterns.js +0 -239
- package/dist/generators/test-generator/patterns/table-patterns.js.map +0 -1
- package/docs/orchestration-spec.md +0 -267
- package/src/generators/test-generator/patterns/assertion-patterns.ts +0 -691
- package/src/generators/test-generator/patterns/capture-patterns.ts +0 -97
- package/src/generators/test-generator/patterns/database-patterns.ts +0 -95
- package/src/generators/test-generator/patterns/form-patterns.ts +0 -167
- package/src/generators/test-generator/patterns/interaction-patterns.ts +0 -465
- package/src/generators/test-generator/patterns/keyboard-patterns.ts +0 -51
- package/src/generators/test-generator/patterns/navigation-patterns.ts +0 -140
- package/src/generators/test-generator/patterns/scope-patterns.ts +0 -40
- package/src/generators/test-generator/patterns/scroll-patterns.ts +0 -27
- package/src/generators/test-generator/patterns/setup-patterns.ts +0 -76
- package/src/generators/test-generator/patterns/table-patterns.ts +0 -279
package/src/harness/audit.ts
CHANGED
|
@@ -11,7 +11,7 @@ import * as fs from 'fs';
|
|
|
11
11
|
import { loadScenarios, parseViewpointOverview, ScenarioInfo, ViewpointEntry } from './parse';
|
|
12
12
|
import {
|
|
13
13
|
loadCatalog, viewpointGate, assertionDepth, dataThemesFor, coverageBalance, duplicateClusters, traceability, claimProof, taxonomyLint,
|
|
14
|
-
GateResult, DepthResult, BalanceResult, DuplicateResult, TraceResult, ClaimProofResult, TaxonomyResult,
|
|
14
|
+
GateResult, DepthResult, BalanceResult, DuplicateResult, TraceResult, ClaimProofResult, TaxonomyResult, Catalog,
|
|
15
15
|
} from './sensors';
|
|
16
16
|
import { readIntent, projectRootFromScreenDir, IntentProfile } from './intent';
|
|
17
17
|
import { getProvenance, Provenance } from './provenance';
|
|
@@ -19,6 +19,9 @@ import { specCoverage, SpecCoverageResult, parseSpecClauses } from './spec-cover
|
|
|
19
19
|
import { downstreamScope, manualOracle, readText, DownstreamResult, ManualOracleResult,
|
|
20
20
|
negativeSideEffect, sourceBacked, crossArtifactOwnership } from './quality-gates';
|
|
21
21
|
import { viewpointLedger, parseViewpointItems, LedgerResult } from './viewpoint-ledger';
|
|
22
|
+
import { capabilityRegistry } from '../capabilities/registry';
|
|
23
|
+
import { discoverAndRegisterCapabilities } from '../capabilities/discover';
|
|
24
|
+
import { contextRouter } from '../capabilities/context-router';
|
|
22
25
|
|
|
23
26
|
export interface AuditReport {
|
|
24
27
|
screen: string;
|
|
@@ -63,13 +66,23 @@ export function runAudit(screenDir: string, screenName: string): AuditReport {
|
|
|
63
66
|
|
|
64
67
|
const scenarios: ScenarioInfo[] = loadScenarios(featurePath);
|
|
65
68
|
const viewpoints: ViewpointEntry[] = parseViewpointOverview(viewpointPath);
|
|
66
|
-
|
|
69
|
+
// The viewpoint catalog is owned by the default (UI) capability via the SPI; falls back to the
|
|
70
|
+
// in-core loader if no capability provides one. Same catalog content → identical scores (R2).
|
|
71
|
+
discoverAndRegisterCapabilities();
|
|
72
|
+
const defaultCap = capabilityRegistry.defaultCapabilityId();
|
|
73
|
+
const catalog = ((defaultCap && capabilityRegistry.get(defaultCap)?.viewpoints?.()) as Catalog | undefined) || loadCatalog();
|
|
67
74
|
const spec = specCoverage(specPath, scenarios, featureText);
|
|
68
75
|
|
|
69
|
-
const gate = viewpointGate(scenarios, viewpoints, catalog);
|
|
70
76
|
// P3 — intent profile from qa/context.md drives the depth threshold (focus).
|
|
71
77
|
const intent = readIntent(projectRootFromScreenDir(screenDir));
|
|
72
|
-
|
|
78
|
+
// The viewpoint coverage gate + assertion depth are owned by the default (UI) capability and
|
|
79
|
+
// obtained via its `gateProvider` (R2.2b). Same functions underneath → byte-identical gate/depth
|
|
80
|
+
// → identical score. Falls back to the in-core functions if no capability provides them.
|
|
81
|
+
const uiGate = (defaultCap && capabilityRegistry.get(defaultCap)?.gateProvider) as
|
|
82
|
+
((i: { scenarios: ScenarioInfo[]; viewpoints: ViewpointEntry[]; catalog: Catalog; focus: typeof intent.focus }) => { gate: GateResult; depth: DepthResult }) | undefined;
|
|
83
|
+
const provided = uiGate?.({ scenarios, viewpoints, catalog, focus: intent.focus });
|
|
84
|
+
const gate = provided?.gate ?? viewpointGate(scenarios, viewpoints, catalog);
|
|
85
|
+
const depth = provided?.depth ?? assertionDepth(scenarios, dataThemesFor(catalog, gate.pageType), intent.focus);
|
|
73
86
|
const claim = claimProof(scenarios, intent.focus);
|
|
74
87
|
const taxonomy = taxonomyLint(scenarios);
|
|
75
88
|
const balance = coverageBalance(scenarios);
|
|
@@ -125,9 +138,7 @@ export function runAudit(screenDir: string, screenName: string): AuditReport {
|
|
|
125
138
|
if (trace.mappedRatio < 0.5) {
|
|
126
139
|
findings.push(`TRACE: ${trace.note}`);
|
|
127
140
|
}
|
|
128
|
-
|
|
129
|
-
findings.push(`UNIVERSAL: missing theme(s): ${gate.universalGaps.join(', ')} (low priority reminder).`);
|
|
130
|
-
}
|
|
141
|
+
// (UNIVERSAL viewpoint-gap finding now emitted by the `ui` gate sensor — see the gate block below.)
|
|
131
142
|
for (const g of spec.triggerGaps) {
|
|
132
143
|
findings.push(`TRIGGER-UNCOVERED: spec validates "${g.constraint}"${g.code ? ` (${g.code})` : ''} on [${g.required.join(', ')}] but scenarios only exercise it on [${g.found.join(', ') || 'none'}] → add a ${g.missing.join(', ')}-trigger scenario for this constraint (don't collapse the trigger × input matrix).`);
|
|
133
144
|
}
|
|
@@ -157,6 +168,24 @@ export function runAudit(screenDir: string, screenName: string): AuditReport {
|
|
|
157
168
|
findings.push(`UNSOURCEABLE-SCENARIO: "${u}" doesn't trace to any FR / viewpoint item — link it to a source, or tag it @exploration (not part of the official suite).`);
|
|
158
169
|
}
|
|
159
170
|
|
|
171
|
+
// Capability gate sensors (Capability SPI): the ContextRouter scopes WHICH gate sensors run to
|
|
172
|
+
// the capabilities this feature actually uses — generic ('core') + the default UI + any whose
|
|
173
|
+
// annotation tags appear (e.g. @query). Today core+ui gate sensors are always in scope, so this
|
|
174
|
+
// is behaviour-identical; it bounds the set as capability-specific gate sensors (@api, …) are
|
|
175
|
+
// added. Each runs over the audit context; an 'error' finding fails the gate.
|
|
176
|
+
const featureTags = [
|
|
177
|
+
...(scenarios.some((s) => s.queryRefs && s.queryRefs.length) ? ['@query'] : []),
|
|
178
|
+
...(scenarios.some((s) => s.apiRefs && s.apiRefs.length) ? ['@api'] : []),
|
|
179
|
+
];
|
|
180
|
+
const routedGateIds = contextRouter.route({ target: { kind: 'screen', id: screenName }, artifact: 'feature', tags: featureTags }).gateSensorIds;
|
|
181
|
+
const gateSensorFindings = capabilityRegistry.sensors('gate')
|
|
182
|
+
.filter((s) => routedGateIds.includes(s.id))
|
|
183
|
+
.flatMap((s) => s.run({ screenName, cwd: projectRootFromScreenDir(screenDir), featureText, scenarios, universalGaps: gate.universalGaps }));
|
|
184
|
+
// Each gate sensor's message carries its own code prefix (VERIFICATION-FAIL / UNIVERSAL / …)
|
|
185
|
+
// → push verbatim.
|
|
186
|
+
for (const f of gateSensorFindings) findings.push(f.message);
|
|
187
|
+
const gateSensorError = gateSensorFindings.some((f) => f.severity === 'error');
|
|
188
|
+
|
|
160
189
|
// #8 — multi-axis calibration: a high overall must not hide a weak axis.
|
|
161
190
|
const manualCompleteness = manualOracleResult.manualTotal
|
|
162
191
|
? 1 - manualOracleResult.insufficient.length / manualOracleResult.manualTotal : 1;
|
|
@@ -180,7 +209,7 @@ export function runAudit(screenDir: string, screenName: string): AuditReport {
|
|
|
180
209
|
// Gate spans coverage (viewpoint themes), depth, claim-proof, spec-clause coverage,
|
|
181
210
|
// AND taxonomy-match (scenarios must use the project's viewpoint IDs when defined).
|
|
182
211
|
const gateStatus: 'PASS' | 'FAIL' =
|
|
183
|
-
gate.gaps.length === 0 && depth.verdict !== 'fail' && claim.verdict !== 'fail' && spec.verdict !== 'fail' && !taxonomyMismatch ? 'PASS' : 'FAIL';
|
|
212
|
+
gate.gaps.length === 0 && depth.verdict !== 'fail' && claim.verdict !== 'fail' && spec.verdict !== 'fail' && !taxonomyMismatch && !gateSensorError ? 'PASS' : 'FAIL';
|
|
184
213
|
|
|
185
214
|
return {
|
|
186
215
|
screen: screenName,
|
|
@@ -1,21 +1,37 @@
|
|
|
1
1
|
# Driver Catalog (metadata only — NO driver code is bundled here).
|
|
2
2
|
# Lets Sungen RECOMMEND/RESOLVE a driver that may not be installed yet, and tells
|
|
3
|
-
# `sungen capability add` which package to install. See docs/spec/sungen_phase2a_spec.md
|
|
3
|
+
# `sungen capability add` which package to install. See docs/spec/sungen_phase2a_spec.md
|
|
4
|
+
# and docs/spec/sungen_packaging_spec.md (R5 — the capability SPI + npm packages).
|
|
4
5
|
#
|
|
5
|
-
#
|
|
6
|
-
#
|
|
7
|
-
#
|
|
6
|
+
# Two axes:
|
|
7
|
+
# kind: platform → HOW tests run (runtime/codegen adapter). Pick ONE per project.
|
|
8
|
+
# kind: capability → WHAT extra thing is verified, added on top of a platform.
|
|
9
|
+
# Fields:
|
|
10
|
+
# status: shipped → published as an npm package (R5); planned → not built yet.
|
|
11
|
+
# bundled: true → installed automatically (a dependency of @sun-asterisk/sungen),
|
|
12
|
+
# so `capability add` is unnecessary.
|
|
13
|
+
# unblocks: manual-reason codes (M1–M9) this driver resolves (Phase 2b taxonomy).
|
|
14
|
+
#
|
|
15
|
+
# R5 status: the three real capabilities ship as packages — @sungen/driver-ui (web UI,
|
|
16
|
+
# bundled as the default), @sungen/driver-db, @sungen/driver-api. The web *platform*
|
|
17
|
+
# entry below points at @sungen/driver-ui (the UI step vocabulary + viewpoint gate); the
|
|
18
|
+
# Playwright codegen *adapter* itself is still in-core (Phase 2a). Mobile + the remaining
|
|
19
|
+
# capabilities are planned. See the "Platform axis & mobile evolution" section of the
|
|
20
|
+
# packaging spec for the `sungen init --platform <web|mobile>` roadmap.
|
|
8
21
|
|
|
9
22
|
drivers:
|
|
10
23
|
web:
|
|
11
24
|
kind: platform
|
|
12
|
-
package: "@sungen/driver-
|
|
13
|
-
|
|
14
|
-
|
|
25
|
+
package: "@sungen/driver-ui" # R5: web UI capability (step patterns + viewpoint gate)
|
|
26
|
+
status: shipped
|
|
27
|
+
bundled: true # @sun-asterisk/sungen depends on it → UI works out of the box
|
|
28
|
+
runtime: playwright # codegen adapter still in-core (Phase 2a)
|
|
29
|
+
adapter: web # registry adapter name
|
|
15
30
|
capabilities: ["@ui"]
|
|
16
31
|
mobile:
|
|
17
32
|
kind: platform
|
|
18
33
|
package: "@sungen/driver-mobile"
|
|
34
|
+
status: planned # PoC on the feat/mobile branch (Appium / Flutter)
|
|
19
35
|
runtime: appium
|
|
20
36
|
adapter: mobile
|
|
21
37
|
capabilities: ["@ui"]
|
|
@@ -23,35 +39,42 @@ drivers:
|
|
|
23
39
|
api:
|
|
24
40
|
kind: capability
|
|
25
41
|
package: "@sungen/driver-api"
|
|
42
|
+
status: shipped
|
|
26
43
|
capabilities: ["@api", "@apiAssert", "@hybrid"]
|
|
27
44
|
unblocks: [M2]
|
|
28
|
-
data-factory:
|
|
29
|
-
kind: capability
|
|
30
|
-
package: "@sungen/driver-data-factory"
|
|
31
|
-
capabilities: ["@dataFactory"]
|
|
32
|
-
unblocks: [M1]
|
|
33
45
|
db:
|
|
34
46
|
kind: capability
|
|
35
47
|
package: "@sungen/driver-db"
|
|
48
|
+
status: shipped
|
|
36
49
|
capabilities: ["@dbAssert"]
|
|
37
50
|
unblocks: [M2]
|
|
51
|
+
data-factory:
|
|
52
|
+
kind: capability
|
|
53
|
+
package: "@sungen/driver-data-factory"
|
|
54
|
+
status: planned
|
|
55
|
+
capabilities: ["@dataFactory"]
|
|
56
|
+
unblocks: [M1]
|
|
38
57
|
mock:
|
|
39
58
|
kind: capability
|
|
40
59
|
package: "@sungen/driver-mock"
|
|
60
|
+
status: planned
|
|
41
61
|
capabilities: ["@mock", "@network"]
|
|
42
62
|
unblocks: [M3]
|
|
43
63
|
mail-file:
|
|
44
64
|
kind: capability
|
|
45
65
|
package: "@sungen/driver-mail-file"
|
|
66
|
+
status: planned
|
|
46
67
|
capabilities: ["@mail", "@file"]
|
|
47
68
|
unblocks: [M5]
|
|
48
69
|
contract:
|
|
49
70
|
kind: capability
|
|
50
71
|
package: "@sungen/driver-contract"
|
|
72
|
+
status: planned
|
|
51
73
|
capabilities: ["@contract"]
|
|
52
74
|
unblocks: [M5]
|
|
53
75
|
specialized:
|
|
54
76
|
kind: capability
|
|
55
77
|
package: "@sungen/driver-specialized"
|
|
78
|
+
status: planned
|
|
56
79
|
capabilities: ["@specialized"]
|
|
57
80
|
unblocks: [M6]
|
package/src/harness/challenge.ts
CHANGED
|
@@ -34,6 +34,8 @@ export interface ChallengeReport {
|
|
|
34
34
|
shallowThemes: string[];
|
|
35
35
|
// Depth critic
|
|
36
36
|
collectionClaimSingular: ChallengeFinding[];
|
|
37
|
+
// Data-driven critic — @cases-worthy gaps (spec-independent)
|
|
38
|
+
dataDriven: ChallengeFinding[];
|
|
37
39
|
// Novelty critic (deterministic prompts → AI agent fills candidates)
|
|
38
40
|
noveltyPrompts: string[];
|
|
39
41
|
// Roll-up
|
|
@@ -47,6 +49,10 @@ const PLURAL_NOUN = /\b(cards|items|products|rows|results|prices|entries|records
|
|
|
47
49
|
const QUANTIFIER = /\b(all|every|each)\b/i;
|
|
48
50
|
const DISPLAY_VERB = /\b(displays?|shows?|lists?|grid|contains?)\b/i;
|
|
49
51
|
|
|
52
|
+
// Title lexicon implying a CLASS of inputs (→ a data-driven @cases candidate). Used by
|
|
53
|
+
// Detector A. Spec-independent — reads the title, not the spec.
|
|
54
|
+
const INPUT_CLASS_LEXICON = /\b(invalid|formats?|boundary|range|min|max|too (?:long|short)|special char|each|various|classes)\b/i;
|
|
55
|
+
|
|
50
56
|
/** Risk lenses the Novelty critic prompts the AI to explore (beyond the catalog). */
|
|
51
57
|
const NOVELTY_LENSES = [
|
|
52
58
|
'double-submit / rapid repeat of the primary action (duplicate side-effects?)',
|
|
@@ -88,17 +94,49 @@ export function buildChallenge(screenDir: string, screenName: string): Challenge
|
|
|
88
94
|
}
|
|
89
95
|
}
|
|
90
96
|
|
|
91
|
-
// 3.
|
|
97
|
+
// 3. Data-driven critic — surface @cases-worthy gaps (spec-independent).
|
|
98
|
+
const dataDriven: ChallengeFinding[] = [];
|
|
99
|
+
const clustered = new Set<string>();
|
|
100
|
+
// Detector B (high precision): ≥2 non-@cases scenarios sharing the SAME step skeleton
|
|
101
|
+
// (stepSkeleton normalizes {{vars}}/quoted values) → they differ only by data → collapse.
|
|
102
|
+
const bySkeleton = new Map<string, ScenarioInfo[]>();
|
|
103
|
+
for (const s of scenarios) {
|
|
104
|
+
if (s.manual || s.casesDataset || !s.stepSkeleton) continue;
|
|
105
|
+
(bySkeleton.get(s.stepSkeleton) ?? bySkeleton.set(s.stepSkeleton, []).get(s.stepSkeleton)!).push(s);
|
|
106
|
+
}
|
|
107
|
+
for (const group of bySkeleton.values()) {
|
|
108
|
+
if (group.length < 2) continue;
|
|
109
|
+
group.forEach((s) => clustered.add(s.name));
|
|
110
|
+
dataDriven.push({
|
|
111
|
+
scenario: group.map((s) => s.name).join(' | '),
|
|
112
|
+
issue: `${group.length} scenarios share the same steps and differ only by input value (data-variants).`,
|
|
113
|
+
suggestion: `Collapse into ONE \`@cases:<dataset>\` — ${group.length} rows in test-data, each \`{{col}}\` a column. See Gherkin → Advanced → Data-driven.`,
|
|
114
|
+
});
|
|
115
|
+
}
|
|
116
|
+
// Detector A (advisory): a lone scenario whose title implies a class of inputs → suggest @cases.
|
|
117
|
+
for (const s of scenarios) {
|
|
118
|
+
if (s.manual || s.casesDataset || clustered.has(s.name)) continue;
|
|
119
|
+
if (INPUT_CLASS_LEXICON.test(s.name)) {
|
|
120
|
+
dataDriven.push({
|
|
121
|
+
scenario: s.name,
|
|
122
|
+
issue: 'Title reads like a CLASS of inputs (validation/boundary/error) but tests a single value.',
|
|
123
|
+
suggestion: 'Consider `@cases` to cover the EP/boundary classes (one row per valid/invalid class), not just one value.',
|
|
124
|
+
});
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
// 4. Novelty critic — deterministic prompts; the AI agent expands these into candidates.
|
|
92
129
|
const noveltyPrompts = NOVELTY_LENSES.map((l) => `Find 1 non-obvious, valuable scenario via: ${l}`);
|
|
93
130
|
|
|
94
131
|
// Roll-up — exploration readiness signals (not a fake score).
|
|
95
132
|
const explorationReadiness: string[] = [];
|
|
133
|
+
if (dataDriven.length) explorationReadiness.push(`${dataDriven.length} data-driven gap(s) — scenarios that should be one \`@cases\` (collapse data-variants / cover EP-boundary classes).`);
|
|
96
134
|
if (collectionClaimSingular.length) explorationReadiness.push(`${collectionClaimSingular.length} title↔assertion gap(s) — deterministic depth critic flagged these; an AI Business-Depth critic should confirm + fix.`);
|
|
97
135
|
if (overCovered.length) explorationReadiness.push(`${overCovered.length} possibly over-covered area(s) — rebalance toward correctness.`);
|
|
98
136
|
if (shallowThemes.length) explorationReadiness.push(`Shallow themes: ${shallowThemes.join(', ')}.`);
|
|
99
137
|
explorationReadiness.push('Novelty candidates are NOT generated deterministically — run the `sungen-challenge` agent (Claude) or its inline criteria (Copilot) to propose them, then QA accept/reject (≤20% of official, no auto-merge).');
|
|
100
138
|
|
|
101
|
-
return { screen: screenName, overCovered, shallowThemes, collectionClaimSingular, noveltyPrompts, explorationReadiness };
|
|
139
|
+
return { screen: screenName, overCovered, shallowThemes, collectionClaimSingular, dataDriven, noveltyPrompts, explorationReadiness };
|
|
102
140
|
}
|
|
103
141
|
|
|
104
142
|
/** Render the Challenge Report as Markdown (advisory — not part of the official suite). */
|
|
@@ -114,6 +152,13 @@ export function renderChallengeMarkdown(r: ChallengeReport): string {
|
|
|
114
152
|
} else lines.push('_none_');
|
|
115
153
|
lines.push('');
|
|
116
154
|
|
|
155
|
+
lines.push('## Data-driven — scenarios that should be `@cases` (one test case, many inputs)');
|
|
156
|
+
if (r.dataDriven.length) {
|
|
157
|
+
lines.push('| Scenario(s) | Issue | Suggested |', '|---|---|---|');
|
|
158
|
+
for (const f of r.dataDriven) lines.push(`| ${f.scenario} | ${f.issue} | ${f.suggestion} |`);
|
|
159
|
+
} else lines.push('_none_');
|
|
160
|
+
lines.push('');
|
|
161
|
+
|
|
117
162
|
lines.push('## Coverage — possibly over-covered / shallow');
|
|
118
163
|
if (r.overCovered.length) for (const o of r.overCovered) lines.push(`- **${o.bucket}** — ${o.note}`);
|
|
119
164
|
if (r.shallowThemes.length) lines.push(`- Shallow themes: ${r.shallowThemes.join(', ')}`);
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Deterministic lint for the data-driven features (`@cases` + `@query`).
|
|
3
|
+
*
|
|
4
|
+
* Advisory (warn, never block) — surfaced after `sungen generate`. Catches the silent
|
|
5
|
+
* footguns: a `{{var}}` in a @cases scenario that is neither a row column nor a top-level
|
|
6
|
+
* key, a name collision that shadows a top-level value, inconsistent dataset rows, and a
|
|
7
|
+
* @query whose param has no value / whose name doesn't resolve.
|
|
8
|
+
*/
|
|
9
|
+
import * as fs from 'fs';
|
|
10
|
+
import * as path from 'path';
|
|
11
|
+
import { parse as parseYaml } from 'yaml';
|
|
12
|
+
import { GherkinParser, ParsedScenario } from '../generators/gherkin-parser';
|
|
13
|
+
import { resolveQuery, lintCatalog } from './query-catalog';
|
|
14
|
+
|
|
15
|
+
export interface DataDrivenWarning {
|
|
16
|
+
scenario?: string;
|
|
17
|
+
message: string;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
const ROW_META = new Set(['case', 'name', 'label', '__label']);
|
|
21
|
+
|
|
22
|
+
/** Collect the inner text of every `{{…}}` reference in a scenario's steps. */
|
|
23
|
+
function collectRefs(sc: ParsedScenario): string[] {
|
|
24
|
+
const refs = new Set<string>();
|
|
25
|
+
for (const st of sc.steps || []) {
|
|
26
|
+
for (const m of (st.text || '').matchAll(/\{\{\s*([^}]+?)\s*\}\}/g)) refs.add(m[1]);
|
|
27
|
+
}
|
|
28
|
+
return [...refs];
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
/** Keys overridden in a `@query:name(a=…,b=…)` annotation. */
|
|
32
|
+
function overrideKeys(raw?: string): Set<string> {
|
|
33
|
+
const out = new Set<string>();
|
|
34
|
+
if (!raw) return out;
|
|
35
|
+
for (const part of raw.split(',')) {
|
|
36
|
+
const eq = part.indexOf('=');
|
|
37
|
+
if (eq > 0) out.add(part.slice(0, eq).trim());
|
|
38
|
+
}
|
|
39
|
+
return out;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
/** Lint the @cases/@query usage of one screen/flow directory. Returns advisory warnings. */
|
|
43
|
+
export function lintDataDriven(screenDir: string, cwd: string = process.cwd()): DataDrivenWarning[] {
|
|
44
|
+
const base = path.basename(screenDir);
|
|
45
|
+
const isFlow = path.basename(path.dirname(screenDir)) === 'flows';
|
|
46
|
+
const screenName = isFlow ? `flows/${base}` : base;
|
|
47
|
+
const featurePath = path.join(screenDir, 'features', `${base}.feature`);
|
|
48
|
+
if (!fs.existsSync(featurePath)) return [];
|
|
49
|
+
|
|
50
|
+
let scenarios: ParsedScenario[];
|
|
51
|
+
try {
|
|
52
|
+
scenarios = new GherkinParser().parseFeatureFile(featurePath).scenarios || [];
|
|
53
|
+
} catch {
|
|
54
|
+
return [];
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
const tdPath = path.join(screenDir, 'test-data', `${base}.yaml`);
|
|
58
|
+
const td: Record<string, any> = tdPath && fs.existsSync(tdPath) ? parseYaml(fs.readFileSync(tdPath, 'utf8')) || {} : {};
|
|
59
|
+
const topKeys = new Set(Object.keys(td));
|
|
60
|
+
const warns: DataDrivenWarning[] = [];
|
|
61
|
+
|
|
62
|
+
for (const sc of scenarios) {
|
|
63
|
+
const tags: string[] = sc.tags || [];
|
|
64
|
+
const refs = collectRefs(sc);
|
|
65
|
+
|
|
66
|
+
// --- @cases -----------------------------------------------------------
|
|
67
|
+
const casesTag = tags.find((t) => t.startsWith('@cases:'));
|
|
68
|
+
if (casesTag) {
|
|
69
|
+
const ds = casesTag.slice('@cases:'.length).trim();
|
|
70
|
+
const rows = td[ds];
|
|
71
|
+
if (!Array.isArray(rows)) {
|
|
72
|
+
warns.push({ scenario: sc.name, message: `@cases:${ds} → dataset "${ds}" is missing or not a list in test-data.` });
|
|
73
|
+
} else {
|
|
74
|
+
const colSets = rows.map((r) => new Set(Object.keys(r || {}).filter((k) => !ROW_META.has(k))));
|
|
75
|
+
const allCols = new Set<string>();
|
|
76
|
+
colSets.forEach((s) => s.forEach((c) => allCols.add(c)));
|
|
77
|
+
for (const c of allCols) {
|
|
78
|
+
const missing = colSets.filter((s) => !s.has(c)).length;
|
|
79
|
+
if (missing) warns.push({ scenario: sc.name, message: `@cases:${ds} → column "${c}" is missing in ${missing}/${rows.length} row(s) — rows are inconsistent.` });
|
|
80
|
+
if (topKeys.has(c)) warns.push({ scenario: sc.name, message: `@cases:${ds} → "${c}" is both a dataset column and a top-level test-data key — the row value shadows the top-level one.` });
|
|
81
|
+
}
|
|
82
|
+
for (const r of refs) {
|
|
83
|
+
const head = r.split(/[.[]/)[0];
|
|
84
|
+
if (!allCols.has(head) && !topKeys.has(head)) {
|
|
85
|
+
warns.push({ scenario: sc.name, message: `@cases:${ds} → {{${r}}} is neither a dataset column nor a top-level test-data key.` });
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
// --- @query -----------------------------------------------------------
|
|
92
|
+
for (const t of tags) {
|
|
93
|
+
const m = t.match(/^@query:([A-Za-z_][A-Za-z0-9_]*)(?:\((.*)\))?$/);
|
|
94
|
+
if (!m) continue;
|
|
95
|
+
const name = m[1];
|
|
96
|
+
const overrides = overrideKeys(m[2]);
|
|
97
|
+
let entry;
|
|
98
|
+
try {
|
|
99
|
+
entry = resolveQuery(name, screenName, cwd);
|
|
100
|
+
} catch (e: any) {
|
|
101
|
+
warns.push({ scenario: sc.name, message: e?.message || `@query:${name} → cannot resolve query.` });
|
|
102
|
+
continue;
|
|
103
|
+
}
|
|
104
|
+
for (const p of entry.params || []) {
|
|
105
|
+
if (!overrides.has(p) && !topKeys.has(p)) {
|
|
106
|
+
warns.push({ scenario: sc.name, message: `@query:${name} → param "${p}" has no value: not in the annotation and not a top-level test-data key.` });
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
// Catalog-level lint (SELECT-only, params declared/used, datasource present).
|
|
113
|
+
try {
|
|
114
|
+
for (const e of lintCatalog(screenName, null, cwd).errors) warns.push({ message: e });
|
|
115
|
+
} catch {
|
|
116
|
+
/* no catalog → nothing to lint */
|
|
117
|
+
}
|
|
118
|
+
return warns;
|
|
119
|
+
}
|
package/src/harness/parse.ts
CHANGED
|
@@ -30,6 +30,9 @@ export interface ScenarioInfo {
|
|
|
30
30
|
haystack: string; // lowercase name + steps text (for keyword coverage)
|
|
31
31
|
stepsText: string; // lowercase steps ONLY (name excluded) — for claim-proof
|
|
32
32
|
vpId?: string; // raw leading ID token of the title (project's scheme: VP0-001, MS-HP-001, VP-LIST-001)
|
|
33
|
+
casesDataset?: string; // @cases:<dataset> — data-driven; one scenario expands to N row-tests
|
|
34
|
+
queryRefs?: string[]; // named queries referenced by this scenario (inline `query [name]` + @query: tags)
|
|
35
|
+
apiRefs?: string[]; // named API endpoints referenced by this scenario (@api: tags)
|
|
33
36
|
}
|
|
34
37
|
|
|
35
38
|
/** Format-tolerant: is this token an ID (project's scheme), not a prose word?
|
|
@@ -98,6 +101,17 @@ const PRIORITY_TAGS: Record<string, Priority> = { '@high': 'high', '@normal': 'n
|
|
|
98
101
|
function classifyScenario(sc: ParsedScenario): ScenarioInfo {
|
|
99
102
|
const tags = sc.tags || [];
|
|
100
103
|
const manual = tags.includes('@manual');
|
|
104
|
+
const casesTag = tags.find((t) => t.startsWith('@cases:'));
|
|
105
|
+
const casesDataset = casesTag ? casesTag.slice('@cases:'.length).trim() : undefined;
|
|
106
|
+
// Named-query references: @query:<name>[(overrides)] tags + inline `query [name]` step refs.
|
|
107
|
+
const queryRefs = new Set<string>();
|
|
108
|
+
for (const t of tags) if (t.startsWith('@query:')) { const m = t.slice('@query:'.length).match(/^([A-Za-z_][A-Za-z0-9_]*)/); if (m) queryRefs.add(m[1]); }
|
|
109
|
+
for (const step of (sc.steps as ParsedStep[]) || []) {
|
|
110
|
+
for (const m of (step.text || '').matchAll(/\bquery\s+\[([A-Za-z_][A-Za-z0-9_]*)\]/gi)) queryRefs.add(m[1]);
|
|
111
|
+
}
|
|
112
|
+
// Named-API references: @api:<name>[(overrides)] tags.
|
|
113
|
+
const apiRefs = new Set<string>();
|
|
114
|
+
for (const t of tags) if (t.startsWith('@api:')) { const m = t.slice('@api:'.length).match(/^([A-Za-z_][A-Za-z0-9_]*)/); if (m) apiRefs.add(m[1]); }
|
|
101
115
|
let priority: Priority = 'unknown';
|
|
102
116
|
for (const t of tags) if (PRIORITY_TAGS[t]) priority = PRIORITY_TAGS[t];
|
|
103
117
|
|
|
@@ -152,6 +166,9 @@ function classifyScenario(sc: ParsedScenario): ScenarioInfo {
|
|
|
152
166
|
haystack: textParts.join(' ').toLowerCase(),
|
|
153
167
|
stepsText: stepTextParts.join(' ').toLowerCase(),
|
|
154
168
|
vpId,
|
|
169
|
+
casesDataset,
|
|
170
|
+
queryRefs: queryRefs.size ? [...queryRefs] : undefined,
|
|
171
|
+
apiRefs: apiRefs.size ? [...apiRefs] : undefined,
|
|
155
172
|
};
|
|
156
173
|
}
|
|
157
174
|
|
|
Binary file
|
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
import * as fs from 'fs';
|
|
16
16
|
import * as path from 'path';
|
|
17
17
|
import * as os from 'os';
|
|
18
|
-
import { loadScenarios } from './parse';
|
|
18
|
+
import { loadScenarios, ScenarioInfo } from './parse';
|
|
19
19
|
|
|
20
20
|
export interface ScriptCheckResult {
|
|
21
21
|
screen: string;
|
|
@@ -146,10 +146,13 @@ export async function runScriptCheck(screenDir: string, screenName: string, flow
|
|
|
146
146
|
}
|
|
147
147
|
|
|
148
148
|
// A. Structural 1:1
|
|
149
|
+
// A @cases scenario emits ONE source test() inside a per-row loop, titled
|
|
150
|
+
// `<name> — ${__row.__label}` — match that literal title, not the bare name.
|
|
151
|
+
const expectedTitle = (s: ScenarioInfo) => (s.casesDataset ? `${s.name} — ${'${'}__row.__label}` : s.name);
|
|
149
152
|
const specTitleSet = new Set(specTitles);
|
|
150
|
-
const
|
|
151
|
-
const missingInSpec = automated.filter((s) => !specTitleSet.has(s
|
|
152
|
-
const extraInSpec = specTitles.filter((t) => !
|
|
153
|
+
const expectedSet = new Set(automated.map(expectedTitle));
|
|
154
|
+
const missingInSpec = automated.filter((s) => !specTitleSet.has(expectedTitle(s))).map((s) => s.name);
|
|
155
|
+
const extraInSpec = specTitles.filter((t) => !expectedSet.has(t));
|
|
153
156
|
const countMatch = committedSpec ? automated.length === specTitles.length : false;
|
|
154
157
|
if (committedSpec && !countMatch) {
|
|
155
158
|
findings.push(`Count mismatch: ${automated.length} automated scenarios vs ${specTitles.length} test() blocks.`);
|
|
@@ -193,7 +196,7 @@ export async function runScriptCheck(screenDir: string, screenName: string, flow
|
|
|
193
196
|
|
|
194
197
|
// C. Anti-bypass / faithfulness
|
|
195
198
|
const { assertionlessTests, hollowSteps } = committedSpec
|
|
196
|
-
? analyzeFaithfulness(specSrc,
|
|
199
|
+
? analyzeFaithfulness(specSrc, expectedSet)
|
|
197
200
|
: { assertionlessTests: [], hollowSteps: [] };
|
|
198
201
|
for (const t of assertionlessTests) {
|
|
199
202
|
findings.push(`BYPASS: test "${t}" has 0 assertions (action-only — proves nothing). The testcase is not really automated.`);
|
package/src/index.ts
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Public API of `@sun-asterisk/sungen` — the capability SPI plus the shared compiler/harness surface
|
|
3
|
+
* that capability drivers (`@sungen/driver-*`) build against. Drivers import from here; core never
|
|
4
|
+
* imports from a driver (discovery loads them at runtime). Keep this surface small and intentional.
|
|
5
|
+
*/
|
|
6
|
+
|
|
7
|
+
// --- Capability SPI ---
|
|
8
|
+
export { capabilityRegistry, CapabilityRegistry } from './capabilities/registry';
|
|
9
|
+
export type { CapabilityDescriptor } from './capabilities/registry';
|
|
10
|
+
export type { Sensor, SensorFinding, AdvisoryScanInput, GateInput } from './capabilities/sensor';
|
|
11
|
+
export type { Context, DiscoveryProvider, ContextMapper, GenerationUnit } from './capabilities/context';
|
|
12
|
+
|
|
13
|
+
// --- Step-pattern authoring (a driver contributes step patterns via its descriptor) ---
|
|
14
|
+
export type { PatternContext, StepPattern, StepTemplateData } from './generators/test-generator/patterns/types';
|
|
15
|
+
export type { MappedStep } from './generators/test-generator/step-mapper';
|
|
16
|
+
export type { ParsedStep } from './generators/gherkin-parser';
|
|
17
|
+
export { getPathCode, inferPath, resolvePathVariables } from './generators/test-generator/utils/path-inference';
|
|
18
|
+
|
|
19
|
+
// --- Precondition-annotation override grammar (shared by the @query / @api driver codegen) ---
|
|
20
|
+
export { parseQueryOverrides } from './harness/annotation-overrides';
|
|
21
|
+
|
|
22
|
+
// --- Named-query catalog (shared: the DB driver's codegen + core's data-driven advisory lint) ---
|
|
23
|
+
export { resolveQuery, compileQuery, lintCatalog } from './harness/query-catalog';
|
|
24
|
+
export type { QueryEntry } from './harness/query-catalog';
|
|
25
|
+
|
|
26
|
+
// --- Shared harness: viewpoint catalog + coverage gate / assertion depth ---
|
|
27
|
+
// (the UI capability's gateProvider composes these; they also back core's ingest + audit fallback)
|
|
28
|
+
export { loadCatalog, viewpointGate, assertionDepth, dataThemesFor } from './harness/sensors';
|
|
29
|
+
export type { Catalog, GateResult, DepthResult } from './harness/sensors';
|
|
30
|
+
export type { ScenarioInfo, ViewpointEntry } from './harness/parse';
|
|
@@ -15,11 +15,12 @@ Run `sungen challenge --screen <name>` (Bash) and read its report (`.sungen/repo
|
|
|
15
15
|
- `.sungen/reports/<name>-audit.json` — what the gate already measured.
|
|
16
16
|
- Blind-spot patterns — run `sungen blindspot list --prompt` (Bash) and check the suite against each known pattern.
|
|
17
17
|
|
|
18
|
-
##
|
|
18
|
+
## Four critics
|
|
19
19
|
|
|
20
20
|
1. **Coverage critic** — viewpoints that are missing or covered only shallowly; areas over-covered with low value (e.g. many subscription edge cases while cart correctness is thin). Recommend rebalancing, not just adding.
|
|
21
21
|
2. **Business-Depth critic** — scenarios whose **title claims more than the steps prove** (a set/collection asserted by one element; "correct X" asserted by mere visibility). For each, give the exact deep step to add. Confirm or dismiss the deterministic flags from `sungen challenge`.
|
|
22
|
-
3. **
|
|
22
|
+
3. **Data-driven critic** — surface `@cases`-worthy gaps, *spec-independent*: (a) **confirm the deterministic collapse suggestions** (`sungen challenge` flags ≥2 scenarios with the same step skeleton → propose the one `@cases:<dataset>` with the rows + a `case` label each); (b) for any action reaching a backend/logic (login, search, create, an API/error path), propose the **corner/error matrix** as `@cases` rows — *invalid · empty · boundary · injection · duplicate · not-found · unauthorized · malformed · rate-limit* — picking the family that fits the screen. These are the cases a spec/viewpoint usually under-specifies.
|
|
23
|
+
4. **Novelty critic** — 3–5 **non-obvious, valuable** scenarios outside the existing pattern, via risk lenses (double-submit, partial-load, boundary/unusual data, concurrency/back-button, historical incidents). Each must map to a risk or viewpoint and explain why it isn't a duplicate.
|
|
23
24
|
|
|
24
25
|
## Guardrails (hard)
|
|
25
26
|
- **Read-only.** Never edit the feature or any file. You return findings; the QA/orchestrator decides.
|
|
@@ -31,9 +31,10 @@ Parse **name** from `$ARGUMENTS`. If missing, ask the user.
|
|
|
31
31
|
**Screen**: Verify `qa/screens/<name>/` exists. If not → `/sungen:add-screen` first.
|
|
32
32
|
2. Check if `.feature` file already has scenarios.
|
|
33
33
|
- If yes → use `AskUserQuestion` to ask the update mode (see `sungen-tc-generation` skill — mode depends on which tiers already exist).
|
|
34
|
-
- If no → fresh creation. Use `AskUserQuestion` to ask generation scope:
|
|
35
|
-
- **Tier 1 — Critical & High priority** — ~10-15 scenarios/section
|
|
36
|
-
- **Full coverage
|
|
34
|
+
- If no → fresh creation. **Write the feature file incrementally** (successive `Write`/`Edit`, ≈10-15 scenarios per call) — never emit the whole suite in one response, or it can exceed the model's output-token cap (`API Error: Claude's response exceeded the N output token maximum`). Use `AskUserQuestion` to ask generation scope:
|
|
35
|
+
- **Tier 1 — Critical & High priority** — ~10-15 scenarios/section: happy paths, core validation, security basics **(Recommended)**
|
|
36
|
+
- **Full coverage (incremental)** — Tier 1 + 2 + 3, written tier-by-tier in batches (`Write` T1 → `Edit` append T2 → `Edit` append T3). Safe on any output-token budget.
|
|
37
|
+
- **Full coverage (single pass)** — generate everything in one go (~40-60 scenarios/section). Faster, but **only if you raised your output cap** (`CLAUDE_CODE_MAX_OUTPUT_TOKENS ≥ 64000`) — otherwise it errors mid-generation. For power users on a high-token model/config.
|
|
37
38
|
3. **Read project context + screen requirements**
|
|
38
39
|
|
|
39
40
|
**Project context** — check `qa/context.md` (project root, not screen-specific):
|
|
@@ -102,6 +102,22 @@ User see [Table] table match data:
|
|
|
102
102
|
|
|
103
103
|
Row scope: `see [Ref] row in [Table] table with {{v}}` enters scope. Subsequent `see [Col] column with {{v}}` checks cell in that row. Use `table match data:` for multi-row verification.
|
|
104
104
|
|
|
105
|
+
### Database verification (optional Data Driver)
|
|
106
|
+
|
|
107
|
+
Read-only DB-state checks. **Prefer named queries** — SQL lives in `qa/screens/<screen>/database/queries.yaml` (reviewed once, parameterized). Invoke with the `@query:<name>` annotation; it binds the result rows to `{{name}}`, then assert with `expect`:
|
|
108
|
+
|
|
109
|
+
```gherkin
|
|
110
|
+
@query:active_user # precondition: run query, bind {{active_user}}
|
|
111
|
+
@query:orders(buyer={{email}}) # …with explicit param override
|
|
112
|
+
Scenario: ...
|
|
113
|
+
Then expect {{active_user.count}} is at least {{one}} # ≥1 row
|
|
114
|
+
And expect {{active_user.first.status}} is "active" # first row's column
|
|
115
|
+
And expect {{orders.count}} is {{expected}} # exact count
|
|
116
|
+
And User see [Total] text is {{orders.first.total}} # UI ↔ DB
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Path access on a bound result: `{{q.count}}`/`{{q.length}}`, `{{q.first.col}}`, `{{q.last.col}}`, `{{q[2].col}}`, `{{q.col}}` (= first row's col). `expect A is B` also supports `is at least` / `is at most` / `is not`. Tier-2 declarative (trivial inline, no catalog): `User see [<table>] row where [<col>] is {{v}} [has [<col2>] = "x"]`, `… no row where …`, `… count is {{n}}`. Full grammar + catalog/datasource/secret rules → **Advanced → Database** doc. Only emit DB steps when the project has a `database/` catalog / `datasources.yaml`.
|
|
120
|
+
|
|
105
121
|
### States
|
|
106
122
|
|
|
107
123
|
`hidden` `visible` `disabled` `enabled` `checked` `unchecked` `focused` `empty` `loading` `selected` `sorted ascending` `sorted descending`
|
|
@@ -195,6 +211,31 @@ Options: `nth` `exact` `scope` `match` `variant` `frame` `contenteditable` `colu
|
|
|
195
211
|
| `@afterEach` | Hook: runs after each test → `test.afterEach()` (custom cleanup) |
|
|
196
212
|
| `@afterAll` | Hook: runs once after all tests → `test.afterAll()` |
|
|
197
213
|
| `@flow` | Mark feature as E2E flow (cross-screen testing) |
|
|
214
|
+
| `@cases:dataset` | Data-driven: run the scenario once per row of the `dataset` LIST in test-data → one `test()` per row |
|
|
215
|
+
| `@query:name` | Database: run the named query from `database/queries.yaml` (precondition) and bind its rows to `{{name}}`; assert with `expect {{name.count}} …` + path access. Override params `@query:name(p={{v}})`. Repeatable. (Optional Data Driver — see Database verification above) |
|
|
216
|
+
| `@api:name` | API: run the named request from `api/apis.yaml` (precondition) and bind the response to `{{name}}`; assert with `expect {{name.status}} …` + path access (`{{name.body.<path>}}`). Override params `@api:name(p={{v}})`. Repeatable. (Optional API Driver) |
|
|
217
|
+
|
|
218
|
+
### Data-driven scenarios (`@cases`)
|
|
219
|
+
|
|
220
|
+
For one test case × many inputs (email/format/boundary validation, decision tables), tag the
|
|
221
|
+
scenario `@cases:<dataset>` and reference each row's columns as `{{col}}`. Put the rows as a LIST
|
|
222
|
+
in test-data — NOT inline; data stays runtime + env-overlayable.
|
|
223
|
+
|
|
224
|
+
```gherkin
|
|
225
|
+
@high @cases:email_validation
|
|
226
|
+
Scenario: VP-VAL-001 The email field rejects invalid formats
|
|
227
|
+
When User fill [Email] field with {{email}}
|
|
228
|
+
Then User see [Login Error] message with {{expected_error}}
|
|
229
|
+
```
|
|
230
|
+
```yaml
|
|
231
|
+
# test-data/<screen>.yaml
|
|
232
|
+
email_validation:
|
|
233
|
+
- { case: "no @", email: "plainaddress", expected_error: "Invalid email" }
|
|
234
|
+
- { case: "valid", email: "ok@x.com", expected_error: "" }
|
|
235
|
+
```
|
|
236
|
+
An optional `case`/`name`/`label` column labels each run. Each row → its own pass/fail. Prefer
|
|
237
|
+
`@cases` over duplicating a scenario per value. (Gherkin `Scenario Outline`/`Examples` is NOT
|
|
238
|
+
supported — use `@cases`.)
|
|
198
239
|
|
|
199
240
|
### Pass-through tags (filter at runtime via Playwright --grep)
|
|
200
241
|
|
|
@@ -6,6 +6,9 @@ user-invocable: false
|
|
|
6
6
|
|
|
7
7
|
## ⚠️ Gotchas — read before generating
|
|
8
8
|
|
|
9
|
+
- **Write incrementally — never emit the whole suite in one response.** Build the `.feature` in batches via successive `Write`/`Edit` (≈10–15 scenarios per call). For **Full coverage**, write tier-by-tier: `Write` Tier 1 → `Edit` append Tier 2 → `Edit` append Tier 3.
|
|
10
|
+
→ One huge `Write` can exceed the model's output-token cap → `API Error: Claude's response exceeded the N output token maximum`. Single-pass full coverage only fits when `CLAUDE_CODE_MAX_OUTPUT_TOKENS ≥ 64000`; otherwise batch. Batching also lets the audit/reviewer run per batch — higher quality.
|
|
11
|
+
|
|
9
12
|
- `spec_figma.md` exists → read file only, **NEVER** call `mcp__figma__*`
|
|
10
13
|
→ PAT auth flow already done by `sungen-capture` (mode figma-pat); re-calling fails or duplicates work.
|
|
11
14
|
|
|
@@ -54,6 +57,25 @@ user-invocable: false
|
|
|
54
57
|
OR condition: generate 1 scenario per branch where that branch alone triggers the outcome.
|
|
55
58
|
→ Happy-path only = missing the most common multi-condition implementation bug.
|
|
56
59
|
|
|
60
|
+
- **Many inputs, same steps → ONE data-driven scenario (`@cases`), not N copies:**
|
|
61
|
+
When a rule needs lots of inputs with the *same* step shape (email/format validation,
|
|
62
|
+
BVA boundary triples, EP classes, decision-table rows), tag one scenario `@cases:<dataset>`,
|
|
63
|
+
reference each row's columns as `{{col}}`, and put the rows as a LIST in test-data:
|
|
64
|
+
```gherkin
|
|
65
|
+
@high @cases:email_validation
|
|
66
|
+
Scenario: VP-VAL-001 The email field rejects invalid formats
|
|
67
|
+
When User fill [Email] field with {{email}}
|
|
68
|
+
Then User see [Error] message with {{expected_error}}
|
|
69
|
+
```
|
|
70
|
+
```yaml
|
|
71
|
+
email_validation:
|
|
72
|
+
- { case: "no @", email: "plainaddress", expected_error: "Invalid email" }
|
|
73
|
+
- { case: "valid", email: "ok@x.com", expected_error: "" }
|
|
74
|
+
```
|
|
75
|
+
→ one `test()` per row, each labelled by `case`. Adding inputs = editing test-data (no recompile),
|
|
76
|
+
and env overlays apply. Prefer this over duplicating a scenario per value. (Gherkin
|
|
77
|
+
`Scenario Outline`/`Examples` is NOT supported — use `@cases`.)
|
|
78
|
+
|
|
57
79
|
---
|
|
58
80
|
|
|
59
81
|
## Tier System
|
|
@@ -120,6 +120,7 @@ Build a mapping table: for each applicable group, does the feature have a matchi
|
|
|
120
120
|
- **EP**: keep only **one representative** per invalid class; same-class duplicates → flag as redundant.
|
|
121
121
|
- **BVA**: spec defines min/max → cover `min-1`, `min`, `max`, `max+1` (Maxlength, counts…).
|
|
122
122
|
- Error messages must match the spec **word-for-word**, not generic.
|
|
123
|
+
- **Data-driven (`@cases`)**: a `@cases:<dataset>` scenario legitimately covers many inputs in ONE scenario (one row per EP class / boundary / rule). Do **not** flag it as "too few negative cases" or as duplication — instead review the **dataset rows**: are all EP classes / boundary triples present, each labelled, expected values exact? N near-identical scenarios that differ only by input value → flag and recommend collapsing to `@cases`.
|
|
123
124
|
|
|
124
125
|
---
|
|
125
126
|
|