@yasserkhanorg/e2e-agents 1.7.2 → 1.7.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +73 -10
- package/dist/engine/ai_enrichment.d.ts.map +1 -1
- package/dist/engine/ai_enrichment.js +4 -3
- package/dist/engine/impact_engine.d.ts +12 -0
- package/dist/engine/impact_engine.d.ts.map +1 -1
- package/dist/engine/impact_engine.js +52 -2
- package/dist/engine/plan_builder.d.ts.map +1 -1
- package/dist/engine/plan_builder.js +54 -4
- package/dist/esm/engine/ai_enrichment.js +4 -3
- package/dist/esm/engine/impact_engine.js +52 -2
- package/dist/esm/engine/plan_builder.js +54 -4
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -56,7 +56,67 @@ npx e2e-ai-agents llm-health
|
|
|
56
56
|
|
|
57
57
|
## Route-Families Training
|
|
58
58
|
|
|
59
|
-
|
|
59
|
+
### What it produces
|
|
60
|
+
|
|
61
|
+
The `train` command builds a **knowledge map** of your codebase — a single JSON file (`route-families.json`) that maps source files to features, test directories, and user flows. This is not ML training; no model is trained. It's building a structured manifest like:
|
|
62
|
+
|
|
63
|
+
```json
|
|
64
|
+
{
|
|
65
|
+
"id": "channels",
|
|
66
|
+
"routes": ["/{team}/channels/{channel}"],
|
|
67
|
+
"priority": "P0",
|
|
68
|
+
"webappPaths": ["src/components/channel_header/**"],
|
|
69
|
+
"serverPaths": ["server/channels/api4/channel*.go", "server/channels/app/channel*.go"],
|
|
70
|
+
"specDirs": ["specs/functional/channels/"],
|
|
71
|
+
"userFlows": ["Create channel", "Archive channel", "Search in channel"],
|
|
72
|
+
"components": ["ChannelHeader", "ChannelSidebar"]
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
### Why the tool needs this
|
|
77
|
+
|
|
78
|
+
When a PR changes `server/channels/app/channel.go`, the tool needs to answer: **"which E2E tests should I run?"** Without the manifest, it has no idea. With it:
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
channel.go changed
|
|
82
|
+
→ belongs to "channels" family
|
|
83
|
+
→ specs are in specs/functional/channels/
|
|
84
|
+
→ run those tests
|
|
85
|
+
→ flag if coverage is missing for the affected user flows
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Every downstream command (`impact`, `plan`, `generate`, `heal`, `e2e-qa-agent`) reads this manifest to understand the codebase.
|
|
89
|
+
|
|
90
|
+
### How scanning works
|
|
91
|
+
|
|
92
|
+
The scanner uses 4 strategies to build the `file → family` mapping:
|
|
93
|
+
|
|
94
|
+
1. **Directory matching** — `src/channels/` + `tests/channels/` share a name → channels family
|
|
95
|
+
2. **Test-derived** — `specs/functional/channels/drafts/` exists with spec files → drafts family (even if source code is scattered across components/actions/reducers)
|
|
96
|
+
3. **Server-derived** — `api4/channel.go` + `app/channel.go` + `store/channel_store.go` span 3 backend tiers → channel family (related files like `channel_bookmark.go` are grouped under the parent)
|
|
97
|
+
4. **Name-matched** — `src/utils/channels.ts` or `server/public/model/channel.go` basename matches → add to channels family's paths
|
|
98
|
+
|
|
99
|
+
### What LLM enrichment adds
|
|
100
|
+
|
|
101
|
+
The scanner finds files. The LLM reads code samples and adds **semantic metadata** the scanner can't determine:
|
|
102
|
+
- Accurate URL routes (`/{team}/channels/{channel}` instead of guessed `/channels`)
|
|
103
|
+
- Priority classification (P0 critical user flow vs P2 nice-to-have)
|
|
104
|
+
- Human-readable user flows ("Create channel", "Search messages")
|
|
105
|
+
- React component and page object names
|
|
106
|
+
|
|
107
|
+
This metadata makes impact analysis smarter — it can prioritize P0 flows and suggest specific test scenarios.
|
|
108
|
+
|
|
109
|
+
### What validation does
|
|
110
|
+
|
|
111
|
+
The `--validate` flag measures manifest accuracy against **real git history**. It's not training data — it's a quality check:
|
|
112
|
+
|
|
113
|
+
```
|
|
114
|
+
835 commits → 5105 changed files → 3223 bound to a family = 63% coverage
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
This tells you the manifest is complete enough. If coverage were 30%, impact analysis would be blind to most code changes.
|
|
118
|
+
|
|
119
|
+
### Usage
|
|
60
120
|
|
|
61
121
|
```bash
|
|
62
122
|
# Scan your codebase + LLM enrichment (default)
|
|
@@ -72,18 +132,19 @@ npx e2e-ai-agents train --path /path/to/project --validate --since HEAD~50
|
|
|
72
132
|
npx e2e-ai-agents train --path /path/to/project --validate --since HEAD~20
|
|
73
133
|
```
|
|
74
134
|
|
|
75
|
-
**Why LLM enrichment is on by default:** The manifest
|
|
135
|
+
**Why LLM enrichment is on by default:** The manifest gives AI context for impact analysis, scenario suggestion, and bug detection. AI-generated context produces better AI reasoning downstream. Use `--no-enrich` for offline/free operation or to avoid sending code snippets to third-party LLM APIs.
|
|
76
136
|
|
|
77
|
-
**Training loop:** Run `train` → review
|
|
137
|
+
**Training loop:** Run `train` → review `route-families.json` → run `train --validate` to check coverage % → fix gaps → repeat.
|
|
78
138
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
5. **Validates** against git history to measure accuracy
|
|
139
|
+
**Additional flags:**
|
|
140
|
+
- `--verbose` / `-v` — DEBUG-level output with timing for each phase
|
|
141
|
+
- `--json` — structured JSON log output (for CI pipelines)
|
|
142
|
+
- `--server-path` — explicit path to backend server root
|
|
143
|
+
- `--budget-usd` — max LLM spend (default: $0.50, max: $10)
|
|
85
144
|
|
|
86
|
-
Output
|
|
145
|
+
**Output:**
|
|
146
|
+
- `<testsRoot>/.e2e-ai-agents/route-families.json` — the manifest
|
|
147
|
+
- `<testsRoot>/.e2e-ai-agents/train-report.json` — timing data, family counts, coverage stats, LLM metrics
|
|
87
148
|
|
|
88
149
|
## Configuration
|
|
89
150
|
|
|
@@ -227,6 +288,8 @@ Schemas: [schemas/traceability-input.schema.json](schemas/traceability-input.sch
|
|
|
227
288
|
|
|
228
289
|
| File | Written by | Purpose |
|
|
229
290
|
|------|-----------|---------|
|
|
291
|
+
| `route-families.json` | `train` | Route family manifest |
|
|
292
|
+
| `train-report.json` | `train` | Training timings, coverage, LLM metrics |
|
|
230
293
|
| `plan.json` | `plan` | Coverage plan with gaps, decisions, metrics |
|
|
231
294
|
| `ci-summary.md` | `plan` | Markdown for PR comments |
|
|
232
295
|
| `metrics.jsonl` | `plan` | Append-only run metrics |
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"ai_enrichment.d.ts","sourceRoot":"","sources":["../../src/engine/ai_enrichment.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAC,WAAW,EAAC,MAAM,0BAA0B,CAAC;AAC1D,OAAO,KAAK,EAAC,YAAY,EAAmB,iBAAiB,EAAC,MAAM,oBAAoB,CAAC;AACzF,OAAO,KAAK,EAAC,eAAe,EAAC,MAAM,gCAAgC,CAAC;AAGpE,MAAM,WAAW,eAAe;IAC5B,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,eAAe,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,cAAc,EAAE,MAAM,CAAC;IACvB,eAAe,EAAE,MAAM,EAAE,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,kBAAkB,EAAE,MAAM,EAAE,CAAC;IAC7B,WAAW,EAAE,MAAM,EAAE,CAAC;CACzB;AAED,MAAM,WAAW,kBAAkB;IAC/B,gBAAgB,EAAE,eAAe,EAAE,CAAC;IACpC,mBAAmB,EAAE,KAAK,CAAC;QAAC,IAAI,EAAE,MAAM,CAAC;QAAC,aAAa,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAC,CAAC,CAAC;IAClF,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,YAAY,EAAE,MAAM,CAAC;IACrB,UAAU,EAAE;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAC,CAAC;CAC/C;AAED,MAAM,WAAW,mBAAmB;IAChC,mBAAmB,EAAE,YAAY,CAAC;IAClC,KAAK,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAC3B,QAAQ,EAAE,WAAW,CAAC;IACtB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,WAAW,CAAC,EAAE,iBAAiB,EAAE,CAAC;IAClC,eAAe,CAAC,EAAE,MAAM,CAAC;CAC5B;
|
|
1
|
+
{"version":3,"file":"ai_enrichment.d.ts","sourceRoot":"","sources":["../../src/engine/ai_enrichment.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAC,WAAW,EAAC,MAAM,0BAA0B,CAAC;AAC1D,OAAO,KAAK,EAAC,YAAY,EAAmB,iBAAiB,EAAC,MAAM,oBAAoB,CAAC;AACzF,OAAO,KAAK,EAAC,eAAe,EAAC,MAAM,gCAAgC,CAAC;AAGpE,MAAM,WAAW,eAAe;IAC5B,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,eAAe,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,cAAc,EAAE,MAAM,CAAC;IACvB,eAAe,EAAE,MAAM,EAAE,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,kBAAkB,EAAE,MAAM,EAAE,CAAC;IAC7B,WAAW,EAAE,MAAM,EAAE,CAAC;CACzB;AAED,MAAM,WAAW,kBAAkB;IAC/B,gBAAgB,EAAE,eAAe,EAAE,CAAC;IACpC,mBAAmB,EAAE,KAAK,CAAC;QAAC,IAAI,EAAE,MAAM,CAAC;QAAC,aAAa,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAC,CAAC,CAAC;IAClF,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,YAAY,EAAE,MAAM,CAAC;IACrB,UAAU,EAAE;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAC,CAAC;CAC/C;AAED,MAAM,WAAW,mBAAmB;IAChC,mBAAmB,EAAE,YAAY,CAAC;IAClC,KAAK,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAC3B,QAAQ,EAAE,WAAW,CAAC;IACtB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,WAAW,CAAC,EAAE,iBAAiB,EAAE,CAAC;IAClC,eAAe,CAAC,EAAE,MAAM,CAAC;CAC5B;AAmLD;;;GAGG;AACH,wBAAsB,kBAAkB,CAAC,OAAO,EAAE,mBAAmB,GAAG,OAAO,CAAC,kBAAkB,CAAC,CAkIlG"}
|
|
@@ -89,8 +89,9 @@ function buildPrompt(options) {
|
|
|
89
89
|
lines.push('');
|
|
90
90
|
lines.push('Rules for missingScenarios:');
|
|
91
91
|
lines.push('- Cross-reference the scenario titles in Existing Test Coverage. If a scenario already exists that covers the behavior, do NOT suggest it — instead list it in coveredBy.');
|
|
92
|
-
lines.push('- For coverage=uncovered: list
|
|
92
|
+
lines.push('- For coverage=uncovered: list ONLY scenarios that test the SPECIFIC behavior change in this diff — NOT generic scenarios for the entire feature. A permission check fix needs permission-denial test scenarios, not general CRUD tests.');
|
|
93
93
|
lines.push('- For coverage=covered or coverage=partial: ONLY list scenarios introduced by THIS diff that have NO matching scenario in existing coverage. If the diff adds no new user-visible behavior, return []. Do not pad with generic scenarios.');
|
|
94
|
+
lines.push('- If the source code diff is trivial (single-line fix, type change, field rename, or the PR is primarily adding tests), return missingScenarios=[] and explain in reasons that no new user-visible behavior was introduced.');
|
|
94
95
|
lines.push('');
|
|
95
96
|
lines.push(JSON.stringify({
|
|
96
97
|
impactedFlows: [
|
|
@@ -99,10 +100,10 @@ function buildPrompt(options) {
|
|
|
99
100
|
name: '<human-readable flow name>',
|
|
100
101
|
priority: 'P0|P1|P2',
|
|
101
102
|
reasons: [
|
|
102
|
-
'<EXACTLY 1-2 sentences
|
|
103
|
+
'<EXACTLY 1-2 sentences explaining what SPECIFICALLY changed for users in THIS diff. Reference the actual behavior modification — NOT a general description of what the feature does. BAD: "Users can create and view posts." GOOD: "Users editing posts now require create_post permission, which may block edits for restricted roles.">',
|
|
103
104
|
],
|
|
104
105
|
coveredBy: ['<spec file paths that cover this flow>'],
|
|
105
|
-
missingScenarios: ['<concrete scenario title for a
|
|
106
|
+
missingScenarios: ['<concrete scenario title for a NEW or CHANGED behavior in THIS diff. Must be SPECIFIC to the code change — never generic feature tests. BAD: "User can create a new post". GOOD: "User without create_post permission sees error when editing a post". If the diff is trivial (typo, field rename, test-only) return [].>'],
|
|
106
107
|
},
|
|
107
108
|
],
|
|
108
109
|
unboundFileAnalysis: [
|
|
@@ -17,12 +17,19 @@ export interface ImpactedFeature {
|
|
|
17
17
|
userFlows: string[];
|
|
18
18
|
coverageStatus: CoverageStatus;
|
|
19
19
|
}
|
|
20
|
+
export type PrTestFileType = 'playwright' | 'cypress' | 'unit' | 'snapshot';
|
|
21
|
+
export interface PrTestFile {
|
|
22
|
+
file: string;
|
|
23
|
+
type: PrTestFileType;
|
|
24
|
+
}
|
|
20
25
|
export interface ImpactResult {
|
|
21
26
|
changedFiles: string[];
|
|
22
27
|
expandedFiles: string[];
|
|
23
28
|
impactedFeatures: ImpactedFeature[];
|
|
24
29
|
unboundFiles: string[];
|
|
25
30
|
warnings: string[];
|
|
31
|
+
/** Test files that were in the original PR changeset but filtered from analysis. */
|
|
32
|
+
prIncludedTestFiles: PrTestFile[];
|
|
26
33
|
}
|
|
27
34
|
export interface ImpactEngineOptions {
|
|
28
35
|
testsRoot: string;
|
|
@@ -38,6 +45,11 @@ export declare function extractScenarios(filePath: string, framework: 'playwrigh
|
|
|
38
45
|
export declare function analyzeImpact(changedFiles: string[], options: ImpactEngineOptions): ImpactResult;
|
|
39
46
|
/**
|
|
40
47
|
* Get gaps: P0/P1 features with 'uncovered' status.
|
|
48
|
+
*
|
|
49
|
+
* Suppresses family-level (generic) gaps when ALL their changed files are
|
|
50
|
+
* already covered by feature-level (specific) matches in other families.
|
|
51
|
+
* This prevents double-counting when a file like `policies.tsx` matches both
|
|
52
|
+
* a generic family (`config`) and a specific feature (`system_console/permissions`).
|
|
41
53
|
*/
|
|
42
54
|
export declare function getGaps(result: ImpactResult): ImpactedFeature[];
|
|
43
55
|
/**
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"impact_engine.d.ts","sourceRoot":"","sources":["../../src/engine/impact_engine.ts"],"names":[],"mappings":"AAMA,OAAO,KAAK,EAGR,eAAe,EAClB,MAAM,gCAAgC,CAAC;AASxC,OAAO,KAAK,EAAC,mBAAmB,EAAC,MAAM,oBAAoB,CAAC;AAE5D,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,SAAS,GAAG,WAAW,CAAC;AAEjE,MAAM,WAAW,iBAAiB;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,SAAS,EAAE,MAAM,EAAE,CAAC;CACvB;AAED,MAAM,WAAW,eAAe;IAC5B,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,eAAe,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,eAAe,EAAE,MAAM,EAAE,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,qBAAqB,EAAE,iBAAiB,EAAE,CAAC;IAC3C,kBAAkB,EAAE,iBAAiB,EAAE,CAAC;IACxC,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,cAAc,EAAE,cAAc,CAAC;CAClC;AAED,MAAM,WAAW,YAAY;IACzB,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,gBAAgB,EAAE,eAAe,EAAE,CAAC;IACpC,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,QAAQ,EAAE,MAAM,EAAE,CAAC;
|
|
1
|
+
{"version":3,"file":"impact_engine.d.ts","sourceRoot":"","sources":["../../src/engine/impact_engine.ts"],"names":[],"mappings":"AAMA,OAAO,KAAK,EAGR,eAAe,EAClB,MAAM,gCAAgC,CAAC;AASxC,OAAO,KAAK,EAAC,mBAAmB,EAAC,MAAM,oBAAoB,CAAC;AAE5D,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,SAAS,GAAG,WAAW,CAAC;AAEjE,MAAM,WAAW,iBAAiB;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,SAAS,EAAE,MAAM,EAAE,CAAC;CACvB;AAED,MAAM,WAAW,eAAe;IAC5B,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,eAAe,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,eAAe,EAAE,MAAM,EAAE,CAAC;IAC1B,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,qBAAqB,EAAE,iBAAiB,EAAE,CAAC;IAC3C,kBAAkB,EAAE,iBAAiB,EAAE,CAAC;IACxC,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,cAAc,EAAE,cAAc,CAAC;CAClC;AAED,MAAM,MAAM,cAAc,GAAG,YAAY,GAAG,SAAS,GAAG,MAAM,GAAG,UAAU,CAAC;AAE5E,MAAM,WAAW,UAAU;IACvB,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,cAAc,CAAC;CACxB;AAED,MAAM,WAAW,YAAY;IACzB,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,gBAAgB,EAAE,eAAe,EAAE,CAAC;IACpC,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,oFAAoF;IACpF,mBAAmB,EAAE,UAAU,EAAE,CAAC;CACrC;AAED,MAAM,WAAW,mBAAmB;IAChC,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,aAAa,CAAC,EAAE,mBAAmB,CAAC;IACpC,aAAa,CAAC,EAAE,MAAM,EAAE,CAAC;CAC5B;AA+CD;;;GAGG;AACH,wBAAgB,gBAAgB,CAAC,QAAQ,EAAE,MAAM,EAAE,SAAS,EAAE,YAAY,GAAG,SAAS,GAAG,MAAM,EAAE,CAgBhG;AA0GD,wBAAgB,aAAa,CACzB,YAAY,EAAE,MAAM,EAAE,EACtB,OAAO,EAAE,mBAAmB,GAC7B,YAAY,CAoFd;AAYD;;;;;;;GAOG;AACH,wBAAgB,OAAO,CAAC,MAAM,EAAE,YAAY,GAAG,eAAe,EAAE,CAuB/D;AAED;;GAEG;AACH,wBAAgB,cAAc,CAAC,MAAM,EAAE,YAAY,GAAG,eAAe,EAAE,CAItE"}
|
|
@@ -150,16 +150,39 @@ function groupBindings(fileBindings) {
|
|
|
150
150
|
function isTestFile(file) {
|
|
151
151
|
const normalized = file.replace(/\\/g, '/');
|
|
152
152
|
return /\.(spec|test)\.(ts|tsx|js|jsx)$/.test(normalized) ||
|
|
153
|
+
/\.snap$/.test(normalized) ||
|
|
153
154
|
/_test\.go$/.test(normalized) ||
|
|
154
155
|
normalized.includes('__tests__/') ||
|
|
156
|
+
normalized.includes('__snapshots__/') ||
|
|
155
157
|
normalized.includes('/tests/') ||
|
|
156
158
|
normalized.includes('/test/');
|
|
157
159
|
}
|
|
160
|
+
/** Classify filtered test files by type for downstream decision-making. */
|
|
161
|
+
function classifyPrTestFiles(allFiles, sourceFiles) {
|
|
162
|
+
const sourceSet = new Set(sourceFiles);
|
|
163
|
+
return allFiles
|
|
164
|
+
.filter((f) => !sourceSet.has(f))
|
|
165
|
+
.map((f) => {
|
|
166
|
+
const n = f.replace(/\\/g, '/');
|
|
167
|
+
if (/\.snap$/.test(n) || n.includes('__snapshots__/')) {
|
|
168
|
+
return { file: f, type: 'snapshot' };
|
|
169
|
+
}
|
|
170
|
+
if (/\.spec\.(ts|tsx|js|jsx)$/.test(n)) {
|
|
171
|
+
return { file: f, type: 'playwright' };
|
|
172
|
+
}
|
|
173
|
+
if (n.includes('/cypress/') && /\.(js|ts)$/.test(n)) {
|
|
174
|
+
return { file: f, type: 'cypress' };
|
|
175
|
+
}
|
|
176
|
+
return { file: f, type: 'unit' };
|
|
177
|
+
});
|
|
178
|
+
}
|
|
158
179
|
function analyzeImpact(changedFiles, options) {
|
|
159
180
|
const { testsRoot, routeFamilies } = options;
|
|
160
181
|
const warnings = [];
|
|
161
|
-
//
|
|
182
|
+
// Partition into source files and test files
|
|
183
|
+
const allOriginalFiles = [...changedFiles];
|
|
162
184
|
changedFiles = changedFiles.filter((f) => !isTestFile(f));
|
|
185
|
+
const prIncludedTestFiles = classifyPrTestFiles(allOriginalFiles, changedFiles);
|
|
163
186
|
// Load manifest
|
|
164
187
|
const manifest = (0, route_families_js_1.loadRouteFamilyManifest)(testsRoot, routeFamilies);
|
|
165
188
|
if (!manifest) {
|
|
@@ -169,6 +192,7 @@ function analyzeImpact(changedFiles, options) {
|
|
|
169
192
|
impactedFeatures: [],
|
|
170
193
|
unboundFiles: [...changedFiles],
|
|
171
194
|
warnings: ['Route family manifest not found. All files are unbound.'],
|
|
195
|
+
prIncludedTestFiles,
|
|
172
196
|
};
|
|
173
197
|
}
|
|
174
198
|
// Combine original + expanded files
|
|
@@ -222,6 +246,7 @@ function analyzeImpact(changedFiles, options) {
|
|
|
222
246
|
impactedFeatures,
|
|
223
247
|
unboundFiles,
|
|
224
248
|
warnings,
|
|
249
|
+
prIncludedTestFiles,
|
|
225
250
|
};
|
|
226
251
|
}
|
|
227
252
|
function inferCypressRoot(testsRoot) {
|
|
@@ -235,9 +260,34 @@ function inferCypressRoot(testsRoot) {
|
|
|
235
260
|
}
|
|
236
261
|
/**
|
|
237
262
|
* Get gaps: P0/P1 features with 'uncovered' status.
|
|
263
|
+
*
|
|
264
|
+
* Suppresses family-level (generic) gaps when ALL their changed files are
|
|
265
|
+
* already covered by feature-level (specific) matches in other families.
|
|
266
|
+
* This prevents double-counting when a file like `policies.tsx` matches both
|
|
267
|
+
* a generic family (`config`) and a specific feature (`system_console/permissions`).
|
|
238
268
|
*/
|
|
239
269
|
function getGaps(result) {
|
|
240
|
-
|
|
270
|
+
// Collect files that are covered via feature-level matches (more specific)
|
|
271
|
+
const filesCoveredByFeatures = new Set();
|
|
272
|
+
for (const f of result.impactedFeatures) {
|
|
273
|
+
if (f.featureId && f.coverageStatus !== 'uncovered') {
|
|
274
|
+
for (const file of f.changedFiles) {
|
|
275
|
+
filesCoveredByFeatures.add(file);
|
|
276
|
+
}
|
|
277
|
+
}
|
|
278
|
+
}
|
|
279
|
+
return result.impactedFeatures.filter((f) => {
|
|
280
|
+
if (f.priority !== 'P0' && f.priority !== 'P1')
|
|
281
|
+
return false;
|
|
282
|
+
if (f.coverageStatus !== 'uncovered')
|
|
283
|
+
return false;
|
|
284
|
+
// Only suppress FAMILY-level gaps (no featureId = generic match).
|
|
285
|
+
// If it's a feature-level gap, keep it — it's specific and intentional.
|
|
286
|
+
if (!f.featureId && f.changedFiles.every((file) => filesCoveredByFeatures.has(file))) {
|
|
287
|
+
return false;
|
|
288
|
+
}
|
|
289
|
+
return true;
|
|
290
|
+
});
|
|
241
291
|
}
|
|
242
292
|
/**
|
|
243
293
|
* Get partial gaps: P0/P1 features with 'partial' status (advisory).
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"plan_builder.d.ts","sourceRoot":"","sources":["../../src/engine/plan_builder.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAC,YAAY,EAAC,MAAM,oBAAoB,CAAC;AAErD,OAAO,KAAK,EAAC,YAAY,EAAkB,MAAM,oBAAoB,CAAC;AAEtE,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,oBAAoB,CAAC;AAC3D,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,sBAAsB,CAAC;AAG7D,OAAO,KAAK,EACR,UAAU,EACV,SAAS,EACT,kBAAkB,EAIrB,MAAM,kBAAkB,CAAC;AAE1B,YAAY,EAAC,UAAU,EAAE,SAAS,EAAE,kBAAkB,EAAC,CAAC;
|
|
1
|
+
{"version":3,"file":"plan_builder.d.ts","sourceRoot":"","sources":["../../src/engine/plan_builder.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAC,YAAY,EAAC,MAAM,oBAAoB,CAAC;AAErD,OAAO,KAAK,EAAC,YAAY,EAAkB,MAAM,oBAAoB,CAAC;AAEtE,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,oBAAoB,CAAC;AAC3D,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,sBAAsB,CAAC;AAG7D,OAAO,KAAK,EACR,UAAU,EACV,SAAS,EACT,kBAAkB,EAIrB,MAAM,kBAAkB,CAAC;AAE1B,YAAY,EAAC,UAAU,EAAE,SAAS,EAAE,kBAAkB,EAAC,CAAC;AAgQxD,wBAAgB,mBAAmB,CAC/B,MAAM,EAAE,YAAY,EACpB,cAAc,CAAC,EAAE,OAAO,CAAC,YAAY,CAAC,EACtC,YAAY,CAAC,EAAE,kBAAkB,EACjC,kBAAkB,CAAC,EAAE,kBAAkB,GACxC,UAAU,CAyKZ;AAED,wBAAgB,eAAe,CAAC,OAAO,EAAE,MAAM,EAAE,IAAI,EAAE,UAAU,GAAG,MAAM,CAMzE;AAED,wBAAgB,uBAAuB,CAAC,IAAI,EAAE,UAAU,GAAG,MAAM,CA4IhE;AAED,wBAAgB,cAAc,CAAC,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,YAAY,SAAiC,GAAG,MAAM,CAMvH"}
|
|
@@ -114,6 +114,16 @@ function pickRunSet(impact, confidence, policy) {
|
|
|
114
114
|
function buildDecision(impact, runSet, confidence, policy) {
|
|
115
115
|
const gaps = (0, impact_engine_js_1.getGaps)(impact);
|
|
116
116
|
if (gaps.length > 0) {
|
|
117
|
+
// Check if PR already includes E2E test files that likely cover the gaps
|
|
118
|
+
const prE2ESpecCount = (impact.prIncludedTestFiles ?? [])
|
|
119
|
+
.filter((t) => t.type === 'playwright' || t.type === 'cypress').length;
|
|
120
|
+
if (prE2ESpecCount > 0) {
|
|
121
|
+
return {
|
|
122
|
+
action: 'run-now',
|
|
123
|
+
title: 'Run now',
|
|
124
|
+
summary: `Detected ${gaps.length} coverage gap(s), but the PR includes ${prE2ESpecCount} E2E test file(s). Verify the new tests cover impacted flows.`,
|
|
125
|
+
};
|
|
126
|
+
}
|
|
117
127
|
return {
|
|
118
128
|
action: 'must-add-tests',
|
|
119
129
|
title: 'Must add tests',
|
|
@@ -251,8 +261,19 @@ function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adaptiveThres
|
|
|
251
261
|
? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
|
|
252
262
|
: aiFeatureByFamilyId.get(f.familyId);
|
|
253
263
|
const baseReasons = [`No E2E tests found for ${label}`];
|
|
254
|
-
|
|
255
|
-
|
|
264
|
+
let aiReasonsList = [];
|
|
265
|
+
if (aiFeature) {
|
|
266
|
+
if (aiFeature.aiReasons.length > 0) {
|
|
267
|
+
aiReasonsList = aiFeature.aiReasons.slice(0, 2);
|
|
268
|
+
}
|
|
269
|
+
else {
|
|
270
|
+
// Fallback: LLM returned scenarios but no reasons — synthesize a description
|
|
271
|
+
const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
|
|
272
|
+
aiReasonsList = [`Changes to ${fileHint} affect the ${label} feature, which currently lacks E2E coverage.`];
|
|
273
|
+
}
|
|
274
|
+
}
|
|
275
|
+
const reasons = aiReasonsList.length > 0
|
|
276
|
+
? [...baseReasons, ...aiReasonsList]
|
|
256
277
|
: baseReasons;
|
|
257
278
|
const missingScenarios = aiFeature && aiFeature.aiMissingScenarios.length > 0
|
|
258
279
|
? aiFeature.aiMissingScenarios
|
|
@@ -274,8 +295,18 @@ function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adaptiveThres
|
|
|
274
295
|
? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
|
|
275
296
|
: aiFeatureByFamilyId.get(f.familyId);
|
|
276
297
|
const baseReasons = [`${label} is covered by Cypress only — consider adding Playwright tests`];
|
|
277
|
-
|
|
278
|
-
|
|
298
|
+
let partialAiReasons = [];
|
|
299
|
+
if (aiFeature) {
|
|
300
|
+
if (aiFeature.aiReasons.length > 0) {
|
|
301
|
+
partialAiReasons = aiFeature.aiReasons.slice(0, 2);
|
|
302
|
+
}
|
|
303
|
+
else {
|
|
304
|
+
const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
|
|
305
|
+
partialAiReasons = [`Changes to ${fileHint} affect the ${label} feature, which has Cypress but no Playwright coverage.`];
|
|
306
|
+
}
|
|
307
|
+
}
|
|
308
|
+
const reasons = partialAiReasons.length > 0
|
|
309
|
+
? [...baseReasons, ...partialAiReasons]
|
|
279
310
|
: baseReasons;
|
|
280
311
|
gapDetails.push({
|
|
281
312
|
id: label,
|
|
@@ -360,6 +391,7 @@ function renderCiSummaryMarkdown(plan) {
|
|
|
360
391
|
const lines = [];
|
|
361
392
|
const { uncoveredP0P1Flows, changedFiles, impactedFlows, coveredFlows: coveredCount, partialFlows: partialCount, unboundFiles: unboundCount } = plan.metrics;
|
|
362
393
|
const mustAddTests = plan.decision.action === 'must-add-tests';
|
|
394
|
+
const hasGapsButPrHasSpecs = !mustAddTests && plan.gapDetails.filter((g) => !g.name.includes('(partial)')).length > 0;
|
|
363
395
|
const flowsWithAdvisory = plan.coveredFlows.filter((f) => f.advisoryScenarios && f.advisoryScenarios.length > 0);
|
|
364
396
|
const cleanFlows = plan.coveredFlows.filter((f) => !f.advisoryScenarios || f.advisoryScenarios.length === 0);
|
|
365
397
|
const statusEmoji = mustAddTests ? '🔴' : plan.decision.action === 'safe-to-merge' ? '🟢' : '🟡';
|
|
@@ -411,6 +443,24 @@ function renderCiSummaryMarkdown(plan) {
|
|
|
411
443
|
}
|
|
412
444
|
}
|
|
413
445
|
}
|
|
446
|
+
// ── Informational gaps (PR includes E2E specs) ─────────────────────────────
|
|
447
|
+
if (hasGapsButPrHasSpecs) {
|
|
448
|
+
const infoGaps = plan.gapDetails.filter((g) => !g.name.includes('(partial)'));
|
|
449
|
+
lines.push('');
|
|
450
|
+
lines.push(`### ℹ️ Coverage gaps detected (PR includes E2E tests)`);
|
|
451
|
+
lines.push('');
|
|
452
|
+
lines.push('> The PR adds E2E test files. Verify they cover these flows:');
|
|
453
|
+
lines.push('');
|
|
454
|
+
for (const gap of infoGaps) {
|
|
455
|
+
const aiLabel = gap.source === 'ai+deterministic' ? ' ✦ AI-enriched' : '';
|
|
456
|
+
lines.push(`- **${gap.name}** · ${gap.priority}${aiLabel}`);
|
|
457
|
+
const aiReasons = gap.reasons.slice(1);
|
|
458
|
+
if (aiReasons.length > 0) {
|
|
459
|
+
lines.push(` ${aiReasons.join(' ')}`);
|
|
460
|
+
}
|
|
461
|
+
}
|
|
462
|
+
lines.push('');
|
|
463
|
+
}
|
|
414
464
|
// ── Advisory: covered flows with new behavior ─────────────────────────────
|
|
415
465
|
if (flowsWithAdvisory.length > 0) {
|
|
416
466
|
lines.push('');
|
|
@@ -86,8 +86,9 @@ function buildPrompt(options) {
|
|
|
86
86
|
lines.push('');
|
|
87
87
|
lines.push('Rules for missingScenarios:');
|
|
88
88
|
lines.push('- Cross-reference the scenario titles in Existing Test Coverage. If a scenario already exists that covers the behavior, do NOT suggest it — instead list it in coveredBy.');
|
|
89
|
-
lines.push('- For coverage=uncovered: list
|
|
89
|
+
lines.push('- For coverage=uncovered: list ONLY scenarios that test the SPECIFIC behavior change in this diff — NOT generic scenarios for the entire feature. A permission check fix needs permission-denial test scenarios, not general CRUD tests.');
|
|
90
90
|
lines.push('- For coverage=covered or coverage=partial: ONLY list scenarios introduced by THIS diff that have NO matching scenario in existing coverage. If the diff adds no new user-visible behavior, return []. Do not pad with generic scenarios.');
|
|
91
|
+
lines.push('- If the source code diff is trivial (single-line fix, type change, field rename, or the PR is primarily adding tests), return missingScenarios=[] and explain in reasons that no new user-visible behavior was introduced.');
|
|
91
92
|
lines.push('');
|
|
92
93
|
lines.push(JSON.stringify({
|
|
93
94
|
impactedFlows: [
|
|
@@ -96,10 +97,10 @@ function buildPrompt(options) {
|
|
|
96
97
|
name: '<human-readable flow name>',
|
|
97
98
|
priority: 'P0|P1|P2',
|
|
98
99
|
reasons: [
|
|
99
|
-
'<EXACTLY 1-2 sentences
|
|
100
|
+
'<EXACTLY 1-2 sentences explaining what SPECIFICALLY changed for users in THIS diff. Reference the actual behavior modification — NOT a general description of what the feature does. BAD: "Users can create and view posts." GOOD: "Users editing posts now require create_post permission, which may block edits for restricted roles.">',
|
|
100
101
|
],
|
|
101
102
|
coveredBy: ['<spec file paths that cover this flow>'],
|
|
102
|
-
missingScenarios: ['<concrete scenario title for a
|
|
103
|
+
missingScenarios: ['<concrete scenario title for a NEW or CHANGED behavior in THIS diff. Must be SPECIFIC to the code change — never generic feature tests. BAD: "User can create a new post". GOOD: "User without create_post permission sees error when editing a post". If the diff is trivial (typo, field rename, test-only) return [].>'],
|
|
103
104
|
},
|
|
104
105
|
],
|
|
105
106
|
unboundFileAnalysis: [
|
|
@@ -144,16 +144,39 @@ function groupBindings(fileBindings) {
|
|
|
144
144
|
function isTestFile(file) {
|
|
145
145
|
const normalized = file.replace(/\\/g, '/');
|
|
146
146
|
return /\.(spec|test)\.(ts|tsx|js|jsx)$/.test(normalized) ||
|
|
147
|
+
/\.snap$/.test(normalized) ||
|
|
147
148
|
/_test\.go$/.test(normalized) ||
|
|
148
149
|
normalized.includes('__tests__/') ||
|
|
150
|
+
normalized.includes('__snapshots__/') ||
|
|
149
151
|
normalized.includes('/tests/') ||
|
|
150
152
|
normalized.includes('/test/');
|
|
151
153
|
}
|
|
154
|
+
/** Classify filtered test files by type for downstream decision-making. */
|
|
155
|
+
function classifyPrTestFiles(allFiles, sourceFiles) {
|
|
156
|
+
const sourceSet = new Set(sourceFiles);
|
|
157
|
+
return allFiles
|
|
158
|
+
.filter((f) => !sourceSet.has(f))
|
|
159
|
+
.map((f) => {
|
|
160
|
+
const n = f.replace(/\\/g, '/');
|
|
161
|
+
if (/\.snap$/.test(n) || n.includes('__snapshots__/')) {
|
|
162
|
+
return { file: f, type: 'snapshot' };
|
|
163
|
+
}
|
|
164
|
+
if (/\.spec\.(ts|tsx|js|jsx)$/.test(n)) {
|
|
165
|
+
return { file: f, type: 'playwright' };
|
|
166
|
+
}
|
|
167
|
+
if (n.includes('/cypress/') && /\.(js|ts)$/.test(n)) {
|
|
168
|
+
return { file: f, type: 'cypress' };
|
|
169
|
+
}
|
|
170
|
+
return { file: f, type: 'unit' };
|
|
171
|
+
});
|
|
172
|
+
}
|
|
152
173
|
export function analyzeImpact(changedFiles, options) {
|
|
153
174
|
const { testsRoot, routeFamilies } = options;
|
|
154
175
|
const warnings = [];
|
|
155
|
-
//
|
|
176
|
+
// Partition into source files and test files
|
|
177
|
+
const allOriginalFiles = [...changedFiles];
|
|
156
178
|
changedFiles = changedFiles.filter((f) => !isTestFile(f));
|
|
179
|
+
const prIncludedTestFiles = classifyPrTestFiles(allOriginalFiles, changedFiles);
|
|
157
180
|
// Load manifest
|
|
158
181
|
const manifest = loadRouteFamilyManifest(testsRoot, routeFamilies);
|
|
159
182
|
if (!manifest) {
|
|
@@ -163,6 +186,7 @@ export function analyzeImpact(changedFiles, options) {
|
|
|
163
186
|
impactedFeatures: [],
|
|
164
187
|
unboundFiles: [...changedFiles],
|
|
165
188
|
warnings: ['Route family manifest not found. All files are unbound.'],
|
|
189
|
+
prIncludedTestFiles,
|
|
166
190
|
};
|
|
167
191
|
}
|
|
168
192
|
// Combine original + expanded files
|
|
@@ -216,6 +240,7 @@ export function analyzeImpact(changedFiles, options) {
|
|
|
216
240
|
impactedFeatures,
|
|
217
241
|
unboundFiles,
|
|
218
242
|
warnings,
|
|
243
|
+
prIncludedTestFiles,
|
|
219
244
|
};
|
|
220
245
|
}
|
|
221
246
|
function inferCypressRoot(testsRoot) {
|
|
@@ -229,9 +254,34 @@ function inferCypressRoot(testsRoot) {
|
|
|
229
254
|
}
|
|
230
255
|
/**
|
|
231
256
|
* Get gaps: P0/P1 features with 'uncovered' status.
|
|
257
|
+
*
|
|
258
|
+
* Suppresses family-level (generic) gaps when ALL their changed files are
|
|
259
|
+
* already covered by feature-level (specific) matches in other families.
|
|
260
|
+
* This prevents double-counting when a file like `policies.tsx` matches both
|
|
261
|
+
* a generic family (`config`) and a specific feature (`system_console/permissions`).
|
|
232
262
|
*/
|
|
233
263
|
export function getGaps(result) {
|
|
234
|
-
|
|
264
|
+
// Collect files that are covered via feature-level matches (more specific)
|
|
265
|
+
const filesCoveredByFeatures = new Set();
|
|
266
|
+
for (const f of result.impactedFeatures) {
|
|
267
|
+
if (f.featureId && f.coverageStatus !== 'uncovered') {
|
|
268
|
+
for (const file of f.changedFiles) {
|
|
269
|
+
filesCoveredByFeatures.add(file);
|
|
270
|
+
}
|
|
271
|
+
}
|
|
272
|
+
}
|
|
273
|
+
return result.impactedFeatures.filter((f) => {
|
|
274
|
+
if (f.priority !== 'P0' && f.priority !== 'P1')
|
|
275
|
+
return false;
|
|
276
|
+
if (f.coverageStatus !== 'uncovered')
|
|
277
|
+
return false;
|
|
278
|
+
// Only suppress FAMILY-level gaps (no featureId = generic match).
|
|
279
|
+
// If it's a feature-level gap, keep it — it's specific and intentional.
|
|
280
|
+
if (!f.featureId && f.changedFiles.every((file) => filesCoveredByFeatures.has(file))) {
|
|
281
|
+
return false;
|
|
282
|
+
}
|
|
283
|
+
return true;
|
|
284
|
+
});
|
|
235
285
|
}
|
|
236
286
|
/**
|
|
237
287
|
* Get partial gaps: P0/P1 features with 'partial' status (advisory).
|
|
@@ -108,6 +108,16 @@ function pickRunSet(impact, confidence, policy) {
|
|
|
108
108
|
function buildDecision(impact, runSet, confidence, policy) {
|
|
109
109
|
const gaps = getGaps(impact);
|
|
110
110
|
if (gaps.length > 0) {
|
|
111
|
+
// Check if PR already includes E2E test files that likely cover the gaps
|
|
112
|
+
const prE2ESpecCount = (impact.prIncludedTestFiles ?? [])
|
|
113
|
+
.filter((t) => t.type === 'playwright' || t.type === 'cypress').length;
|
|
114
|
+
if (prE2ESpecCount > 0) {
|
|
115
|
+
return {
|
|
116
|
+
action: 'run-now',
|
|
117
|
+
title: 'Run now',
|
|
118
|
+
summary: `Detected ${gaps.length} coverage gap(s), but the PR includes ${prE2ESpecCount} E2E test file(s). Verify the new tests cover impacted flows.`,
|
|
119
|
+
};
|
|
120
|
+
}
|
|
111
121
|
return {
|
|
112
122
|
action: 'must-add-tests',
|
|
113
123
|
title: 'Must add tests',
|
|
@@ -245,8 +255,19 @@ export function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adapti
|
|
|
245
255
|
? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
|
|
246
256
|
: aiFeatureByFamilyId.get(f.familyId);
|
|
247
257
|
const baseReasons = [`No E2E tests found for ${label}`];
|
|
248
|
-
|
|
249
|
-
|
|
258
|
+
let aiReasonsList = [];
|
|
259
|
+
if (aiFeature) {
|
|
260
|
+
if (aiFeature.aiReasons.length > 0) {
|
|
261
|
+
aiReasonsList = aiFeature.aiReasons.slice(0, 2);
|
|
262
|
+
}
|
|
263
|
+
else {
|
|
264
|
+
// Fallback: LLM returned scenarios but no reasons — synthesize a description
|
|
265
|
+
const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
|
|
266
|
+
aiReasonsList = [`Changes to ${fileHint} affect the ${label} feature, which currently lacks E2E coverage.`];
|
|
267
|
+
}
|
|
268
|
+
}
|
|
269
|
+
const reasons = aiReasonsList.length > 0
|
|
270
|
+
? [...baseReasons, ...aiReasonsList]
|
|
250
271
|
: baseReasons;
|
|
251
272
|
const missingScenarios = aiFeature && aiFeature.aiMissingScenarios.length > 0
|
|
252
273
|
? aiFeature.aiMissingScenarios
|
|
@@ -268,8 +289,18 @@ export function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adapti
|
|
|
268
289
|
? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
|
|
269
290
|
: aiFeatureByFamilyId.get(f.familyId);
|
|
270
291
|
const baseReasons = [`${label} is covered by Cypress only — consider adding Playwright tests`];
|
|
271
|
-
|
|
272
|
-
|
|
292
|
+
let partialAiReasons = [];
|
|
293
|
+
if (aiFeature) {
|
|
294
|
+
if (aiFeature.aiReasons.length > 0) {
|
|
295
|
+
partialAiReasons = aiFeature.aiReasons.slice(0, 2);
|
|
296
|
+
}
|
|
297
|
+
else {
|
|
298
|
+
const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
|
|
299
|
+
partialAiReasons = [`Changes to ${fileHint} affect the ${label} feature, which has Cypress but no Playwright coverage.`];
|
|
300
|
+
}
|
|
301
|
+
}
|
|
302
|
+
const reasons = partialAiReasons.length > 0
|
|
303
|
+
? [...baseReasons, ...partialAiReasons]
|
|
273
304
|
: baseReasons;
|
|
274
305
|
gapDetails.push({
|
|
275
306
|
id: label,
|
|
@@ -354,6 +385,7 @@ export function renderCiSummaryMarkdown(plan) {
|
|
|
354
385
|
const lines = [];
|
|
355
386
|
const { uncoveredP0P1Flows, changedFiles, impactedFlows, coveredFlows: coveredCount, partialFlows: partialCount, unboundFiles: unboundCount } = plan.metrics;
|
|
356
387
|
const mustAddTests = plan.decision.action === 'must-add-tests';
|
|
388
|
+
const hasGapsButPrHasSpecs = !mustAddTests && plan.gapDetails.filter((g) => !g.name.includes('(partial)')).length > 0;
|
|
357
389
|
const flowsWithAdvisory = plan.coveredFlows.filter((f) => f.advisoryScenarios && f.advisoryScenarios.length > 0);
|
|
358
390
|
const cleanFlows = plan.coveredFlows.filter((f) => !f.advisoryScenarios || f.advisoryScenarios.length === 0);
|
|
359
391
|
const statusEmoji = mustAddTests ? '🔴' : plan.decision.action === 'safe-to-merge' ? '🟢' : '🟡';
|
|
@@ -405,6 +437,24 @@ export function renderCiSummaryMarkdown(plan) {
|
|
|
405
437
|
}
|
|
406
438
|
}
|
|
407
439
|
}
|
|
440
|
+
// ── Informational gaps (PR includes E2E specs) ─────────────────────────────
|
|
441
|
+
if (hasGapsButPrHasSpecs) {
|
|
442
|
+
const infoGaps = plan.gapDetails.filter((g) => !g.name.includes('(partial)'));
|
|
443
|
+
lines.push('');
|
|
444
|
+
lines.push(`### ℹ️ Coverage gaps detected (PR includes E2E tests)`);
|
|
445
|
+
lines.push('');
|
|
446
|
+
lines.push('> The PR adds E2E test files. Verify they cover these flows:');
|
|
447
|
+
lines.push('');
|
|
448
|
+
for (const gap of infoGaps) {
|
|
449
|
+
const aiLabel = gap.source === 'ai+deterministic' ? ' ✦ AI-enriched' : '';
|
|
450
|
+
lines.push(`- **${gap.name}** · ${gap.priority}${aiLabel}`);
|
|
451
|
+
const aiReasons = gap.reasons.slice(1);
|
|
452
|
+
if (aiReasons.length > 0) {
|
|
453
|
+
lines.push(` ${aiReasons.join(' ')}`);
|
|
454
|
+
}
|
|
455
|
+
}
|
|
456
|
+
lines.push('');
|
|
457
|
+
}
|
|
408
458
|
// ── Advisory: covered flows with new behavior ─────────────────────────────
|
|
409
459
|
if (flowsWithAdvisory.length > 0) {
|
|
410
460
|
lines.push('');
|
package/dist/index.d.ts
CHANGED
|
@@ -21,7 +21,7 @@ export type { HybridConfig } from './provider_factory.js';
|
|
|
21
21
|
export { analyzeImpactDeterministic, recommendTestsDeterministic, handoffGeneratedTests, ingestTraceability, captureTraceability } from './api.js';
|
|
22
22
|
export type { AgentApiOptions, RecommendTestsV2Result, TraceabilityIngestApiOptions, TraceabilityCaptureApiOptions, } from './api.js';
|
|
23
23
|
export { analyzeImpact as analyzeImpactV2, getGaps, getPartialGaps } from './engine/impact_engine.js';
|
|
24
|
-
export type { ImpactResult, ImpactedFeature, CoverageStatus, ImpactEngineOptions, SpecWithScenarios } from './engine/impact_engine.js';
|
|
24
|
+
export type { ImpactResult, ImpactedFeature, CoverageStatus, ImpactEngineOptions, SpecWithScenarios, PrTestFile, PrTestFileType } from './engine/impact_engine.js';
|
|
25
25
|
export { extractScenarios } from './engine/impact_engine.js';
|
|
26
26
|
export { buildPlanFromImpact } from './engine/plan_builder.js';
|
|
27
27
|
export { appendFeedbackAndRecompute, readCalibration, readFlakyTests, getAdaptiveThresholds } from './agent/feedback.js';
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAGA;;;;;;;;;;;GAWG;AAGH,YAAY,EACR,WAAW,EACX,eAAe,EACf,UAAU,EACV,WAAW,EACX,UAAU,EACV,oBAAoB,EACpB,kBAAkB,EAClB,cAAc,EACd,eAAe,EACf,YAAY,EACZ,YAAY,EACZ,YAAY,GACf,MAAM,yBAAyB,CAAC;AAEjC,OAAO,EAAC,gBAAgB,EAAE,0BAA0B,EAAC,MAAM,yBAAyB,CAAC;AAGrF,OAAO,EAAC,iBAAiB,EAAE,mBAAmB,EAAC,MAAM,yBAAyB,CAAC;AAC/E,OAAO,EAAC,cAAc,EAAE,gBAAgB,EAAC,MAAM,sBAAsB,CAAC;AACtE,OAAO,EAAC,cAAc,EAAE,gBAAgB,EAAC,MAAM,sBAAsB,CAAC;AACtE,OAAO,EAAC,cAAc,EAAC,MAAM,sBAAsB,CAAC;AAGpD,OAAO,EAAC,kBAAkB,EAAE,qBAAqB,EAAC,MAAM,uBAAuB,CAAC;AAChF,YAAY,EAAC,YAAY,EAAC,MAAM,uBAAuB,CAAC;AAGxD,OAAO,EAAC,0BAA0B,EAAE,2BAA2B,EAAE,qBAAqB,EAAE,kBAAkB,EAAE,mBAAmB,EAAC,MAAM,UAAU,CAAC;AACjJ,YAAY,EACR,eAAe,EACf,sBAAsB,EACtB,4BAA4B,EAC5B,6BAA6B,GAChC,MAAM,UAAU,CAAC;AAGlB,OAAO,EAAC,aAAa,IAAI,eAAe,EAAE,OAAO,EAAE,cAAc,EAAC,MAAM,2BAA2B,CAAC;AACpG,YAAY,EAAC,YAAY,EAAE,eAAe,EAAE,cAAc,EAAE,mBAAmB,EAAE,iBAAiB,EAAC,MAAM,2BAA2B,CAAC;
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAGA;;;;;;;;;;;GAWG;AAGH,YAAY,EACR,WAAW,EACX,eAAe,EACf,UAAU,EACV,WAAW,EACX,UAAU,EACV,oBAAoB,EACpB,kBAAkB,EAClB,cAAc,EACd,eAAe,EACf,YAAY,EACZ,YAAY,EACZ,YAAY,GACf,MAAM,yBAAyB,CAAC;AAEjC,OAAO,EAAC,gBAAgB,EAAE,0BAA0B,EAAC,MAAM,yBAAyB,CAAC;AAGrF,OAAO,EAAC,iBAAiB,EAAE,mBAAmB,EAAC,MAAM,yBAAyB,CAAC;AAC/E,OAAO,EAAC,cAAc,EAAE,gBAAgB,EAAC,MAAM,sBAAsB,CAAC;AACtE,OAAO,EAAC,cAAc,EAAE,gBAAgB,EAAC,MAAM,sBAAsB,CAAC;AACtE,OAAO,EAAC,cAAc,EAAC,MAAM,sBAAsB,CAAC;AAGpD,OAAO,EAAC,kBAAkB,EAAE,qBAAqB,EAAC,MAAM,uBAAuB,CAAC;AAChF,YAAY,EAAC,YAAY,EAAC,MAAM,uBAAuB,CAAC;AAGxD,OAAO,EAAC,0BAA0B,EAAE,2BAA2B,EAAE,qBAAqB,EAAE,kBAAkB,EAAE,mBAAmB,EAAC,MAAM,UAAU,CAAC;AACjJ,YAAY,EACR,eAAe,EACf,sBAAsB,EACtB,4BAA4B,EAC5B,6BAA6B,GAChC,MAAM,UAAU,CAAC;AAGlB,OAAO,EAAC,aAAa,IAAI,eAAe,EAAE,OAAO,EAAE,cAAc,EAAC,MAAM,2BAA2B,CAAC;AACpG,YAAY,EAAC,YAAY,EAAE,eAAe,EAAE,cAAc,EAAE,mBAAmB,EAAE,iBAAiB,EAAE,UAAU,EAAE,cAAc,EAAC,MAAM,2BAA2B,CAAC;AACjK,OAAO,EAAC,gBAAgB,EAAC,MAAM,2BAA2B,CAAC;AAC3D,OAAO,EAAC,mBAAmB,EAAC,MAAM,0BAA0B,CAAC;AAC7D,OAAO,EAAC,0BAA0B,EAAE,eAAe,EAAE,cAAc,EAAE,qBAAqB,EAAC,MAAM,qBAAqB,CAAC;AACvH,YAAY,EAAC,2BAA2B,EAAE,kBAAkB,EAAE,YAAY,EAAE,kBAAkB,EAAC,MAAM,qBAAqB,CAAC;AAC3H,OAAO,EAAC,sBAAsB,EAAC,MAAM,oBAAoB,CAAC;AAC1D,YAAY,EAAC,6BAA6B,EAAE,4BAA4B,EAAC,MAAM,oBAAoB,CAAC;AACpG,OAAO,EAAC,uBAAuB,EAAC,MAAM,gCAAgC,CAAC;AACvE,YAAY,EAAC,yBAAyB,EAAE,wBAAwB,EAAE,uBAAuB,EAAC,MAAM,gCAAgC,CAAC;AACjI,OAAO,EAAC,wBAAwB,EAAC,MAAM,iCAAiC,CAAC;AACzE,YAAY,EAAC,0BAA0B,EAAE,yBAAyB,EAAC,MAAM,iCAAiC,CAAC;AAG3G,OAAO,EAAC,WAAW,EAAC,MAAM,4BAA4B,CAAC;AACvD,YAAY,EAAC,cAAc,EAAE,cAAc,EAAC,MAAM,4BAA4B,CAAC;AAC/E,YAAY,EAAC,YAAY,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,UAAU,EAAE,cAAc,EAAC,MAAM,+BAA+B,CAAC;AACrI,OAAO,EAAC,kBAAkB,EAAC,MAAM,iCAAiC,CAAC;AACnE,YAAY,EAAC,gBAAgB,EAAE,gBAAgB,EAAE,aAAa,EAAC,MAAM,iCAAiC,CAAC;AACvG,OAAO,EAAC,qBAAqB,EAAE,uBAAuB,EAAE,yBAAyB,EAAC,MAAM,yBAAyB,CAAC;AAClH,YAAY,EAAC,uBAAuB,EAAE,uBAAuB,EAAC,MAAM,yBAAyB,CAAC;AAC9F,OAAO,EAAC,YAAY,EAAE,cAAc,EAAE,kBAAkB,EAAE,kBAAkB,EAAC,MAAM,2BAA2B,CAAC;AAC/G,YAAY,EAAC,UAAU,EAAE,UAAU,EAAE,UAAU,EAAC,MAAM,2BAA2B,CAAC;AAClF,OAAO,EAAC,eAAe,EAAE,qBAAqB,EAAC,MAAM,mBAAmB,CAAC;AACzE,YAAY,EAAC,iBAAiB,EAAC,MAAM,mBAAmB,CAAC;AAGzD,OAAO,EAAC,uBAAuB,EAAE,mBAAmB,EAAE,4BAA4B,EAAE,qBAAqB,EAAE,sBAAsB,EAAC,MAAM,+BAA+B,CAAC;AACxK,YAAY,EAAC,WAAW,EAAE,YAAY,EAAE,mBAAmB,EAAE,WAAW,EAAE,eAAe,EAAC,MAAM,+BAA+B,CAAC;AAChI,OAAO,EAAC,eAAe,EAAE,qBAAqB,EAAC,MAAM,4BAA4B,CAAC;AAClF,YAAY,EAAC,iBAAiB,EAAE,iBAAiB,EAAC,MAAM,4BAA4B,CAAC;AACrF,OAAO,EAAC,cAAc,EAAE,iBAAiB,EAAC,MAAM,2BAA2B,CAAC;AAC5E,YAAY,EAAC,SAAS,EAAE,SAAS,EAAC,MAAM,2BAA2B,CAAC;AAGpE,YAAY,EAAC,UAAU,EAAE,YAAY,EAAE,YAAY,EAAE,OAAO,EAAE,WAAW,EAAC,MAAM,kBAAkB,CAAC;AACnG,YAAY,EAAC,UAAU,EAAC,MAAM,iBAAiB,CAAC;AAGhD,OAAO,EAAC,oBAAoB,EAAC,MAAM,qBAAqB,CAAC;AACzD,YAAY,EAAC,aAAa,EAAE,iBAAiB,EAAC,MAAM,qBAAqB,CAAC;AAC1E,YAAY,EAAC,aAAa,EAAE,aAAa,EAAE,cAAc,EAAE,mBAAmB,EAAE,WAAW,EAAC,MAAM,oBAAoB,CAAC;AAGvH,OAAO,EAAC,WAAW,EAAC,MAAM,uBAAuB,CAAC;AAClD,OAAO,EAAC,aAAa,EAAE,mBAAmB,EAAC,MAAM,sBAAsB,CAAC;AACxE,OAAO,EAAC,cAAc,EAAC,MAAM,wBAAwB,CAAC;AACtD,OAAO,EAAC,cAAc,EAAE,cAAc,EAAE,qBAAqB,EAAE,sBAAsB,EAAC,MAAM,yBAAyB,CAAC;AACtH,YAAY,EACR,UAAU,EAAE,aAAa,EAAE,cAAc,EAAE,aAAa,EACxD,gBAAgB,EAAE,gBAAgB,EAAE,gBAAgB,EAAE,WAAW,EAAE,YAAY,GAClF,MAAM,qBAAqB,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@yasserkhanorg/e2e-agents",
|
|
3
|
-
"version": "1.7.
|
|
3
|
+
"version": "1.7.4",
|
|
4
4
|
"description": "AI-powered E2E test impact analysis, generation, and healing. Analyzes code changes to identify affected Playwright tests, detects coverage gaps, and generates or repairs specs using pluggable LLM providers (Claude, OpenAI, Ollama). Includes MCP server, traceability, and CI/CD integration.",
|
|
5
5
|
"main": "dist/index.js",
|
|
6
6
|
"module": "dist/esm/index.js",
|