@yasserkhanorg/e2e-agents 1.7.1 → 1.7.3

package/README.md

@@ -56,7 +56,67 @@ npx e2e-ai-agents llm-health
 
 ## Route-Families Training
 
-Route-families map your source files to features, test directories, and user flows. They are the context that powers accurate impact analysis. The `train` command bootstraps and maintains this manifest.
+### What it produces
+
+The `train` command builds a **knowledge map** of your codebase — a single JSON file (`route-families.json`) that maps source files to features, test directories, and user flows. This is not ML training; no model is trained. It's building a structured manifest like:
+
+```json
+{
+  "id": "channels",
+  "routes": ["/{team}/channels/{channel}"],
+  "priority": "P0",
+  "webappPaths": ["src/components/channel_header/**"],
+  "serverPaths": ["server/channels/api4/channel*.go", "server/channels/app/channel*.go"],
+  "specDirs": ["specs/functional/channels/"],
+  "userFlows": ["Create channel", "Archive channel", "Search in channel"],
+  "components": ["ChannelHeader", "ChannelSidebar"]
+}
+```
+
+### Why the tool needs this
+
+When a PR changes `server/channels/app/channel.go`, the tool needs to answer: **"which E2E tests should I run?"** Without the manifest, it has no idea. With it:
+
+```
+channel.go changed
+  → belongs to "channels" family
+    → specs are in specs/functional/channels/
+      → run those tests
+      → flag if coverage is missing for the affected user flows
+```
+
+Every downstream command (`impact`, `plan`, `generate`, `heal`, `e2e-qa-agent`) reads this manifest to understand the codebase.
+
+### How scanning works
+
+The scanner uses 4 strategies to build the `file → family` mapping:
+
+1. **Directory matching** — `src/channels/` + `tests/channels/` share a name → channels family
+2. **Test-derived** — `specs/functional/channels/drafts/` exists with spec files → drafts family (even if source code is scattered across components/actions/reducers)
+3. **Server-derived** — `api4/channel.go` + `app/channel.go` + `store/channel_store.go` span 3 backend tiers → channel family (related files like `channel_bookmark.go` are grouped under the parent)
+4. **Name-matched** — `src/utils/channels.ts` or `server/public/model/channel.go` basename matches → add to channels family's paths
+
+### What LLM enrichment adds
+
+The scanner finds files. The LLM reads code samples and adds **semantic metadata** the scanner can't determine:
+- Accurate URL routes (`/{team}/channels/{channel}` instead of guessed `/channels`)
+- Priority classification (P0 critical user flow vs P2 nice-to-have)
+- Human-readable user flows ("Create channel", "Search messages")
+- React component and page object names
+
+This metadata makes impact analysis smarter — it can prioritize P0 flows and suggest specific test scenarios.
+
+### What validation does
+
+The `--validate` flag measures manifest accuracy against **real git history**. It's not training data — it's a quality check:
+
+```
+835 commits → 5105 changed files → 3223 bound to a family = 63% coverage
+```
+
+This tells you the manifest is complete enough. If coverage were 30%, impact analysis would be blind to most code changes.
+
+### Usage
 
 ```bash
 # Scan your codebase + LLM enrichment (default)
@@ -72,18 +132,19 @@ npx e2e-ai-agents train --path /path/to/project --validate --since HEAD~50
 npx e2e-ai-agents train --path /path/to/project --validate --since HEAD~20
 ```
 
-**Why LLM enrichment is on by default:** The manifest exists to give AI context for impact analysis, scenario suggestion, and bug detection. AI-generated context produces better AI reasoning downstream. Use `--no-enrich` for offline/free operation or to avoid sending code snippets to third-party LLM APIs.
+**Why LLM enrichment is on by default:** The manifest gives AI context for impact analysis, scenario suggestion, and bug detection. AI-generated context produces better AI reasoning downstream. Use `--no-enrich` for offline/free operation or to avoid sending code snippets to third-party LLM APIs.
 
-**Training loop:** Run `train` → review the generated `route-families.json` → run `train --validate` to check coverage % → fix gaps → repeat until 95%+.
+**Training loop:** Run `train` → review `route-families.json` → run `train --validate` to check coverage % → fix gaps → repeat.
 
-The `train` command:
-1. **Scans** your project structure (frontend `src/`, backend `server/`, test dirs)
-2. **Matches** source directories to test directories by name
-3. **Enriches** with LLM (priority, user flows, routes, components)
-4. **Merges** intelligently with any existing manifest (preserves human curation)
-5. **Validates** against git history to measure accuracy
+**Additional flags:**
+- `--verbose` / `-v` — DEBUG-level output with timing for each phase
+- `--json` — structured JSON log output (for CI pipelines)
+- `--server-path` — explicit path to backend server root
+- `--budget-usd` — max LLM spend (default: $0.50, max: $10)
 
-Output is written to `<testsRoot>/.e2e-ai-agents/route-families.json`.
+**Output:**
+- `<testsRoot>/.e2e-ai-agents/route-families.json` — the manifest
+- `<testsRoot>/.e2e-ai-agents/train-report.json` — timing data, family counts, coverage stats, LLM metrics
 
 ## Configuration
 
@@ -227,6 +288,8 @@ Schemas: [schemas/traceability-input.schema.json](schemas/traceability-input.sch
 
 | File | Written by | Purpose |
 |------|-----------|---------|
+| `route-families.json` | `train` | Route family manifest |
+| `train-report.json` | `train` | Training timings, coverage, LLM metrics |
 | `plan.json` | `plan` | Coverage plan with gaps, decisions, metrics |
 | `ci-summary.md` | `plan` | Markdown for PR comments |
 | `metrics.jsonl` | `plan` | Append-only run metrics |

package/dist/engine/plan_builder.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"plan_builder.d.ts","sourceRoot":"","sources":["../../src/engine/plan_builder.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAC,YAAY,EAAC,MAAM,oBAAoB,CAAC;AAErD,OAAO,KAAK,EAAC,YAAY,EAAkB,MAAM,oBAAoB,CAAC;AAEtE,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,oBAAoB,CAAC;AAC3D,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,sBAAsB,CAAC;AAG7D,OAAO,KAAK,EACR,UAAU,EACV,SAAS,EACT,kBAAkB,EAIrB,MAAM,kBAAkB,CAAC;AAE1B,YAAY,EAAC,UAAU,EAAE,SAAS,EAAE,kBAAkB,EAAC,CAAC;AAoPxD,wBAAgB,mBAAmB,CAC/B,MAAM,EAAE,YAAY,EACpB,cAAc,CAAC,EAAE,OAAO,CAAC,YAAY,CAAC,EACtC,YAAY,CAAC,EAAE,kBAAkB,EACjC,kBAAkB,CAAC,EAAE,kBAAkB,GACxC,UAAU,CAsJZ;AAED,wBAAgB,eAAe,CAAC,OAAO,EAAE,MAAM,EAAE,IAAI,EAAE,UAAU,GAAG,MAAM,CAMzE;AAED,wBAAgB,uBAAuB,CAAC,IAAI,EAAE,UAAU,GAAG,MAAM,CAwHhE;AAED,wBAAgB,cAAc,CAAC,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,YAAY,SAAiC,GAAG,MAAM,CAMvH"}
+{"version":3,"file":"plan_builder.d.ts","sourceRoot":"","sources":["../../src/engine/plan_builder.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAC,YAAY,EAAC,MAAM,oBAAoB,CAAC;AAErD,OAAO,KAAK,EAAC,YAAY,EAAkB,MAAM,oBAAoB,CAAC;AAEtE,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,oBAAoB,CAAC;AAC3D,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,sBAAsB,CAAC;AAG7D,OAAO,KAAK,EACR,UAAU,EACV,SAAS,EACT,kBAAkB,EAIrB,MAAM,kBAAkB,CAAC;AAE1B,YAAY,EAAC,UAAU,EAAE,SAAS,EAAE,kBAAkB,EAAC,CAAC;AAoPxD,wBAAgB,mBAAmB,CAC/B,MAAM,EAAE,YAAY,EACpB,cAAc,CAAC,EAAE,OAAO,CAAC,YAAY,CAAC,EACtC,YAAY,CAAC,EAAE,kBAAkB,EACjC,kBAAkB,CAAC,EAAE,kBAAkB,GACxC,UAAU,CAyKZ;AAED,wBAAgB,eAAe,CAAC,OAAO,EAAE,MAAM,EAAE,IAAI,EAAE,UAAU,GAAG,MAAM,CAMzE;AAED,wBAAgB,uBAAuB,CAAC,IAAI,EAAE,UAAU,GAAG,MAAM,CAwHhE;AAED,wBAAgB,cAAc,CAAC,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,YAAY,SAAiC,GAAG,MAAM,CAMvH"}

package/dist/engine/plan_builder.js

@@ -251,8 +251,19 @@ function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adaptiveThres
             ? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
             : aiFeatureByFamilyId.get(f.familyId);
         const baseReasons = [`No E2E tests found for ${label}`];
-        const reasons = aiFeature && aiFeature.aiReasons.length > 0
-            ? [...baseReasons, ...aiFeature.aiReasons.slice(0, 2)]
+        let aiReasonsList = [];
+        if (aiFeature) {
+            if (aiFeature.aiReasons.length > 0) {
+                aiReasonsList = aiFeature.aiReasons.slice(0, 2);
+            }
+            else {
+                // Fallback: LLM returned scenarios but no reasons — synthesize a description
+                const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
+                aiReasonsList = [`Changes to ${fileHint} affect the ${label} feature, which currently lacks E2E coverage.`];
+            }
+        }
+        const reasons = aiReasonsList.length > 0
+            ? [...baseReasons, ...aiReasonsList]
             : baseReasons;
         const missingScenarios = aiFeature && aiFeature.aiMissingScenarios.length > 0
             ? aiFeature.aiMissingScenarios
@@ -274,8 +285,18 @@ function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adaptiveThres
             ? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
             : aiFeatureByFamilyId.get(f.familyId);
         const baseReasons = [`${label} is covered by Cypress only — consider adding Playwright tests`];
-        const reasons = aiFeature && aiFeature.aiReasons.length > 0
-            ? [...baseReasons, ...aiFeature.aiReasons.slice(0, 2)]
+        let partialAiReasons = [];
+        if (aiFeature) {
+            if (aiFeature.aiReasons.length > 0) {
+                partialAiReasons = aiFeature.aiReasons.slice(0, 2);
+            }
+            else {
+                const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
+                partialAiReasons = [`Changes to ${fileHint} affect the ${label} feature, which has Cypress but no Playwright coverage.`];
+            }
+        }
+        const reasons = partialAiReasons.length > 0
+            ? [...baseReasons, ...partialAiReasons]
             : baseReasons;
         gapDetails.push({
             id: label,

package/dist/esm/engine/plan_builder.js

@@ -245,8 +245,19 @@ export function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adapti
             ? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
             : aiFeatureByFamilyId.get(f.familyId);
         const baseReasons = [`No E2E tests found for ${label}`];
-        const reasons = aiFeature && aiFeature.aiReasons.length > 0
-            ? [...baseReasons, ...aiFeature.aiReasons.slice(0, 2)]
+        let aiReasonsList = [];
+        if (aiFeature) {
+            if (aiFeature.aiReasons.length > 0) {
+                aiReasonsList = aiFeature.aiReasons.slice(0, 2);
+            }
+            else {
+                // Fallback: LLM returned scenarios but no reasons — synthesize a description
+                const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
+                aiReasonsList = [`Changes to ${fileHint} affect the ${label} feature, which currently lacks E2E coverage.`];
+            }
+        }
+        const reasons = aiReasonsList.length > 0
+            ? [...baseReasons, ...aiReasonsList]
             : baseReasons;
         const missingScenarios = aiFeature && aiFeature.aiMissingScenarios.length > 0
             ? aiFeature.aiMissingScenarios
@@ -268,8 +279,18 @@ export function buildPlanFromImpact(impact, policyOverride, aiEnrichment, adapti
             ? (aiFeatureByFeatureId.get(f.featureId) ?? aiFeatureByFamilyId.get(f.familyId))
             : aiFeatureByFamilyId.get(f.familyId);
         const baseReasons = [`${label} is covered by Cypress only — consider adding Playwright tests`];
-        const reasons = aiFeature && aiFeature.aiReasons.length > 0
-            ? [...baseReasons, ...aiFeature.aiReasons.slice(0, 2)]
+        let partialAiReasons = [];
+        if (aiFeature) {
+            if (aiFeature.aiReasons.length > 0) {
+                partialAiReasons = aiFeature.aiReasons.slice(0, 2);
+            }
+            else {
+                const fileHint = f.changedFiles.slice(0, 3).map((p) => p.split('/').pop()).join(', ');
+                partialAiReasons = [`Changes to ${fileHint} affect the ${label} feature, which has Cypress but no Playwright coverage.`];
+            }
+        }
+        const reasons = partialAiReasons.length > 0
+            ? [...baseReasons, ...partialAiReasons]
             : baseReasons;
         gapDetails.push({
             id: label,

package/dist/esm/training/validator.js

@@ -15,6 +15,9 @@ const INFRA_GLOBS = [
     '**/.github/*', '**/.ci/*', '**/scripts/*',
     '**/docker-compose*',
     '**/__fixtures__/*', '**/test_templates/*',
+    '**/__snapshots__/*', '**/retrylayer/*', '**/timerlayer/*',
+    '**/cmd/mmctl/*',
+    '**/mattermost-redux/src/reducers/*',
     'playwright.config.ts', 'global-setup.ts',
 ];
 /**

package/dist/training/validator.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"validator.d.ts","sourceRoot":"","sources":["../../src/training/validator.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAC,mBAAmB,EAAC,MAAM,gCAAgC,CAAC;AAExE,OAAO,KAAK,EAAC,gBAAgB,EAAE,gBAAgB,EAAC,MAAM,YAAY,CAAC;AAiBnE;;;GAGG;AACH,wBAAgB,WAAW,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CA6BrD;AAED,wBAAgB,WAAW,CAAC,GAAG,EAAE,MAAM,GAAG,KAAK,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,EAAE,CAAA;CAAC,CAAC,CA6BhG;AAED,wBAAgB,cAAc,CAAC,WAAW,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,KAAK,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,EAAE,CAAA;CAAC,CAAC,CAgB1H;AA+CD,wBAAgB,cAAc,CAC1B,QAAQ,EAAE,mBAAmB,EAC7B,KAAK,EAAE,MAAM,EAAE,EACf,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,MAAM,EACf,YAAY,CAAC,EAAE,MAAM,EAAE,GACxB,gBAAgB,CA+BlB;AAED,wBAAgB,qBAAqB,CACjC,OAAO,EAAE,gBAAgB,EAAE,EAC3B,QAAQ,EAAE,mBAAmB,GAC9B,gBAAgB,CAkDlB;AAED,wBAAgB,sBAAsB,CAAC,MAAM,EAAE,gBAAgB,GAAG,MAAM,CAgCvE"}
+{"version":3,"file":"validator.d.ts","sourceRoot":"","sources":["../../src/training/validator.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAC,mBAAmB,EAAC,MAAM,gCAAgC,CAAC;AAExE,OAAO,KAAK,EAAC,gBAAgB,EAAE,gBAAgB,EAAC,MAAM,YAAY,CAAC;AAoBnE;;;GAGG;AACH,wBAAgB,WAAW,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CA6BrD;AAED,wBAAgB,WAAW,CAAC,GAAG,EAAE,MAAM,GAAG,KAAK,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,EAAE,CAAA;CAAC,CAAC,CA6BhG;AAED,wBAAgB,cAAc,CAAC,WAAW,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,KAAK,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,EAAE,CAAA;CAAC,CAAC,CAgB1H;AA+CD,wBAAgB,cAAc,CAC1B,QAAQ,EAAE,mBAAmB,EAC7B,KAAK,EAAE,MAAM,EAAE,EACf,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,MAAM,EACf,YAAY,CAAC,EAAE,MAAM,EAAE,GACxB,gBAAgB,CA+BlB;AAED,wBAAgB,qBAAqB,CACjC,OAAO,EAAE,gBAAgB,EAAE,EAC3B,QAAQ,EAAE,mBAAmB,GAC9B,gBAAgB,CAkDlB;AAED,wBAAgB,sBAAsB,CAAC,MAAM,EAAE,gBAAgB,GAAG,MAAM,CAgCvE"}

package/dist/training/validator.js

@@ -23,6 +23,9 @@ const INFRA_GLOBS = [
     '**/.github/*', '**/.ci/*', '**/scripts/*',
     '**/docker-compose*',
     '**/__fixtures__/*', '**/test_templates/*',
+    '**/__snapshots__/*', '**/retrylayer/*', '**/timerlayer/*',
+    '**/cmd/mmctl/*',
+    '**/mattermost-redux/src/reducers/*',
     'playwright.config.ts', 'global-setup.ts',
 ];
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yasserkhanorg/e2e-agents",
-  "version": "1.7.1",
+  "version": "1.7.3",
   "description": "AI-powered E2E test impact analysis, generation, and healing. Analyzes code changes to identify affected Playwright tests, detects coverage gaps, and generates or repairs specs using pluggable LLM providers (Claude, OpenAI, Ollama). Includes MCP server, traceability, and CI/CD integration.",
   "main": "dist/index.js",
   "module": "dist/esm/index.js",