@vijaypjavvadi/bdd2pw 3.8.1 → 3.10.0

package/CHANGELOG.md

@@ -9,6 +9,177 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 _Nothing yet._
 
+## [3.10.0] — 2026-06-06
+
+### Added — self-healing stats sidecar + `bdd2pw heal-stats` CLI
+
+v1.1+ has shipped a runtime `healOrThrow` helper that emits
+`register` / `heal_attempt` / `healed` / `heal_unavailable` events
+to `<repo>/artefacts/heal-events.jsonl`. Until now those events
+sat in a JSONL file with no offline analyzer — operators had to
+grep the file themselves to see if healing was working.
+
+v3.10.0 adds:
+
+**1. `bdd2pw heal-stats <repo>` CLI** — reads
+`<repo>/artefacts/heal-events.jsonl` and writes
+`<repo>/artefacts/heal-stats.json` with:
+
+    {
+      "version": "3.10.0",
+      "generatedAt": "2026-06-06T...",
+      "source": "<events file>",
+      "totals": {
+        "registrations": 47,
+        "healAttempts": 6,
+        "healed": 4,
+        "healUnavailable": 2,
+        "healRate": 0.6667,
+        "uniqueFields": 23,
+        "uniquePages": 3
+      },
+      "topFailingFields": [
+        { "page": "LoginPage", "name": "submitButton",
+          "attempts": 3, "healed": 2 },
+        ...
+      ],
+      "topErrors": [
+        { "error": "timeout <n>ms exceeded", "count": 4 },
+        ...
+      ],
+      "topCandidates": [
+        { "selector": "[data-testid='submit']", "promotions": 2,
+          "averageConfidence": 0.87 },
+        ...
+      ],
+      "retryLatencyMs": { "p50": 1240, "p95": 3010,
+                          "min": 880, "max": 3010 },
+      "perScenario": [
+        { "scenario": "User can log in", "attempts": 1, "healed": 1 },
+        ...
+      ]
+    }
+
+Where v3.9.0's `llm-stats.json` measures the LLM batching ROI per
+scaffold, v3.10.0's `heal-stats.json` measures the self-healing
+ROI per TEST RUN.
+
+**2. New module `src/reports/healStats.ts`** — pure
+`aggregate(events, source, topN)` function plus the wrapper
+`analyseHealStats(opts)` that resolves the input path
+(file / repo-dir / artefacts-subdir) and writes the sidecar.
+
+**3. Error normalisation.** The aggregator collapses error messages
+into stable buckets (lowercase, strip "Error:"/"TimeoutError:"
+prefixes, replace runs of 3+ digits with `<n>`) so transient
+timestamps / IDs don't fragment the histogram.
+
+**4. Retry latency derivation.** Pairs each `heal_attempt` with its
+matching `healed` (by `page+name+method`) and reports the timestamp
+delta as the retry latency. Computes p50 / p95 / min / max across
+the run.
+
+**5. Graceful empty-input behaviour.** Pointing the CLI at a repo
+with no heal-events.jsonl writes a zero-event summary instead of
+failing — `bdd2pw heal-stats` is now safe to run unconditionally
+in CI after `npx playwright test`.
+
+### Tests
+
+`tests/unit/v3100HealStats.test.ts`:
+- Empty input → all-zero summary.
+- Mixed register / heal_attempt / healed / heal_unavailable sequence
+  aggregates correctly (counts, healRate, topFailingFields,
+  topErrors, topCandidates, p50/p95 latency, per-scenario rollup).
+- `topN` honored — top-3 returns 3 entries from 5 distinct fields.
+- End-to-end: write JSONL → `analyseHealStats(file)` → read JSON
+  back and verify shape + version stamp.
+- Missing-file safety: pointing at a directory with no
+  heal-events.jsonl returns total=0 and writes an empty summary.
+- Repo-path resolution: passing a directory finds
+  `artefacts/heal-events.jsonl` automatically.
+
+## [3.9.0] — 2026-06-05
+
+### Added — opt-in LLM telemetry sidecar
+
+`scaffold({ llmStats: true })` now writes
+`<repo>/artefacts/llm-stats.json` capturing per-scaffold LLM activity:
+
+    {
+      "version": "3.9.0",
+      "scaffoldId": "scaffold-...",
+      "generatedAt": "2026-06-05T...",
+      "totals": {
+        "callsAttempted": 5,
+        "callsSuccessful": 5,
+        "bindingsGenerated": 23,
+        "cacheHits": 12,
+        "cacheMisses": 11,
+        "cacheHitRate": 0.522,
+        "inputTokens": 24500,
+        "outputTokens": 8200,
+        "estimatedCostUsd": 0.0735
+      },
+      "calls": [
+        { "index": 1, "model": "...", "inputTokens": 4900,
+          "outputTokens": 1820, "latencyMs": 2310,
+          "fromCache": false, "batchSize": 5 },
+        ...
+      ],
+      "latencyMs": { "p50": 2100, "p95": 3450, "min": 1200, "max": 3450 },
+      "pricing": { "model": "claude-sonnet-4-6",
+                   "inputUsdPerMillion": 3, "outputUsdPerMillion": 15 }
+    }
+
+**Why it exists.** v3.5.0 introduced per-scenario LLM batching with
+the promise of ~75% cost reduction on unmatched-heavy scaffolds.
+Until now, the savings were invisible — operators had no per-scaffold
+record of cache hit rate, batch size distribution, token counts, or
+estimated cost. This sidecar makes the ROI measurable per run.
+
+**How it works.** New `src/llm/telemetry.ts` ships an `LLMTelemetry`
+class that subscribes to the existing `LLMLogEvent` stream
+(`cache_hit` / `cache_miss` / `provider_call_start` /
+`provider_call_done` / `binding_parsed` events that
+`AnthropicLLMClient` already emits). The scaffold orchestrator fans
+the log callback to BOTH the existing pino logger AND the telemetry
+collector. Zero overhead when disabled — no telemetry instance is
+constructed and the fan-out is a single `?.` null check.
+
+**Pricing.** Cost is an ESTIMATE — provider prices change. The
+sidecar records the per-million rates used (default snapshot for
+Claude Sonnet 4.6: $3/M input, $15/M output) so a reviewer can
+re-derive the number against current pricing. Override via
+`new LLMTelemetry(scaffoldId, version, pricingOverride)` if you
+have negotiated rates or a different model.
+
+**New exports** from `@vijaypjavvadi/bdd2pw`:
+- `LLMTelemetry` — the collector class.
+- `DEFAULT_PRICING` — per-million USD pricing snapshot. Pre-seeded
+  for `claude-sonnet-4-6`, `claude-3-7-sonnet-latest`,
+  `claude-opus-4-6`, `claude-haiku-4-5-20251001`.
+- Types: `TelemetryCall`, `TelemetrySummary`.
+
+### Default behavior
+
+`llmStats` is **off** by default — pure-UI scaffolds and existing
+LLM-enabled scaffolds emit the same files as v3.8.x. Operators opt
+in when they want to track spend or debug latency.
+
+### Tests
+
+`tests/unit/v390Telemetry.test.ts`:
+- Single-batch aggregates correctly (cache misses incremented,
+  tokens captured, latency recorded, bindings counted).
+- Cache hit rate computed across mixed cached + uncached calls.
+- p50 / p95 latency from a 10-element sample.
+- Cost estimation against the known Sonnet pricing snapshot.
+- Scaffold context (id, version, generated timestamp) stamped
+  into the summary.
+- Zero-event scaffold produces a clean empty summary (no crash,
+  no NaN).
+
 ## [3.8.0] — 2026-05-24
 
 ### Added — four more opt-in domain rule packs