tamash-playwright 0.13.0-beta.2 → 0.13.0-beta.3

package/.env.example

@@ -95,3 +95,23 @@ OLLAMA_API_KEY=
 # no vision fallback, no action-recovery tactics. A fast, free, deterministic first line of
 # defense, not a full replacement for an AI provider. Just set:
 # HEALER_PROVIDER=tamash
+
+# --- Failure analysis: classify a genuinely-failed expect() assertion ---
+# A separate capability from healing above — never fixes anything, only explains a test failure
+# that already happened after every retry was exhausted: likely-defect vs. likely-wrong-locator
+# vs. likely-timing-or-environment vs. inconclusive, with a short explanation. Covers any final
+# failure, not just expect() -- an action (click/fill/etc.) healing already tried and reported on
+# is analyzed too, with healing's own diagnosis folded in as extra context rather than skipped.
+# Reuses whichever HEALER_PROVIDER is already configured above (tamash always declines — no
+# reasoning capability). No reporter to add to playwright.config.ts, runs automatically the same
+# way healing does.
+#
+# On by default, same polarity as HEALER_ENABLED above (unset or anything other than false/0
+# leaves it on) -- independent of HEALER_ENABLED, though: turning healing off does not also turn
+# this off, they check two separate flags. Costs a real AI call on every genuinely-failed test
+# (unlike healing, which only spends when an action fails and is opportunistic about it), so a
+# suite with several broken tests means several calls, every run, until each is fixed. Each
+# call's token usage shows up as its own `failure-analysis-tokens-used` annotation in the report
+# -- deliberately a different annotation type from healing's own `llm-tokens-used`, so the two
+# costs can be weighed against each other, not just healing's. Set to false to turn it off:
+# FAILURE_ANALYSIS_ENABLED=false

package/CHANGELOG.md

@@ -5,12 +5,20 @@ All notable changes to this project are documented here. Format loosely follows
 
 ## [Unreleased]
 
+### Added
+
+- **AI-powered analysis of a genuinely-failed test.** Healing has always deliberately left `expect()` untouched — silently "fixing" an assertion could mask a real defect. This adds a separate capability instead: once a test's retries are exhausted and it's still failing, classify *why* — `likely-defect`, `likely-wrong-locator`, `likely-timing-or-environment`, or `inconclusive` — with a short explanation naming what was actually found, so it's clear whether that's worth filing a bug for, fixing the test's selector, or investigating test stability. Covers any final failure, not just `expect()`: an action (click/fill/etc.) that healing already tried and reported on is analyzed too, folding healing's own diagnosis (provider, failure stage, its own reason for declining or for a replay that still failed) in as extra context for the classifier — e.g. healing declining because nothing plausible existed anywhere on the page is itself a strong signal toward `likely-defect` that the raw error message alone doesn't carry — rather than skipping the action failure or re-deriving everything from scratch. One unified verdict per genuinely-failed test, whatever kind of failure it was, not two separate, potentially conflicting explanations of the same thing. `likely-wrong-locator` is checked first, deliberately, ahead of `likely-defect`: it's the one verdict the AI is uniquely positioned to catch that skimming a plain error message would miss — the full ARIA snapshot is checked for something that plausibly matches under different text/structure before concluding the app itself is broken. On by default (`FAILURE_ANALYSIS_ENABLED`, same polarity as `HEALER_ENABLED` — unset or anything but `false`/`0` leaves it on; independent of `HEALER_ENABLED` itself, so turning healing off doesn't also silence this) — no reporter to register, no config file to edit: it's a `test.extend()` auto-fixture, the same mechanism `bindContext`/`bindPageActions`/`bindBrowser` already use, so it runs automatically the moment `test` is imported from this package. Chose default-on over mirroring `HEALER_ACTION_RECOVERY_ENABLED`'s opt-in-off polarity deliberately: this is a standalone capability, not a second-order escalation on top of an already-running mechanism the way action-recovery is — structurally closer to `HEALER_ENABLED` itself. The real tradeoff: unlike healing, which only spends tokens on a genuine action failure and is opportunistic about it (most actions succeed), this spends on *every* genuinely-failed test regardless of outcome — a red run with several broken tests means several AI calls, every run, until each is fixed. Each call's usage is its own `failure-analysis-tokens-used` annotation, deliberately a different type from healing's own `llm-tokens-used`, so the two costs can be told apart and weighed against each other rather than requiring free-text parsing to separate. (An earlier draft of this shipped as a standalone Playwright Reporter instead — reverted before release: it would have meant a *second* reporter line for anyone already running `tamash-playwright-dashboard`, on top of a real bug found only by actually running it — a fire-and-forget async call inside `onTestEnd` was silently abandoned when the process exited before the AI call resolved, since that hook's return type is strictly `void`, never awaited.) Reuses whichever `HEALER_PROVIDER` is already configured; `tamash` always declines (no reasoning capability). Exactly one AI call per genuinely-failed test when retries are configured at the project level (`playwright.config.ts`'s own `retries:`, or `--retries`) — the common case; a per-file `test.describe.configure({ retries })` override isn't visible to a fixture, so that specific case falls back to one call per attempt instead of one overall, confirmed safe (no crash, no wrong verdict) but not exactly-once. Reported the same way healing's own reports already are (`attachReportToRunningTest`, `healer/index.ts`): a `failure-analysis` annotation summarizing the verdict, the token-usage annotation described above, and a JSON attachment with the full attempt history — all visible in Playwright's own HTML report, not just the console. Deliberately no separate results log file: unlike `heals.jsonl` (a real cache, and read by `apply-heals`), nothing consumes one for this feature, and `tamash-playwright-dashboard` already reads healing's own reports straight off the test result's attachments rather than `heals.jsonl` — confirmed directly against its source (`reporter-helpers.ts`'s `extractHealReports`) — so a future dashboard integration needs no separate file either. Reads the exact same ARIA snapshot Playwright itself already captures at failure time for its own "Copy prompt for AI" feature when available, falling back to a live snapshot (the page is still open from a fixture, unlike from a Reporter) otherwise. Same non-negotiable property as healing itself: `expect()` is never touched, so this can only ever add information next to a failure that already happened, never change pass/fail.
+
+  Four real correctness bugs found and fixed via deliberate corner-case testing after the initial implementation, not caught by the happy-path tests alone: (1) the per-attempt working log was originally keyed by `testFile:testLine`, mirroring `heal-log.ts`'s own keying for a locator's call site — but that does NOT uniquely identify a *test*, since a loop generating parameterized tests (`for (const x of [...]) test(...)`) produces several tests sharing the exact same source line; four such tests failing concurrently corrupted each other's attempt history (one ended up with 5 attempts pulled from 3 different tests) before switching the key to `testInfo.testId` (confirmed stable across retries, genuinely unique even when the line collides). (2) A flaky test — fails once, recovers on retry — used to leave that failed attempt's row behind forever, since cleanup only happened after a genuine final-attempt analysis, which a recovered test never reaches; fixed by clearing a test's own rows on the passed branch too. (3) `testInfo.status !== 'passed'` fires for *any* reason a test fails, not just a genuine `expect()` assertion — a plain action (click/fill) that healing already tried and reported on (`self-heal-failed` already pushed onto `testInfo.annotations` synchronously, mid-test-body, by healing's own catch block) reached this same fixture and got explained a second, redundant time by an AI call reasoning about the same failure with strictly less context. An intermediate fix skipped analysis whenever `self-heal-failed` was already present, scoping the whole feature to `expect()` only — reconsidered before release: that meant an action failure never got the "is this worth filing a bug for" verdict either, just healing's own (different) diagnosis of why the *locator* couldn't be fixed. Landed instead as described above — analyze every final failure, folding healing's own diagnosis in as context — which needed the same stale-row cleanup discipline as bug (2), just reached via a different path: a mixed sequence (attempt 0 a genuine `expect()` failure, attempt 1 an action failure) must still end up with both attempts in the one final analysis, and the working log fully cleared afterward either way. (4) `readFailureAttempts` rebuilt each entry via an explicit field list (`{ retry, errorMessage, ariaSnapshot }`) that predated the `healingAttempt` field added for bug (3)'s fix — `appendFailureAttempt` wrote it to disk correctly the whole time, but every *read* silently dropped it again, so an action failure's healing context never actually reached the AI despite being captured; fixed by adding `healingAttempt` to that same explicit list, now covered by a unit test asserting the round trip specifically (not just that *a* value round-trips, since that would have passed before this fix too, given the other fields did).
+
 ### Fixed
 
 - **A context/page built off `browser` in `test.beforeAll` (shared across tests) was never healing-aware.** Only the `context`/`page` fixtures were wrapped with the healing proxy — a consumer using `test.beforeAll(async ({ browser }) => { context = await browser.newContext(); page = await context.newPage(); })` to log in once and reuse the session across every test in a file bypassed both, so nothing built that way ever healed. Found via a real support report. Fixed: the `browser` fixture is now wrapped too (`bindBrowser()`, `src/bindings/locator.binding.ts`) — `browser.newContext()`, `browser.newPage()`, and `browser.contexts()` all now hand back healing-aware objects, exactly like `context.newPage()` already did. Doesn't cover a browser obtained entirely outside the fixture system (e.g. `chromium.launch()` in `globalSetup`) — see "What else it heals" in README.md/usage.md for the manual `bindContext()`/`bindPageActions()` escape hatch for that case.
 
 ### Docs
 
+- **README.md rewritten for readability — 666 lines down to ~220, no content lost.** The published npm listing had grown into dense, run-on paragraphs padded with self-qualifying phrases ("confirmed live," "confirmed by real testing, not just docs") — an audit-trail tone that belongs in this CHANGELOG and RELEASE-TESTING.md, not in a first-time visitor's landing page. Rewritten with short sentences and a plain, confident tone; the top-level "get started" flow (install → connect a provider → set `actionTimeout` → use it) now reads in under a minute instead of being buried under ~230 lines of provider-auth edge cases and a 110-line raw GitHub Actions YAML block. Nothing was deleted: the subscription-provider deep-dives (Claude/Copilot/Cursor/Kiro/Codex, including the Copilot org-vs-personal-token gotcha) moved into usage.md's "2. Connect an AI model" section as new subsections (it hadn't documented them at all before — README was the only copy), and README now links out to usage.md's existing sections for CI YAML, caching, and everything else usage.md already covered in full. Also fixed a real gap found while doing this: the `.env`/`usage.md` "pick one" provider comment was missing `cursor-subscription`/`kiro-subscription`/`codex-subscription` even though all three are real, documented providers.
+
 - **README.md/usage.md/feature.md now lead with the actual rule, not just examples of it.** The `browser`-in-`beforeAll` fix above was documented as a bullet point alongside popups/iframes, which left the real boundary implicit: *any* browser/context/page from Playwright's own fixtures heals automatically, however many hops away and however it's reused, while anything built outside the fixture system (`chromium.launch()` directly, or `globalSetup`) still needs manual `bindContext()`/`bindPageActions()`. Now stated as the first thing in "What else it heals," not inferred from bullets.
 
 ## [0.12.0] - 2026-09-13