tamash-playwright 0.7.0-beta.7 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,431 +1,431 @@
1
- # tamash-playwright
2
-
3
- `tamash-playwright` is a plug and play self-healing solution for any Playwright test framework. All you need to do is install the package, update your AI API key details, and import `test` from `tamash-playwright`.
4
-
5
- That's it. No code changes required if you're following standard Playwright best practices.
6
-
7
- ### Why you need this
8
-
9
- Websites change often. A button gets renamed or moved, and your test can't find it anymore — even though the app still works fine for real users. Normally, that just means a broken test.
10
-
11
- `tamash-playwright` fixes this automatically. When a test can't find an element, it asks an AI model to find it on the current page and tries again. If it succeeds, your test keeps going. If not, it fails normally, just like before.
12
-
13
- **Want to see it working before you set it up yourself?** Clone the sample repo — [github.com/qtpsudhakarproducts/tamash-playwright-typescript-playwright](https://github.com/qtpsudhakarproducts/tamash-playwright-typescript-playwright) — a full worked example with both a plain-locator test and a Page Object Model test, an intentionally broken selector, and step-by-step setup instructions.
14
-
15
- **Looking for the complete reference** — caching, `apply-heals`, sharded-CI and PR automation, every env var and CLI flag? See [usage.md](usage.md).
16
-
17
- Here are the detailed steps to use this package.
18
-
19
- ## Step 1: Install it
20
-
21
- ```sh
22
- npm install tamash-playwright
23
- ```
24
-
25
- You also need Playwright's own test package, if you don't already have it:
26
-
27
- ```sh
28
- npm install -D @playwright/test
29
- ```
30
-
31
- ## Step 2: Connect an AI model
32
-
33
- `tamash-playwright` needs an AI model to decide where a broken element actually went. Pick one of Ollama, OpenAI, Anthropic (Claude), or Google Gemini, and give it an API key.
34
-
35
- Create a file named `.env` in your project folder:
36
-
37
- ```sh
38
- # Master on/off switch. Leave this as true, or remove the line entirely.
39
- HEALER_ENABLED=true
40
-
41
- # Pick one: ollama | openai | anthropic | gemini
42
- HEALER_PROVIDER=ollama
43
-
44
- # Optional, off by default — see "Action recovery" below.
45
- # HEALER_ACTION_RECOVERY_ENABLED=true
46
-
47
- # --- Ollama Cloud (https://ollama.com) ---
48
- OLLAMA_MODEL=gpt-oss:120b
49
- OLLAMA_API_KEY=
50
-
51
- # --- OpenAI ---
52
- # OPENAI_MODEL=gpt-4.1-mini
53
- # OPENAI_API_KEY=
54
-
55
- # --- Anthropic (Claude) ---
56
- # ANTHROPIC_MODEL=claude-haiku-4-5
57
- # ANTHROPIC_API_KEY=
58
-
59
- # --- Google Gemini ---
60
- # GEMINI_MODEL=
61
- # GEMINI_API_KEY=
62
- ```
63
-
64
- Just fill in the API key and model for whichever one you want to use, and leave the rest as-is (or delete them).
65
-
66
- ### Getting a free Ollama key (fastest way to get started)
67
-
68
- Ollama Cloud is a quick, free way to get an API key without signing up for OpenAI/Anthropic/Gemini billing.
69
-
70
- 1. Go to [ollama.com](https://ollama.com/) and create an account.
71
- 2. Once signed in, go to [ollama.com/settings/keys](https://ollama.com/settings/keys).
72
- 3. Create a new API key and copy it.
73
- 4. Paste it into your `.env` file:
74
-
75
- ```sh
76
- HEALER_ENABLED=true
77
- HEALER_PROVIDER=ollama
78
- OLLAMA_MODEL=gpt-oss:120b
79
- OLLAMA_API_KEY=paste_your_key_here
80
- ```
81
-
82
- That's all you need — no other variables required.
83
-
84
- ### Important: set `actionTimeout` in your `playwright.config.ts`
85
-
86
- By default, Playwright lets a broken locator retry silently for your *entire* test timeout before it ever throws an error — which means self-healing never gets a turn at all, since it only kicks in once an action actually fails. Set `actionTimeout` to something well below your test timeout so a broken locator fails fast, leaving real time for healing to run:
87
-
88
- ```ts
89
- export default defineConfig({
90
- timeout: 60000, // your overall test timeout
91
- use: {
92
- actionTimeout: 8000, // must be comfortably less than the test timeout above
93
- },
94
- });
95
- ```
96
-
97
- Without this, healing attempts will show `stage=no_snapshot` in the console and never recover anything — not because healing failed, but because it never had time to run before the whole test was torn down.
98
-
99
- ## Step 3: Check your setup
100
-
101
- Run the built-in doctor command to confirm everything's wired up correctly before you rely on it:
102
-
103
- ```sh
104
- npx tamash-playwright doctor
105
- ```
106
-
107
- It checks:
108
-
109
- 1. **AI connectivity** — confirms `HEALER_ENABLED`/`HEALER_PROVIDER` are set correctly and actually calls your configured provider to make sure the API key and model work.
110
- 2. **`actionTimeout` configuration** — checks your `playwright.config.ts` for an `actionTimeout` set well below your test `timeout` (see above); flags it if missing or too close to the test timeout, since that silently starves self-healing of any time to run.
111
- 3. **Action recovery status** — whether `HEALER_ACTION_RECOVERY_ENABLED` is on (see below).
112
- 4. **Vision capability** — whether your configured model is expected to support the screenshot-based fallback (see below), based on its name.
113
- 5. **Missing `.describe()` labels** — scans your test files (`tests/` by default, or pass `--dir <path>`) for locators that don't have a `.describe('...')` label, and flags the ones most worth fixing (raw CSS/XPath selectors first).
114
- 6. **Locators written directly in test files** — flags any locator defined inline in a test rather than inside a Page Object class, which is a Playwright best practice regardless of self-healing: it keeps tests readable and means a UI change only needs a fix in one place.
115
-
116
- If it finds issues, the fastest fix is to open the project in an AI coding assistant (Claude Code, Cursor, GitHub Copilot, etc.) and ask it to address what it flagged — add `.describe()` calls, or extract locators into Page Object classes. You can also add a standing rule to that assistant's instructions/skill file (e.g. `CLAUDE.md`, `.cursor/rules`, `.github/copilot-instructions.md`) so it follows both practices automatically on any new test code going forward.
117
-
118
- ## Step 4: Use it in your tests
119
-
120
- Change one line at the top of your test file — everything else about how you write tests stays exactly the same:
121
-
122
- ```ts
123
- // Before
124
- import { test, expect } from '@playwright/test';
125
-
126
- // After
127
- import { test, expect } from 'tamash-playwright';
128
- ```
129
-
130
- That's it. Write your tests as normal:
131
-
132
- ```ts
133
- import { test, expect } from 'tamash-playwright';
134
-
135
- test('logs in', async ({ page }) => {
136
- await page.goto('/');
137
- const txtUserName = page.locator('input[name="username"]').describe('User Name Textbox');
138
- await txtUserName.fill('testadmin');
139
-
140
- const txtPassword = page.locator('input[placeholder="Password"]').describe('Password Textbox');
141
- await txtPassword.fill('secret');
142
-
143
- const btnLogin = page.locator('button[type="submit"]').describe('Login Button');
144
- await btnLogin.click();
145
-
146
- await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
147
- });
148
- ```
149
-
150
- ### A quick tip for better results
151
-
152
- If you're using plain CSS selectors (like `page.locator('input[name="username"]')`) rather than Playwright's more descriptive locators (`getByRole`, `getByPlaceholder`, etc.), it helps to add a short, human-readable label so the healer knows what it's actually looking for. Chain `.describe('...')` right onto the locator:
153
-
154
- ```ts
155
- test('login test using CSS Selectors', async ({ page }) => {
156
- await page.goto('https://example.com/auth/login');
157
-
158
- const txtUserName = page.locator('input[name="username"]').describe('User Name Textbox');
159
- await txtUserName.fill('testadmin');
160
-
161
- const txtPassword = page.locator('input[placeholder="Password"]').describe('Password Textbox');
162
- await txtPassword.fill('secret');
163
-
164
- const btnLogin = page.locator('button[type="submit"]').describe('Login Button');
165
- await btnLogin.click();
166
-
167
- await expect(page.locator('h6')).toHaveText('Dashboard');
168
- });
169
- ```
170
-
171
- This step is optional, but recommended — without it, the healer has to guess purely from a broken CSS selector, which gives it a lot less to work with.
172
-
173
- ## Local vs CI, at a glance
174
-
175
- Everything below applies identically in both places — same package, same behavior — but a few things are worth knowing up front before you get to the details:
176
-
177
- | | Locally | In CI |
178
- | --- | --- | --- |
179
- | **Healing** | Runs automatically on every `npx playwright test`, using the `.env` file from Step 2. | Runs automatically the same way — set the same variables as CI secrets/environment variables on your test job instead of a `.env` file (there's a real example in [Running `apply-heals` in CI](#running-apply-heals-in-ci-sharded-or-not)). |
180
- | **Caching** ([below](#not-paying-for-the-same-heal-twice)) | Persists on disk across every run — a real, ongoing saving the longer you keep working. | Only helps *within* one run — most runners start from a fresh checkout each time, so it doesn't carry over between separate CI runs. `apply-heals` landing the real fix is what actually stops repeat AI calls in CI, not the cache. |
181
- | **`apply-heals`** ([below](#making-a-heal-permanent-apply-heals)) | Run manually whenever you want, preview with `--dry-run`, review the diff yourself, commit when ready. | Runs automatically after your test job and opens a PR for review — nothing ever auto-commits to your branch. |
182
-
183
- ## What else it heals — no extra setup needed
184
-
185
- Beyond a single broken `click`/`fill`/`getByRole` on the main page, all of this works automatically once you've done Steps 1–2:
186
-
187
- - **Popups and extra tabs.** A page opened via `context.newPage()`, `window.open`, or a `target="_blank"` link is just as healing-aware as your main `page` — no manual wrapping needed.
188
- - **Elements inside `<iframe>`s.** `page.frameLocator('#my-iframe')` and anything chained off it heals the same way, scoped correctly to the iframe's own document.
189
- - **Most of the Playwright API surface**, not just clicks and fills — `check`, `selectOption`, `dragTo`, `dispatchEvent`, read methods like `textContent`/`getAttribute`/`isChecked`, `screenshot`, and more. Methods that can't be safely healed by guessing a replacement element (`dragTo`, `drop`) are still reported honestly on failure, they're just never silently retried with a different element.
190
-
191
- ## When there's no name to match: finding elements by structure
192
-
193
- Sometimes the broken element has no useful identity of its own — a plain `<input>` with no name, no working placeholder, and a label that's visually right next to it but never actually linked (no `<label for>`, no `aria-labelledby`). A human finds it instantly by sight; matching purely on accessible name has nothing to grab onto.
194
-
195
- For exactly these cases, `tamash-playwright` reads a structural map of the whole page — not just a flat list of named elements — so the AI can point at the exact element even when it has no name of its own. Once it has that, a separate step (no extra AI call) works out the most stable way to describe it for next time:
196
-
197
- - If the element has real identity — an id, test id, accessible role and name, label, or placeholder — that's used directly.
198
- - If not, but a label sits right next to it in the page's own structure, the fix anchors on that nearby text instead — and only after confirming it genuinely points at the *same* element, not just one that happens to match.
199
- - If neither applies, it falls back to whatever Playwright's own locator-generation logic can produce, even a positional one — but flags it for review rather than treating it as fully trusted (see below), since a selector that depends on element order can silently point at the wrong thing later if the page changes.
200
-
201
- You don't configure any of this — it happens automatically, and every candidate is verified against the live page (real DOM identity, not just "a match was found") before anything gets used.
202
-
203
- ### When a fix needs a second look
204
-
205
- Not every healed selector is equally durable. One with a real id, test id, or accessible name is about as solid as anything a human would write by hand. One that only had a nearby label or a positional fallback to work with is correct *right now*, but worth a glance before you fully rely on it — the page could change in a way that fools it later.
206
-
207
- `tamash-playwright` tells you which is which — look for `needsReview=yes` in the console line, a `self-heal-needs-review` annotation in the HTML report, or a `[NEEDS REVIEW]` tag in `apply-heals`'s output (see below). Nothing is blocked or held back because of it; it's a hint about where to look first, not a gate.
208
-
209
- ## When text alone isn't enough: vision fallback
210
-
211
- Sometimes an element has nothing useful to match on by text — an icon-only button with no label, or several visually distinct elements that all look identical in the accessibility tree. If your configured model supports image input (e.g. `gpt-4o`, `claude-haiku-4-5`, `gemini-2.0-flash`), `tamash-playwright` automatically falls back to a screenshot-based search after the normal text-based attempt fails — no separate setup, it just uses the same provider and API key from Step 2. Run `npx tamash-playwright doctor` to check whether your configured model is expected to support this.
212
-
213
- ## Action recovery (optional)
214
-
215
- Occasionally a locator heals correctly — the AI found the right element — but the *action* on it still fails, e.g. it's covered by an overlay or needs scrolling into view first. Set `HEALER_ACTION_RECOVERY_ENABLED=true` to let the AI pick a recovery tactic from a fixed, safe set (scroll into view, retry bypassing actionability checks, wait briefly and retry, or dispatch the DOM event directly) before giving up. It's off by default since it's a second, more speculative layer of intervention beyond selector healing — the AI only ever picks from that fixed menu, it never decides how to interact with the page on its own.
216
-
217
- ## Not paying for the same heal twice
218
-
219
- Once a locator heals successfully, `tamash-playwright` remembers the fix in `.tamash-playwright/heals.jsonl` — the same file `apply-heals` reads (see below). The next time that exact locator breaks the same way, it tries the previously-confirmed selector *first*, with no ARIA snapshot and no AI call. Only if that no longer works (the page changed again) does it fall through to a fresh snapshot-and-AI-call, exactly as before — so there's no correctness risk in trying the cached selector, only a cost/time saving when it still works.
220
-
221
- **Locally**, this is a real, ongoing saving: `.tamash-playwright/heals.jsonl` lives on your disk and persists across every `npx playwright test` you run, for as long as you keep working on that checkout — the same broken locator only ever costs one AI call, no matter how many times you re-run your tests afterward.
222
-
223
- **In CI, this only helps *within* one run**, not across separate ones — most CI runners start from a fresh checkout every time, so `.tamash-playwright/` (gitignored, never committed) doesn't carry over from yesterday's run to today's. What actually eliminates repeat AI calls in CI is `apply-heals` merging the real fix into your source — once that lands, the locator isn't broken anymore and healing never needs to run for it again. The cache still earns its keep within a single run, though: if the same broken locator shows up in several tests in one run (a shared Page Object method, say), only the first one pays for a fresh AI call — every other occurrence in that same run reuses it for free. You'll see it in the console line as `provider=cache` with no token count, instead of the real provider name:
224
-
225
- ```
226
- [self-healer] tests/sampletest.spec.ts:13 — locator.fill "User Name Textbox (placeholder "xyz")" -> HEALED [provider=cache, vision=no, actionRecovery=no, suggested="role:textbox:Username"] — locator.fill: Timeout 8000ms exceeded.
227
- ```
228
-
229
- A cache hit doesn't re-log itself (there's nothing new to record), so the log doesn't grow just from repeated confirmations of the same already-known fix — it only grows when a *new* AI call produces a fresh suggestion. If the original heal was flagged `needsReview` (see above), the cached replay keeps flagging it on every run that reuses it, not just the first — a fragile fix doesn't quietly stop being worth a look just because it's been cached.
230
-
231
- ## Making a heal permanent: `apply-heals`
232
-
233
- Runtime healing (including the caching above) never touches your source code — the original locator stays broken in your test file or Page Object forever, healed at runtime on every run, until you fix it yourself. `apply-heals` closes that loop: it rewrites the original broken locator to the selector that actually worked, so the next run doesn't need healing — from cache or otherwise — at all.
234
-
235
- ```sh
236
- npx playwright test # heals at runtime, and records what it healed
237
- npx tamash-playwright apply-heals --dry-run # preview the source changes it would make
238
- npx tamash-playwright apply-heals # write them
239
- ```
240
-
241
- ```
242
- [FIX] src/pages/loginpage.ts:11
243
- - .locator('input[name="username1"]')
244
- + .getByRole("textbox", { name: "Username" })
245
-
246
- 1 fix(es) applied to 1 file(s), 0 skipped.
247
- Review the changes (e.g. `git diff`) before committing.
248
- ```
249
-
250
- A fix derived from a nearby label or a positional fallback (see [above](#when-theres-no-name-to-match-finding-elements-by-structure)) is marked the same way as everywhere else:
251
-
252
- ```
253
- [FIX] [NEEDS REVIEW] tests/employee-id.spec.ts:31
254
- - .getByPlaceholder('Employee')
255
- + .locator('div').filter({ hasText: 'Employee Id' }).getByRole('textbox')
256
- ⚠ No stable identity of its own — durable selector anchors on nearby text instead. Please verify this still targets the right element if the page layout changes.
257
- ```
258
-
259
- A few things worth knowing:
260
-
261
- - **Nothing is applied automatically.** `apply-heals` is a separate, deliberate command — a test run never edits your source on its own.
262
- - **Only real selector fixes are eligible.** A heal only qualifies if it's text/ARIA-based (not just an ephemeral visual-match-only fallback that never resolved to anything reusable — see below) and the locator itself was actually replaced (not an action-recovery heal, where the original locator was already correct and only the action needed help).
263
- - **Only the matched call is touched.** `.describe('...')` and everything else on the line is left exactly as written; only the `.locator(...)`/`.getByRole(...)`/etc. call itself is replaced.
264
- - **Always review the diff before committing** — this rewrites your source files, so treat it like any other automated code change: check `git diff`, run your tests again, and commit deliberately.
265
-
266
- ### Proving a fix actually works: `verify-heals.cjs`
267
-
268
- Every real (non-`--dry-run`) `apply-heals` run also writes `.tamash-playwright/verify-heals.cjs` — a ready-to-run script that re-runs exactly the tests affected by that run, with healing turned off:
269
-
270
- ```sh
271
- node .tamash-playwright/verify-heals.cjs
272
- ```
273
-
274
- A pass proves the rewritten selectors work **on their own** — not just "worked while healing was still there to catch a mistake." It's a plain Node script (portable across Windows/macOS/Linux, no shell-specific env var syntax to get wrong) that calls Playwright's own CLI with your existing config, so there's nothing to configure — just run it after `apply-heals`, locally or as a CI step.
275
-
276
- Every run also writes a before/after report to `.tamash-playwright/` — `apply-heals-report.json` (machine-readable) and `apply-heals-report.md` (human-readable, one section per fix with the exact before/after code). This is what feeds the PR body in the CI pattern below, so a reviewer sees the actual change without digging through logs — worth knowing about even if you only ever run this locally, since it's a persisted record of every fix beyond whatever's still in your scrollback.
277
-
278
- Those two filenames always mean "the latest run" — each run overwrites them. Since `.tamash-playwright/` is gitignored (never committed), that would otherwise mean a second run erases all trace of the first. To keep a real history, every run *also* archives a timestamped copy of both reports, plus the raw `heals.jsonl` that produced them, under `.tamash-playwright/history/` — nothing in there is ever overwritten or deleted by a later run, so it's safe to just leave it accumulating, or clean it out yourself whenever you want.
279
-
280
- ### Running `apply-heals` in CI (sharded or not)
281
-
282
- `apply-heals` never touches git itself — in CI, that means the fix only exists in that job's ephemeral checkout unless something turns it into a real, reviewable change. The recommended shape: a separate job that runs *after* your test job(s), downloads whatever got healed, and opens a PR rather than pushing straight to a branch.
283
-
284
- If your suite runs sharded (`--shard=N/M` across multiple CI machines), each shard only sees its own slice of what got healed — `.tamash-playwright/heals.jsonl` ends up fragmented, one partial file per shard. `--logs-dir` is built for exactly this: point it at a directory containing any number of `heals.jsonl` files, nested however you like, and it merges all of them before planning fixes (deduplicating by keeping only the newest entry per file:line, so two shards healing the same line is harmless):
285
-
286
- ```sh
287
- npx tamash-playwright apply-heals --logs-dir shard-logs
288
- ```
289
-
290
- A GitHub Actions example — each test job uploads its own log as an artifact; a separate job merges them, applies the fixes on a fresh branch, **re-runs the suite against just those fixes to prove they actually work**, and only then opens a PR (labeled with whether verification passed, either way):
291
-
292
- ```yaml
293
- jobs:
294
- test:
295
- # ...your existing test job(s), sharded or not — the part that matters here is that healing
296
- # needs the same variables from Step 2, set as CI secrets/environment variables instead of a
297
- # local .env file (which won't exist on the runner and shouldn't be committed anyway).
298
- runs-on: ubuntu-latest
299
- env:
300
- HEALER_ENABLED: true
301
- HEALER_PROVIDER: ollama
302
- OLLAMA_MODEL: gpt-oss:120b
303
- OLLAMA_API_KEY: ${{ secrets.OLLAMA_API_KEY }} # set via `gh secret set OLLAMA_API_KEY`
304
- steps:
305
- - run: npx playwright test
306
- - uses: actions/upload-artifact@v4
307
- if: ${{ !cancelled() }}
308
- with:
309
- name: heals-log-${{ strategy.job-index }}
310
- path: .tamash-playwright/heals.jsonl
311
- if-no-files-found: ignore
312
-
313
- apply-heals:
314
- needs: test
315
- if: ${{ !cancelled() && github.event_name == 'push' }} # not pull_request — see note below
316
- runs-on: ubuntu-latest
317
- # Two pushes close together would otherwise race on the same heal branch/PR — whichever
318
- # finished last could clobber the other mid-update. Queues instead (never cancels an
319
- # in-progress run) so each one always starts from a clean, fully-finished state.
320
- concurrency:
321
- group: apply-heals-${{ github.ref }}
322
- cancel-in-progress: false
323
- permissions:
324
- contents: write
325
- pull-requests: write
326
- steps:
327
- - uses: actions/checkout@v4
328
- - uses: actions/setup-node@v4
329
- with: { node-version: lts/* }
330
- - run: npm ci
331
-
332
- - uses: actions/download-artifact@v4
333
- with:
334
- pattern: heals-log-*
335
- path: shard-logs
336
- continue-on-error: true # no artifact at all when nothing needed healing — the common case
337
- - run: npx tamash-playwright apply-heals --logs-dir shard-logs
338
-
339
- - name: Check whether any fixes were applied
340
- id: check
341
- run: echo "changed=$(git diff --quiet || echo true)" >> "$GITHUB_OUTPUT"
342
-
343
- # apply-heals already wrote .tamash-playwright/verify-heals.cjs — it knows exactly which
344
- # tests were affected and sets HEALER_ENABLED=false itself, so there's nothing to parse or
345
- # configure here. A pass proves the *written* fix works standalone — leaving healing on
346
- # could let a still-broken selector get silently re-healed at runtime again, reporting green
347
- # without ever proving the applied source fix was actually correct.
348
- - name: Verify the healed selectors work on their own
349
- id: verify
350
- if: steps.check.outputs.changed == 'true'
351
- run: node .tamash-playwright/verify-heals.cjs
352
- continue-on-error: true
353
-
354
- # Captured via the step's own id so Compose PR body can link straight to it — otherwise it's
355
- # uploaded correctly but nobody reviewing the PR would know it exists.
356
- - name: Upload verification report
357
- id: upload-verification-report
358
- if: steps.check.outputs.changed == 'true' && !cancelled()
359
- uses: actions/upload-artifact@v4
360
- with:
361
- name: apply-heals-verification-report
362
- path: playwright-report/
363
-
364
- # apply-heals already wrote .tamash-playwright/apply-heals-report.md with a before/after per
365
- # fix — this prepends the verification result so the PR body is one linked story (what
366
- # broke, what changed, whether it's proven to work) instead of three things to go find.
367
- - name: Compose PR body
368
- if: steps.check.outputs.changed == 'true'
369
- run: |
370
- {
371
- echo "Auto-generated by \`tamash-playwright apply-heals\` after self-healing kicked in during CI."
372
- echo ""
373
- if [ "${{ steps.verify.outcome }}" = "success" ]; then
374
- echo "**Verification run (healing disabled): ✅ passed.**"
375
- else
376
- echo "**Verification run (healing disabled): ⚠️ FAILED — review carefully before merging.**"
377
- fi
378
- echo "[Full test execution report](${{ steps.upload-verification-report.outputs.artifact-url }})"
379
- echo ""
380
- cat .tamash-playwright/apply-heals-report.md
381
- } > .tamash-playwright/pr-body.md
382
-
383
- # branch is suffixed with the ref so two different branches healing at the same time (if
384
- # your trigger isn't restricted to a single branch) never fight over one physical heal
385
- # branch — the concurrency group above only serializes runs on the *same* ref.
386
- - name: Open PR with healed selectors
387
- if: steps.check.outputs.changed == 'true'
388
- uses: peter-evans/create-pull-request@v6
389
- with:
390
- commit-message: "fix: apply self-healed selectors from CI"
391
- title: "Apply self-healed selectors on ${{ github.ref_name }} (verification ${{ steps.verify.outcome == 'success' && 'passed' || 'FAILED' }})"
392
- body-path: .tamash-playwright/pr-body.md
393
- branch: tamash-playwright/apply-heals-${{ github.ref_name }}
394
- delete-branch: true
395
-
396
- - name: Fail the job if verification didn't pass
397
- if: steps.check.outputs.changed == 'true' && steps.verify.outcome != 'success'
398
- run: exit 1 # PR is still opened above for review — this just keeps CI status honest
399
- ```
400
-
401
- One more thing worth flagging if you copy this: `.tamash-playwright/` needs to be in your `.gitignore`. `peter-evans/create-pull-request` commits whatever differs from `HEAD` in the working tree — without the ignore, the report/log files themselves would get swept into the PR alongside the actual source fix.
402
-
403
- A few choices worth calling out:
404
-
405
- - **Gated on `push`, not `pull_request`.** A `pull_request` run from a fork gets a read-only `GITHUB_TOKEN` (so it couldn't open a PR anyway), and "open a PR to fix this still-open PR" isn't a sensible flow regardless. Running after merges to `main`/`master` avoids both problems.
406
- - **The PR is opened either way**, verification passed or failed — a failed verification is still worth a human's attention (maybe the fix is right and something else was flaky); it just gets an honest label instead of a silent false-positive. The final step fails the *job* itself when verification fails, so CI status stays truthful even though the PR still exists for review.
407
- - [`peter-evans/create-pull-request`](https://github.com/peter-evans/create-pull-request) is a no-op if there's nothing to commit, so the job is safe to run on every push — it only ever opens a PR when there's an actual fix to review, and reuses the same branch/PR on subsequent runs rather than piling up duplicates.
408
-
409
- **Want a browsable dashboard, not just PR diffs and CI logs?** You can add a third job that publishes a combined report — the initial run, what got healed, and the post-fix verification, plus every past run archived and browsable — to GitHub Pages. See the "Publishing a healing dashboard to GitHub Pages" section of [usage.md](usage.md) for the full recipe.
410
-
411
- ## Checking what actually happened
412
-
413
- Every healing attempt — whether it succeeded or not — shows up in Playwright's own HTML report (`npx playwright show-report`), no separate report to check:
414
-
415
- - An annotation on the test summarizing what happened, e.g. `Recovered using ollama:gpt-oss:120b (role:button:Submit)` — plus a separate `self-heal-needs-review` annotation when the fix is the kind worth a second look (see [above](#when-a-fix-needs-a-second-look)).
416
- - A JSON attachment with the full detail: which provider was used, whether the vision or action-recovery fallback was involved, the AI's suggested selector, token cost, and — if it didn't heal — which stage it stopped at (e.g. `ai_declined`, `replay_failed`).
417
- - Exactly where in your own code the locator was created — a test file or a Page Object class, whichever it really is — so you know which line to go fix even if you never look at the healing report again.
418
-
419
- The same detail is also printed to the console as it happens, one line per attempt:
420
-
421
- ```
422
- [self-healer] src/pages/loginpage.ts:11 — locator.fill "Username Textbox" -> HEALED [provider=ollama:gpt-oss:120b, vision=no, actionRecovery=no, suggested="role:textbox:Username", 620 tokens (489 input + 131 output)] — locator.fill: Timeout 8000ms exceeded.
423
- ```
424
-
425
- ## License
426
-
427
- Free to use, including commercially. The source code may not be copied, modified, redistributed, or resold without prior written permission. See the LICENSE file included in this package for the full terms.
428
-
429
- ## Support
430
-
431
- For questions or concerns, contact us at support@vibetestq.com.
1
+ # tamash-playwright
2
+
3
+ `tamash-playwright` is a plug and play self-healing solution for any Playwright test framework. All you need to do is install the package, update your AI API key details, and import `test` from `tamash-playwright`.
4
+
5
+ That's it. No code changes required if you're following standard Playwright best practices.
6
+
7
+ ### Why you need this
8
+
9
+ Websites change often. A button gets renamed or moved, and your test can't find it anymore — even though the app still works fine for real users. Normally, that just means a broken test.
10
+
11
+ `tamash-playwright` fixes this automatically. When a test can't find an element, it asks an AI model to find it on the current page and tries again. If it succeeds, your test keeps going. If not, it fails normally, just like before.
12
+
13
+ **Want to see it working before you set it up yourself?** Clone the sample repo — [github.com/qtpsudhakarproducts/tamash-playwright-typescript-playwright](https://github.com/qtpsudhakarproducts/tamash-playwright-typescript-playwright) — a full worked example with both a plain-locator test and a Page Object Model test, an intentionally broken selector, and step-by-step setup instructions.
14
+
15
+ **Looking for the complete reference** — caching, `apply-heals`, sharded-CI and PR automation, every env var and CLI flag? See [usage.md](usage.md).
16
+
17
+ Here are the detailed steps to use this package.
18
+
19
+ ## Step 1: Install it
20
+
21
+ ```sh
22
+ npm install tamash-playwright
23
+ ```
24
+
25
+ You also need Playwright's own test package, if you don't already have it:
26
+
27
+ ```sh
28
+ npm install -D @playwright/test
29
+ ```
30
+
31
+ ## Step 2: Connect an AI model
32
+
33
+ `tamash-playwright` needs an AI model to decide where a broken element actually went. Pick one of Ollama, OpenAI, Anthropic (Claude), or Google Gemini, and give it an API key.
34
+
35
+ Create a file named `.env` in your project folder:
36
+
37
+ ```sh
38
+ # Master on/off switch. Leave this as true, or remove the line entirely.
39
+ HEALER_ENABLED=true
40
+
41
+ # Pick one: ollama | openai | anthropic | gemini
42
+ HEALER_PROVIDER=ollama
43
+
44
+ # Optional, off by default — see "Action recovery" below.
45
+ # HEALER_ACTION_RECOVERY_ENABLED=true
46
+
47
+ # --- Ollama Cloud (https://ollama.com) ---
48
+ OLLAMA_MODEL=gpt-oss:120b
49
+ OLLAMA_API_KEY=
50
+
51
+ # --- OpenAI ---
52
+ # OPENAI_MODEL=gpt-4.1-mini
53
+ # OPENAI_API_KEY=
54
+
55
+ # --- Anthropic (Claude) ---
56
+ # ANTHROPIC_MODEL=claude-haiku-4-5
57
+ # ANTHROPIC_API_KEY=
58
+
59
+ # --- Google Gemini ---
60
+ # GEMINI_MODEL=
61
+ # GEMINI_API_KEY=
62
+ ```
63
+
64
+ Just fill in the API key and model for whichever one you want to use, and leave the rest as-is (or delete them).
65
+
66
+ ### Getting a free Ollama key (fastest way to get started)
67
+
68
+ Ollama Cloud is a quick, free way to get an API key without signing up for OpenAI/Anthropic/Gemini billing.
69
+
70
+ 1. Go to [ollama.com](https://ollama.com/) and create an account.
71
+ 2. Once signed in, go to [ollama.com/settings/keys](https://ollama.com/settings/keys).
72
+ 3. Create a new API key and copy it.
73
+ 4. Paste it into your `.env` file:
74
+
75
+ ```sh
76
+ HEALER_ENABLED=true
77
+ HEALER_PROVIDER=ollama
78
+ OLLAMA_MODEL=gpt-oss:120b
79
+ OLLAMA_API_KEY=paste_your_key_here
80
+ ```
81
+
82
+ That's all you need — no other variables required.
83
+
84
+ ### Important: set `actionTimeout` in your `playwright.config.ts`
85
+
86
+ By default, Playwright lets a broken locator retry silently for your *entire* test timeout before it ever throws an error — which means self-healing never gets a turn at all, since it only kicks in once an action actually fails. Set `actionTimeout` to something well below your test timeout so a broken locator fails fast, leaving real time for healing to run:
87
+
88
+ ```ts
89
+ export default defineConfig({
90
+ timeout: 60000, // your overall test timeout
91
+ use: {
92
+ actionTimeout: 8000, // must be comfortably less than the test timeout above
93
+ },
94
+ });
95
+ ```
96
+
97
+ Without this, healing attempts will show `stage=no_snapshot` in the console and never recover anything — not because healing failed, but because it never had time to run before the whole test was torn down.
98
+
99
+ ## Step 3: Check your setup
100
+
101
+ Run the built-in doctor command to confirm everything's wired up correctly before you rely on it:
102
+
103
+ ```sh
104
+ npx tamash-playwright doctor
105
+ ```
106
+
107
+ It checks:
108
+
109
+ 1. **AI connectivity** — confirms `HEALER_ENABLED`/`HEALER_PROVIDER` are set correctly and actually calls your configured provider to make sure the API key and model work.
110
+ 2. **`actionTimeout` configuration** — checks your `playwright.config.ts` for an `actionTimeout` set well below your test `timeout` (see above); flags it if missing or too close to the test timeout, since that silently starves self-healing of any time to run.
111
+ 3. **Action recovery status** — whether `HEALER_ACTION_RECOVERY_ENABLED` is on (see below).
112
+ 4. **Vision capability** — whether your configured model is expected to support the screenshot-based fallback (see below), based on its name.
113
+ 5. **Missing `.describe()` labels** — scans your test files (`tests/` by default, or pass `--dir <path>`) for locators that don't have a `.describe('...')` label, and flags the ones most worth fixing (raw CSS/XPath selectors first).
114
+ 6. **Locators written directly in test files** — flags any locator defined inline in a test rather than inside a Page Object class, which is a Playwright best practice regardless of self-healing: it keeps tests readable and means a UI change only needs a fix in one place.
115
+
116
+ If it finds issues, the fastest fix is to open the project in an AI coding assistant (Claude Code, Cursor, GitHub Copilot, etc.) and ask it to address what it flagged — add `.describe()` calls, or extract locators into Page Object classes. You can also add a standing rule to that assistant's instructions/skill file (e.g. `CLAUDE.md`, `.cursor/rules`, `.github/copilot-instructions.md`) so it follows both practices automatically on any new test code going forward.
117
+
118
+ ## Step 4: Use it in your tests
119
+
120
+ Change one line at the top of your test file — everything else about how you write tests stays exactly the same:
121
+
122
+ ```ts
123
+ // Before
124
+ import { test, expect } from '@playwright/test';
125
+
126
+ // After
127
+ import { test, expect } from 'tamash-playwright';
128
+ ```
129
+
130
+ That's it. Write your tests as normal:
131
+
132
+ ```ts
133
+ import { test, expect } from 'tamash-playwright';
134
+
135
+ test('logs in', async ({ page }) => {
136
+ await page.goto('/');
137
+ const txtUserName = page.locator('input[name="username"]').describe('User Name Textbox');
138
+ await txtUserName.fill('testadmin');
139
+
140
+ const txtPassword = page.locator('input[placeholder="Password"]').describe('Password Textbox');
141
+ await txtPassword.fill('secret');
142
+
143
+ const btnLogin = page.locator('button[type="submit"]').describe('Login Button');
144
+ await btnLogin.click();
145
+
146
+ await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
147
+ });
148
+ ```
149
+
150
+ ### A quick tip for better results
151
+
152
+ If you're using plain CSS selectors (like `page.locator('input[name="username"]')`) rather than Playwright's more descriptive locators (`getByRole`, `getByPlaceholder`, etc.), it helps to add a short, human-readable label so the healer knows what it's actually looking for. Chain `.describe('...')` right onto the locator:
153
+
154
+ ```ts
155
+ test('login test using CSS Selectors', async ({ page }) => {
156
+ await page.goto('https://example.com/auth/login');
157
+
158
+ const txtUserName = page.locator('input[name="username"]').describe('User Name Textbox');
159
+ await txtUserName.fill('testadmin');
160
+
161
+ const txtPassword = page.locator('input[placeholder="Password"]').describe('Password Textbox');
162
+ await txtPassword.fill('secret');
163
+
164
+ const btnLogin = page.locator('button[type="submit"]').describe('Login Button');
165
+ await btnLogin.click();
166
+
167
+ await expect(page.locator('h6')).toHaveText('Dashboard');
168
+ });
169
+ ```
170
+
171
+ This step is optional, but recommended — without it, the healer has to guess purely from a broken CSS selector, which gives it a lot less to work with.
172
+
173
+ ## Local vs CI, at a glance
174
+
175
+ Everything below applies identically in both places — same package, same behavior — but a few things are worth knowing up front before you get to the details:
176
+
177
+ | | Locally | In CI |
178
+ | --- | --- | --- |
179
+ | **Healing** | Runs automatically on every `npx playwright test`, using the `.env` file from Step 2. | Runs automatically the same way — set the same variables as CI secrets/environment variables on your test job instead of a `.env` file (there's a real example in [Running `apply-heals` in CI](#running-apply-heals-in-ci-sharded-or-not)). |
180
+ | **Caching** ([below](#not-paying-for-the-same-heal-twice)) | Persists on disk across every run — a real, ongoing saving the longer you keep working. | Only helps *within* one run — most runners start from a fresh checkout each time, so it doesn't carry over between separate CI runs. `apply-heals` landing the real fix is what actually stops repeat AI calls in CI, not the cache. |
181
+ | **`apply-heals`** ([below](#making-a-heal-permanent-apply-heals)) | Run manually whenever you want, preview with `--dry-run`, review the diff yourself, commit when ready. | Runs automatically after your test job and opens a PR for review — nothing ever auto-commits to your branch. |
182
+
183
+ ## What else it heals — no extra setup needed
184
+
185
+ Beyond a single broken `click`/`fill`/`getByRole` on the main page, all of this works automatically once you've done Steps 1–2:
186
+
187
+ - **Popups and extra tabs.** A page opened via `context.newPage()`, `window.open`, or a `target="_blank"` link is just as healing-aware as your main `page` — no manual wrapping needed.
188
+ - **Elements inside `<iframe>`s.** `page.frameLocator('#my-iframe')` and anything chained off it heals the same way, scoped correctly to the iframe's own document.
189
+ - **Most of the Playwright API surface**, not just clicks and fills — `check`, `selectOption`, `dragTo`, `dispatchEvent`, read methods like `textContent`/`getAttribute`/`isChecked`, `screenshot`, and more. Methods that can't be safely healed by guessing a replacement element (`dragTo`, `drop`) are still reported honestly on failure, they're just never silently retried with a different element.
190
+
191
+ ## When there's no name to match: finding elements by structure
192
+
193
+ Sometimes the broken element has no useful identity of its own — a plain `<input>` with no name, no working placeholder, and a label that's visually right next to it but never actually linked (no `<label for>`, no `aria-labelledby`). A human finds it instantly by sight; matching purely on accessible name has nothing to grab onto.
194
+
195
+ For exactly these cases, `tamash-playwright` reads a structural map of the whole page — not just a flat list of named elements — so the AI can point at the exact element even when it has no name of its own. Once it has that, a separate step (no extra AI call) works out the most stable way to describe it for next time:
196
+
197
+ - If the element has real identity — an id, test id, accessible role and name, label, or placeholder — that's used directly.
198
+ - If not, but a label sits right next to it in the page's own structure, the fix anchors on that nearby text instead — and only after confirming it genuinely points at the *same* element, not just one that happens to match.
199
+ - If neither applies, it falls back to whatever Playwright's own locator-generation logic can produce, even a positional one — but flags it for review rather than treating it as fully trusted (see below), since a selector that depends on element order can silently point at the wrong thing later if the page changes.
200
+
201
+ You don't configure any of this — it happens automatically, and every candidate is verified against the live page (real DOM identity, not just "a match was found") before anything gets used.
202
+
203
+ ### When a fix needs a second look
204
+
205
+ Not every healed selector is equally durable. One with a real id, test id, or accessible name is about as solid as anything a human would write by hand. One that only had a nearby label or a positional fallback to work with is correct *right now*, but worth a glance before you fully rely on it — the page could change in a way that fools it later.
206
+
207
+ `tamash-playwright` tells you which is which — look for `needsReview=yes` in the console line, a `self-heal-needs-review` annotation in the HTML report, or a `[NEEDS REVIEW]` tag in `apply-heals`'s output (see below). Nothing is blocked or held back because of it; it's a hint about where to look first, not a gate.
208
+
209
+ ## When text alone isn't enough: vision fallback
210
+
211
+ Sometimes an element has nothing useful to match on by text — an icon-only button with no label, or several visually distinct elements that all look identical in the accessibility tree. If your configured model supports image input (e.g. `gpt-4o`, `claude-haiku-4-5`, `gemini-3.6-flash`), `tamash-playwright` automatically falls back to a screenshot-based search after the normal text-based attempt fails — no separate setup, it just uses the same provider and API key from Step 2. Run `npx tamash-playwright doctor` to check whether your configured model is expected to support this.
212
+
213
+ ## Action recovery (optional)
214
+
215
+ Occasionally a locator heals correctly — the AI found the right element — but the *action* on it still fails, e.g. it's covered by an overlay or needs scrolling into view first. Set `HEALER_ACTION_RECOVERY_ENABLED=true` to let the AI pick a recovery tactic from a fixed, safe set (scroll into view, retry bypassing actionability checks, wait briefly and retry, or dispatch the DOM event directly) before giving up. It's off by default since it's a second, more speculative layer of intervention beyond selector healing — the AI only ever picks from that fixed menu, it never decides how to interact with the page on its own.
216
+
217
+ ## Not paying for the same heal twice
218
+
219
+ Once a locator heals successfully, `tamash-playwright` remembers the fix in `.tamash-playwright/heals.jsonl` — the same file `apply-heals` reads (see below). The next time that exact locator breaks the same way, it tries the previously-confirmed selector *first*, with no ARIA snapshot and no AI call. Only if that no longer works (the page changed again) does it fall through to a fresh snapshot-and-AI-call, exactly as before — so there's no correctness risk in trying the cached selector, only a cost/time saving when it still works.
220
+
221
+ **Locally**, this is a real, ongoing saving: `.tamash-playwright/heals.jsonl` lives on your disk and persists across every `npx playwright test` you run, for as long as you keep working on that checkout — the same broken locator only ever costs one AI call, no matter how many times you re-run your tests afterward.
222
+
223
+ **In CI, this only helps *within* one run**, not across separate ones — most CI runners start from a fresh checkout every time, so `.tamash-playwright/` (gitignored, never committed) doesn't carry over from yesterday's run to today's. What actually eliminates repeat AI calls in CI is `apply-heals` merging the real fix into your source — once that lands, the locator isn't broken anymore and healing never needs to run for it again. The cache still earns its keep within a single run, though: if the same broken locator shows up in several tests in one run (a shared Page Object method, say), only the first one pays for a fresh AI call — every other occurrence in that same run reuses it for free. You'll see it in the console line as `provider=cache` with no token count, instead of the real provider name:
224
+
225
+ ```
226
+ [self-healer] tests/sampletest.spec.ts:13 — locator.fill "User Name Textbox (placeholder "xyz")" -> HEALED [provider=cache, vision=no, actionRecovery=no, suggested="role:textbox:Username"] — locator.fill: Timeout 8000ms exceeded.
227
+ ```
228
+
229
+ A cache hit doesn't re-log itself (there's nothing new to record), so the log doesn't grow just from repeated confirmations of the same already-known fix — it only grows when a *new* AI call produces a fresh suggestion. If the original heal was flagged `needsReview` (see above), the cached replay keeps flagging it on every run that reuses it, not just the first — a fragile fix doesn't quietly stop being worth a look just because it's been cached.
230
+
231
+ ## Making a heal permanent: `apply-heals`
232
+
233
+ Runtime healing (including the caching above) never touches your source code — the original locator stays broken in your test file or Page Object forever, healed at runtime on every run, until you fix it yourself. `apply-heals` closes that loop: it rewrites the original broken locator to the selector that actually worked, so the next run doesn't need healing — from cache or otherwise — at all.
234
+
235
+ ```sh
236
+ npx playwright test # heals at runtime, and records what it healed
237
+ npx tamash-playwright apply-heals --dry-run # preview the source changes it would make
238
+ npx tamash-playwright apply-heals # write them
239
+ ```
240
+
241
+ ```
242
+ [FIX] src/pages/loginpage.ts:11
243
+ - .locator('input[name="username1"]')
244
+ + .getByRole("textbox", { name: "Username" })
245
+
246
+ 1 fix(es) applied to 1 file(s), 0 skipped.
247
+ Review the changes (e.g. `git diff`) before committing.
248
+ ```
249
+
250
+ A fix derived from a nearby label or a positional fallback (see [above](#when-theres-no-name-to-match-finding-elements-by-structure)) is marked the same way as everywhere else:
251
+
252
+ ```
253
+ [FIX] [NEEDS REVIEW] tests/employee-id.spec.ts:31
254
+ - .getByPlaceholder('Employee')
255
+ + .locator('div').filter({ hasText: 'Employee Id' }).getByRole('textbox')
256
+ ⚠ No stable identity of its own — durable selector anchors on nearby text instead. Please verify this still targets the right element if the page layout changes.
257
+ ```
258
+
259
+ A few things worth knowing:
260
+
261
+ - **Nothing is applied automatically.** `apply-heals` is a separate, deliberate command — a test run never edits your source on its own.
262
+ - **Only real selector fixes are eligible.** A heal only qualifies if it's text/ARIA-based (not just an ephemeral visual-match-only fallback that never resolved to anything reusable — see below) and the locator itself was actually replaced (not an action-recovery heal, where the original locator was already correct and only the action needed help).
263
+ - **Only the matched call is touched.** `.describe('...')` and everything else on the line is left exactly as written; only the `.locator(...)`/`.getByRole(...)`/etc. call itself is replaced.
264
+ - **Always review the diff before committing** — this rewrites your source files, so treat it like any other automated code change: check `git diff`, run your tests again, and commit deliberately.
265
+
266
+ ### Proving a fix actually works: `verify-heals.cjs`
267
+
268
+ Every real (non-`--dry-run`) `apply-heals` run also writes `.tamash-playwright/verify-heals.cjs` — a ready-to-run script that re-runs exactly the tests affected by that run, with healing turned off:
269
+
270
+ ```sh
271
+ node .tamash-playwright/verify-heals.cjs
272
+ ```
273
+
274
+ A pass proves the rewritten selectors work **on their own** — not just "worked while healing was still there to catch a mistake." It's a plain Node script (portable across Windows/macOS/Linux, no shell-specific env var syntax to get wrong) that calls Playwright's own CLI with your existing config, so there's nothing to configure — just run it after `apply-heals`, locally or as a CI step.
275
+
276
+ Every run also writes a before/after report to `.tamash-playwright/` — `apply-heals-report.json` (machine-readable) and `apply-heals-report.md` (human-readable, one section per fix with the exact before/after code). This is what feeds the PR body in the CI pattern below, so a reviewer sees the actual change without digging through logs — worth knowing about even if you only ever run this locally, since it's a persisted record of every fix beyond whatever's still in your scrollback.
277
+
278
+ Those two filenames always mean "the latest run" — each run overwrites them. Since `.tamash-playwright/` is gitignored (never committed), that would otherwise mean a second run erases all trace of the first. To keep a real history, every run *also* archives a timestamped copy of both reports, plus the raw `heals.jsonl` that produced them, under `.tamash-playwright/history/` — nothing in there is ever overwritten or deleted by a later run, so it's safe to just leave it accumulating, or clean it out yourself whenever you want.
279
+
280
+ ### Running `apply-heals` in CI (sharded or not)
281
+
282
+ `apply-heals` never touches git itself — in CI, that means the fix only exists in that job's ephemeral checkout unless something turns it into a real, reviewable change. The recommended shape: a separate job that runs *after* your test job(s), downloads whatever got healed, and opens a PR rather than pushing straight to a branch.
283
+
284
+ If your suite runs sharded (`--shard=N/M` across multiple CI machines), each shard only sees its own slice of what got healed — `.tamash-playwright/heals.jsonl` ends up fragmented, one partial file per shard. `--logs-dir` is built for exactly this: point it at a directory containing any number of `heals.jsonl` files, nested however you like, and it merges all of them before planning fixes (deduplicating by keeping only the newest entry per file:line, so two shards healing the same line is harmless):
285
+
286
+ ```sh
287
+ npx tamash-playwright apply-heals --logs-dir shard-logs
288
+ ```
289
+
290
+ A GitHub Actions example — each test job uploads its own log as an artifact; a separate job merges them, applies the fixes on a fresh branch, **re-runs the suite against just those fixes to prove they actually work**, and only then opens a PR (labeled with whether verification passed, either way):
291
+
292
+ ```yaml
293
+ jobs:
294
+ test:
295
+ # ...your existing test job(s), sharded or not — the part that matters here is that healing
296
+ # needs the same variables from Step 2, set as CI secrets/environment variables instead of a
297
+ # local .env file (which won't exist on the runner and shouldn't be committed anyway).
298
+ runs-on: ubuntu-latest
299
+ env:
300
+ HEALER_ENABLED: true
301
+ HEALER_PROVIDER: ollama
302
+ OLLAMA_MODEL: gpt-oss:120b
303
+ OLLAMA_API_KEY: ${{ secrets.OLLAMA_API_KEY }} # set via `gh secret set OLLAMA_API_KEY`
304
+ steps:
305
+ - run: npx playwright test
306
+ - uses: actions/upload-artifact@v4
307
+ if: ${{ !cancelled() }}
308
+ with:
309
+ name: heals-log-${{ strategy.job-index }}
310
+ path: .tamash-playwright/heals.jsonl
311
+ if-no-files-found: ignore
312
+
313
+ apply-heals:
314
+ needs: test
315
+ if: ${{ !cancelled() && github.event_name == 'push' }} # not pull_request — see note below
316
+ runs-on: ubuntu-latest
317
+ # Two pushes close together would otherwise race on the same heal branch/PR — whichever
318
+ # finished last could clobber the other mid-update. Queues instead (never cancels an
319
+ # in-progress run) so each one always starts from a clean, fully-finished state.
320
+ concurrency:
321
+ group: apply-heals-${{ github.ref }}
322
+ cancel-in-progress: false
323
+ permissions:
324
+ contents: write
325
+ pull-requests: write
326
+ steps:
327
+ - uses: actions/checkout@v4
328
+ - uses: actions/setup-node@v4
329
+ with: { node-version: lts/* }
330
+ - run: npm ci
331
+
332
+ - uses: actions/download-artifact@v4
333
+ with:
334
+ pattern: heals-log-*
335
+ path: shard-logs
336
+ continue-on-error: true # no artifact at all when nothing needed healing — the common case
337
+ - run: npx tamash-playwright apply-heals --logs-dir shard-logs
338
+
339
+ - name: Check whether any fixes were applied
340
+ id: check
341
+ run: echo "changed=$(git diff --quiet || echo true)" >> "$GITHUB_OUTPUT"
342
+
343
+ # apply-heals already wrote .tamash-playwright/verify-heals.cjs — it knows exactly which
344
+ # tests were affected and sets HEALER_ENABLED=false itself, so there's nothing to parse or
345
+ # configure here. A pass proves the *written* fix works standalone — leaving healing on
346
+ # could let a still-broken selector get silently re-healed at runtime again, reporting green
347
+ # without ever proving the applied source fix was actually correct.
348
+ - name: Verify the healed selectors work on their own
349
+ id: verify
350
+ if: steps.check.outputs.changed == 'true'
351
+ run: node .tamash-playwright/verify-heals.cjs
352
+ continue-on-error: true
353
+
354
+ # Captured via the step's own id so Compose PR body can link straight to it — otherwise it's
355
+ # uploaded correctly but nobody reviewing the PR would know it exists.
356
+ - name: Upload verification report
357
+ id: upload-verification-report
358
+ if: steps.check.outputs.changed == 'true' && !cancelled()
359
+ uses: actions/upload-artifact@v4
360
+ with:
361
+ name: apply-heals-verification-report
362
+ path: playwright-report/
363
+
364
+ # apply-heals already wrote .tamash-playwright/apply-heals-report.md with a before/after per
365
+ # fix — this prepends the verification result so the PR body is one linked story (what
366
+ # broke, what changed, whether it's proven to work) instead of three things to go find.
367
+ - name: Compose PR body
368
+ if: steps.check.outputs.changed == 'true'
369
+ run: |
370
+ {
371
+ echo "Auto-generated by \`tamash-playwright apply-heals\` after self-healing kicked in during CI."
372
+ echo ""
373
+ if [ "${{ steps.verify.outcome }}" = "success" ]; then
374
+ echo "**Verification run (healing disabled): ✅ passed.**"
375
+ else
376
+ echo "**Verification run (healing disabled): ⚠️ FAILED — review carefully before merging.**"
377
+ fi
378
+ echo "[Full test execution report](${{ steps.upload-verification-report.outputs.artifact-url }})"
379
+ echo ""
380
+ cat .tamash-playwright/apply-heals-report.md
381
+ } > .tamash-playwright/pr-body.md
382
+
383
+ # branch is suffixed with the ref so two different branches healing at the same time (if
384
+ # your trigger isn't restricted to a single branch) never fight over one physical heal
385
+ # branch — the concurrency group above only serializes runs on the *same* ref.
386
+ - name: Open PR with healed selectors
387
+ if: steps.check.outputs.changed == 'true'
388
+ uses: peter-evans/create-pull-request@v6
389
+ with:
390
+ commit-message: "fix: apply self-healed selectors from CI"
391
+ title: "Apply self-healed selectors on ${{ github.ref_name }} (verification ${{ steps.verify.outcome == 'success' && 'passed' || 'FAILED' }})"
392
+ body-path: .tamash-playwright/pr-body.md
393
+ branch: tamash-playwright/apply-heals-${{ github.ref_name }}
394
+ delete-branch: true
395
+
396
+ - name: Fail the job if verification didn't pass
397
+ if: steps.check.outputs.changed == 'true' && steps.verify.outcome != 'success'
398
+ run: exit 1 # PR is still opened above for review — this just keeps CI status honest
399
+ ```
400
+
401
+ One more thing worth flagging if you copy this: `.tamash-playwright/` needs to be in your `.gitignore`. `peter-evans/create-pull-request` commits whatever differs from `HEAD` in the working tree — without the ignore, the report/log files themselves would get swept into the PR alongside the actual source fix.
402
+
403
+ A few choices worth calling out:
404
+
405
+ - **Gated on `push`, not `pull_request`.** A `pull_request` run from a fork gets a read-only `GITHUB_TOKEN` (so it couldn't open a PR anyway), and "open a PR to fix this still-open PR" isn't a sensible flow regardless. Running after merges to `main`/`master` avoids both problems.
406
+ - **The PR is opened either way**, verification passed or failed — a failed verification is still worth a human's attention (maybe the fix is right and something else was flaky); it just gets an honest label instead of a silent false-positive. The final step fails the *job* itself when verification fails, so CI status stays truthful even though the PR still exists for review.
407
+ - [`peter-evans/create-pull-request`](https://github.com/peter-evans/create-pull-request) is a no-op if there's nothing to commit, so the job is safe to run on every push — it only ever opens a PR when there's an actual fix to review, and reuses the same branch/PR on subsequent runs rather than piling up duplicates.
408
+
409
+ **Want a browsable dashboard, not just PR diffs and CI logs?** You can add a third job that publishes a combined report — the initial run, what got healed, and the post-fix verification, plus every past run archived and browsable — to GitHub Pages. See the "Publishing a healing dashboard to GitHub Pages" section of [usage.md](usage.md) for the full recipe.
410
+
411
+ ## Checking what actually happened
412
+
413
+ Every healing attempt — whether it succeeded or not — shows up in Playwright's own HTML report (`npx playwright show-report`), no separate report to check:
414
+
415
+ - An annotation on the test summarizing what happened, e.g. `Recovered using ollama:gpt-oss:120b (role:button:Submit)` — plus a separate `self-heal-needs-review` annotation when the fix is the kind worth a second look (see [above](#when-a-fix-needs-a-second-look)).
416
+ - A JSON attachment with the full detail: which provider was used, whether the vision or action-recovery fallback was involved, the AI's suggested selector, token cost, and — if it didn't heal — which stage it stopped at (e.g. `ai_declined`, `replay_failed`).
417
+ - Exactly where in your own code the locator was created — a test file or a Page Object class, whichever it really is — so you know which line to go fix even if you never look at the healing report again.
418
+
419
+ The same detail is also printed to the console as it happens, one line per attempt:
420
+
421
+ ```
422
+ [self-healer] src/pages/loginpage.ts:11 — locator.fill "Username Textbox" -> HEALED [provider=ollama:gpt-oss:120b, vision=no, actionRecovery=no, suggested="role:textbox:Username", 620 tokens (489 input + 131 output)] — locator.fill: Timeout 8000ms exceeded.
423
+ ```
424
+
425
+ ## License
426
+
427
+ Free to use, including commercially. The source code may not be copied, modified, redistributed, or resold without prior written permission. See the LICENSE file included in this package for the full terms.
428
+
429
+ ## Support
430
+
431
+ For questions or concerns, contact us at support@vibetestq.com.