elementus-ai 1.1.1 → 1.4.0

package/CHANGELOG.md

@@ -0,0 +1,49 @@
+# Changelog
+
+All notable changes are documented here. This project adheres to [Semantic Versioning](https://semver.org).
+
+## 1.4.0
+
+### Added
+
+- `HealEvent.resolved` — `{ tag, text, role }` of the element a heal resolved to (when known), so a
+  consumer can compose a **suggested replacement locator** (e.g. `getByRole(role, { name: text })`) and
+  show "selector X failed → replace with Y" in reports. Absent on coordinate-only vision heals.
+
+### Changed
+
+- Additive on the existing `onHeal` event; no behavior change.
+
+## 1.3.0
+
+### Added
+
+- `onHeal(event)` config callback — fires once per heal with `{ description, selector, method, framework }`
+  so consumers can surface healed locators (report annotations, CI fail-on-heal). Framework-agnostic:
+  elementus emits the event; wiring it to a reporter is consumer-side (README "Detecting heals" has
+  Playwright + WebdriverIO/Appium recipes).
+- `HealEvent` exported type (`index.d.ts`); `onHeal` added to `ElementusOptions`.
+- Test `T47` covering onHeal; docs in README, `docs/API.md`, `docs/ARCHITECTURE.md`.
+
+### Changed
+
+- No change to existing behavior. `onHeal` is opt-in; unset ⇒ byte-for-byte the same as 1.2.0
+  (zero-overhead path untouched, single export preserved).
+
+## 1.2.0
+
+### Added
+
+- Bundled TypeScript type definitions (`index.d.ts`), exposed via the package `types` field — no `@types/...` package or `declare module` shim required.
+- Exported types: `createElementus`, `ElementusOptions`, `Elementus`, `ElementusPage`, `AiLocatorOptions`.
+- `ElementusPage` types `page.locator(selector, { ai })` by extending Playwright's own locator options, so the `{ ai }` hint type-checks while plain locators and native options keep working.
+- `@playwright/test` types resolve as an optional peer — WDIO/Appium-only projects don't need Playwright installed.
+- README: new **TypeScript** section (typed fixture + Page Object Model patterns); the One-Prompt Setup is now TypeScript-aware.
+
+### Changed
+
+- No runtime changes. `elementus.js` is untouched; this release is type definitions and documentation only.
+
+## 1.1.1
+
+- Previous release.

package/README.md

@@ -24,6 +24,7 @@ I just installed the npm package "elementus-ai" — a self-healing element resol
    - Search for: playwright.config, wdio.conf, appium config files
    - Check package.json for: @playwright/test, playwright, webdriverio, wdio, appium
    - Read a few existing test files to understand the test structure
+   - Note whether the project is TypeScript (tsconfig.json or .ts test files) — this changes the fixture syntax (see step 3)
    - If none found, tell me you can't detect a supported framework and stop
 
 2. CHOOSE THE LLM PROVIDER
@@ -33,10 +34,24 @@ I just installed the npm package "elementus-ai" — a self-healing element resol
 
 3. INTEGRATE BASED ON MY FRAMEWORK
 
+   First create the elementus instance using the provider chosen in step 2:
+     const { createElementus } = require('elementus-ai')   // ESM/TS: import { createElementus } from 'elementus-ai'
+     const el = createElementus({ provider: 'gemini', geminiApiKey: process.env.GEMINI_API_KEY })
+     // LM Studio instead: createElementus({ provider: 'lmstudio', lmStudioUrl: 'http://localhost:1234/v1/chat/completions', model: 'holo-3.1-9b' })
+   For exact, copy-pasteable code (TypeScript fixture, onHeal recipe), read the installed package's
+   node_modules/elementus-ai/README.md ("Quick Start", "TypeScript", "Detecting heals") and docs/API.md.
+
    For Playwright:
-   - Create or update a fixtures file that wraps page with el.wrapPage(page)
+   - Create or update a fixtures file that overrides the page fixture with el.wrapPage(page)
    - Make sure all tests import from the fixtures file instead of @playwright/test
+   - TypeScript projects: use import/export and type the override as base.extend<{ page: ElementusPage }>({ ... }) (import ElementusPage from elementus-ai) so { ai } is autocompleted and documented. Types are bundled — do NOT add @types or a "declare module" shim
    - Set actionTimeout: 10000 in playwright config (Elementus respects framework timeouts)
+   - Optional heal visibility: only if the user wants healed locators surfaced, wire onHeal to the
+     framework's reporter. Build a suggested replacement from e.resolved ({ tag, text, role }), e.g.
+     getByRole(role, { name: text }). Playwright: push a testInfo "healed" annotation (HTML report) AND
+     testInfo.attach (so it also reaches Allure + custom reporters); optionally fail CI on heal via an
+     env-gated afterEach/fixture. WebdriverIO/Appium: collect in onHeal, fail in afterTest. See
+     "Detecting heals" in the README. Default off
 
    For WebDriverIO:
    - In wdio.conf.js before hook, wrap browser and override global $:
@@ -90,6 +105,8 @@ await p.locator('#submit-btn', { ai: 'Submit order button' }).click()
 await p.locator('#stable-element').click()
 ```
 
+> **Using TypeScript or ESM?** `import { createElementus } from 'elementus-ai'` — type definitions are bundled. See [TypeScript](#typescript) for the typed fixture pattern.
+
 ## LLM Provider Setup
 
 ### Option A: Local LLM via LM Studio (free, private)
@@ -193,6 +210,55 @@ await d.$('~emailField', { ai: 'Email input' }).setValue('test@test.com')
 
 Works with Flutter, React Native, native Android/iOS — any Appium driver.
 
+## TypeScript
+
+Type definitions are bundled — there is no `@types/elementus-ai` package to install and no `declare module` shim to write. Because `@playwright/test` is an *optional* peer, WDIO/Appium-only projects can use the types without installing Playwright.
+
+```ts
+import { createElementus, type ElementusPage } from 'elementus-ai'
+```
+
+**Typed Playwright fixture.** `wrapPage` changes the page's runtime value but not its static type, so override the `page` fixture's type with `ElementusPage` — then `{ ai }` is recognized and autocompleted (with docs) in your tests:
+
+```ts
+// fixtures.ts
+import { test as base, expect } from '@playwright/test'
+import { createElementus, type ElementusPage } from 'elementus-ai'
+
+const el = createElementus({ provider: 'gemini', geminiApiKey: process.env.GEMINI_API_KEY })
+
+export const test = base.extend<{ page: ElementusPage }>({
+  page: async ({ page }, use) => {
+    await use(el.wrapPage(page))
+  },
+})
+export { expect }
+
+// In tests — page is already wrapped and typed:
+test('example', async ({ page }) => {
+  await page.locator('#btn', { ai: 'Submit button' }).click() // { ai } type-checks
+  await page.locator('#btn').click()                          // plain locator, zero overhead
+})
+```
+
+> The override is for editor support — IntelliSense and inline docs for `{ ai }`. It heals at runtime either way, and because Playwright's `locator()` options are permissive, `{ ai }` compiles with or without the override; the override just surfaces it as a documented option.
+
+**Page Object Model.** Type the page your objects receive as `ElementusPage`:
+
+```ts
+import { type ElementusPage } from 'elementus-ai'
+
+abstract class BasePage {
+  constructor(protected readonly page: ElementusPage) {}
+}
+
+class LoginPage extends BasePage {
+  readonly submit = this.page.locator('#submit', { ai: 'Submit button' })
+}
+```
+
+**Exported types:** `ElementusOptions`, `Elementus`, `ElementusPage`, `AiLocatorOptions`. `AiLocatorOptions` is Playwright's own `locator()` option type plus `ai?: string`, derived from the installed Playwright version so it never drifts.
+
 ## API Reference
 
 ### `el.wrapPage(page)`
@@ -267,9 +333,91 @@ createElementus({
 
   // Custom stop words
   stopWords: null,          // Set of words to ignore in descriptions
+
+  // Heal telemetry (opt-in) — called once per heal with { description, selector, method, framework, resolved }.
+  // Best-effort & isolated: a throwing callback never breaks a test. See "Detecting heals" below.
+  onHeal: null,             // e.g. (e) => console.log('healed', e.selector, '→', e.description)
 })
 ```
 
+## Detecting heals (reporting & CI)
+
+A heal is silent by default — the test still passes. To surface it, pass an `onHeal(event)` callback;
+elementus calls it **once per heal** with `{ description, selector, method, framework, resolved }`. The library only
+*emits* the event — turning it into a report annotation or a CI failure is framework-specific (a few lines
+on your side). It fires when the AI resolves a replacement element, so it signals "the selector drifted and
+AI stepped in", not "the action ultimately passed".
+
+**Playwright** — annotate the test (shows in the Playwright HTML report), and optionally fail
+CI on heal:
+
+```javascript
+// fixtures.js
+const { test: base } = require('@playwright/test')
+const { createElementus } = require('elementus-ai')
+
+const el = createElementus({
+  provider: 'gemini',
+  geminiApiKey: process.env.GEMINI_API_KEY,
+  onHeal: (e) => {
+    // Build a suggested replacement from the element elementus resolved to (e.resolved)
+    const r = e.resolved
+    const role = r && (r.role ?? { a: 'link', button: 'button' }[r.tag])
+    const suggestion = !r ? '(located visually — no DOM suggestion)'
+      : role && r.text ? `getByRole('${role}', { name: '${r.text}' })`
+      : r.text ? `getByText('${r.text}')`
+      : `locator('${r.tag}')`
+    const line = `${e.selector}  →  ${suggestion}` // failed locator → suggested replacement
+    try {
+      const info = base.info()
+      info.annotations.push({ type: 'healed', description: line })          // Playwright HTML report
+      void info.attach('healed', { body: line, contentType: 'text/plain' }) // Allure + custom reporters
+    } catch {}  // ignored outside a running test
+  },
+})
+
+const test = base.extend({
+  page: async ({ page }, use) => { await use(el.wrapPage(page)) },
+  // Optional: fail CI when a selector only passed via self-healing (drift you should fix).
+  failOnHeal: [async ({}, use, testInfo) => {
+    await use()
+    const heals = testInfo.annotations.filter(a => a.type === 'healed')
+    if (process.env.ELEMENTUS_FAIL_ON_HEAL && heals.length) {
+      throw new Error('Passed only via self-healing — fix the selector(s):\n  ' + heals.map(h => h.description).join('\n  '))
+    }
+  }, { auto: true }],
+})
+module.exports = { test, expect: require('@playwright/test').expect }
+```
+
+(TypeScript: import the `HealEvent` type from `elementus-ai`; the annotation/fixture code is identical.)
+
+**WebdriverIO / Appium** — collect in `onHeal`, fail (or report) in `afterTest`:
+
+```javascript
+// wdio.conf.js
+const { createElementus } = require('elementus-ai')
+let heals = []
+const el = createElementus({ provider: 'gemini', geminiApiKey: '...', onHeal: (e) => heals.push(e) })
+
+exports.config = {
+  beforeTest() { heals = [] },
+  afterTest()  {
+    if (process.env.ELEMENTUS_FAIL_ON_HEAL && heals.length) {
+      throw new Error('healed: ' + heals.map(h => h.description).join(', '))
+    }
+  },
+}
+```
+
+**Where each report shows it:** the annotation appears in the **Playwright HTML report** (test → _Annotations_).
+In **Allure**, the heal shows in the test's **step tree** (the broken locator → the healed `[data-elementus=…]`
+element) and, with fail-on-heal on, the **failure message** — note `allure-playwright` does not convert
+runtime annotations into labels/steps, so for a dedicated Allure marker also
+`testInfo.attach('healed', { body: \`${e.selector} → ${e.description}\`, contentType: 'text/plain' })`
+(attachments surface in Allure and most custom reporters). A custom HTML reporter surfaces the heal via the
+failure message and captured stdout.
+
 ## Security Notes
 
 - **Debug screenshots** capture the full page — including any sensitive data visible on it. Keep `debugDir` out of version control.

package/elementus.js

@@ -377,6 +377,7 @@ const REGION_LABELS = [
  * @param {number} [userConfig.visionMaxWidth=1280] - max screenshot width (px) sent to vision LLM
  * @param {string|null} [userConfig.cacheFile=null] - opt-in fingerprint cache file (e.g. './elementus-cache.json')
  * @param {string|null} [userConfig.embeddingModel=null] - opt-in embedding model for semantic matching
+ * @param {(event: { description: string, selector: string, method: string, framework: string, resolved?: { tag: string, text: string, role: string|null } }) => void} [userConfig.onHeal] - called after a heal resolves a replacement element (original selector failed, AI found it). `resolved` is the element it healed to (when known), for suggesting a replacement. Best-effort; throwing never breaks a test.
  * @returns {{ wrap, wrapPage, wrapBrowser, locate, find, click }}
  */
 function createElementus(userConfig = {}) {
@@ -1988,6 +1989,31 @@ function createElementus(userConfig = {}) {
     return { record: null, somCandidates: out.somCandidates || null }
   }
 
+  // Facts about the element the last heal resolved to (tag/text/role) — set during resolution,
+  // read once by _notifyHeal. Framework-agnostic; consumers format a suggested replacement locator.
+  let _lastResolved = null
+  function _resolvedFacts(record) {
+    if (!record) return null
+    return { tag: record.tag || '', text: (record.text || '').trim(), role: record.role || null }
+  }
+
+  // Fire the user's onHeal(event) after a real heal — the original selector failed and the AI
+  // pipeline resolved a replacement. Best-effort and fully isolated: a throwing or rejecting
+  // callback must never break a test. No-op with zero overhead when onHeal is not configured.
+  function _notifyHeal(ctx, description, selectorKey, method) {
+    const resolved = _lastResolved
+    _lastResolved = null
+    const cb = config.onHeal
+    if (typeof cb !== 'function') return
+    const framework = _isNative(ctx) ? 'appium' : _isPlaywright(ctx) ? 'playwright' : 'wdio'
+    try {
+      const event = { description, selector: selectorKey, method, framework }
+      if (resolved) event.resolved = resolved
+      const r = cb(event)
+      if (r && typeof r.then === 'function') r.then(undefined, () => {})
+    } catch {}
+  }
+
   async function _findByDescription(ctx, description, selectorKey = '') {
     const { record, somCandidates } = await _resolveElement(ctx, description, selectorKey)
     if (record) {
@@ -1995,6 +2021,7 @@ function createElementus(userConfig = {}) {
         const mark = {}
         const locator = record._locator || await markByElement(ctx, record, mark)
         await _cacheStore(ctx, description, selectorKey, record, record._uid || mark.uid || null)
+        _lastResolved = _resolvedFacts(record)
         return locator
       } catch (err) {
         console.log(`[Resolve] Mark failed (${err.message}) — trying vision`)
@@ -2006,11 +2033,13 @@ function createElementus(userConfig = {}) {
         const mark = {}
         const locator = await markByElement(ctx, result.element, mark)
         await _cacheStore(ctx, description, selectorKey, result.element, mark.uid || null)
+        _lastResolved = _resolvedFacts(result.element)
         return locator
       }
       const mark = {}
       const locator = await markAtCoordinates(ctx, result.coords.docX, result.coords.docY, mark)
       await _cacheStore(ctx, description, selectorKey, result.coords, mark.uid || null)
+      _lastResolved = null
       return locator
     } catch (err) {
       throw new Error(`All fallback paths exhausted for "${description}": ${err.message}`)
@@ -2038,7 +2067,10 @@ function createElementus(userConfig = {}) {
     } catch {
       console.log(`\u2717 Locator failed \u2014 searching for: "${description}"`)
     }
-    return _findByDescription(ctx, description, _selectorKey(locator))
+    const selectorKey = _selectorKey(locator)
+    const resolved = await _findByDescription(ctx, description, selectorKey)
+    _notifyHeal(ctx, description, selectorKey, 'locate')
+    return resolved
   }
 
   /**
@@ -2088,6 +2120,8 @@ function createElementus(userConfig = {}) {
       // Store before clicking \u2014 the click may navigate away from the page
       await _cacheStore(ctx, description, selectorKey, record)
       await scrollAndClick(ctx, record)
+      _lastResolved = _resolvedFacts(record)
+      _notifyHeal(ctx, description, selectorKey, 'click')
       return
     }
     try {
@@ -2095,9 +2129,13 @@ function createElementus(userConfig = {}) {
       if (result.element) {
         await _cacheStore(ctx, description, selectorKey, result.element)
         await scrollAndClick(ctx, result.element)
+        _lastResolved = _resolvedFacts(result.element)
+        _notifyHeal(ctx, description, selectorKey, 'click')
         return
       }
       await clickAtCoords(ctx, result.coords)
+      _lastResolved = null
+      _notifyHeal(ctx, description, selectorKey, 'click')
     } catch (err) {
       throw new Error(`All fallback paths exhausted for "${description}": ${err.message}`)
     }
@@ -2160,6 +2198,7 @@ function createElementus(userConfig = {}) {
             if (!_resolved) {
               console.log(`[AI] ${prop}() \u2014 resolving via AI first for "${description}"`)
               _resolved = await _findByDescription(driverContext, description, wrapSelectorKey)
+              _notifyHeal(driverContext, description, wrapSelectorKey, String(prop))
             }
             return _resolved[prop](...args)
           }
@@ -2168,7 +2207,10 @@ function createElementus(userConfig = {}) {
             return await original.apply(target, args)
           } catch (firstError) {
             console.log(`[AI] ${String(prop)}() failed \u2014 AI fallback for "${description}"`)
-            if (!_resolved) _resolved = await _findByDescription(driverContext, description, wrapSelectorKey)
+            if (!_resolved) {
+              _resolved = await _findByDescription(driverContext, description, wrapSelectorKey)
+              _notifyHeal(driverContext, description, wrapSelectorKey, String(prop))
+            }
 
             const resolvedMethod = _resolved[prop]
             if (typeof resolvedMethod !== 'function') {

package/index.d.ts

@@ -0,0 +1,104 @@
+// Type definitions for elementus-ai
+// Project: https://github.com/Morph93/elementus
+//
+// Self-healing element resolution for Playwright, WebdriverIO & Appium.
+// These types describe the Playwright/core API. WebdriverIO's global `$`
+// augmentation lives in the separate, opt-in `wdio.d.ts`.
+//
+// `@playwright/test` is an OPTIONAL peer dependency. The `@ts-ignore` below lets
+// WDIO/Appium-only consumers (who have no Playwright installed) fall back to
+// `any` for these types instead of failing module resolution.
+// @ts-ignore -- optional peer dependency
+import type { Page, Locator } from '@playwright/test'
+
+export interface ElementusOptions {
+  /** LLM provider. @default 'lmstudio' */
+  provider?: 'lmstudio' | 'gemini'
+  /** LM Studio chat-completions endpoint. @default 'http://localhost:1234/v1/chat/completions' */
+  lmStudioUrl?: string
+  /** LM Studio model name. @default 'holo-3.1-9b' */
+  model?: string
+  /** Google Gemini API key (or set the GEMINI_API_KEY env var). @default null */
+  geminiApiKey?: string | null
+  /** Gemini model id. @default 'gemini-3.5-flash' */
+  geminiModel?: string
+  /** Max elements sent to the LLM for disambiguation. @default 20 */
+  maxCandidates?: number
+  /** Save debug screenshots to `debugDir`. @default false */
+  debug?: boolean
+  /** Directory for debug screenshots (required when `debug` is true). @default null */
+  debugDir?: string | null
+  /** Custom stop words to ignore in descriptions (replaces the defaults). @default null */
+  stopWords?: Set<string> | null
+  /** Max screenshot width (px) sent to the vision LLM. @default 1280 */
+  visionMaxWidth?: number
+  /** Opt-in fingerprint cache file, e.g. './elementus-cache.json'. @default null */
+  cacheFile?: string | null
+  /** Opt-in embedding model for semantic paraphrase matching. @default null */
+  embeddingModel?: string | null
+  /**
+   * Called after a heal resolves a replacement element (the original selector failed and the AI
+   * pipeline found a match). Best-effort and isolated — a throwing/rejecting callback never breaks
+   * a test. Fires at resolution time, so it does NOT guarantee the subsequent action succeeded.
+   */
+  onHeal?: (event: HealEvent) => void | Promise<void>
+}
+
+/** Event passed to `onHeal` when a broken selector is healed. */
+export interface HealEvent {
+  /** The natural-language `{ ai }` description that resolved the element. */
+  description: string
+  /** The original selector that failed (selector key). */
+  selector: string
+  /** The action that triggered the heal: 'click', 'fill', 'isVisible', 'locate', etc. */
+  method: string
+  /** Which driver the heal happened on. */
+  framework: 'playwright' | 'wdio' | 'appium'
+  /**
+   * The element elementus resolved to (when known — absent on coordinate-only vision heals).
+   * Use it to compose a suggested replacement locator, e.g. getByRole(role, { name: text }).
+   */
+  resolved?: { tag: string; text: string; role: string | null }
+}
+
+/**
+ * Playwright's own `locator()` options, plus the Elementus `ai` hint.
+ * Derived from the installed Playwright types so it never drifts.
+ */
+export type AiLocatorOptions = NonNullable<Parameters<Page['locator']>[1]> & {
+  /** Natural-language description; the self-healing fallback used when `selector` breaks. */
+  ai?: string
+}
+
+/**
+ * A Playwright Page whose `locator()` also accepts `{ ai }`. Locators created
+ * with an `ai` hint self-heal when the selector breaks; locators without it are
+ * returned unchanged (zero overhead).
+ */
+export type ElementusPage = Page & {
+  locator(selector: string, options?: AiLocatorOptions): Locator
+}
+
+export interface Elementus {
+  /**
+   * Wrap a Playwright Page so `page.locator(selector, { ai })` self-heals.
+   * Call once per test, or in a fixture for the whole suite.
+   */
+  wrapPage(page: Page): ElementusPage
+  /**
+   * Wrap a WebdriverIO/Appium browser so `$(selector, { ai })` self-heals.
+   * Returns the same object it was given (now AI-aware).
+   */
+  wrapBrowser<T>(browser: T): T
+  /** Try `locator` first; fall back to AI resolution if it fails. */
+  locate(ctx: Page, locator: Locator, description: string): Promise<Locator>
+  /** Resolve an element from a natural-language description alone. */
+  find(ctx: Page, description: string): Promise<Locator>
+  /** Click with an optimized fallback (goto for links, JS click for buttons). */
+  click(ctx: Page, locator: Locator, description: string): Promise<void>
+  /** Low-level: wrap a single locator with AI fallback. Prefer wrapPage(). */
+  wrap(ctx: Page, locator: Locator, description: string): Locator
+}
+
+/** Create an Elementus instance with the given configuration. */
+export function createElementus(options?: ElementusOptions): Elementus

package/package.json

@@ -1,8 +1,9 @@
 {
   "name": "elementus-ai",
-  "version": "1.1.1",
+  "version": "1.4.0",
   "description": "Self-healing element resolution for Playwright, WDIO & Appium. AI-powered fallback when selectors break.",
   "main": "elementus.js",
+  "types": "index.d.ts",
   "scripts": {
     "test": "playwright test test/playwright.spec.js",
     "test:smoke": "playwright test test/playwright.spec.js -g \"T01 |T02 |T09 |T17 |T23 \""
@@ -51,8 +52,10 @@
   },
   "files": [
     "elementus.js",
+    "index.d.ts",
     "wdio.d.ts",
     "README.md",
+    "CHANGELOG.md",
     "LICENSE"
   ]
 }