playwright-genie 1.0.1 → 2.0.0

package/README.md

@@ -12,9 +12,14 @@
 - **Natural Language** — describe elements in plain English, no selectors needed
 - **40+ Playwright Actions** — click, fill, check, hover, drag, wait, screenshot and more
 - **Any LLM Provider** — OpenAI, Claude, Ollama, Azure, or any OpenAI-compatible API
-- **Smart Caching** — persistent disk cache + in-memory cache to minimize LLM calls
+- **Adaptive Page Analysis** — automatically switches between ARIA, hybrid, and DOM-only modes based on page accessibility quality
+- **DOM Tree Pipeline** — generates XPath and CSS selectors in-browser for pages with poor accessibility
+- **Smart Caching** — two-tier cache (memory + disk `.locator-cache.json`) to minimize LLM calls
+- **Fallback Locator Chains** — LLM returns multiple locator strategies; if the primary fails, fallbacks are tried automatically
 - **Action-Aware** — `fill('username')` targets the input, not the label
-- **Auto-Retry** — stale cached locators are automatically re-resolved
+- **Auto-Retry** — stale cached locators are automatically invalidated and re-resolved
+- **Batch Resolution** — `prefetch()` resolves multiple queries in a single LLM call
+- **iframe Support** — automatically detects and resolves elements inside iframes
 - **TypeScript Support** — full type definitions included
 - **Single Page Object** — one `createSmartLocator(page)` works across all navigations
 
@@ -95,6 +100,20 @@ await smart.click('submit button');
 await browser.close();
 ```
 
+## How It Works
+
+When you call `smart.click('login button')`, the library:
+
+1. **Collects page structure** — gathers the ARIA accessibility tree, interactive elements, special attributes (`data-testid`, `placeholder`, `aria-label`), and a full DOM tree with XPath/CSS selectors
+2. **Evaluates ARIA quality** — scores the page as `good`, `sparse`, or `none` based on how many named interactive elements the ARIA tree contains
+3. **Builds an adaptive payload** — selects the best strategy:
+   - **`aria` mode** — rich ARIA tree with good accessibility; uses ARIA snapshot + special elements
+   - **`hybrid` mode** — sparse ARIA; combines the ARIA tree with DOM tree nodes for better coverage
+   - **`dom` mode** — no useful ARIA; sends DOM tree with XPaths and CSS selectors generated in-browser
+4. **Queries the LLM** — sends the payload with your natural language query; the LLM returns a Playwright locator string (e.g., `getByRole('button', { name: 'Login' })`) along with fallback locators
+5. **Validates and caches** — verifies the locator resolves to a real element, caches it to memory and disk, and returns a `SmartAction` for interaction
+6. **Auto-recovers** — if a cached locator goes stale, it's invalidated and re-resolved; if the primary locator fails, fallback locators are tried automatically
+
 ## API Reference
 
 ### `createSmartLocator(page, options?)`
@@ -114,6 +133,7 @@ const smart = createSmartLocator(page, { verbose: true });
 | `model` | `string` | env var | Override LLM model |
 | `temperature` | `number` | `0` | LLM temperature |
 | `maxTokens` | `number` | `1024` | Max response tokens |
+| `actionTimeout` | `number` | `10000` | Timeout for actions in ms |
 
 ---
 
@@ -282,6 +302,113 @@ smart.clearCache();       // clear in-memory cache only
 smart.clearAllCaches();   // clear both memory + disk (.locator-cache.json)
 ```
 
+Use `clearCache()` after SPA navigations where the DOM changes significantly (e.g., after login) to force fresh page analysis.
+
+## Low-Level API
+
+For advanced use cases, you can use the lower-level exports directly.
+
+### `findLocator(page, query, options?)`
+
+Resolves a single natural language query to a Playwright locator string without performing any action.
+
+```js
+import { findLocator } from 'playwright-genie';
+
+const result = await findLocator(page, 'submit button');
+console.log(result);
+// {
+//   found: true,
+//   strategy: 'role',
+//   locatorString: "getByRole('button', { name: 'Submit' })",
+//   confidence: 0.95,
+//   fallbackLocators: ["getByTestId('submit-btn')", "locator('#submit')"],
+//   isInFrame: false,
+//   frameSelector: null
+// }
+```
+
+### `findAllMatches(page, query, options?)`
+
+Returns an array of all matching locator results for a query.
+
+```js
+import { findAllMatches } from 'playwright-genie';
+
+const matches = await findAllMatches(page, 'navigation link');
+```
+
+### `getPageStructure(page, forceRefresh?)`
+
+Returns the full page structure used for LLM resolution. Useful for debugging or building custom pipelines.
+
+```js
+import { getPageStructure } from 'playwright-genie';
+
+const structure = await getPageStructure(page);
+console.log(structure.mainFrame.ariaQuality); // 'good' | 'sparse' | 'none'
+console.log(structure.mainFrame.ariaTree);    // ARIA snapshot (YAML string)
+console.log(structure.mainFrame.domTree);     // Array of DOM nodes with XPath/CSS
+console.log(structure.mainFrame.interactiveElements); // Interactive element metadata
+console.log(structure.mainFrame.specialElements);     // Elements with data-testid, etc.
+console.log(structure.frames);                // iframe structures
+```
+
+### `resolveLocator(page, query, options?)`
+
+Low-level resolver that checks memory cache → disk cache → LLM. Returns the raw result object without creating a Playwright locator.
+
+```js
+import { resolveLocator } from 'playwright-genie';
+
+const result = await resolveLocator(page, 'login button', { action: 'click' });
+// result.source is 'memory', 'disk', or 'llm'
+```
+
+### `getLocator(page, query, options?)`
+
+Resolves a query and returns both the Playwright `Locator` object and the result metadata. Handles stale cache invalidation and fallback chains.
+
+```js
+import { getLocator } from 'playwright-genie';
+
+const { locator, result } = await getLocator(page, 'email input', { action: 'fill' });
+await locator.fill('user@example.com');
+```
+
+### `clearCache()` / `clearAllCaches()`
+
+Module-level cache clearing functions.
+
+```js
+import { clearCache, clearAllCaches } from 'playwright-genie';
+
+clearCache();       // memory only
+clearAllCaches();   // memory + disk
+```
+
+## Adaptive Page Analysis
+
+The library automatically adapts to the accessibility quality of each page:
+
+| ARIA Quality | Criteria | Mode | What Gets Sent to LLM |
+|---|---|---|---|
+| **`good`** | ARIA tree has 3+ named interactive elements, 20+ lines | `aria` | Trimmed ARIA tree + special elements + interactive elements |
+| **`sparse`** | ARIA tree exists but fewer named elements than the page has | `hybrid` | ARIA tree + DOM tree nodes (XPath/CSS) + interactive elements |
+| **`none`** | ARIA tree has < 5 lines or is missing | `dom` | DOM tree with XPaths and CSS selectors + interactive elements |
+
+### DOM Tree Pipeline
+
+For pages with poor or no accessibility markup, the library walks the DOM in-browser and:
+
+- Traverses up to **300 visible nodes** (headings, links, buttons, inputs, landmarks, etc.)
+- Generates **XPath** for each node (e.g., `//*[@id="login"]`, `//form/div[2]/input[1]`)
+- Generates **unique CSS selectors** (e.g., `#login`, `[data-testid="submit"]`, `button.primary`)
+- Extracts text content, ARIA labels, placeholders, `data-testid` attributes, and parent context
+- Filters nodes by **relevance scoring** against your query before sending to the LLM
+
+This means the library works on any page — not just accessible ones.
+
 ## Caching
 
 **playwright-genie** uses a two-level cache to minimize LLM calls:
@@ -291,7 +418,9 @@ smart.clearAllCaches();   // clear both memory + disk (.locator-cache.json)
 
 Cache keys are scoped by **URL pathname + action + query**, so `fill('username')` on `/login` won't collide with `click('username')` on `/dashboard`.
 
-If a cached locator becomes stale (element no longer exists), it's automatically invalidated and re-resolved via LLM.
+If a cached locator becomes stale (element no longer exists), the library:
+1. Tries **fallback locators** returned by the LLM
+2. If all fallbacks fail, **invalidates the cache** and re-queries the LLM with fresh page structure
 
 Set `LOCATOR_CACHE_FILE` env var to customize the cache file path.
 
@@ -353,6 +482,51 @@ test('shopping flow', async ({ page }) => {
 });
 ```
 
+### SPA Navigation with Cache Clearing
+
+```js
+test('SPA login and navigate', async ({ page }) => {
+  const smart = createSmartLocator(page);
+  await page.goto('https://spa-app.com/login');
+
+  await smart.fill('username', 'admin');
+  await smart.fill('password', 'secret');
+  await smart.click('login button');
+
+  // After SPA navigation, clear cache to force fresh page analysis
+  await page.waitForURL('**/dashboard');
+  smart.clearCache();
+
+  await smart.click('settings tab');
+  await smart.waitForVisible('settings panel');
+});
+```
+
+### Batch Pre-fetch for Performance
+
+```js
+test('prefetch for faster tests', async ({ page }) => {
+  const smart = createSmartLocator(page);
+  await page.goto('https://myapp.com/form');
+
+  // Resolve all locators in one LLM call
+  await smart.prefetch(
+    'first name input',
+    'last name input',
+    'email field',
+    'phone number',
+    'submit button'
+  );
+
+  // All cached — zero LLM calls from here
+  await smart.fill('first name input', 'John');
+  await smart.fill('last name input', 'Doe');
+  await smart.fill('email field', 'john@example.com');
+  await smart.fill('phone number', '555-0123');
+  await smart.click('submit button');
+});
+```
+
 ### Dynamic Content & Modals
 
 ```js
@@ -372,35 +546,46 @@ test('handle dynamic content', async ({ page }) => {
 });
 ```
 
-### Form Validation
+### Using Low-Level API for Debugging
 
 ```js
-test('form validation', async ({ page }) => {
-  const smart = createSmartLocator(page);
-  await page.goto('https://myapp.com/signup');
-
-  await smart.click('submit button');
-  await smart.waitForVisible('email error message');
-
-  const error = await smart.getText('email error message');
-  expect(error).toContain('required');
-
-  await smart.fill('email field', 'user@example.com');
-  const enabled = await smart.isEnabled('submit button');
-  expect(enabled).toBe(true);
+import { getPageStructure, findLocator } from 'playwright-genie';
+
+test('debug locator resolution', async ({ page }) => {
+  await page.goto('https://myapp.com');
+
+  // Inspect page analysis
+  const structure = await getPageStructure(page);
+  console.log('ARIA quality:', structure.mainFrame.ariaQuality);
+  console.log('DOM nodes:', structure.mainFrame.domTree.length);
+  console.log('Interactive elements:', structure.mainFrame.interactiveElements.length);
+
+  // See what the LLM resolves without acting
+  const result = await findLocator(page, 'submit button');
+  console.log('Strategy:', result.strategy);
+  console.log('Locator:', result.locatorString);
+  console.log('Fallbacks:', result.fallbackLocators);
 });
 ```
 
-### Drag & Drop
+## Exports
 
 ```js
-test('kanban board', async ({ page }) => {
-  const smart = createSmartLocator(page);
-  await page.goto('https://app.example.com/board');
-
-  await smart.dragTo('first task card', 'done column');
-  await smart.waitForVisible('task moved toast');
-});
+import {
+  createSmartLocator,   // Main entry — creates smart locator with 40+ action methods
+  findLocator,          // Resolve a query to a locator string (no action)
+  findAllMatches,       // Get all matching locator results
+  getPageStructure,     // Get the full page structure (ARIA + DOM + interactive elements)
+  getLocator,           // Resolve + validate + create Playwright Locator object
+  resolveLocator,       // Low-level: cache lookup → LLM resolution
+  clearCache,           // Clear in-memory cache
+  clearAllCaches,       // Clear memory + disk cache
+  SmartAction,          // Class wrapping a Playwright Locator with 40+ methods
+  chatCompletion,       // Direct LLM call (for custom pipelines)
+  getConfig,            // Get current LLM configuration
+  loadDiskCache,        // Load disk cache manually
+  invalidateDiskEntry,  // Invalidate a specific disk cache entry
+} from 'playwright-genie';
 ```
 
 ## Best Practices
@@ -441,6 +626,18 @@ await el.press('Enter');
 // 1 LLM call instead of 2
 ```
 
+**Use prefetch() for forms and multi-element pages:**
+```js
+await smart.prefetch('name', 'email', 'password', 'submit');
+// 1 LLM call instead of 4
+```
+
+**Clear cache after SPA navigation:**
+```js
+await page.waitForURL('**/dashboard');
+smart.clearCache();
+```
+
 ## TypeScript
 
 ```ts
@@ -465,14 +662,28 @@ LLM_LOCATOR_DEBUG=true npx playwright test
 ```
 
 This logs:
+- Payload mode selected (`aria`/`hybrid`/`dom`) and payload size
 - LLM queries and responses
 - Cache hits/misses (memory and disk)
-- Page structure payload sizes
-- Stale cache invalidations
+- Stale cache invalidations and fallback attempts
+- Page structure collection timing
+
+## Environment Variables
+
+| Variable | Description |
+|---|---|
+| `LLM_API_KEY` | API key for LLM provider |
+| `LLM_BASE_URL` | Base URL for OpenAI-compatible API |
+| `LLM_MODEL` | Model name (e.g., `gpt-4o-mini`) |
+| `OPENAI_API_KEY` | Fallback API key |
+| `ANTHROPIC_API_KEY` | Fallback API key |
+| `ROUTELLM_API_KEY` | Fallback API key |
+| `LOCATOR_CACHE_FILE` | Custom path for disk cache file |
+| `LLM_LOCATOR_DEBUG` | Set to `true` to enable debug logging |
 
 ## Security Notes
 
-- The library sends the page's accessibility tree to your configured LLM API
+- The library sends the page's accessibility tree and/or DOM structure to your configured LLM API
 - Sensitive data visible in the DOM may be sent to the API
 - Use environment variables for API keys — never hardcode them
 - For sensitive environments, use a local LLM (e.g., Ollama)
@@ -484,9 +695,3 @@ Contributions are welcome! Please feel free to submit a Pull Request.
 ## License
 
 MIT License — see the [LICENSE](LICENSE) file for details.
-
-## Acknowledgments
-
-- [Playwright](https://playwright.dev/) — browser automation framework
-- [OpenAI](https://openai.com/) — LLM API support
-- [Claude Code](https://docs.anthropic.com/en/docs/claude-code) — AI-powered code generation and architecture design