stably 4.9.0 → 4.10.0

package/dist/stably-plugin-cli/skills/playwright-config-auth/SKILL.md

@@ -0,0 +1,217 @@
+---
+name: playwright-config-auth
+description: Guide for Playwright config management including cache invalidation, auth project dependencies, and avoiding auth setup chains. Use when encountering config issues, auth/login setup, project dependencies configuration, cache problems, or unexpected auth behavior. Triggers on playwright.config, auth project, dependencies, storageState, config cache, cache-bust, auth setup chain, project selection.
+---
+
+# Playwright Config & Auth Guide
+
+This skill provides guidance for Playwright configuration management and authentication setup for the CLI environment.
+
+## Key Difference: No Catch-All Project
+
+**The CLI does NOT have a `stably-internal-all-tests-project` catch-all project.** That project only exists in the web environment.
+
+When you need to run tests without auth dependencies, you must add an isolated project to `playwright.config.ts` (see "Adding an Isolated Project" section below).
+
+## Troubleshooting: Stale Config Cache
+
+If you edit `playwright.config.ts` and changes don't take effect, the cache may be stale:
+
+```bash
+rm -rf /tmp/playwright-transform-cache-* 2>/dev/null || true
+```
+
+This is rare in CLI - only try this if config changes are clearly not being picked up.
+
+## Avoiding Auth Setup Chains
+
+Some repositories use multi-user authentication patterns with project dependencies. This can cause unexpected behavior when using setup tools.
+
+### The Problem
+
+Projects with `dependencies: ['setup-xyz']` will run all dependency setup tests (auth flows) before your seed test - even if you don't need that auth state.
+
+### How to Recognize It
+
+Look for:
+- Multiple `setup-*` projects in `playwright.config.ts`
+- `storageState` configurations
+- `.auth/*.json` files
+
+### The Solution
+
+Since the CLI has no catch-all project, you must add an isolated project to avoid auth chains.
+
+1. Use `tests/seed.spec.ts` for browser setup
+2. Add an isolated project to `playwright.config.ts` (see below)
+3. Use that project when running tests that don't need auth
+
+### Before Creating Test Files
+
+Check if the target project's `testMatch` pattern will cover your new test file:
+
+1. Read `playwright.config.ts` and find the `isolated` project's `testMatch` pattern
+2. If your new file won't match (e.g., `testMatch: ['**/seed.spec.ts']` won't match `my-test.spec.ts`):
+   - Either update `testMatch` to include the new file (e.g., `['**/seed.spec.ts', '**/my-test.spec.ts']`)
+   - Or use a broader pattern (e.g., `tests/isolated/*.spec.ts` and place files there)
+3. After editing config: clear cache, restart MCP, run `test_list` to verify
+
+### Adding an Isolated Project (REQUIRED for No-Auth Tests)
+
+Add an isolated project to `playwright.config.ts` with specific `testMatch` and NO dependencies:
+
+```typescript
+{
+  name: 'isolated',
+  testMatch: '**/seed.spec.ts',  // Or any pattern matching your seed file
+  use: {
+    ...devices['Desktop Chrome'],
+    // Optionally use existing auth state if available
+    storageState: '.auth/user.json',
+  },
+  // NO dependencies - this is key!
+}
+```
+
+This ensures only the isolated project matches your seed file, avoiding dependency chains.
+
+Common patterns: `**/seed.spec.ts`, `**/setup.spec.ts`, or `tests/isolated/*.spec.ts`
+
+### After Adding or Editing the Project
+
+1. Clear Playwright cache: `rm -rf /tmp/playwright-transform-cache-* 2>/dev/null || true`
+2. Restart MCP servers to pick up the config change
+3. **Run `test_list` and verify your test appears under the correct project** before running `test_debug` or `test_run`
+4. If your test only appears under a project with auth dependencies (e.g., `all`), do NOT proceed - fix the project matching first
+5. **If test appears under multiple projects**, add `testIgnore` to the unintended projects to exclude your test file
+6. Use the new project: `test_run with projects: ['isolated']`
+
+### If Setup Takes Too Long or Runs Auth Unexpectedly
+
+The seed file likely matched a user project with dependencies. Check `playwright.config.ts` for which project patterns match your seed file location, and add an isolated project.
+
+## Leveraging Auth Project Dependencies
+
+When the repository uses authentication via project dependencies (a common pattern), follow these steps:
+
+### How to Recognize This Pattern
+
+- `playwright.config.ts` has an `auth` project with `testMatch: /.*\.auth\.ts/`
+- Other projects have `dependencies: ["auth"]` (e.g., `chromium`, `mobile-safari`)
+- Auth tests create `storageState` files (e.g., `.auth/US.json`, `.auth/EU.json`)
+- Test fixtures use `storageState` to load pre-authenticated sessions
+
+### The Problem
+
+- The CLI has no catch-all project; the user's default project is used
+- If you don't specify a project with auth dependencies, auth never runs
+- Tests fail with "No auth file found" or missing storageState
+- Even with right project, some auth setups may have skip conditions
+
+### The Solution
+
+1. **Specify a project with auth dependencies**: When calling `generator_setup_page` or `test_run`, explicitly specify a project that has `dependencies: ["auth"]` (e.g., `chromium`, `mobile-safari`)
+
+2. **Check for auth skip conditions**: If auth tests are being skipped, inspect auth test file (e.g., `*.auth.ts`) and related setup files for skip conditions. Look for environment variables or configuration that forces re-authentication.
+
+3. **Create tests using normal pattern**: With auth working via project dependencies, tests don't need manual login workarounds. They can use repository's standard auth pattern with `storageState`.
+
+### Example Workflow
+
+1. Read `playwright.config.ts` and identify projects with `dependencies: ["auth"]` (e.g., `chromium`, `mobile-safari`)
+2. Check auth test files for skip conditions or environment variables
+3. Create test file using repository's normal auth pattern (via storageState from fixtures)
+4. Call `generator_setup_page` with identified project (e.g., `project: "chromium"`)
+5. Auth project runs automatically as dependency -> creates auth files -> test runs authenticated
+
+### What NOT to Do
+
+- Don't leave project unspecified and expect auth to run
+- Don't create `seed.spec.ts` files with manual `login()` calls as workaround for auth issues
+- Don't add `@noauth` tag unless user specifically wants tests that handle their own authentication independently
+
+## Project Selection for Auth (CRITICAL)
+
+If config has auth project dependencies (e.g., `chromium` with `dependencies: ["auth"]`), you MUST specify that project when calling setup tools.
+
+```
+generator_setup_page({ project: "chromium", seedFile: "tests/my-test.spec.ts", ... })
+```
+
+**Do NOT leave the project param blank** - auth will not run without specifying a project that has auth dependencies.
+
+## stably.yaml Scheduled Runs
+
+When creating or editing `stably.yaml` schedules, use this exact shape:
+
+```yaml
+schedules:
+  schedule-name:
+    cron: "0 9 * * *"
+    # Optional: stablyTestArgs: "--project smoke"
+    # Optional: timezone: "America/Los_Angeles"
+```
+
+### Rules
+
+- `schedules` MUST be an object/map keyed by schedule name
+- Do NOT use array format like `schedules: [{ name: ..., cron: ... }]`
+- Do NOT use `testMatch` inside `stably.yaml` schedules
+- Use `stablyTestArgs` for filtering test selection
+
+## Test File Format Guidelines
+
+- Stably supports all Playwright default test file extensions: `.spec.ts`, `.test.ts`, `.spec.js`, `.test.js`, `.spec.tsx`, `.test.tsx`, `.spec.jsx`, `.test.jsx`, and CommonJS/ESM variants
+- For new tests, prefer `.spec.ts` or `.test.ts` (TypeScript) for best IDE support
+- If user requests specific extension, proceed with their preference if Playwright-supported
+- For consistency, check existing test files to match their convention
+- Do NOT modify `testMatch` option in `playwright.config.ts` to use custom patterns (e.g., `testMatch: ["**/*.e2e.ts"]`)
+- You CAN read and edit `playwright.config.ts` for other legitimate purposes (adjusting timeouts, adding projects, configuring reporters)
+
+## Test Execution Project Selection (CRITICAL)
+
+This section covers how to select the right project when running tests with `test_run`.
+
+### Rule 1: Add an Isolated Project for Independent Tests
+
+When running tests that DON'T need auth or other project dependencies (e.g., tests for public websites, tests with `test.use({ storageState: { cookies: [], origins: [] } })`):
+
+1. Add an isolated project to `playwright.config.ts` with NO dependencies (see "Adding an Isolated Project" above)
+2. Use that project:
+   ```
+   test_run with projects: ['isolated'] and locations: ['tests/my-test.spec.ts']
+   ```
+
+**Why**: Playwright runs tests under ALL matching projects, not just one. If your test matches a project with `dependencies: ['setup-xyz']`, auth/setup flows will run unexpectedly.
+
+### Rule 2: Verify Project Config Before Specifying
+
+Before specifying a project like `chromium`, `firefox`, etc., read `playwright.config.ts` and check that project for:
+
+- **grep** - regex filter that restricts which test names can run
+- **testMatch** - file pattern filter that restricts which test files can run
+- **testIgnore** - file patterns that are excluded from running
+- **dependencies** - other projects (like auth setup) that must run first
+
+If the project has `grep` or `testMatch` filters that would exclude your test file or test name, do NOT use that project. Add an isolated project with no dependencies/filters instead.
+
+### Rule 3: Quick Decision Tree
+
+```
+Does the test need auth/setup dependencies?
++-- NO (public website, no login needed)
+|   +-- Add isolated project to config, then use it
++-- YES (requires login, specific browser config, etc.)
+    +-- Check target project's grep/testMatch/testIgnore filters
+        +-- Test matches filters -> Use that project
+        +-- Test does NOT match filters -> Pick appropriate project
+```
+
+### Common Pitfalls
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Empty test output | Project filters exclude your test | Add isolated project with no filters |
+| Auth runs unexpectedly | Test matches project with auth dependencies | Add isolated project with no dependencies |
+| Test runs multiple times | Multiple projects match the test | Specify single project explicitly |
+| "Project not found" | Tried using `stably-internal-all-tests-project` | This project doesn't exist in CLI; add isolated project |

package/dist/stably-plugin-cli/skills/stably-sdk-reference/SKILL.md

@@ -0,0 +1,307 @@
+---
+name: stably-sdk-reference
+description: Complete reference for Stably SDK features including aiAssert, agent.act(), extract(), getLocatorsByAI(), and email inbox testing. Use when writing Playwright tests with AI-powered assertions, autonomous agent workflows, visual data extraction, AI-based element finding, or email verification flows. Triggers on aiAssert, toMatchScreenshotPrompt, agent.act, page.extract, getLocatorsByAI, @stablyai/email, Inbox.build, model selection.
+---
+
+# Stably SDK Reference
+
+This skill provides comprehensive guidance for using Stably's AI-powered testing features in Playwright tests.
+
+## Import Statement
+
+Always use the Stably SDK import:
+
+```typescript
+import { test, expect } from "@stablyai/playwright-test";
+```
+
+## AI Assertions (aiAssert)
+
+Use `aiAssert` for intent-based visual verification of dynamic UIs:
+
+```typescript
+// Page-level assertion
+await expect(page).aiAssert(
+  "Shows revenue trend chart and spotlight card",
+  { timeout: 30_000 }
+);
+
+// Scoped to specific element (preferred for precision)
+await expect(page.locator(".header"))
+  .aiAssert("Nav with avatar and bell icon");
+```
+
+**Signature:** `expect(page|locator).aiAssert(prompt: string, options?: { timeout?: number, fullPage?: boolean, model?: AIModel })`
+
+**Best Practices:**
+- Use for **dynamic** UIs where deterministic assertions are insufficient
+- Keep prompts specific with labels and units
+- Scope with locators when possible (more precise, less noisy)
+- **Consider `fullPage: true` carefully**: Only use when assertion requires content beyond the visible viewport. Viewport captures are faster and cheaper.
+
+**Note:** `toMatchScreenshotPrompt` is deprecated. Use `aiAssert` instead.
+
+## AI Extraction (extract)
+
+Extract data from visual content:
+
+```typescript
+// Simple string extraction
+const txt = await page.extract("List revenue, active users, and churn rate");
+
+// Typed extraction with Zod schema
+import { z } from "zod";
+const Metrics = z.object({
+  revenue: z.string(),
+  activeUsers: z.number(),
+  churnRate: z.number()
+});
+const m = await page.extract(
+  "Return revenue (currency), active users, churn %",
+  { schema: Metrics }
+);
+```
+
+**Signatures:**
+- `page.extract(prompt: string, options?: { model?: AIModel }): Promise<string>`
+- `page.extract<T extends z.AnyZodObject>(prompt, { schema: T, model?: AIModel }): Promise<z.output<T>>`
+
+## AI Locator Finding (getLocatorsByAI)
+
+Find elements using natural language based on accessibility properties:
+
+```typescript
+// Find a single element
+const { locator: loginBtn, count } = await page.getLocatorsByAI("the login button");
+expect(count).toBe(1);
+await loginBtn.click();
+
+// Find multiple elements
+const { locator: productCards, count: cardCount } = await page.getLocatorsByAI(
+  "all product cards in the grid"
+);
+await expect(productCards).toHaveCount(cardCount);
+```
+
+**Signature:** `page.getLocatorsByAI(prompt: string, options?: { model?: AIModel }): Promise<{ locator: Locator, count: number, reason: string }>`
+
+**Properties:**
+- Returns a Playwright `Locator` usable for interactions and assertions
+- `count` indicates how many elements were found (0 if none)
+- `reason` contains the AI's explanation of what it found
+- Requires Playwright v1.54.1 or higher
+
+**Best Practices:**
+- Describe elements by accessible properties (labels, roles, text) rather than visual attributes (colors, positioning)
+- Best for finding elements when CSS selectors or test IDs are unreliable
+
+## AI Agent (agent.act)
+
+Use the `agent` fixture for complex, autonomous workflows:
+
+```typescript
+test("complex workflow", async ({ agent, page }) => {
+  await page.goto("/orders");
+  await agent.act("Find the first pending order and mark it as shipped", { page });
+});
+
+// Or create manually
+const agent = context.newAgent();
+await agent.act("Your task here", { page, maxCycles: 10 });
+```
+
+**Signature:** `agent.act(prompt: string, options: { page: Page, maxCycles?: number, model?: string }): Promise<{ success: boolean }>`
+
+**Default maxCycles:** 30
+**Supported models:** `anthropic/claude-sonnet-4-6` (default), `google/gemini-2.5-computer-use-preview-10-2025`
+
+### Passing Variables to Prompts
+
+Use template literals to pass variables:
+
+```typescript
+const duration = 24 * 7 * 60;
+await agent.act(`Enter the duration of ${duration} seconds`, { page });
+
+const username = "john.doe@example.com";
+await agent.act(`Login with username ${username}`, { page });
+```
+
+### Self-Contained Prompts (CRITICAL)
+
+All prompts to Stably SDK AI methods must be self-contained with all necessary information:
+
+1. **No implicit references to outside context:**
+   - Bad: `agent.act("Verify the field you just filled in the form is 4", { page })`
+   - Good: `agent.act("Verify the 'timeout' field in the form has value 4", { page })`
+   - Bad: `agent.act("Pick something that's not in the previous step", { page })`
+   - Good: `const selectedItem = "Option A"; await agent.act(\`Pick an option other than ${selectedItem}\`, { page })`
+
+2. **Pass information between AI methods using explicit variables:**
+   ```typescript
+   const orderId = await page.extract("Get the order ID from the first row");
+   await agent.act(`Cancel order with ID ${orderId}`, { page });
+   ```
+
+3. **Include detailed instructions and domain knowledge:**
+   - Bad: `agent.act("Fill in the form", { page })`
+   - Good: `agent.act("Fill in the form with test data. On page 4 you might run into a popup asking for premium features - just click 'Skip' or 'Cancel' to ignore it", { page })`
+
+### Offload Work to Playwright
+
+The less actions/cycles agent.act() needs, the better it performs. Offload work to Playwright code:
+
+1. **Repetition:** Use loops in code, not in prompts
+   - Bad: "Click the button 5 times"
+   - Good: "Click the button" (in a loop that runs 5 times)
+
+2. **Calculations:** Calculate in code, pass result to prompt
+   - Bad: "enter the duration of 24*7*60 seconds"
+   - Good: `const sum = 24*7*60; agent.act(\`enter the duration of ${sum} seconds\`, { page })`
+
+3. **Conditionals:** Use code for if/else when possible
+
+### agent.act() Best Practices
+
+1. **Split complex prompts into smaller tasks:**
+   ```typescript
+   // Bad - too many steps
+   await agent.act('Close popups, click menu, expand panel, find sliders', { page, maxCycles: 15 });
+
+   // Good - single responsibility
+   await agent.act('Close the tutorial popup if visible', { page, maxCycles: 5 });
+   await agent.act('Click the color menu and expand Basic Color panel', { page, maxCycles: 8 });
+   ```
+
+2. **Be specific in prompts** - Include visual hints:
+   ```typescript
+   // Bad
+   await agent.act('Adjust the slider', { page });
+
+   // Good
+   await agent.act('Move the "Brightness" slider (the one with the sun icon) to approximately 75%', { page });
+   ```
+
+3. **Use appropriate maxCycles:**
+   - Simple single action: 3-5 cycles
+   - Multi-step interaction: 8-10 cycles
+   - Complex workflow: 15-20 cycles
+   - Never rely on the default 30 cycles
+
+## Model Selection
+
+`aiAssert`, `extract`, and `getLocatorsByAI` support an optional `model` parameter:
+
+```typescript
+await expect(page).aiAssert("Shows the dashboard", { model: "google/gemini-3-flash-preview" });
+const data = await page.extract("Get the price", { model: "openai/o4-mini" });
+const { locator } = await page.getLocatorsByAI("the submit button", { model: "google/gemini-3-pro-preview" });
+```
+
+**Available models:**
+- `"openai/o4-mini"` - OpenAI's efficient reasoning model
+- `"google/gemini-3-pro-preview"` - Google's most capable model
+- `"google/gemini-3-flash-preview"` - Google's fast, efficient model (good for simple tasks)
+
+**Tips:**
+- Use `gemini-3-flash-preview` for fast, simple operations
+- Use `gemini-3-pro-preview` or `o4-mini` for complex reasoning
+
+## Email Inbox Testing (@stablyai/email)
+
+Use disposable email inboxes for testing email-dependent flows:
+
+```typescript
+import { Inbox } from "@stablyai/email";
+
+// Create a test-scoped inbox
+const inbox = await Inbox.build({ suffix: `test-${Date.now()}` });
+// inbox.address → "org+test-1706621234567@mail.stably.ai"
+
+// Wait for an email
+const email = await inbox.waitForEmail({
+  from: "noreply@example.com",
+  subject: "verification",
+  timeoutMs: 60_000,
+});
+
+// AI-powered extraction
+const { data: otp } = await inbox.extractFromEmail({
+  id: email.id,
+  prompt: "Extract the 6-digit OTP code",
+});
+
+// Structured extraction with Zod
+import { z } from "zod";
+const { data } = await inbox.extractFromEmail({
+  id: email.id,
+  prompt: "Extract verification URL and expiration",
+  schema: z.object({ url: z.string().url(), expiresIn: z.string() }),
+});
+```
+
+### Playwright Fixture Pattern
+
+For test isolation and automatic cleanup:
+
+```typescript
+import { test as base } from "@stablyai/playwright-test";
+import { Inbox } from "@stablyai/email";
+
+const test = base.extend<{ inbox: Inbox }>({
+  inbox: async ({}, use, testInfo) => {
+    const inbox = await Inbox.build({ suffix: `test-${testInfo.testId}` });
+    await use(inbox);
+    await inbox.deleteAllEmails();
+  },
+});
+```
+
+### Key Options
+
+**waitForEmail:** `from`, `subject`, `subjectMatch` (`'contains'` | `'exact'`), `timeoutMs` (default 120000), `pollIntervalMs` (default 3000)
+
+**extractFromEmail:** `id` (required), `prompt` (required), `schema` (optional Zod). Returns `{ data, reason }`
+
+**Best practices:**
+- Always use unique suffixes for parallel test isolation
+- Use the fixture pattern for automatic cleanup
+- Prefer `waitForEmail` over polling with `listEmails`
+
+## When to Use SDK vs Playwright
+
+**Use Playwright** when:
+- Simple, concrete checks (element visible, text matches, URL correct)
+- Faster and more reliable for deterministic scenarios
+
+**Use Stably SDK** when:
+1. Test accuracy and stability are paramount
+2. Interactions are hard to express in Playwright or too brittle
+3. Canvas-related operations or drag/click requiring coordinates → use `agent.act()`
+4. Visual-heavy assertions → use `aiAssert`
+5. Email verification flows → use `@stablyai/email`
+
+**IMPORTANT:** Do NOT use `toHaveScreenshot()` - this Playwright assertion is not supported. Use `aiAssert` for ALL visual assertions.
+
+## Minimal Template
+
+```typescript
+import { test, expect } from "@stablyai/playwright-test";
+
+test("AI-enhanced dashboard", async ({ page, agent }) => {
+  await page.goto("/dashboard");
+
+  // Use agent for complex workflows
+  await agent.act("Navigate to settings and enable notifications", { page });
+
+  // Use AI assertions for dynamic content
+  await expect(page).aiAssert(
+    "Dashboard shows revenue chart (>= 6 months) and account spotlight card"
+  );
+});
+```
+
+## Troubleshooting
+
+- **Slow assertions** → Scope visuals with locators; reduce viewport
+- **Agent stops early** → Increase `maxCycles` or break task into smaller steps