stably 4.8.9 → 4.10.0

package/dist/stably-plugin-cli/skills/test-creation-workflow/SKILL.md

@@ -0,0 +1,311 @@
+---
+name: test-creation-workflow
+description: CLI workflow for creating Playwright test files with planning phase, User Prompt comments, linkId guardrails, and helper patterns. Use when creating new tests, planning test implementation, writing test skeletons, or creating helper functions. Triggers on new test, create test, plan test, User Prompt comment, helper function.
+---
+
+<!-- CLI-ONLY SKILL: Proceeds directly to implementation after planning.
+     The web version uses AskUserQuestion for ambiguity and requires plan approval. -->
+
+# Test Creation Workflow
+
+Complete workflow for creating high-quality Playwright tests.
+
+## Complexity Assessment (Do This First)
+
+Before starting, classify the task:
+
+| Complexity | Criteria | Planning Approach |
+|------------|----------|-------------------|
+| **Trivial** | ≤5 steps, public site, no auth, no SDK features | Skip formal planning. One-liner plan: "Navigate to X, assert Y, use Z project." |
+| **Moderate** | 6-15 steps, OR auth required, OR SDK features | Lightweight plan: Summary + Assertion Matrix only |
+| **Complex** | 15+ steps, multiple preconditions, helpers needed | Full planning phase (A-F + all matrices) |
+
+**For trivial tasks:** Skip the full planning checklist. Just create the test file and implement directly.
+
+## Workflow Overview
+
+1. **Read context** - Understand existing test files, helpers, and config
+2. **Plan** - Answer planning questions before implementing
+3. **Create test file** - Under `tests/` with descriptive name
+4. **Setup browser** - See `browser-interaction-guide` skill
+5. **Implement flow** - Navigate and interact with browser
+6. **Write intermediate code** - For long flows, checkpoint progress
+7. **Verify with test_run** - Final verification required
+8. **Create helpers** - Extract reusable patterns
+
+## Planning Phase
+
+Answer these questions before implementation.
+
+### Planning Questions
+
+**A. Intent Analysis**
+- What is the intended behavior that needs testing?
+- What user journey or feature is being validated?
+- What is the success criteria for this test?
+
+**B. Prerequisites Discovery**
+- What test data is required? (users, products, specific states)
+- What authentication/authorization is needed?
+- What navigation steps are needed to reach the test starting point?
+- Are there existing helpers that handle these prerequisites?
+- **Hidden preconditions** (discover from source code):
+  * UI state (theme, dark mode, viewport size)
+  * Domain/data state (settings, configuration that must exist first)
+  * Feature flags, permissions, role/tenant gates
+  * Route/query parameters, deep link requirements
+  * Persisted client state (localStorage, cookies, session)
+  * Email verification steps (OTP codes, magic links, confirmation emails)
+
+**C. Test Strategy**
+- Is this test idempotent? (can run multiple times without side effects)
+- How to make this test fast and reliable?
+- What assertions provide the best coverage without being flaky?
+- Are there edge cases worth covering in separate tests?
+
+**D. Codebase Patterns**
+- What existing test patterns should be followed?
+- What helper functions can be reused?
+- What naming conventions are used in this project?
+
+**E. Assertion Quality & Flakiness Contract**
+- What are the 2-4 critical outcomes that must be verified?
+- For each outcome, what deterministic Playwright assertion will be used?
+- For value-critical outcomes, what is the source of truth for the expected value?
+- If an exact value cannot be known ahead of time, how will you control the value in setup?
+- For each outcome, is `aiAssert` needed? If yes, why is Playwright-only assertion insufficient?
+- Which assertions risk flakiness and how will you avoid it?
+- **Auth setup**: Does the config have auth project dependencies? (See `playwright-config-auth` skill)
+
+**BROWSER-VALIDATE BEFORE ASSERTING**: Do not assume text content - verify via browser first.
+- Before writing text assertions, use `test_debug` or `browser_snapshot` to see actual content
+- "I think the heading says X" is not verification - browser inspection is
+- This prevents fix cycles from wrong text assumptions (e.g., "illustrative examples" vs "documentation examples")
+
+**ANTI-HEDGING RULE**: If you discovered the expected value, assert that exact value.
+- If the value is "1m 20s", assert `toHaveText("1m 20s")`, NOT a regex like `/\d+m\s+\d+s/`
+- If the computed style is 12px, assert `toBe(12)`, NOT a range like `toBeLessThanOrEqual(14)`
+- If the PR changes a background color, assert the background color, NOT class presence
+- Regex, ranges, and presence checks are ONLY acceptable when the exact value is genuinely unknowable
+
+**VERIFY DATA CLAIMS**: Precondition values from backend data must be verified through a source independent of the system under test.
+- "I saw X in the browser" verifies the UI renders something, not that the underlying data is X
+- This does NOT apply to deterministic code outputs (CSS values, hardcoded labels)
+- It DOES apply to values from backend data, configuration, or data-routing decisions
+- Verification must be an assertion: query the source, assert it equals expected, THEN assert UI matches
+- If you cannot verify a precondition value, flag it as UNVERIFIED in ASSERTION MATRIX and OPEN UNKNOWNS
+
+**F. Navigation Blueprint**
+- What is the exact ordered navigation path from entry state to each critical assertion point?
+- For each transition, what action triggers it and what locator strategy will be used?
+- What deterministic readiness signal proves each transition succeeded?
+
+### Output the Plan
+
+Output plan in plain text with these sections:
+- **Plan Summary**
+- **Preconditions Matrix** (precondition, source evidence, setup action, verification check)
+- **Assertion Matrix** (critical outcome, Playwright assertion(s), source of truth)
+- **Flake Controls**
+- **Open Unknowns**
+
+### Assertion Type Selection
+
+- **Standard Playwright assertions** (`toHaveText`, `toBeVisible`, `toHaveURL`): Use for simple, concrete checks
+- **`aiAssert`**: Use for complex visual layouts, dynamic content, semantic meaning
+
+Don't use `aiAssert` for things checkable with standard assertions.
+
+### Assertion Timeouts
+
+Prefer relying on Playwright's `expect.timeout` config setting rather than adding explicit timeouts.
+
+```typescript
+// Preferred - uses expect.timeout from playwright.config.ts
+await expect(page.getByText('Welcome')).toBeVisible();
+```
+
+Only add explicit timeouts for operations known to be slow (file uploads, video processing).
+
+## Creating the Test File
+
+Create test file under `tests/` with a descriptive name (e.g., `login-flow.spec.ts`).
+
+- Do NOT put test cases in `seed.spec.ts` - it's just a placeholder
+- Match existing test file conventions in the project
+
+## User Prompt Comment
+
+Every test must have a User Prompt comment recording the user's original request.
+
+### Rules
+
+- Every test must have one, no exceptions
+- Write user's exact words - do not modify or rephrase
+- Place directly above each test block
+
+### Single Test Format
+
+```typescript
+import { test, expect } from '@stablyai/playwright-test';
+
+test.describe('User Flow Description', () => {
+  /**
+   * User Prompt:
+   * - <copy user's exact words here - no modifications>
+   *
+   * [Clarifications: (only include if questions were asked)]
+   * - <question topic>: <user's answer>
+   */
+  test('descriptive test name', async ({ page }) => {
+    // test code
+  });
+});
+```
+
+### Multiple Tests from One Prompt
+
+When creating multiple tests from a single user prompt, split into common context + specific parts for each test:
+
+```typescript
+test.describe('User Flow Description', () => {
+  /**
+   * User Prompt:
+   * - <common context from user's prompt>
+   * - <specific part for this test>
+   */
+  test('first test name', async ({ page }) => {
+    // test code
+  });
+
+  /**
+   * User Prompt:
+   * - <common context from user's prompt>
+   * - <specific part for this test>
+   */
+  test('second test name', async ({ page }) => {
+    // test code
+  });
+});
+```
+
+### When Modifying Tests
+
+- **Intent changed** (e.g., "change this test to also verify X"): Update the User Prompt comment
+- **Intent unchanged** (e.g., "fix this error"): Do not modify the User Prompt comment
+
+## linkId Guardrails
+
+`@stably.linkId` is runtime-owned metadata in runnable test files (`.spec.*`/`.test.*`) only.
+
+- Agent must never invent, rewrite, or delete `@stably.linkId`
+- During regenerate/overwrite flows, preserve existing `@stably.linkId` header line exactly
+- If no `@stably.linkId` exists, proceed with content edits but do not add a guessed ID
+- Runtime finalize is responsible for inserting or validating `@stably.linkId`
+- Do not finish with duplicate or malformed `@stably.linkId` headers
+
+## Browser Setup
+
+### Tool Decision Tree
+
+```
+What do you need to do?
+├── Create/develop a test (keep browser open to see state)
+│   └── Use: generator_setup_page
+│       - Requires: seedFile with top-level test() call
+│       - Browser stays open regardless of pass/fail
+│
+├── Debug a failing test (step through, inspect on failure)
+│   └── Use: test_debug
+│       - Browser closes on success, stays open on failure
+│       - Good for iterative fix cycles
+│
+└── Verify a test passes (final validation)
+    └── Use: test_run
+        - Browser closes after run
+        - Use for final verification before completing task
+```
+
+### Common Mistake
+
+`generator_setup_page` requires a seed file with a **top-level `test()` call**. Tests wrapped in `test.describe()` return "seed test not found".
+
+**Workaround:** Use `tests/seed.spec.ts` as seedFile, then navigate manually with `browser_navigate`.
+
+### Project Selection
+
+For auth dependencies, see `playwright-config-auth` skill.
+
+## Long User Flows (15+ actions)
+
+For flows with more than 15 actions, checkpoint progress:
+
+1. Reach a stable intermediate state
+2. Call `generator_read_log` to retrieve action code
+3. Write intermediate code with `generator_write_test`
+   - fileName must be a relative path (e.g., `tests/my-test.spec.ts`)
+4. Use `test_debug` to verify intermediate code works
+5. Repeat until entire flow is complete
+
+## Helper Functions
+
+### When to Create
+
+Extract repetitive code patterns: login flows, navigation patterns, form filling, common assertions.
+
+### Location & Naming
+
+- Place in `tests/helpers/`
+- Use format `<name>.helper.ts` (e.g., `login.helper.ts`)
+
+### Requirements
+
+```typescript
+import { Page } from '@playwright/test';
+
+/**
+ * Logs in a user with the provided credentials.
+ */
+export async function login({
+  page,
+  email,
+  password,
+}: {
+  page: Page;
+  email: string;
+  password: string;
+}): Promise<void> {
+  // implementation
+}
+```
+
+- Export functions with clear, descriptive names
+- Use single-object parameters
+- Include TypeScript types
+- Add JSDoc comments
+
+### Using Existing Helpers
+
+Before creating new helpers, check `tests/helpers/` for existing ones.
+
+### Debugging Helper Errors
+
+1. Inspect other test files using the same helper to verify your usage pattern
+2. If your implementation is correct, modify the helper itself
+3. Only if steps 1-2 fail, eject helper code into your test or create new helper
+
+## Final Verification
+
+You must run `test_run` for final verification before the task is complete.
+
+- A test without `test_run` verification is not finished
+- If it fails, fix and run `test_run` again until it passes
+- Use `test_debug` for development and debugging, `test_run` for final verification
+
+For project selection guidance, see `playwright-config-auth` skill.
+
+## Completion Checklist
+
+- [ ] Test file created
+- [ ] `test_run` executed and passed
+- [ ] Self-review completed (if code was modified)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stably",
-  "version": "4.8.9",
+  "version": "4.10.0",
   "packageManager": "pnpm@10.24.0",
   "description": "AI-powered E2E Playwright testing CLI. Stably can understand your codebase, edit/run tests, and handle complex test scenarios for you.",
   "main": "dist/index.mjs",
@@ -91,7 +91,9 @@
     "@stablyai/agent-tool-display": "workspace:*",
     "@stablyai/agent-workspace-info": "workspace:*",
     "@stablyai/ci-run-id": "0.1.1",
+    "@stablyai/email": "0.1.2",
     "@stablyai/git-diff-tracker": "workspace:*",
+    "@stablyai/playwright-test": "2.1.10",
     "@stablyai/playwright-trace-scrubber": "workspace:*",
     "@stablyai/proxy": "workspace:*",
     "@tsconfig/node20": "^20.1.4",
@@ -102,6 +104,7 @@
     "dotenv-cli": "^11.0.0",
     "execa": "^9.5.3",
     "node-pty": "1.2.0-beta.10",
+    "playwright": "1.57.0",
     "terser": "^5.44.1",
     "tsup": "^8.5.0",
     "tsx": "^4.20.5",

package/dist/stably-plugin/.claude-plugin/plugin.json

@@ -1,5 +0,0 @@
-{
-  "name": "stably-agent",
-  "description": "Internal skills for the Stably CLI agent",
-  "version": "1.0.0"
-}

package/dist/stably-plugin/skills/playwright-best-practices/references/accessibility.md

@@ -1,359 +0,0 @@
-# Accessibility Testing
-
-## Table of Contents
-
-1. [Axe-Core Integration](#axe-core-integration)
-2. [Keyboard Navigation](#keyboard-navigation)
-3. [ARIA Validation](#aria-validation)
-4. [Focus Management](#focus-management)
-5. [Color & Contrast](#color--contrast)
-
-## Axe-Core Integration
-
-### Setup
-
-```bash
-npm install -D @axe-core/playwright
-```
-
-### Basic A11y Test
-
-```typescript
-import { test, expect } from "@playwright/test";
-import AxeBuilder from "@axe-core/playwright";
-
-test("homepage should have no a11y violations", async ({ page }) => {
-  await page.goto("/");
-
-  const results = await new AxeBuilder({ page }).analyze();
-
-  expect(results.violations).toEqual([]);
-});
-```
-
-### Scoped Analysis
-
-```typescript
-test("form accessibility", async ({ page }) => {
-  await page.goto("/contact");
-
-  // Analyze only the form
-  const results = await new AxeBuilder({ page })
-    .include("#contact-form")
-    .analyze();
-
-  expect(results.violations).toEqual([]);
-});
-
-test("ignore known issues", async ({ page }) => {
-  await page.goto("/legacy-page");
-
-  const results = await new AxeBuilder({ page })
-    .exclude(".legacy-widget") // Skip legacy component
-    .disableRules(["color-contrast"]) // Disable specific rule
-    .analyze();
-
-  expect(results.violations).toEqual([]);
-});
-```
-
-### A11y Fixture
-
-```typescript
-// fixtures/a11y.fixture.ts
-import { test as base } from "@playwright/test";
-import AxeBuilder from "@axe-core/playwright";
-
-type A11yFixtures = {
-  makeAxeBuilder: () => AxeBuilder;
-};
-
-export const test = base.extend<A11yFixtures>({
-  makeAxeBuilder: async ({ page }, use) => {
-    await use(() =>
-      new AxeBuilder({ page }).withTags([
-        "wcag2a",
-        "wcag2aa",
-        "wcag21a",
-        "wcag21aa",
-      ]),
-    );
-  },
-});
-
-// Usage
-test("dashboard a11y", async ({ page, makeAxeBuilder }) => {
-  await page.goto("/dashboard");
-  const results = await makeAxeBuilder().analyze();
-  expect(results.violations).toEqual([]);
-});
-```
-
-### Detailed Violation Reporting
-
-```typescript
-test("report a11y issues", async ({ page }) => {
-  await page.goto("/");
-
-  const results = await new AxeBuilder({ page }).analyze();
-
-  // Custom failure message with details
-  const violations = results.violations.map((v) => ({
-    id: v.id,
-    impact: v.impact,
-    description: v.description,
-    nodes: v.nodes.map((n) => n.html),
-  }));
-
-  expect(violations, JSON.stringify(violations, null, 2)).toHaveLength(0);
-});
-```
-
-## Keyboard Navigation
-
-### Tab Order Testing
-
-```typescript
-test("correct tab order in form", async ({ page }) => {
-  await page.goto("/signup");
-
-  // Start from the beginning
-  await page.keyboard.press("Tab");
-  await expect(page.getByLabel("Email")).toBeFocused();
-
-  await page.keyboard.press("Tab");
-  await expect(page.getByLabel("Password")).toBeFocused();
-
-  await page.keyboard.press("Tab");
-  await expect(page.getByRole("button", { name: "Sign up" })).toBeFocused();
-});
-```
-
-### Keyboard-Only Interaction
-
-```typescript
-test("complete flow with keyboard only", async ({ page }) => {
-  await page.goto("/products");
-
-  // Navigate to product with keyboard
-  await page.keyboard.press("Tab"); // Skip to main content
-  await page.keyboard.press("Tab"); // First product
-  await page.keyboard.press("Enter"); // Open product
-
-  await expect(page).toHaveURL(/\/products\/\d+/);
-
-  // Add to cart with keyboard
-  await page.keyboard.press("Tab");
-  await page.keyboard.press("Tab"); // Navigate to "Add to Cart"
-  await page.keyboard.press("Enter");
-
-  await expect(page.getByRole("alert")).toContainText("Added to cart");
-});
-```
-
-### Skip Links
-
-```typescript
-test("skip link works", async ({ page }) => {
-  await page.goto("/");
-
-  await page.keyboard.press("Tab");
-  const skipLink = page.getByRole("link", { name: /skip to main/i });
-  await expect(skipLink).toBeFocused();
-
-  await page.keyboard.press("Enter");
-
-  // Focus should move to main content
-  await expect(page.getByRole("main")).toBeFocused();
-});
-```
-
-### Escape Key Handling
-
-```typescript
-test("escape closes modal", async ({ page }) => {
-  await page.goto("/dashboard");
-  await page.getByRole("button", { name: "Settings" }).click();
-
-  const modal = page.getByRole("dialog");
-  await expect(modal).toBeVisible();
-
-  await page.keyboard.press("Escape");
-
-  await expect(modal).toBeHidden();
-  // Focus should return to trigger
-  await expect(page.getByRole("button", { name: "Settings" })).toBeFocused();
-});
-```
-
-## ARIA Validation
-
-### Role Verification
-
-```typescript
-test("correct ARIA roles", async ({ page }) => {
-  await page.goto("/dashboard");
-
-  // Verify landmark roles
-  await expect(page.getByRole("navigation")).toBeVisible();
-  await expect(page.getByRole("main")).toBeVisible();
-  await expect(page.getByRole("contentinfo")).toBeVisible(); // footer
-
-  // Verify interactive roles
-  await expect(page.getByRole("button", { name: "Menu" })).toBeVisible();
-  await expect(page.getByRole("search")).toBeVisible();
-});
-```
-
-### ARIA States
-
-```typescript
-test("aria-expanded updates correctly", async ({ page }) => {
-  await page.goto("/faq");
-
-  const accordion = page.getByRole("button", { name: "Shipping" });
-
-  // Initially collapsed
-  await expect(accordion).toHaveAttribute("aria-expanded", "false");
-
-  await accordion.click();
-
-  // Now expanded
-  await expect(accordion).toHaveAttribute("aria-expanded", "true");
-
-  // Content is visible
-  const panel = page.getByRole("region", { name: "Shipping" });
-  await expect(panel).toBeVisible();
-});
-```
-
-### Live Regions
-
-```typescript
-test("live region announces updates", async ({ page }) => {
-  await page.goto("/checkout");
-
-  // Find live region
-  const liveRegion = page.locator('[aria-live="polite"]');
-
-  await page.getByLabel("Quantity").fill("3");
-
-  // Live region should update with new total
-  await expect(liveRegion).toContainText("Total: $29.97");
-});
-```
-
-## Focus Management
-
-### Focus Trap in Modal
-
-```typescript
-test("focus trapped in modal", async ({ page }) => {
-  await page.goto("/");
-  await page.getByRole("button", { name: "Open Modal" }).click();
-
-  const modal = page.getByRole("dialog");
-  await expect(modal).toBeVisible();
-
-  // Get all focusable elements in modal
-  const focusableElements = modal.locator(
-    'button, [href], input, select, textarea, [tabindex]:not([tabindex="-1"])',
-  );
-  const count = await focusableElements.count();
-
-  // Tab through all elements, should stay in modal
-  for (let i = 0; i < count + 1; i++) {
-    await page.keyboard.press("Tab");
-    const focused = page.locator(":focus");
-    await expect(modal).toContainText((await focused.textContent()) || "");
-  }
-});
-```
-
-### Focus Restoration
-
-```typescript
-test("focus returns after modal close", async ({ page }) => {
-  await page.goto("/");
-
-  const trigger = page.getByRole("button", { name: "Delete Item" });
-  await trigger.click();
-
-  await page.getByRole("button", { name: "Cancel" }).click();
-
-  // Focus should return to the trigger
-  await expect(trigger).toBeFocused();
-});
-```
-
-## Color & Contrast
-
-### High Contrast Mode
-
-```typescript
-test("works in high contrast mode", async ({ page }) => {
-  await page.emulateMedia({ forcedColors: "active" });
-  await page.goto("/");
-
-  // Verify key elements are visible
-  await expect(page.getByRole("navigation")).toBeVisible();
-  await expect(page.getByRole("button", { name: "Sign In" })).toBeVisible();
-
-  // Take screenshot for visual verification
-  await expect(page).toHaveScreenshot("high-contrast.png");
-});
-```
-
-### Reduced Motion
-
-```typescript
-test("respects reduced motion preference", async ({ page }) => {
-  await page.emulateMedia({ reducedMotion: "reduce" });
-  await page.goto("/");
-
-  // Animations should be disabled
-  const hero = page.getByTestId("hero-animation");
-  const animation = await hero.evaluate(
-    (el) => getComputedStyle(el).animationDuration,
-  );
-
-  expect(animation).toBe("0s");
-});
-```
-
-## CI Integration
-
-### A11y as CI Gate
-
-```typescript
-// playwright.config.ts
-export default defineConfig({
-  projects: [
-    {
-      name: "a11y",
-      testMatch: /.*\.a11y\.spec\.ts/,
-      use: { ...devices["Desktop Chrome"] },
-    },
-  ],
-});
-```
-
-```yaml
-# .github/workflows/a11y.yml
-- name: Run accessibility tests
-  run: npx playwright test --project=a11y
-```
-
-## Anti-Patterns to Avoid
-
-| Anti-Pattern                  | Problem                      | Solution                                   |
-| ----------------------------- | ---------------------------- | ------------------------------------------ |
-| Testing a11y only on homepage | Misses issues on other pages | Test all critical user flows               |
-| Ignoring all violations       | No value from tests          | Address or explicitly exclude known issues |
-| Only automated testing        | Misses many a11y issues      | Combine with manual testing                |
-| Testing without screen reader | Misses interaction issues    | Test with VoiceOver/NVDA periodically      |
-
-## Related References
-
-- **Locators**: See [locators.md](locators.md) for role-based selectors
-- **Visual testing**: See [test-organization.md](test-organization.md) for screenshot comparison