shiplightai 0.1.49 → 0.1.50

package/README.md

@@ -1,672 +1,88 @@
 # shiplightai
 
-The best way to write UI tests — YAML with natural language steps. Easy for AI agents to create and maintain, yet clear for humans to understand and control.
+AI-powered end-to-end testing for Playwright. Write tests as YAML with natural-language steps, run them alongside your existing `.test.ts` files, get self-healing locators and a visual debugger. One package ships both the `shiplight` CLI and the library you wire into `playwright.config.ts`.
 
-AI-powered execution with self-healing locators means near-zero maintenance — no more broken tests every time the UI changes. Compatible with Playwright, running alongside your existing `.test.ts` files with `npx playwright test` — no separate tooling needed.
+**Full documentation: [docs.shiplight.ai](https://docs.shiplight.ai)**
 
-**AI does the work. Humans have visibility and control.**
+## Quick start
 
-## Quick Start
+Scaffold a new test project in under a minute:
 
-### 1. Install
-
-```bash
-npm install -D shiplightai @playwright/test
-```
-
-### 2. Configure
-
-In your `playwright.config.ts`:
-
-```ts
-import { defineConfig, shiplightConfig } from 'shiplightai';
-
-export default defineConfig({
-  ...shiplightConfig(),
-
-  testDir: './tests',
-  use: {
-    headless: true,
-    viewport: { width: 1280, height: 720 },
-  },
-});
-```
-
-### 3. Set up API keys
-
-At least one AI API key is required. You can either export it directly or use a `.env` file.
-
-**Option A: Environment variable**
-```bash
-export ANTHROPIC_API_KEY=sk-ant-...
-# or export GOOGLE_API_KEY=...
-```
-
-**Option B: `.env` file** (in your test directory or project root)
 ```bash
-ANTHROPIC_API_KEY=sk-ant-...
-# GOOGLE_API_KEY=...
+npx shiplightai@latest create ./my-tests
+cd my-tests
+cp .env.example .env            # then set one of GOOGLE_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY
+npm install
+npx playwright install chromium
+npx shiplight test              # runs the scaffolded starter test
 ```
 
-`shiplightConfig()` auto-discovers `.env` files by walking up the directory tree — no manual `dotenv` setup needed.
-
-The AI model is auto-detected from your API key (`ANTHROPIC_API_KEY` → `claude-haiku-4-5`, `GOOGLE_API_KEY` → `gemini-3.1-flash-lite-preview`, `OPENAI_API_KEY` → `gpt-5.4-mini`). Set `WEB_AGENT_MODEL` to override.
-
-### 4. Write a YAML test
-
-Create `tests/login.test.yaml`:
-
-```yaml
-goal: Verify user can log in
-url: https://example.com/login
-
-statements:
-  - Click on the username field and type "testuser"
-  - Click on the password field and type "secret123"
-  - Click the Login button
-  - "VERIFY: Dashboard page is visible"
-```
+The scaffolder writes `package.json`, `playwright.config.ts`, `.env.example`, `.gitignore`, and a runnable `tests/example.test.yaml` that exercises a live site. Open `shiplight-report/index.html` after the run to see per-step screenshots, videos, and traces.
 
-### 5. Run
+Or add Shiplight to an existing Playwright project:
 
 ```bash
-npx playwright test
+npm install shiplightai @playwright/test
 ```
 
-Playwright discovers both `*.test.ts` and `*.test.yaml` files. YAML files are transparently transpiled to `.yaml.spec.ts` files next to the source.
-
-### 6. Gitignore generated files
-
-Add to your `.gitignore`:
+## CLI
 
-```
-*.yaml.spec.ts
-auth.setup.ts
-.auth/
-.env
-```
+Every command is invoked via `npx shiplight <command>` from inside a project that has `shiplightai` installed.
 
-## Project Structure
+| Command | Purpose |
+|---|---|
+| `shiplight create <path>` | Scaffold a new test project |
+| `shiplight test [args]` | Run your YAML and Playwright test suite (forwards all Playwright flags) |
+| `shiplight debug <file>` | Launch the interactive visual debugger for a YAML test |
+| `shiplight report [folder]` | Regenerate or merge HTML reports |
+| `shiplight transpile [glob]` | Transpile YAML tests to `.yaml.spec.ts` (usually automatic) |
 
-A typical Shiplight test project follows standard Playwright conventions — a `playwright.config.ts` at the root, test files in subdirectories, and a `.gitignore`. On top of that, Shiplight adds two config files: `.env` for API keys (shared across all projects) and `shiplight.config.json` for per-project login credentials (only needed if the app requires authentication).
+Full reference with flags, options, and troubleshooting: **[docs.shiplight.ai/local/cli-reference](https://docs.shiplight.ai/local/cli-reference)**.
 
-```
-my-tests/
-├── playwright.config.ts
-├── package.json
-├── .env                            # API keys — shared by ALL projects (gitignored)
-├── .env.example                    # Checked in, documents required keys
-├── .gitignore
-│
-├── airbnb/                         # Project 1: public site, no login needed
-│   ├── search.test.yaml
-│   ├── filter.test.yaml
-│   └── listing.test.yaml
-│
-├── my-saas-app/                    # Project 2: requires login
-│   ├── shiplight.config.json       # {"url":"https://my-saas.com","username":"...","password":"..."}
-│   ├── dashboard.test.yaml
-│   ├── settings.test.yaml
-│   └── billing.test.yaml
-│
-└── admin-portal/                   # Project 3: different app, different login
-    ├── shiplight.config.json       # {"url":"https://admin.my-saas.com","username":"...","password":"..."}
-    ├── users.test.yaml
-    └── audit-log.test.yaml
-```
+## Library
 
-### Root files
+Wire up YAML support and the Shiplight reporter in `playwright.config.ts`:
 
-**`playwright.config.ts`** — Playwright config with `shiplightConfig()`:
 ```ts
 import { defineConfig, shiplightConfig } from 'shiplightai';
 
 export default defineConfig({
   ...shiplightConfig(),
 
-  testDir: '.',
-  timeout: 120_000,
+  testDir: './tests',
   use: {
-    headless: false,
+    headless: true,
     viewport: { width: 1280, height: 720 },
   },
 });
 ```
 
-`shiplightConfig()` runs during Playwright config loading and:
-1. Walks up from `scanDir` to the project root looking for `.env` files and loads them (closer files take precedence)
-2. Scans for `**/*.test.yaml` files
-3. Transpiles each to a `*.yaml.spec.ts` file next to the source
-
-Playwright then discovers the generated `.yaml.spec.ts` files through its default `testMatch` pattern. If you override `testMatch`, make sure it includes `*.spec.ts`.
-
-**`.env`** — API keys, shared by all projects (gitignored):
-```bash
-# At least one required. Model is auto-detected from the key.
-ANTHROPIC_API_KEY=sk-ant-...
-# GOOGLE_API_KEY=...
-
-# Optional: override auto-detected model
-# WEB_AGENT_MODEL=claude-haiku-4-5
-```
-
-**`.gitignore`**:
-```
-node_modules/
-test-results/
-*.yaml.spec.ts
-auth.setup.ts
-.auth/
-.env
-```
-
-### Subdirectories
-
-Each subdirectory is a separate project that can test a different application. Subdirectories that require login include a `shiplight.config.json` with credentials. Subdirectories that don't need login (e.g., public sites) simply omit it.
-
-**`my-saas-app/shiplight.config.json`** — login + variables for one app:
-```json
-{
-  "url": "https://my-saas.com",
-  "username": "qa@my-saas.com",
-  "password": "test-password",
-  "variables": {
-    "BASE_URL": "https://my-saas.com",
-    "API_TOKEN": { "value": "sk-test-...", "sensitive": true }
-  }
-}
-```
-
-**`admin-portal/shiplight.config.json`** — different app, different credentials:
-```json
-{
-  "url": "https://admin.my-saas.com",
-  "username": "admin@my-saas.com",
-  "password": "admin-password"
-}
-```
-
-### Running tests
-
-**Run all tests:**
-```bash
-npx playwright test
-```
-
-**Run one project:**
-```bash
-npx playwright test my-saas-app/
-```
-
-**Debug a single test:**
-```bash
-shiplight debug my-saas-app/dashboard.test.yaml
-```
-
-The visual debugger uses the same config discovery — it finds `my-saas-app/shiplight.config.json` for login and `.env` at the root for API keys.
-
-## Environment Variables
-
-| Variable | Description | Default |
-|---|---|---|
-| `ANTHROPIC_API_KEY` | Anthropic API key (for Claude models) | — |
-| `GOOGLE_API_KEY` | Google AI API key (for Gemini models) | — |
-| `OPENAI_API_KEY` | OpenAI API key (for GPT/o-series models) | — |
-| `WEB_AGENT_MODEL` | AI model override | Auto-detected from API key |
-| `OPENAI_BASE_URL` | Custom base URL for OpenAI-compatible APIs | — |
-| `SHIPLIGHT_LOGIN_EMAIL` | Login email (with `_PASSWORD`, overrides `shiplight.config.json`) | — |
-| `SHIPLIGHT_LOGIN_PASSWORD` | Login password (with `_EMAIL`, overrides `shiplight.config.json`) | — |
-| `SHIPLIGHT_LOGIN_URL` | Login page URL (overrides `shiplight.config.json` `url`) | — |
-| `SHIPLIGHT_LOGIN_TOTP_SECRET` | TOTP secret for 2FA (overrides `shiplight.config.json` `totp_secret`) | — |
-| `PLAYWRIGHT_STARTING_URL` | Override the starting URL for all tests | — |
-
-At least one AI API key is required. The model is auto-detected: `ANTHROPIC_API_KEY` defaults to `claude-haiku-4-5`, `GOOGLE_API_KEY` defaults to `gemini-3.1-flash-lite-preview`, `OPENAI_API_KEY` defaults to `gpt-5.4-mini`. Set `WEB_AGENT_MODEL` to override.
-
-## Configuration Options
-
-```ts
-shiplightConfig({
-  // Directory to scan for .test.yaml files (default: process.cwd())
-  scanDir: './tests',
-
-  // API key for cloud features (optional)
-  apiKey: process.env.SHIPLIGHT_API_KEY,
-
-  // Auto-discover .env files by walking up the directory tree (default: true)
-  // Set to false for CI pipelines where env vars are injected externally
-  dotenv: false,
-})
-```
-
-### Agent Fixture
-
-The package exports a custom `test` that extends Playwright's `test` with an additional `agent` fixture. It works exactly like Playwright's `test` in every other way — same API, same hooks, same assertions. `expect` is re-exported from Playwright unchanged. Generated YAML tests use this automatically:
-
-```ts
-import { test, expect } from 'shiplightai/fixture';
-
-test('my test', async ({ page, agent }) => {
-  // `agent` is a pre-configured WebAgent instance
-  // `page` is the standard Playwright page
-});
-```
-
-You can also use the fixture in your hand-written `.test.ts` files to get the same `agent` instance:
-
-```ts
-// tests/custom.test.ts
-import { test, expect } from 'shiplightai/fixture';
-
-test('custom test with agent', async ({ page, agent }) => {
-  await page.goto('https://example.com');
-  await agent.run(page, 'Click the login button', 'step-1');
-  await agent.assert(page, 'User is on the dashboard', 'step-2');
-});
-```
-
-## Authentication (Optional)
-
-If your app requires login, add credentials to `shiplight.config.json` and wire up a Playwright setup project. `shiplightConfig()` auto-generates an `auth.setup.ts` file in each directory that has credentials. Skip this section if you're testing public pages.
-
-### 1. Create `shiplight.config.json`
-
-Place this in your test subdirectory (see [Project Structure](#project-structure) above):
-
-```json
-{
-  "url": "https://your-app.com",
-  "username": "testuser@example.com",
-  "password": "your-password",
-  "totp_secret": "JBSWY3DPEHPK3PXP"
-}
-```
-
-The `totp_secret` field is optional — only needed if your app uses 2FA.
-
-### 2. Add setup project to `playwright.config.ts`
-
-Use Playwright's standard project dependencies to run the auto-generated `auth.setup.ts` before authenticated tests:
-
-```ts
-export default defineConfig({
-  ...shiplightConfig(),
-
-  projects: [
-    // Setup project — runs auto-generated auth.setup.ts
-    { name: 'my-app-setup', testDir: './my-app', testMatch: 'auth.setup.ts' },
-    {
-      name: 'my-app',
-      testDir: './my-app',
-      dependencies: ['my-app-setup'],
-      use: { storageState: './my-app/.auth/storage-state.json' },
-    },
-  ],
-});
-```
-
-That's it. No `global-setup.ts` needed — `shiplightConfig()` generates `auth.setup.ts` automatically from your `shiplight.config.json`.
-
-### Credential resolution order
-
-Login credentials are resolved in this order (first match wins):
-
-1. **Environment variables**: `SHIPLIGHT_LOGIN_EMAIL` + `SHIPLIGHT_LOGIN_PASSWORD` (+ optional `SHIPLIGHT_LOGIN_URL`)
-2. **Config file**: `shiplight.config.json` or `login.config.json`, discovered by walking up from the test directory
-
-This means you can use `shiplight.config.json` for local development and override with env vars in CI.
-
-### How it works
-
-1. `shiplightConfig()` scans for `shiplight.config.json` files with credentials and generates `auth.setup.ts` next to each
-2. Playwright runs the setup project before dependent projects
-3. The AI agent logs in using `shiplightai`'s `WebAgent.loginPage()` — no fragile selectors
-4. The resulting cookies/localStorage are saved to `<dir>/.auth/storage-state.json`
-5. All tests in the dependent project load this storage state — every test starts already authenticated
-6. Each directory is self-contained: its own credentials, its own auth state, its own setup
-
-### 2FA / TOTP support
-
-Set `totp_secret` in `shiplight.config.json` or `SHIPLIGHT_LOGIN_TOTP_SECRET` as an env var, and the agent will generate and enter the 2FA code automatically.
-
-## YAML Test Format
-
-### Basic Structure
-
-```yaml
-goal: Description of what this test verifies
-url: https://your-app.com/starting-page
-
-statements:
-  - Step described in natural language
-  - Another step
-  - "VERIFY: Expected outcome"
-
-teardown:
-  - Clean up step
-```
-
-| Field | Required | Description |
-|---|---|---|
-| `goal` | Yes | Test description (used as the Playwright test name) |
-| `url` | Yes | Starting URL to navigate to |
-| `statements` | Yes | List of test steps |
-| `teardown` | No | Steps that always run after the test (like `finally`) |
-
-### Statement Types
-
-#### Draft (natural language)
-
-Plain strings are AI-resolved steps. The agent figures out what to click, type, etc.
-
-```yaml
-statements:
-  - Navigate to the settings page
-  - Click the "Delete Account" button
-  - Type "confirm" in the confirmation dialog
-```
-
-#### VERIFY
-
-Asserts a condition using AI. Must be a quoted string prefixed with `VERIFY:`.
-
-```yaml
-statements:
-  - "VERIFY: The success message is displayed"
-  - "VERIFY: User is redirected to the dashboard"
-```
-
-#### ACTION (with action entity)
-
-Deterministic actions with explicit locators. These replay fast (~1s) without AI.
-
-```yaml
-statements:
-  - description: Click the Submit button
-    action_entity:
-      action_description: Click the Submit button
-      locator: "getByRole('button', { name: 'Submit' })"
-      action_data:
-        action_name: click
-        kwargs: {}
-
-  - description: Type email address
-    action_entity:
-      action_description: Type email address
-      locator: "getByLabel('Email')"
-      action_data:
-        action_name: input_text
-        kwargs:
-          text: "user@example.com"
-
-  - description: Press Enter
-    action_entity:
-      action_data:
-        action_name: press
-        kwargs:
-          keys: Enter
-```
-
-#### STEP (grouping)
-
-Groups related statements under a label.
-
-```yaml
-statements:
-  - STEP: Fill in the registration form
-    statements:
-      - Type "John" in the first name field
-      - Type "Doe" in the last name field
-      - Type "john@example.com" in the email field
-```
+`shiplightConfig()` returns a partial Playwright config that wires in YAML transpilation, auto-discovered `.env` files, and the HTML reporter. Spread it into your own `defineConfig` and override whatever you need.
 
-#### IF / ELSE (conditional)
+For the YAML statement language, authentication, variables, templates, and custom functions: **[docs.shiplight.ai/local](https://docs.shiplight.ai/local/yaml-tests)**.
 
-Conditional branching with AI or JavaScript conditions.
+## Environment variables
 
-```yaml
-statements:
-  # AI condition
-  - IF: A cookie consent banner is visible
-    THEN:
-      - Click the Accept button
-    ELSE:
-      - "VERIFY: No banner is blocking the page"
+Set at least one LLM provider API key in `.env` before running tests:
 
-  # JavaScript condition
-  - IF: "js: page.url().includes('/login')"
-    THEN:
-      - Enter credentials and log in
 ```
-
-#### WHILE loop
-
-Repeating steps with a timeout.
-
-```yaml
-statements:
-  - WHILE: There are more items in the list
-    DO:
-      - Click the next item
-      - "VERIFY: Item details are shown"
-    timeout_ms: 30000
+GOOGLE_API_KEY=...
+# or ANTHROPIC_API_KEY=sk-ant-...
+# or OPENAI_API_KEY=sk-...
 ```
 
-### Supported Actions
+The AI model is auto-selected from the first key set. Override with `WEB_AGENT_MODEL=<model>`.
 
-Actions used in `action_entity.action_data.action_name`:
+Shiplight uses an **explicit env var allowlist**: only the vars it forwards into its internal SDK config are accessible to the agent. Any env var not on the list is invisible to the SDK by design. Full list (`OPENAI_BASE_URL`, Vertex AI routing, Mailgun keys, etc.): **[docs.shiplight.ai/local/cli-reference#environment-variables](https://docs.shiplight.ai/local/cli-reference)**.
 
-| Action | kwargs | Description |
-|---|---|---|
-| `click` | — | Click an element |
-| `double_click` | — | Double-click an element |
-| `right_click` | — | Right-click an element |
-| `hover` | — | Hover over an element |
-| `input_text` | `text` | Type text into an input |
-| `clear_input` | — | Clear an input field |
-| `press` | `keys` | Press a keyboard key (e.g., `Enter`, `Tab`) |
-| `send_keys_on_element` | `keys` | Press a key on a specific element |
-| `select_dropdown_option` | `text` | Select a dropdown option by text |
-| `scroll` | `down`, `num_pages` | Scroll the page |
-| `scroll_to_text` | `text` | Scroll to text on the page |
-| `go_to_url` | `url`, `new_tab` | Navigate to a URL |
-| `go_back` | — | Browser back |
-| `reload_page` | — | Reload the page |
-| `wait` | `seconds` | Wait for a duration |
-| `wait_for_page_ready` | — | Wait for page load |
-| `verify` | `statement` or `code` | Assert a condition (AI or JS) |
-| `js_code` | `code` | Run inline JavaScript |
-| `function` | `functionName`, `parameterNames`, `parameterValues` | Call a function |
-| `switch_tab` | `tab_index` | Switch browser tab |
-| `close_tab` | — | Close current tab |
-| `upload_file` | `file_path` | Upload a file |
-| `save_variable` | `name`, `value` | Save a variable for later use |
+## Requirements
 
-### Locators
+- **Node.js >= 22**
+- **`@playwright/test` 1.58.2** — declared as a peer dependency and installed automatically by `npm install`
+- **Chromium** — install on demand with `npx playwright install chromium`
 
-Action entities can specify element locators in two ways:
-
-```yaml
-# Playwright locator (preferred)
-locator: "getByRole('button', { name: 'Submit' })"
-
-# XPath
-xpath: "//button[@id='submit']"
-```
+## Links
 
-If both are present, `locator` takes priority. If neither is present, the AI agent resolves the element from the action description.
-
-### Frames
-
-For elements inside iframes:
-
-```yaml
-action_entity:
-  frame_path:
-    - "iframe#main"
-  locator: "getByText('Hello')"
-  action_data:
-    action_name: click
-    kwargs: {}
-```
-
-## Extensions
-
-### Custom Test Name
-
-Override the Playwright test name (defaults to `goal`):
-
-```yaml
-name: Login with valid credentials
-goal: Verify login flow works
-url: https://example.com
-statements:
-  - ...
-```
-
-### Tags
-
-Add Playwright tags for filtering with `--grep`:
-
-```yaml
-tags:
-  - smoke
-  - auth
-goal: Login test
-url: https://example.com
-statements:
-  - ...
-```
-
-Run: `npx playwright test --grep @smoke`
-
-### Playwright Fixtures
-
-Pass options to `test.use()`:
-
-```yaml
-use:
-  viewport:
-    width: 375
-    height: 812
-  locale: fr-FR
-goal: Mobile French layout
-url: https://example.com
-statements:
-  - ...
-```
-
-## Variables
-
-Use `{{VAR_NAME}}` syntax in YAML tests to reference variables. Variables are resolved at runtime.
-
-```yaml
-statements:
-  - description: Type username
-    action_entity:
-      locator: "getByLabel('Username')"
-      action_data:
-        action_name: input_text
-        kwargs:
-          text: "{{TEST_USER}}"
-```
-
-### Defining variables in `shiplight.config.json`
-
-Declare variable defaults in your project's `shiplight.config.json`. These are loaded before each test runs.
-
-```json
-{
-  "url": "https://my-app.com",
-  "username": "qa@my-app.com",
-  "password": "test-password",
-  "variables": {
-    "TEST_USER": "standard_user",
-    "TEST_PASS": { "value": "secret_sauce", "sensitive": true }
-  }
-}
-```
-
-Variables can be either:
-- **Plain string** — `"TEST_USER": "standard_user"`
-- **Object with sensitive flag** — `"TEST_PASS": { "value": "secret_sauce", "sensitive": true }` (masked in logs)
-
-
-## Templates
-
-Extract reusable flows into template files and include them with `template:`.
-
-### Template file (`templates/login.yaml`):
-
-```yaml
-params:
-  - username
-  - password
-
-statements:
-  - description: Enter username
-    action_entity:
-      locator: "getByLabel('Username')"
-      action_data:
-        action_name: input_text
-        kwargs:
-          text: "{{username}}"
-  - description: Enter password
-    action_entity:
-      locator: "getByLabel('Password')"
-      action_data:
-        action_name: input_text
-        kwargs:
-          text: "{{password}}"
-  - description: Click login
-    action_entity:
-      locator: "getByRole('button', { name: 'Log in' })"
-      action_data:
-        action_name: click
-        kwargs: {}
-```
-
-### Using the template:
-
-```yaml
-goal: Purchase flow
-url: https://example.com
-
-statements:
-  - template: ../templates/login.yaml
-    params:
-      username: "{{TEST_USER}}"
-      password: "{{TEST_PASS}}"
-  - Navigate to the checkout page
-  - "VERIFY: Order summary is displayed"
-```
-
-Template params (`{{username}}`) are substituted at transpile time. Variables (`{{TEST_USER}}`) pass through and are resolved at runtime.
-
-Templates can be nested (max depth: 5) and circular references are detected.
-
-## Custom Functions
-
-Call TypeScript functions from YAML using the `function` action with `file#export` syntax:
-
-```yaml
-statements:
-  - description: Seed test data
-    action_entity:
-      action_data:
-        action_name: function
-        kwargs:
-          functionName: "../helpers/seed.ts#createTestUser"
-          parameterNames:
-            - page
-            - email
-          parameterValues:
-            - page
-            - "test@example.com"
-```
-
-This generates:
-```ts
-import { createTestUser } from '../helpers/seed';
-// ...
-await createTestUser(page, "test@example.com");
-```
+- **Documentation:** [docs.shiplight.ai](https://docs.shiplight.ai)
+- **VS Code extension:** [github.com/ShiplightAI/vscode-extension](https://github.com/ShiplightAI/vscode-extension)
+- **Cloud platform:** [shiplight.ai](https://www.shiplight.ai)