expect-sdk 0.0.24

package/README.md

@@ -0,0 +1,536 @@
+# expect-sdk
+
+> **Alpha:** This package is currently in alpha. APIs may change between releases.
+
+**Expect** tests your app in a browser so you don't have to.
+
+- Spawns agents simulating real logged-in users to find issues and regressions.
+- No more writing Playwright by hand or token-hungry computer use tools.
+- Get video recordings and GitHub Actions out of the box.
+
+```ts
+import { Expect } from "expect-sdk";
+
+const result = await Expect.test({
+  url: "http://localhost:3000/login",
+  tests: ["valid credentials redirect to the dashboard"],
+});
+```
+
+`expect-sdk` is the TypeScript SDK for [Expect](https://expect.dev). Use it when you need test results in code: CI scripts, test suites, custom tooling. Use [expect-cli](https://npmjs.com/package/expect-cli) for everything else.
+
+## Install
+
+```bash
+npm install expect-sdk
+```
+
+## Set up
+
+Expect works with any [supported coding agent](https://github.com/millionco/expect#supported-agents), including Claude Code, Codex, Gemini CLI, GitHub Copilot, Cursor, OpenCode, and Factory Droid. Install at least one:
+
+```bash
+npm install -g @anthropic-ai/claude-code   # default
+```
+
+Set your API key:
+
+```bash
+export ANTHROPIC_API_KEY=your-api-key
+```
+
+The SDK uses Claude Code by default. See the [full list of supported agents](https://github.com/millionco/expect#supported-agents) for other providers and their setup instructions.
+
+Playwright is an optional peer dependency, only needed for `page`/`browserContext` interoperability and function-based `setup`/`teardown`.
+
+## Run your first test
+
+```ts
+import { Expect } from "expect-sdk";
+
+const result = await Expect.test({
+  url: "http://localhost:3000",
+  tests: ["the homepage loads and shows a welcome heading"],
+});
+
+console.log(result.status); // "passed" or "failed"
+```
+
+The SDK launches a browser, navigates to your URL, and uses AI to verify each test. No test framework is required.
+
+## Capabilities
+
+### Tests as plain English
+
+Tests are written as strings describing expected behavior. The AI agent handles navigation, interaction, and verification on your behalf.
+
+```ts
+await Expect.test({
+  url: "http://localhost:3000/signup",
+  tests: [
+    "submitting empty form shows validation errors on all fields",
+    "mismatched passwords show 'passwords do not match'",
+    "valid submission redirects to the dashboard with welcome message",
+  ],
+});
+```
+
+### Detailed test prompts
+
+For tests that need more context than a single sentence, use the `{ title, prompt }` object form. The `title` is displayed in results, and the `prompt` gives the AI detailed instructions:
+
+```ts
+await Expect.test({
+  url: "http://localhost:3000/dashboard",
+  tests: [
+    "sidebar navigation works",
+    {
+      title: "dashboard data loads correctly",
+      prompt:
+        "verify the user's name appears in the header, the sidebar shows Settings/Projects/Team links, no loading spinners remain after 3 seconds, and there are no console errors",
+    },
+  ],
+});
+```
+
+### Live streaming
+
+`Expect.test()` returns a `TestRun`, which is both a `PromiseLike<TestResult>` and an `AsyncIterable<TestEvent>`. You can await it for the final result, or iterate over it for live progress updates:
+
+```ts
+const run = Expect.test({
+  url: "http://localhost:3000",
+  tests: ["login works", "dashboard loads"],
+});
+
+for await (const event of run) {
+  if (event.type === "step:passed") console.log(`✅ ${event.step.title}`);
+  if (event.type === "step:failed") console.log(`❌ ${event.step.title}`);
+}
+
+const result = await run;
+```
+
+### Playwright interoperability
+
+You can use Playwright for deterministic setup steps and let the AI handle fuzzy verification. Pass a string for AI-driven actions, or a callback that receives the Playwright `Page`:
+
+```ts
+await Expect.test({
+  url: "/admin",
+  cookies: "chrome",
+  setup: async (page) => {
+    await page.fill("#email", "admin@test.com");
+    await page.click("button[type=submit]");
+    await page.waitForURL("**/dashboard");
+  },
+  tests: ["admin panel shows user management table"],
+  teardown: "delete any test data created during this test",
+});
+```
+
+You can also pass an existing Playwright `Page` if you've already set up the browser yourself:
+
+```ts
+import { chromium } from "playwright";
+
+const browser = await chromium.launch();
+const page = await browser.newPage();
+await page.goto("http://localhost:3000/login");
+await page.fill("#email", "test@test.com");
+
+const result = await Expect.test({
+  page,
+  tests: ["login form is pre-filled with test@test.com"],
+});
+```
+
+### Sessions
+
+Sessions create a persistent browser context where cookies and localStorage carry across multiple `.test()` calls. Each call opens a fresh page within the same context.
+
+```ts
+const session = Expect.session({ url: "http://localhost:3000", cookies: "chrome" });
+
+await session.test({ url: "/login", tests: ["login form loads"] });
+await session.test({ url: "/dashboard", tests: ["dashboard loads while authenticated"] });
+await session.test({ url: "/settings", tests: ["settings page is accessible"] });
+
+await session.close();
+```
+
+### Cookie extraction
+
+`Expect.cookies()` extracts cookies from a local browser profile for authenticated testing. It returns a Playwright-compatible `Cookie[]` that you can spread into your test input.
+
+```ts
+const cookies = await Expect.cookies("chrome");
+
+await Expect.test({
+  url: "/dashboard",
+  cookies: [
+    ...cookies,
+    {
+      name: "feature_flag",
+      value: "new_ui",
+      domain: "localhost",
+      path: "/",
+      expires: -1,
+      httpOnly: false,
+      secure: false,
+      sameSite: "Lax",
+    },
+  ],
+  tests: ["dashboard loads with authentication"],
+});
+```
+
+As a shorthand, you can pass a browser name or `true` directly to `cookies` on `Expect.test()` instead of calling `Expect.cookies()` separately:
+
+```ts
+await Expect.test({ url: "/dashboard", cookies: "chrome", tests: ["loads"] });
+await Expect.test({ url: "/dashboard", cookies: true, tests: ["loads"] });
+```
+
+### Custom tools
+
+You can provide custom tools that the AI agent can call during test execution. This is useful for database setup, API calls, or any server-side action the agent needs to perform as part of a test.
+
+```ts
+import { tool } from "expect-sdk";
+
+const createUser = tool(
+  "create_user",
+  "Create a test user in the database",
+  { type: "object", properties: { email: { type: "string" } }, required: ["email"] },
+  async (input) => {
+    const user = await db.users.create({ email: input.email as string });
+    return `Created user ${user.id}`;
+  },
+);
+
+await Expect.test({
+  url: "/admin/users",
+  tools: [createUser],
+  tests: ["new user appears in the users table"],
+});
+```
+
+### Configuration
+
+`configure()` sets global defaults that apply to all subsequent `Expect.test()` and `Expect.session()` calls, so you don't need to repeat common options:
+
+```ts
+import { Expect, configure } from "expect-sdk";
+
+configure({ baseUrl: "http://localhost:3000", mode: "headless" });
+
+await Expect.test({ url: "/login", tests: ["login page loads"] });
+await Expect.test({ url: "/signup", tests: ["signup page loads"] });
+```
+
+### Config file
+
+For project-wide defaults, create an `expect.config.ts` file in your project root. The SDK automatically loads this file when running tests, so you don't need to call `configure()` in every test script.
+
+```ts
+// expect.config.ts
+import { defineConfig } from "expect-sdk";
+
+export default defineConfig({
+  baseUrl: "http://localhost:3000",
+  browser: "chrome",
+  mode: "headless",
+  cookies: "chrome",
+  timeout: 120_000,
+});
+```
+
+With this file in place, all `Expect.test()` calls resolve relative URLs against `baseUrl` and use Chrome cookies automatically:
+
+```ts
+// No need to specify baseUrl or cookies in each call
+await Expect.test({ url: "/login", tests: ["login page loads"] });
+await Expect.test({ url: "/signup", tests: ["signup page loads"] });
+```
+
+Inline `configure()` calls take precedence over config file values, so you can override specific options per-script when needed.
+
+### Error handling
+
+The SDK throws `ExpectConfigError` for invalid input (missing URL, empty tests array) and `ExpectTimeoutError` when a test exceeds its timeout. Both are exported from the package:
+
+```ts
+import { Expect, ExpectConfigError } from "expect-sdk";
+
+try {
+  const result = await Expect.test({
+    url: "http://localhost:3000",
+    tests: ["login works"],
+    timeout: 60_000,
+  });
+  if (result.status === "failed") {
+    console.error("Test failures:");
+    for (const error of result.errors) {
+      console.error(`  ${error.title}: ${error.summary}`);
+    }
+    process.exit(1);
+  }
+} catch (error) {
+  if (error instanceof ExpectConfigError) {
+    console.error("Configuration error:", error.message);
+  } else {
+    console.error("Unexpected error:", error);
+  }
+  process.exit(1);
+}
+```
+
+### Parallel execution
+
+Tests run sequentially by default. To run multiple tests in parallel, use `Promise.all` or `Promise.allSettled`:
+
+```ts
+const [login, signup, dashboard] = await Promise.all([
+  Expect.test({ url: "/login", tests: ["login page loads"] }),
+  Expect.test({ url: "/signup", tests: ["signup page loads"] }),
+  Expect.test({ url: "/dashboard", tests: ["dashboard loads"] }),
+]);
+```
+
+Each test launches its own browser instance, so they don't interfere with each other.
+
+### CI/CD
+
+The SDK works in any CI environment that has Node.js and a supported agent installed. Here's an example GitHub Actions workflow:
+
+```yaml
+# .github/workflows/expect.yml
+name: Browser Tests
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+      - run: npm install -g @anthropic-ai/claude-code
+      - run: npm start & npx wait-on http://localhost:3000
+      - run: npx tsx tests/browser.ts
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+Where `tests/browser.ts` is a script that uses the SDK:
+
+```ts
+import { Expect, configure } from "expect-sdk";
+
+configure({ baseUrl: "http://localhost:3000" });
+
+const result = await Expect.test({
+  url: "/login",
+  tests: ["login form validates required fields", "valid credentials redirect to the dashboard"],
+});
+
+if (result.status === "failed") {
+  console.error(result.errors.map((e) => e.summary).join("\n"));
+  process.exit(1);
+}
+```
+
+## Compare to other testing tools
+
+|                | expect-sdk                      | Playwright / Cypress     | Manual QA      |
+| -------------- | ------------------------------- | ------------------------ | -------------- |
+| Test authoring | Plain English                   | Selectors + assertions   | Click around   |
+| Maintenance    | Tests don't break on UI changes | Brittle selectors        | N/A            |
+| Setup          | `npm install expect-sdk`        | Page objects, fixtures   | Hire people    |
+| Flakiness      | AI retries intelligently        | Explicit waits           | Human judgment |
+| Best for       | Behavior verification           | Pixel-precise assertions | Exploratory    |
+
+Use expect-sdk when you care about **what** the app does. Use Playwright when you care about **how** it renders.
+
+---
+
+## API reference
+
+### `Expect.test(input): TestRun`
+
+Runs a one-shot test. The SDK launches a browser, navigates to the URL, executes all tests, and closes the browser when finished.
+
+```ts
+const result = await Expect.test({
+  url: "http://localhost:3000/login",
+  cookies: "chrome",
+  tests: ["login form validates email", "valid credentials redirect to dashboard"],
+  setup: "navigate to the login page",
+  timeout: 60_000,
+});
+```
+
+| Field         | Type          | Default      | Description                                                      |
+| ------------- | ------------- | ------------ | ---------------------------------------------------------------- | ------------------ |
+| `url`         | `string`      | -            | URL to navigate to (absolute or relative to `baseUrl`)           |
+| `page`        | `Page`        | -            | Existing Playwright page (skips browser creation, `url` ignored) |
+| `cookies`     | `CookieInput` | -            | `true`, browser name, browser name array, or `Cookie[]`          |
+| `tools`       | `Tool[]`      | -            | Custom tools the AI can call                                     |
+| `tests`       | `Test[]`      | **required** | Test descriptions as strings or `{ title?, prompt }` objects     |
+| `setup`       | `Action`      | -            | String instruction or `(page: Page) => Promise<void              | string>`           |
+| `teardown`    | `Action`      | -            | String instruction or `(page: Page) => Promise<void              | string>`           |
+| `mode`        | `"headed"     | "headless"`  | `"headless"`                                                     | Browser visibility |
+| `timeout`     | `number`      | `300000`     | Timeout in milliseconds                                          |
+| `isRecording` | `boolean`     | `false`      | Enable session recording                                         |
+
+### `TestRun`
+
+The object returned by `Expect.test()`. It implements both `PromiseLike<TestResult>` and `AsyncIterable<TestEvent>`, so you can either await the final result or iterate over events as they stream in.
+
+```ts
+interface TestRun extends PromiseLike<TestResult> {
+  [Symbol.asyncIterator](): AsyncIterator<TestEvent>;
+}
+```
+
+### `TestResult`
+
+The final result returned when a `TestRun` resolves.
+
+```ts
+interface TestResult {
+  status: "pending" | "passed" | "failed";
+  url: string;
+  duration: number;
+  recordingPath?: string;
+  steps: StepResult[];
+  errors: StepResult[]; // steps.filter(s => s.status === "failed")
+}
+```
+
+### `StepResult`
+
+Represents the outcome of a single test step within a run.
+
+```ts
+interface StepResult {
+  title: string;
+  status: "pending" | "passed" | "failed";
+  summary: string;
+  screenshotPath?: string;
+  duration: number;
+}
+```
+
+### `TestEvent`
+
+Events emitted during test execution. Use `for await (const event of run)` to consume them:
+
+```ts
+type TestEvent =
+  | { type: "run:started"; title: string; baseUrl?: string }
+  | { type: "step:started"; title: string }
+  | { type: "step:passed"; step: StepResult }
+  | { type: "step:failed"; step: StepResult }
+  | { type: "step:skipped"; title: string; reason: string }
+  | { type: "screenshot"; title: string; path: string }
+  | { type: "completed"; result: TestResult };
+```
+
+### `Expect.session(config): ExpectSession`
+
+Creates a persistent browser context where cookies and localStorage are preserved across multiple `.test()` calls.
+
+```ts
+const session = Expect.session({
+  url: "http://localhost:3000",
+  cookies: "chrome",
+  mode: "headless",
+});
+
+await session.test({ url: "/login", tests: ["login works"] });
+await session.test({ url: "/dashboard", tests: ["loads while authenticated"] });
+await session.close();
+```
+
+| Field     | Type          | Default     | Description                       |
+| --------- | ------------- | ----------- | --------------------------------- | ------------------ |
+| `url`     | `string`      | -           | Base URL for the session          |
+| `cookies` | `CookieInput` | -           | Cookies to inject                 |
+| `mode`    | `"headed"     | "headless"` | `"headless"`                      | Browser visibility |
+| `timeout` | `number`      | `300000`    | Default timeout for session tests |
+
+### `ExpectSession`
+
+```ts
+interface ExpectSession {
+  test(input: SessionTestInput): TestRun;
+  close(): Promise<void>;
+  [Symbol.asyncDispose](): Promise<void>;
+}
+```
+
+### `Expect.cookies(browser): Promise<Cookie[]>`
+
+Extracts cookies from a local browser profile and returns them as a Playwright-compatible `Cookie[]`.
+
+```ts
+const cookies = await Expect.cookies("chrome");
+const multi = await Expect.cookies(["chrome", "firefox"]);
+const autoDetect = await Expect.cookies(true);
+```
+
+### `tool(name, description, schema, handler): Tool`
+
+Creates a custom tool that the AI agent can invoke during test execution. The schema follows the MCP tool input schema format.
+
+```ts
+import { tool } from "expect-sdk";
+
+const createUser = tool(
+  "create_user",
+  "Create a test user",
+  { type: "object", properties: { email: { type: "string" } }, required: ["email"] },
+  async (input) => `Created user ${input.email}`,
+);
+```
+
+### `configure(config): void`
+
+Sets global configuration defaults. Each call shallow-merges into the existing config, so you can call it multiple times to build up options incrementally.
+
+```ts
+import { configure } from "expect-sdk";
+configure({ baseUrl: "http://localhost:3000", mode: "headless", timeout: 60_000 });
+```
+
+### `defineConfig(config): ExpectConfig`
+
+An identity function that provides type inference when writing `expect.config.ts` files. It returns the config object unchanged.
+
+```ts
+import { defineConfig } from "expect-sdk";
+export default defineConfig({ baseUrl: "http://localhost:3000", cookies: "chrome" });
+```
+
+### `ExpectConfig`
+
+```ts
+interface ExpectConfig {
+  baseUrl?: string;
+  browser?: BrowserName;
+  mode?: "headed" | "headless";
+  cookies?: CookieInput;
+  timeout?: number;
+  model?: string;
+  apiKey?: string;
+  rootDir?: string;
+}
+```
+
+## License
+
+FSL-1.1-MIT © [Million Software, Inc.](https://million.dev)

package/dist/constants-DFJAD4-F.mjs

@@ -0,0 +1,152 @@
+import { DateTime, Layer, Option, References, Schema } from "effect";
+import { Executor, Git } from "@expect/supervisor";
+import { Agent } from "@expect/agent";
+//#region src/errors.ts
+var ExpectTimeoutError = class extends Schema.ErrorClass("ExpectTimeoutError")({
+	_tag: Schema.tag("ExpectTimeoutError"),
+	timeoutMs: Schema.Number
+}) {
+	message = `expect execution timed out after ${this.timeoutMs}ms`;
+};
+var ExpectConfigError = class extends Error {
+	constructor(message, fix) {
+		super(`${message}\n\nFix: ${fix}`);
+		this.name = "ExpectConfigError";
+	}
+};
+//#endregion
+//#region src/build-instruction.ts
+const resolveUrl = (url, baseUrl) => {
+	if (typeof url !== "string") throw new ExpectConfigError(`Expected a URL string, got ${typeof url}.`, `Expect.test({ url: "http://localhost:3000/login", tests: [...] })`);
+	try {
+		return new URL(url).href;
+	} catch {}
+	if (!baseUrl) throw new ExpectConfigError(`No baseUrl configured and URL "${url}" is relative.`, `configure({ baseUrl: "http://localhost:3000" })\nOr use a full URL: Expect.test({ url: "http://localhost:3000${url}", tests: [...] })`);
+	return new URL(url, baseUrl).href;
+};
+const buildInstruction = (url, tests) => {
+	const lines = [`Navigate to ${url} and verify the following requirements:`, ""];
+	for (const [index, title] of tests.entries()) lines.push(`${index + 1}. ${title}`);
+	return lines.join("\n");
+};
+//#endregion
+//#region src/layers.ts
+const layerSdk = (agentBackend, rootDir) => {
+	const gitLayer = Git.withRepoRoot(rootDir);
+	const agentLayer = Agent.layerFor(agentBackend);
+	const executorLayer = Executor.layer.pipe(Layer.provide(gitLayer));
+	return Layer.mergeAll(executorLayer, gitLayer).pipe(Layer.provideMerge(agentLayer), Layer.provideMerge(Layer.succeed(References.MinimumLogLevel, "Error")));
+};
+//#endregion
+//#region src/result-builder.ts
+const REPLAY_SESSION_PREFIX = "rrweb replay:";
+const SCREENSHOT_PREFIX = "Screenshot:";
+const stepDurationMs = (step) => {
+	if (Option.isNone(step.startedAt)) return 0;
+	if (Option.isNone(step.endedAt)) return Date.now() - Number(DateTime.toEpochMillis(step.startedAt.value));
+	return Number(DateTime.toEpochMillis(step.endedAt.value)) - Number(DateTime.toEpochMillis(step.startedAt.value));
+};
+const stepStatusToResultStatus = (status) => {
+	if (status === "passed") return "passed";
+	if (status === "failed" || status === "skipped") return "failed";
+	return "pending";
+};
+const extractArtifacts = (events) => {
+	const closeResult = events.slice().reverse().find((event) => event._tag === "ToolResult" && event.toolName === "close" && !event.isError && event.result.length > 0);
+	if (!closeResult || closeResult._tag !== "ToolResult") return {
+		recordingPath: void 0,
+		screenshotPaths: []
+	};
+	const lines = closeResult.result.split("\n").map((line) => line.trim()).filter((line) => line.length > 0);
+	return {
+		recordingPath: lines.find((line) => line.startsWith(REPLAY_SESSION_PREFIX))?.replace(REPLAY_SESSION_PREFIX, "").trim() || void 0,
+		screenshotPaths: lines.filter((line) => line.startsWith(SCREENSHOT_PREFIX)).map((line) => line.replace(SCREENSHOT_PREFIX, "").trim()).filter((value) => value.length > 0)
+	};
+};
+const buildStepResult = (step, screenshotPaths, stepIndex) => ({
+	title: step.title,
+	status: stepStatusToResultStatus(step.status),
+	summary: Option.getOrElse(step.summary, () => ""),
+	screenshotPath: screenshotPaths[stepIndex],
+	duration: stepDurationMs(step)
+});
+const buildTestResult = (executed, url, startedAt, artifacts) => {
+	const steps = executed.steps.map((step, index) => buildStepResult(step, artifacts.screenshotPaths, index));
+	const errors = steps.filter((step) => step.status === "failed");
+	const hasFailure = errors.length > 0;
+	const hasPending = steps.some((step) => step.status === "pending");
+	let status = "passed";
+	if (hasFailure) status = "failed";
+	else if (hasPending || steps.length === 0) status = "pending";
+	return {
+		status,
+		url,
+		duration: Date.now() - startedAt,
+		recordingPath: artifacts.recordingPath,
+		steps,
+		errors
+	};
+};
+const mapExecutionEvent = (event, context) => {
+	switch (event._tag) {
+		case "RunStarted": return {
+			type: "run:started",
+			title: event.plan.title,
+			baseUrl: Option.getOrUndefined(event.plan.baseUrl)
+		};
+		case "StepStarted": return {
+			type: "step:started",
+			title: event.title
+		};
+		case "StepCompleted": {
+			const step = context.stepMap.get(event.stepId);
+			const index = context.stepIndexMap.get(event.stepId) ?? -1;
+			return step ? {
+				type: "step:passed",
+				step: buildStepResult(step, context.artifacts.screenshotPaths, index)
+			} : void 0;
+		}
+		case "StepFailed": {
+			const step = context.stepMap.get(event.stepId);
+			const index = context.stepIndexMap.get(event.stepId) ?? -1;
+			return step ? {
+				type: "step:failed",
+				step: buildStepResult(step, context.artifacts.screenshotPaths, index)
+			} : void 0;
+		}
+		case "StepSkipped": return {
+			type: "step:skipped",
+			title: context.stepMap.get(event.stepId)?.title ?? event.stepId,
+			reason: event.reason
+		};
+		case "ToolResult": return event.toolName.endsWith("__screenshot") && !event.isError ? {
+			type: "screenshot",
+			title: event.toolName,
+			path: event.result
+		} : void 0;
+		case "RunFinished": return {
+			type: "completed",
+			result: buildTestResult(context.executed, context.url, context.startedAt, context.artifacts)
+		};
+		default: return;
+	}
+};
+const diffEvents = (previous, current, executed, url, startedAt) => {
+	const context = {
+		stepMap: new Map(executed.steps.map((step) => [step.id, step])),
+		stepIndexMap: new Map(executed.steps.map((step, index) => [step.id, index])),
+		artifacts: extractArtifacts(executed.events),
+		executed,
+		url,
+		startedAt
+	};
+	return current.slice(previous.length).map((event) => mapExecutionEvent(event, context)).filter((event) => event !== void 0);
+};
+//#endregion
+//#region src/constants.ts
+const DEFAULT_TIMEOUT_MS = 300 * 1e3;
+const DEFAULT_AGENT_BACKEND = "claude";
+//#endregion
+export { diffEvents as a, buildInstruction as c, ExpectTimeoutError as d, buildTestResult as i, resolveUrl as l, DEFAULT_TIMEOUT_MS as n, extractArtifacts as o, buildStepResult as r, layerSdk as s, DEFAULT_AGENT_BACKEND as t, ExpectConfigError as u };
+
+//# sourceMappingURL=constants-DFJAD4-F.mjs.map