superghost 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Luis Morales
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,172 @@
+# SuperGhost
+
+Plain English test cases with AI execution and instant cached replay for CI/CD.
+
+Write tests in YAML. An AI agent executes them in a real browser or via API calls. Results are cached step-by-step so re-runs are instant and deterministic -- no flaky tests, no test code to maintain.
+
+![SuperGhost Demo](demo/demo.gif)
+
+## Install
+
+### Zero-install (recommended)
+
+```bash
+bunx superghost --config tests.yaml
+```
+
+### Global install
+
+```bash
+bun install -g superghost
+superghost --config tests.yaml
+```
+
+### Standalone binary
+
+Download the latest binary for your platform from [GitHub Releases](https://github.com/lacion/superghost/releases).
+
+```bash
+chmod +x superghost-darwin-arm64
+./superghost-darwin-arm64 --config tests.yaml
+```
+
+On first run, the standalone binary automatically installs MCP server dependencies to `~/.superghost/`.
+
+## Quick Start
+
+Create a `tests.yaml` file:
+
+```yaml
+baseUrl: https://example.com
+model: claude-sonnet-4-20250514
+
+tests:
+  - name: Homepage loads
+    case: Navigate to the homepage and verify the page title contains "Example"
+
+  - name: API health check
+    case: Send a GET request to /api/health and verify the response status is 200
+```
+
+Run it:
+
+```bash
+bunx superghost --config tests.yaml
+```
+
+## CLI
+
+```
+Usage: superghost [options]
+
+Options:
+  -c, --config <path>  Path to YAML config file (required)
+  -V, --version        Output the version number
+  -h, --help           Display help
+```
+
+## Provider Setup
+
+SuperGhost supports four AI providers. Set the appropriate environment variable for your chosen provider.
+
+### Anthropic (default)
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+```yaml
+model: claude-sonnet-4-20250514
+```
+
+### OpenAI
+
+```bash
+export OPENAI_API_KEY=sk-...
+```
+
+```yaml
+model: gpt-4o
+modelProvider: openai
+```
+
+### Google Gemini
+
+```bash
+export GOOGLE_GENERATIVE_AI_API_KEY=...
+```
+
+```yaml
+model: gemini-2.5-flash
+modelProvider: gemini
+```
+
+### OpenRouter
+
+```bash
+export OPENROUTER_API_KEY=sk-or-...
+```
+
+```yaml
+model: anthropic/claude-sonnet-4-20250514
+modelProvider: openrouter
+```
+
+## Configuration
+
+All fields in `tests.yaml`:
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `baseUrl` | `string` | (required) | Base URL for all tests |
+| `model` | `string` | (required) | AI model identifier |
+| `modelProvider` | `string` | `"anthropic"` | Provider: `anthropic`, `openai`, `gemini`, `openrouter` |
+| `browser` | `string` | `"chromium"` | Browser engine: `chromium`, `firefox`, `webkit` |
+| `headless` | `boolean` | `false` | Run browser in headless mode |
+| `cacheDir` | `string` | `".superghost-cache"` | Directory for cached test steps |
+| `context` | `string` | `undefined` | Global context passed to every test |
+| `tests` | `array` | (required) | Array of test definitions |
+| `tests[].name` | `string` | `undefined` | Display name for the test |
+| `tests[].case` | `string` | (required) | Plain English test instruction |
+| `tests[].context` | `string` | `undefined` | Per-test context for the AI agent |
+
+## How It Works
+
+1. **First run:** The AI agent reads your plain English test case and executes it step-by-step in a real browser (via Playwright MCP) or via API calls (via curl MCP). Each step is recorded to a cache file.
+
+2. **Subsequent runs:** Cached steps are replayed directly against the browser/API without calling the AI. This makes re-runs instant and deterministic.
+
+3. **Self-healing:** If a cached step fails during replay (e.g., a selector changed), SuperGhost automatically falls back to the AI agent to re-execute that test. The new steps replace the stale cache.
+
+## Example App (E2E)
+
+The `e2e/` directory contains a fullstack Task Manager app that validates SuperGhost end-to-end and serves as a reference for writing test configs.
+
+```bash
+# Start the example app
+bun run e2e:app
+# Open http://localhost:3777
+
+# Run smoke tests (2 tests — requires an AI API key)
+bun run e2e:smoke
+
+# Run browser UI tests (7 tests)
+bun run e2e:browser
+
+# Run API endpoint tests (7 tests)
+bun run e2e:api
+
+# Run all 16 tests
+bun run e2e:all
+```
+
+The test runner exits gracefully when no API key is configured, making it safe for CI environments. See [`e2e/README.md`](e2e/README.md) for details.
+
+## Standalone Binary
+
+When running as a standalone compiled binary (downloaded from GitHub Releases), SuperGhost cannot use `bunx` to spawn MCP server packages. Instead:
+
+- On first run, MCP dependencies (`@playwright/mcp`, `@calibress/curl-mcp`) are automatically installed to `~/.superghost/`
+- Subsequent runs skip the install step
+- You must have a Playwright-compatible browser installed on your system (Chromium, Firefox, or WebKit)
+- SuperGhost does **not** auto-install browser binaries -- if Playwright cannot find a browser, it will display its own error message with install instructions

package/package.json

@@ -0,0 +1,77 @@
+{
+  "name": "superghost",
+  "version": "0.1.0",
+  "description": "Plain English test cases with AI execution and instant cached replay for CI/CD",
+  "type": "module",
+  "bin": {
+    "superghost": "src/cli.ts"
+  },
+  "files": [
+    "src/",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "test": "bun test",
+    "typecheck": "bunx tsc --noEmit",
+    "build:binary": "bun run scripts/build-binaries.ts",
+    "prepublishOnly": "bun test && bunx tsc --noEmit",
+    "e2e": "bun run e2e/run-e2e.ts",
+    "e2e:smoke": "bun run e2e/run-e2e.ts smoke",
+    "e2e:browser": "bun run e2e/run-e2e.ts browser",
+    "e2e:api": "bun run e2e/run-e2e.ts api",
+    "e2e:all": "bun run e2e/run-e2e.ts all",
+    "e2e:headed": "bun run e2e/run-e2e.ts smoke --headed",
+    "e2e:smoke:headed": "bun run e2e/run-e2e.ts smoke --headed",
+    "e2e:browser:headed": "bun run e2e/run-e2e.ts browser --headed",
+    "e2e:api:headed": "bun run e2e/run-e2e.ts api --headed",
+    "e2e:all:headed": "bun run e2e/run-e2e.ts all --headed",
+    "e2e:app": "bun run e2e/app/server.ts"
+  },
+  "keywords": [
+    "testing",
+    "ai",
+    "browser",
+    "e2e",
+    "playwright",
+    "mcp",
+    "cli"
+  ],
+  "license": "MIT",
+  "author": "Luis Morales (https://github.com/lacion)",
+  "engines": {
+    "bun": ">=1.2.0"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/lacion/superghost.git"
+  },
+  "homepage": "https://github.com/lacion/superghost#readme",
+  "bugs": {
+    "url": "https://github.com/lacion/superghost/issues"
+  },
+  "dependencies": {
+    "@ai-sdk/anthropic": "^3.0.58",
+    "@ai-sdk/google": "^3.0.37",
+    "@ai-sdk/mcp": "^1.0.25",
+    "@ai-sdk/openai": "^3.0.41",
+    "@modelcontextprotocol/sdk": "^1.27.1",
+    "@openrouter/ai-sdk-provider": "^2.2.5",
+    "ai": "^6.0.116",
+    "commander": "^14.0.3",
+    "nanospinner": "^1.2.2",
+    "picocolors": "^1.1.1",
+    "zod": "^4.3.6"
+  },
+  "devDependencies": {
+    "@types/bun": "^1.3.10",
+    "@types/react": "^19.0.0",
+    "@types/react-dom": "^19.0.0",
+    "react": "^19.0.0",
+    "react-dom": "^19.0.0",
+    "typescript": "^5.9.3"
+  }
+}

package/src/agent/agent-runner.ts

@@ -0,0 +1,69 @@
+import { generateText, Output, stepCountIs } from "ai";
+import { z } from "zod";
+import { StepRecorder } from "../cache/step-recorder.ts";
+import type { AgentExecutionResult } from "./types.ts";
+import { buildSystemPrompt } from "./prompt.ts";
+
+/**
+ * Schema for structured agent output.
+ * The agent must produce a { passed, message } JSON object.
+ */
+const TestResultSchema = z.object({
+  passed: z.boolean().describe("Whether the test case passed"),
+  message: z
+    .string()
+    .describe("Brief diagnostic: what happened and what the page showed"),
+});
+
+/**
+ * Execute a single test case using the AI agent with MCP tools.
+ *
+ * Uses Vercel AI SDK's generateText with:
+ * - Output.object() for structured { passed, message } responses
+ * - stopWhen: stepCountIs(recursionLimit) for loop control
+ * - StepRecorder tool wrapping for cache step capture
+ *
+ * @returns AgentExecutionResult with pass/fail status, diagnostic message, and recorded steps
+ */
+export async function executeAgent(config: {
+  model: any;
+  tools: Record<string, any>;
+  testCase: string;
+  baseUrl: string;
+  recursionLimit: number;
+  globalContext?: string;
+  testContext?: string;
+}): Promise<AgentExecutionResult> {
+  const recorder = new StepRecorder();
+  const wrappedTools = recorder.wrapTools(config.tools);
+
+  const systemPrompt = buildSystemPrompt(
+    config.testCase,
+    config.baseUrl,
+    config.globalContext,
+    config.testContext,
+  );
+
+  const { output } = await generateText({
+    model: config.model,
+    tools: wrappedTools,
+    system: systemPrompt,
+    prompt: `Execute the test case: "${config.testCase}"`,
+    stopWhen: stepCountIs(config.recursionLimit),
+    output: Output.object({ schema: TestResultSchema }),
+  });
+
+  if (output === null) {
+    return {
+      passed: false,
+      message: `Agent did not produce a structured result — it may have exceeded the ${config.recursionLimit} step limit`,
+      steps: recorder.getSteps(),
+    };
+  }
+
+  return {
+    passed: output.passed,
+    message: output.message,
+    steps: recorder.getSteps(),
+  };
+}

package/src/agent/mcp-manager.ts

@@ -0,0 +1,78 @@
+import { createMCPClient } from "@ai-sdk/mcp";
+import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
+import type { Config } from "../config/types.ts";
+import { getMcpCommand } from "../dist/paths.ts";
+
+/**
+ * Manages the lifecycle of Playwright and curl MCP servers.
+ *
+ * MCP servers are shared across the test suite (not restarted per test).
+ * Fresh browser context per test comes from the `--isolated` flag on
+ * Playwright MCP. Both tool sets are merged and provided to the agent
+ * regardless of test type.
+ */
+export class McpManager {
+  private playwrightClient: Awaited<ReturnType<typeof createMCPClient>> | null =
+    null;
+  private curlClient: Awaited<ReturnType<typeof createMCPClient>> | null = null;
+
+  constructor(private readonly config: Pick<Config, "browser" | "headless">) {}
+
+  /**
+   * Spawn Playwright MCP and curl MCP servers via stdio transport.
+   * Must be called before getTools().
+   */
+  async initialize(): Promise<void> {
+    // Resolve MCP spawn commands (bunx in npm mode, path-based in standalone)
+    const playwrightCmd = getMcpCommand("@playwright/mcp");
+    const curlCmd = getMcpCommand("@calibress/curl-mcp");
+
+    const playwrightArgs = [
+      ...playwrightCmd.args,
+      "--isolated",
+      `--browser=${this.config.browser}`,
+    ];
+
+    if (this.config.headless) {
+      playwrightArgs.splice(playwrightCmd.args.length, 0, "--headless");
+    }
+
+    this.playwrightClient = await createMCPClient({
+      transport: new StdioClientTransport({
+        command: playwrightCmd.command,
+        args: playwrightArgs,
+      }),
+    });
+
+    this.curlClient = await createMCPClient({
+      transport: new StdioClientTransport({
+        command: curlCmd.command,
+        args: [...curlCmd.args],
+      }),
+    });
+  }
+
+  /**
+   * Get merged tool set from both Playwright and curl MCP servers.
+   * Provides ALL tools to the agent regardless of test type.
+   */
+  async getTools(): Promise<Record<string, any>> {
+    const playwrightTools = await this.playwrightClient!.tools();
+    const curlTools = await this.curlClient!.tools();
+    return { ...playwrightTools, ...curlTools };
+  }
+
+  /**
+   * Close both MCP server connections.
+   * Uses Promise.allSettled to ensure both servers are cleaned up
+   * even if one fails to close.
+   */
+  async close(): Promise<void> {
+    await Promise.allSettled([
+      this.playwrightClient?.close(),
+      this.curlClient?.close(),
+    ]);
+    this.playwrightClient = null;
+    this.curlClient = null;
+  }
+}

package/src/agent/model-factory.ts

@@ -0,0 +1,71 @@
+import { anthropic } from "@ai-sdk/anthropic";
+import { openai } from "@ai-sdk/openai";
+import { google } from "@ai-sdk/google";
+import { createOpenRouter } from "@openrouter/ai-sdk-provider";
+
+/** Supported LLM provider names */
+export type ProviderName = "anthropic" | "openai" | "google" | "openrouter";
+
+/** Environment variable names for each provider's API key */
+export const ENV_VARS: Record<ProviderName, string> = {
+  anthropic: "ANTHROPIC_API_KEY",
+  openai: "OPENAI_API_KEY",
+  google: "GOOGLE_GENERATIVE_AI_API_KEY",
+  openrouter: "OPENROUTER_API_KEY",
+};
+
+// Auto-inference rules: model name prefix -> provider
+const MODEL_PREFIX_MAP: Array<[RegExp, ProviderName]> = [
+  [/^claude-/, "anthropic"],
+  [/^gpt-/, "openai"],
+  [/^o\d/, "openai"],
+  [/^gemini-/, "google"],
+  [/\//, "openrouter"],
+];
+
+/**
+ * Infer the LLM provider from a model name string.
+ * Falls back to "anthropic" if no pattern matches.
+ */
+export function inferProvider(modelName: string): ProviderName {
+  for (const [pattern, provider] of MODEL_PREFIX_MAP) {
+    if (pattern.test(modelName)) return provider;
+  }
+  return "anthropic";
+}
+
+/**
+ * Validate that the API key environment variable is set for the given provider.
+ * Throws a descriptive error if the key is missing.
+ */
+export function validateApiKey(provider: ProviderName): void {
+  const envVar = ENV_VARS[provider];
+  if (!Bun.env[envVar]) {
+    throw new Error(
+      `Missing API key for ${provider}.\n` +
+        `  Set the ${envVar} environment variable:\n` +
+        `    export ${envVar}=your-key-here\n` +
+        `  Or add it to your .env file.`,
+    );
+  }
+}
+
+/**
+ * Create an AI SDK model instance for the given model name and provider.
+ */
+export function createModel(modelName: string, providerName: ProviderName) {
+  switch (providerName) {
+    case "anthropic":
+      return anthropic(modelName);
+    case "openai":
+      return openai(modelName);
+    case "google":
+      return google(modelName);
+    case "openrouter": {
+      const openrouter = createOpenRouter({
+        apiKey: Bun.env.OPENROUTER_API_KEY!,
+      });
+      return openrouter.chat(modelName);
+    }
+  }
+}

package/src/agent/prompt.ts

@@ -0,0 +1,47 @@
+/**
+ * Build the system prompt for the QA automation agent.
+ *
+ * Includes test case, base URL, tool usage instructions, and optional
+ * global/per-test context fields.
+ */
+export function buildSystemPrompt(
+  testCase: string,
+  baseUrl: string,
+  globalContext?: string,
+  testContext?: string,
+): string {
+  const lines: string[] = [
+    "You are a QA automation agent. Execute the following test case and determine if it passes or fails.",
+    "",
+    `Test case: "${testCase}"`,
+    `Base URL: "${baseUrl}"`,
+    "",
+    "You have access to both browser automation tools and HTTP/curl tools.",
+    "Choose the appropriate tools based on the test case.",
+    "",
+    "For browser/UI tests:",
+    "- Navigate to the base URL first",
+    "- Use browser_snapshot to understand page state before acting",
+    "- Use browser_click, browser_type for interactions",
+    "",
+    "For API tests:",
+    "- Use the curl_request tool to make HTTP requests",
+    "- Check status codes, headers, and response body",
+    "",
+    "Instructions:",
+    "1. Analyze the test case and decide which tools to use.",
+    "2. Execute the actions needed to verify the test case.",
+    "3. Be methodical. If something doesn't work, try alternative approaches before declaring failure.",
+    "4. When finished, provide your verdict as structured output with passed (boolean) and message (brief diagnostic).",
+  ];
+
+  if (globalContext) {
+    lines.push("", "Additional context from the user:", globalContext);
+  }
+
+  if (testContext) {
+    lines.push("", "Test-specific context:", testContext);
+  }
+
+  return lines.join("\n");
+}

package/src/agent/types.ts

@@ -0,0 +1,28 @@
+import type { CachedStep } from "../cache/types.ts";
+import type { ProviderName } from "./model-factory.ts";
+
+/** Result of a single AI agent execution */
+export interface AgentExecutionResult {
+  /** Whether the test case passed */
+  passed: boolean;
+  /** Diagnostic message describing the outcome */
+  message: string;
+  /** Recorded tool call steps for caching */
+  steps: CachedStep[];
+}
+
+/** Configuration for a single agent run */
+export interface AgentConfig {
+  /** Model identifier (e.g., "claude-sonnet-4-6", "gpt-4o") */
+  model: string;
+  /** LLM provider */
+  provider: ProviderName;
+  /** Maximum number of agent steps */
+  recursionLimit: number;
+  /** Plain English test case description */
+  testCase: string;
+  /** Base URL for the application under test */
+  baseUrl: string;
+  /** Optional per-test context appended to system prompt */
+  context?: string;
+}

package/src/cache/cache-manager.ts

@@ -0,0 +1,105 @@
+import { join } from "node:path";
+import { mkdir, rename } from "node:fs/promises";
+import type { CacheEntry, CachedStep } from "./types.ts";
+
+/**
+ * Manages file-based cache entries for test step recordings.
+ * Each entry is a JSON file keyed by a deterministic SHA-256 hash of (testCase + baseUrl).
+ * Uses atomic write-then-rename to prevent corrupted cache files.
+ */
+export class CacheManager {
+  private readonly cacheDir: string;
+
+  constructor(cacheDir: string) {
+    this.cacheDir = cacheDir;
+  }
+
+  /**
+   * Generate a deterministic 16-char hex hash key.
+   * Uses Bun-native CryptoHasher for SHA-256 hashing.
+   */
+  static hashKey(testCase: string, baseUrl: string): string {
+    const input = `${testCase}|${baseUrl}`;
+    const hasher = new Bun.CryptoHasher("sha256");
+    hasher.update(input);
+    return hasher.digest("hex").slice(0, 16);
+  }
+
+  /**
+   * Save a cache entry for the given test case.
+   * Creates the cache directory if it does not exist.
+   * Uses atomic write (tmp file + rename) to prevent corruption.
+   * Preserves createdAt from existing entry when updating.
+   */
+  async save(
+    testCase: string,
+    baseUrl: string,
+    steps: CachedStep[],
+    diagnostics: {
+      model: string;
+      provider: string;
+      stepCount: number;
+      aiMessage: string;
+      durationMs: number;
+    },
+  ): Promise<void> {
+    await mkdir(this.cacheDir, { recursive: true });
+
+    const hash = CacheManager.hashKey(testCase, baseUrl);
+    const now = new Date().toISOString();
+
+    // Load existing entry to preserve createdAt
+    const existing = await this.load(testCase, baseUrl);
+
+    const entry: CacheEntry = {
+      version: 1,
+      testCase,
+      baseUrl,
+      steps,
+      model: diagnostics.model,
+      provider: diagnostics.provider,
+      stepCount: diagnostics.stepCount,
+      aiMessage: diagnostics.aiMessage,
+      durationMs: diagnostics.durationMs,
+      createdAt: existing?.createdAt ?? now,
+      updatedAt: now,
+    };
+
+    const filePath = join(this.cacheDir, `${hash}.json`);
+    const tmpPath = `${filePath}.tmp`;
+
+    // Atomic write: write to tmp file, then rename
+    await Bun.write(tmpPath, JSON.stringify(entry, null, 2));
+    await rename(tmpPath, filePath);
+  }
+
+  /**
+   * Load a cache entry for the given test case.
+   * Returns null if the file does not exist or contains invalid JSON.
+   */
+  async load(testCase: string, baseUrl: string): Promise<CacheEntry | null> {
+    const hash = CacheManager.hashKey(testCase, baseUrl);
+    const filePath = join(this.cacheDir, `${hash}.json`);
+
+    try {
+      return await Bun.file(filePath).json() as CacheEntry;
+    } catch {
+      return null;
+    }
+  }
+
+  /**
+   * Delete a cache entry for the given test case.
+   * No-op if the file does not exist.
+   */
+  async delete(testCase: string, baseUrl: string): Promise<void> {
+    const hash = CacheManager.hashKey(testCase, baseUrl);
+    const filePath = join(this.cacheDir, `${hash}.json`);
+
+    try {
+      await Bun.file(filePath).delete();
+    } catch {
+      // No-op if file doesn't exist
+    }
+  }
+}