skeptic-cli 0.2.0

package/README.md

@@ -0,0 +1,393 @@
+# skeptic
+
+CLI-first end-to-end testing for TypeScript specs. Skeptic runs Playwright tests,
+adds agent-friendly page discovery, and captures QA evidence that is useful in
+local debugging, CI, and coding-agent workflows.
+
+> Agent authors should also read [AGENTS.md](./AGENTS.md) for the recommended
+> inspect → author → run loop and the full fixture API.
+
+## Install For Development
+
+```bash
+git clone https://github.com/iamjr15/skeptic
+cd skeptic/cli
+npm install
+npm run build
+node dist/skeptic.mjs --help
+```
+
+To use the local checkout as a command:
+
+```bash
+npm link
+skeptic --help
+```
+
+## Quick Start
+
+```bash
+# Initialize a project
+skeptic init
+
+# Discover stable selectors before authoring a test
+skeptic inspect https://example.com
+
+# Capture an ad hoc QA evidence bundle without writing a spec
+skeptic observe https://example.com
+
+# Run TypeScript specs with full QA evidence
+skeptic run tests/homepage.spec.ts --observability --video --trace
+
+# Generate a validated TypeScript spec with AI
+skeptic generate -m "test the login page"
+
+# Check local setup
+skeptic doctor
+```
+
+`skeptic init` creates:
+
+- `package.json` when missing, or adds a `test:e2e` script and `skeptic-cli` dev dependency when present
+- `tests/`
+- `tests/package.json` with `type: "module"` so specs can use ESM without changing your app package mode
+- `tests/example.spec.ts`
+- `skeptic.config.yaml`
+- `tsconfig.json`
+- `.skeptic/.gitignore`
+- root `.gitignore` entries for `.skeptic/` and `skeptic-output/`
+
+## Agent Skills
+
+The npm package includes a `skeptic` skill for Claude Code, Codex, Cursor, and
+OpenCode. During `npm install`, Skeptic installs that skill into user-level agent
+skill directories when possible:
+
+| Agent | User skill directory |
+|---|---|
+| Claude Code | `~/.claude/skills/skeptic` |
+| Codex | `${CODEX_HOME:-~/.codex}/skills/skeptic` |
+| Cursor | `~/.cursor/skills/skeptic` |
+| OpenCode | `~/.opencode/skills/skeptic` |
+
+The installer only replaces skills previously managed by `skeptic-cli`; existing
+custom skills are left untouched. To skip automatic installation, set
+`SKEPTIC_SKIP_AGENT_SKILL_INSTALL=1` or `SKEPTIC_INSTALL_AGENT_SKILLS=0`.
+Automatic installation is skipped in CI unless `SKEPTIC_INSTALL_AGENT_SKILLS=1`
+is set. To install only selected agents, set
+`SKEPTIC_AGENT_SKILLS=claude,codex`.
+
+For a repository-scoped skill that should be committed with a project, run:
+
+```bash
+skeptic add skill --agent all --scope project
+```
+
+For an explicit user-level reinstall, run:
+
+```bash
+skeptic add skill --agent all --scope user
+```
+
+## Test Format
+
+Skeptic specs are ordinary TypeScript files that import from `skeptic-cli`.
+
+```ts
+import { test, expect } from "skeptic-cli";
+
+test("homepage smoke", async ({ page, snapshot, screenshot, observability }) => {
+  await page.goto("https://example.com");
+  await expect(page).toHaveTitle(/Example Domain/);
+
+  const tree = await snapshot(page);
+  await expect(tree.byRole("heading", { name: "Example Domain" })).toBeVisible();
+
+  await screenshot("homepage", { fullPage: true });
+  await observability.expectNoConsoleErrors();
+});
+```
+
+The fixture exposes:
+
+| Member | Purpose |
+|---|---|
+| `page` | Playwright `Page`, wrapped for cursor/action markers when video is enabled |
+| `snapshot` | ARIA + cursor-interactive discovery with refs and locator helpers |
+| `screenshot` | PNG screenshots, including annotated numbered-ref captures |
+| `settle` | Network-idle settle helper |
+| `observability` | Performance, network, console, and accessibility assertions |
+| `ai` | Vision-backed assertions, defect checks, and text extraction |
+| `ctx` | Per-test execution context and artifact paths |
+
+`expect` is re-exported from Playwright Test, so matchers like
+`toHaveURL`, `toBeVisible`, and `toHaveText` are available without a second
+import.
+
+## Discovery
+
+`skeptic inspect <url>` opens the page, captures an ARIA/cursor snapshot, and
+prints stable `selectorHint:` lines.
+
+```bash
+skeptic inspect https://example.com --interactive --compact
+skeptic inspect https://example.com --json
+skeptic inspect https://example.com --annotated --annotate-output inspect.png
+```
+
+Useful flags:
+
+| Flag | Purpose |
+|---|---|
+| `--interactive` | Show only ref-bearing entries |
+| `--compact` | Show interactive entries with minimal ancestors |
+| `--selector <css>` | Scope discovery to part of the page |
+| `--json` | Emit machine-readable refs, hints, and stats |
+| `--device <id>` | Inspect under a configured device profile |
+| `--connect <url>` | Attach to an existing browser over CDP |
+| `--with-playwright-hints` | Emit equivalent Playwright locator snippets |
+| `--annotated` | Save a numbered-ref screenshot |
+
+Refs like `e3` are runtime handles. Copy `selectorHint` strings into durable
+tests, or use `tree.byRef("e3")` only after a matching `snapshot(page)` call in
+the same test.
+
+## Running Tests
+
+```bash
+skeptic tui
+skeptic tui tests/login.spec.ts
+skeptic run
+skeptic run tests/login.spec.ts
+skeptic run tests/**/*.spec.ts --tag smoke
+skeptic run --parallel 4
+skeptic run --shard-split 4 --shard-index 1
+skeptic run --watch
+```
+
+`skeptic tui` is the discoverable interactive entrypoint. `skeptic run` opens
+the same TUI automatically in an interactive terminal when the console reporter
+is active; use `skeptic run --no-tui` for plain console output.
+
+Important flags:
+
+| Flag | Purpose |
+|---|---|
+| `--headed` | Show the browser |
+| `--ci` | Force headless CI behavior |
+| `--bail` | Stop after the first failing test |
+| `--retries <n>` | Retry failed tests |
+| `--timeout <ms>` | Playwright default action timeout |
+| `--hard-timeout <ms>` | Per-test ceiling enforced by the runner |
+| `--parallel <n>` | Run up to N spec-file workers concurrently |
+| `--shard-split <n>` | Split tests across N independent shard runs |
+| `--shard-all <n>` | Run all tests on each shard for variance checks |
+| `--reporter <format...>` | `console`, `json`, `junit`, `html` |
+| `--output <dir>` | Report and artifact directory |
+| `--list` | Discover tests without launching a browser |
+
+`--parallel` runs different spec files concurrently. Tests inside one file stay
+ordered so hooks, module state, and duplicate names remain predictable.
+
+## Observability
+
+`--observability` enables the full QA bundle:
+
+- visual settle before screenshots
+- full-page screenshots by default
+- performance metrics
+- network capture and issue detection
+- console capture
+- accessibility audit with automatic per-test `audit.md`
+- sidecar artifacts when report defaults allow them
+
+Use `--observability-write-sidecars` to force sidecars even when the reporter
+profile would not otherwise write them.
+
+```ts
+test("checkout stays healthy", async ({ page, observability }) => {
+  await page.goto("/checkout");
+  await observability.expectPerformance({ lcp: "<2500ms", cls: "<0.1" });
+  await observability.expectNoNetworkErrors();
+  await observability.expectNoConsoleErrors();
+  await observability.expectAccessible({ standard: "WCAG21AA" });
+});
+```
+
+Artifacts can include:
+
+- `results.json`
+- `report.html`
+- `junit.xml`
+- screenshots
+- WebM videos
+- Playwright trace zips
+- `perf-trace.md`
+- `network.json`
+- `console.json`
+- `accessibility.json`
+- `audit.md`
+
+## Observe
+
+`skeptic observe <url>` is the one-command evidence path for exploratory QA.
+
+```bash
+skeptic observe https://example.com --full-page
+skeptic observe https://example.com --no-video --no-trace
+```
+
+It writes an output directory containing an HTML report, JSON report,
+screenshots, annotated screenshots, snapshot text/JSON, console/network data,
+performance summary, accessibility JSON, and an accessibility markdown audit.
+
+## AI Features
+
+Skeptic supports Gemini, OpenAI, and Anthropic.
+
+```yaml
+ai:
+  provider: openai
+  model: gpt-4o
+```
+
+Set the matching provider key with `GEMINI_API_KEY`, `OPENAI_API_KEY`, or
+`ANTHROPIC_API_KEY`. You can also use `SKEPTIC_AI_PROVIDER` and
+`SKEPTIC_AI_API_KEY` to override config in CI.
+
+Available AI paths:
+
+- `ai.assert("the dashboard greets the user")`
+- `ai.assertNoDefects()`
+- `ai.extract("the invoice total")`
+- `skeptic generate --message "test checkout"`
+- `skeptic generate --diff`
+- `skeptic run --analyze`
+
+Generated specs are typechecked and imported before being written.
+
+## MCP And ACP
+
+`skeptic mcp` exposes testing and browser QA tools over stdio:
+
+| Tool | Purpose |
+|---|---|
+| `list_tests` | Discover specs |
+| `validate_tests` | Typecheck and import-check specs |
+| `generate_test` | Generate a validated TypeScript spec |
+| `run_test` | Run specs and stream progress |
+| `browser_open` | Open a page with config-driven browser/auth/safety |
+| `browser_snapshot` | Capture ARIA/cursor refs |
+| `browser_playwright` | Run focused Playwright code |
+| `browser_screenshot` | Capture PNG, annotated PNG, or snapshot-only output |
+| `browser_console_logs` | Read console messages |
+| `browser_network_requests` | Read requests and computed issues |
+| `browser_performance_metrics` | Capture Web Vitals, LoAF, resources, and `perf-trace.md` |
+| `browser_accessibility_audit` | Run axe-core plus IBM Equal Access when available |
+| `browser_close` | Close the browser session |
+
+`skeptic acp` exposes a testing-focused agent server for editors that support
+Agent Client Protocol.
+
+## Configuration
+
+`skeptic.config.yaml` lives in the project root.
+
+```yaml
+url: http://localhost:3000
+tests: "tests/**/*.spec.ts"
+
+browser:
+  engine: chromium
+  headless: true
+  timeout: 30000
+  viewport:
+    width: 1280
+    height: 720
+
+execution:
+  retries: 0
+  bail: false
+  parallel: 1
+
+output:
+  dir: ./skeptic-output
+  reporters: [console]
+
+observability:
+  collectors: []
+  defaultsForReports: passive
+  networkCaptureLimit: 500
+  duplicateWindowMs: 500
+  accessibilityDualEngine: false
+  autoAccessibilityAudit: false
+
+safety:
+  allowedDomains: []
+  confirmActions: []
+  maxOutputChars: 120000
+
+env:
+  BASE_URL: http://localhost:3000
+```
+
+## Cookie Injection
+
+```bash
+skeptic cookies list
+skeptic run --cookies
+skeptic run --cookies-from chrome
+```
+
+Cookie extraction is opt-in. Cookies are injected into local test browser
+contexts and are not sent to Skeptic services.
+
+## CI
+
+```bash
+skeptic add github-action
+skeptic add github-action --ai --provider openai
+```
+
+The generated workflow installs dependencies, installs Chromium, starts your dev
+server, runs Skeptic, uploads artifacts, and posts a PR comment with
+`skeptic comment`.
+
+## Notifications
+
+Optional Slack and webhook notifications are configured under `notifications`.
+Notification failures warn but do not fail the test run.
+
+```yaml
+notifications:
+  slack:
+    webhookUrl: ${SLACK_WEBHOOK_URL}
+    onFailure: true
+    onSuccess: false
+
+  webhook:
+    url: ${SKEPTIC_WEBHOOK_URL}
+    onFailure: true
+    onSuccess: false
+```
+
+Webhook payloads use a `tests` array with name, file, status, duration, error,
+and optional shard metadata.
+
+## Diagnostics
+
+```bash
+skeptic doctor
+skeptic doctor --json --quick
+skeptic browsers install chromium
+skeptic daemon status
+skeptic daemon stop
+```
+
+`skeptic doctor` checks config, output directories, browser installs, optional
+accessibility/cookie engines, daemon state, cookie profiles, and AI provider
+setup.
+
+## License
+
+MIT

package/agent-skills/skeptic/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: skeptic
+description: Use Skeptic for CLI-first browser QA and TypeScript E2E tests. Use when asked to inspect pages, write or run skeptic-cli specs, validate UI changes, capture observability evidence, or use Skeptic MCP tools. Not for unit-only logic with no browser behavior.
+---
+
+<!-- skeptic-agent-skill: managed by skeptic-cli -->
+
+# Skeptic
+
+Use Skeptic when a coding agent needs browser evidence: page inspection, TypeScript E2E specs, one-off QA captures, AI-backed checks, or MCP browser validation. Do not claim a UI/browser change works until you have run a relevant Skeptic command or MCP tool and checked the evidence.
+
+## Choose The Surface
+
+- One-off QA or bug hunt: run `skeptic observe <url> --full-page`.
+- Persistent regression coverage: run `skeptic inspect <url> --interactive --compact --with-playwright-hints`, write a `tests/*.spec.ts`, then run `skeptic run`.
+- Changed-code verification: run `skeptic run --diff` when the project has specs, or use `skeptic generate --diff` to create one.
+- Agent-integrated browser work: if Skeptic MCP tools are available, use `browser_open`, `browser_snapshot`, `browser_playwright`, `browser_screenshot`, `browser_console_logs`, `browser_network_requests`, `browser_performance_metrics`, `browser_accessibility_audit`, and `browser_close`.
+
+If the `skeptic` binary is not on PATH, try `npx skeptic-cli` or `npx --yes skeptic-cli@latest`.
+
+## Fast Loop
+
+```bash
+skeptic doctor --quick
+skeptic inspect <url> --interactive --compact --with-playwright-hints
+skeptic run tests/<scenario>.spec.ts --observability --video --trace
+```
+
+For a page with no existing spec:
+
+```bash
+skeptic observe <url> --full-page --video --trace
+```
+
+Use the generated `results.json`, `report.html`, screenshots, videos, traces, `network.json`, `console.json`, `accessibility.json`, and `perf-trace.md` as the evidence source. Reference artifact paths from `results.json` instead of guessing filenames.
+
+## Writing Specs
+
+Skeptic specs import from `skeptic-cli`.
+
+```ts
+import { test, expect } from "skeptic-cli";
+
+test("homepage smoke", async ({ page, snapshot, screenshot, observability }) => {
+  await page.goto("https://example.com");
+  await expect(page).toHaveTitle(/Example Domain/);
+
+  const tree = await snapshot(page, { interactive: true, compact: true });
+  await tree.byRole("link", { name: "More information..." }).click();
+
+  await screenshot("homepage", { fullPage: true });
+  await observability.expectNoConsoleErrors();
+});
+```
+
+Rules:
+
+- Put browser side effects inside `test(...)`, hooks, or helper functions called from tests.
+- Prefer role, label, text, and test-id locators over CSS.
+- Use `snapshot(page)` before interacting through refs or snapshot helpers.
+- Re-snapshot after navigation, route changes, modal open/close, or major DOM mutation.
+- Do not paste CLI `@eN` refs directly into specs. Use `selectorHint` from `inspect`, or use `tree.byRef("eN")` only for refs returned by the same in-test `snapshot(page)` call.
+- Add `screenshot("name")` for states that would help debug a failure.
+
+## Observability Checks
+
+Use `--observability` for real QA evidence. In specs, assert the signals that match the risk:
+
+```ts
+await observability.expectNoConsoleErrors();
+await observability.expectNoNetworkErrors({ allow: [/analytics/] });
+await observability.expectPerformance({ lcp: "<2500ms", cls: "<0.1" });
+await observability.expectAccessible({ standard: "WCAG21AA" });
+```
+
+If an observability artifact reports a failure, fix the product or the test and re-run the same flow immediately.
+
+## MCP Workflow
+
+When Skeptic is exposed through MCP:
+
+1. `browser_open` the target URL.
+2. `browser_snapshot` or `browser_screenshot` with snapshot mode to get refs.
+3. Use one `browser_playwright` call for actions that share the same DOM state. Use the `ref` helper for snapshot refs and `return` structured evidence.
+4. After DOM-changing actions, request a fresh snapshot.
+5. Check `browser_console_logs`, `browser_network_requests`, `browser_accessibility_audit`, and `browser_performance_metrics`.
+6. `browser_close` when done so video and trace artifacts flush.
+
+Batch fills, clicks, and data collection when the DOM is stable. Do not take a new snapshot between plain text fills unless the page structure changed.
+
+## Verification Standard
+
+Before reporting completion for browser-facing work:
+
+- Run the smallest Skeptic command or MCP workflow that actually exercises the changed behavior.
+- Test at least one adjacent or negative path when forms, routing, validation, auth, persistence, or shared components changed.
+- Read the full command/tool output. Passing navigation alone is not enough.
+- If there are console errors, network failures, serious accessibility issues, poor Web Vitals, or visible regressions, fix and re-run.
+- State the exact command/tool run and the main artifact path in the final report.

package/agent-skills/skeptic/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Skeptic"
+  short_description: "Use Skeptic to inspect pages, write TypeScript E2E tests, run browser QA, and collect observability evidence."
+  default_prompt: "Use Skeptic to verify this UI or browser behavior with real evidence and report the relevant artifacts."

package/bin/launcher.mjs

@@ -0,0 +1,18 @@
+#!/usr/bin/env node
+// JS package launcher.
+//
+// The published package always includes the JS bundle at `dist/skeptic.mjs`.
+//
+// This file is intentionally NOT processed by tsup — it's hand-written and
+// stays under `bin/` in the published tarball (declared in package.json
+// `files`). tsup output goes to `dist/skeptic.mjs`.
+
+import { fileURLToPath, pathToFileURL } from "node:url";
+import { dirname, join } from "node:path";
+
+const here = dirname(fileURLToPath(import.meta.url));
+const entrypoint = join(here, "..", "dist", "skeptic.mjs");
+
+// ESM `import()` rejects raw Windows paths like `C:\...`; convert to
+// `file://` URL first.
+await import(pathToFileURL(entrypoint).href);