explorbot 0.2.3 → 0.2.5

package/docs/contributing/npm-package.md

@@ -0,0 +1,138 @@
+# Building and Publishing the npm Package
+
+Explorbot develops on Bun but ships to npm as a Node.js-compatible package. This page covers how the build works and how to publish it.
+
+## Prerequisites
+
+- Bun (for development and running the build)
+- Node.js >= 24 (for verifying the build output)
+- npm account with publish access to `explorbot` package
+
+## How the Build Works
+
+The source is TypeScript with `.ts` imports (enabled by `allowImportingTsExtensions` in `tsconfig.json`). Bun runs these natively, but Node.js needs compiled `.js` files.
+
+The build runs the TypeScript compiler (`tsc`) with a dedicated `tsconfig.build.json`:
+
+1. **TypeScript compilation** - Compiles `src/`, `bin/`, and `boat/` to `dist/`, preserving the directory structure.
+2. **Import rewriting** - `rewriteRelativeImportExtensions` rewrites `.ts` imports to `.js` in the output (a TypeScript 5.7+ feature).
+3. **Type declarations** - `scripts/build-types.ts` emits `.d.ts` files for the library API (see [Type Declarations](#type-declarations)).
+4. **Asset copying** - Copies `rules/` and `assets/sample-files/` into `dist/` so runtime path resolution works.
+5. **Shebang replacement** - Replaces `#!/usr/bin/env bun` with `#!/usr/bin/env node` in every CLI entry point declared in `bin`: `dist/bin/explorbot-cli.js` and `dist/boat/prima/bin/prima-cli.js`.
+
+### Build Configuration
+
+**`tsconfig.build.json`** extends the base `tsconfig.json` with:
+
+| Option | Value | Purpose |
+|--------|-------|---------|
+| `noEmit` | `false` | Enable output (base config has `true`) |
+| `outDir` | `dist` | Compilation output directory |
+| `rewriteRelativeImportExtensions` | `true` | Rewrite `.ts` → `.js` in imports |
+| `declaration` | `false` | The JS build emits no `.d.ts`; declarations are built separately (see below) |
+| `sourceMap` | `false` | No source maps in the published package |
+| `skipLibCheck` | `true` | Skip type checking of dependencies |
+
+The build skips type checking (`--noCheck` flag) because Bun is more permissive than `tsc` strict mode. Bun enforces type safety during development.
+
+### Package Structure
+
+After the build, the npm package contains:
+
+```
+dist/
+├── bin/explorbot-cli.js    # CLI entry point (#!/usr/bin/env node)
+├── src/                    # Compiled application code (.js) + type declarations (.d.ts)
+│   ├── index.js            # Library entry point
+│   └── index.d.ts          # Library type declarations
+├── boat/                   # Compiled API tester module
+├── rules/                  # Agent rule files (markdown)
+└── assets/sample-files/    # Sample files for testing
+```
+
+### Type Declarations
+
+`declaration: true` doesn't work directly on this codebase: the Researcher agent is built from generic mixin factories (`WithDeepAnalysis(Base)` etc.) that return anonymous classes with `private` members, which TypeScript can't serialize into a `.d.ts` (`TS4094`). Rather than refactor those hot-path agents, `scripts/build-types.ts` generates declarations from a transformed copy of the source:
+
+1. Copies `src/` into a temporary tree, rewriting every `private`/`protected` class modifier to `public` at its exact AST position (members are preserved; only the visibility keyword changes, which removes `TS4094`).
+2. Runs `tsc --emitDeclarationOnly` over the copy into `dist/src/`.
+3. Rewrites `.ts`/`.tsx` module specifiers to `.js` in the emitted `.d.ts` so they resolve for consumers.
+4. Deletes the temporary tree.
+
+The transform touches only the intermediate copy — the shipped `.js` keeps its real `private`/`protected` visibility. The `.d.ts` types are exact (unions, option shapes, and return types are all preserved), so Node.js/TypeScript consumers get full type-checking. Bun consumers resolve the TypeScript source directly via the `bun` export condition.
+
+Key `package.json` fields:
+
+```json
+{
+  "bin": {
+    "explorbot": "./dist/bin/explorbot-cli.js",
+    "prima": "./dist/boat/prima/bin/prima-cli.js"
+  },
+  "main": "dist/src/index.js",
+  "types": "dist/src/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/src/index.d.ts",
+      "bun": "./src/index.ts",
+      "import": "./dist/src/index.js"
+    }
+  },
+  "files": [
+    "dist/",
+    "src/**/*.ts",
+    "src/**/*.tsx",
+    "bin/**/*.ts",
+    "boat/api-tester/src/**/*.ts",
+    "boat/doc-collector/src/**/*.ts",
+    "boat/doc-collector/bin/**/*.ts",
+    "boat/doc-collector/package.json",
+    "boat/prima/src/**/*.ts",
+    "boat/prima/bin/**/*.ts",
+    "boat/prima/package.json",
+    "rules/",
+    "assets/sample-files/"
+  ],
+  "engines": { "node": ">=24.0.0" }
+}
+```
+
+The package ships two commands: `explorbot`, and `prima` for the [prima boat](../reference/commands.md#prima-boat), so `npx -p explorbot prima <command>` works without a separate install.
+
+Explorbot is both a CLI (`bin`) and a library (`exports`). The `.` entry point is `src/index.ts`, a side-effect-free barrel that re-exports the public API (`ExplorBot`, `Plan`, `Test`, and their types). The `exports` conditions are ordered so each consumer gets the right entry: `types` (the emitted `.d.ts`) for type-checking, `bun` (the TypeScript source) under Bun, and `import` (the compiled JS) under Node.js. This is why the source `src/**` files ship alongside `dist/`.
+
+## Building Locally
+
+```bash
+# Build the npm package
+bun run build:npm
+
+# Verify the CLIs work on Node.js
+node dist/bin/explorbot-cli.js --help
+node dist/boat/prima/bin/prima-cli.js --help
+
+# Check what would be published
+npm pack --dry-run
+```
+
+## Publishing
+
+GitHub Actions publishes automatically (see below), but you can also publish manually:
+
+```bash
+# Bump version
+npm version patch  # or minor, major
+
+# Build and publish (prepublishOnly runs build:npm automatically)
+npm publish
+```
+
+## Known Limitations
+
+- **Type declarations are transform-generated** - Declarations come from a transformed copy of the source (see [Type Declarations](#type-declarations)), not from `tsc --declaration` directly, because the mixin-based agents can't emit declarations as written. The published `.d.ts` types are exact; the workaround only concerns how they're produced.
+
+## CI/CD
+
+The `test.yml` workflow verifies the npm build on every push. On Node.js 24 it runs `bun run build:npm`, then the Node smoke tests: `node --test tests/node/*.mjs`. The `publish.yml` workflow additionally checks `node dist/bin/explorbot-cli.js --help` before publishing.
+
+The `publish.yml` workflow publishes to npm when you push a version tag (`v*` or a bare `1.2.3`-style tag). It overwrites the package version from the tag; tags containing `beta`, `alpha`, `pre`, or `rc` publish to the `beta` dist-tag instead of `latest`.

package/docs/contributing/observability.md

@@ -0,0 +1,227 @@
+# Observability & Debugging
+
+Explorbot integrates with [Langfuse](https://langfuse.com) for tracing. Use it to see what happened during a session: what data each agent received, which tools it called, and how it decided.
+
+![Langfuse Trace View](../assets/langfuse-trace.png)
+
+## Why Observability?
+
+Without traces, you only see the final test result and basic logs. When Explorbot runs on its own, you need to see:
+
+- What prompts went to the AI.
+- What tools were called, and with what parameters.
+- Token usage and cost per session.
+- How long each operation took.
+- Errors and retries.
+
+Use this data to:
+
+- Debug failed tests. See what the AI saw and decided.
+- Create Knowledge fixes. Find the context that was missing.
+- Tune prompts and agent performance.
+- Understand why a test passed or failed.
+- Export sessions for the `/explorbot-debug` skill.
+
+## Setting Up Langfuse
+
+### 1. Create a Langfuse Account
+
+Sign up at [langfuse.com](https://langfuse.com) (free tier available) or self-host.
+
+### 2. Get Your API Keys
+
+From your Langfuse project settings, copy:
+- **Public Key**
+- **Secret Key**
+
+### 3. Configure Explorbot
+
+Add credentials to your `.env` file:
+
+```bash
+LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxx
+LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxx
+```
+
+Or configure in `explorbot.config.js`:
+
+```javascript
+export default {
+  ai: {
+    model: groq('gpt-oss-20b'),
+    langfuse: {
+      enabled: true,
+      publicKey: process.env.LANGFUSE_PUBLIC_KEY,
+      secretKey: process.env.LANGFUSE_SECRET_KEY,
+      baseUrl: 'https://cloud.langfuse.com', // or your self-hosted URL
+    },
+  },
+};
+```
+
+### 4. Run Explorbot
+
+Once configured, Explorbot traces every AI call. No code changes needed.
+
+## What Gets Traced
+
+Explorbot uses the [Vercel AI SDK integration](https://langfuse.com/docs/integrations/vercel-ai-sdk) with Langfuse. Each session captures:
+
+| Trace | Description |
+|-------|-------------|
+| `test: <scenario>` | Full test execution cycle |
+| `researcher: <url>` | Page analysis by Researcher agent |
+| `planner: <url>` | Test scenario generation |
+| `driller: <url>` | Component drilling |
+| `ai.generateText` | Text generation calls |
+| `ai.generateObject` | Structured output calls |
+| `codeceptjs.step` | Individual browser actions |
+
+Navigator has no span of its own — its AI calls appear as `ai.*` spans under the parent trace.
+
+## Analyzing Sessions
+
+### In Langfuse Dashboard
+
+1. Open your Langfuse project
+2. Find the session by timestamp or name
+3. Click to see the full trace tree
+4. Inspect individual spans for:
+   - Input prompts
+   - Output responses
+   - Token counts
+   - Duration
+   - Errors
+
+## Debugging with Claude Code
+
+Explorbot includes a Claude Code skill that analyzes failed sessions.
+
+### Using the Debug Skill
+
+Find the failed `test: <scenario>` trace in Langfuse and copy its trace ID. Then, in Claude Code, run:
+
+```
+/explorbot-debug
+```
+
+Give it the trace ID — the skill fetches the trace with all its observations via `bun .claude/skills/explorbot-debug/langfuse-export.ts <trace-id>`. The trace holds the full context: prompts, tool calls, page states, and AI decisions. Without a trace ID, the skill analyzes `output/explorbot.log` instead.
+
+### What the Skill Analyzes
+
+The skill looks for three failure patterns:
+
+| Pattern | Symptoms | Solution |
+|---------|----------|----------|
+| **Missing Context** | Wrong element clicked, didn't understand UI | Add Knowledge file with disambiguation rules |
+| **Wrong Prompts** | Incorrect assumptions, wrong flow | Add Knowledge with business context |
+| **Wrong Tool Choice** | Used click when form needed, typing issues | Add Knowledge with CodeceptJS code examples |
+
+### How It Helps
+
+1. Extracts key data from the trace with jq:
+   - Failed tool calls
+   - URLs visited
+   - Prompts sent to the AI
+
+2. Identifies the root cause of failures.
+
+3. Suggests Knowledge files to fix the issue:
+   ```markdown
+   ---
+   url: /admin/users/*
+   ---
+
+   ## User Table
+   Each row has same buttons. Use container:
+   I.click('Delete', '[data-user-id="123"]')
+   ```
+
+4. Can try interactions with browser tools, if available, and record working CodeceptJS code.
+
+### Example Workflow
+
+```bash
+# 1. Test fails
+./bin/explorbot-cli.ts explore /admin/users
+
+# 2. Open Langfuse, find the failed "test: ..." trace, copy its trace ID
+
+# 3. In Claude Code:
+/explorbot-debug
+# Provide the trace ID
+
+# 4. Skill analyzes and suggests Knowledge fix
+# 5. Create knowledge file
+./bin/explorbot-cli.ts learn "/admin/users/*" "Use container context for table actions"
+
+# 6. Re-run test
+```
+
+## Debugging Tips
+
+### Enable Verbose Logging
+
+```bash
+./bin/explorbot-cli.ts explore /admin/users --verbose
+```
+
+Or set the environment variable:
+
+```bash
+DEBUG=explorbot:* ./bin/explorbot-cli.ts explore /admin/users
+```
+
+This shows detailed logs:
+
+- Prompts sent to the AI
+- Tool calls and results
+- State transitions
+
+### Specific Debug Namespaces
+
+```bash
+# AI provider calls only
+DEBUG=explorbot:provider ./bin/explorbot-cli.ts explore /admin/users
+
+# Navigator agent only
+DEBUG=explorbot:navigator ./bin/explorbot-cli.ts explore /admin/users
+
+# Multiple namespaces
+DEBUG=explorbot:tester,explorbot:navigator ./bin/explorbot-cli.ts explore /admin/users
+```
+
+### Available Namespaces
+
+| Namespace | What it shows |
+|-----------|---------------|
+| `explorbot:provider` | AI API calls, responses |
+| `explorbot:provider:out` | Outgoing prompts |
+| `explorbot:provider:in` | Incoming responses |
+| `explorbot:navigator` | Navigation decisions |
+| `explorbot:researcher` | Page analysis |
+| `explorbot:planner` | Test scenario generation |
+| `explorbot:tester` | Test execution |
+| `explorbot:historian` | Experience saving |
+| `explorbot:quartermaster` | A11y analysis |
+
+## Cost Tracking
+
+Langfuse tracks token usage per call. Use it to:
+
+- Monitor cost across sessions
+- Compare model efficiency
+- Find expensive operations
+- Tune prompts to reduce tokens
+
+## Self-Hosting Langfuse
+
+For privacy or compliance, you can self-host Langfuse. Langfuse v3 requires docker compose with Postgres, ClickHouse, and Redis — follow the [self-hosting docs](https://langfuse.com/self-hosting).
+
+Then set `baseUrl` in your config:
+
+```javascript
+langfuse: {
+  baseUrl: 'http://localhost:3000',
+}
+```

package/docs/contributing/regression-tests.md

@@ -0,0 +1,103 @@
+# Self-Regression Tests
+
+The regression harness runs Explorbot **end-to-end with real AI models** against a controlled local fixture app, then asserts on the artifacts Explorbot produces (research, plans, test results). Unlike the integration tests in `tests/integration/` — which mock the LLM — these runs use a live provider, so they are nondeterministic and gated behind owner approval in CI.
+
+The harness lives in `tests/regression/` and is driven by `regression:*` commands in the repo `Bunoshfile.js`.
+
+## Scenarios
+
+### Scenario A — fresh explore (`regression:basic`)
+
+Runs `explorbot explore /issues --headless` against the Trackly fixture and asserts:
+
+- **Login evidence** — a plan targets a post-login route (`/issues`, `/settings`) and at least one research file describes a post-login page (proves the agent applied the seeded credentials and got past the gate).
+- **Research** — at least one research file with a heading, a UI-map table, and enough domain keywords.
+- **Scenarios identified** — the plan holds at least `MIN_PLANNED_TESTS` tests covering at least `MIN_FEATURE_GROUPS` feature areas.
+- **Tests passed** — the `Results: N passed, N failed` stdout line shows no failures and at least `MIN_PASSED` passing tests, cross-checked against the reporter markdown.
+
+Credentials are supplied through a seeded knowledge file (`tests/regression/seeds/knowledge/login.md`) matched to `/login`, so this scenario also exercises knowledge rules.
+
+### Scenario B — experience reuse (`regression:experience`)
+
+Proves Explorbot reuses prior context to pass a test it cannot pass cold. The Trackly Archive vault (`/vault`) is unlocked only by an access code that exists nowhere in the DOM — only in the server and in the seed files.
+
+- **Control run** (always, once) — runs the seed plan with **empty** knowledge and experience dirs. It must NOT pass. A fully-passing control means the vault gate is broken and fails the harness.
+- **Seeded run** (retried) — runs the same plan with the seed knowledge and experience dirs. It must pass.
+
+The plan (`tests/regression/seeds/vault-plan.md`) describes *what* to do but never contains the code. The seeded context supplies it two ways: `seeds/knowledge/vault.md` carries the access **code** (data), and `seeds/experience/vault.md` carries the interaction **recipe** (fill field, click Unlock). Both are prior seeded context; the control has neither and fails.
+
+> The code is carried in a knowledge file, not only experience, on purpose. Knowledge is Explorbot's prominent, deterministic channel for page-specific data — the same mechanism scenario A uses for login credentials. Empirically, cheap models followed an experience-only code recipe only occasionally, so an experience-only gate was a coin-flip; knowledge makes the gate reliable while the experience file still exercises recipe reuse.
+
+## The fixture app — "Trackly"
+
+A self-contained issue tracker served in-process by `tests/regression/fixture/server.ts` (`Bun.serve`, ephemeral port, fresh in-memory store per attempt). No external network, no database.
+
+| Route | Auth | Contents |
+|-------|------|----------|
+| `/login` | no | credentials form (always native HTML) |
+| `/issues` | yes | issue list, search, status filter, label menu (start page) |
+| `/issues/new` | yes | create form: title, description, priority, labels + assignees multiselects |
+| `/issues/:id` | yes | detail, change-status menu, comment form, delete modal |
+| `/settings` | yes | Profile/Preferences tabs + an activity iframe |
+| `/vault` | no | scenario B access-code gate |
+| `/api/*` | cookie | REST API over the same store, `+ /api/openapi.json` |
+
+The REST API mirrors the store and is served for a future API-testing scenario; the current scenarios do not exercise it.
+
+### Component variants (ARIA on/off)
+
+Every widget (button, text field, select, multiselect, dropdown menu, modal, tabs) renders in one of three variants, selectable per run:
+
+- `native` — semantic HTML (`<button>`, `<select multiple>`, `<dialog>`, `<details>`).
+- `aria` — custom `<div>` widgets with correct roles (`role="combobox"`, `aria-modal`, `role="menu"`, …).
+- `plain` — bare `<div>` widgets with no roles or labels (the hostile case).
+- `random` — a seeded RNG picks a variant per widget; the same `--seed` reproduces identical markup.
+
+Pass `--variant` and `--seed` to any scenario. The login page is always native so variant runs don't fail at the door.
+
+## Running locally
+
+```bash
+# No-AI sanity check: fixture routes, auth, API, vault gate, seed parsing, variant rendering
+bunx bunosh regression:smoke
+
+# Serve the fixture for manual inspection (append ?variant=plain&seed=7 to any page)
+bunx bunosh regression:serve
+
+# Real-AI runs (need a provider key)
+export OPENROUTER_API_KEY=sk-...
+bunx bunosh regression:experience --retries 0   # cheapest AI path
+bunx bunosh regression:basic --retries 0
+bunx bunosh regression:all                       # both scenarios, default retries
+```
+
+Each attempt runs in an isolated throwaway directory under `tests/regression/.runs/` (gitignored). Inspect `tests/regression/.runs/<scenario>-<variant>-a<n>/output/` for the research, plans, and reporter markdown a run produced, and `tests/regression/.runs/report.md` for the summary.
+
+### Variant matrix
+
+`regression:variants` runs a scenario across variants and reports the outcome per variant **without gating** — a `plain`-variant failure is data about model robustness, not a regression. Only a crashed or timed-out run fails it.
+
+```bash
+bunx bunosh regression:variants --scenario basic --variants native,aria,plain
+```
+
+## CI and the approval gate
+
+`.github/workflows/regression.yml` runs on `pull_request` and `workflow_dispatch`. The job uses the `regression` GitHub environment, which holds `OPENROUTER_API_KEY` behind a **required reviewer**. Every run — including fork PRs — pauses at the environment gate before the secret is exposed. The job runs the PR's own code, so the reviewer must read the PR diff (workflow, Bunoshfile, fixture, lib) before approving; that review is the security boundary. The workflow never uses `pull_request_target`.
+
+The report is posted as a sticky PR comment (on `pull_request`) or a `Regression Reports` GitHub Discussion (on `workflow_dispatch`). Fork PRs get a read-only token, so comment posting may fail; the report is also written to the job step summary and uploaded as the `regression-runs` artifact.
+
+The report also embeds the **Session Analysis** from the fresh-explore run — Explorbot's Analyst agent writes a prose summary of what works, defects, UX issues, and execution issues, which the harness reads from `output/reports/<label>.md` and appends to the comment. The Analyst is enabled in the config template (`ai.agents.analyst.enabled: true`); it adds one AI call per basic run.
+
+## Tuning
+
+- **Thresholds** (`MIN_PLANNED_TESTS`, `MIN_PASSED`, `MIN_KEYWORD_HITS`, `MIN_FEATURE_GROUPS`) live at the top of `tests/regression/lib/assertions.ts`.
+- **Models** live in `tests/regression/fixture/explorbot.config.js`. If cheap models are too flaky, promote a stronger model to `ai.model` or lower `MIN_PASSED`.
+- **Run size** — adjust `--max-tests` in `runBasicScenario` and the per-CLI `timeout` values in `Bunoshfile.js`.
+
+## Owner prerequisites (one-time)
+
+1. Settings → Environments → create `regression`; add the repo owner as a **Required reviewer**; deployment branch policy "No restriction".
+2. Add the environment secret `OPENROUTER_API_KEY` (a dedicated key with a low spend limit).
+3. Create a Discussions category named exactly `Regression Reports`.
+4. Ensure the OpenRouter account has credit for the models in the config template.

package/docs/contributing/testing.md

@@ -0,0 +1,95 @@
+# Testing Guide
+
+## Running Tests
+
+### Unit Tests
+```bash
+# Run all unit tests
+bun test tests/unit
+
+# Run specific test file
+bun test tests/unit/state-manager.test.ts
+
+# Run tests with coverage
+bun test tests/unit --coverage
+```
+
+### Coverage Reports
+
+#### Text Coverage Report
+```bash
+# Get coverage with text output
+bun run test:coverage
+```
+
+#### Coverage Summary Only
+```bash
+# Get just the coverage summary
+bun run test:coverage:summary
+```
+
+#### LCOV Coverage File
+The `--coverage` flag writes coverage data to `coverage/lcov.info`.
+
+## Coverage Configuration
+
+Coverage is configured in `bunfig.toml`:
+
+- **Coverage Directory**: `coverage/`
+- **Coverage Threshold**: 80%
+- **Included Files**: `src/**/*.{ts,tsx,js,jsx}`
+- **Excluded Files**: Test files, TUI components, build artifacts
+
+## Test Structure
+
+```
+tests/
+├── unit/                    # Unit tests (Bun)
+│   ├── state-manager.test.ts
+│   ├── experience-tracker.test.ts
+│   └── ...
+├── integration/             # AI agent tests with mocked LLM
+├── node/                    # Node.js build smoke tests
+├── regression/              # End-to-end runs with real AI
+└── mocks/                   # Test mocks
+```
+
+This page covers unit tests. The other suites: integration tests exercise AI agents against a mocked LLM via aimock — see [AI integration tests](./ai-integration-tests.md). Node smoke tests in `tests/node/` verify the compiled npm build and run in CI (`bun run test:node`). Regression tests run Explorbot with real AI against a local fixture app — see [regression tests](./regression-tests.md).
+
+## Writing Tests
+
+### Best Practices
+
+1. **Happy path focus**: Test successful scenarios and core functionality.
+2. **Test isolation**: Make each test independent and clean up after itself.
+3. **Descriptive names**: Name tests for what they check.
+4. **Mock external dependencies**: Use `MockAIProvider` for AI-related tests.
+5. **Temp directories**: Use `/tmp/` paths for file system tests.
+
+### Mock AI Provider
+
+```typescript
+import { MockAIProvider } from '../mocks/ai-provider.mock';
+
+const mockAI = new MockAIProvider();
+mockAI.setResponses([
+  { text: 'Mock response 1' },
+  { text: 'Mock response 2' }
+]);
+
+// Use mockAI.getModel() in your tests
+```
+
+## CI/CD Integration
+
+Generate coverage reports for your CI pipeline:
+
+```bash
+# Generate coverage for CI
+bun test tests/unit --coverage --coverage-reporter=lcov
+
+# Check coverage threshold (exits with error if below 80%)
+bun test tests/unit --coverage --coverage-reporter=text
+```
+
+Upload the LCOV file (`coverage/lcov.info`) to a coverage service like Codecov or Coveralls.