explorbot 0.2.3 → 0.2.4

package/docs/workflow/agentic-usage.md

@@ -0,0 +1,233 @@
+# Agentic Usage
+
+Explorbot is a terminal command, so a coding agent — Claude Code, Cursor, Codex, or your own script — can drive it the same way it drives `git` or `npm`. This page covers the two things an agent needs: getting Explorbot configured once, and handing it a test plan it wrote itself.
+
+The division of labour that works best: the agent decides *what* to test and writes it down as a plan; Explorbot figures out *how* to click through the app and reports what actually happened.
+
+## Start here: the global installation
+
+Configure models and keys once for the machine, and every later command works in any directory with no environment variables to remember:
+
+```bash
+npx explorbot init --global --provider openrouter --api-key sk-...
+npx explorbot explore https://app.example.com/login --max-tests 3
+```
+
+`init --global` writes `~/.explorbot/config.js` with the recommended model ids of this Explorbot version and stores the key in `~/.explorbot/.env`. It needs no terminal — with `--provider` there is no wizard, so an agent can run it unattended.
+
+The global config holds models and keys, never a site. Most commands carry the site themselves — `explore`, `plan`, `research`, `navigate`, `context`, `shell`, `freesail`, `docs collect`. The ones that take no URL argument — `test`, `learn`, `knows`, `experience`, `compact` — read it from `EXPLORBOT_URL`, and stop with `No site to explore` when it is unset:
+
+```bash
+EXPLORBOT_URL=https://app.example.com npx explorbot learn "/login" 'Sign in as ${env.APP_USER}'
+```
+
+This is the form to prefer. Each site explored gets its own folder under `~/.explorbot/sites/<host>/` holding `knowledge/`, `experience/`, and `output/`, so what Explorbot learns about an app is still there on the next run — the agent that explored `/checkout` yesterday does not start from zero today. Later runs can name the site by host instead of repeating the URL:
+
+```bash
+npx explorbot explore app.example.com/dashboard
+npx explorbot sites                                # what is registered, and when it last ran
+```
+
+The `EXPLORBOT_*` variables below still win over it: setting `EXPLORBOT_AI_PROVIDER` or `EXPLORBOT_AI_MODEL` builds the configuration from the environment and the global config is skipped. So an agent can install once and still override models or the URL per command.
+
+## One-liner API
+
+When nothing can be installed — a CI job, a container, someone else's machine — set `EXPLORBOT_AI_PROVIDER` and Explorbot builds a config from `EXPLORBOT_*` environment variables instead. Name a provider and you get its recommended models:
+
+```bash
+EXPLORBOT_URL=https://app.example.com \
+EXPLORBOT_AI_PROVIDER=openrouter \
+  npx explorbot explore /login --max-tests 3
+```
+
+No `init`, no config file, no project directory, no model IDs to look up. These variables win over the global installation, so a run can always be pinned to its own models. A project `explorbot.config.js` still wins over them, so adding these variables never changes the behavior of an existing project.
+
+### Variables
+
+<!-- START env -->
+| Variable | Required | Meaning |
+|---|---|---|
+| `EXPLORBOT_AI_PROVIDER` | yes | Provider name; fills every model role from its recommended models. Turns on config-free mode |
+| `EXPLORBOT_AI_MODEL` | no | Pins the main model — a model id for the provider, or a standalone provider/model-id |
+| `EXPLORBOT_URL` | yes | Base URL to test; the API boat reads it as the base endpoint |
+| `EXPLORBOT_VISION_MODEL` | no | Screenshot analysis; overrides the provider recommendation |
+| `EXPLORBOT_AGENTIC_MODEL` | no | Captain and Pilot decisions; overrides the provider recommendation |
+| `EXPLORBOT_OUTPUT` | no | Output root for states, plans, research, and reports. Defaults to the site dir under ~/.explorbot/sites |
+| `EXPLORBOT_EPHEMERAL` | no | Keep no state between runs — output goes to a fresh temp directory instead of the site dir |
+| `EXPLORBOT_KNOWLEDGE` | no | Inline knowledge text, applied to every page |
+| `EXPLORBOT_KNOWLEDGE_FILE` | no | Path to a knowledge markdown file |
+| `EXPLORBOT_API_SPEC` | no | OpenAPI spec path for the API boat |
+| `EXPLORBOT_NO_BANNER` | no | Suppress the startup banner, for machine-readable output |
+<!-- END env -->
+
+`EXPLORBOT_URL` is optional when the command itself carries an absolute URL, as `docs collect https://…` does. The [API boat](../api-testing/basics.md) reads it as the base endpoint.
+
+This table is generated from the registry in `src/config.ts`, which also feeds `explorbot --help` — so `npx explorbot --help` lists the same variables on any command, and an agent can discover them without reading these docs.
+
+### Naming models
+
+Set `EXPLORBOT_AI_PROVIDER` to a provider name and Explorbot uses that provider's recommended model for every role — the same IDs listed in [Providers](../basics/providers.md), maintained in [`models.json`](../../models.json):
+
+```bash
+EXPLORBOT_AI_PROVIDER=openrouter   # model, visionModel, and agenticModel all filled in
+```
+
+This is the form to reach for when you do not care which model runs, only that the run works. Recommendations change as models are released, so a provider name keeps up while a pinned ID does not.
+
+To pin the main model, add `EXPLORBOT_AI_MODEL`. With a provider set, it is the model id for that provider, used verbatim — slashes and all:
+
+```bash
+EXPLORBOT_AI_PROVIDER=openrouter \
+EXPLORBOT_AI_MODEL=openai/gpt-oss-120b:nitro \
+  npx explorbot explore /checkout
+```
+
+On its own, without a provider, `EXPLORBOT_AI_MODEL` must carry the provider as `provider/model-id`, and it sets only the main `model` — `visionModel` and `agenticModel` stay unset unless you add `EXPLORBOT_AI_PROVIDER` or set them explicitly. It splits on the **first** slash, so provider-qualified IDs survive intact:
+
+```
+openrouter/openai/gpt-oss-120b:nitro   → openrouter, model "openai/gpt-oss-120b:nitro"
+groq/openai/gpt-oss-20b                → groq, model "openai/gpt-oss-20b"
+anthropic/claude-haiku-4-5-20251001    → anthropic, model "claude-haiku-4-5-20251001"
+```
+
+`EXPLORBOT_VISION_MODEL` and `EXPLORBOT_AGENTIC_MODEL` override those roles the same way — a provider name for its recommendation, or `provider/model-id` to pin one. Mix the forms to take a provider's recommendations and override one role:
+
+```bash
+EXPLORBOT_AI_PROVIDER=groq \
+EXPLORBOT_AGENTIC_MODEL=anthropic \
+  npx explorbot explore /checkout
+```
+
+Supported providers: `openai`, `anthropic`, `google`, `groq`, `mistral`, `openrouter`, `sambanova`. Each is created with its conventional API-key variable — `OPENROUTER_API_KEY`, `GROQ_API_KEY`, `MISTRAL_API_KEY`, and so on.
+
+Not every provider has a recommendation for every role — Anthropic is recommended only for `agenticModel`, since Claude models are accurate but costly for token-heavy page reading. Naming a provider that has no recommendation for a role you asked for is an error that names the role, so combine providers as in the example above.
+
+A `.env` file in the working directory is loaded before the config lookup, so `EXPLORBOT_*` variables and API keys can live there instead of on the command line.
+
+### Knowledge without a project
+
+Both knowledge variables write into the run's knowledge directory, and both can be set at once.
+
+`EXPLORBOT_KNOWLEDGE` is the fast path for credentials — it applies to every page:
+
+```bash
+EXPLORBOT_KNOWLEDGE="Log in as admin@example.com / secret123. Dismiss the cookie banner first." \
+EXPLORBOT_URL=https://app.example.com \
+EXPLORBOT_AI_PROVIDER=openrouter \
+  npx explorbot explore /admin/users
+```
+
+`EXPLORBOT_KNOWLEDGE_FILE` points at a markdown file the agent wrote. Its frontmatter is preserved, so it can target specific URLs — see [Knowledge](./knowledge.md) for the format:
+
+```bash
+EXPLORBOT_KNOWLEDGE_FILE=./checkout-knowledge.md npx explorbot explore /checkout
+```
+
+### What this mode changes
+
+Config-free runs leave no trace in the working directory:
+
+- **Output goes to the site folder** — `~/.explorbot/sites/<host>/`, the same folder the global installation uses, so states, plans, research, and reports for one app collect in one place however the run was configured. `EXPLORBOT_OUTPUT` points them somewhere else. Read the path Explorbot resolved from the `Configuration built from EXPLORBOT_* environment variables. Output: …` line.
+- **`EXPLORBOT_EPHEMERAL=1` keeps nothing between runs** — output goes to a fresh temp directory instead, for throwaway CI jobs and demos. The [prima boat](../reference/commands.md#prima-boat) exposes the same switch as `--ephemeral`.
+- **Experience is written into the site folder.** What worked on a page is remembered and reused by later runs against the same host. `EXPLORBOT_EPHEMERAL=1` turns writing off, so an ephemeral run stays reproducible.
+- **The Historian is off.** No generated CodeceptJS or Playwright test files — that is the one thing the global installation gives that this mode does not. Plans and reports are still written.
+
+### Reading results
+
+Everything lands under the output root:
+
+| Path | Contents |
+|---|---|
+| `reports/<mode>-<session>.md` | Session report: coverage, defects, execution issues |
+| `plans/<page>.md` | The plan that was generated or executed |
+| `states/` | Per-state HTML, ARIA snapshots, and screenshots |
+| `research/` | UI maps produced by the Researcher |
+
+The report is the artifact to parse. It clusters findings by root cause and is written for a reader, not a machine.
+
+`explore` and `test` exit `0` whenever the session completes, and non-zero only when the run itself fails to start — a failing scenario is a result, not a crash. Do not read pass/fail from their exit code; read the report. `navigate` is the exception and exits `1` when a URL is unreachable, which makes it a useful pre-flight check.
+
+## Running agent-prepared plans
+
+A [test plan](./test-plans.md) is plain markdown. An agent that has read the codebase usually knows what a feature is supposed to do better than an agent looking at rendered HTML, so writing the plan and executing it are worth separating.
+
+Write the plan:
+
+```markdown
+<!-- suite -->
+# Checkout
+
+### Prerequisite
+
+* URL: /cart
+
+<!-- test
+priority: critical
+-->
+# Customer completes checkout with a saved card
+
+## Requirements
+/cart
+
+## Steps
+* Proceed to checkout from the cart
+* Pick the saved card as the payment method
+* Confirm the order
+
+## Expected
+* The order confirmation page shows an order number
+* The cart is empty afterwards
+```
+
+Then hand it to Explorbot:
+
+```bash
+EXPLORBOT_URL=https://app.example.com \
+EXPLORBOT_AI_PROVIDER=openrouter \
+  npx explorbot test checkout-plan.md '*'
+```
+
+The index argument selects tests: `1`, `1,3`, `1-5`, or `*` for all. The plan file is input only — Explorbot never rewrites it, so plans stay in version control next to the code they cover.
+
+Steps are guidance, not a script. The Tester adapts them to what the page actually shows, which is why steps should describe intent rather than selectors. Expected outcomes are the strict part: a test passes only when every one of them is verified. See the [Planner's outcome guidance](../web-testing/planner.md#built-in-styles) for what makes an outcome verifiable.
+
+To have Explorbot invent the scenarios instead, run `explorbot plan <path>` and read the generated file from `plans/`.
+
+## Inspecting a page without spending tokens
+
+Two commands help an agent orient itself before committing to a run:
+
+```bash
+npx explorbot context /login        # URL, headings, knowledge, interactive elements
+npx explorbot shell /login 'I.click("Sign in")'   # run one CodeceptJS command
+```
+
+`context` makes no AI calls. Use it to check that a page loads, that login knowledge applies, and that the elements a plan assumes are actually there.
+
+## The other boats
+
+The same variables drive API testing and doc collection.
+
+```bash
+EXPLORBOT_URL=https://api.example.com \
+EXPLORBOT_API_SPEC=./openapi.yaml \
+EXPLORBOT_AI_PROVIDER=openrouter \
+  npx explorbot api explore
+```
+
+```bash
+EXPLORBOT_AI_PROVIDER=openrouter \
+  npx explorbot docs collect https://app.example.com/dashboard --max-pages 20
+```
+
+`docs collect` takes its base URL from the absolute path argument, so `EXPLORBOT_URL` is optional there.
+
+Knowledge written by `EXPLORBOT_KNOWLEDGE` carries `endpoint: '*'` frontmatter alongside `url: '*'`, matching the convention `api init` and `api know` use. The API boat does not read knowledge at runtime yet; the frontmatter is there for when it does, and the web side ignores it.
+
+## See Also
+
+- [Test Plans](./test-plans.md) — the plan format in full
+- [Knowledge](./knowledge.md) — teaching Explorbot about your app
+- [Commands](../reference/commands.md) — every CLI command
+- [Continuous integration](./ci.md) — scheduled runs with cached experience
+- [Scripting](../reference/scripting.md) — the programmatic API when a CLI call is not enough

package/docs/workflow/application-spec.md

@@ -0,0 +1,73 @@
+# Application Specs
+
+An application spec is a versioned Markdown bundle that gives Explorbot previously collected information about an application. It can be produced by Docbot, another documentation tool, or by hand.
+
+Live HTML, ARIA, and screenshots remain the source of truth. Explorbot uses matching spec pages as supporting context and does not load the whole bundle into every prompt.
+
+## Configure
+
+Set the bundle directory in `explorbot.config.js`:
+
+```javascript
+export default {
+  dirs: {
+    spec: 'spec',
+  },
+};
+```
+
+Paths are resolved from the project directory. Use `--spec <path>` on a web command to override the configured bundle for one run. Both the bundle directory and its `index.md` path are accepted.
+
+## Bundle structure
+
+```text
+spec/
+|-- index.md
+`-- pages/
+    |-- home.md
+    `-- users.md
+```
+
+`index.md` is required and serves as a human-readable entry point. Its contents are not injected into agents. Page files may be nested anywhere below `pages/` and must use the contract below.
+
+## Page contract
+
+Every page is a Markdown file with YAML front matter:
+
+```markdown
+---
+format: explorbot-application-spec
+version: 1
+url: /users
+---
+
+# Users
+
+## Purpose
+
+Lists the application's users.
+
+## User Can
+
+- user can search users by name
+  Proof: A search field is visible above the user list.
+
+## User Might
+
+- user might export the user list
+  Signal: An unlabeled download control is present.
+```
+
+The front matter fields are mandatory:
+
+- `format` must be `explorbot-application-spec`.
+- `version` must be `1`.
+- `url` is the URL pattern used to select the page for the current browser state. It supports the same patterns as [knowledge files](./knowledge.md#url-patterns).
+
+The Markdown body is supplied to agents as written, so headings beyond those shown above are allowed. Use `User Can` only for observed capabilities and transitions. Put inferred or unverified capabilities under `User Might`; Explorbot will require confirmation from the live UI before relying on them.
+
+Screenshots and other relative links may be included for readers, but Explorbot currently consumes the Markdown text only.
+
+## Validation
+
+Explorbot rejects a bundle when `index.md` or `pages/` is missing, when it contains no page files, or when a page has an unsupported format, version, or missing URL.

package/docs/workflow/ci.md

@@ -0,0 +1,202 @@
+# Continuous Integration
+
+Once your Explorbot setup works locally — it logs in, explores, and produces sensible tests — the next step is to run it on a schedule, so the whole app gets re-tested continuously without anyone starting it by hand. This page shows what a CI run needs and gives a worked pipeline for the common providers. If you haven't run Explorbot headlessly yet, read [Running Explorbot](../basics/running.md) first.
+
+## What a CI run needs
+
+- **A provider API key from the environment.** Your local `.env` file is not in the repo; in CI, set the key (for example `OPENROUTER_API_KEY`) as a pipeline secret and pass it as an environment variable.
+- **Playwright browsers.** Install them in the job: `npx playwright install --with-deps chromium`.
+- **A headless browser.** That's the default — don't pass `--show` and there is nothing to configure.
+- **A start path and a test budget.** `npx explorbot explore / --max-tests 10` keeps the run bounded and predictable.
+- **A hard timeout as backstop.** AI runs can stall; a job-level timeout guarantees the pipeline never hangs.
+
+For login, commit your `knowledge/` directory to the repo — it's the input you control, and Explorbot reads it on every run. Keep credentials out of the files with `${env.NAME}` interpolation, as described in [Knowledge](./knowledge.md), and set those variables as pipeline secrets too. Add `--session` to the run command so Explorbot logs in once and reuses the saved session.
+
+## Cache, commit, or upload
+
+Three directories, three fates:
+
+- **Commit `knowledge/`.** It's authored by you and versioned like code.
+- **Cache `experience/` and `output/`.** `experience/` carries lessons between runs — fewer repeated failures, faster runs — so a cached CI job gets smarter every night instead of starting from zero. `output/` carries `session.json` plus previous plans and generated tests; with `--configure` the next run reloads a saved plan and re-runs its tests regression-style instead of planning everything fresh (see the [`--configure` reference](../reference/commands.md#explore)).
+- **Upload `output/reports/`, `output/tests/`, and `output/screencasts/` as artifacts.** Reports and screencasts are for humans to review after the run; generated tests are code you may want to review and commit. If you send results to Testomat.io with artifact storage, screencasts are already attached to tests there — see [Reporting on CI](#reporting-on-ci).
+
+## Exit codes and gating
+
+As covered in [Running Explorbot](../basics/running.md#exit-codes), `explore` exits `1` only when the run itself crashes — bad config, unreachable app, provider failure. Failing tests do not fail the job; they are findings, not crashes. So a green pipeline means "Explorbot ran", not "no bugs found".
+
+To see what it found, read the session report in `output/reports/` — the Analyst writes a markdown summary that clusters defects by root cause. For per-test pass/fail in machine-checkable form, enable the markdown test report or send results to Testomat.io with the `TESTOMATIO` project key; both are described in [Reporting](./reporting.md). API tests are stricter: `api test` and `api explore` exit `1` when any test fails, so they gate natively.
+
+## Reporting on CI
+
+A scheduled run is only worth having if its results are easy to review the next morning. The recommended stack gives you a report you read top-down — start at the overview, drill into a failing test, watch exactly what happened:
+
+1. **Send results to Testomat.io.** Add the `TESTOMATIO` project key as a pipeline secret. Every run lands in the dashboard with steps, statuses, and history — nothing to download from CI.
+2. **Enable artifact storage.** Connect an S3-compatible bucket under Settings > Artifacts in Testomat.io (credentials via `S3_*` pipeline variables — see [artifacts in cloud reports](./reporting.md#artifacts-in-cloud-reports)). Screenshots and screencasts attached to tests upload automatically.
+3. **Record screencasts.** Every test becomes a video with the executed actions and the current scenario step overlaid — reviewing one beats clicking through a screenshot trail.
+4. **Enable the Analyst.** Its summary becomes the run description on Testomat.io, so the run opens with defects clustered by root cause, coverage, and what works.
+
+The two config switches, next to your reporter settings:
+
+```js
+export default {
+  ai: {
+    agents: {
+      historian: { screencast: true },
+      analyst: { enabled: true },
+    },
+  },
+};
+```
+
+The review flow this buys: open the run, read the Analyst overview in the description, scan the failed tests, open one, watch its screencast.
+
+Without the cloud reporter, the fallback is `html: true` and `markdown: true` under `reporter` plus uploading `output/reports/` and `output/screencasts/` as CI artifacts — the pipelines below do. You lose history and the overview-to-video flow, but every run is still reviewable.
+
+**API runs** report through the same reporter, so the same `TESTOMATIO` key covers `api test` and `api explore`. There are no screencasts (no browser) and the Analyst covers web sessions only; upload `output/requests/` as a CI artifact instead — every request and response is logged there as YAML.
+
+## GitHub Actions
+
+A nightly run with caching, artifacts, and a manual trigger:
+
+```yaml
+name: Explorbot Nightly
+
+on:
+  schedule:
+    - cron: '0 3 * * *'
+  workflow_dispatch:
+
+jobs:
+  explore:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 24
+
+      - run: npm ci
+      - run: npx playwright install --with-deps chromium
+
+      - uses: actions/cache@v4
+        with:
+          path: |
+            experience
+            output
+          key: explorbot-${{ github.run_id }}
+          restore-keys: |
+            explorbot-
+
+      - run: npx explorbot explore / --max-tests 10 --session
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: explorbot-results
+          path: |
+            output/reports
+            output/tests
+            output/screencasts
+```
+
+The cache key includes the run id, so every run saves an updated cache and the next one restores the latest via `restore-keys`. To report runs to Testomat.io, add `TESTOMATIO: ${{ secrets.TESTOMATIO }}` to the `env` block — see [Reporting](./reporting.md).
+
+## GitLab CI
+
+Create the schedule under **CI/CD → Pipeline schedules** and set `OPENROUTER_API_KEY` as a masked CI/CD variable. The job itself:
+
+```yaml
+explorbot:
+  image: node:24
+  rules:
+    - if: $CI_PIPELINE_SOURCE == "schedule"
+  timeout: 1h
+  cache:
+    key: explorbot
+    paths:
+      - experience/
+      - output/
+  script:
+    - npm ci
+    - npx playwright install --with-deps chromium
+    - npx explorbot explore / --max-tests 10 --session
+  artifacts:
+    when: always
+    paths:
+      - output/reports/
+      - output/tests/
+      - output/screencasts/
+```
+
+## Jenkins
+
+Jenkins has no built-in cross-run cache, but the workspace persists between builds on the same agent — so `experience/` and `output/` carry over as long as you don't wipe the workspace (skip `cleanWs`, or use the Job Cacher plugin on those two directories).
+
+```groovy
+pipeline {
+  agent any
+  triggers { cron('H 3 * * *') }
+  options { timeout(time: 60, unit: 'MINUTES') }
+  environment { OPENROUTER_API_KEY = credentials('openrouter-api-key') }
+  stages {
+    stage('Explore') {
+      steps {
+        sh 'npm ci'
+        sh 'npx playwright install --with-deps chromium'
+        sh 'npx explorbot explore / --max-tests 10 --session'
+      }
+    }
+  }
+  post {
+    always {
+      archiveArtifacts artifacts: 'output/reports/**, output/tests/**, output/screencasts/**', allowEmptyArchive: true
+    }
+  }
+}
+```
+
+## Azure Pipelines
+
+Scheduled triggers are declared in the pipeline itself; `always: true` runs it even without new commits. Set the API key as a secret pipeline variable. Azure caches one path per `Cache@2` task and keys are immutable, so use two tasks with the same rolling-key pattern as the GitHub example.
+
+```yaml
+schedules:
+  - cron: '0 3 * * *'
+    branches:
+      include: [main]
+    always: true
+
+pool:
+  vmImage: ubuntu-latest
+
+steps:
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '24.x'
+  - task: Cache@2
+    inputs:
+      key: 'explorbot | "$(Build.BuildId)"'
+      restoreKeys: 'explorbot'
+      path: experience
+  - task: Cache@2
+    inputs:
+      key: 'explorbot-output | "$(Build.BuildId)"'
+      restoreKeys: 'explorbot-output'
+      path: output
+  - script: |
+      npm ci
+      npx playwright install --with-deps chromium
+      npx explorbot explore / --max-tests 10 --session
+    timeoutInMinutes: 60
+    env:
+      OPENROUTER_API_KEY: $(OPENROUTER_API_KEY)
+  - task: PublishPipelineArtifact@1
+    condition: always()
+    inputs:
+      targetPath: output/reports
+      artifact: explorbot-reports
+```