ghostlab 0.2.1__tar.gz → 0.3.0__tar.gz

ghostlab-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,114 @@
+# Contributing to Ghostlab
+
+Thanks for helping improve Ghostlab! This guide is written to be friendly to both
+humans and coding agents. For a machine-readable map of the project, see
+[`llms.txt`](llms.txt).
+
+## Project shape
+
+- Python package: `rehearsal` · installed CLI: `ghostlab` (alias `rehearsal`).
+- Pipeline: **understand → generate → run → evaluate**, plus optional SQLite
+  persistence, a Streamlit UI, and an MCP Apps render layer.
+- The MCP client uses only the standard library; coding-agent CLIs (Codex /
+  Claude) are the agent backends.
+- NVIDIA OpenShell is the default execution backend for real runner sessions
+  and local stdio MCPs; `--sandbox local` is the explicit trusted-host opt-out.
+
+## Setup
+
+```bash
+python3 -m venv .venv                   # Python 3.10+
+.venv/bin/pip install -e '.[dev]'       # add ',ui' and/or ',apps' for those features
+.venv/bin/pytest                        # run the test suite
+```
+
+Optional extras:
+
+- `.[ui]` — the Streamlit pipeline UI (`ghostlab ui`).
+- `.[apps]` — Playwright, for rendering MCP Apps `ui://` widgets
+  (`ghostlab apps-render`). After install: `playwright install chrome`.
+
+OpenShell integration work also needs a supported compute driver (Docker
+Desktop is the simplest local option) and a connected gateway:
+
+```bash
+openshell status
+ghostlab doctor
+```
+
+For documentation work:
+
+```bash
+.venv/bin/mkdocs serve                  # live-preview the wiki locally
+.venv/bin/mkdocs build --strict         # what CI enforces
+```
+
+## Conventions
+
+- **Support Python 3.10–3.13.** Keep syntax and dependencies compatible with
+  the declared `requires-python` range and CI matrix.
+- **Test your change.** Most logic is pure and unit-tested with `unittest` under
+  `tests/`. Browser/network-dependent paths should degrade gracefully and be
+  guarded so the suite passes without them.
+- **Keep artifacts hybrid.** Commands write human-readable `.md` + machine
+  `.json`; SQLite is the system of record but never required for a run to work.
+- **Keep commits focused** on one logical change, with tests or docs alongside
+  behavior changes. Don't commit generated output (`jobs/`, `runs/`, `datasets/`,
+  `dist/`, `site/`), caches, or `.venv` — they're gitignored.
+- Match the surrounding style: small modules, docstrings that explain *why*, and
+  no new heavyweight dependencies in the core (use an optional extra instead).
+
+## Making a change
+
+1. Branch off `main` (e.g. `feat/…`, `fix/…`, `docs/…`).
+2. Make the change with tests.
+3. Open a PR against `main` with a clear description of what and why. Reference
+   the issue it addresses (`Closes #NN`).
+
+When you **add or change a CLI command**, update all of:
+
+- `rehearsal/cli.py` (the command)
+- `tests/` (coverage)
+- `README.md` (the command reference)
+- `docs/cli.md` (the wiki page)
+
+Use `ghostlab` (not `rehearsal`) in public examples.
+
+## Pull request checklist
+
+Run the checks that match your change:
+
+```bash
+.venv/bin/pytest                        # always
+.venv/bin/mkdocs build --strict         # if you touched docs/ or mkdocs.yml
+.venv/bin/python -m build               # if you touched packaging
+.venv/bin/twine check dist/*            # "
+```
+
+Run the full set for release, packaging, workflow, or public-CLI changes.
+
+## Documentation site
+
+The wiki lives in `docs/` and is built with MkDocs. The **Pages** workflow
+(`.github/workflows/pages.yml`) deploys it to GitHub Pages on pushes to `main`,
+release tags, and manual runs. Public docs: <https://sajjadgg.github.io/Ghostlab/>.
+
+## Releases
+
+Publishing to PyPI is automated by `.github/workflows/publish.yml` via
+[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/) (no tokens).
+It runs when you **publish a GitHub Release** (or dispatch the workflow manually).
+Version tags use `v*.*.*`.
+
+To cut a release:
+
+1. Bump `__version__` in `rehearsal/__init__.py`.
+2. Validate locally: `.venv/bin/pytest && .venv/bin/python -m build && .venv/bin/twine check dist/*`.
+3. `gh release create v<x.y.z> --generate-notes` — this triggers `publish.yml`.
+
+## Where to start
+
+Browse the [open issues](https://github.com/sajjadGG/Ghostlab/issues) — they are
+labeled by pipeline stage (`pipeline:understand`, `pipeline:run`,
+`pipeline:evaluate`, …). Good first contributions: a new target/scenario/runner
+preset, an additional lint or assertion, or a per-widget MCP Apps assertion.

{ghostlab-0.2.1 → ghostlab-0.3.0}/PKG-INFO

@@ -1,13 +1,13 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: ghostlab
-Version: 0.2.1
-Summary: Local end-to-end testing lab for any MCP server: coding agents role-play real users, drive your tools over multiple turns, score runs, and render/interact with MCP Apps ui:// widgets.
+Version: 0.3.0
+Summary: OpenShell-first end-to-end evaluation lab for configured AI agents, MCP servers, skills, and MCP Apps.
 Project-URL: Documentation, https://sajjadgg.github.io/Ghostlab/
 Project-URL: Homepage, https://github.com/sajjadGG/Ghostlab
 Project-URL: Repository, https://github.com/sajjadGG/Ghostlab
 Project-URL: Issues, https://github.com/sajjadGG/Ghostlab/issues
 Project-URL: Changelog, https://github.com/sajjadGG/Ghostlab/releases
-Author: Sajjad Gholamzadeh
+Author: SajjadGG
 License: MIT
 License-File: LICENSE
 Keywords: agents,ai-agents,claude,cli,codex,end-to-end-testing,evaluation,llm,llm-evaluation,mcp,mcp-apps,model-context-protocol,testing
@@ -25,6 +25,8 @@ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Software Development :: Quality Assurance
 Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.10
+Requires-Dist: questionary<3,>=2.1
+Requires-Dist: rich<16,>=14
 Provides-Extra: apps
 Requires-Dist: playwright>=1.40; extra == 'apps'
 Provides-Extra: dev
@@ -40,7 +42,7 @@ Description-Content-Type: text/markdown
 
 # Ghostlab
 
-> A local, end-to-end **testing lab for any MCP server** — coding agents role-play
+> A local, end-to-end **testing lab for configured AI agents** — coding agents role-play
 > real users, drive your tools over multiple turns, and the harness captures
 > traces, scores outcomes, and even **renders and clicks through MCP Apps UI
 > widgets**.
@@ -51,12 +53,13 @@ Description-Content-Type: text/markdown
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 [![llms.txt](https://img.shields.io/badge/llms.txt-✓-purple)](llms.txt)
 
-**Test your MCP server the way it's actually used** — not with unit tests against
+**Test an agent the way it is actually used** — not only with unit tests against
 the protocol, but with a real coding agent (Codex / Claude) that picks tools,
 makes mistakes, and tries to accomplish goals, while a second agent plays the
 user. Protocol-level checks (schema errors, a tool call that 500s) are useful
 sanity checks, but they aren't the real test — the real test is whether an
-agent can actually get a task done through your MCP, end to end.
+agent can actually get a task done with its configured MCPs, skills, workspace,
+instructions, and runner, end to end.
 
 📖 **Docs wiki:** https://sajjadgg.github.io/Ghostlab/ · 🤖 **For agents:** [`llms.txt`](llms.txt) · 🛠 **Contributing:** [`CONTRIBUTING.md`](CONTRIBUTING.md)
 
@@ -74,21 +77,67 @@ agent can actually get a task done through your MCP, end to end.
 python3 -m venv .venv                # Python 3.10+
 .venv/bin/pip install -e .            # add '.[ui]' for the web UI, '.[apps]' for widget rendering
 
-ghostlab create
+ghostlab create                       # evaluate an MCP server
+ghostlab lab                          # evaluate a configured agent (model, skills, MCPs, code)
 ```
 
-That's the whole flow. `ghostlab create` walks you through everything, end to end:
+Pick `create` when the thing under test is an **MCP server**, and
+[`lab`](#evaluate-a-configured-agent-ghostlab-lab) when it is an **agent** — a
+model plus instructions, skills, MCPs, permissions, and a codebase.
 
-1. **Name + target** — the only two prompts. A target is an MCP URL, or a path
-   to a target JSON / standard `mcpServers` config.
+Ghostlab uses [NVIDIA OpenShell](https://docs.nvidia.com/openshell/latest/)
+as its default execution boundary. Install OpenShell, start a supported compute
+driver (Docker Desktop is the simplest local option), and confirm the gateway:
+
+```bash
+openshell status
+ghostlab doctor
+```
+
+`openshell status` must say `Connected`. On a Homebrew installation, a refused
+connection commonly means Docker is stopped or the gateway needs restarting:
+
+```bash
+open -a Docker                       # macOS, when Docker Desktop is installed
+brew services restart openshell
+openshell status
+```
+
+OpenShell is the default; there is no `--local` flag. Use the explicit
+`--sandbox local` escape hatch only for trusted code that you intentionally
+want to execute directly on the host:
+
+```bash
+ghostlab create --name trusted --agent agent.yaml --sandbox local --yes
+ghostlab discover --job trusted --sandbox local
+ghostlab test --job trusted --sandbox local
+ghostlab run ... --sandbox local
+```
+
+With OpenShell, Ghostlab creates separate sandboxes for the agent under test
+and user emulator, stages only declared files, forwards only allowlisted
+environment variables, attaches named OpenShell providers, captures sandbox
+logs with the run artifacts, and deletes the sandboxes at teardown. Local stdio
+MCP processes in the job pipeline are routed through the same boundary. OpenShell failures remain
+`harness_error`s and never silently fall back to host execution.
+
+That's the whole flow. Interactive `ghostlab create` guides you through the
+evaluation subject (agent/MCP/skill), OpenShell image/providers, generation
+size, four model roles (AUT, user emulator, generation, and judge), Codex
+approval/sandbox policy, runner lifecycle/timeout, release gate, and whether to
+run immediately. The Questionary/Rich terminal UI provides arrow-key selection,
+multi-select suite picking, color, and numbered progress. The same choices
+remain available as flags for scripts and CI.
+
+`ghostlab create` walks through everything, end to end:
+
+1. **Name + agent/target** — an agent JSON/YAML can compose a runner, MCPs,
+   skills, workspace, and assets. `--target` and `--skill` remain simple shorthands.
 2. **Discover** — connects to the target, lints its contract (schema errors,
    risk labels), and probes any MCP Apps `ui://` widgets.
-3. **Configure semantic testing** — if `codex` is on your `$PATH`, offers to
-   wire it up as the **agent-under-test**: a real coding-agent session with
-   your MCP's tools available, so the dual-agent conversations below have
-   something to actually drive. (No codex? The wizard still finishes —
-   protocol-level suites still run, semantic/security just skip until a host
-   is configured; see [Runner Configs](#runner-configs).)
+3. **Configure semantic testing** — wires the configured runner (Codex by
+   default) as the **agent-under-test** and displays its exact command, model,
+   approval mode, nested sandbox, parser, and timeout.
 4. **Generate a test plan** — personas × scenarios for the semantic/security
    suites, plus deterministic coverage for every discovered tool
    (`test-plan.yaml`), all editable afterward.
@@ -104,15 +153,44 @@ repeating the whole wizard:
 
 ```bash
 ghostlab discover --job <name>    # re-inspect after the target changes
+ghostlab config --job <name>      # exact resolved runner/models/sandbox config
 ghostlab plan --job <name>        # regenerate/curate the test plan
 ghostlab test --job <name>        # rerun (add --suite semantic to narrow it)
+ghostlab test --job <name> --resume  # keep completed cases; retry harness outages
+ghostlab create --name <name> --resume --yes  # continue the full job pipeline
 ghostlab review --job <name>      # the readiness/gate report on its own
 ```
 
-A job is a self-contained folder: `jobs/<name>/job.yaml` (target, hosts,
+The end-to-end creator has a strict semantic contract: it only prints
+`Evaluation ready` after at least one semantic/security conversation actually
+runs. Missing model access, an unavailable OpenShell provider, failed scenario
+generation, and placeholder-only plans produce a non-zero exit with corrective
+details. Standalone automation can opt into the same contract with
+`ghostlab plan --require-semantic` and `ghostlab test --require-semantic`.
+
+A job is a self-contained folder: `jobs/<name>/job.yaml` (agent, target, sandbox, hosts,
 generation/test defaults, gates — all editable), `test-plan.yaml`, `workspace/`
 (discover/generated/test artifacts + a local sqlite db), and `runs/`.
 
+To evaluate a skill instead of an MCP server:
+
+```bash
+ghostlab create --name release-notes-skill --skill ./skills/release-notes --yes
+```
+
+Skill discovery reads `SKILL.md`; planning generates persona-grounded semantic
+and adversarial cases; testing injects the skill instructions into the AUT and
+judges observable compliance. MCP-only protocol and Apps suites are omitted.
+
+To evaluate a composed agent:
+
+```bash
+ghostlab create --name my-agent --agent examples/agent.json --yes
+```
+
+The agent definition is the canonical evaluation subject. A one-MCP or
+one-skill job is normalized into the same shape.
+
 ## What you get
 
 | Stage | What it produces |
@@ -121,6 +199,11 @@ generation/test defaults, gates — all editable), `test-plan.yaml`, `workspace/
 | **Plan** | A coverage-driven `test-plan.yaml`: deterministic protocol cases for every tool, plus generated persona/scenario cases for the semantic/security suites |
 | **Test** | Multi-host execution results (`results.json`/`results.md`), a standalone HTML dashboard, and — for conversational cases — full dual-agent transcripts with structured tool-call capture |
 | **Review** | A readiness report: pass/fail gate verdict, failure clusters, and prioritized repairs |
+| **Rollout** | With `--pdf`, one document per run: configuration, inferred purpose, personas, transcript with tool calls, judge evidence, and critique |
+
+For a configured agent, `ghostlab lab` adds an inferred **purpose profile** —
+what the agent is for, its workflows, and its risk surface — and drives
+generation from that instead of from the tool inventory.
 
 ## See it in action
 
@@ -150,8 +233,9 @@ reasoning, and the full dual-agent transcript with inline tool calls:
 
 Build a repeatable, sandboxed tester that can:
 
-- Run any target MCP app in an isolated environment.
-- Launch one coding-agent session as the **agent-under-test** (with target MCP injected).
+- Run arbitrary configured agents inside NVIDIA OpenShell.
+- Compose zero or more MCPs, skills, instructions, workspace files, and assets.
+- Launch one coding-agent session as the **agent-under-test**.
 - Launch another coding-agent session as the **user emulator** (persona + goal driven).
 - Drive multi-turn interactions between them.
 - Capture full traces, tool activity, failures, and outcomes.
@@ -162,9 +246,11 @@ This lets you test with your existing Codex/Claude usage path, instead of wiring
 
 Ghostlab is intentionally **app-agnostic**:
 
-- Works with any MCP server reachable by stdio/SSE/streamable HTTP.
+- Treats an agent configuration—not a single MCP—as the evaluation boundary.
+- Works with MCP servers reachable by stdio/SSE/streamable HTTP and local skills.
 - Supports local or remote MCP endpoints.
-- Supports multiple coding-agent runners (Codex, Claude Code, and future adapters).
+- Supports multiple coding-agent runners (Codex, OpenCode, Claude Code, and future adapters).
+- Expresses an OpenCode agent's full configuration — model, instructions, skills, subagents, tool permissions, and any number of MCPs — and runs all of it inside the sandbox.
 
 No Cortex-specific assumptions are required in the core harness.
 
@@ -202,11 +288,12 @@ jobs/<name>/
 
 ### Core dual-harness architecture
 
-1. **AUT Harness (Agent Under Test)** — starts a coding-agent session (Codex or
-   Claude Code), injects the target MCP server config into it, and exposes a
+1. **AUT Harness (Agent Under Test)** — starts the configured runner (Codex,
+   Claude Code, or another process) inside OpenShell, supplies its complete
+   MCP/skill/workspace composition, and exposes a
    controlled I/O bridge so it can receive user messages and return
    replies/tool results.
-2. **User Emulator Harness** — starts a second coding-agent session, gives it a
+2. **User Emulator Harness** — starts a second isolated coding-agent session, gives it a
    scenario file (persona, goals, constraints, success criteria), and asks it
    to act like a realistic user, sending messages turn-by-turn to the AUT.
 3. **Orchestrator** — coordinates turn-taking, timeouts, retries, and stop
@@ -215,18 +302,26 @@ jobs/<name>/
 
 ### Target configuration model
 
-Each test run points to a target definition:
+Each test run points to an agent definition. Legacy target fields remain the
+primary discovery input for one-MCP/one-skill jobs:
 
 - `target.id`: unique name (`filesystem-mcp-local`, `my-app-staging`)
-- `transport`: `stdio` | `sse` | `streamable-http`
+- `transport`: `stdio` | `sse` | `streamable-http` | `skill`
 - `connection`: command+args+env (stdio) or URL+headers (network transports)
 - `capabilities`: optional expected tools/resources/prompts
 - `startup`: optional health checks and boot timeout
 
+The canonical `agent` section contains `runner`, `instructions`, and composable
+`inputs.mcps`, `inputs.skills`, and assets. The sibling `sandbox` section defaults
+to OpenShell and controls the image, uploads, workdir, policy, resource limits,
+network mode, environment allowlist, providers, logs, and cleanup.
+`backend: local` is explicit unsandboxed compatibility—not a fallback.
+
 ### Commands
 
 The package installs two equivalent console scripts: `ghostlab` and `rehearsal`.
 
+- `ghostlab lab` — guided setup for a **configured agent** (model, instructions, skills, MCPs, permissions, code), then generate scenarios from its inferred purpose and run them fully sandboxed.
 - `ghostlab create` — the end-to-end wizard described above.
 - `ghostlab init` — advanced: scaffold a standalone `ghostlab.yaml` **spec** from a target JSON (see [spec vs job](#spec-vs-job) — most users want `ghostlab create`).
 - `ghostlab discover` — inspect the job's target, lint its contract, refresh capabilities.
@@ -243,11 +338,125 @@ The package installs two equivalent console scripts: `ghostlab` and `rehearsal`.
 - `ghostlab scorecard` — roll run verdicts and critiques into a summary scorecard.
 - `ghostlab compare` — diff two dataset runs for regressions.
 - `ghostlab apps-probe` / `apps-render` — probe/render MCP Apps `ui://` widgets.
-- `ghostlab doctor` — check codex and validate runner presets.
+- `ghostlab doctor` — check the sandbox and both LLM backends (`--probe` for a live check).
 - `ghostlab dashboard` — build a standalone HTML dashboard for a `ghostlab test` run.
 - `ghostlab ui` — launch the Streamlit pipeline UI.
 - `ghostlab db` — manage the SQLite persistence database.
 
+### Evaluate a configured agent: `ghostlab lab`
+
+`ghostlab create` evaluates an **MCP server**. `ghostlab lab` evaluates an
+**agent** — the thing you actually ship: a model plus instructions, skills, MCP
+servers, tool permissions, subagents, and a codebase it operates on.
+
+```bash
+ghostlab lab --name release-bot
+```
+
+It walks ten steps, and nothing generated is used before you have seen it:
+
+| Step | |
+| --- | --- |
+| 1 Source | An existing `opencode.json`, an agent config, or from scratch |
+| 2 Purpose | Your description — authoritative over anything inferred |
+| 3 Model | Only models your OpenCode install can actually reach |
+| 4 Capabilities | Import from a standard `mcpServers` config, pick per server |
+| 5 Instructions, skills, code | Instruction files, skill folders, the workspace |
+| 6 Permissions | `read-only` / `edit-workspace` / `full-shell`, blast radius spelled out |
+| 7 Sandbox | Image, and the explicit credential opt-in |
+| 8 Profile | Review the inferred purpose, workflows, and risk surface |
+| 9 Scenarios | Generated from that profile; drop any you do not want |
+| 10 Run | Execute, judge, and report |
+
+Every answer is written to `job.yaml`, so the result is a reproducible file
+rather than a conversation that happened once.
+
+#### Purpose-driven generation
+
+Persona and scenario generation for an MCP job works from the tool inventory.
+For an agent that is the wrong question: an agent whose purpose lives in its
+prompt would get scenarios about tool families instead of about its job.
+
+Ghostlab instead reads the agent's description, instruction files, skill
+definitions, subagent prompts, permission posture, and MCP inventory, and infers
+what it is *for* — purpose, audience, workflows, and a grounded **risk surface**
+that seeds the adversarial scenarios. From a small release assistant it produced
+risks like *"prompt injection through changelog contents"* and *"claiming a
+release was published, violating the core instruction"*.
+
+The result is written to `workspace/agent-profile.json` and shown for review
+before anything is generated from it.
+
+#### Everything runs in the sandbox
+
+A configured agent is only worth testing with real permissions, and those are
+exactly the ones you do not want on your machine. So the whole agent goes
+inside OpenShell:
+
+- the **CLI** comes from `docker/agent-sandbox.Dockerfile` (a Linux image with
+  OpenCode installed — the host binary is platform-specific and cannot be
+  uploaded), and runs over the SSH channel;
+- its **MCPs** are launched by OpenCode inside that same container;
+- its **code** is an uploaded copy at `/sandbox/workspace`, so `edit` and `bash`
+  act on a throwaway;
+- **credentials** are an explicit opt-in, uploaded outside the workspace at mode
+  600, and redacted from every report;
+- **network** is default-deny, with a generated policy allowing only the model
+  provider and OpenCode's model catalog. Anything else the agent reaches for is
+  denied and shows up in the sandbox log — which is itself a finding.
+
+#### Rollout report
+
+`--pdf` assembles one document per run: the resolved configuration (secrets
+redacted), the inferred purpose, personas and scenarios, the full transcript
+with every tool call and its latency, the judge verdict with per-criterion
+evidence, and the tool-usability critique.
+
+```bash
+ghostlab test --job release-bot --pdf     # rollout.html + rollout.pdf per run
+```
+
+PDF rendering uses the browser the MCP Apps host already needs
+(`pip install 'ghostlab[apps]'`); without it the HTML is still written.
+
+See [Configured Agent Lab](docs/configured-agent-lab.md) for the full design.
+
+### Drive Ghostlab from a coding agent: the `ghostlab` skill
+
+`skills/ghostlab/` packages Ghostlab as a skill, so Claude Code (or any harness
+that loads `SKILL.md`) can set up an evaluation, run it, and read the results
+without being handed the CLI surface each time.
+
+```bash
+mkdir -p ~/.claude/skills
+ln -s "$PWD/skills/ghostlab" ~/.claude/skills/ghostlab    # or .claude/skills/ for one project
+```
+
+Then just ask: *"test whether this MCP actually works with a real agent"* or
+*"evaluate this agent config and tell me what breaks"*.
+
+The skill steers toward the **file-driven path** rather than the interactive
+wizard, because that is what a harness can actually drive and a reviewer can
+actually diff:
+
+```bash
+ghostlab create --name release-bot --agent ./agent.json --no-discover --yes
+ghostlab discover --job release-bot
+ghostlab plan --job release-bot --llm-backend opencode --model github-copilot/claude-sonnet-4.5
+ghostlab test --job release-bot --llm-backend opencode --model github-copilot/claude-sonnet-4.5 --pdf
+ghostlab review --job release-bot
+```
+
+An `agent.json` carries the full declarative runtime — model, instructions,
+skills, subagents, permissions, and any number of MCPs — with every path
+resolved relative to the file. `examples/agent-lab/` is a complete working one.
+
+It also encodes the judgement calls that are easy to get wrong: that a `fail` is
+usually a real finding rather than a harness bug, that the deterministic
+tool-call record outranks the judge's narrative when they disagree, and that
+`--sandbox local` is *required* (not optional) for MCPs needing host-only
+resources. See [`skills/README.md`](skills/README.md).
+
 ### The UI: `ghostlab ui`
 
 Run the whole pipeline from a browser instead of the CLI:
@@ -257,11 +466,12 @@ pip install 'ghostlab[ui]'       # installs streamlit
 ghostlab ui                      # opens http://localhost:8501
 ```
 
-The app mirrors the same job-based flow as `ghostlab create`: pick or create a
-job, discover its target, configure the agent-under-test host, generate and
-curate the test plan, run selected suites, and review colored results and full
-conversation traces — the same `job.yaml`/`test-plan.yaml`/`results.json` a CLI
-run of the same job would produce.
+The app mirrors the same job-based flow as `ghostlab create`. Its new-evaluation
+screen accepts composed agents, MCPs, or skills and configures OpenShell,
+providers, generation size, and gates in one place. An overview shows pipeline
+completion and resolved configuration; long stages stream their CLI logs live;
+results can be filtered and exported as the standalone dashboard. It reads and
+writes the same `job.yaml`/`test-plan.yaml`/`results.json` artifacts as the CLI.
 
 ### Understand a new MCP: `inspect`
 
@@ -383,12 +593,15 @@ ghostlab run-dataset \
   --target target.json \
   --aut-runner runners/codex-cortex-aut.json \
   --user-runner runners/codex-user-emulator.json \
+  --provider openai \
   --limit 2
 ```
 
 Each case runs through the orchestrator (with its persona) into its own run
 directory, and a dataset-level `summary.md` + `results.json` capture per-case
-status and turn counts.
+status and turn counts. `--provider` attaches an existing OpenShell provider to
+the runner sandboxes (and judge when `--evaluate` is used); repeat it when the
+composition needs more than one provider.
 
 ### Tool-call capture & output hygiene
 
@@ -469,26 +682,73 @@ ghostlab run --target target.json --scenario <scenario.json> \
 ```bash
 ghostlab doctor               # validates runners/*.json
 ghostlab doctor --runners runners/codex-cortex-local-session.json
+ghostlab doctor --sandbox local  # trusted local mode; skips OpenShell checks
 ```
 
-Reports the codex binary + version and validates each runner's kind, command,
-and parser.
+Reports both LLM backends (codex and opencode) with the selected one marked,
+validates each runner's kind, command, and parser, and checks the OpenShell
+CLI/gateway by default. Add `--probe` to verify a backend can actually answer
+instead of only confirming its binary exists.
+
+### Coding-agent backends (codex or opencode)
+
+Ghostlab drives a coding-agent CLI for generation, the agent-under-test, and
+judging. `inspect` needs no agent — it is a direct MCP client.
+
+| Backend | Select with | Models |
+| --- | --- | --- |
+| `codex` (default) | `--llm-backend codex` | Your ChatGPT/Codex plan |
+| `opencode` | `--llm-backend opencode` | GitHub Copilot, Azure, and any other provider you have authenticated |
 
-### Default agent backend
+The codex binary is auto-detected from `$PATH`, then the macOS app bundle
+(`/Applications/Codex.app/Contents/Resources/codex`); override with
+`$REHEARSAL_CODEX_BIN` or `--codex-bin`. The opencode binary is auto-detected
+from `$PATH` then `~/.opencode/bin/opencode`; override with
+`$GHOSTLAB_OPENCODE_BIN`.
 
-`codex` is the default coding-agent backend for generation, the
-agent-under-test, and judging. `inspect` needs no agent — it is a direct MCP
-client. The codex binary is auto-detected from `$PATH`, then the macOS app
-bundle (`/Applications/Codex.app/Contents/Resources/codex`); override with
-`$REHEARSAL_CODEX_BIN` or `--codex-bin`.
+Use opencode when codex is unavailable — no plan, exhausted quota, or a CLI too
+old for the model your account is pinned to. With Copilot already authenticated
+(`opencode auth login`), the whole loop runs on it:
+
+```bash
+ghostlab plan --job my-job --llm-backend opencode --model github-copilot/claude-sonnet-4.5
+ghostlab test --job my-job --llm-backend opencode --model github-copilot/claude-sonnet-4.5
+```
+
+Set it once per job instead of per command via `generation.backend` in
+`job.yaml`, or globally with `$GHOSTLAB_LLM_BACKEND`. Precedence is
+`--llm-backend` > `job.yaml` > env var > `codex`. Pick any model
+`opencode models github-copilot` lists; Ghostlab always pins one explicitly
+rather than inheriting opencode's own default, which may not exist on your
+provider.
+
+Verify a backend can actually answer — not just that its binary exists —
+with `ghostlab doctor --probe`. A `--version` check cannot detect an expired
+quota or a CLI/model mismatch, and those otherwise surface much later as
+`generation skipped`.
+
+This is separate from the execution backend: Codex is the default agent
+program, while OpenShell is the default sandbox in which agent programs run.
+Standalone `ghostlab inspect` is a low-level direct protocol command; for an
+untrusted local stdio MCP, create a job and use `ghostlab discover` so its
+declared uploads and sandbox policy are applied.
 
 ### Colored output
 
-CLI output is colored automatically on a TTY (dual-agent transcripts,
+Interactive `ghostlab create` uses Questionary for arrow-key choices and
+checkbox multi-select, with Rich panels and progress presentation. The rest of
+the CLI is colored automatically on a TTY (dual-agent transcripts,
 pass/fail/skip verdicts, gate failures). Set `NO_COLOR=1` (or
 `GHOSTLAB_COLOR=0`) to disable it, `GHOSTLAB_COLOR=1` to force it on (e.g.
 piping into a pager that groks ANSI).
 
+Use `ghostlab config --job <name>` to see the exact effective Codex command,
+model, approval policy, nested sandbox, runner parser/timeout, all four model
+roles, composed MCP/skill inputs, and OpenShell configuration. Add `--json` for
+machine-readable output. If a command omits `-m`, Ghostlab resolves the
+top-level model from Codex's `config.toml` and identifies that source. The
+Streamlit Overview and Configure tab expose the same values.
+
 ## Runner Configs
 
 Mock runner (no agent, free):