ghostlab 0.1.0__tar.gz → 0.2.0__tar.gz

{ghostlab-0.1.0 → ghostlab-0.2.0}/.gitignore

@@ -222,3 +222,12 @@ __marimo__/
 
 # Streamlit UI workspace (generated)
 ghostlab_workspace/
+
+# ghostlab spec-flow local artifacts
+.ghostlab/
+ghostlab.sqlite3
+
+# ghostlab jobs — track job.yaml + test-plan.*, ignore per-job run artifacts + db
+jobs/*/workspace/
+jobs/*/runs/
+jobs/*/ghostlab.sqlite3*

ghostlab-0.2.0/PKG-INFO

@@ -0,0 +1,506 @@
+Metadata-Version: 2.4
+Name: ghostlab
+Version: 0.2.0
+Summary: Local end-to-end testing lab for any MCP server: coding agents role-play real users, drive your tools over multiple turns, score runs, and render/interact with MCP Apps ui:// widgets.
+Project-URL: Documentation, https://sajjadgg.github.io/Rehearsal/
+Project-URL: Homepage, https://github.com/sajjadGG/Rehearsal
+Project-URL: Repository, https://github.com/sajjadGG/Rehearsal
+Project-URL: Issues, https://github.com/sajjadGG/Rehearsal/issues
+Project-URL: Changelog, https://github.com/sajjadGG/Rehearsal/releases
+Author: Sajjad Gholamzadeh
+License: MIT
+License-File: LICENSE
+Keywords: agents,ai-agents,claude,cli,codex,end-to-end-testing,evaluation,llm,llm-evaluation,mcp,mcp-apps,model-context-protocol,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Provides-Extra: apps
+Requires-Dist: playwright>=1.40; extra == 'apps'
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: mkdocs>=1.6; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: twine>=5.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.6; extra == 'docs'
+Provides-Extra: ui
+Requires-Dist: streamlit>=1.30; extra == 'ui'
+Description-Content-Type: text/markdown
+
+# MCP Rehearsal / Ghostlab
+
+> A local, end-to-end **testing lab for any MCP server** — coding agents role-play
+> real users, drive your tools over multiple turns, and the harness captures
+> traces, scores outcomes, and even **renders and clicks through MCP Apps UI
+> widgets**.
+
+[![CI](https://github.com/sajjadGG/Rehearsal/actions/workflows/ci.yml/badge.svg)](https://github.com/sajjadGG/Rehearsal/actions)
+[![Docs](https://img.shields.io/badge/docs-wiki-blue)](https://sajjadgg.github.io/Rehearsal/)
+[![Python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue)](pyproject.toml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![llms.txt](https://img.shields.io/badge/llms.txt-✓-purple)](llms.txt)
+
+**Test your MCP server the way it's actually used** — not with unit tests against
+the protocol, but with a real coding agent (Codex / Claude) that picks tools,
+makes mistakes, and tries to accomplish goals, while a second agent plays the
+user. Protocol-level checks (schema errors, a tool call that 500s) are useful
+sanity checks, but they aren't the real test — the real test is whether an
+agent can actually get a task done through your MCP, end to end.
+
+📖 **Docs wiki:** https://sajjadgg.github.io/Rehearsal/ · 🤖 **For agents:** [`llms.txt`](llms.txt) · 🛠 **Contributing:** [`CONTRIBUTING.md`](CONTRIBUTING.md)
+
+## Quickstart
+
+```bash
+python3.13 -m venv .venv
+.venv/bin/pip install -e .            # add '.[ui]' for the web UI, '.[apps]' for widget rendering
+
+ghostlab create
+```
+
+That's the whole flow. `ghostlab create` walks you through everything, end to end:
+
+1. **Name + target** — the only two prompts. A target is an MCP URL, or a path
+   to a target JSON / standard `mcpServers` config.
+2. **Discover** — connects to the target, lints its contract (schema errors,
+   risk labels), and probes any MCP Apps `ui://` widgets.
+3. **Configure semantic testing** — if `codex` is on your `$PATH`, offers to
+   wire it up as the **agent-under-test**: a real coding-agent session with
+   your MCP's tools available, so the dual-agent conversations below have
+   something to actually drive. (No codex? The wizard still finishes —
+   protocol-level suites still run, semantic/security just skip until a host
+   is configured; see [Runner Configs](#runner-configs).)
+4. **Generate a test plan** — personas × scenarios for the semantic/security
+   suites, plus deterministic coverage for every discovered tool
+   (`test-plan.yaml`), all editable afterward.
+5. **Pick which suites to run** — defaults to everything; narrow it to just
+   `semantic` while you're iterating, or the full set for a release check.
+6. **Run + review** — executes the plan against your configured host(s),
+   writes a colored pass/fail summary plus a dashboard, and prints the
+   readiness/gate verdict.
+
+Everything the wizard does is one of `discover` / `plan` / `test` / `review`
+under the hood — run any of them standalone afterward to iterate without
+repeating the whole wizard:
+
+```bash
+ghostlab discover --job <name>    # re-inspect after the target changes
+ghostlab plan --job <name>        # regenerate/curate the test plan
+ghostlab test --job <name>        # rerun (add --suite semantic to narrow it)
+ghostlab review --job <name>      # the readiness/gate report on its own
+```
+
+A job is a self-contained folder: `jobs/<name>/job.yaml` (target, hosts,
+generation/test defaults, gates — all editable), `test-plan.yaml`, `workspace/`
+(discover/generated/test artifacts + a local sqlite db), and `runs/`.
+
+## What you get
+
+| Stage | What it produces |
+| --- | --- |
+| **Discover** | A deterministic `contract.json` (schema lint, risk labels, MCP Apps metadata checks) and a refreshed `capabilities:` section in `job.yaml` |
+| **Plan** | A coverage-driven `test-plan.yaml`: deterministic protocol cases for every tool, plus generated persona/scenario cases for the semantic/security suites |
+| **Test** | Multi-host execution results (`results.json`/`results.md`), a standalone HTML dashboard, and — for conversational cases — full dual-agent transcripts with structured tool-call capture |
+| **Review** | A readiness report: pass/fail gate verdict, failure clusters, and prioritized repairs |
+
+## Goal
+
+Build a repeatable, sandboxed tester that can:
+
+- Run any target MCP app in an isolated environment.
+- Launch one coding-agent session as the **agent-under-test** (with target MCP injected).
+- Launch another coding-agent session as the **user emulator** (persona + goal driven).
+- Drive multi-turn interactions between them.
+- Capture full traces, tool activity, failures, and outcomes.
+
+This lets you test with your existing Codex/Claude usage path, instead of wiring a separate LLM provider deployment just for E2E testing.
+
+## Scope
+
+Rehearsal is intentionally **app-agnostic**:
+
+- Works with any MCP server reachable by stdio/SSE/streamable HTTP.
+- Supports local or remote MCP endpoints.
+- Supports multiple coding-agent runners (Codex, Claude Code, and future adapters).
+
+No Cortex-specific assumptions are required in the core harness.
+
+## Reference
+
+Everything below is the individual-command reference and advanced usage —
+useful once you're past the first `ghostlab create` run, or scripting CI.
+
+### Job folder layout
+
+```text
+jobs/<name>/
+  job.yaml          # target, setup, hosts, generation, test, prompts, gates
+  test-plan.yaml    # produced by `ghostlab plan`
+  workspace/        # discover/, generated/, test/ artifacts + ghostlab.sqlite3
+  runs/             # dual-agent run output
+```
+
+### Core dual-harness architecture
+
+1. **AUT Harness (Agent Under Test)** — starts a coding-agent session (Codex or
+   Claude Code), injects the target MCP server config into it, and exposes a
+   controlled I/O bridge so it can receive user messages and return
+   replies/tool results.
+2. **User Emulator Harness** — starts a second coding-agent session, gives it a
+   scenario file (persona, goals, constraints, success criteria), and asks it
+   to act like a realistic user, sending messages turn-by-turn to the AUT.
+3. **Orchestrator** — coordinates turn-taking, timeouts, retries, and stop
+   conditions; logs every message/event in structured format; produces a run
+   report with bug candidates and reproduction context.
+
+### Target configuration model
+
+Each test run points to a target definition:
+
+- `target.id`: unique name (`filesystem-mcp-local`, `my-app-staging`)
+- `transport`: `stdio` | `sse` | `streamable-http`
+- `connection`: command+args+env (stdio) or URL+headers (network transports)
+- `capabilities`: optional expected tools/resources/prompts
+- `startup`: optional health checks and boot timeout
+
+### Commands
+
+The package installs two equivalent console scripts: `ghostlab` and `rehearsal`.
+
+- `ghostlab create` — the end-to-end wizard described above.
+- `ghostlab init` — create a `job.yaml`/`ghostlab.yaml` from an existing target JSON, without the wizard prompts.
+- `ghostlab discover` — inspect the job's target, lint its contract, refresh capabilities.
+- `ghostlab plan` — generate (or curate) the coverage-driven test plan.
+- `ghostlab test` — execute the test plan across the job's host adapters.
+- `ghostlab review` — readiness report over discover + plan + test artifacts (release gate).
+- `ghostlab inspect` — connect to a target MCP and capture what it exposes (no job needed).
+- `ghostlab profile` — turn an `inspect.json` into a capability profile (codex).
+- `ghostlab generate-scenarios` / `generate-personas` / `generate-dataset` — build reusable persona×scenario datasets outside the job model.
+- `ghostlab review-dataset` / `run-dataset` — curate and run a standalone dataset.
+- `ghostlab run` — run one dual-agent E2E scenario directly.
+- `ghostlab evaluate` — score a run into a pass/fail verdict (codex judge).
+- `ghostlab compare` — diff two dataset runs for regressions.
+- `ghostlab apps-probe` / `apps-render` — probe/render MCP Apps `ui://` widgets.
+- `ghostlab doctor` — check codex and validate runner presets.
+- `ghostlab dashboard` — build a standalone HTML dashboard for a `ghostlab test` run.
+- `ghostlab ui` — launch the Streamlit pipeline UI.
+- `ghostlab db` — manage the SQLite persistence database.
+
+### The UI: `ghostlab ui`
+
+Run the whole pipeline from a browser instead of the CLI:
+
+```bash
+pip install 'ghostlab[ui]'       # installs streamlit
+ghostlab ui                      # opens http://localhost:8501
+```
+
+The app mirrors the same job-based flow as `ghostlab create`: pick or create a
+job, discover its target, configure the agent-under-test host, generate and
+curate the test plan, run selected suites, and review colored results and full
+conversation traces — the same `job.yaml`/`test-plan.yaml`/`results.json` a CLI
+run of the same job would produce.
+
+### Understand a new MCP: `inspect`
+
+Point it at a target and it introspects the server without any coding-agent
+credits or manual `curl`:
+
+```bash
+ghostlab inspect --target targets/cortex-local.json
+```
+
+This connects over the configured transport (stdio / streamable-HTTP / SSE),
+runs the `initialize` handshake, and pages through `tools/list`,
+`resources/list`, `resources/templates/list`, and `prompts/list`. It writes
+`runs/<id>-inspect/inspect.json` (raw) and `inspect.md` (readable), and **lints**
+tool/resource descriptions for references to tools the server does not actually
+expose. This capability dump is the input to capability profiling and scenario
+generation.
+
+### Profile a new MCP: `profile`
+
+Turn the raw `inspect.json` into a structured **capability profile** — the
+bridge between Understand and Generate. Deterministic structure (tool taxonomy
+by name family, read/write state surfaces, gaps) is computed locally; a domain
+summary and inferred multi-step workflows are generated by codex:
+
+```bash
+ghostlab profile \
+  --inspect runs/<id>-inspect/inspect.json
+```
+
+It writes `capabilities.json` + `capabilities.md` next to the `inspect.json`.
+Generated workflow steps are filtered to real tool names, so the profile never
+references hallucinated or non-exposed tools.
+
+### Generate scenarios: `generate-scenarios`
+
+Generate grounded use-case scenarios the MCP supports, derived from the
+capability profile:
+
+```bash
+ghostlab generate-scenarios \
+  --profile runs/<id>-inspect/capabilities.json \
+  --n 3 \
+  --output-dir scenarios
+```
+
+Scenarios are spread across intents (`happy_path` / `edge_case` / `adversarial`)
+and each declares an `exercises` list of the tools it should drive the assistant
+to use. Tool references are filtered to real tool names.
+
+### Build a persona library: `generate-personas`
+
+Personas are reusable **user profiles** decoupled from scenarios, so the same
+persona can be paired with many scenarios (the basis for the dataset matrix).
+Generate a domain-relevant library from a capability profile:
+
+```bash
+ghostlab generate-personas \
+  --profile runs/<id>-inspect/capabilities.json \
+  --n 4 \
+  --output-dir personas
+```
+
+Each persona has a `summary`, behavioral `traits` (terse, impatient, easily
+confused, non-native, ...), and a domain `context` map (native_language,
+target_exam, level, ...). Pass one to a run with `--persona`:
+
+```bash
+ghostlab run ... --persona personas/ielts-power-user.json
+```
+
+### Build a dataset: `generate-dataset`
+
+A dataset is a **persona x scenario matrix** — different users, and different
+scenarios tailored to each of them. For every persona, codex generates
+persona-specific scenarios, and the pairs become runnable cases:
+
+```bash
+ghostlab generate-dataset \
+  --profile runs/<id>-inspect/capabilities.json \
+  --personas 3 --scenarios-per-persona 3 --seed 7 \
+  --name cortex
+```
+
+This writes a self-contained dataset directory:
+
+```text
+datasets/cortex/
+  dataset.json          manifest: mcp, seed, cases[]
+  personas/<id>.json
+  scenarios/<id>.json    persona-namespaced; inline `persona` is a situational note
+```
+
+### Review & curate a dataset: `review-dataset`
+
+Before spending agent credits, check that the dataset makes sense:
+
+```bash
+ghostlab review-dataset \
+  --dataset datasets/cortex \
+  --profile runs/<id>-inspect/capabilities.json
+```
+
+This writes `review.md` + `review.json` with a tool-coverage matrix, per-case
+previews, and flags (near-duplicate cases, scenarios exercising non-exposed
+tools, personas with no scenarios). Curation is file-first — each case gets a
+`status` in `dataset.json` (`pending` / `approved` / `rejected` / `needs-edit`):
+
+```bash
+ghostlab review-dataset --dataset datasets/cortex \
+  --approve case-a case-b --reject case-c
+```
+
+### Run a dataset: `run-dataset`
+
+```bash
+ghostlab run-dataset \
+  --dataset datasets/cortex \
+  --target targets/cortex-local.json \
+  --aut-runner runners/codex-cortex-aut.json \
+  --user-runner runners/codex-user-emulator.json \
+  --limit 2
+```
+
+Each case runs through the orchestrator (with its persona) into its own run
+directory, and a dataset-level `summary.md` + `results.json` capture per-case
+status and turn counts.
+
+### Tool-call capture & output hygiene
+
+Every run captures structured MCP tool calls from the agent host. The codex AUT
+runners set `"parser": "codex-json"` and run `codex exec --json`, so the
+orchestrator parses the JSONL stream and records each `mcp_tool_call` with its
+arguments, result, error, and status into `events.jsonl`, with a per-turn table
+in `report.md`. stdout and stderr are kept separate: only stdout (with known
+host noise redacted) becomes the conversational message handed to the other
+agent, while raw stderr is logged for debugging.
+
+### Evaluate a run: `evaluate`
+
+```bash
+ghostlab evaluate --run runs/<id> --capabilities runs/<id>-inspect/capabilities.json
+```
+
+Combines deterministic checks over captured tool calls with a codex LLM-judge
+that scores each `success_criterion`/`failure_signal` from the transcript.
+Writes `verdict.json` + `verdict.md`; exits non-zero unless the verdict is
+`pass`.
+
+### Compare two runs: `compare`
+
+```bash
+ghostlab compare --base runs/<base>-summary --candidate runs/<cand>-summary \
+  --output comparison.md
+```
+
+Diffs case-by-case on verdict, listing regressions first, then fixes, then
+other changes. Exits non-zero when there are regressions, so it can gate CI.
+
+### MCP Apps: `apps-probe` / `apps-render`
+
+Some MCPs ship **MCP Apps UI** resources — a tool's `_meta.ui.resourceUri`
+points to a `ui://…` HTML widget a compatible host is expected to render.
+
+`apps-probe` connects to a target, finds every UI-producing tool, fetches each
+`ui://` resource, and reports render-readiness and CSP diagnostics:
+
+```bash
+ghostlab apps-probe --target targets/cortex-local.json
+```
+
+`apps-render` actually renders a `ui://` widget in headless Chrome, proving a
+user can see and use it — it implements the MCP Apps host bridge, mounts the
+widget in a sandboxed iframe, completes the `ui/initialize` handshake, feeds it
+real tool input/result, and can drive a sequence of UI intents:
+
+```bash
+pip install 'ghostlab[apps]' && playwright install chrome    # one-time
+ghostlab apps-render --target targets/cortex-local.json \
+  --tool views_generate_sentence_scramble \
+  --arguments '{"target_sentence":"The cat sat on the mat","shuffled_elements":["mat","The","on","sat","cat","the"]}' \
+  --intent '{"type":"reorder","value":["The","cat","sat","on","the","mat"]}' \
+  --intent '{"type":"reveal"}'
+```
+
+It writes `apps-render.json` + `apps-render.md`, a `widget.png` of the initial
+render, and `widget-final.png` after the intents run. Exit status is non-zero
+if the render errored or any assertion failed.
+
+### Session runner (one live agent across turns)
+
+By default each turn spawns a fresh agent process and the orchestrator replays
+the transcript. The session runner (`"kind": "codex-session"`) instead keeps
+one codex session alive: turn 1 records the `thread_id`, and later turns run
+`codex exec resume <thread_id>` so codex retains context — fewer tokens, no
+repeated cold-start noise.
+
+```bash
+ghostlab run --target targets/cortex-local.json --scenario <scenario.json> \
+  --aut-runner runners/codex-cortex-local-session.json --user-runner <user.json>
+```
+
+### Validate your setup: `doctor`
+
+```bash
+ghostlab doctor               # validates runners/*.json
+ghostlab doctor --runners runners/codex-cortex-local-session.json
+```
+
+Reports the codex binary + version and validates each runner's kind, command,
+and parser.
+
+### Default agent backend
+
+`codex` is the default coding-agent backend for generation, the
+agent-under-test, and judging. `inspect` needs no agent — it is a direct MCP
+client. The codex binary is auto-detected from `$PATH`, then the macOS app
+bundle (`/Applications/Codex.app/Contents/Resources/codex`); override with
+`$REHEARSAL_CODEX_BIN` or `--codex-bin`.
+
+### Colored output
+
+CLI output is colored automatically on a TTY (dual-agent transcripts,
+pass/fail/skip verdicts, gate failures). Set `NO_COLOR=1` (or
+`GHOSTLAB_COLOR=0`) to disable it, `GHOSTLAB_COLOR=1` to force it on (e.g.
+piping into a pager that groks ANSI).
+
+## Runner Configs
+
+Mock runner (no agent, free):
+
+```json
+{ "kind": "mock" }
+```
+
+Process runner:
+
+```json
+{
+  "kind": "process",
+  "command": ["codex", "exec", "-"],
+  "env": {},
+  "timeout_seconds": 300,
+  "prompt_mode": "stdin"
+}
+```
+
+The process runner starts one fresh process per turn. `prompt_mode` can be
+`stdin`, `append-arg`, or `replace-placeholder`. `ghostlab create` synthesizes
+one of these automatically for the agent-under-test host (wiring the target
+MCP in via codex's `-c mcp_servers.<id>...` overrides, including
+`bearer_token_env_var` for `Bearer ${VAR}`-style auth headers) — see
+`jobs/<name>/runners/aut.json` after running it. To use Claude Code or another
+agent as the AUT instead, hand-write a runner JSON (see
+`runners/claude-process.example.json`) and pass it via `--aut-runner` to
+`ghostlab create`/`ghostlab plan`, or add it directly under `hosts:` in
+`job.yaml`.
+
+## Install from PyPI
+
+```bash
+pip install ghostlab            # add [ui] and/or [apps] for those extras
+ghostlab --help
+```
+
+## Packaging & Release
+
+Build and validate distributions locally:
+
+```bash
+.venv/bin/python -m pytest
+.venv/bin/python -m build
+.venv/bin/twine check dist/*
+```
+
+CI runs tests on Python 3.10 through 3.13 and verifies that the package builds.
+Releases are automated: the **`publish.yml`** workflow builds the sdist + wheel,
+publishes them to PyPI via **Trusted Publishing**, and attaches them to the
+GitHub Release — triggered when you **publish a GitHub Release** (or run the
+workflow manually). Cut a release like:
+
+```bash
+# bump rehearsal/__init__.py __version__ first, then:
+gh release create v0.1.0 --generate-notes
+```
+
+To enable publishing, create the PyPI project **`ghostlab`** and add a Trusted
+Publisher for this repository, workflow `.github/workflows/publish.yml`,
+environment `pypi`. No PyPI username or token is committed.
+
+The Pages workflow builds the docs wiki with MkDocs and deploys it to GitHub
+Pages on pushes to `main`, `v*.*.*` release tags, and manual workflow runs. In
+the GitHub repository settings, set Pages to use GitHub Actions as the source.