ghostlab 0.2.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {ghostlab-0.2.0 → ghostlab-0.2.1}/.gitignore +5 -4
- {ghostlab-0.2.0 → ghostlab-0.2.1}/CONTRIBUTING.md +2 -2
- ghostlab-0.2.0/README.md → ghostlab-0.2.1/PKG-INFO +104 -13
- ghostlab-0.2.0/PKG-INFO → ghostlab-0.2.1/README.md +64 -53
- {ghostlab-0.2.0 → ghostlab-0.2.1}/docs/cli.md +6 -6
- {ghostlab-0.2.0 → ghostlab-0.2.1}/docs/datasets-evaluation.md +1 -1
- {ghostlab-0.2.0 → ghostlab-0.2.1}/docs/getting-started.md +6 -6
- {ghostlab-0.2.0 → ghostlab-0.2.1}/docs/ghostlab-vision-gap-and-roadmap.md +1 -1
- {ghostlab-0.2.0 → ghostlab-0.2.1}/docs/index.md +2 -2
- {ghostlab-0.2.0 → ghostlab-0.2.1}/docs/publishing.md +1 -1
- ghostlab-0.2.1/examples/README.md +51 -0
- ghostlab-0.2.1/examples/mcp-config.json +7 -0
- ghostlab-0.2.0/scenarios/basic-discovery.json → ghostlab-0.2.1/examples/scenario.json +0 -1
- ghostlab-0.2.0/targets/cortex-local.json → ghostlab-0.2.1/examples/target.json +2 -2
- ghostlab-0.2.1/llms.txt +60 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/mkdocs.yml +3 -3
- {ghostlab-0.2.0 → ghostlab-0.2.1}/pyproject.toml +6 -8
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/__init__.py +1 -1
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/cli.py +12 -4
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/report.py +1 -1
- ghostlab-0.2.0/llms.txt +0 -61
- ghostlab-0.2.0/personas/careful-celpip-beginner.json +0 -21
- ghostlab-0.2.0/personas/general-learning-edge-case.json +0 -23
- ghostlab-0.2.0/personas/ielts-power-user.json +0 -23
- ghostlab-0.2.0/personas/skeptical-toefl-retaker.json +0 -22
- ghostlab-0.2.0/scenarios/cortex-onboarding-status.json +0 -19
- ghostlab-0.2.0/scenarios/cortex-practice-generation.json +0 -19
- ghostlab-0.2.0/scenarios/error-recovery.json +0 -19
- ghostlab-0.2.0/targets/cortex-ngrok.json +0 -29
- ghostlab-0.2.0/targets/example-http.json +0 -16
- ghostlab-0.2.0/targets/example-stdio.json +0 -18
- {ghostlab-0.2.0 → ghostlab-0.2.1}/LICENSE +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/docs/concepts.md +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/ghostlab/__init__.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/ghostlab/__main__.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/ghostlab/cli.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/__main__.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/apps_host/__init__.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/apps_host/assertions.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/apps_host/executor.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/apps_host/live.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/apps_host/protocol.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/apps_host/renderer.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/apps_host/report.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/codex_backend.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/compare.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/config.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/contract.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/critique.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/dashboard.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/dataset.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/evaluate.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/generate.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/hosts/__init__.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/hosts/base.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/hosts/direct.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/hosts/runner.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/inspect.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/jobs.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/logging.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/mcp_apps.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/mcp_client.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/mcp_config.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/mcp_targets.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/orchestrator.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/personas.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/plan.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/plan_generate.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/profile.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/prompts.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/readiness.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/review.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/runner_presets.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/runners.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/sampling.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/scorecard.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/setup_runtime.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/spec.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/storage/__init__.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/storage/db.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/storage/hashing.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/storage/ids.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/storage/migrations/0001_initial.sql +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/storage/redact.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/storage/repository.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/termcolor.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/testrun.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/tool_capture.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/types.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/ui/__init__.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/rehearsal/ui/app.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/claude-process.example.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/codex-cortex-aut.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/codex-cortex-local-aut.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/codex-cortex-local-session.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/codex-process.example.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/codex-user-emulator.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/mock-aut.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/runners/mock-user.json +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_apps_host.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_apps_live.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_cli_spec_flow.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_compare.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_config.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_contract.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_critique.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_dataset.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_evaluate.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_generate.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_hosts_runner.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_inspect_lint.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_jobs.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_mcp_apps.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_mcp_client.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_mcp_targets.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_orchestrator.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_personas.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_plan.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_plan_generate.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_profile.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_readiness.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_review.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_runner_presets.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_sampling.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_scorecard.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_session_runner.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_setup_runtime.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_spec.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_storage.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_testrun.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_tool_capture.py +0 -0
- {ghostlab-0.2.0 → ghostlab-0.2.1}/tests/test_widgets_and_dashboard.py +0 -0
|
@@ -227,7 +227,8 @@ ghostlab_workspace/
|
|
|
227
227
|
.ghostlab/
|
|
228
228
|
ghostlab.sqlite3
|
|
229
229
|
|
|
230
|
-
# ghostlab jobs —
|
|
231
|
-
jobs
|
|
232
|
-
|
|
233
|
-
|
|
230
|
+
# ghostlab jobs — runtime output (created by `ghostlab create`), not tracked
|
|
231
|
+
jobs/
|
|
232
|
+
|
|
233
|
+
# macOS
|
|
234
|
+
.DS_Store
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Contributing to
|
|
1
|
+
# Contributing to Ghostlab
|
|
2
2
|
|
|
3
3
|
Thanks for helping improve Ghostlab! This guide is written to be friendly to both
|
|
4
4
|
humans and coding agents. For a machine-readable map of the project, see
|
|
@@ -50,7 +50,7 @@ Optional extras:
|
|
|
50
50
|
|
|
51
51
|
## Where to start
|
|
52
52
|
|
|
53
|
-
Browse the [open issues](https://github.com/sajjadGG/
|
|
53
|
+
Browse the [open issues](https://github.com/sajjadGG/Ghostlab/issues) — they are
|
|
54
54
|
labeled by pipeline stage (`pipeline:understand`, `pipeline:run`,
|
|
55
55
|
`pipeline:evaluate`, …). Good first contributions: a new target/scenario/runner
|
|
56
56
|
preset, an additional lint or assertion, or a per-widget MCP Apps assertion.
|
|
@@ -1,12 +1,52 @@
|
|
|
1
|
-
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ghostlab
|
|
3
|
+
Version: 0.2.1
|
|
4
|
+
Summary: Local end-to-end testing lab for any MCP server: coding agents role-play real users, drive your tools over multiple turns, score runs, and render/interact with MCP Apps ui:// widgets.
|
|
5
|
+
Project-URL: Documentation, https://sajjadgg.github.io/Ghostlab/
|
|
6
|
+
Project-URL: Homepage, https://github.com/sajjadGG/Ghostlab
|
|
7
|
+
Project-URL: Repository, https://github.com/sajjadGG/Ghostlab
|
|
8
|
+
Project-URL: Issues, https://github.com/sajjadGG/Ghostlab/issues
|
|
9
|
+
Project-URL: Changelog, https://github.com/sajjadGG/Ghostlab/releases
|
|
10
|
+
Author: Sajjad Gholamzadeh
|
|
11
|
+
License: MIT
|
|
12
|
+
License-File: LICENSE
|
|
13
|
+
Keywords: agents,ai-agents,claude,cli,codex,end-to-end-testing,evaluation,llm,llm-evaluation,mcp,mcp-apps,model-context-protocol,testing
|
|
14
|
+
Classifier: Development Status :: 3 - Alpha
|
|
15
|
+
Classifier: Environment :: Console
|
|
16
|
+
Classifier: Intended Audience :: Developers
|
|
17
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Classifier: Programming Language :: Python :: 3
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
24
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
25
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
26
|
+
Classifier: Topic :: Software Development :: Testing
|
|
27
|
+
Requires-Python: >=3.10
|
|
28
|
+
Provides-Extra: apps
|
|
29
|
+
Requires-Dist: playwright>=1.40; extra == 'apps'
|
|
30
|
+
Provides-Extra: dev
|
|
31
|
+
Requires-Dist: build>=1.2; extra == 'dev'
|
|
32
|
+
Requires-Dist: mkdocs>=1.6; extra == 'dev'
|
|
33
|
+
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
34
|
+
Requires-Dist: twine>=5.0; extra == 'dev'
|
|
35
|
+
Provides-Extra: docs
|
|
36
|
+
Requires-Dist: mkdocs>=1.6; extra == 'docs'
|
|
37
|
+
Provides-Extra: ui
|
|
38
|
+
Requires-Dist: streamlit>=1.30; extra == 'ui'
|
|
39
|
+
Description-Content-Type: text/markdown
|
|
40
|
+
|
|
41
|
+
# Ghostlab
|
|
2
42
|
|
|
3
43
|
> A local, end-to-end **testing lab for any MCP server** — coding agents role-play
|
|
4
44
|
> real users, drive your tools over multiple turns, and the harness captures
|
|
5
45
|
> traces, scores outcomes, and even **renders and clicks through MCP Apps UI
|
|
6
46
|
> widgets**.
|
|
7
47
|
|
|
8
|
-
[](https://sajjadgg.github.io/
|
|
48
|
+
[](https://github.com/sajjadGG/Ghostlab/actions)
|
|
49
|
+
[](https://sajjadgg.github.io/Ghostlab/)
|
|
10
50
|
[](pyproject.toml)
|
|
11
51
|
[](LICENSE)
|
|
12
52
|
[](llms.txt)
|
|
@@ -18,12 +58,20 @@ user. Protocol-level checks (schema errors, a tool call that 500s) are useful
|
|
|
18
58
|
sanity checks, but they aren't the real test — the real test is whether an
|
|
19
59
|
agent can actually get a task done through your MCP, end to end.
|
|
20
60
|
|
|
21
|
-
📖 **Docs wiki:** https://sajjadgg.github.io/
|
|
61
|
+
📖 **Docs wiki:** https://sajjadgg.github.io/Ghostlab/ · 🤖 **For agents:** [`llms.txt`](llms.txt) · 🛠 **Contributing:** [`CONTRIBUTING.md`](CONTRIBUTING.md)
|
|
62
|
+
|
|
63
|
+
> **Naming:** the project and repo are **Ghostlab** (formerly *Rehearsal*). The CLI
|
|
64
|
+
> is `ghostlab`, with `rehearsal` kept as an alias, and the installed Python
|
|
65
|
+
> package is `rehearsal` — all the same project.
|
|
66
|
+
|
|
67
|
+
<p align="center">
|
|
68
|
+
<img src="static/ghostlablogo.png" width="400" alt="Logo" align="center">
|
|
69
|
+
</p>
|
|
22
70
|
|
|
23
71
|
## Quickstart
|
|
24
72
|
|
|
25
73
|
```bash
|
|
26
|
-
python3
|
|
74
|
+
python3 -m venv .venv # Python 3.10+
|
|
27
75
|
.venv/bin/pip install -e . # add '.[ui]' for the web UI, '.[apps]' for widget rendering
|
|
28
76
|
|
|
29
77
|
ghostlab create
|
|
@@ -74,6 +122,30 @@ generation/test defaults, gates — all editable), `test-plan.yaml`, `workspace/
|
|
|
74
122
|
| **Test** | Multi-host execution results (`results.json`/`results.md`), a standalone HTML dashboard, and — for conversational cases — full dual-agent transcripts with structured tool-call capture |
|
|
75
123
|
| **Review** | A readiness report: pass/fail gate verdict, failure clusters, and prioritized repairs |
|
|
76
124
|
|
|
125
|
+
## See it in action
|
|
126
|
+
|
|
127
|
+
**Watch a real coding-agent drive your MCP, turn by turn** — every tool call is
|
|
128
|
+
captured with its pass/fail status. Below is the live trace of a Hugging Face
|
|
129
|
+
MCP run, including two `hf_hub_query` calls that failed against the server:
|
|
130
|
+
|
|
131
|
+
<p align="center">
|
|
132
|
+
<img src="static/cli-run-trace.png" width="900" alt="CLI run trace showing per-tool-call status">
|
|
133
|
+
</p>
|
|
134
|
+
|
|
135
|
+
**Get a standalone HTML dashboard** — pass rate, per-case verdicts, and
|
|
136
|
+
suite/host tags at a glance:
|
|
137
|
+
|
|
138
|
+
<p align="center">
|
|
139
|
+
<img src="static/results-dashboard.png" width="760" alt="HTML results dashboard with pass-rate KPIs and per-case list">
|
|
140
|
+
</p>
|
|
141
|
+
|
|
142
|
+
**Drill into any case** — the goal and persona, the judge's verdict with its
|
|
143
|
+
reasoning, and the full dual-agent transcript with inline tool calls:
|
|
144
|
+
|
|
145
|
+
<p align="center">
|
|
146
|
+
<img src="static/case-report-transcript.png" width="760" alt="Per-case report with judge verdict and conversation transcript">
|
|
147
|
+
</p>
|
|
148
|
+
|
|
77
149
|
## Goal
|
|
78
150
|
|
|
79
151
|
Build a repeatable, sandboxed tester that can:
|
|
@@ -88,7 +160,7 @@ This lets you test with your existing Codex/Claude usage path, instead of wiring
|
|
|
88
160
|
|
|
89
161
|
## Scope
|
|
90
162
|
|
|
91
|
-
|
|
163
|
+
Ghostlab is intentionally **app-agnostic**:
|
|
92
164
|
|
|
93
165
|
- Works with any MCP server reachable by stdio/SSE/streamable HTTP.
|
|
94
166
|
- Supports local or remote MCP endpoints.
|
|
@@ -101,6 +173,23 @@ No Cortex-specific assumptions are required in the core harness.
|
|
|
101
173
|
Everything below is the individual-command reference and advanced usage —
|
|
102
174
|
useful once you're past the first `ghostlab create` run, or scripting CI.
|
|
103
175
|
|
|
176
|
+
### Spec vs job
|
|
177
|
+
|
|
178
|
+
There are two ways to hold an evaluation's config; **for almost everyone the
|
|
179
|
+
answer is a job.**
|
|
180
|
+
|
|
181
|
+
- **Job** (recommended) — a self-contained `jobs/<name>/` folder created by
|
|
182
|
+
`ghostlab create`. Every command takes `--job <name>` (or auto-detects
|
|
183
|
+
`job.yaml` in the current dir). This is the mainstream path the whole
|
|
184
|
+
Quickstart uses.
|
|
185
|
+
- **Spec** (advanced) — a single standalone `ghostlab.yaml` produced by
|
|
186
|
+
`ghostlab init`, addressed with `--spec <file>`. Useful for scripting or
|
|
187
|
+
keeping config outside the `jobs/` layout. Unless you specifically need that,
|
|
188
|
+
prefer a job.
|
|
189
|
+
|
|
190
|
+
The commands overlap (`discover`/`plan`/`test`/`review` accept either `--job` or
|
|
191
|
+
`--spec`); pick one model per evaluation and stay with it.
|
|
192
|
+
|
|
104
193
|
### Job folder layout
|
|
105
194
|
|
|
106
195
|
```text
|
|
@@ -139,7 +228,7 @@ Each test run points to a target definition:
|
|
|
139
228
|
The package installs two equivalent console scripts: `ghostlab` and `rehearsal`.
|
|
140
229
|
|
|
141
230
|
- `ghostlab create` — the end-to-end wizard described above.
|
|
142
|
-
- `ghostlab init` —
|
|
231
|
+
- `ghostlab init` — advanced: scaffold a standalone `ghostlab.yaml` **spec** from a target JSON (see [spec vs job](#spec-vs-job) — most users want `ghostlab create`).
|
|
143
232
|
- `ghostlab discover` — inspect the job's target, lint its contract, refresh capabilities.
|
|
144
233
|
- `ghostlab plan` — generate (or curate) the coverage-driven test plan.
|
|
145
234
|
- `ghostlab test` — execute the test plan across the job's host adapters.
|
|
@@ -150,6 +239,8 @@ The package installs two equivalent console scripts: `ghostlab` and `rehearsal`.
|
|
|
150
239
|
- `ghostlab review-dataset` / `run-dataset` — curate and run a standalone dataset.
|
|
151
240
|
- `ghostlab run` — run one dual-agent E2E scenario directly.
|
|
152
241
|
- `ghostlab evaluate` — score a run into a pass/fail verdict (codex judge).
|
|
242
|
+
- `ghostlab critique` — rate a run's tool ergonomics from the agent's perspective (codex).
|
|
243
|
+
- `ghostlab scorecard` — roll run verdicts and critiques into a summary scorecard.
|
|
153
244
|
- `ghostlab compare` — diff two dataset runs for regressions.
|
|
154
245
|
- `ghostlab apps-probe` / `apps-render` — probe/render MCP Apps `ui://` widgets.
|
|
155
246
|
- `ghostlab doctor` — check codex and validate runner presets.
|
|
@@ -178,7 +269,7 @@ Point it at a target and it introspects the server without any coding-agent
|
|
|
178
269
|
credits or manual `curl`:
|
|
179
270
|
|
|
180
271
|
```bash
|
|
181
|
-
ghostlab inspect --target
|
|
272
|
+
ghostlab inspect --target examples/target.json
|
|
182
273
|
```
|
|
183
274
|
|
|
184
275
|
This connects over the configured transport (stdio / streamable-HTTP / SSE),
|
|
@@ -239,7 +330,7 @@ confused, non-native, ...), and a domain `context` map (native_language,
|
|
|
239
330
|
target_exam, level, ...). Pass one to a run with `--persona`:
|
|
240
331
|
|
|
241
332
|
```bash
|
|
242
|
-
ghostlab run ... --persona
|
|
333
|
+
ghostlab run ... --persona persona.json
|
|
243
334
|
```
|
|
244
335
|
|
|
245
336
|
### Build a dataset: `generate-dataset`
|
|
@@ -289,7 +380,7 @@ ghostlab review-dataset --dataset datasets/cortex \
|
|
|
289
380
|
```bash
|
|
290
381
|
ghostlab run-dataset \
|
|
291
382
|
--dataset datasets/cortex \
|
|
292
|
-
--target
|
|
383
|
+
--target target.json \
|
|
293
384
|
--aut-runner runners/codex-cortex-aut.json \
|
|
294
385
|
--user-runner runners/codex-user-emulator.json \
|
|
295
386
|
--limit 2
|
|
@@ -339,7 +430,7 @@ points to a `ui://…` HTML widget a compatible host is expected to render.
|
|
|
339
430
|
`ui://` resource, and reports render-readiness and CSP diagnostics:
|
|
340
431
|
|
|
341
432
|
```bash
|
|
342
|
-
ghostlab apps-probe --target
|
|
433
|
+
ghostlab apps-probe --target target.json
|
|
343
434
|
```
|
|
344
435
|
|
|
345
436
|
`apps-render` actually renders a `ui://` widget in headless Chrome, proving a
|
|
@@ -349,7 +440,7 @@ real tool input/result, and can drive a sequence of UI intents:
|
|
|
349
440
|
|
|
350
441
|
```bash
|
|
351
442
|
pip install 'ghostlab[apps]' && playwright install chrome # one-time
|
|
352
|
-
ghostlab apps-render --target
|
|
443
|
+
ghostlab apps-render --target target.json \
|
|
353
444
|
--tool views_generate_sentence_scramble \
|
|
354
445
|
--arguments '{"target_sentence":"The cat sat on the mat","shuffled_elements":["mat","The","on","sat","cat","the"]}' \
|
|
355
446
|
--intent '{"type":"reorder","value":["The","cat","sat","on","the","mat"]}' \
|
|
@@ -369,7 +460,7 @@ one codex session alive: turn 1 records the `thread_id`, and later turns run
|
|
|
369
460
|
repeated cold-start noise.
|
|
370
461
|
|
|
371
462
|
```bash
|
|
372
|
-
ghostlab run --target
|
|
463
|
+
ghostlab run --target target.json --scenario <scenario.json> \
|
|
373
464
|
--aut-runner runners/codex-cortex-local-session.json --user-runner <user.json>
|
|
374
465
|
```
|
|
375
466
|
|
|
@@ -1,52 +1,12 @@
|
|
|
1
|
-
|
|
2
|
-
Name: ghostlab
|
|
3
|
-
Version: 0.2.0
|
|
4
|
-
Summary: Local end-to-end testing lab for any MCP server: coding agents role-play real users, drive your tools over multiple turns, score runs, and render/interact with MCP Apps ui:// widgets.
|
|
5
|
-
Project-URL: Documentation, https://sajjadgg.github.io/Rehearsal/
|
|
6
|
-
Project-URL: Homepage, https://github.com/sajjadGG/Rehearsal
|
|
7
|
-
Project-URL: Repository, https://github.com/sajjadGG/Rehearsal
|
|
8
|
-
Project-URL: Issues, https://github.com/sajjadGG/Rehearsal/issues
|
|
9
|
-
Project-URL: Changelog, https://github.com/sajjadGG/Rehearsal/releases
|
|
10
|
-
Author: Sajjad Gholamzadeh
|
|
11
|
-
License: MIT
|
|
12
|
-
License-File: LICENSE
|
|
13
|
-
Keywords: agents,ai-agents,claude,cli,codex,end-to-end-testing,evaluation,llm,llm-evaluation,mcp,mcp-apps,model-context-protocol,testing
|
|
14
|
-
Classifier: Development Status :: 3 - Alpha
|
|
15
|
-
Classifier: Environment :: Console
|
|
16
|
-
Classifier: Intended Audience :: Developers
|
|
17
|
-
Classifier: License :: OSI Approved :: MIT License
|
|
18
|
-
Classifier: Operating System :: OS Independent
|
|
19
|
-
Classifier: Programming Language :: Python :: 3
|
|
20
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
21
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
22
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
23
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
24
|
-
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
25
|
-
Classifier: Topic :: Software Development :: Quality Assurance
|
|
26
|
-
Classifier: Topic :: Software Development :: Testing
|
|
27
|
-
Requires-Python: >=3.10
|
|
28
|
-
Provides-Extra: apps
|
|
29
|
-
Requires-Dist: playwright>=1.40; extra == 'apps'
|
|
30
|
-
Provides-Extra: dev
|
|
31
|
-
Requires-Dist: build>=1.2; extra == 'dev'
|
|
32
|
-
Requires-Dist: mkdocs>=1.6; extra == 'dev'
|
|
33
|
-
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
34
|
-
Requires-Dist: twine>=5.0; extra == 'dev'
|
|
35
|
-
Provides-Extra: docs
|
|
36
|
-
Requires-Dist: mkdocs>=1.6; extra == 'docs'
|
|
37
|
-
Provides-Extra: ui
|
|
38
|
-
Requires-Dist: streamlit>=1.30; extra == 'ui'
|
|
39
|
-
Description-Content-Type: text/markdown
|
|
40
|
-
|
|
41
|
-
# MCP Rehearsal / Ghostlab
|
|
1
|
+
# Ghostlab
|
|
42
2
|
|
|
43
3
|
> A local, end-to-end **testing lab for any MCP server** — coding agents role-play
|
|
44
4
|
> real users, drive your tools over multiple turns, and the harness captures
|
|
45
5
|
> traces, scores outcomes, and even **renders and clicks through MCP Apps UI
|
|
46
6
|
> widgets**.
|
|
47
7
|
|
|
48
|
-
[](https://sajjadgg.github.io/
|
|
8
|
+
[](https://github.com/sajjadGG/Ghostlab/actions)
|
|
9
|
+
[](https://sajjadgg.github.io/Ghostlab/)
|
|
50
10
|
[](pyproject.toml)
|
|
51
11
|
[](LICENSE)
|
|
52
12
|
[](llms.txt)
|
|
@@ -58,12 +18,20 @@ user. Protocol-level checks (schema errors, a tool call that 500s) are useful
|
|
|
58
18
|
sanity checks, but they aren't the real test — the real test is whether an
|
|
59
19
|
agent can actually get a task done through your MCP, end to end.
|
|
60
20
|
|
|
61
|
-
📖 **Docs wiki:** https://sajjadgg.github.io/
|
|
21
|
+
📖 **Docs wiki:** https://sajjadgg.github.io/Ghostlab/ · 🤖 **For agents:** [`llms.txt`](llms.txt) · 🛠 **Contributing:** [`CONTRIBUTING.md`](CONTRIBUTING.md)
|
|
22
|
+
|
|
23
|
+
> **Naming:** the project and repo are **Ghostlab** (formerly *Rehearsal*). The CLI
|
|
24
|
+
> is `ghostlab`, with `rehearsal` kept as an alias, and the installed Python
|
|
25
|
+
> package is `rehearsal` — all the same project.
|
|
26
|
+
|
|
27
|
+
<p align="center">
|
|
28
|
+
<img src="static/ghostlablogo.png" width="400" alt="Logo" align="center">
|
|
29
|
+
</p>
|
|
62
30
|
|
|
63
31
|
## Quickstart
|
|
64
32
|
|
|
65
33
|
```bash
|
|
66
|
-
python3
|
|
34
|
+
python3 -m venv .venv # Python 3.10+
|
|
67
35
|
.venv/bin/pip install -e . # add '.[ui]' for the web UI, '.[apps]' for widget rendering
|
|
68
36
|
|
|
69
37
|
ghostlab create
|
|
@@ -114,6 +82,30 @@ generation/test defaults, gates — all editable), `test-plan.yaml`, `workspace/
|
|
|
114
82
|
| **Test** | Multi-host execution results (`results.json`/`results.md`), a standalone HTML dashboard, and — for conversational cases — full dual-agent transcripts with structured tool-call capture |
|
|
115
83
|
| **Review** | A readiness report: pass/fail gate verdict, failure clusters, and prioritized repairs |
|
|
116
84
|
|
|
85
|
+
## See it in action
|
|
86
|
+
|
|
87
|
+
**Watch a real coding-agent drive your MCP, turn by turn** — every tool call is
|
|
88
|
+
captured with its pass/fail status. Below is the live trace of a Hugging Face
|
|
89
|
+
MCP run, including two `hf_hub_query` calls that failed against the server:
|
|
90
|
+
|
|
91
|
+
<p align="center">
|
|
92
|
+
<img src="static/cli-run-trace.png" width="900" alt="CLI run trace showing per-tool-call status">
|
|
93
|
+
</p>
|
|
94
|
+
|
|
95
|
+
**Get a standalone HTML dashboard** — pass rate, per-case verdicts, and
|
|
96
|
+
suite/host tags at a glance:
|
|
97
|
+
|
|
98
|
+
<p align="center">
|
|
99
|
+
<img src="static/results-dashboard.png" width="760" alt="HTML results dashboard with pass-rate KPIs and per-case list">
|
|
100
|
+
</p>
|
|
101
|
+
|
|
102
|
+
**Drill into any case** — the goal and persona, the judge's verdict with its
|
|
103
|
+
reasoning, and the full dual-agent transcript with inline tool calls:
|
|
104
|
+
|
|
105
|
+
<p align="center">
|
|
106
|
+
<img src="static/case-report-transcript.png" width="760" alt="Per-case report with judge verdict and conversation transcript">
|
|
107
|
+
</p>
|
|
108
|
+
|
|
117
109
|
## Goal
|
|
118
110
|
|
|
119
111
|
Build a repeatable, sandboxed tester that can:
|
|
@@ -128,7 +120,7 @@ This lets you test with your existing Codex/Claude usage path, instead of wiring
|
|
|
128
120
|
|
|
129
121
|
## Scope
|
|
130
122
|
|
|
131
|
-
|
|
123
|
+
Ghostlab is intentionally **app-agnostic**:
|
|
132
124
|
|
|
133
125
|
- Works with any MCP server reachable by stdio/SSE/streamable HTTP.
|
|
134
126
|
- Supports local or remote MCP endpoints.
|
|
@@ -141,6 +133,23 @@ No Cortex-specific assumptions are required in the core harness.
|
|
|
141
133
|
Everything below is the individual-command reference and advanced usage —
|
|
142
134
|
useful once you're past the first `ghostlab create` run, or scripting CI.
|
|
143
135
|
|
|
136
|
+
### Spec vs job
|
|
137
|
+
|
|
138
|
+
There are two ways to hold an evaluation's config; **for almost everyone the
|
|
139
|
+
answer is a job.**
|
|
140
|
+
|
|
141
|
+
- **Job** (recommended) — a self-contained `jobs/<name>/` folder created by
|
|
142
|
+
`ghostlab create`. Every command takes `--job <name>` (or auto-detects
|
|
143
|
+
`job.yaml` in the current dir). This is the mainstream path the whole
|
|
144
|
+
Quickstart uses.
|
|
145
|
+
- **Spec** (advanced) — a single standalone `ghostlab.yaml` produced by
|
|
146
|
+
`ghostlab init`, addressed with `--spec <file>`. Useful for scripting or
|
|
147
|
+
keeping config outside the `jobs/` layout. Unless you specifically need that,
|
|
148
|
+
prefer a job.
|
|
149
|
+
|
|
150
|
+
The commands overlap (`discover`/`plan`/`test`/`review` accept either `--job` or
|
|
151
|
+
`--spec`); pick one model per evaluation and stay with it.
|
|
152
|
+
|
|
144
153
|
### Job folder layout
|
|
145
154
|
|
|
146
155
|
```text
|
|
@@ -179,7 +188,7 @@ Each test run points to a target definition:
|
|
|
179
188
|
The package installs two equivalent console scripts: `ghostlab` and `rehearsal`.
|
|
180
189
|
|
|
181
190
|
- `ghostlab create` — the end-to-end wizard described above.
|
|
182
|
-
- `ghostlab init` —
|
|
191
|
+
- `ghostlab init` — advanced: scaffold a standalone `ghostlab.yaml` **spec** from a target JSON (see [spec vs job](#spec-vs-job) — most users want `ghostlab create`).
|
|
183
192
|
- `ghostlab discover` — inspect the job's target, lint its contract, refresh capabilities.
|
|
184
193
|
- `ghostlab plan` — generate (or curate) the coverage-driven test plan.
|
|
185
194
|
- `ghostlab test` — execute the test plan across the job's host adapters.
|
|
@@ -190,6 +199,8 @@ The package installs two equivalent console scripts: `ghostlab` and `rehearsal`.
|
|
|
190
199
|
- `ghostlab review-dataset` / `run-dataset` — curate and run a standalone dataset.
|
|
191
200
|
- `ghostlab run` — run one dual-agent E2E scenario directly.
|
|
192
201
|
- `ghostlab evaluate` — score a run into a pass/fail verdict (codex judge).
|
|
202
|
+
- `ghostlab critique` — rate a run's tool ergonomics from the agent's perspective (codex).
|
|
203
|
+
- `ghostlab scorecard` — roll run verdicts and critiques into a summary scorecard.
|
|
193
204
|
- `ghostlab compare` — diff two dataset runs for regressions.
|
|
194
205
|
- `ghostlab apps-probe` / `apps-render` — probe/render MCP Apps `ui://` widgets.
|
|
195
206
|
- `ghostlab doctor` — check codex and validate runner presets.
|
|
@@ -218,7 +229,7 @@ Point it at a target and it introspects the server without any coding-agent
|
|
|
218
229
|
credits or manual `curl`:
|
|
219
230
|
|
|
220
231
|
```bash
|
|
221
|
-
ghostlab inspect --target
|
|
232
|
+
ghostlab inspect --target examples/target.json
|
|
222
233
|
```
|
|
223
234
|
|
|
224
235
|
This connects over the configured transport (stdio / streamable-HTTP / SSE),
|
|
@@ -279,7 +290,7 @@ confused, non-native, ...), and a domain `context` map (native_language,
|
|
|
279
290
|
target_exam, level, ...). Pass one to a run with `--persona`:
|
|
280
291
|
|
|
281
292
|
```bash
|
|
282
|
-
ghostlab run ... --persona
|
|
293
|
+
ghostlab run ... --persona persona.json
|
|
283
294
|
```
|
|
284
295
|
|
|
285
296
|
### Build a dataset: `generate-dataset`
|
|
@@ -329,7 +340,7 @@ ghostlab review-dataset --dataset datasets/cortex \
|
|
|
329
340
|
```bash
|
|
330
341
|
ghostlab run-dataset \
|
|
331
342
|
--dataset datasets/cortex \
|
|
332
|
-
--target
|
|
343
|
+
--target target.json \
|
|
333
344
|
--aut-runner runners/codex-cortex-aut.json \
|
|
334
345
|
--user-runner runners/codex-user-emulator.json \
|
|
335
346
|
--limit 2
|
|
@@ -379,7 +390,7 @@ points to a `ui://…` HTML widget a compatible host is expected to render.
|
|
|
379
390
|
`ui://` resource, and reports render-readiness and CSP diagnostics:
|
|
380
391
|
|
|
381
392
|
```bash
|
|
382
|
-
ghostlab apps-probe --target
|
|
393
|
+
ghostlab apps-probe --target target.json
|
|
383
394
|
```
|
|
384
395
|
|
|
385
396
|
`apps-render` actually renders a `ui://` widget in headless Chrome, proving a
|
|
@@ -389,7 +400,7 @@ real tool input/result, and can drive a sequence of UI intents:
|
|
|
389
400
|
|
|
390
401
|
```bash
|
|
391
402
|
pip install 'ghostlab[apps]' && playwright install chrome # one-time
|
|
392
|
-
ghostlab apps-render --target
|
|
403
|
+
ghostlab apps-render --target target.json \
|
|
393
404
|
--tool views_generate_sentence_scramble \
|
|
394
405
|
--arguments '{"target_sentence":"The cat sat on the mat","shuffled_elements":["mat","The","on","sat","cat","the"]}' \
|
|
395
406
|
--intent '{"type":"reorder","value":["The","cat","sat","on","the","mat"]}' \
|
|
@@ -409,7 +420,7 @@ one codex session alive: turn 1 records the `thread_id`, and later turns run
|
|
|
409
420
|
repeated cold-start noise.
|
|
410
421
|
|
|
411
422
|
```bash
|
|
412
|
-
ghostlab run --target
|
|
423
|
+
ghostlab run --target target.json --scenario <scenario.json> \
|
|
413
424
|
--aut-runner runners/codex-cortex-local-session.json --user-runner <user.json>
|
|
414
425
|
```
|
|
415
426
|
|
|
@@ -10,8 +10,8 @@ the project flow (`init` → `discover` → future `plan`/`test`/`review` stages
|
|
|
10
10
|
every other command keeps accepting raw target/scenario JSON as before.
|
|
11
11
|
|
|
12
12
|
```bash
|
|
13
|
-
ghostlab init --target
|
|
14
|
-
ghostlab init --target
|
|
13
|
+
ghostlab init --target target.json # writes ghostlab.yaml
|
|
14
|
+
ghostlab init --target target.json --out cortex.ghostlab.json
|
|
15
15
|
```
|
|
16
16
|
|
|
17
17
|
Options: `--name` (display name), `--workspace` (artifact directory, default
|
|
@@ -215,7 +215,7 @@ Writes `readiness.json` / `readiness.md` next to the test results:
|
|
|
215
215
|
Introspect a target MCP server.
|
|
216
216
|
|
|
217
217
|
```bash
|
|
218
|
-
ghostlab inspect --target
|
|
218
|
+
ghostlab inspect --target examples/target.json
|
|
219
219
|
```
|
|
220
220
|
|
|
221
221
|
## profile
|
|
@@ -282,8 +282,8 @@ Run one scenario.
|
|
|
282
282
|
|
|
283
283
|
```bash
|
|
284
284
|
ghostlab run \
|
|
285
|
-
--target
|
|
286
|
-
--scenario
|
|
285
|
+
--target examples/target.json \
|
|
286
|
+
--scenario examples/scenario.json \
|
|
287
287
|
--aut-runner runners/mock-aut.json \
|
|
288
288
|
--user-runner runners/mock-user.json
|
|
289
289
|
```
|
|
@@ -295,7 +295,7 @@ Run every case in a dataset. Use `--limit` for small development runs and `--app
|
|
|
295
295
|
```bash
|
|
296
296
|
ghostlab run-dataset \
|
|
297
297
|
--dataset datasets/cortex \
|
|
298
|
-
--target
|
|
298
|
+
--target target.json \
|
|
299
299
|
--aut-runner runners/codex-cortex-aut.json \
|
|
300
300
|
--user-runner runners/codex-user-emulator.json \
|
|
301
301
|
--limit 2
|
|
@@ -76,7 +76,7 @@ run contains a call to that tool whose arguments include the given key/value pai
|
|
|
76
76
|
|
|
77
77
|
```bash
|
|
78
78
|
ghostlab run-dataset --dataset datasets/cortex \
|
|
79
|
-
--target
|
|
79
|
+
--target target.json \
|
|
80
80
|
--aut-runner runners/codex-cortex-local-session.json \
|
|
81
81
|
--evaluate --capabilities runs/<id>-inspect/capabilities.json
|
|
82
82
|
```
|
|
@@ -3,11 +3,11 @@
|
|
|
3
3
|
## Install From A Checkout
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
|
-
python3
|
|
7
|
-
.venv/bin/pip install -
|
|
6
|
+
python3 -m venv .venv # Python 3.10+
|
|
7
|
+
.venv/bin/pip install -e .
|
|
8
8
|
```
|
|
9
9
|
|
|
10
|
-
This installs the package in editable mode and adds the `ghostlab` and `rehearsal` console scripts. Prefer `ghostlab` in new docs and scripts.
|
|
10
|
+
This installs the package in editable mode and adds the `ghostlab` and `rehearsal` console scripts. Prefer `ghostlab` in new docs and scripts. (Contributors who also want the test/build/docs toolchain use `pip install -r requirements-dev.txt` instead — see `CONTRIBUTING.md`.)
|
|
11
11
|
|
|
12
12
|
```bash
|
|
13
13
|
ghostlab --help
|
|
@@ -95,8 +95,8 @@ Mock runners let you exercise the orchestrator without spending coding-agent cre
|
|
|
95
95
|
|
|
96
96
|
```bash
|
|
97
97
|
ghostlab run \
|
|
98
|
-
--target
|
|
99
|
-
--scenario
|
|
98
|
+
--target examples/target.json \
|
|
99
|
+
--scenario examples/scenario.json \
|
|
100
100
|
--aut-runner runners/mock-aut.json \
|
|
101
101
|
--user-runner runners/mock-user.json
|
|
102
102
|
```
|
|
@@ -110,7 +110,7 @@ Run output is written under `runs/<run-id>/`:
|
|
|
110
110
|
## Inspect A Real MCP Target
|
|
111
111
|
|
|
112
112
|
```bash
|
|
113
|
-
ghostlab inspect --target
|
|
113
|
+
ghostlab inspect --target examples/target.json
|
|
114
114
|
# or a standard MCP client config:
|
|
115
115
|
ghostlab inspect --target ./mcp.json --server obsidian
|
|
116
116
|
```
|
|
@@ -283,7 +283,7 @@ review:
|
|
|
283
283
|
This artifact becomes the center of the system. All commands can still exist, but a new top-level flow should be:
|
|
284
284
|
|
|
285
285
|
```bash
|
|
286
|
-
ghostlab init --target
|
|
286
|
+
ghostlab init --target target.json --out ghostlab.yaml
|
|
287
287
|
ghostlab discover --spec ghostlab.yaml
|
|
288
288
|
ghostlab plan --spec ghostlab.yaml
|
|
289
289
|
ghostlab test --spec ghostlab.yaml --suite all --hosts codex-cli,claude-code
|
|
@@ -16,11 +16,11 @@ Use it when you want to test an MCP server through the same Codex or Claude Code
|
|
|
16
16
|
## Common Flow
|
|
17
17
|
|
|
18
18
|
```bash
|
|
19
|
-
ghostlab inspect --target
|
|
19
|
+
ghostlab inspect --target examples/target.json
|
|
20
20
|
ghostlab profile --inspect runs/<id>-inspect/inspect.json
|
|
21
21
|
ghostlab generate-dataset --profile runs/<id>-inspect/capabilities.json --name cortex
|
|
22
22
|
ghostlab review-dataset --dataset datasets/cortex --profile runs/<id>-inspect/capabilities.json
|
|
23
|
-
ghostlab run-dataset --dataset datasets/cortex --target
|
|
23
|
+
ghostlab run-dataset --dataset datasets/cortex --target target.json --approved-only
|
|
24
24
|
```
|
|
25
25
|
|
|
26
26
|
Start with [Getting Started](getting-started.md), then use the [CLI Reference](cli.md) for the full command map.
|
|
@@ -14,7 +14,7 @@ The `Release` workflow publishes the `ghostlab` package to PyPI when a `v*.*.*`
|
|
|
14
14
|
|
|
15
15
|
PyPI publishing uses Trusted Publishing. Create or claim the `ghostlab` project on PyPI and add a trusted publisher for:
|
|
16
16
|
|
|
17
|
-
- Repository: `sajjadGG/
|
|
17
|
+
- Repository: `sajjadGG/Ghostlab`
|
|
18
18
|
- Workflow: `.github/workflows/release.yml`
|
|
19
19
|
- Environment: `pypi`
|
|
20
20
|
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Examples
|
|
2
|
+
|
|
3
|
+
Ready-to-run sample configs so you can try Ghostlab without writing any JSON
|
|
4
|
+
first. They point at the public Hugging Face MCP (`https://huggingface.co/mcp`),
|
|
5
|
+
which can be **introspected without a token**, so the read-only commands below
|
|
6
|
+
work with zero setup.
|
|
7
|
+
|
|
8
|
+
| File | Shape | Used by |
|
|
9
|
+
| --- | --- | --- |
|
|
10
|
+
| `target.json` | Native Ghostlab target (`id` + `transport` + `connection`) | `ghostlab inspect`, `ghostlab run` |
|
|
11
|
+
| `scenario.json` | A dual-agent scenario (persona, goal, success/failure criteria) | `ghostlab run` |
|
|
12
|
+
| `mcp-config.json` | Standard `mcpServers` client config | `ghostlab create`, `ghostlab inspect` |
|
|
13
|
+
|
|
14
|
+
## Try it
|
|
15
|
+
|
|
16
|
+
Introspect the target with no auth (writes `inspect.json` + `inspect.md`):
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
ghostlab inspect --target examples/target.json
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Run the mock dual-agent scenario end to end — no coding-agent credits, no
|
|
23
|
+
network to the target (the mock runners don't call it):
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
ghostlab run \
|
|
27
|
+
--target examples/target.json \
|
|
28
|
+
--scenario examples/scenario.json \
|
|
29
|
+
--aut-runner runners/mock-aut.json \
|
|
30
|
+
--user-runner runners/mock-user.json
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Scaffold a job straight from a standard `mcpServers` config:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
ghostlab create --target examples/mcp-config.json
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## Adding auth
|
|
40
|
+
|
|
41
|
+
Most Hugging Face tools (beyond introspection) need a token. Add a bearer header
|
|
42
|
+
to either config and export the referenced env var — the value is an env-var
|
|
43
|
+
*reference*, never a literal secret in the file:
|
|
44
|
+
|
|
45
|
+
```json
|
|
46
|
+
"headers": { "Authorization": "Bearer ${HF_TOKEN}" }
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
export HF_TOKEN=hf_... # your token, kept out of the config file
|
|
51
|
+
```
|