@roleplay-sh/cli 0.1.5 → 0.1.7

package/.env.example

@@ -1,7 +1,7 @@
-# Optional agent credentials used by your own HTTP/CLI target.
+# Agent credentials used by your own HTTP/CLI target.
 AGENT_API_KEY=
 
-# Cloud workbench upload settings. Requires a trial workspace and project API key.
+# Workbench project settings. Create these after starting a Builder or Team trial.
 ROLEPLAY_CLOUD_URL=https://app.roleplay.sh
 ROLEPLAY_PROJECT_ID=
 ROLEPLAY_API_KEY=
@@ -11,14 +11,17 @@ ROLEPLAY_AGENT_NAME=
 ROLEPLAY_TARGET_URL=http://localhost:3000/agent
 ROLEPLAY_TARGET_COMMAND=
 
-# Optional LLM provider settings for adaptive attacker turns and semantic judging.
-# Provider choices: mock, openai, anthropic, google, openai-compatible.
-ROLEPLAY_LLM_PROVIDER=mock
+# Adaptive attacker and judge configuration.
+# Provider choices: openai, anthropic, google, openai-compatible.
+ROLEPLAY_LLM_PROVIDER=<provider>
 ROLEPLAY_LLM_MODEL=
+ROLEPLAY_JUDGE_MODE=semantic
+ROLEPLAY_JUDGE_PROVIDER=<provider>
+ROLEPLAY_JUDGE_MODEL=
 ROLEPLAY_ATTACKER_PROVIDER=
 ROLEPLAY_ATTACKER_MODEL=
-ROLEPLAY_JUDGE_PROVIDER=
-ROLEPLAY_JUDGE_MODEL=
+
+# Provider API keys. Set only the one you use; do not commit real secrets.
 ROLEPLAY_OPENAI_API_KEY=
 ROLEPLAY_ANTHROPIC_API_KEY=
 ROLEPLAY_GOOGLE_API_KEY=

package/CHANGELOG.md

@@ -4,11 +4,32 @@ All notable changes to roleplay.sh will be documented in this file.
 
 This project follows semantic versioning after the public `0.1.0` release.
 
-## 0.1.4 - Unreleased
+## 0.1.7 - Unreleased
+
+### Added
+
+- Guided `roleplay setup` for Workbench project, target, provider, and judge configuration.
+- Explicit judge modes: `rules`, `semantic`, and `hybrid`.
+- Command-specific help for `run`, `doctor`, and `setup`.
+- Judge metadata in saved reports so users can see how evidence was evaluated.
+
+### Changed
+
+- Real targets now require an explicit provider and judge choice instead of silently defaulting to a named provider.
+- Public README and release copy now present roleplay.sh as a provider-neutral Workbench runner.
+- `doctor` now separates attacker provider readiness, judge readiness, entitlement, and upload readiness.
+
+## 0.1.6 - 2026-06-14
+
+### Changed
+
+- Aligned CLI copy with the paid roleplay.sh Workbench model.
+
+## 0.1.4 - 2026-06-14
 
 ### Changed
 
-- Updated CLI upload, doctor, and setup copy for the paid roleplay.sh cloud workbench.
+- Updated CLI upload, doctor, and setup copy for the paid roleplay.sh Workbench.
 - Clarified that production uploads require a Builder or Team trial, project API key, and sanitized upload policy.
 - Kept public command syntax stable while preserving mock smoke tests and BYO provider usage for real runs.
 
@@ -16,14 +37,14 @@ This project follows semantic versioning after the public `0.1.0` release.
 
 ### Added
 
-- Adaptive LLM attacker providers for OpenAI, Anthropic, Google Gemini, and OpenAI-compatible APIs.
+- Adaptive attacker providers for OpenAI, Anthropic, Google Gemini, and OpenAI-compatible APIs.
 - LLM transcript judging against scenario success and failure criteria.
 - `--provider`, `--attacker-provider`, `--judge-provider`, model, and OpenAI-compatible base URL flags.
 - Scenario YAML support for attacker and judge provider settings.
 
 ### Changed
 
-- Real HTTP and CLI targets default to LLM provider mode for `social-engineering-core`.
+- Real HTTP and CLI targets use provider-backed mode for `social-engineering-core`.
 - Mock mode remains available as an explicit deterministic smoke-test path with `--target mock --provider mock`.
 
 ## 0.1.2 - 2026-06-03
@@ -39,7 +60,7 @@ This project follows semantic versioning after the public `0.1.0` release.
 - Dedicated public CLI package for local attack-pack execution.
 - Built-in `social-engineering-core` attack pack.
 - Local reports and replayable transcripts.
-- Sanitized cloud workbench upload support.
+- Sanitized workbench upload support.
 
 ## 0.1.0 - 2026-05-17
 

package/CONTRIBUTING.md

@@ -11,7 +11,13 @@ pnpm test
 pnpm build
 ```
 
-Use local attack-pack execution for tests and examples. External model-provider behavior is now part of the public CLI; keep provider additions explicit, tested, and documented.
+Use local attack-pack execution for tests and examples. External provider behavior is part of the public CLI; keep provider additions explicit, tested, documented, and vendor-neutral in user-facing examples.
+
+Judge changes must preserve all three user-facing modes:
+
+- `rules` for deterministic smoke/offline checks.
+- `semantic` for provider-backed security evaluation.
+- `hybrid` for semantic evaluation plus deterministic guardrails.
 
 ## Pull requests
 

package/README.md

@@ -1,8 +1,8 @@
 # roleplay.sh CLI
 
-Social-engineering regression tests for AI agents.
+Included local runner for roleplay.sh social-engineering tests.
 
-`roleplay` runs adversarial roleplay scenarios against local, HTTP, CLI, or mock agents, records replayable evidence, and uploads sanitized findings to the roleplay.sh cloud workbench.
+`roleplay` runs attack packs against your local, HTTP, CLI, or mock AI agent target, saves replayable evidence, and uploads sanitized proof to the roleplay.sh Workbench.
 
 ## Install
 
@@ -16,23 +16,30 @@ Or run without installing:
 npx @roleplay-sh/cli --help
 ```
 
-## Quickstart
+## Smoke Test Only
+
+Use mock mode to confirm the CLI is installed and can save local evidence. This does not test a real agent.
 
 ```bash
 roleplay init
-roleplay run social-engineering-core --target mock --provider mock --fail-on critical
+roleplay run social-engineering-core --target mock --provider mock --judge rules --fail-on critical
 roleplay report latest
 roleplay replay latest
 ```
 
-## Test A Local Agent
+## Run A Real Local Test
+
+Start a Builder or Team trial in the roleplay.sh Workbench, create a project API key, choose your provider, choose how results should be judged, then run the included local runner against your agent.
 
 HTTP target:
 
 ```bash
 roleplay run social-engineering-core \
   --target http://localhost:3000/agent \
-  --provider openai \
+  --provider <provider> \
+  --judge semantic \
+  --project <project-id> \
+  --api-key <project-api-key> \
   --fail-on critical
 ```
 
@@ -41,22 +48,49 @@ CLI target:
 ```bash
 roleplay run social-engineering-core \
   --target-command "node ./agent.js" \
-  --provider openai \
+  --provider <provider> \
+  --judge hybrid \
+  --project <project-id> \
+  --api-key <project-api-key> \
   --fail-on critical \
   --yes
 ```
 
-Set the provider API key before running a real attack pack:
+## Judge Choices
+
+- `--judge rules`: deterministic local rule judge. Best for smoke tests and offline checks.
+- `--judge semantic`: provider-backed security judge. Recommended for real agent tests.
+- `--judge hybrid`: semantic judge plus deterministic guardrails. Recommended for CI once your provider is configured.
+
+Rules-only judging can be used against real targets only with `--allow-rules-only`, so it is never mistaken for full semantic evaluation.
+
+## Provider Configuration
+
+roleplay.sh is provider-neutral. Pick the provider you want to use for adaptive attacker turns and semantic judging.
 
 ```bash
-export ROLEPLAY_OPENAI_API_KEY="your-openai-key"
+export ROLEPLAY_PROJECT_ID="<project-id>"
+export ROLEPLAY_API_KEY="<project-api-key>"
+export ROLEPLAY_LLM_PROVIDER="<provider>"
+export ROLEPLAY_JUDGE_MODE="hybrid"
+export ROLEPLAY_JUDGE_PROVIDER="<provider>"
+export ROLEPLAY_<PROVIDER>_API_KEY="your-provider-key"
 ```
 
-Supported providers are `openai`, `anthropic`, `google`, and `openai-compatible`. Use `--attacker-provider` and `--judge-provider` when you want different providers for adaptive attacker turns and transcript judging. Use `--target mock --provider mock` for deterministic local smoke tests.
+Supported provider identifiers: `openai`, `anthropic`, `google`, and `openai-compatible`.
 
-## Upload Sanitized Findings To The Cloud Workbench
+Use `--attacker-provider` and `--judge-provider` when you want different providers for attacker turns and transcript judging.
 
-Start a Builder or Team trial at `https://app.roleplay.sh`, create a workspace project and API key, then run:
+## Guided Setup
+
+```bash
+roleplay setup
+roleplay doctor --cloud
+```
+
+`roleplay setup` writes safe placeholders to `.env.example`. It does not store raw provider or Workbench API keys by default.
+
+## Upload Sanitized Proof
 
 ```bash
 ROLEPLAY_CLOUD_URL=https://app.roleplay.sh \
@@ -69,26 +103,31 @@ Sanitized upload is the default. Full transcripts, raw scenario YAML, and local
 
 ## Commands
 
+- `roleplay setup` guides Workbench and local runner setup.
 - `roleplay init` creates local config and starter scenarios.
 - `roleplay run` runs a scenario file or built-in attack pack.
 - `roleplay report` prints a saved run report.
 - `roleplay replay` replays transcript evidence.
-- `roleplay upload` uploads sanitized findings to the roleplay.sh cloud workbench.
+- `roleplay upload` uploads sanitized findings to the Workbench.
 - `roleplay list` lists local runs.
-- `roleplay doctor` checks local and Cloud configuration.
+- `roleplay doctor` checks install, Workbench, provider, judge, and upload readiness.
 - `roleplay mcp` exposes roleplay.sh through MCP.
 
 ## CI Example
 
 ```yaml
 - name: Run roleplay.sh attack pack
-  run: pnpm dlx @roleplay-sh/cli run social-engineering-core --fail-on critical
+  run: pnpm dlx @roleplay-sh/cli run social-engineering-core --judge hybrid --fail-on critical
   env:
     ROLEPLAY_TARGET_URL: ${{ secrets.ROLEPLAY_TARGET_URL }}
-    ROLEPLAY_LLM_PROVIDER: openai
-    ROLEPLAY_OPENAI_API_KEY: ${{ secrets.ROLEPLAY_OPENAI_API_KEY }}
+    ROLEPLAY_PROJECT_ID: ${{ secrets.ROLEPLAY_PROJECT_ID }}
+    ROLEPLAY_API_KEY: ${{ secrets.ROLEPLAY_API_KEY }}
+    ROLEPLAY_LLM_PROVIDER: ${{ secrets.ROLEPLAY_LLM_PROVIDER }}
+    ROLEPLAY_JUDGE_MODE: hybrid
+    ROLEPLAY_JUDGE_PROVIDER: ${{ secrets.ROLEPLAY_JUDGE_PROVIDER }}
+    ROLEPLAY_LLM_API_KEY: ${{ secrets.ROLEPLAY_LLM_API_KEY }}
 
-- name: Upload sanitized findings
+- name: Upload sanitized proof
   if: always()
   run: pnpm dlx @roleplay-sh/cli upload all --source ci --mode sanitized_findings
   env:

package/RELEASE.md

@@ -29,8 +29,8 @@ The publish workflow uses GitHub OIDC and intentionally does not require an npm
 Create a GitHub release or push a version tag:
 
 ```bash
-git tag v0.1.4
-git push origin v0.1.4
+git tag v0.1.7
+git push origin v0.1.7
 ```
 
 The publish workflow runs checks and then publishes with:
@@ -46,19 +46,24 @@ npm view @roleplay-sh/cli version
 npm install -g @roleplay-sh/cli
 roleplay --help
 roleplay init
-roleplay run social-engineering-core --target mock --provider mock --fail-on critical
+roleplay run social-engineering-core --target mock --provider mock --judge rules --fail-on critical
 roleplay report latest
 roleplay replay latest
 ```
 
-For real LLM-backed verification:
+For real provider-backed verification:
 
 ```bash
-export ROLEPLAY_OPENAI_API_KEY=<openai-key>
-roleplay run social-engineering-core --target http://localhost:3000/agent --provider openai --max-turns 1 --fail-on critical
+export ROLEPLAY_PROJECT_ID=<project-id>
+export ROLEPLAY_API_KEY=<project-api-key>
+export ROLEPLAY_LLM_PROVIDER=<provider>
+export ROLEPLAY_JUDGE_MODE=semantic
+export ROLEPLAY_JUDGE_PROVIDER=<provider>
+export ROLEPLAY_<PROVIDER>_API_KEY=<provider-key>
+roleplay run social-engineering-core --target http://localhost:3000/agent --provider <provider> --judge semantic --max-turns 1 --fail-on critical
 ```
 
-For cloud workbench upload verification, start a Builder or Team trial, create a project API key at `https://app.roleplay.sh`, and run:
+For workbench upload verification, start a Builder or Team trial, create a project API key at `https://app.roleplay.sh`, and run:
 
 ```bash
 ROLEPLAY_CLOUD_URL=https://app.roleplay.sh \

package/SECURITY.md

@@ -12,7 +12,9 @@ Do not include real API keys, customer data, private prompts, transcripts, or pr
 
 ## Data Handling
 
-roleplay.sh stores runs locally under `.roleplay/runs`. Scenario files, hidden context, transcripts, and reports may contain sensitive information. Full transcripts stay local unless you explicitly upload them to the cloud workbench with full-transcript mode enabled in both the project policy and the CLI command.
+roleplay.sh stores runs locally under `.roleplay/runs`. Scenario files, hidden context, transcripts, and reports may contain sensitive information. Full transcripts stay local unless you explicitly upload them to the workbench with full-transcript mode enabled in both the project policy and the CLI command.
+
+Provider API keys should stay in your local environment or CI secret store. `roleplay setup` writes placeholders only and does not store raw provider or Workbench API keys by default.
 
 ## CLI Target Execution