@kestrel-agents/ruhroh 0.5.0-beta.0 → 0.5.0-beta.1

package/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+## 0.5.0-beta.1
+
+- Reworked the README as a consumer-focused introduction.
+- Added getting-started, scenario-authoring, and adapter-authoring guides.
+- Added lightweight public contribution, security, and issue-routing docs.
+- Added a VitePress documentation site for GitHub Pages.
+- Kept package APIs and runtime behavior unchanged from `0.5.0-beta.0`.
+
+## 0.5.0-beta.0
+
+- First public beta of Ruhroh.
+- Published JSON scenario discovery, validation, and Harbor task generation.
+- Published package-owned Python Harbor runtime support for command adapters.
+- Shipped bundled scenarios, public examples, logo assets, docs, and smoke CI.
+- Kept generated verifiers app-agnostic and live agent runs optional/manual.

package/README.md

@@ -6,12 +6,27 @@
 
 Ruhroh is the **Real-User Harness for Repair-Oriented Harbor**.
 
-Ruhroh runs real-user task scenarios against coding agents through adapters,
-preserves the full implementation journey, and runs a terminal evaluator over
-the final delivered workspace.
+It runs realistic software tasks against coding agents, preserves the full
+implementation journey, and judges the final delivered workspace through a
+terminal evaluator.
 
-Ruhroh is agent-agnostic. Kestrel is one reference run-agent adapter, not the
-benchmark itself. Harbor is the execution substrate.
+Ruhroh exists because most agent benchmarks are either too static or too easy to
+overfit. Real users do not ask for a required filename or a magic route; they ask
+for an outcome, watch the agent iterate, and care whether the finished workspace
+actually works. Ruhroh packages that loop into repeatable Harbor tasks while
+keeping the benchmark itself agent-agnostic.
+
+Use Ruhroh when you want to:
+
+- turn product-like user requests into repeatable agent tasks;
+- compare agents or prompts on delivered outcomes, not source-text heuristics;
+- preserve transcripts, intermediate attempts, final workspaces, and eval
+  judgments for review;
+- generate Harbor-compatible task directories from portable JSON scenarios.
+
+Ruhroh is not a native agent runner. You bring the run-agent adapter. The public
+package ships the scenario format, generator, CLI, result contracts, and a
+package-owned Python Harbor runtime for command-backed adapters.
 
 ## Install
 
@@ -19,37 +34,70 @@ benchmark itself. Harbor is the execution substrate.
 pnpm add -D @kestrel-agents/ruhroh
 ```
 
-## Quickstart
+## Quickstart: Inspect and Generate Tasks
+
+Ruhroh ships bundled scenarios so you can inspect the package before wiring a
+live agent:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --list
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --generate-only
+```
+
+In this repository, build first and use the local CLI output:
+
+```bash
+pnpm build
+node dist/cli.js --scenario-dir examples/scenarios --list
+node dist/cli.js --scenario-dir examples/scenarios --scenario simple-newsletter --generate-only
+```
+
+Generated Harbor task directories are written under:
+
+```text
+.generated/ruhroh/harbor/tasks/<scenario-id>/
+```
 
-Create scenarios under `ruhroh/scenarios/<id>/`, or use the bundled scenarios
-under `node_modules/@kestrel-agents/ruhroh/scenarios`, then run:
+Use `--dry-run` to see the Harbor command without starting a benchmark:
 
 ```bash
-pnpm ruhroh --list
-pnpm ruhroh --scenario simple-newsletter --generate-only
-pnpm ruhroh --scenario simple-newsletter --adapter ./path/to/agent-adapter --dry-run
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --adapter custom-shell --dry-run
 ```
 
-In this repo, the same package CLI is available with:
+## Run an Agent
+
+Ruhroh selects agents at runtime. For shell-based agents, pass a command path as
+the adapter:
 
 ```bash
-pnpm ruhroh --scenario-dir examples/scenarios --list
-pnpm ruhroh --scenario-dir examples/scenarios --scenario simple-newsletter --generate-only
+pnpm exec ruhroh \
+  --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios \
+  --scenario simple-newsletter \
+  --adapter ./path/to/agent-wrapper.sh
 ```
 
-This package currently contains the portable TypeScript surfaces:
+When the adapter value looks like a command or path, the CLI wires it through
+`RUHROH_RUN_AGENT_COMMAND` for the package runtime. The command receives the
+workspace, goal, iteration metadata, and result path. When the goal is satisfied,
+it should exit successfully and emit the completion signal described in
+[`docs/custom-shell.md`](docs/custom-shell.md).
+
+Use `custom-shell` directly when you want to provide the command through the
+environment:
+
+```bash
+export RUHROH_RUN_AGENT_COMMAND=./path/to/agent-wrapper.sh
+export RUHROH_RUN_AGENT_COMPLETION_PROTOCOL=json-final-line
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --adapter custom-shell
+```
 
-- scenario schema and validation;
-- run-agent adapter interfaces and capability compatibility helpers;
-- eval and final result types;
-- verdict mapping helpers;
-- env forwarding and redaction helpers;
-- Harbor command construction helpers;
-- JSON scenario discovery and Harbor task generation helpers.
+Live agent runs require whatever credentials that agent needs. Default CI and
+package smoke tests should stay credential-free and use `--dry-run` or fixture
+evals.
 
-## Scenario Generation
+## Write Scenarios
 
-Ruhroh can load JSON scenarios from:
+Create scenarios under `ruhroh/scenarios/<id>/`:
 
 ```text
 ruhroh/scenarios/<id>/
@@ -58,15 +106,66 @@ ruhroh/scenarios/<id>/
   assets/
 ```
 
-and generate local Harbor task directories under:
+The scenario JSON names the user prompt, runtime requirements, loop settings,
+and evaluation rubric. Scenario prompts should read like real user requests:
+describe the desired outcome, relevant context, and success criteria. Avoid
+encoding implementation shortcuts such as "create exactly this file" unless that
+is genuinely part of the user request.
 
-```text
-.generated/ruhroh/harbor/tasks/<scenario-id>/
-```
+Good scenarios usually include:
+
+- a concrete user goal;
+- constraints the agent must respect;
+- assets or seed data needed by the task;
+- a rubric that tells the evaluator how to judge the final workspace;
+- evidence guidance for transcripts, logs, commands, screenshots, or generated
+  files.
+
+See the [scenario format guide](https://lumicorp.github.io/ruhroh/scenario-format)
+for the full schema.
+
+## How Judging Works
+
+Ruhroh intentionally keeps generated Harbor verifiers app-agnostic. The verifier
+checks the structured Ruhroh result and reward mapping; it does not inspect
+source text, required filenames, routes, or hard-coded commands.
+
+The evaluation boundary is:
+
+1. the run-agent iterates in a benchmark workspace;
+2. Ruhroh preserves iteration records, transcripts, event logs, and the final
+   workspace snapshot;
+3. a terminal evaluator reviews the final delivered workspace and journey
+   evidence;
+4. the generic Harbor verifier maps the structured result to reward.
 
-The generated Harbor verifier is app-agnostic. It only validates that the
-structured Ruhroh result exists and maps to a passing score/reward; it does not
-perform required-file, route, command, or source-text checks.
+This separation is the point: scenario-specific judgment belongs in the eval
+rubric and evaluator, not in brittle generator logic.
+
+Core artifacts include:
+
+- `ruhroh-loop-result.json`
+- `ruhroh-loop-iterations.jsonl`
+- `ruhroh-loop-journey.json`
+- `ruhroh-loop-eval.json`
+- `ruhroh-workspace.tar.gz`
+
+See the [artifacts guide](https://lumicorp.github.io/ruhroh/artifacts) for the
+complete artifact list.
+
+## Use It Well
+
+- Start with smoke-tier scenarios that are small but realistic.
+- Keep scenarios agent-agnostic; select adapters at runtime.
+- Prefer outcome rubrics over file-name or source-text checks.
+- Treat prompts and assets as untrusted input, and run agents only in benchmark
+  workspaces.
+- Keep live model credentials out of default CI.
+- Review preserved artifacts when a score looks surprising; the journey often
+  explains whether the failure is an agent issue, adapter issue, or evaluator
+  issue.
+
+## Public API
 
 The public API exports:
 
@@ -75,35 +174,28 @@ The public API exports:
 - `generateHarborTask()`
 - `generateHarborDataset()`
 
-The package CLI exposes this through:
-
-```bash
-pnpm ruhroh --scenario-dir ruhroh/scenarios --generate-only
-pnpm ruhroh --scenario-dir ruhroh/scenarios --dry-run
-```
-
-This package ships the reusable scenario source under `scenarios/` and the
-package-owned Python Harbor runtime under `python/ruhroh`. Run-agents are wired
-into this runtime as command adapters through
-`RUHROH_RUN_AGENT_COMMAND`; terminal evaluation can be supplied through
-`RUHROH_EVAL_COMMAND` or fixture eval variables.
+It also exports TypeScript contracts for scenarios, adapters, results, verdict
+mapping, env forwarding/redaction, and Harbor command construction.
 
-Kestrel is a consumer adapter, not a Ruhroh package dependency. The Harbor
-harness is package-owned for generated Ruhroh tasks.
+Kestrel is one consumer adapter, not a Ruhroh package dependency. Harbor is the
+execution substrate.
 
 ## Docs
 
-- Architecture: `docs/architecture.md`
-- Scenario format: `docs/scenario-format.md`
-- Adapter protocol: `docs/adapter-protocol.md`
-- Custom-shell adapter: `docs/custom-shell.md`
-- Harbor: `docs/harbor.md`
-- Eval-agent: `docs/eval-agent.md`
-- Artifacts: `docs/artifacts.md`
-- CI: `docs/ci.md`
-- Security: `docs/security.md`
-- Limitations: `docs/limitations.md`
-- Public repo layout: `docs/public-repo-layout.md`
+- Getting started: <https://lumicorp.github.io/ruhroh/getting-started>
+- Write a scenario: <https://lumicorp.github.io/ruhroh/write-a-scenario>
+- Write an adapter: <https://lumicorp.github.io/ruhroh/write-an-adapter>
+- Architecture: <https://lumicorp.github.io/ruhroh/architecture>
+- Scenario format: <https://lumicorp.github.io/ruhroh/scenario-format>
+- Adapter protocol: <https://lumicorp.github.io/ruhroh/adapter-protocol>
+- Custom-shell adapter: <https://lumicorp.github.io/ruhroh/custom-shell>
+- Harbor: <https://lumicorp.github.io/ruhroh/harbor>
+- Eval-agent: <https://lumicorp.github.io/ruhroh/eval-agent>
+- Artifacts: <https://lumicorp.github.io/ruhroh/artifacts>
+- CI: <https://lumicorp.github.io/ruhroh/ci>
+- Security: <https://lumicorp.github.io/ruhroh/security>
+- Limitations: <https://lumicorp.github.io/ruhroh/limitations>
+- Public repo layout: <https://lumicorp.github.io/ruhroh/public-repo-layout>
 
 ## Security
 

package/docs/adapter-protocol.md

@@ -0,0 +1,56 @@
+---
+id: ruhroh-adapter-protocol
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - src/adapters.ts
+  - python/ruhroh/loop_controller.py
+---
+
+# Adapter Protocol
+
+Run-agent adapters own agent-specific behavior. Ruhroh core owns orchestration
+and result mapping.
+
+The TypeScript adapter contract is exported from `@kestrel-agents/ruhroh`:
+
+- `prepare()`
+- `startSession()`
+- `runTurn()`
+- `detectCompletion()`
+- `collectArtifacts()`
+- `cleanup()`
+
+Adapters report their continuity level:
+
+- `native_session`
+- `workspace_plus_transcript`
+- `workspace_only`
+
+For shell-based public agents, use the generic command adapter protocol. The
+current command adapter passes:
+
+- `RUHROH_MESSAGE`
+- `RUHROH_ITERATION`
+- `RUHROH_WORKSPACE`
+- `RUHROH_GOAL_PATH`
+- `RUHROH_WORKSPACE_PATH`
+- `RUHROH_RESULT_PATH`
+- `RUHROH_SESSION_HANDLE`
+- `RUHROH_SCENARIO_ID`
+- `RUHROH_RUN_ROOT`
+- `RUHROH_ADAPTER_ID`
+
+Wrappers should emit a final JSON line:
+
+```json
+{"status":"goal_satisfied"}
+```
+
+Wrappers may also write `RUHROH_RESULT_PATH` with
+`version: "ruhroh_run_agent_result_v1"`. The adapter reads that file when
+present and maps `goal_satisfied`, `continue`, `cannot_satisfy`,
+`policy_blocked`, `runtime_failure`, and `infra_failure` into the generic
+completion contract.

package/docs/architecture.md

@@ -0,0 +1,46 @@
+---
+id: ruhroh-architecture
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - src/index.ts
+  - python/ruhroh/loop_controller.py
+---
+
+# Ruhroh Architecture
+
+Ruhroh is the Real-User Harness for Repair-Oriented Harbor. It runs real-user
+task scenarios against coding agents through adapters, preserves the full
+implementation journey, and runs a terminal evaluator over the final delivered
+workspace.
+
+## Components
+
+- Ruhroh core: scenario discovery, scenario validation, Harbor task generation,
+  Harbor command construction, artifact naming, result typing, and verdict
+  mapping.
+- Package Harbor runtime: the installable Python controller used for portable
+  custom-shell benchmarks.
+- Run-agent adapter: the agent-specific bridge that starts or continues a
+  coding agent in a benchmark workspace.
+- Harbor: the execution substrate that installs the benchmark agent, runs the
+  generated task, collects artifacts, and reads verifier reward output.
+- Eval-agent: the terminal evaluator that inspects a copied final workspace and
+  journey evidence after implementation is complete.
+
+Kestrel is one reference run-agent adapter. It is not the benchmark itself.
+
+## Lifecycle
+
+1. Discover JSON scenarios.
+2. Generate Harbor task directories under `.generated/ruhroh/harbor/tasks`.
+3. Run Harbor against the selected task.
+4. The installed Ruhroh controller asks the selected run-agent adapter to work.
+5. The adapter continues until it reports `goal_satisfied` or the iteration cap
+   is reached.
+6. The eval-agent reviews the full journey once.
+7. The generic Harbor verifier maps the structured Ruhroh result to reward.
+
+Ruhroh core does not perform brittle app-specific checks.

package/docs/artifacts.md

@@ -0,0 +1,30 @@
+---
+id: ruhroh-artifacts
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - src/results.ts
+  - python/ruhroh/loop_controller.py
+---
+
+# Artifacts
+
+Ruhroh preserves the implementation journey and final judgment as Harbor
+artifacts.
+
+Core artifacts:
+
+- `ruhroh-loop-result.json`: final Harbor-facing verdict.
+- `ruhroh-loop-iterations.jsonl`: one implementation-run record per run-agent
+  turn.
+- `ruhroh-loop-journey.json`: full implementation journey summary.
+- `ruhroh-loop-eval.json`: terminal eval-agent judgment.
+- `ruhroh-workspace.tar.gz`: final implementation workspace snapshot.
+- `ruhroh-loop-events.tar.gz`: per-iteration adapter event logs when available.
+- `ruhroh-loop-transcripts.tar.gz`: per-iteration run-agent transcripts.
+
+Adapter-specific artifacts may include bridge logs, prompts, transcripts, or
+result files. They should be referenced from structured result metadata instead
+of inferred by filename heuristics.

package/docs/ci.md

@@ -0,0 +1,31 @@
+---
+id: ruhroh-ci
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - ../.github/workflows/ruhroh-smoke.yml
+  - package.json
+---
+
+# CI Usage
+
+Default CI should exercise deterministic Ruhroh surfaces:
+
+- package build;
+- package unit tests;
+- scenario discovery and task generation fixtures;
+- dry-run Harbor command construction;
+- custom-shell wrapper protocol tests.
+
+Default CI should not require external model credentials or live public-agent
+runs.
+
+Manual workflows may run live adapters when credentials are available. Upload
+the generated Harbor job directory, Ruhroh summary JSON, transcripts, and
+workspace archive as artifacts.
+
+The repo-local `Ruhroh Smoke` workflow keeps live agent execution off by
+default. On manual dispatch, set `live_gemini=true` and provide a
+`GEMINI_API_KEY` secret to run the Gemini CLI custom-shell smoke.

package/docs/custom-shell.md

@@ -0,0 +1,48 @@
+---
+id: ruhroh-custom-shell
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - python/ruhroh/loop_controller.py
+  - ../../examples/adapters/gemini-cli/run.sh
+---
+
+# Custom-Shell Adapter
+
+`custom-shell` is the public escape hatch for agents that can run from a shell
+and write files in a workspace.
+
+Configure it with the generic command adapter protocol:
+
+```bash
+export RUHROH_RUN_AGENT_COMMAND=examples/adapters/gemini-cli/run.sh
+export RUHROH_RUN_AGENT_COMPLETION_PROTOCOL=json-final-line
+```
+
+The current adapter invokes the command with:
+
+- `RUHROH_MESSAGE`
+- `RUHROH_ITERATION`
+- `RUHROH_WORKSPACE`
+- `RUHROH_GOAL_PATH`
+- `RUHROH_WORKSPACE_PATH`
+- `RUHROH_RESULT_PATH`
+- `RUHROH_SESSION_HANDLE`
+- `RUHROH_SCENARIO_ID`
+- `RUHROH_RUN_ROOT`
+- `RUHROH_ADAPTER_ID`
+
+The command must exit `0` for a successful turn. To tell Ruhroh the goal is
+done, print a final JSON line:
+
+```json
+{"status":"goal_satisfied"}
+```
+
+If the final JSON line is absent, Ruhroh treats the turn as incomplete and may
+continue until the iteration cap.
+
+The example Gemini wrapper also writes the `ruhroh_run_agent_result_v1` result
+file at `RUHROH_RESULT_PATH`.

package/docs/eval-agent.md

@@ -0,0 +1,32 @@
+---
+id: ruhroh-eval-agent
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - src/results.ts
+---
+
+# Eval-Agent
+
+The eval-agent is terminal-only in V1. It runs after the implementation loop,
+not after every run-agent turn.
+
+Inputs include:
+
+- original task;
+- scenario context and rubric;
+- implementation run ids;
+- transcripts, event logs, and bridge logs when available;
+- copied final workspace;
+- implementation stop reason.
+
+The eval-agent may inspect files, run commands, start the app, and gather
+evidence. It must not mutate the original implementation workspace.
+
+Expected output is `ruhroh_eval_result_v1` with status `passed`, `failed`,
+`review`, or `infra_failed`. Only `passed` maps to a passing Harbor reward.
+
+Ruhroh core never treats source keywords, required generic filenames, or generic
+routes as app success proxies.

package/docs/getting-started.md

@@ -0,0 +1,59 @@
+---
+id: ruhroh-getting-started
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-23
+depends_on:
+  - README.md
+  - package.json
+---
+
+# Getting Started
+
+Install Ruhroh in a project where you want to generate and run repeatable agent
+tasks:
+
+```bash
+pnpm add -D @kestrel-agents/ruhroh
+```
+
+List the bundled scenarios:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --list
+```
+
+Generate a Harbor task without running an agent:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --generate-only
+```
+
+The generated task appears under:
+
+```text
+.generated/ruhroh/harbor/tasks/simple-newsletter/
+```
+
+Preview the Harbor command:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --adapter custom-shell --dry-run
+```
+
+That dry run should print a `harbor run` command and placeholder secret values
+such as `${OPENAI_API_KEY}`. It should not start Harbor or call a live model.
+
+To run a live agent, provide a command-backed adapter:
+
+```bash
+pnpm exec ruhroh \
+  --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios \
+  --scenario simple-newsletter \
+  --adapter ./path/to/agent-wrapper.sh
+```
+
+Use the artifacts from the Harbor run to review what happened: the final result,
+iteration records, transcripts, event logs, eval judgment, and workspace
+snapshot.

package/docs/harbor.md

@@ -0,0 +1,43 @@
+---
+id: ruhroh-harbor
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - src/harbor.ts
+  - src/generate.ts
+---
+
+# Harbor Usage
+
+Ruhroh generates local Harbor task directories:
+
+```text
+.generated/ruhroh/harbor/tasks/<scenario-id>/
+  task.toml
+  instruction.md
+  tests/test.sh
+  environment/Dockerfile
+  solution/solve.sh
+  assets/
+```
+
+Generated `task.toml` includes schema version, artifacts, task metadata,
+scenario id, verifier timeout, agent timeout, and environment config.
+
+Generated `tests/test.sh` is generic. It reads the final Ruhroh result JSON,
+checks structured completion and score/reward mapping, and does not inspect app
+files, routes, build commands, or source text.
+
+Dry-run command:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --adapter custom-shell --dry-run
+```
+
+Generate without running Harbor:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --generate-only
+```

package/docs/index.md

@@ -0,0 +1,40 @@
+---
+layout: home
+
+hero:
+  name: Ruhroh
+  text: Real-user tasks for coding-agent evaluation.
+  tagline: Generate Harbor-compatible benchmark tasks from realistic user scenarios, run agents through adapters, and judge delivered outcomes.
+  image:
+    src: /ruhroh/ruhroh-logo.png
+    alt: Ruhroh logo
+  actions:
+    - theme: brand
+      text: Get Started
+      link: /getting-started
+    - theme: alt
+      text: Write a Scenario
+      link: /write-a-scenario
+    - theme: alt
+      text: npm
+      link: https://www.npmjs.com/package/@kestrel-agents/ruhroh
+
+features:
+  - title: Outcome-focused
+    details: Evaluate finished workspaces and full implementation journeys instead of brittle source-text heuristics.
+  - title: Agent-agnostic
+    details: Bring your own run-agent adapter. Ruhroh owns scenarios, generation, runtime contracts, and artifacts.
+  - title: Harbor-compatible
+    details: Generate repeatable local Harbor task directories from portable JSON scenarios.
+---
+
+## First commands
+
+```bash
+pnpm add -D @kestrel-agents/ruhroh
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --list
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --generate-only
+```
+
+Ruhroh is for benchmark authors who want tasks that behave more like real user
+requests: goals, constraints, assets, iteration evidence, and outcome judgment.

package/docs/limitations.md

@@ -0,0 +1,24 @@
+---
+id: ruhroh-limitations
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - README.md
+  - python/ruhroh/harbor_agent.py
+---
+
+# Limitations
+
+- The package-owned Python Harbor runtime supports command-backed adapters
+  selected at runtime. First-class provider packages for Kestrel and model eval
+  are still future work.
+- The Kestrel adapter is a consumer integration, not the public benchmark
+  boundary; consumers wire adapters through `RUHROH_RUN_AGENT_COMMAND`.
+- Model-backed eval is supplied by consumers through `RUHROH_EVAL_COMMAND`.
+  Fixture eval remains available for deterministic package smoke tests.
+- Public agent wrappers using `custom-shell` have `workspace_only` continuity
+  unless the wrapper implements stronger session preservation.
+- Live public-agent runs require credentials and are intentionally excluded from
+  default CI.

package/docs/public-repo-layout.md

@@ -0,0 +1,72 @@
+---
+id: ruhroh-public-repo-layout
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - package.json
+  - docs/.vitepress/config.ts
+  - examples/scenarios/simple-newsletter/scenario.json
+  - examples/adapters/gemini-cli/README.md
+---
+
+# Public Repo Layout
+
+Repository shape:
+
+```text
+ruhroh/
+  assets/
+  python/
+    ruhroh/
+  examples/
+    scenarios/
+      simple-newsletter/
+      grocery-budget-planner/
+    adapters/
+      gemini-cli/
+  docs/
+    index.md
+    getting-started.md
+    write-a-scenario.md
+    write-an-adapter.md
+    architecture.md
+    scenario-format.md
+    adapter-protocol.md
+    custom-shell.md
+    harbor.md
+    eval-agent.md
+    artifacts.md
+    ci.md
+    security.md
+    limitations.md
+    public-repo-layout.md
+    .vitepress/
+      config.ts
+  .github/workflows/
+    ruhroh-smoke.yml
+    docs-pages.yml
+  .github/ISSUE_TEMPLATE/
+    bug_report.md
+    scenario_request.md
+    adapter_request.md
+  CHANGELOG.md
+  CONTRIBUTING.md
+  SECURITY.md
+  README.md
+  package.json
+```
+
+Ownership:
+
+- root package contains portable TypeScript APIs and the package CLI.
+- `python` contains the package-owned Harbor runtime.
+- `scenarios` contains bundled benchmark scenarios.
+- `examples/scenarios` contains small public-friendly scenarios.
+- `examples/adapters/gemini-cli` demonstrates a real public-agent wrapper.
+- `docs` contains Markdown docs and the VitePress GitHub Pages site.
+- `.github/ISSUE_TEMPLATE` routes public bug, scenario, and adapter requests.
+
+Downstream projects install `@kestrel-agents/ruhroh`, supply adapter commands,
+and keep project-specific run-agent code outside this repository.

package/docs/scenario-format.md

@@ -0,0 +1,50 @@
+---
+id: ruhroh-scenario-format
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - src/scenarios.ts
+  - src/generate.ts
+---
+
+# Scenario Format
+
+Ruhroh supports JSON scenario directories:
+
+```text
+ruhroh/scenarios/<id>/
+  scenario.json
+  instruction.md
+  assets/
+```
+
+`scenario.json` uses `version: "ruhroh_scenario_v2"` and points at the
+Harbor-visible task prompt with `userPromptPath`.
+
+Required fields:
+
+- `id`, `title`, `tier`, `kind`
+- `userPromptPath`
+- `run.timeoutSeconds` and optional `run.mode`
+- `requires.continuity`, `requires.tools`, `requires.network`
+- `loop.defaultMaxIterations`, `loop.stopPolicy`
+- `evaluation.mode`, `evaluation.scenarioContext`, `evaluation.goalRubric`,
+  `evaluation.evidenceGuidance`
+
+Assets are copied into the generated Harbor task under `assets/`. Scenario
+prompts and assets are untrusted input; run them only inside benchmark
+workspaces.
+
+Generate tasks with:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --generate-only
+```
+
+Adapter selection is runtime configuration, for example:
+
+```bash
+pnpm exec ruhroh --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios --scenario simple-newsletter --adapter ./adapters/my-agent
+```

package/docs/security.md

@@ -0,0 +1,29 @@
+---
+id: ruhroh-security
+domain: security
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-22
+depends_on:
+  - src/env.ts
+  - src/generate.ts
+---
+
+# Security Model
+
+Ruhroh handles untrusted benchmark material and untrusted agent output.
+
+Rules:
+
+- Treat scenario prompts and assets as untrusted input.
+- Run-agents mutate only benchmark workspaces.
+- Eval-agent inspection should happen against a copied workspace or constrained
+  output area.
+- Secrets must be passed through explicit environment allowlists.
+- Dry-run output must print placeholders such as `${OPENAI_API_KEY}`, never
+  secret values.
+- Generated Harbor verifiers do not perform app-goal checks.
+- Public agent examples must not require live credentials in default CI.
+
+When using public coding agents, install them from official sources and review
+their command execution permissions before running on untrusted scenarios.

package/docs/write-a-scenario.md

@@ -0,0 +1,65 @@
+---
+id: ruhroh-write-a-scenario
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-23
+depends_on:
+  - docs/scenario-format.md
+  - src/scenarios.ts
+---
+
+# Write a Scenario
+
+A Ruhroh scenario is a realistic user task plus the metadata needed to run and
+judge it repeatedly.
+
+Create a directory:
+
+```text
+ruhroh/scenarios/my-task/
+  scenario.json
+  instruction.md
+  assets/
+```
+
+The prompt in `instruction.md` should read like a user request. It should state
+the desired outcome, useful constraints, and any domain context the agent needs.
+
+Good prompt:
+
+```md
+Build a small CSV reconciliation tool for the attached people exports. The user
+needs to upload two CSVs, see unmatched records, and download a merged CSV.
+Prioritize a clear workflow and explain any records that cannot be matched.
+```
+
+Poor prompt:
+
+```md
+Create `src/App.tsx`, add a route at `/reconcile`, and include the text
+`Download merged CSV`.
+```
+
+The second prompt overfits implementation details. Use those details only when
+they are genuinely part of the user's goal.
+
+The scenario JSON should define:
+
+- the scenario id, title, tier, and kind;
+- `userPromptPath`;
+- runtime requirements such as continuity, tools, and network;
+- loop defaults such as max iterations;
+- evaluation context, rubric, and evidence guidance.
+
+Keep adapter choice out of new `ruhroh_scenario_v2` scenarios. Select adapters
+at runtime with `--adapter`.
+
+Use the rubric to describe outcome quality. The generated Harbor verifier stays
+generic; it should not become a scenario-specific file or source-code checker.
+
+Validate by generating the task:
+
+```bash
+pnpm exec ruhroh --scenario-dir ruhroh/scenarios --scenario my-task --generate-only
+```

package/docs/write-an-adapter.md

@@ -0,0 +1,72 @@
+---
+id: ruhroh-write-an-adapter
+domain: benchmarks
+status: active
+owner: ruhroh-maintainers
+last_verified_at: 2026-06-23
+depends_on:
+  - docs/custom-shell.md
+  - python/ruhroh/loop_controller.py
+---
+
+# Write an Adapter
+
+Ruhroh is agent-agnostic. A run-agent adapter is the bridge between Ruhroh's
+iteration loop and the coding agent you want to evaluate.
+
+For public usage, the simplest adapter is a shell command:
+
+```bash
+pnpm exec ruhroh \
+  --scenario-dir node_modules/@kestrel-agents/ruhroh/scenarios \
+  --scenario simple-newsletter \
+  --adapter ./adapters/my-agent.sh
+```
+
+When `--adapter` looks like a path or command, Ruhroh sets
+`RUHROH_RUN_AGENT_COMMAND` for the package runtime.
+
+Minimal wrapper:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+cd "$RUHROH_WORKSPACE"
+
+printf '%s\n' "$RUHROH_MESSAGE" > .ruhroh-current-goal.md
+
+# Replace this with your agent invocation.
+# The agent should edit files inside $RUHROH_WORKSPACE.
+my-agent --prompt-file .ruhroh-current-goal.md
+
+printf '{"status":"goal_satisfied"}\n'
+```
+
+The command receives environment variables including:
+
+- `RUHROH_MESSAGE`
+- `RUHROH_ITERATION`
+- `RUHROH_WORKSPACE`
+- `RUHROH_GOAL_PATH`
+- `RUHROH_RESULT_PATH`
+- `RUHROH_SCENARIO_ID`
+- `RUHROH_RUN_ROOT`
+
+The wrapper must exit `0` for a successful turn. If the goal is complete, emit a
+final JSON line:
+
+```json
+{"status":"goal_satisfied"}
+```
+
+If the wrapper does not emit completion, Ruhroh may continue until the iteration
+cap. For richer results, write a `ruhroh_run_agent_result_v1` JSON result file
+to `RUHROH_RESULT_PATH`.
+
+Keep wrappers conservative:
+
+- operate only inside `RUHROH_WORKSPACE`;
+- store prompts, logs, and transcripts under the run root or workspace;
+- avoid printing secrets;
+- keep live credentials out of default CI.

package/package.json

@@ -1,13 +1,13 @@
 {
   "name": "@kestrel-agents/ruhroh",
-  "version": "0.5.0-beta.0",
+  "version": "0.5.0-beta.1",
   "description": "Real-User Harness for Repair-Oriented Harbor",
   "license": "MIT",
   "repository": {
     "type": "git",
     "url": "git+https://github.com/LumiCorp/ruhroh.git"
   },
-  "homepage": "https://github.com/LumiCorp/ruhroh",
+  "homepage": "https://lumicorp.github.io/ruhroh/",
   "bugs": {
     "url": "https://github.com/LumiCorp/ruhroh/issues"
   },
@@ -41,6 +41,8 @@
     "python/**/*.py",
     "python/**/*.sh",
     "scenarios/**/*",
+    "docs/**/*.md",
+    "CHANGELOG.md",
     "README.md",
     "LICENSE"
   ],
@@ -54,13 +56,17 @@
   "scripts": {
     "clean": "node --input-type=module -e \"import { rmSync } from 'node:fs'; rmSync('dist', { recursive: true, force: true });\"",
     "build": "pnpm run clean && tsc -p tsconfig.json",
+    "docs:dev": "vitepress dev docs",
+    "docs:build": "vitepress build docs",
+    "docs:preview": "vitepress preview docs",
     "prepare": "pnpm run build",
     "test": "node --import tsx --test tests/*.test.ts"
   },
   "devDependencies": {
     "@types/node": "^22.13.10",
     "tsx": "^4.19.3",
-    "typescript": "^5.8.2"
+    "typescript": "^5.8.2",
+    "vitepress": "^1.6.4"
   },
   "packageManager": "pnpm@9.12.3"
 }