robot_lab-to 0.2.6

data/docs/local-models/tools.md

@@ -0,0 +1,92 @@
+# Built-in Tools
+
+When `--local-guards` is enabled, the per-iteration robot is given a small set of
+built-in workspace tools in addition to `submit_iteration_result`. They are
+named with short, conventional names (`read`, `write`, `edit`, `bash`) — small
+models do measurably better with short tool names than with long namespaced ones.
+
+!!! note "Why these are bundled"
+    Without file tools, a robot can only *report* — it can't change code. The
+    built-in tools give a local model a predictable, guarded surface to do real
+    work. With a cloud model you'd typically supply richer tools yourself; these
+    are a dependable baseline for the local-assistant use case.
+
+## The tools
+
+### `read`
+
+Read a UTF-8 text file, optionally a line range.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `path` | yes | File to read. |
+| `offset` | no | 1-based first line to return. |
+| `limit` | no | Maximum number of lines to return. |
+
+Output is capped (2 000 lines) so one oversized read can't blow a small model's
+context, and a truncation note tells the model how to read more. Reading a file
+also satisfies the [read-before-edit](guardrails.md#read-before-edit) guard.
+
+### `write`
+
+Create a **new** file. Parent directories are created as needed.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `path` | yes | File to create. |
+| `content` | yes | Full text content. |
+
+`write` is for new files only. The [write-guard](guardrails.md#write-guard)
+refuses it on a file that already exists and redirects the model to `edit` — small
+models otherwise rewrite whole files and lose content.
+
+### `edit`
+
+Exact-string replacement in an existing file.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `path` | yes | File to edit. |
+| `old_text` | yes | Exact text to replace (whitespace included). |
+| `new_text` | yes | Replacement text. |
+| `replace_all` | no | Replace every occurrence (default false). |
+
+`old_text` must match the current file contents exactly and be unique unless
+`replace_all` is set; otherwise the edit is refused with a clear message. The
+[read-before-edit](guardrails.md#read-before-edit) guard requires the file to have
+been `read` this run first, so `old_text` reflects the real contents rather than a
+guess.
+
+### `bash`
+
+Run a shell command in the project directory.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `command` | yes | Shell command to run. |
+| `timeout` | no | Seconds before the command is killed (default 120). |
+
+Returns combined stdout+stderr and the exit status (`[exit 0]\n...`). Output is
+capped (~30 000 chars) and a runaway process is killed at the timeout so it can't
+stall an overnight loop.
+
+## How they're wired in
+
+The orchestrator builds them per iteration and attaches them through RobotLab's
+`local_tools:`:
+
+```ruby
+# only when local_guards is on
+[submit_tool, Tools::Read.new, Tools::Write.new, Tools::Edit.new, Tools::Bash.new]
+```
+
+Without `--local-guards`, the robot gets only `submit_iteration_result` — you're
+expected to supply your own tools (typical for cloud-model setups).
+
+The short names come from a small `FileTool` base class that overrides RubyLLM's
+default namespaced tool name. The guardrail hooks match on exactly these names
+(`read`/`write`/`edit`/`bash`).
+
+---
+
+Next: [Guardrails](guardrails.md).

data/docs/reference/architecture.md

@@ -0,0 +1,139 @@
+# Architecture
+
+`robot_lab-to` is a thin, well-separated orchestration layer over RobotLab. The
+`Orchestrator` owns the loop and the git history; everything else is a
+single-purpose collaborator it drives.
+
+![robot-to architecture](../assets/architecture.svg)
+
+## Component map
+
+| Component | File | Responsibility |
+|-----------|------|----------------|
+| `CLI` | `cli.rb` | `OptionParser`-based `robot-to` entry point. |
+| `Config` | `config.rb` | `MywayConfig::Base` settings cascade. |
+| `Orchestrator` | `orchestrator.rb` | The main loop: setup → iterate → eval-gated commit/rollback → stop → finalize. |
+| `PromptBuilder` | `prompt_builder.rb` | Builds each iteration's system prompt (objective, notes, task, submit, repair, score feedback, stop-when). |
+| `Run` | `run.rb` | Mutable run-state value object (iteration, counters, tokens, timing, paths, `last_score_value`, `iterations_since_improvement`). |
+| `SubmitResult` | `tools/submit_result.rb` | Tool the robot calls to report its result. |
+| `IterationResult` | `iteration_result.rb` | `Data.define` value object for a submitted result. |
+| `CommitManager` | `commit_manager.rb` | All git operations via `Open3` (no shell interpolation); also diffs the working tree vs. a ref for `Evals::Prose`. |
+| `NotesManager` | `notes_manager.rb` | Reads/writes the cross-iteration `notes.md`. |
+| `StopConditions` | `stop_conditions.rb` | Evaluates iterations / tokens / failures / plateau / stop-when. |
+| `Verifier` | `verifier.rb` | Low-level command runner (timeout, process-group kill); composed by `Evals::Code`. |
+| `Backoff` | `backoff.rb` | Interruptible exponential backoff for retries. |
+| `JsonlLogger` | `jsonl_logger.rb` | Append-only JSONL event log with a pre-init buffer. |
+| `ExitSummary` | `exit_summary.rb` | Prints the post-run report. |
+| `AtomicFile` | `atomic_file.rb` | Atomic writes/appends for `notes.md`. |
+
+## Evals — orchestrator-owned scoring
+
+The eval subsystem generalizes the verify-command gate into a pluggable, scalar
+scoring loop. See [Evals](../concepts/evals.md) for the full behavioral model —
+this table is the file map.
+
+| Component | File | Responsibility |
+|-----------|------|-----------------|
+| `Evals::Score` | `evals/score.rb` | `Data.define(gate_ok, improved, met_target, value, detail, output)` — the verdict for one iteration. |
+| `Evals::Context` | `evals/context.rb` | `Data.define(work_dir, previous_ref, previous_value, objective, iteration)` handed to every eval. |
+| `Evals::Base` | `evals/base.rb` | Abstract eval; `#score(ctx)` and `#protected_paths` (default `[]`). |
+| `Evals::Null` | `evals/null.rb` | Default when nothing is configured: always `gate_ok: true, improved: true, met_target: false`. |
+| `Evals::Code` | `evals/code.rb` | Measured/deterministic: `--verify-command` floor + optional `--measure`/`--target`. Composes `Verifier`. |
+| `Evals::Prose` | `evals/prose.rb` | Judged/pairwise: an LLM judge compares the working tree vs. the parent commit; optional `--floor`. `met_target` always `false`. |
+| `Evals::ProcEval` | `evals/factory.rb` | Wraps a bare proc passed as `eval:` so it responds to `#score`. |
+| `Evals.build` / `Evals.from_name` | `evals/factory.rb` | Resolves `config.eval` (instance / proc / registered Symbol / `"code"` / `"null"` / `"prose"` / `nil`) into an eval. |
+| `Guards::GraderLock` | `guards/grader_lock.rb` | Always-on `RobotLab::Hook` (independent of `--local-guards`) refusing write/edit/bash on an eval's `protected_paths` + `--protect-path`. |
+
+## Local-model layer
+
+These components are active only when `--local-guards` is set:
+
+| Component | File | Responsibility |
+|-----------|------|----------------|
+| `Tools::FileTool` | `tools/file_tool.rb` | Base class giving file tools short names. |
+| `Tools::Read/Write/Edit/Bash` | `tools/*.rb` | Built-in workspace tools. |
+| `Guards` | `guards.rb` | Registers all guards on a robot via `install(robot, run:, except:)`. |
+| `Guards::WriteGuard` | `guards/write_guard.rb` | Refuse `write` on existing files; normalize paths. Excludable per-run via `Config#write_guard = false` (Ruby-only, default `true`) — useful when the robot is iteratively rewriting a whole file, e.g. a prose draft. |
+| `Guards::ReadBeforeEdit` | `guards/read_before_edit.rb` | Refuse `edit` before `read`. |
+| `Guards::Checkpoint` | `guards/checkpoint.rb` | Snapshot files before mutation. |
+| `Guards::QualityMonitor` | `guards/quality_monitor.rb` | Detect repeated-tool-call loops. |
+| `Guards::RunStore` | `guards/run_store.rb` | Thread-local per-run guard state. |
+| `Guards::PathResolution` | `guards/path_resolution.rb` | Pure path helpers shared by the guards. |
+
+See [Guardrails](../local-models/guardrails.md) and
+[Built-in Tools](../local-models/tools.md).
+
+## Key design decisions
+
+### Fresh robot per iteration
+
+Each iteration constructs a new `RobotLab::Robot`. There is no shared chat history;
+continuity is carried entirely by `notes.md`. This keeps every iteration a clean,
+independent attempt and bounds context growth.
+
+### The orchestrator owns git, the robot owns work
+
+The robot never commits. It makes changes and reports a result; the orchestrator
+decides whether to commit (gated by the configured eval) or roll back. This
+separation is what makes the git history trustworthy.
+
+### The eval, not the robot, decides "better" and "done"
+
+`should_fully_stop` and a claimed `success: true` are the robot grading its own
+homework. `Evals::Base#score` is the orchestrator-owned authority instead:
+`gate_ok`/`improved` decide commit vs. rollback, and `met_target` — not the
+robot's self-report — decides when a measurable objective is done. `--stop-when`
+remains as a fallback for objectives an eval can't measure. See
+[Evals](../concepts/evals.md).
+
+### Grader lockdown is separate from the local-model guards
+
+`Guards::GraderLock` protects the eval's own criteria (a spec, a floor script,
+`--protect-path` globs) from robot edits. It is wired in unconditionally when
+protected paths exist, unlike the small-model guardrail set
+(`Guards.install`, gated by `--local-guards`) — a frontier model can reward-hack
+its grading criteria just as easily as a small one can mangle a file.
+
+### Tools attach via `local_tools:`
+
+Tool *instances* are passed to `RobotLab.build` as `local_tools:`. (RobotLab's
+`tools:` parameter is a name *allowlist filter*, not an instance list — passing
+instances there silently attaches nothing.) A regression test asserts the built
+robot actually exposes the submit tool, guarding against that class of mistake.
+
+### Streaming is optional
+
+By default the robot streams, enabling per-chunk token accounting and mid-stream
+budget enforcement. Local Ollama models run non-streaming (`stream: false`), so
+tokens are accounted from each iteration's result and the budget is enforced at
+iteration boundaries. See [Ollama Setup](../local-models/ollama.md#streaming-and-tool-calls).
+
+### Interruptible by design
+
+Signal handlers, the backoff sleep, and the per-iteration runner thread all
+cooperate so a `SIGINT`/`SIGTERM` stops the run promptly and cleanly, with the
+exit summary still printed.
+
+## Run lifecycle (control flow)
+
+```
+To.run(objective, **opts)
+└─ Orchestrator#run
+   ├─ setup_run            branch, notes, run dir, collaborators (incl. Evals.build)
+   ├─ main_loop
+   │  └─ per iteration:
+   │     ├─ build fresh robot (PromptBuilder + tools [+ guards] [+ GraderLock])
+   │     ├─ robot.run  → SubmitResult captured
+   │     ├─ nudge if not submitted
+   │     ├─ evaluate (Eval#score) if success reported
+   │     │  ├─ gate fails  → repair_until_gate (R2), else reset --hard
+   │     │  ├─ gate ok, not improved → reset --hard ([NO IMPROVEMENT])
+   │     │  └─ gate ok, improved → commit; check met_target?
+   │     ├─ NotesManager append
+   │     └─ StopConditions check (incl. plateau)
+   └─ finalize_run         JSONL close, ExitSummary
+```
+
+---
+
+Next: [Changelog](../about/changelog.md).

data/examples/01_basic_usage/.gitignore

@@ -0,0 +1,9 @@
+# Artifacts produced when the example runs — never commit them into the
+# robot_lab-to repository. (gitignore has no inline comments, so each note
+# is on its own line.)
+
+# the throwaway git repo the robot works in (recreated each run)
+/project/
+
+# the run's event log and cross-iteration notes.md
+/.robot_lab_to/

data/examples/01_basic_usage/README.md

@@ -0,0 +1,173 @@
+# 01 — Basic Usage
+
+An end-to-end use of `robot_lab-to`, driven programmatically with
+`RobotLab::To.run`.
+
+The script ([`basic_usage.rb`](basic_usage.rb)):
+
+1. Creates a fresh **`project/`** git repository next to the script,
+   seeded with a **failing Minitest suite** for a Roman-numeral library
+   (`test/roman_numeral_test.rb`). It's git-ignored and re-created on each run.
+2. Runs an autonomous loop with the objective *"implement `lib/roman_numeral.rb`
+   until the seeded test suite passes."* Because the tests are fixed and the robot
+   may not edit them, it has to satisfy real, externally-defined requirements it
+   can't game.
+3. Gates each iteration on `ruby -Ilib -Itest test/roman_numeral_test.rb` — a
+   change is committed only when the suite is green; otherwise it's rolled back.
+4. Stops early (via `stop_when`) once the suite passes, and prints the resulting
+   branch, commit history, the generated implementation, and a final test run.
+
+It exercises the full loop over several iterations: **branch → iterate → verify →
+commit on green / roll back on red**.
+
+### The seeded spec
+
+The library must provide `RomanNumeral.to_roman(int)` and
+`RomanNumeral.from_roman(str)` with: subtractive notation (`IV`, `IX`, `XL`, …),
+the `1..3999` range, round-trip correctness, case-insensitive parsing, and
+`ArgumentError` on invalid input — including **non-canonical** numerals like
+`"IIII"`. The full expectations live in the seeded `test/roman_numeral_test.rb`.
+
+## Prerequisites
+
+Either a **local Ollama** server (default, no API key) or a **cloud** provider key.
+
+=== "Local Ollama (default)"
+
+    ```bash
+    ollama serve &                # if not already running
+    ollama pull qwen3.6:latest    # a tool-capable model
+    ```
+
+=== "Cloud provider"
+
+    ```bash
+    export ANTHROPIC_API_KEY="sk-ant-..."   # or OPENAI_API_KEY
+    ```
+
+## Run it
+
+From the gem root:
+
+```bash
+# Local Ollama (default)
+bundle exec ruby examples/01_basic_usage/basic_usage.rb
+
+# A cloud model instead
+RLTO_LOCAL=false RLTO_PROVIDER=anthropic RLTO_MODEL=claude-sonnet-4-6 \
+  bundle exec ruby examples/01_basic_usage/basic_usage.rb
+```
+
+## Configuration
+
+All optional, set via environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `RLTO_LOCAL` | `true` | Use a local Ollama model. Set `false` for a cloud provider. |
+| `RLTO_PROVIDER` | `openai` (local) | LLM provider passed to `RobotLab::To.run`. |
+| `RLTO_MODEL` | `qwen3.6:latest` (local) | Model id. Any tool-capable model works. |
+| `OLLAMA_BASE` | `http://localhost:11434/v1` | Ollama's OpenAI-compatible endpoint. |
+
+## What you'll see
+
+A `RobotLab::Hook` (`FeedbackHook` in the script) narrates the robot's actions
+live, so the run never sits silent while the model works:
+
+```
+Cleaning leftover: .../examples/01_basic_usage/project
+Project dir:     examples/01_basic_usage/project
+Provider/model:  openai/qwen3.6:latest (local Ollama)
+Objective:       implement lib/roman_numeral.rb to pass the test suite
+
+      🤔 thinking…
+      📖 read test/roman_numeral_test.rb
+      📝 write lib/roman_numeral.rb
+      💻 bash: ruby -Ilib -Itest test/roman_numeral_test.rb
+         ↳ exit 1
+      ✏️ edit lib/roman_numeral.rb
+      💻 bash: ruby -Ilib -Itest test/roman_numeral_test.rb
+         ↳ exit 0
+      ✅ submit result: Implemented RomanNumeral.to_roman/from_roman; all 9 tests pass.
+
+=== Result ===
+Branch:  robot-to/implement-a-roman-numeral-library-...
+
+Commits:
+4a59bbc robot-to 1: Implemented lib/roman_numeral.rb ... all 9 tests pass.
+72b06ec initial: failing roman-numeral test suite
+
+lib/roman_numeral.rb:
+module RomanNumeral
+  ...
+end
+
+Final test run:
+9 runs, 594 assertions, 0 failures, 0 errors, 0 skips
+
+Notes:    examples/01_basic_usage/.robot_lab_to/runs/.../notes.md
+Project:  examples/01_basic_usage/project
+Run logs: examples/01_basic_usage/.robot_lab_to
+```
+
+### Live feedback via the hook system
+
+`FeedbackHook < RobotLab::Hook` is registered globally with `RobotLab.on(...)`,
+so it applies to every per-iteration robot the orchestrator builds — no change to
+the orchestrator needed. It implements three hook methods:
+
+| Hook method | Reports |
+|-------------|---------|
+| `before_llm_generation` | `🤔 thinking…` — the model started generating |
+| `before_tool_call` | the action: read / write / edit / bash / submit |
+| `after_tool_call` | a shell exit code, or a tool error |
+
+This is a compact, practical example of the robot_lab hook system; the guardrails
+shipped with `--local-guards` use the same mechanism.
+
+> **Don't want to write your own?** robot_lab ships a ready-made narrator —
+> `RobotLab::Narrator.enable!` — which does exactly this. The
+> [02_advanced_usage](../02_advanced_usage/README.md#live-feedback) example uses
+> it. We hand-write the hook here to teach the API.
+
+> **Note:** robot_lab-to's own `robot-to: …` progress lines use `Kernel#warn`,
+> which is silenced when Ruby warnings are disabled (`$VERBOSE` is `nil`, common
+> under `bundle exec`). The hook writes to `$stderr` directly, so its narration
+> always shows.
+
+Everything the example produces stays under `examples/01_basic_usage/`:
+
+```
+examples/01_basic_usage/
+├── basic_usage.rb
+├── project/     # the throwaway git repo (re-created on every run)
+└── .robot_lab_to/       # run event logs + cross-iteration notes.md
+```
+
+Each run **starts with a clean slate** — `project/` and `.robot_lab_to/`
+left over from a previous execution are deleted before setup, then rebuilt. The
+most recent run's repo, logs, and `notes.md` remain afterward for inspection
+(until the next run clears them). Both are git-ignored, so you never need to
+clean them up by hand.
+
+## How it maps to the library
+
+The whole example is one call:
+
+```ruby
+RobotLab::To.run(
+  objective,
+  provider:       :openai,    # routes to Ollama for local runs
+  model:          "qwen3.6:latest",
+  local_guards:   true,       # built-in file tools + guardrails
+  stream:         false,      # Ollama tool calls need non-streaming
+  max_iterations: 6,
+  verify_command: "ruby -Ilib -Itest test/roman_numeral_test.rb",
+  stop_when:      "test/roman_numeral_test.rb passes with 0 failures and 0 errors"
+)
+```
+
+For the meaning of each option, see the
+[Settings Reference](https://madbomber.github.io/robot_lab-to/configuration/settings/)
+and, for the local-model specifics, the
+[Local Models guide](https://madbomber.github.io/robot_lab-to/local-models/).