robot_lab-to 0.2.6

data/docs/concepts/evals.md

@@ -0,0 +1,254 @@
+# Evals: Scoring Iterations
+
+A robot reporting `success: true` is a claim; a `--verify-command` that must exit
+`0` answers "did it break?". Neither answers the harder question: **is this
+iteration actually better than the last one?** An agent will happily report
+success — and pass a verify command — on a change that made no real progress.
+
+An **eval** replaces the robot's self-report as the deciding authority. It is an
+orchestrator-owned, pluggable scorer that judges the *uncommitted working tree*
+after every iteration and returns a `Score`. The eval decides commit vs. rollback
+(via `gate_ok` / `improved`) and, for measurable objectives, decides when the run
+is *done* (`met_target`) — instead of the robot's self-reported
+`should_fully_stop`.
+
+This generalizes the [Verification Gate](verification.md): `--verify-command`
+still exists and still works exactly as before, but it is now one input to the
+default eval (`Evals::Code`) rather than the whole story.
+
+## The `Score`
+
+Every eval returns a `RobotLab::To::Evals::Score` — a `Data.define` with six
+fields:
+
+| Field | Meaning |
+|-------|---------|
+| `gate_ok` | The floor holds (correctness). Must be `true` to commit at all. |
+| `improved` | Better than the parent commit? The descent signal. |
+| `met_target` | The objective is measurably reached — an orchestrator-owned stop. |
+| `value` | The score itself (higher = better); `nil` for pure-pairwise evals. |
+| `detail` | One-line human summary, written to `notes.md` and the log. |
+| `output` | Diagnostic text (gate output, judge rationale) fed to repair prompts; `nil` when there is none. |
+
+`gate_ok?`, `improved?`, and `met_target?` predicate readers are also defined.
+
+## How it drives the loop
+
+```
+robot says success ──► Eval#score ──► gate_ok? ──► improved (or --no-require-improvement)? ──► COMMIT
+                                          │                        │
+                                          │ no                     │ no
+                                          ▼                        ▼
+                              repair_until_gate (R2)         ROLLBACK  ([NO IMPROVEMENT])
+                              then ROLLBACK if still failing
+```
+
+- **Gate fails** (`gate_ok: false`) — the failure output is handed back to the
+  *same* robot up to `--max-verify-repairs` times, re-scoring after each attempt
+  (this is the same R2 repair loop the verify gate always had — see
+  [Verification Gate](verification.md)). If the budget runs out, the tree is
+  rolled back and the notes get a `[VERIFY FAILED]` entry.
+- **Gate passes but didn't improve** (`improved: false`) — rolled back with a
+  `[NO IMPROVEMENT]` entry in `notes.md`, *unless* `--no-require-improvement` is
+  passed (see [`require_improvement`](#require_improvement-strict-descent)
+  below). This is **not** counted as a failure — it does not trip
+  `--max-consecutive-failures` — but it does count toward
+  [`--stop-on-plateau`](stop-conditions.md#plateau-no-improvement).
+- **Gate passes and improved** — committed. `@run.last_score_value` is updated,
+  the plateau counter resets, and `met_target?` is checked: if true, the run
+  aborts with `target met: <detail>` — this **supersedes** the robot's
+  self-reported `should_fully_stop`; `--stop-when` remains as a fallback.
+
+Each iteration's verdict is logged as an `eval` event in `run.log` (`gate`,
+`improved`, `value`, `met_target`, `detail`) — see
+[Run State & Event Log](run-state.md).
+
+## `Evals::Null` — the default, unscored
+
+When nothing is configured, `Evals::Null` reports `gate_ok: true, improved: true,
+met_target: false` on every call — every reported success commits, and the run
+stops only via `--stop-when` or the other stop conditions. This is exactly
+`robot_lab-to`'s original behavior; adding an eval is entirely opt-in.
+
+## `Evals::Code` — measured descent
+
+For software, "better" is measurable. `Evals::Code` composes two independent
+commands:
+
+| Option | Role |
+|--------|------|
+| `--verify-command CMD` | **Floor** — must exit `0` (correctness). Same gate as before. |
+| `--measure CMD` | **Descent signal** — a command that prints a number to stdout; higher is better. |
+| `--target FLOAT` | Stop once the measured value reaches this. |
+
+```bash
+robot-to "Raise parser coverage to 90%" \
+  --verify  "bundle exec rake test" \      # floor: must stay green
+  --measure "bundle exec rake coverage" \  # descent signal
+  --target  90                             # stop once the score reaches this
+```
+
+`Evals::Code#score`:
+
+- `gate_ok` — `true` when `--verify-command` is unset, or the command exits `0`.
+- `value` — the **first number** matched in the `--measure` command's stdout
+  (`/-?\d+(?:\.\d+)?/`), or `nil` if `--measure` is unset.
+- `improved` — with no `value`, equals `gate_ok` (any gate-passing change is
+  progress, matching legacy behavior). With a `value`, `true` only if it strictly
+  beats `previous_value` (or there is no prior committed value yet).
+- `met_target` — `true` once `value >= --target`.
+
+**Backward compatible by construction:** with no `--measure`, `Evals::Code`
+behaves exactly like the original verify-only gate — `improved == gate_ok`,
+`met_target` never fires, and the run stops via `--stop-when` / the other stop
+conditions as before.
+
+The `--measure` command just has to print a number: `rake coverage`, a benchmark
+harness, `ruby -e 'puts passing_count'`, `grep -c` of a lint report — anything.
+
+## `Evals::Prose` — pairwise judgement
+
+For a document, an opinion piece, or a book there is no `rake coverage`. Absolute
+scores from an LLM are too noisy to descend against — but "is B better than A?"
+is a reliable judgement. `Evals::Prose` asks a **judge model** to compare the
+working tree to the last *committed* version:
+
+```bash
+robot-to "Write an opinionated guide to the Viable Systems Model" \
+  --eval  prose \
+  --spec  outline.md \          # the spec the judge measures against
+  --floor "rake docs:lint" \    # optional mechanizable checks (links, TODOs)
+  --judge-model claude-opus-4 \ # defaults to the doer's --model
+  --stop-on-plateau 3           # stop after 3 drafts with no improvement
+```
+
+`Evals::Prose#score`:
+
+- `gate_ok` — `true` when `--floor` is unset, or the floor command exits `0`. If
+  the floor fails, the judge is **not** called (saves a request) and the verdict
+  is forced to `:worse`.
+- The judge is shown `VERSION A` (the parent commit) and `VERSION B` (the working
+  tree) for every file that changed, plus the objective and — if `--spec` is set —
+  its contents, and replies exactly one word: `better`, `worse`, or `same`.
+  Anything ambiguous is treated as `:same` (does not count as improvement).
+- **Brand-new document shortcut:** if the parent version is empty and the draft
+  is not, the judge is skipped and the verdict is `:better` automatically — an
+  "empty vs. content" pairwise comparison is degenerate and models frequently
+  (and wrongly) call it `:same`.
+- `improved` — `true` only when the verdict is `:better`.
+- `met_target` — **always `false`.** There is no absolute milestone for prose;
+  runs end via `--stop-on-plateau`, `--max-iterations`, or a human stopping it.
+- `value` — always `nil` (pairwise verdicts have no scalar).
+
+By default the judge **reuses the doer's `--model`** — no separate judge model is
+required to get pairwise scoring. Pass `--judge-model` to use a different (often
+cheaper, or deliberately more skeptical) model for the judge role. See
+`examples/04_prose` in the gem for a worked two-model (doer + judge) setup.
+
+## `require_improvement` — strict descent
+
+- **Default: `true`.** A gate-passing iteration that does not beat the parent is
+  rolled back (`[NO IMPROVEMENT]`), so the branch descends monotonically.
+- `--no-require-improvement` restores "commit any gate-passing success," useful
+  when you want every non-regressing attempt kept rather than only strict
+  improvements.
+- This is a hardcoded Ruby/CLI-only default (`Config#require_improvement?`), not
+  in `defaults.yml` — set it per-run via `--no-require-improvement` or
+  `RobotLab::To.run(require_improvement: false)`.
+
+## Plateau: the primary stop for judged evals
+
+`Evals::Prose` never sets `met_target`, so its natural terminator is
+`--stop-on-plateau N` — stop after `N` iterations with no committed improvement.
+Plateau is tracked independently of `--max-consecutive-failures` (which is for
+the robot being genuinely broken — errors, gate failures, no submit); a
+gate-passing "same" verdict is normal, healthy exploration for prose, not a
+failure. See [Stop Conditions](stop-conditions.md#plateau-no-improvement).
+
+## Score feedback in the prompt
+
+When a run is scored (`--measure`, `--target`, `--eval prose`, or `--spec` is
+set), `PromptBuilder` adds a **Score Feedback** section to every iteration's
+system prompt:
+
+- The best committed score so far and the target, if any: *"Best score committed
+  so far: 84.0 (target: 90.0)."*
+- A plateau warning after a run of non-improving iterations: *"The last 3
+  iteration(s) did not improve and were rolled back... try a DIFFERENT approach
+  now."*
+- Otherwise, positive reinforcement: *"Your last change improved the result and
+  was committed. Build on it."*
+
+This turns each attempt into a testable hypothesis against the target rather
+than a blind guess. Unscored runs (the `Evals::Null` default) get no such
+section — behavior is unchanged.
+
+## Guarding the grader (`GraderLock`)
+
+Whatever the eval, the robot must not be able to "win" by editing the criteria
+it is scored against. `Guards::GraderLock` — a `RobotLab::Hook` — refuses any
+`write`/`edit` targeting a locked path, and refuses any `bash` command whose
+string references one (matched conservatively, by absolute path or basename).
+
+Locked paths come from two sources, merged:
+
+- `Evals::Base#protected_paths` — `Evals::Prose#protected_paths` returns
+  `[spec]` (a floor *command* isn't reliably a single file, so lock it
+  separately with `--protect-path` if it's a script); `Evals::Code` locks
+  nothing by default — **the test suite is deliberately editable**, since
+  adding tests is how coverage legitimately rises.
+- `--protect-path GLOB` (repeatable) — user-specified grader files for any eval,
+  code or custom (e.g. a floor script, a scoring rubric).
+
+`GraderLock` is installed in `Orchestrator#build_robot` whenever any protected
+paths resolve, **independent of `--local-guards`** — unlike the small-model
+guardrails (see [Guardrails](../local-models/guardrails.md)), a frontier model
+can reward-hack too, so this one is always on.
+
+```bash
+robot-to "..." --eval code --measure "ruby score.rb" --target 100 \
+  --protect-path score.rb --protect-path test/scoring_test.rb
+```
+
+## Custom evals
+
+Three escape hatches, all via the Ruby API (`RobotLab::To.run` / `Config.new`):
+
+```ruby
+# (a) subclass RobotLab::To::Evals::Base
+class BenchmarkEval < RobotLab::To::Evals::Base
+  def score(ctx) = RobotLab::To::Evals::Score.new(gate_ok: true, improved: ..., met_target: ..., value: ..., detail: ..., output: nil)
+end
+RobotLab::To.run(objective, eval: BenchmarkEval.new)
+
+# (b) a bare proc — wrapped in Evals::ProcEval automatically
+RobotLab::To.run(objective, eval: ->(ctx) { RobotLab::To::Evals::Score.new(...) })
+
+# (c) a registered symbol
+RobotLab::To.register_eval(:benchmark) { |config| BenchmarkEval.new(target: config.eval_target) }
+RobotLab::To.run(objective, eval: :benchmark)
+```
+
+!!! warning "`--eval` on the CLI only resolves `code` / `null` / `prose`"
+    The CLI's `--eval NAME` flag always produces a **String**. The factory's
+    registry lookup only matches a **Symbol** (`Config.new(eval: :benchmark)`),
+    so a registered custom eval is reachable from the Ruby API but not yet from
+    `--eval benchmark` on the command line — passing an unrecognized string
+    raises `unknown eval strategy`. Likewise, pointing `--eval` at a `.rb` file
+    path is not implemented; use the Ruby API's instance/proc/registered-symbol
+    forms instead.
+
+## Backward compatibility
+
+| Invocation | Eval selected | Behavior |
+|---|---|---|
+| No `--verify-command`, no `--eval`, no `--measure`/`--target` | `Evals::Null` | Commit any reported success; stop via `--stop-when`. **Identical to pre-Evals behavior.** |
+| `--verify-command "rake test"` only | `Evals::Code` (gate-only) | Commit on pass; stop via `--stop-when`. **Identical.** |
+| Old `run.json` (`--resume`) | — | Loads; `last_score_value` / `iterations_since_improvement` default to `nil` / `0`. |
+
+New behavior only activates when `--measure`, `--target`, `--eval prose`, or
+`--stop-on-plateau` is supplied.
+
+---
+
+Next: [Run State & Event Log](run-state.md).

data/docs/concepts/iteration-loop.md

@@ -0,0 +1,128 @@
+# The Iteration Loop
+
+The orchestrator is a loop. Each pass is an independent attempt at **one**
+improvement toward the objective. This page describes the full lifecycle and the
+decisions made at each step.
+
+![The robot-to iteration loop](../assets/iteration-loop.svg)
+
+## Setup (once per run)
+
+Before the loop begins, `setup_run`:
+
+- Confirms `HEAD` exists — aborts with guidance if the repo has no commits.
+- Creates the run branch `robot-to/<slug>-<timestamp>` and records `base_commit`.
+- Creates the run directory and seeds `notes.md` with the objective header.
+- Adds the run directory to `.git/info/exclude`.
+- Builds the per-run collaborators: `PromptBuilder`, `Backoff`, `StopConditions`,
+  and the **eval** (`Evals.build(config, git:)` — `Evals::Code` when
+  `--verify-command`/`--measure`/`--target` is set, `Evals::Prose` for
+  `--eval prose`, otherwise the unscored `Evals::Null`; see [Evals](evals.md)).
+
+## One iteration
+
+### 1. Build a fresh robot
+
+A **new** `RobotLab::Robot` is created every iteration — there is no shared chat
+history between iterations. Continuity comes entirely from `notes.md`, which is
+injected into the system prompt. The prompt (built by `PromptBuilder`) contains:
+
+- the objective and current iteration number,
+- the accumulated notes ("prior work"),
+- the task instructions (make one focused change; run tests/build; a no-op is not
+  success),
+- the requirement to call `submit_iteration_result`,
+- optional repair and stop-when sections.
+
+The robot is given the `submit_iteration_result` tool. With `--local-guards`, it
+also receives built-in file tools (`read`, `write`, `edit`, `bash`) and the
+guardrail hooks. See [Built-in Tools](../local-models/tools.md). If the
+configured eval (or `--protect-path`) has protected paths, `Guards::GraderLock`
+is attached too — independent of `--local-guards` — refusing edits to the
+grader's own criteria. See [Evals](evals.md#guarding-the-grader-graderlock).
+
+### 2. The robot works
+
+`robot.run` executes. The robot makes changes using whatever tools it has, then
+calls `submit_iteration_result` with:
+
+| Field | Meaning |
+|-------|---------|
+| `success` | Did this iteration make meaningful progress? |
+| `summary` | One sentence describing what was done or why it stopped. |
+| `key_changes` | Files or changes made this iteration. |
+| `key_learnings` | Insights to carry forward in the notes. |
+| `should_fully_stop` | Set when a `--stop-when` condition is judged met. |
+
+### 3. Handle "thinks but doesn't act"
+
+If the robot ends its turn **without** calling `submit_iteration_result`, the
+orchestrator re-prompts the *same* robot (preserving its chat) up to
+`--max-submit-nudges` times, asking only for the result. This recovers the common
+degenerate case where a model does work but forgets the final report. If it still
+doesn't submit, the iteration is treated as a failure.
+
+### 4. Decide: commit or roll back
+
+```
+result.success?  ──► Eval#score  ──► gate_ok? ──► improved (or --no-require-improvement)? ──► COMMIT
+       │                                │                         │
+       │ no                            │ no                       │ no
+       ▼                                ▼ (after R2 repair budget) ▼
+    ROLLBACK                        ROLLBACK                   ROLLBACK
+```
+
+- **Robot reported failure (or no submit)** → `git reset --hard`;
+  consecutive-failure counter increments. The eval is never called.
+- **Robot reported success** → the configured **eval** scores the working tree
+  (`Evals::Null` by default — see [Evals](evals.md)):
+    - **Gate fails** (`gate_ok: false`) — handed back to the *same* robot for up
+      to `--max-verify-repairs` repair attempts, re-scoring each time; if the
+      gate is still failing when the budget runs out, `git reset --hard` and
+      recorded as `[VERIFY FAILED]`.
+    - **Gate passes, didn't improve** (`improved: false`, and
+      `require_improvement?` is true — the default) → `git reset --hard`;
+      recorded as `[NO IMPROVEMENT]`. Not counted as a failure.
+    - **Gate passes and improved** → `git add -A` and commit (if anything is
+      staged). The consecutive-failure/-error counters reset, and the eval's
+      `met_target?` is checked — if true, the run stops immediately.
+
+See [Evals](evals.md) for the full scoring model (this generalizes what used to
+be a verify-command-only gate — see [Verification Gate](verification.md)).
+
+### 5. Update the notes
+
+Either way, the orchestrator appends an entry to `notes.md` — success, failure,
+no-improvement, verify/gate-failure, or error — so the next iteration's robot
+sees what happened. See [Cross-Iteration Memory](notes.md).
+
+### 6. Check stop conditions
+
+Stop conditions are evaluated **before** and **after** each iteration. The loop
+ends if iterations, tokens, consecutive failures, or the eval's plateau counter
+hit their limits; if the eval reports `met_target`; or if the robot set
+`should_fully_stop` for a `--stop-when` condition. See
+[Stop Conditions](stop-conditions.md) and [Evals](evals.md).
+
+## Errors, retries, and backoff
+
+Exceptions during an iteration (e.g. transient API errors) are caught:
+
+- **Authentication errors** are permanent — the run aborts immediately.
+- Other errors roll back the tree, record the error in notes, and **retry** after
+  an interruptible exponential backoff (`60 × 2ⁿ` seconds), up to `--max-retries`
+  consecutive errors.
+
+A `SIGINT`/`SIGTERM` (Ctrl-C) requests a graceful stop: the in-flight robot is
+interrupted, the loop exits cleanly, and the exit summary still prints.
+
+## Commit-failure repair
+
+If a commit itself fails (e.g. a pre-commit hook rejects the change), the failure
+is queued. The next iteration's prompt includes a **repair section** asking the
+robot to fix the uncommitted changes first. The working tree is *not* reset while
+a commit failure is pending, so the robot can address it.
+
+---
+
+Next: [Cross-Iteration Memory](notes.md).

data/docs/concepts/notes.md

@@ -0,0 +1,91 @@
+# Cross-Iteration Memory (`notes.md`)
+
+Each iteration runs a **fresh** robot with no memory of previous iterations. The
+only thing that carries forward is `notes.md` — a human-readable log the
+orchestrator maintains and injects into every iteration's system prompt.
+
+This is what lets the loop make progress instead of repeating the same attempt.
+
+## Who writes it, who reads it
+
+- **The orchestrator writes it.** After every iteration it appends an entry
+  (success, failure, verify-failure, or error).
+- **The robot reads it.** The `PromptBuilder` embeds the current notes into the
+  system prompt under a "Prior Work" section.
+- **The robot must not edit it.** The prompt explicitly tells the robot that
+  `notes.md` is maintained automatically. Writes are atomic (`AtomicFile`).
+
+## Location
+
+```
+.robot_lab_to/runs/<run_id>/notes.md
+```
+
+The path is also passed to the robot in the prompt so it can read the file
+directly if it has file tools.
+
+## Structure
+
+The file starts with a header and grows one section per iteration:
+
+```markdown
+# robot-to run: 20260623-144554-3e5c27
+
+Objective: Migrate the codebase from Minitest to RSpec
+
+## Iteration Log
+
+### Iteration 1
+
+**Summary:** Converted spec_helper and the parser tests to RSpec
+
+**Changes:**
+- spec/spec_helper.rb
+- spec/parser_spec.rb
+
+**Learnings:**
+- The project uses `assert_equal`; RSpec equivalent is `expect(x).to eq(y)`
+
+### Iteration 2 [FAIL]
+
+**Summary:** Attempted to convert the integration tests but broke the suite
+
+**Learnings:**
+- The integration tests depend on a Rack test helper not yet ported
+```
+
+## Entry types
+
+| Marker | When | Includes |
+|--------|------|----------|
+| *(none)* | Successful, committed iteration | summary, changes, learnings |
+| `[FAIL]` | Robot reported failure or didn't submit | summary, learnings |
+| `[VERIFY FAILED]` | Robot claimed success but the eval's gate failed (`--verify-command` / `--floor`) | summary, gate output |
+| `[NO IMPROVEMENT]` | Gate passed but the eval didn't rule it better than the parent commit (rolled back unless `--no-require-improvement`) | summary, score detail |
+| `[ERROR]` | An exception was raised during the iteration | error class and message |
+
+`[VERIFY FAILED]` and `[NO IMPROVEMENT]` come from the [Evals](evals.md) scoring
+gate, not the robot — see there for how `gate_ok` and `improved` are decided.
+
+Because failures and their learnings are recorded too, the next robot can avoid
+repeating a dead end — the failure log is as valuable as the success log.
+
+## Designing objectives around the notes
+
+The notes are most effective when:
+
+- The objective is a single, clear goal the robot can chip away at.
+- Each iteration is genuinely incremental — "make ONE focused change" is baked
+  into the prompt.
+- `key_learnings` capture *transferable* facts ("the build uses esbuild, not
+  webpack"), not restatements of the diff.
+
+You can read the notes live during a run to watch the robot's reasoning evolve:
+
+```bash
+tail -f .robot_lab_to/runs/*/notes.md
+```
+
+---
+
+Next: [Stop Conditions](stop-conditions.md).

data/docs/concepts/run-state.md

@@ -0,0 +1,93 @@
+# Run State & Event Log
+
+Everything a run produces — besides the git commits — lives under a single run
+directory. This page documents that layout and the JSONL event log.
+
+## Run directory layout
+
+```
+.robot_lab_to/
+└── runs/
+    └── <run_id>/                 # e.g. 20260623-144554-3e5c27
+        ├── notes.md              # human-readable cross-iteration memory
+        ├── run.log               # JSONL event log (one event per line)
+        └── checkpoints/          # pre-edit file snapshots (local_guards only)
+```
+
+- `<run_id>` is `YYYYMMDD-HHMMSS-<random hex>`.
+- The whole `.robot_lab_to/` directory is added to `.git/info/exclude` at startup,
+  so it never appears in `git status` or commits.
+- Change `--run-dir` to relocate it (default `.robot_lab_to`).
+
+## The JSONL event log (`run.log`)
+
+`run.log` contains one JSON object per line — a structured, append-only trace of
+the run. Every line has an `event` name and a UTC `ts` timestamp; other keys
+depend on the event. The logger buffers up to 100 events before the file is open,
+so the very first setup events are never lost.
+
+A typical successful iteration:
+
+```json
+{"event":"orchestrator:start","ts":"2026-06-23T14:45:54.001-05:00","run_id":"20260623-144554-3e5c27","objective":"...","branch":"robot-to/...","model":"claude-sonnet-4-6"}
+{"event":"iteration:start","ts":"...","iteration":1}
+{"event":"agent:run:start","ts":"...","iteration":1}
+{"event":"agent:run:end","ts":"...","iteration":1}
+{"event":"eval","ts":"...","iteration":1,"gate":true,"improved":true,"value":null,"met_target":false,"detail":"verify=true"}
+{"event":"commit:success","ts":"...","iteration":1,"message":"robot-to 1: ..."}
+```
+
+### Event reference
+
+| Event | Meaning |
+|-------|---------|
+| `orchestrator:start` | Run began. Carries `run_id`, `objective`, `branch`, `model`. |
+| `orchestrator:resume` | A `--resume` re-entered the loop (`run_id`, `iteration`, `branch`). |
+| `iteration:start` | A new iteration began. |
+| `agent:run:start` / `agent:run:end` | The robot's `run` call started / finished. |
+| `agent:nudge` | The robot didn't submit; re-asking (`attempt` N). |
+| `agent:run:error` | An exception occurred during the iteration (`error`, `message`). |
+| `backoff:start` / `backoff:end` | Retry backoff sleep around a transient error. |
+| `eval` | The [eval's](evals.md) verdict for this iteration: `gate`, `improved`, `value`, `met_target`, `detail`. |
+| `eval:repair` | R2 repair attempt after a gate failure (`attempt` N of `--max-verify-repairs`). |
+| `eval:repair_error` | The repair attempt itself raised (`message`); repair loop stops. |
+| `commit:success` | Iteration committed (`message`). |
+| `commit:failed` | `git commit` failed; queued for repair (`output`). |
+| `iteration:failure` | Robot reported failure or never submitted (`summary`). |
+| `iteration:gate_failure` | Robot claimed success but the eval's gate failed (recorded in notes as `[VERIFY FAILED]`). |
+| `iteration:no_improvement` | Gate passed but the eval didn't rule it an improvement (`detail`); rolled back. |
+| `orchestrator:abort` | Run stopped by a stop condition, `met_target`, or permanent error (`reason`). |
+| `orchestrator:fatal` | Unexpected fatal error (`error`, `message`). |
+| `orchestrator:end` | Final totals: `iterations`, `commits`, `input_tokens`, `output_tokens`, `abort_reason`, `elapsed`. |
+
+Decision-file events (`decision:blocking`, `decision:wait:start`,
+`decision:wait:resolved`, `decision:raised`) are also written when decisions are
+enabled — see the README's Human-in-the-Loop section.
+
+### Analyzing a run
+
+Because every line is JSON, the log is easy to query with
+[`jq`](https://stedolan.github.io/jq/):
+
+```bash
+# Watch events live, formatted
+tail -f .robot_lab_to/runs/*/run.log | jq -r '"\(.ts)  \(.event)"'
+
+# How many iterations committed vs failed?
+jq -r 'select(.event=="commit:success")' run.log | wc -l
+jq -r 'select(.event=="iteration:failure") | .summary' run.log
+
+# Final token usage
+jq 'select(.event=="orchestrator:end") | {iterations,commits,input_tokens,output_tokens}' run.log
+```
+
+## Debug logging
+
+By default, RubyLLM/RobotLab logging is suppressed so the console stays clean.
+Pass `--debug` to keep verbose provider logging enabled (useful when diagnosing
+provider or tool-call issues). The JSONL event log is always written regardless
+of `--debug`.
+
+---
+
+Next: [Configuration](../configuration/index.md).

data/docs/concepts/stop-conditions.md

@@ -0,0 +1,111 @@
+# Stop Conditions
+
+Because `robot-to` is meant to run unattended, it must reliably stop on its own.
+Five independent conditions can end a run; whichever trips first wins.
+
+## The five conditions
+
+| Condition | Flag | Checked | Effect |
+|-----------|------|---------|--------|
+| Max iterations | `--max-iterations N` | before & after each iteration | Stop after `N` iterations. |
+| Max tokens | `--max-tokens N` | before & after each iteration (and mid-stream when streaming) | Stop once cumulative tokens reach `N`. |
+| Consecutive failures | `--max-consecutive-failures N` | after each iteration | Abort after `N` failures in a row (default **3**). |
+| Plateau | `--stop-on-plateau N` | after each iteration | Abort after `N` iterations with no committed improvement (eval-gated runs only). |
+| Stop-when | `--stop-when "<text>"` | after each *successful* iteration | The robot judges the condition; sets `should_fully_stop`. |
+
+There is a sixth, orchestrator-owned stop that isn't a `StopConditions` check at
+all: an [eval's](evals.md) `met_target` — reaching a measured `--target` aborts
+the run immediately on that iteration's commit, taking priority over
+`should_fully_stop`. See [Evals](evals.md).
+
+If you set none of the bounded limits, the loop relies on
+`--max-consecutive-failures` (default 3) and `--stop-when` to terminate. For a
+genuinely unattended run, **always set at least `--max-iterations` or
+`--max-tokens`.**
+
+## How each works
+
+### Max iterations
+
+A simple counter. The run aborts with `max iterations reached (N)` once
+`run.iteration >= N`.
+
+### Max tokens
+
+Cumulative input + output tokens across all iterations are compared against the
+limit.
+
+- **Streaming runs** (the default) account tokens per chunk and can interrupt an
+  in-flight iteration the moment the budget is exhausted.
+- **Non-streaming runs** (`--no-stream`, used for local models) account tokens
+  from each iteration's result and stop at the next iteration boundary. See
+  [Local Models](../local-models/ollama.md#streaming-and-tool-calls).
+
+### Consecutive failures
+
+Resets to zero on every successful, committed iteration; increments on every
+failure, verify-failure, or commit-failure. This is the safety net that stops a
+run that is going nowhere. The default is **3**.
+
+### Plateau (no improvement)
+
+`--stop-on-plateau N` aborts once `N` consecutive iterations pass the eval's gate
+but fail to beat the parent commit's score (`iterations_since_improvement >= N`).
+It resets to zero on every committed improvement.
+
+This is the **primary terminator for `Evals::Prose`**, which never sets a
+measurable `met_target` — a pairwise judge has no absolute bar to reach, only
+"better than last time." It's distinct from `--max-consecutive-failures`:
+plateau counts *valid, gate-passing work that just wasn't better* (including
+"same" verdicts, which are common and normal for prose), while consecutive
+failures counts the robot being genuinely broken (errors, gate failures, no
+submit). Conflating the two would throttle legitimate exploration.
+
+```bash
+robot-to "Write an opinionated guide to VSM" \
+  --eval prose --spec outline.md \
+  --stop-on-plateau 3   # stop after 3 drafts that didn't improve
+```
+
+See [Evals](evals.md) for the full scoring model.
+
+### Stop-when (natural language)
+
+`--stop-when` is a goal the robot evaluates. After a successful iteration, the
+robot decides whether the condition is met and, if so, returns
+`should_fully_stop: true` in its result. The orchestrator then aborts with
+`stop condition met: <text>`.
+
+```bash
+robot-to "Refactor the auth module for clarity" \
+  --max-iterations 30 \
+  --stop-when "the auth module has no method longer than 15 lines and tests pass"
+```
+
+`--stop-when` only ends the loop on a **successful** iteration — a failing
+iteration never triggers it, even if the robot thinks the goal is met.
+
+## Graceful interruption
+
+Pressing **Ctrl-C** (`SIGINT`) or sending `SIGTERM` requests a graceful stop: the
+in-flight robot is interrupted, any pending backoff sleep is cancelled, the loop
+exits, and the exit summary still prints. The current iteration's incomplete work
+is rolled back.
+
+## What you'll see
+
+The exit summary header reports the reason:
+
+```
+robot-to stopped — claude-sonnet-4-6 — 41m 08s
+Reason: max iterations reached (30)
+```
+
+Possible reasons include `max iterations reached (N)`, `max tokens reached (N)`,
+`N consecutive failures`, `plateau: N iterations without improvement`,
+`target met: <detail>`, `stop condition met: <text>`, a permanent error, or a
+fatal error.
+
+---
+
+Next: [Verification Gate](verification.md).