lemans 1.4.0 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7ec0760a162ddb4b0fe13d4ddeb4027f934c1ffae9c7acbfda330da9ff68d91f
-  data.tar.gz: 85cd135708fade8a06610c4cd26e4f4556e8d90ec566d3a5e12e02d13e46d351
+  metadata.gz: 4383d276c7923cc0853a1eb42bc179cc753769010893e61fd63dab4304aad2ef
+  data.tar.gz: 6805d08e1946fcc7bc7d06adb41c721daa2aeb184f2093c720a8affa2881b40c
 SHA512:
-  metadata.gz: eb13268e0b98f9e8ca4f82f3e50b0a454366f00f0030fcaed4d03bee6d67eb214370e1317f9ffbba7701beefdb4d4e8e6c5f399cdaad90425208bdbc2b950fdc
-  data.tar.gz: f0a8acc7bc79be0fd546fdc0c59db8e0cfa354aba6806bb09f416bf2d61b898c895a0b94a71b27c122a6a7f65854f51d07fd32f2fd0ce66d91e9e4568163bdeb
+  metadata.gz: d9a0993a5a5ec87224ea241ee0001c4d382b3ee4da2d401840e00275c66952844f25d2f86d504e9fdbd967c115bb59bed8f6366bb6fea33ffd3fe5ca54b0c013
+  data.tar.gz: 291121a4b9cafca127af023db929e3c1a5a162f197e5d306d1e45f6e26aebb63032b56d47f944189392c2cd7a9b18f563d7558a57e07719132b0555b685e0b35

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 ## [Unreleased]
 
+## [1.5.0] - 2026-10-05
+
+- `lemans restart --recover` continues the failed step's agent session
+- `lemans restart --reverify` grades the last verified step again and goes on from there; `--allow-scored` restarts scored runs.
+- miniswen: `--continue-from PATH` continues a session from its `--results-path` file.
+- Docker: support the `allowlist` network mode through a proxy container on an internal network.
+- Support `lemans report path/to/runs`, `lemans clobber path/to/runs`, and `lemans regrade path/to/runs` (in addition to `--runs-dir path/to/runs`)
+- Add `lemans restart <run>` to continue a failed multistep run from its last completed step as a new run.
+- Fix(miniswen): run Bash tool commands through `bash -c` in Local, Jail, and Docker; sandbox images must include Bash.
+
 ## [1.4.0] - 2026-09-24
 
 - miniswen: add built-in proxy and the `--allow-hosts` switch.

data/README.md

@@ -15,6 +15,7 @@ lemans is a harness for benchmarking coding agents, the Ruby way:
 - Ruby 3.4+ is required to run `lemans`
 - Daytona account (API token) or Docker (for local sandboxes)
 - Some LLM provider/proxy credentials (e.g., OpenRouter)
+- Bash in the sandbox image for command execution
 - iproute2 in the sandbox image, so a jailed agent keeps loopback while cut off the internet
 
 ## Getting started
@@ -102,6 +103,8 @@ verifier:
   # restore: [test, bin]                            # [optional] paths restored from the pre-agent snapshot, so grading can't be tampered with
 ```
 
+On the Docker backend, `allowlist` sends traffic through a proxy container. Only tools that use `http_proxy`/`https_proxy` can reach the listed hosts. Host names must match exactly, and IP ranges are not supported. See [the proxy README](lib/lemans/environments/docker/proxy/README.md).
+
 A minimal task example—checking whether an agent can write "Hello, world" into a file:
 
 - `instruction.md`: the task itself (note the preamble about the training corpora and the frontmatter)
@@ -231,6 +234,17 @@ Each trial writes a flat `runs/<model>/<task>__<id>/` directory:
  
 Every `result.json` is stamped with the `lemans_version` that wrote it, and newer lemans keeps reading runs produced by older releases. `lemans run --resume` skips trials that already have a scored result for the same agent and model.
 
+`lemans restart <run>` continues a multistep run that failed for reasons outside the model, such as a provider outage. Give it the run directories or the trial ids: `lemans restart RUN [RUN...]` restarts every run under the same options, `-c` at once, and runs none if any is refused. The new run replays the agent patches of the settled steps in a fresh sandbox and starts at the next step. It copies the settled steps' usage, phases, and artifacts, and records the source in `restarted_from`. The failed run stays as it is. It restarts even if the task or the bench changed since; the new run records the current digests.
+
+Two modes restart inside a step:
+
+- `--recover` continues the failed step's agent session. The new run also replays the step's partial patch, and the agent goes on from the saved `agent.result.json`. The session keeps the steps, tokens, cost, and time it already spent. Only `miniswen` and `miniswen-installed` can recover.
+- `--reverify` runs the last verification again with the current tests. Use it after you change the grading or the harness. The new run replays the graded step's patch and does not run its agent. If the verification was an intermediate gate and it now passes, the trial goes on to the next step. A reverification accepts scored runs.
+
+A scored run restarts in the other modes only with `--allow-scored`.
+
+Only the files that git tracks come back: ignored files (databases, logs, installed dependencies) are not in the patches.
+
 You can also run `lemans report` with various flags to see aggregated results, e.g.:
 
 ```sh
@@ -256,8 +270,9 @@ gpt-5.6-luna  ar-archive-book-access  2/2    2m 23s  $0.0132  12.5   156905
 | `lemans init` | Scaffold a new bench directory: an annotated `bench.yml` and two example tasks |
 | `lemans tasks` | List the tasks in a bench (`--tag` to filter) |
 | `lemans run` | Run tasks and grade them (`--task`, `--tag`, `--agent`, `--model`, `--max-output-tokens`, `-k`, `-c`, `--resume`) |
-| `lemans report` | Summarize `runs/` as a table or CSV (`--task`, `--tag`, `--metadata key:value` to filter, `-A [task-agent-model]` to aggregate, `-S <column>` to sort); repeated attempts add pass@k per model × task, fractional grading a `credit` column |
-| `lemans clobber` | Delete run results (`--task`, `--ttl 10m\|2h\|1d`, `--invalid`, `-f` to skip the confirmation) |
+| `lemans restart <run>...` | Continue failed multistep runs from their last settled step in new runs (`-c`, `--recover` to continue the failed step's session, `--reverify` to grade again, `--allow-scored`, `--backend`, `--max-output-tokens`) |
+| `lemans report [RUNS_DIR]` | Summarize `runs/` (or `RUNS_DIR`) as a table or CSV (`--task`, `--tag`, `--metadata key:value` to filter, `-A [task-agent-model]` to aggregate, `-S <column>` to sort); repeated attempts add pass@k per model × task, fractional grading a `credit` column |
+| `lemans clobber [RUNS_DIR]` | Delete run results under `runs/` (or `RUNS_DIR`) (`--task`, `--ttl 10m\|2h\|1d`, `--invalid`, `-f` to skip the confirmation) |
 
 ## miniswen
 

data/exe/lemans-remote

@@ -29,6 +29,13 @@
 #
 #     exe/lemans-remote run --bench ../ai-evals --retry-runs=./runs
 #
+#   Restart local runs remotely, with the options of `lemans restart` — one
+#   sandbox per run (--run-in-band for a single sandbox, --sync to wait and
+#   download). The runs travel with the bench; the new runs come back via
+#   pull-runs (the shipped copies do not):
+#
+#     exe/lemans-remote restart --bench ../ai-evals runs/b/model/task__AbC1234 task__XyZ5678 --reverify
+#
 #   Then watch, fetch, and clean up:
 #
 #     exe/lemans-remote status [--history] [--running | --complete] [-W [INTERVAL]]
@@ -81,6 +88,7 @@ module LemansRemote # :nodoc: all
   REMOTE_BENCH_DIR = "/task"
   REMOTE_BENCH_ARCHIVE = "/tmp/bench.tgz"
   REMOTE_RUNS_ARCHIVE = "/tmp/runs.tgz"
+  REMOTE_SOURCES_ARCHIVE = "/tmp/sources.tgz"
   REMOTE_LOG = "/tmp/lemans-remote-run.log"
   REMOTE_STATUS = "/tmp/lemans-remote-run.status"
   REMOTE_WRAPPER = "/tmp/lemans-remote-wrapper.sh"
@@ -546,8 +554,11 @@ module LemansRemote # :nodoc: all
     TIMED_OUT = 124
     CREATE_ATTEMPTS = 3
 
+    # `command` replaces the `lemans run` built from tasks/models/args; `sources`
+    # are run dirs (relative to `runs_dir`) shipped into the sandbox's runs/
+    # for the command to read, and dropped again before the results leave it.
     def initialize(bench:, snapshot:, run_id:, tasks:, models:, extra_args:, extra_env:,
-                   timeout_sec:, runs_dir:, keep:, sync:, shell:, launch_interval: 0)
+                   timeout_sec:, runs_dir:, keep:, sync:, shell:, launch_interval: 0, command: nil, sources: [])
       @bench = bench
       @snapshot = snapshot
       @run_id = run_id
@@ -561,6 +572,8 @@ module LemansRemote # :nodoc: all
       @sync = sync
       @shell = shell
       @launch_interval = launch_interval
+      @command = command
+      @sources = sources
       @started_at = Time.now.utc.iso8601
     end
 
@@ -666,6 +679,31 @@ module LemansRemote # :nodoc: all
       exec!(sandbox, "mkdir -p #{REMOTE_BENCH_DIR} && " \
                      "tar -xzf #{REMOTE_BENCH_ARCHIVE} -C #{REMOTE_BENCH_DIR} && " \
                      "rm -f #{REMOTE_BENCH_ARCHIVE}")
+      upload_sources(sandbox) if @sources.any?
+    end
+
+    def upload_sources(sandbox)
+      say :upload, "#{@sources.size} run(s) from #{@runs_dir}"
+      Dir.mktmpdir("lemans-remote") do |tmp|
+        archive = File.join(tmp, "sources.tgz")
+        _, err, status = Open3.capture3("tar", "-czf", archive, "-C", @runs_dir.to_s, *@sources)
+        raise "could not pack the runs: #{err}" unless status.success?
+
+        File.open(archive, "rb") do |file|
+          sandbox.fs.upload_file_stream(file, REMOTE_SOURCES_ARCHIVE, timeout: TRANSFER_TIMEOUT_SEC)
+        end
+      end
+      exec!(sandbox, "mkdir -p #{REMOTE_BENCH_DIR}/runs && " \
+                     "tar -xzf #{REMOTE_SOURCES_ARCHIVE} -C #{REMOTE_BENCH_DIR}/runs && " \
+                     "rm -f #{REMOTE_SOURCES_ARCHIVE}")
+    end
+
+    # The shipped runs are copies of local ones: archiving them back would
+    # resurrect whatever was clobbered locally in the meantime.
+    def drop_sources_command
+      return "true" if @sources.empty?
+
+      "(cd #{REMOTE_BENCH_DIR}/runs && rm -rf -- #{@sources.shelljoin})"
     end
 
     def execute(sandbox)
@@ -683,6 +721,8 @@ module LemansRemote # :nodoc: all
     end
 
     def remote_command
+      return @command if @command
+
       command = %w[lemans run --bench .]
       @tasks.each { command += [ "--task", it ] }
       @models.each { command += [ "--model", it ] }
@@ -718,6 +758,7 @@ module LemansRemote # :nodoc: all
         "tasks" => @tasks,
         "models" => @models,
         "args" => @extra_args,
+        "command" => remote_command,
         "lemans_version" => Lemans::VERSION,
         "snapshot" => @snapshot,
         "sandbox_id" => sandbox.id,
@@ -735,6 +776,7 @@ module LemansRemote # :nodoc: all
         (cd #{REMOTE_BENCH_DIR} && #{remote_command}) > #{REMOTE_LOG} 2>&1
         status=$?
         cp #{REMOTE_LOG} #{vault_dir}/run.log || true
+        #{drop_sources_command}
         if [ -d #{REMOTE_BENCH_DIR}/runs ]; then
           tar -czf #{vault_dir}/runs.tar.gz -C #{REMOTE_BENCH_DIR}/runs .
         fi
@@ -827,7 +869,7 @@ module LemansRemote # :nodoc: all
         return
       end
 
-      exec!(sandbox, "tar -czf #{REMOTE_RUNS_ARCHIVE} -C #{REMOTE_BENCH_DIR}/runs .")
+      exec!(sandbox, "#{drop_sources_command} && tar -czf #{REMOTE_RUNS_ARCHIVE} -C #{REMOTE_BENCH_DIR}/runs .")
       FileUtils.mkdir_p(@runs_dir)
       Dir.mktmpdir("lemans-remote") do |tmp|
         local = File.join(tmp, "runs.tgz")
@@ -975,6 +1017,67 @@ module LemansRemote # :nodoc: all
       raise Thor::Error, "lemans-remote: #{e.message}"
     end
 
+    desc "restart RUN...", "Restart local runs on remote Daytona sandboxes, the way `lemans restart` does (one sandbox per run)"
+    option :bench, default: ".", desc: "Directory holding bench.yml"
+    option :runs_dir, default: "runs", desc: "Local directory holding the runs (sync mode: and receiving the new ones)"
+    option :recover, type: :boolean, default: false, desc: "Continue the failed step's agent session from its saved history"
+    option :reverify, type: :boolean, default: false,
+                      desc: "Grade the last verified step again (with the current tests) and go on from there"
+    option :allow_scored, type: :boolean, default: false, desc: "Restart a scored run (--reverify always may)"
+    option :max_output_tokens, type: :numeric, banner: "TOKENS", desc: "Cap the agent's output per model call"
+    option :concurrency, type: :numeric, aliases: "-c", desc: "Restarts in flight at once inside a sandbox (with --run-in-band)"
+    option :timeout, default: "6h", desc: "Give up on the remote run after this long"
+    option :sync, type: :boolean, default: false, desc: "Restart every run in one sandbox, wait, and download the results"
+    option :run_in_band, type: :boolean, default: false, aliases: %w[--runInBand],
+                         desc: "Async mode: restart every run in one sandbox instead of one sandbox per run"
+    option :keep, type: :boolean, default: false, desc: "Sync mode: leave the sandbox around for debugging"
+    option :env, repeatable: true, desc: "Forward an extra host ENV variable by name"
+    option :launch_interval, type: :numeric, default: 3,
+                             desc: "Async mode: seconds between consecutive launches (0 to disable)"
+    def restart(*runs)
+      raise Thor::Error, "lemans-remote: name the run(s) to restart" if runs.empty?
+      raise Thor::Error, "lemans-remote: --recover and --reverify exclude each other" if options[:recover] && options[:reverify]
+      raise Thor::Error, "lemans-remote: --launch-interval must not be negative" if options[:launch_interval].to_f.negative?
+
+      bench = Lemans::Config.load_file(options[:bench])
+      runs_root = Pathname(options[:runs_dir])
+      store = Lemans::Stores::FS.new(runs_root)
+      ids = runs.map { File.basename(it) }.uniq
+      found = store.fetch.select { ids.include?(it.id) }.to_h { [ it.id, it ] }
+      missing = ids - found.keys
+      raise Thor::Error, "lemans-remote: no run #{missing.join(", ")} under #{runs_root}" if missing.any?
+
+      sources = found.values_at(*ids)
+
+      # The sandbox would refuse them anyway: refuse here, before any sandbox starts
+      bench.load_options(model: sources.map(&:model).uniq)
+      Lemans::Runner.new(bench, bench.tasks, store:, restarts: sources, restart_mode: restart_mode,
+                                             allow_scored: options[:allow_scored]).attempts
+
+      provisioner = build_provisioner
+      raise Thor::Error, "lemans-remote: snapshot #{provisioner.name} not found — run `lemans-remote provision` first" unless provisioner.provisioned?
+
+      jobs = options[:sync] || options[:run_in_band] ? [ sources ] : sources.map { [ it ] }
+      Vault.ensure unless options[:sync]
+
+      say_status :fanout, "#{jobs.size} sandboxes, one per run, one every #{options[:launch_interval]}s" if jobs.size > 1
+      failures = []
+      jobs.each_with_index do |job, index|
+        sleep options[:launch_interval].to_f if index.positive? && options[:launch_interval].to_f.positive?
+        exit_code = launch_restart(bench, job, runs_root, provisioner)
+        exit exit_code if options[:sync] && !exit_code.zero?
+      rescue StandardError => e
+        raise if options[:sync]
+
+        failures << "#{job.map(&:id).join(",")} (#{e.message})"
+      end
+
+      say_status :detached, "`lemans-remote status` to watch, `lemans-remote pull-runs` to fetch results" unless options[:sync]
+      raise Thor::Error, "lemans-remote: failed to launch: #{failures.join("; ")}" if failures.any?
+    rescue Lemans::ConfigError, RuntimeError => e
+      raise Thor::Error, "lemans-remote: #{e.message}"
+    end
+
     desc "status", "List lemans-remote runs (--history adds completed runs whose sandboxes are gone)"
     option :history, type: :boolean, default: false, desc: "Also read the vault manifests (spins a short-lived helper sandbox)"
     option :running, type: :boolean, default: false, desc: "Only show runs still in flight"
@@ -1282,8 +1385,43 @@ module LemansRemote # :nodoc: all
       [ failures, launched ]
     end
 
-    def build_run_id(bench, tasks, models, attempt = nil)
+    def restart_mode = (:recover if options[:recover]) || (:reverify if options[:reverify])
+
+    def launch_restart(bench, sources, runs_root, provisioner)
+      tasks = sources.map(&:task).uniq
+      models = sources.map(&:model).uniq
+      Runner.new(
+        bench: bench,
+        snapshot: provisioner.name,
+        run_id: build_run_id(bench, tasks, models, kind: "restart"),
+        tasks: tasks,
+        models: models,
+        extra_args: "",
+        extra_env: options[:env] || [],
+        timeout_sec: seconds!(options[:timeout]).to_i,
+        runs_dir: runs_root.to_s,
+        keep: options[:keep],
+        sync: options[:sync],
+        shell: shell,
+        launch_interval: options[:launch_interval].to_f,
+        command: restart_command(sources),
+        sources: sources.map { |source| runs_root.glob("**/#{source.id}").find(&:directory?).relative_path_from(runs_root).to_s }
+      ).call
+    end
+
+    def restart_command(sources)
+      argv = [ "lemans", "restart", *sources.map(&:id), "--bench", ".", "--runs-dir", "runs" ]
+      argv << "--recover" if options[:recover]
+      argv << "--reverify" if options[:reverify]
+      argv << "--allow-scored" if options[:allow_scored]
+      argv += [ "--max-output-tokens", options[:max_output_tokens].to_i.to_s ] if options[:max_output_tokens]
+      argv += [ "-c", options[:concurrency].to_i.to_s ] if options[:concurrency]
+      argv.shelljoin
+    end
+
+    def build_run_id(bench, tasks, models, attempt = nil, kind: nil)
       parts = [ bench.root.expand_path.basename.to_s ]
+      parts << kind if kind
       parts << (tasks.size == 1 ? tasks.first : "#{tasks.size}tasks") if tasks.any?
       if models.any?
         short = models.first.split("/").last