lemans 1.1.0 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 44398d66bdcf062d7d5d41ad70e400e76a0031afdcbbdf2df83883ec60aefd33
4
- data.tar.gz: 79dcdeff359498787b2535c5155ae00806c17572cb8359b51a27454f67a1c5d6
3
+ metadata.gz: f74cd8839b49ac3d579c9e8999e398d867b4b4d6510bae6f28546b00e5e427cd
4
+ data.tar.gz: bac93bb29cb5d9c6ef31c99aa13dd845ced0b022088876390e8bda5f075857ee
5
5
  SHA512:
6
- metadata.gz: f6cb208dfd63c0c63631586e84db8e7cb3e1b9c1e9092d59c6c41d4d58d8057793ca15facf7b9ab15a56723606b878b8817ce97024ad852e3e4c1199429382cb
7
- data.tar.gz: 251f9a0142b9464499c3c5219ddcc4c95684609048200f5d48a4c55e4e022370941746a3f8635594be3a53ef5f72fcf84d006caeb2b94af618bc38f4ec40c3f7
6
+ metadata.gz: 65772ed04c753b4109fef2f57d0351506c84cff8b5724f097d634c2f95e2a240b37013a3e86b2cdb6af3893ff630a36a1feb9418305267e37394f7fffbe4e273
7
+ data.tar.gz: 5adfd42c14059d7ca8863617cb1c13ae504cf00c24f82f1dba348f1f89a02c8c7d422de987d5cd714abe14f869883bbd4f3828a8e397083f3464b3c34ee215c7
data/CHANGELOG.md CHANGED
@@ -1,5 +1,12 @@
1
1
  ## [Unreleased]
2
2
 
3
+ ## [1.2.0] - 2026-09-02
4
+
5
+ - `allow_failure { ... }` (`LemansReport::Assertions`) for verification checks that are recorded in `checks.json` but do not grade the run.
6
+ - Add `inherit_from: ../bench.yml` and support per-task overrides via `bench.yml` in a task directory.
7
+ - Multistep tasks support.
8
+ - Support named Docker envs via `environment.profiles` and `environment: <name>` in a task's frontmatter.
9
+
3
10
  ## [1.1.0] - 2026-08-28
4
11
 
5
12
  - A `#<effort>` model suffix (`openrouter/openai/gpt-5.6-luna#xhigh`) pins the reasoning effort; results land in `<model>-<effort>/`.
data/README.md CHANGED
@@ -46,7 +46,7 @@ my-bench/
46
46
  ├── hello-world/
47
47
  │ ├── instruction.md # what the agent is asked to do; YAML frontmatter carries
48
48
  │ │ # name, description, difficulty, tags, metadata — plus the
49
- │ │ # per-task overrides: setup, restore, verifier.setup
49
+ │ │ # per-task overrides: setup, restore, verifier.setup, environment
50
50
  │ ├── environment/Dockerfile # [optional] a task-specific sandbox image, instead of the shared one
51
51
  │ ├── environment.patch # [optional] task setup patch: applied and resealed as a fresh git repo at setup
52
52
  │ ├── verification_test.rb # grades the result
@@ -59,6 +59,9 @@ The `bench.yml` looks like this:
59
59
  ```yaml
60
60
  version: 1
61
61
 
62
+ # inherit_from: ../bench.yml # [optional] start from another bench.yml; the sections below are
63
+ # # deep-merged over it (lists replace; set a key to ~ to drop it)
64
+
62
65
  # setup: # [optional] sandbox preparation, run before the agent starts
63
66
  # files: [fixtures/seed.sql] # uploaded for the commands to consume, then wiped
64
67
  # commands: [bin/sandbox-setup]
@@ -68,6 +71,11 @@ environment:
68
71
  # backend: "daytona" # or "docker"
69
72
  # dockerfile: environment/Dockerfile # the default — or pin a published image instead:
70
73
  # image: ghcr.io/acme/my-bench@sha256:...
74
+ # profiles: # [optional] named alternatives to the shared image; a task
75
+ # campfire: # picks one with `environment: campfire` in its frontmatter
76
+ # dockerfile: docker/campfire/Dockerfile
77
+ # fizzy:
78
+ # image: ghcr.io/acme/fizzy@sha256:...
71
79
  resources: { cpus: 2, memory: 2GB, storage: 5GB }
72
80
  build_timeout: 10m
73
81
  network:
@@ -134,7 +142,7 @@ A minimal task example—checking whether an agent can write "Hello, world" into
134
142
  +Hello, world
135
143
  ```
136
144
 
137
- - `verification_test.rb`: a Ruby test that grades the solution (pass → 1, fail → 0; for partial credit, write a float in 0.0..1.0 to `$LOGS/reward.txt` instead). We use Minitest:
145
+ - `verification_test.rb`: a Ruby test that grades the solution (pass → 1, fail → 0; for partial credit, write a float in 0.0..1.0 to `$LOGS/reward.txt` instead). A check worth recording but not grading goes into `allow_failure { ... }` (`include LemansReport::Assertions`): a failed assertion inside lands in `checks.json` as `fail (allowed)` with its message and leaves the reward alone, while errors and skips stay hard failures. We use Minitest:
138
146
 
139
147
  ```ruby
140
148
  require "minitest/autorun"
@@ -162,6 +170,8 @@ verifier:
162
170
 
163
171
  An `environment.patch` next to `instruction.md` is always applied, declared or not.
164
172
 
173
+ A heavy task may need more than the bench-wide budgets: a `bench.yml` in the task directory is deep-merged over the bench's (as if it had `inherit_from` pointing at it) and applies to that task's trials only — say, `agent: { timeout: 2h, step_limit: 300 }`.
174
+
165
175
  ### 3. Set credentials
166
176
 
167
177
  ```bash
@@ -14,10 +14,16 @@ module Lemans
14
14
  PATCH = "solution.patch"
15
15
 
16
16
  def run(task, environment)
17
- raise ConfigError, "#{task.name}: no solution/ to run — the oracle has nothing to prove" unless task.solution?
17
+ files = task.solution_files
18
+ if files.empty?
19
+ raise ConfigError, "#{task.name}: no solution/ to run — the oracle has nothing to prove" if
20
+ task.verifiable? && !task.solution_applied_earlier?
18
21
 
19
- upload_solution(environment, task)
20
- outcome = environment.exec(command_for(task), timeout:)
22
+ return Response.new(outcome: Result::Outcome.new(:completed), usage: Result::Usage.zero)
23
+ end
24
+
25
+ upload_solution(environment, files)
26
+ outcome = environment.exec(command_for(task, files), timeout:)
21
27
 
22
28
  unless outcome.success?
23
29
  raise InfrastructureError,
@@ -32,8 +38,8 @@ module Lemans
32
38
 
33
39
  # An entrypoint ships only when applying the golden patch is not enough: an executable
34
40
  # `solve` (its shebang picks the language) or solve.sh; otherwise the bare patch is applied.
35
- def command_for(task)
36
- shipped = task.solution_files.map(&:last)
41
+ def command_for(task, files)
42
+ shipped = files.map(&:last)
37
43
  # An upload promises no mode bit, so the executable gets its own.
38
44
  return "chmod +x #{REMOTE_DIR}/#{SOLVE} && #{REMOTE_DIR}/#{SOLVE}" if shipped.include?(SOLVE)
39
45
  return "bash #{REMOTE_DIR}/#{ENTRYPOINT}" if shipped.include?(ENTRYPOINT)
@@ -43,8 +49,8 @@ module Lemans
43
49
  "cd #{Shellwords.escape(task.environment.workdir)} && git apply --binary --whitespace=nowarn #{REMOTE_DIR}/#{PATCH}"
44
50
  end
45
51
 
46
- def upload_solution(environment, task)
47
- task.solution_files.each do |local, remote|
52
+ def upload_solution(environment, files)
53
+ files.each do |local, remote|
48
54
  environment.upload(local, "#{REMOTE_DIR}/#{remote}")
49
55
  end
50
56
  end
@@ -37,5 +37,10 @@ tasks/<name>/
37
37
  ```
38
38
 
39
39
  - `hello-world` — proves the image and the grading pipeline end to end
40
- - `example-task` — a FizzBuzz-style challenge showing the per-task `setup` and
41
- `restore` overrides and tamper-proof grading
40
+ - `example-task` — a two-step FizzBuzz challenge (`multistep: true`: implement,
41
+ then golf it under a size budget) showing the per-task `setup` and `restore`
42
+ overrides, tamper-proof grading, and indexed step files
43
+ (`verification_test.1.rb`, `solution.1.patch`, `solution.2.patch`). Steps that
44
+ share helpers keep them in one `tests/` directory: everything unindexed in it
45
+ ships with every verified step, `tests/verification_test.N.rb` grades step N,
46
+ and `tests/verification_test.rb` the final one
@@ -3,6 +3,12 @@
3
3
  name: example-task
4
4
  description: Classic FizzBuzz under a source-size budget
5
5
  difficulty: medium
6
+ # A multistep task: the instruction sections below are separated by `---`;
7
+ # the first is a shared preamble, each following one is a step. Every step
8
+ # runs a fresh agent session. Step files carry the step index: verification_test.1.rb
9
+ # grades step 1 on the spot, solution.N.patch is step N's solution applied on top of
10
+ # the previous one (a lone solution.patch would be the whole task's solution, applied before step 1).
11
+ multistep: true
6
12
  tags: [ruby, golf]
7
13
  metadata:
8
14
  category: example
@@ -19,14 +25,19 @@ setup:
19
25
  # before grading: rewriting expected.txt buys the agent nothing.
20
26
  restore: [expected.txt]
21
27
  ---
22
- `/app/fizzbuzz.rb` is supposed to print the classic FizzBuzz sequence, but
23
- today it only prints the numbers. Make `ruby fizzbuzz.rb` print, one per
24
- line, the numbers 1 through 100 with every multiple of 3 replaced by `Fizz`,
25
- every multiple of 5 by `Buzz`, and every multiple of both by `FizzBuzz` —
26
- exactly the contents of `expected.txt`.
28
+ You are working on `/app/fizzbuzz.rb`, a tiny Ruby program graded against
29
+ `/app/expected.txt`. The program must compute its output itself: it is graded
30
+ with `expected.txt` out of reach, so reading the file back is not an answer.
27
31
 
28
- Two constraints:
32
+ ---
33
+
34
+ `fizzbuzz.rb` is supposed to print the classic FizzBuzz sequence, but today
35
+ it only prints the numbers. Make `ruby fizzbuzz.rb` print, one per line, the
36
+ numbers 1 through 100 with every multiple of 3 replaced by `Fizz`, every
37
+ multiple of 5 by `Buzz`, and every multiple of both by `FizzBuzz` — exactly
38
+ the contents of `expected.txt`.
39
+
40
+ ---
29
41
 
30
- - `fizzbuzz.rb` must stay at most 150 bytes keep it tight;
31
- - the program must compute the sequence itself: it is graded with
32
- `expected.txt` out of reach, so reading the file back is not an answer.
42
+ Now make it tight: shrink `fizzbuzz.rb` to at most 150 bytes without changing
43
+ what it prints.
@@ -0,0 +1,17 @@
1
+ diff --git a/fizzbuzz.rb b/fizzbuzz.rb
2
+ index 345b032..98ae4cf 100644
3
+ --- a/fizzbuzz.rb
4
+ +++ b/fizzbuzz.rb
5
+ @@ -1 +1,11 @@
6
+ -1.upto(100) { |i| puts i }
7
+ +1.upto(100) do |i|
8
+ + if i % 15 == 0
9
+ + puts "FizzBuzz"
10
+ + elsif i % 3 == 0
11
+ + puts "Fizz"
12
+ + elsif i % 5 == 0
13
+ + puts "Buzz"
14
+ + else
15
+ + puts i
16
+ + end
17
+ +end
@@ -0,0 +1,17 @@
1
+ diff --git a/fizzbuzz.rb b/fizzbuzz.rb
2
+ index 98ae4cf..2c15f7d 100644
3
+ --- a/fizzbuzz.rb
4
+ +++ b/fizzbuzz.rb
5
+ @@ -1,11 +1 @@
6
+ -1.upto(100) do |i|
7
+ - if i % 15 == 0
8
+ - puts "FizzBuzz"
9
+ - elsif i % 3 == 0
10
+ - puts "Fizz"
11
+ - elsif i % 5 == 0
12
+ - puts "Buzz"
13
+ - else
14
+ - puts i
15
+ - end
16
+ -end
17
+ +1.upto(100) { |i| s = "#{"Fizz" if i % 3 == 0}#{"Buzz" if i % 5 == 0}"; puts s.empty? ? i : s }
@@ -0,0 +1,31 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "minitest/autorun"
4
+ require "tmpdir"
5
+
6
+ # Step 1 is gated on correctness alone; the size budget is step 2's problem.
7
+ class FizzbuzzSequenceTest < Minitest::Test
8
+ SOURCE = "/app/fizzbuzz.rb"
9
+ EXPECTED = "/app/expected.txt"
10
+
11
+ def test_the_sequence_matches
12
+ expected = File.read(EXPECTED)
13
+
14
+ # The program must compute the sequence, not read it back: it runs from a
15
+ # scratch directory with expected.txt hidden away.
16
+ output = hiding(EXPECTED) do
17
+ Dir.mktmpdir { |scratch| Dir.chdir(scratch) { IO.popen([ "ruby", SOURCE ], &:read) } }
18
+ end
19
+
20
+ assert_equal expected, output
21
+ end
22
+
23
+ private
24
+
25
+ def hiding(path)
26
+ File.rename(path, "#{path}.hidden")
27
+ yield
28
+ ensure
29
+ File.rename("#{path}.hidden", path) if File.exist?("#{path}.hidden")
30
+ end
31
+ end
@@ -13,7 +13,7 @@ class FizzbuzzTest < Minitest::Test
13
13
  # The program must compute the sequence, not read it back: it runs from a
14
14
  # scratch directory with expected.txt hidden away.
15
15
  output = hiding(EXPECTED) do
16
- Dir.mktmpdir { |scratch| Dir.chdir(scratch) { IO.popen(["ruby", SOURCE], &:read) } }
16
+ Dir.mktmpdir { |scratch| Dir.chdir(scratch) { IO.popen([ "ruby", SOURCE ], &:read) } }
17
17
  end
18
18
 
19
19
  assert_equal expected, output
@@ -35,6 +35,18 @@ module Lemans
35
35
 
36
36
  def model = models.first
37
37
 
38
+ def to_h
39
+ {
40
+ "name" => name,
41
+ "model" => models,
42
+ "timeout" => timeout,
43
+ "step_limit" => step_limit,
44
+ "cost_limit" => cost_limit,
45
+ "exec_timeout" => exec_timeout,
46
+ "environment" => { "network" => environment.network.to_h }
47
+ }.compact
48
+ end
49
+
38
50
  Environment = Struct.new(:network, keyword_init: true)
39
51
 
40
52
  def initialize(name, model)
@@ -4,6 +4,7 @@ module Lemans
4
4
  class Config
5
5
  class Environment # :nodoc:
6
6
  Resources = Struct.new(:cpus, :memory, :storage, keyword_init: true)
7
+ Profile = Struct.new(:image, :dockerfile, keyword_init: true)
7
8
 
8
9
  class << self
9
10
  include Conversion
@@ -15,7 +16,8 @@ module Lemans
15
16
  conf.backend = data["backend"] if data["backend"]
16
17
  conf.image = data["image"] if data["image"]
17
18
  conf.dockerfile = data["dockerfile"] if data["dockerfile"]
18
- raise ConfigError, "environment.image and environment.dockerfile are mutually exclusive" if conf.image && conf.dockerfile
19
+
20
+ data["profiles"]&.each { |name, entry| conf.profiles[name] = profile!(name, entry) }
19
21
 
20
22
  conf.workdir = absolute_path!(data["workdir"]) if data["workdir"]
21
23
  conf.build_timeout = seconds!(data["build_timeout"]) if data["build_timeout"]
@@ -27,20 +29,44 @@ module Lemans
27
29
 
28
30
  conf
29
31
  end
32
+
33
+ private
34
+
35
+ def profile!(name, entry)
36
+ image, dockerfile = entry&.values_at("image", "dockerfile")
37
+ raise ConfigError, "environment.profiles.#{name}: image and dockerfile are mutually exclusive" if image && dockerfile
38
+ raise ConfigError, "environment.profiles.#{name} must declare image or dockerfile" unless image || dockerfile
39
+
40
+ Profile.new(image:, dockerfile:)
41
+ end
30
42
  end
31
43
 
32
44
  attr_accessor :image, :dockerfile, :workdir, :backend,
33
- :resources, :build_timeout, :network
45
+ :resources, :build_timeout, :network, :profiles
34
46
 
35
47
  def initialize
36
48
  @image = nil
37
49
  @dockerfile = nil
50
+ @profiles = {}
38
51
  @backend = "daytona"
39
52
  @workdir = "/app"
40
53
  @resources = Resources.new(cpus: 2, memory: 2048, storage: 5120)
41
54
  @build_timeout = 10 * 60
42
55
  @network = NetworkPolicy.new
43
56
  end
57
+
58
+ def to_h
59
+ {
60
+ "backend" => backend,
61
+ "image" => image,
62
+ "dockerfile" => dockerfile&.to_s,
63
+ "profiles" => profiles.transform_values { { "image" => it.image, "dockerfile" => it.dockerfile&.to_s }.compact },
64
+ "workdir" => workdir,
65
+ "build_timeout" => build_timeout,
66
+ "network" => network.to_h,
67
+ "resources" => resources.to_h.transform_keys(&:to_s)
68
+ }.compact
69
+ end
44
70
  end
45
71
  end
46
72
  end
@@ -33,6 +33,8 @@ module Lemans
33
33
  @hosts = hosts
34
34
  end
35
35
 
36
+ def to_h = { "mode" => mode, "hosts" => hosts }.compact
37
+
36
38
  # Backends allowlist domains and IP ranges through separate APIs.
37
39
  def domains = partitioned_hosts.last
38
40
 
@@ -13,7 +13,10 @@ module Lemans
13
13
  case data
14
14
  when Hash
15
15
  conf.commands = Array(data["commands"])
16
- conf.files = Array(data["files"]).map { [ root.join(it), it ] }
16
+ conf.files = Array(data["files"]).map do |entry|
17
+ local, remote = entry.is_a?(Array) ? entry : [ entry, entry ]
18
+ [ root.join(local), remote ]
19
+ end
17
20
  else
18
21
  conf.commands = Array(data)
19
22
  end
@@ -38,6 +41,8 @@ module Lemans
38
41
  end
39
42
 
40
43
  def empty? = files.empty? && commands.empty?
44
+
45
+ def to_h = { "commands" => commands, "files" => files.map { |local, remote| [ local.to_s, remote ] } }
41
46
  end
42
47
  end
43
48
  end
@@ -24,6 +24,7 @@ module Lemans
24
24
  conf.preverify = data["preverify"] if data["preverify"]
25
25
  conf.restore_paths = restore_paths!(data["restore"]) if data["restore"]
26
26
  conf.logs_dir = absolute_path!(data["logs_dir"]) if data["logs_dir"]
27
+ conf.verification = root.join(data["verification"]) if data["verification"]
27
28
 
28
29
  conf
29
30
  end
@@ -41,31 +42,41 @@ module Lemans
41
42
  end
42
43
  end
43
44
 
44
- attr_accessor :root, :timeout, :setup, :command, :preverify, :restore_paths, :logs_dir
45
+ attr_accessor :timeout, :setup, :command, :preverify, :restore_paths, :logs_dir, :verification
45
46
 
46
47
  def initialize
47
- @root = Pathname("./")
48
48
  @timeout = 10 * 60
49
49
  @setup = Setup.new
50
50
  @command = DEFAULT_COMMAND
51
51
  @preverify = nil
52
52
  @restore_paths = []
53
53
  @logs_dir = "/logs/verifier"
54
+ @verification = nil
54
55
  end
55
56
 
56
57
  def reward_path = "#{logs_dir.chomp("/")}/reward.txt"
57
58
 
59
+ def to_h
60
+ {
61
+ "timeout" => timeout,
62
+ "setup" => setup.to_h,
63
+ "command" => command,
64
+ "preverify" => preverify,
65
+ "restore" => restore_paths,
66
+ "logs_dir" => logs_dir,
67
+ "verification" => verification&.to_s
68
+ }.compact
69
+ end
70
+
58
71
  # [absolute, remote-relative] pairs. Shared verification files grade
59
72
  # every trial, so they ship alongside each task's own tests.
60
73
  def files
61
- @files ||= begin
62
- dir = root.join(VERIFICATION_DIR)
63
- if dir.directory?
64
- dir.glob("**/*", File::FNM_DOTMATCH).select(&:file?).map { [ it, it.relative_path_from(dir).to_s ] }
74
+ @files ||=
75
+ if verification&.directory?
76
+ verification.glob("**/*", File::FNM_DOTMATCH).select(&:file?).map { [ it, it.relative_path_from(verification).to_s ] }
65
77
  else
66
78
  []
67
79
  end
68
- end
69
80
  end
70
81
  end
71
82
  end
data/lib/lemans/config.rb CHANGED
@@ -7,18 +7,24 @@ require "pathname"
7
7
  module Lemans
8
8
  # Benchmark configuration and task definitions container.
9
9
  class Config
10
- attr_reader :root, :config_path, :tasks_dir,
11
- :tasks, :version,
10
+ attr_reader :root, :config_path, :parent, :tasks_dir, :version,
12
11
  :backend, :concurrency, :attempts
13
12
 
14
13
  # Nested configs
15
14
  attr_reader :environment, :agent, :verifier, :setup
16
15
 
16
+ using Ext::DeepMerge
17
+
18
+ FILENAMES = %w[bench.yaml bench.yml].freeze
19
+
17
20
  class << self
18
- def load_file(path)
21
+ # A task directory's bench.yml is loaded with the bench as its `parent`:
22
+ # inherit_from is implied.
23
+ def load_file(path, parent: nil)
24
+ path = File.dirname(path) if File.file?(path)
19
25
  raise ConfigError, "bench directory not found: #{path}" unless File.directory?(path)
20
26
 
21
- config_name = %w[bench.yaml bench.yml].find { File.file?(File.join(path, it)) }
27
+ config_name = FILENAMES.find { File.file?(File.join(path, it)) }
22
28
 
23
29
  raise ConfigError, "bench.yml not found" unless config_name
24
30
 
@@ -29,6 +35,17 @@ module Lemans
29
35
 
30
36
  root = Pathname(path)
31
37
 
38
+ parent = load_file(root.join(contents["inherit_from"])) if contents["inherit_from"]
39
+
40
+ if parent
41
+ # Local conventions win over the inherited config (unless declared, even as ~)
42
+ environment = (contents["environment"] ||= {})
43
+ environment["dockerfile"] = root.join("environment/Dockerfile").to_s if root.join("environment/Dockerfile").file? && !environment.key?("dockerfile")
44
+ verifier = (contents["verifier"] ||= {})
45
+ verifier["verification"] = root.join(Verifier::VERIFICATION_DIR).to_s if root.join(Verifier::VERIFICATION_DIR).directory? && !verifier.key?("verification")
46
+ contents = parent.to_h.deep_merge(contents)
47
+ end
48
+
32
49
  sections = {}
33
50
  sections[:version] = contents["version"]
34
51
  sections[:tasks_dir] = contents["tasks"]
@@ -44,17 +61,18 @@ module Lemans
44
61
  sections[:setup].commands = Array(commands) + sections[:setup].commands
45
62
  end
46
63
 
47
- new(root, config_path:, **sections)
64
+ new(root, config_path:, parent:, **sections)
48
65
  rescue Psych::Exception => e
49
66
  raise ConfigError, "#{path}: #{e.message}"
50
67
  end
51
68
  end
52
69
 
53
- def initialize(root = Pathname("./"), config_path: Pathname("./bench.yml"), version: nil, tasks_dir: "tasks",
70
+ def initialize(root = Pathname("./"), config_path: Pathname("./bench.yml"), parent: nil, version: nil, tasks_dir: "tasks",
54
71
  setup: nil, agent: Agent.new("miniswen", "openrouter/z-ai/glm-5.2"),
55
72
  environment: Environment.new, verifier: Verifier.new)
56
73
  @root = root
57
74
  @config_path = config_path
75
+ @parent = parent
58
76
  @version = version
59
77
  @tasks_dir = root.join(tasks_dir)
60
78
  @setup = setup || Setup.new
@@ -62,26 +80,41 @@ module Lemans
62
80
  @environment = environment
63
81
  @environment.dockerfile = root.join(@environment.dockerfile) if @environment.dockerfile
64
82
  @environment.dockerfile ||= default_dockerfile unless @environment.image
83
+ @environment.profiles.each_value { it.dockerfile = root.join(it.dockerfile) if it.dockerfile }
65
84
  @verifier = verifier
66
- @verifier.root = root
85
+ @verifier.verification ||= root.join(Verifier::VERIFICATION_DIR)
67
86
  @concurrency = 4
68
87
  @attempts = 1
69
88
  @backend = @environment.backend
70
- @tasks = parse_tasks
71
89
  end
72
90
 
91
+ def tasks = @tasks ||= parse_tasks
92
+
73
93
  def load_options(agent: nil, model: nil, attempts: nil, concurrency: nil, backend: nil, **)
74
94
  @agent.name = agent if agent
75
95
  @agent.models = Array(model) if model
76
96
  @attempts = attempts if attempts
77
97
  @concurrency = concurrency if concurrency
78
98
  @backend = environment.backend = backend if backend
99
+ tasks.each { it.config.load_options(agent:, model:, attempts:, concurrency:, backend:) unless it.config.equal?(self) }
79
100
  end
80
101
 
81
102
  def agent_name = agent.name
82
103
 
83
104
  def models = agent.models
84
105
 
106
+ # The bench.yml this config reads as, paths resolved: what a bench
107
+ # inheriting from it merges over. Tasks are a bench's own.
108
+ def to_h
109
+ {
110
+ "version" => version,
111
+ "setup" => setup.to_h,
112
+ "agent" => agent.to_h,
113
+ "environment" => environment.to_h,
114
+ "verifier" => verifier.to_h
115
+ }.compact
116
+ end
117
+
85
118
  # Resolved once: an hours-long run reports the bench it started from, not
86
119
  # later tree drift.
87
120
  def revision
@@ -92,9 +125,11 @@ module Lemans
92
125
  def digest
93
126
  @digest ||= begin
94
127
  sha = Digest::SHA256.new
95
- sha << TreeDigest.call(root.join(Verifier::VERIFICATION_DIR))
128
+ sha << TreeDigest.call(verifier.verification)
96
129
  sha << TreeDigest.call(environment.dockerfile.dirname) if environment.dockerfile
130
+ environment.profiles.each_value { sha << TreeDigest.call(it.dockerfile.dirname) if it.dockerfile }
97
131
  sha << Digest::SHA256.file(config_path.to_s).hexdigest if config_path.file?
132
+ sha << parent.digest if parent
98
133
  sha.hexdigest[0, 16]
99
134
  end
100
135
  end
@@ -152,7 +152,7 @@ module Lemans
152
152
  end
153
153
 
154
154
  status = wait.value
155
- [ timed_out ? 124 : (status.exitstatus || 1), output.scrub ]
155
+ [ timed_out ? 124 : (status.exitstatus || 1), output.force_encoding(Encoding::UTF_8).scrub ]
156
156
  end
157
157
  rescue Errno::ENOENT
158
158
  raise ConfigError, "docker: CLI not found — install Docker or pick another backend"
@@ -0,0 +1,13 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Lemans
4
+ module Ext
5
+ module DeepMerge
6
+ refine Hash do
7
+ def deep_merge(other)
8
+ merge(other) { |_, base, over| base.is_a?(Hash) && over.is_a?(Hash) ? base.deep_merge(over) : over }
9
+ end
10
+ end
11
+ end
12
+ end
13
+ end