llm-experiment 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 6bb9441528decd9413786b8a2da04d71bafe0b515183d26fbf90e0c36d5e7cb5
+  data.tar.gz: 7bf9c1e2ad2be746c967bf6ebd010c034d9707379f4d1a2fc2a9ae33e462e416
+SHA512:
+  metadata.gz: 85312ed37cbbe34223c90ced1b91805fdf7e80bdec0e252ac1a7962dd4a46b1ef3efd18ac8bfe01796d8bde3943bdf4afd7c95309a137d84388a6650ae262c43
+  data.tar.gz: '039e17b7b29ec5728c8f2c0b6f07ab7de956cbfbc06b0aed6a4a9380e50f113b017d6c28b228d9b8402598393ac69d24a164e93127367bfd7d055b220fafb798'

data/CHANGELOG.md

@@ -0,0 +1,40 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-08-30
+
+### Added
+
+- `llmx new` — scaffold an experiment directory.
+- `llmx doctor` — preflight the `container` CLI, apiserver, builder envelope, disk, auth store, images, and prompts.
+- `llmx build base` — build and smoke-check the base image from pinned toolchains.
+- `llmx build app KEY` — build a per-app image from a git bundle, with history flattened.
+- `llmx login` — interactive plan-subscription login into the host auth store.
+- `llmx run` — one trial, or the whole grid with `--all`.
+- `llmx status` — grid completion per task, agent, and condition.
+- `llmx shell [APP]` — interactive shell in a trial-shaped container.
+- `llmx parse` — transcripts to `events.jsonl` and `metrics.json`, with hermeticity warnings.
+- `llmx metrics` — medians and exact Mann-Whitney U, never pooled across agent or app.
+- `llmx sanitize` — the gate from `results-raw/` to the committed `results/`.
+- `llmx clean` — disk reclamation verified by observation, with the kill-the-VM recovery documented in its output.
+- `llmx version` — gem version and default toolchain pins.
+
+### Notable properties
+
+- Zero runtime dependencies. Stdlib only.
+- Every `container` state change is verified by observation, never by exit code.
+- Image builds refuse to start without room: 10 GB for the base, `LLMX_MIN_FREE_GB` (25 by default) for an app. Measured against a real Rails build that consumed about 24 GB, most of it the builder VM.
+- The builder is provisioned with 6 CPUs and 8 GB, because a starved builder dies mid-build with no error. An existing builder is reused only when it meets both dimensions; CPUs alone accepted a builder with 6 CPUs and 2 GB.
+- Credentials never enter an image. Each trial copies them out of the mount and repoints the CLI at the copy.
+- Prompts cross into a container base64-encoded, never through shell interpolation.
+- App images carry only orphan `trial/*` branches; the build fails if any other commit message survives.
+- A trial aborts (exit 3) unless the suite ran and failed before the agent started.
+- Trials run sequentially, blocked by task, condition order rotated, resumable from `meta.json`.
+- Every trial records requested model, reported model, CLI versions, and gem version.
+- `llmx sanitize` refuses rather than redacts, and publishes raw transcripts only for apps marked `publish_transcripts: true`.
+
+[0.1.0]: https://github.com/lucianghinda/llm-experiment/releases/tag/v0.1.0

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at dev@ghinda.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Lucian Ghinda
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,190 @@
+# llm-experiment
+
+Run controlled, reproducible experiments on coding agents
+
+Each trial runs in a fresh container with no host state. The host only orchestrates.
+
+Supports Claude Code and Codex CLI.
+
+## How it works
+
+`llmx` runs repeatable coding-agent experiments inside disposable containers. Each trial starts from the same code and task. It records the agent's work, reruns the tests, and collects comparable results.
+
+1. Define tasks, agents, and conditions in `experiment.yml`.
+2. Build pinned container images for reproducibility.
+3. Generate the task × agent × condition trial matrix.
+4. Run every trial in a clean, isolated container.
+5. Capture transcripts, diffs, test results, and metrics.
+6. Compare conditions and export sanitized, publishable results.
+
+## Requirements
+
+- macOS on Apple silicon, using Apple's container runtime
+- The `container` CLI installed and available on your `PATH` (verified with 0.5.0)
+- Ruby 3.4+
+
+The gem has no runtime dependencies.
+
+## Installation
+
+Run:
+
+```sh
+gem install llm-experiment
+```
+
+This installs the `llmx` command.
+
+## Quick start
+
+```sh
+llmx new my-question
+cd my-question
+$EDITOR experiment.yml
+export LLMX_APP_CAMPFIRE=/path/to/checkout
+llmx doctor
+llmx build base
+llmx login
+llmx build app campfire
+llmx run --all --dry-run
+llmx run --all
+llmx parse --all
+llmx metrics
+llmx sanitize
+```
+
+Write your prompts to `prompts/<app>/<task-id>/<condition>.txt` before running.
+
+The gem runs trials. It does not write your prompts or plant your tasks.
+
+## Commands
+
+Every command takes `--help`, `--experiment DIR`, and `--verbose`. Use `--verbose` to echo each container command; it is off by default because a grid runs hundreds of them.
+
+Command | Description
+--- | ---
+`llmx new NAME` | Scaffold an experiment directory
+`llmx doctor` | Check CLI, services, disk, auth, images, and prompts
+`llmx build base` | Build and smoke-check the base image
+`llmx build app KEY` | Build a per-app image from a git bundle
+`llmx login` | Log the agent CLIs in to your plan subscriptions
+`llmx run` | Run one trial, or the whole grid with `--all`
+`llmx status` | Show which grid cells have results
+`llmx shell [APP]` | Open a shell in a trial-shaped container
+`llmx parse` | Turn transcripts into `events.jsonl` and `metrics.json`
+`llmx metrics` | Report medians and exact Mann-Whitney U
+`llmx sanitize` | Gate `results-raw/` into the committed `results/`
+`llmx clean` | Reclaim disk, verified by observation
+`llmx version` | Show the gem version and the default pins
+
+Destructive commands take `--dry-run`. The grid resumes by default, so pass `--redo` to re-run a finished cell.
+
+## experiment.yml
+
+One file configures the experiment.
+
+```yaml
+name: at-file-mentions
+question: "Does @path change what an agent does and what it costs?"
+
+# Optional. Defaults ship with the gem; `llmx version` prints them.
+pins:
+  claude_code: "2.1.233"
+  codex: "0.147.0"
+  opencode: "1.18.15"
+  ruby_versions: ["3.4.5", "4.0.1"]
+  node: "22"
+
+# Trial agents. Only claude and codex are supported.
+agents:
+  claude:
+    model: claude-opus-5
+  codex:
+    model: gpt-5.6-sol
+
+# One prompt variant per condition.
+conditions: [none, bare_noline, at_noline]
+
+trial:
+  timeout_seconds: 900
+  memory: 6g              # a unit suffix is required
+  cpus: 4
+
+apps:
+  campfire:
+    ruby: "3.4.5"
+    bundler: "4.0.13"                                 # or "default"
+    database: sqlite3                                 # sqlite3 or postgresql
+    test_command: bin/rails test
+    db_prepare: bin/rails db:test:prepare
+    suite_ran_pattern: '\d+ runs?, \d+ assertions?'   # default shown
+    publish_transcripts: true                         # false for private code
+    branch_prefix: exp/path-hints                     # source branches on the host
+    neutralize: [CLAUDE.md, AGENTS.md, .mcp.json, .claude]
+
+tasks:
+  - id: campfire-01
+    app: campfire
+    test_file: test/lib/restricted_http/private_network_guard_test.rb
+    impl_files: [lib/restricted_http/private_network_guard.rb]
+```
+
+Unknown keys are errors. A typo that silently does nothing is how an experiment measures the wrong thing.
+
+`postgresql` is available in the config format, but that full in-container path has not yet been validated end to end against a real app. Prefer `sqlite3` unless you plan to verify the PostgreSQL path yourself.
+
+Host checkout paths never go in this file. They come from `LLMX_APP_<KEY>`.
+
+Set one per app:
+
+```sh
+export LLMX_APP_CAMPFIRE=/path/to/campfire
+```
+
+The image builder expects two branch names per app: `<branch_prefix>/base` and `<branch_prefix>/<task-id>`.
+
+## How a trial stays clean
+
+A trial measures the agent, not the machine it ran on.
+
+- Every trial gets a fresh container. Nothing survives it.
+- No host state crosses: no user `CLAUDE.md`, no MCP servers, no hooks, no project memory.
+- Credentials live in `~/.llmx/auth` and are mounted, never baked into an image.
+- The mount is a seed, not a home. Each trial copies the credential files and repoints `CLAUDE_CONFIG_DIR` or `CODEX_HOME` at the copy.
+- Codex also runs with `--ignore-user-config --ephemeral`.
+- Application code enters as a git bundle, so untracked secrets and local bundler overrides stay behind.
+- History flattens to one anonymous orphan commit per branch, named "Import application source". The build fails if any other message survives.
+- Coaching files listed under `neutralize` are deleted during the image build.
+- Prompts cross as base64 in `LLMX_PROMPT_B64`, never as shell text.
+- A trial aborts with exit 3 unless the suite ran and failed before the agent started.
+- `llmx parse` exits non-zero on any trial that reports an MCP server, or a memory path inside the shared mount. It checks where a path points, not whether one exists. Pass `--lenient` to report and continue.
+
+## Disk
+
+Budget about 10 GB for the base image and 4-8 GB per app image. The BuildKit cache reached 9.8 GB building one Rails app.
+
+Plan against the unpacked snapshot, not the sizes `container image inspect` reports. Deleting three app images returned about 16 GB against manifests summing to 1.2-1.6 GB each.
+
+Run `llmx clean` for a report that measures the snapshot:
+
+```sh
+llmx clean                  # report only, deletes nothing
+llmx clean --images         # every llmx-app-* image, about 5 GB each
+llmx clean --builder        # the BuildKit cache, which is regenerable
+```
+
+Both `llmx build base` and `llmx build app` refuse to start without room: 10 GB for the base, `LLMX_MIN_FREE_GB` (25 by default) for an app. Running out mid-build does not fail cleanly. The 25 is measured, not guessed — a Rails app image took about 24 GB, roughly 17 GB of it the BuildKit builder VM rather than the image.
+
+`llmx clean --builder` verifies with `container ls -a` afterwards. `container builder stop` can exit 0 and do nothing, so the exit code is never the evidence.
+
+## Private code
+
+Images built from private repositories are never pushed to a registry.
+
+`llmx sanitize` is the gate between `results-raw/` and the committed `results/`. It refuses the whole publish on a host path or anything shaped like a secret. It refuses rather than redacts, because a surprise in a transcript should stop the commit.
+
+Set `publish_transcripts: false` for a private app. Only derived measurements cross; raw transcripts and diffs stay in `results-raw/`, which is gitignored.
+
+## License
+
+MIT

data/exe/llmx

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative "../lib/llm-experiment"
+
+exit LLMExperiment::CLI.run(ARGV)

data/lib/llm-experiment.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative "llm_experiment"

data/lib/llm_experiment/auth.rb

@@ -0,0 +1,143 @@
+# frozen_string_literal: true
+
+module LLMExperiment
+  # One-time interactive step: log the agent CLIs in to the user's plan
+  # subscriptions, and keep the credentials in a directory that every later
+  # trial mounts read-write.
+  #
+  # Nothing is baked into an image and no API key is required. Re-run this when
+  # a refresh token expires; `llmx run` stops with that instruction rather than
+  # burning trials against a dead login.
+  #
+  # Claude and Codex are pointed at the credential store through
+  # CLAUDE_CONFIG_DIR and CODEX_HOME, so their logins land there directly.
+  # opencode takes no such variable: it keeps credentials in XDG_DATA_HOME next
+  # to its sessions and its database, so logging in writes a whole state
+  # directory. This points XDG_DATA_HOME at a container-local directory that
+  # dies with the container and copies out the one credential file afterwards,
+  # which gets opencode the same shape as the other two without carrying its
+  # session store along.
+  class Auth
+    AGENTS = %w[claude codex opencode].freeze
+
+    def initialize(container: nil, shell: Shell)
+      @container = container || Container.new
+      @shell = shell
+    end
+
+    def login(agents: AGENTS)
+      wanted = wanted_agents(agents)
+      image = ready_image!
+
+      print_intro(wanted)
+      # Interactive: hand the terminal to the container so the device flows work.
+      @shell.interactive(*run_argv(image, login_script, interactive: true))
+
+      puts "\nVerifying what persisted..."
+      @shell.sh(*run_argv(image, verify_script), allow_failure: true)
+      print_persisted
+    end
+
+    def verify
+      @shell.sh(*run_argv(ready_image!, verify_script))
+    end
+
+    private
+
+    def wanted_agents(agents)
+      wanted = Array(agents).map(&:to_s)
+      wanted = AGENTS.dup if wanted.empty?
+      unknown = wanted - AGENTS
+      raise Error, "unknown agent: #{unknown.join(", ")} (supported: #{AGENTS.join(", ")})" if unknown.any?
+
+      wanted
+    end
+
+    def ready_image!
+      @container.ensure_system!
+      image = LLMExperiment.base_image
+      raise Error, "base image #{image} not found; run `llmx build base` first" unless @container.image?(image)
+
+      image
+    end
+
+    def run_argv(image, script, interactive: false)
+      @container.run_argv(image, script,
+                          memory: LLMExperiment.default_memory,
+                          cpus: LLMExperiment.default_cpus,
+                          interactive: interactive)
+    end
+
+    # opencode's credentials live in XDG_DATA_HOME with its sessions and
+    # database, so the login runs against a directory inside the container and
+    # only auth.json is copied back out to the mounted store.
+    def opencode_login_home = "/home/#{LLMExperiment.agent_user}/.opencode-login"
+    def opencode_auth_out = "/home/#{LLMExperiment.agent_user}/.agent-auth/opencode/auth.json"
+
+    def verify_script
+      <<~SH
+        echo "--- claude ---"
+        claude auth status 2>&1 | head -5 || echo "claude: no status subcommand on this build"
+        echo "--- codex ---"
+        codex login status 2>&1 | head -5 || echo "codex: not logged in"
+        echo "--- opencode ---"
+        if [ -s "#{opencode_auth_out}" ]; then
+          echo "opencode: auth.json present ($(wc -c < "#{opencode_auth_out}") bytes)"
+        else
+          echo "opencode: no credential file in the store"
+        fi
+      SH
+    end
+
+    # `exec bash` would replace this shell and the copy-out would never run, so
+    # the shell is called and the file is moved after it returns.
+    def login_script
+      <<~SH
+        export XDG_DATA_HOME=#{opencode_login_home}
+        mkdir -p "$XDG_DATA_HOME"
+        echo "CLAUDE_CONFIG_DIR=$CLAUDE_CONFIG_DIR"
+        echo "CODEX_HOME=$CODEX_HOME"
+        echo "XDG_DATA_HOME=$XDG_DATA_HOME  (opencode; dies with this container)"
+        echo
+        bash
+        if [ -s "$XDG_DATA_HOME/opencode/auth.json" ]; then
+          mkdir -p "$(dirname #{opencode_auth_out})"
+          cp "$XDG_DATA_HOME/opencode/auth.json" "#{opencode_auth_out}"
+          chmod 600 "#{opencode_auth_out}"
+          echo "copied opencode auth.json into the credential store"
+        fi
+      SH
+    end
+
+    def print_intro(wanted)
+      puts <<~INTRO
+
+        Logging the agent CLIs in to your plan subscriptions.
+
+        A container will open with a shell. Run the login command(s) shown below and
+        complete the browser device flow, then type `exit`.
+
+        Credentials are written to #{LLMExperiment.auth_dir} on this Mac and mounted into every
+        trial. They never enter an image and are never committed.
+
+      INTRO
+
+      steps = []
+      if wanted.include?("claude")
+        steps << "claude    ->  run: claude   (then /login, complete the browser flow, then /exit)"
+      end
+      steps << "codex     ->  run: codex login" if wanted.include?("codex")
+      steps << "opencode  ->  run: opencode auth login" if wanted.include?("opencode")
+      puts steps.map { |s| "  #{s}" }.join("\n")
+      puts "\n  When you are done, type: exit\n\n"
+    end
+
+    def print_persisted
+      counts = AGENTS.map do |name|
+        "#{name}/ has #{Dir.glob(File.join(LLMExperiment.auth_dir, name, "**", "*")).size} file(s)"
+      end
+      puts "\n#{LLMExperiment.auth_dir}: #{counts.join(", ")}"
+      puts "If one is empty, that login did not persist - re-run `llmx login --agent NAME` for it."
+    end
+  end
+end

data/lib/llm_experiment/cleaner.rb

@@ -0,0 +1,183 @@
+# frozen_string_literal: true
+
+module LLMExperiment
+  # Disk reclamation, planned and verified by observation.
+  #
+  # Two measured facts shape this class. First, what an image costs is its
+  # unpacked snapshot, not the layer sizes `container image inspect` reports:
+  # deleting three app images returned about 16 GB against the 1.2-1.6 GB the
+  # manifests sum to. So the plan comes from `du` on the data directory.
+  #
+  # Second, `container builder stop` can report success and do nothing. On
+  # 0.5.0, with a builder up for three days, `builder stop`, `delete --force
+  # buildkit` and `system stop` all exited 0 while `container ls -a` went on
+  # reporting the builder as running and its 6.8 GB directory stayed put. So
+  # the builder is checked afterwards, and an exit code is never the evidence.
+  class Cleaner
+    UNITS = { "B" => 1, "K" => 1024, "M" => 1024**2, "G" => 1024**3, "T" => 1024**4 }.freeze
+
+    APP_PREFIX = "llmx-app-"
+    BASE_PREFIX = "llmx-base"
+
+    # Roughly what one app image costs unpacked, measured on 0.5.0.
+    APP_IMAGE_GB = 5
+
+    SNAPSHOT_NOTE = <<~NOTE
+      Sizes above are the unpacked snapshot, which is what an image actually
+      costs. `container image inspect` sums layers instead and reports about a
+      third of it: three app images that summed to 1.2-1.6 GB each returned
+      about 16 GB when deleted. Plan against this listing, not the manifest.
+    NOTE
+
+    # Printed instead of run. The runtime line is unambiguous because it names
+    # `--uuid buildkit`, but the VM line is a bare Virtualization.framework XPC
+    # service that looks identical for any VM on the Mac, so a pattern-kill on
+    # the name can take out Docker, UTM, Parallels or a Simulator.
+    RECOVERY = <<~TEXT
+      The builder is still running. Its exit codes lied; end the VM directly:
+
+        ps -eo pid,etime,command | grep -E 'container-runtime-linux|Virtualization' | grep -v grep
+        kill -TERM <runtime-pid> <vm-pid>
+        container delete --force buildkit
+
+      Read that `ps` output before killing anything. The runtime line names
+      `--uuid buildkit`, so it is unambiguous. The VM line is a bare
+      `Virtualization.framework` XPC service and looks identical for ANY VM on
+      this Mac (Docker, UTM, Parallels, a Simulator). Match it to the runtime
+      process; never pattern-kill on the name.
+
+      A plain TERM was enough when this happened; -9 was not needed. The
+      apiserver does not need to be touched. The builder is recreated on the
+      next build, which re-downloads the builder image once.
+    TEXT
+
+    def initialize(container:, shell: Shell)
+      @container = container
+      @shell = shell
+    end
+
+    # Never deletes anything. Shows what is on disk and what it would cost to
+    # get back.
+    def report
+      puts "container data: #{data_dir}"
+      print_disk
+      puts
+      print_images
+      puts
+      puts SNAPSHOT_NOTE
+      puts "  llmx clean --builder    the BuildKit cache; regenerable, often the largest item"
+      puts "  llmx clean --images     every #{APP_PREFIX}* image, about #{APP_IMAGE_GB} GB each unpacked"
+      puts "  add --dry-run to either to see the list without deleting"
+    end
+
+    # App images only. The base image takes 20-40 minutes to rebuild, so it
+    # goes only when asked for by name.
+    def clean_images(dry_run:, base: false)
+      targets = app_images
+      targets += base_images if base
+      if targets.empty?
+        puts "no llmx images to delete"
+        return
+      end
+
+      if dry_run
+        targets.each { |image| puts "would delete #{image} (about #{APP_IMAGE_GB} GB unpacked)" }
+        puts "nothing deleted (--dry-run)"
+        return
+      end
+
+      before = @container.free_gb
+      targets.each do |image|
+        puts "deleting #{image}"
+        @container.delete_image(image)
+      end
+      after = @container.free_gb
+      puts "free space: #{before} GB -> #{after} GB"
+    end
+
+    # Stops the builder, then checks whether it actually stopped. The exit
+    # codes of the two commands are ignored on purpose.
+    def clean_builder(dry_run:)
+      if dry_run
+        puts "would run: container builder stop"
+        puts "would run: container delete --force buildkit"
+        puts "then verify with `container ls -a`, because those two can exit 0 and do nothing"
+        return
+      end
+
+      @shell.sh("container", "builder", "stop", allow_failure: true)
+      @shell.sh("container", "delete", "--force", "buildkit", allow_failure: true)
+      verify_builder_gone!
+    end
+
+    private
+
+    def data_dir = LLMExperiment.container_data_dir
+
+    def verify_builder_gone!
+      if builder_running?
+        puts
+        puts RECOVERY
+        raise Error, "builder still reported as running by `container ls -a` after stop and delete"
+      end
+
+      puts "builder gone (verified with `container ls -a`)"
+      puts "images are untouched; the builder is recreated on the next build"
+    end
+
+    def builder_running?
+      @container.containers_all.lines.any? { |line| line.match?(/^buildkit\b.*\brunning\b/) }
+    end
+
+    def app_images
+      @container.images.select { |image| image.start_with?(APP_PREFIX) }
+    end
+
+    def base_images
+      @container.images.select { |image| image.start_with?(BASE_PREFIX) }
+    end
+
+    def print_images
+      apps = app_images
+      puts "app images (#{apps.size}):"
+      if apps.empty?
+        puts "  none"
+      else
+        apps.each { |image| puts "  #{image}" }
+      end
+      base = base_images
+      puts "base images (kept unless you pass --base):"
+      puts(base.empty? ? "  none" : base.map { |image| "  #{image}" })
+    end
+
+    def print_disk
+      entries = disk_entries
+      if entries.empty?
+        puts "  nothing on disk, or the CLI keeps its data somewhere else"
+        puts "  (set LLMX_CONTAINER_DATA_DIR if this path is wrong)"
+        return
+      end
+
+      entries.each { |size, path| puts format("  %-8s %s", size, File.basename(path)) }
+    end
+
+    # `du -sh` on each entry, largest first. Human sizes are what a person
+    # compares against a disk, so they are parsed only to sort them.
+    def disk_entries
+      return [] unless Dir.exist?(data_dir)
+
+      paths = Dir.children(data_dir).sort.map { |child| File.join(data_dir, child) }
+      return [] if paths.empty?
+
+      rows = @shell.capture("du", "-sh", *paths, allow_failure: true).lines.filter_map do |line|
+        size, path = line.chomp.split("\t", 2)
+        [size, path] if path
+      end
+      rows.sort_by { |size, _path| -bytes(size) }
+    end
+
+    def bytes(size)
+      size.to_f * UNITS.fetch(size[-1].to_s.upcase, 1)
+    end
+  end
+end