judgelint 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c59739f54cb0b198f538842c411554b101b4a3a836f641f57e8f1ebd97fca0de
+  data.tar.gz: 42b6bec77cc90b1a2f94b4d8a0782531f21d97ec543ad83df0cebcf67ca83cc9
+SHA512:
+  metadata.gz: ee5af8e99281ac1f1946512a485171f6bf9ad3afdbc4eba3ba251f1694c5315668c0d5185895a057f0f38007c6451dff27f3fedf11635bde51dcace18326e188
+  data.tar.gz: e84ca3de964ffdf65a92f92d6ea095bf0282cc091e4c7b20183a30b50fdab65292e9140511e5e402ffe623962ff4c8b0456a562f06278cc75a8b5c69ea992ed8

data/CHANGELOG.md

@@ -0,0 +1,24 @@
+# Changelog
+
+## 0.1.0 (2026-10-23)
+
+- Specs as praxis reads them: a README (or `*.judgelint.md`) with `paths:`
+  frontmatter, each top-level bullet a rule. The model reads the spec's prose
+  with the file and is asked whether the code breaks each bullet and whether
+  it follows it; optional `violation:` and `compliant:` lines sharpen a
+  bullet. Question rules in `.judgelint/rules/` for conventions a bullet
+  can't carry.
+- Window, file, and directory scopes; `context:` files the model reads but
+  never judges; line ranges on findings.
+- `check`, `test` (examples under `.judgelint/examples/`), `rules`,
+  `providers`, and `init`; `--base` and `--pr` for changes; text, JSON,
+  GitHub annotations, and SARIF.
+- Every request planned and priced before it's sent, with `max_cost_usd`; a
+  cost line on every run; answers cached per model.
+- 16 provider presets over the System One and OpenAI Decisions protocols, and
+  `models:` to average several.
+- Inline `judgelint:disable` comments, with a required reason.
+- No runtime dependencies; platform gems for macOS and Linux carry a native
+  binary built with Spinel.
+- examples/scoop-society: praxis's TypeScript demo, checked against praxis's
+  verdicts.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 JP Camara
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,225 @@
+# judgelint
+
+Lint for the conventions a linter can't check.
+
+Write your team's conventions as bullets in a README. judgelint asks a System
+One decision model, such as TypeSafe's Jev, whether each file breaks each
+one, and reports violations with line ranges and exit codes, like a linter.
+Answers take a few hundred milliseconds and cost a fraction of a cent.
+
+Inspired by [praxis](https://github.com/zarpay/praxis-cli), which reads the
+same kind of specs with chat models. judgelint trades praxis's written
+findings for speed and cost, and reads praxis's specs as they are. Ruby with
+no dependencies; also a native binary, built with
+[Spinel](https://github.com/matz/spinel).
+
+## Quick start
+
+```sh
+gem install judgelint
+judgelint init                # .judgelint.yml and an example spec
+export TYPESAFE_API_KEY=...
+judgelint test                # judge the spec's examples
+judgelint --base main         # check what changed since main
+judgelint                     # or every file the specs govern
+```
+
+On [praxis's demo](examples/scoop-society) (long messages cut here):
+
+```text
+$ judgelint src/services/apply-refund.ts src/features/flavor-of-day
+src/services/apply-refund.ts
+  x error      src/services/domain-failures-are-values-never-exceptions  1-21    0.95  Domain failures are values, never exceptions.
+  x error      src/services/error-messages-are-written-for-the           1-21    0.94  Error messages are written for the API consumer.
+  x error      src/services/every-service-has-a-doc-comment              1-21    0.97  Every service has a doc comment …
+src/features/flavor-of-day
+  x error      src/features/exactly-one-entry-point-index-ts                     0.81  Exactly one entry point, `index.ts`, …
+  x error      src/features/logic-files-are-verb-first-kebab                     0.87  Logic files are verb-first kebab-case …
+  ? uncertain  src/features/no-orphaned-files                                    0.60  No orphaned files.
+  ? uncertain  src/features/the-feature-s-name-describes-the                     0.44  The feature's name describes the capability …
+
+2 files, 5 violations, 2 uncertain
+judgelint: 2 requests, 3,320 input tokens, $0.0001
+```
+
+Exit status: 0 clean, 1 violations (or misjudged examples for `judgelint
+test`), 2 for usage, configuration, or model errors. Everything that can be
+checked is checked before the first paid request.
+
+## Specs
+
+A README, or any `*.judgelint.md`, whose frontmatter has `paths:` is a spec.
+Each top-level bullet is a rule.
+
+```markdown
+---
+paths: ["src/services/*.ts"]
+excludes: ["src/services/legacy-import.ts"]
+---
+# Service conventions
+
+- **Domain failures are values, never exceptions.** A service returns
+  `{ ok: false, error }` for anything a caller could plausibly cause.
+- **No I/O except through the injected Store.** No timers, no fetch, no console.
+```
+
+The model reads the spec's prose and the whole file, with numbered lines, and
+is asked each bullet two ways: "Do lines 1-40 of the file clearly violate the
+following team rule?" and "Do they follow it, or does it not apply?" The two
+answers lean opposite ways, so their average is steadier than either. Rules
+are named from their lead words (`src/services/no-i-o-except-through-the`),
+and a bullet that says "should" or "prefer" is a warning, anything else an
+error.
+
+To sharpen a bullet, say what breaking and following it look like, on
+indented lines under it:
+
+```markdown
+- **No I/O except through the injected Store.** No timers, no fetch, no console.
+  violation: calls fetch, console, or a timer directly
+  compliant: reads and writes only through the Store it's given
+```
+
+They make the model more decisive, and like any wording, they're worth
+checking against examples.
+
+| Frontmatter | |
+|---|---|
+| `paths`, `excludes` | Globs for the files the spec governs, from the project root. |
+| `context` | Files the model reads alongside, such as shared types, but never judges. |
+| `scope` | `window` (default) asks about each 40 lines, so findings name lines; `file` asks once per file; `directory` asks once per directory `paths` matches, with every file in it. Praxis's `cohort: by_file` and `by_directory` work too. |
+| `severity`, `threshold` | Override the wording's severity, or the config's threshold, for every rule in the spec. |
+
+A rule's answer is the probability the code breaks it: at or above the
+threshold (0.7) it's a violation, within 0.3 below it's uncertain (reported,
+never failing).
+
+**When a bullet isn't enough**, write that rule as a question in
+`.judgelint/rules/<id>.md`, with criteria the model weighs its answer against:
+
+```markdown
+---
+question: Is any file in this directory unreachable from index.ts through its imports?
+yes_means: A file nothing in the feature imports, or a directory with no index.ts at all.
+no_means: Every file is imported, directly or indirectly, by index.ts.
+scope: directory
+paths: ["src/features/*"]
+---
+Every file in a feature is reachable from index.ts.
+```
+
+`fail_when: no` inverts a question like "Does this service validate its
+input first?".
+
+**Suppress** a finding where a person has decided the code is fine, with a
+reason: `# judgelint:disable <rule> -- <reason>` covers the 40 lines it's in,
+and `judgelint:disable-file` the whole file. Any comment style works.
+
+## Examples
+
+Decision models classify well and apply policies less well, so the way to
+know a rule works is to try it. Put examples under
+`.judgelint/examples/<rule id>/pass/` (must not be violations) and `fail/`
+(must be), and `judgelint test` reports what the model gets wrong:
+
+```text
+lib/no-silent-rescues  9/10
+  ok   pass 0.03  .judgelint/examples/lib/no-silent-rescues/pass/logs_and_reraises.rb
+  MISS pass 0.79  .judgelint/examples/lib/no-silent-rescues/pass/retries_then_raises.rb
+  ok   fail 0.94  .judgelint/examples/lib/no-silent-rescues/fail/assigns_default.rb
+```
+
+A miss means the bullet needs rewording or the convention is too broad for
+one question. (Saying a retry that eventually raises is fine took this repo's
+own rule, in [lib/README.md](lib/README.md), to 10/10.) The model sees each
+example's path relative to `pass/` or `fail/`, so the directory never gives
+the answer away. For a directory rule, each example is a directory.
+
+## In CI
+
+```sh
+judgelint --base main            # since the merge base with main, uncommitted work included
+judgelint --pr                   # the pull request's base branch
+judgelint --pr --format github   # annotations on the changed lines (also: json, sarif)
+```
+
+`--base` asks only about the 40-line windows a change touched, and about
+directories with a changed file, so old code can't block new work. `--pr`
+reads the base branch from the CI (GitHub Actions, GitLab, Buildkite,
+Bitbucket, Azure Pipelines, or `JUDGELINT_BASE_BRANCH`), then `gh`. Check out
+enough history for the merge base.
+
+## Configuration
+
+`.judgelint.yml` has four settings:
+
+```yaml
+models: [typesafe]       # a model is provider[:model]; list two to average their answers
+threshold: 0.7
+ignore: ["vendor/**"]    # files no rule judges, and READMEs that are never specs
+max_cost_usd: 1.00       # refuse a run estimated over this, unless --yes
+```
+
+Flags: `--base`, `--pr`, `--format`, `--model` (replaces `models` for one
+run), `--no-cache`, `--yes`, `--config`.
+
+Every run ends with its cost. Answers are cached in `.judgelint/cache.json`
+by file, question, and model, so an unchanged file costs nothing; commit the
+cache to share it.
+
+**Two models are more certain than one.** On praxis's demo, Jev alone leaves
+9 to 11 findings uncertain; averaging Jev and Cloudflare's Clef leaves 4, with
+the same 12 of 14 violations found and none false, for $0.02 a run
+(`models: [typesafe, cloudflare]`). Each model's answers are cached on their
+own, and all of a file's requests go out at once. The
+[example](examples/scoop-society) compares the models one at a time.
+
+### Providers
+
+`judgelint providers` lists them. Live-tested: `typesafe` (Jev, the default),
+`openrouter`, `celeris`, `cloudflare`, and `openai` (its Decisions API). From
+their docs: `perplexity`, `vercel`, `digitalocean`, `upstage`, `hanzo`,
+`foundry`, `databricks`, and local `ollama`, `llamacpp`, `vllm`, and
+`litellm`. Each reads its key from the usual variable (`TYPESAFE_API_KEY`,
+`CLOUDFLARE_API_TOKEN`, and so on).
+
+Any other System One or Decisions endpoint is a mapping in `models`:
+
+```yaml
+models:
+  - api_base: https://gateway.example.com/${TEAM}   # ${VAR} reads the environment
+    model: some-decider
+    api_key_env: GATEWAY_KEY
+    # path: /v1/systemone   protocol: system_one|decisions   auth: bearer|basic|none|header:<name>
+    # headers: {}   provider_options: {}   price_per_million: 0.04   batch_size: 32
+```
+
+## Native binary
+
+```sh
+git clone https://github.com/matz/spinel && (cd spinel && make deps && make)
+SPINEL=../spinel/bin/spinel rake spinel        # build/judgelint
+SPINEL=../spinel/bin/spinel rake test:spinel   # the tests, against it too
+```
+
+On macOS and Linux, `gem install` picks a platform gem with the binary
+inside, statically linked, and links the `judgelint` command straight to it;
+elsewhere, Windows included, the gem runs as plain Ruby. The binary starts in
+4 ms against CRuby's 110, and in a 300-file benchmark it was 1.4x faster than
+CRuby on one thread and 2x on eight. Under Bundler, `bundle exec` spends about
+100 ms loading Bundler first; to skip it, link the binary
+(`ln -s "$(bundle info judgelint --path)/libexec/judgelint" bin/judgelint`). The code stays inside what Spinel
+compiles: plain classes instead of `Data.define`, a hand-written option
+parser, no gems.
+
+## Development
+
+```sh
+bundle install
+bundle exec rake test
+bundle exec rake example      # re-ask Jev about praxis's demo and check it
+```
+
+## License
+
+MIT

data/exe/judgelint

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# A platform gem (judgelint-*-arm64-darwin and so on) carries the Spinel-built
+# binary in libexec/; run it when it's there. Spinel compiles this file too,
+# and drops this branch.
+if RUBY_ENGINE != "spinel"
+  native = File.expand_path("../libexec/judgelint", __dir__)
+  exec(native, *ARGV) if File.executable?(native)
+end
+
+require "judgelint"
+
+exit Judgelint::CLI.new(ARGV).run

data/lib/judgelint/cache.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module Judgelint
+  # Stores each answer under a hash of everything that produced it: the state
+  # the model saw (path and numbered contents), the question asked of it, and
+  # the model. Changing any of them is a miss. Safe to share across threads.
+  class Cache
+    def initialize(path)
+      @path = path
+      @entries = load_entries
+      @dirty = false
+      @lock = Mutex.new
+    end
+
+    def self.key(question_digest, state_digest, model_label)
+      Digest::SHA256.hexdigest([question_digest, state_digest, model_label].join(":"))
+    end
+
+    def fetch(key)
+      @lock.synchronize { @entries[key] }
+    end
+
+    def store(key, yes_probability)
+      @lock.synchronize do
+        @entries[key] = yes_probability
+        @dirty = true
+      end
+    end
+
+    def save
+      @lock.synchronize do
+        return unless @path && @dirty
+
+        FileUtils.mkdir_p(File.dirname(@path))
+        File.write(@path, "#{JSON.pretty_generate(@entries.sort.to_h)}\n")
+        @dirty = false
+      end
+    end
+
+    private
+
+    def load_entries
+      return {} unless @path && File.exist?(@path)
+
+      data = JSON.parse(File.read(@path))
+      raise ConfigError, "#{@path} isn't a judgelint cache; delete it to start over" unless data.is_a?(Hash)
+
+      data
+    rescue JSON::ParserError
+      raise ConfigError, "#{@path} isn't valid JSON; delete it to start over"
+    end
+  end
+end

data/lib/judgelint/changes.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+module Judgelint
+  # The lines a change touched, from git: everything since the merge base with
+  # a ref, uncommitted work included. Old code outside the change isn't asked
+  # about, so it can't block new work.
+  module Changes
+    # A branch, tag, or commit (HEAD~1 too), and nothing git could read as an
+    # option or a range.
+    REF = %r{\A[A-Za-z0-9][A-Za-z0-9._/~^-]*\z}
+    HUNK = /\A@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/
+
+    # Returns {path => [changed line numbers]}, or :all for a new file.
+    def self.since(ref)
+      check_ref(ref)
+      base = git("merge-base", ref, "HEAD").strip
+      changed = parse_diff(git("-c", "core.quotePath=false", "diff", "-U0", "--no-color", "--no-ext-diff",
+                               "--relative", "--diff-filter=d", base))
+      git("ls-files", "--others", "--exclude-standard").lines.each { |path| changed[path.chomp] = :all }
+      changed
+    end
+
+    # CI variables that name a pull request's base branch.
+    BASE_BRANCH_VARIABLES = %w[
+      JUDGELINT_BASE_BRANCH GITHUB_BASE_REF CI_MERGE_REQUEST_TARGET_BRANCH_NAME
+      BUILDKITE_PULL_REQUEST_BASE_BRANCH BITBUCKET_PR_DESTINATION_BRANCH SYSTEM_PULLREQUEST_TARGETBRANCH
+    ].freeze
+
+    # The pull request's base branch, fetched, as origin/<branch>: from
+    # JUDGELINT_BASE_BRANCH, then GitHub Actions, GitLab, Buildkite, Bitbucket,
+    # or Azure Pipelines, then the gh CLI.
+    def self.pr_base
+      branch = pr_branch
+      raise ConfigError, "Couldn't find the pull request's base branch" if branch.empty?
+
+      check_ref(branch)
+      git("fetch", "--no-tags", "--quiet", "origin", branch)
+      "origin/#{branch}"
+    end
+
+    def self.pr_branch
+      BASE_BRANCH_VARIABLES.each do |name|
+        # Buildkite sets "false" outside a pull request; Azure names a full ref.
+        branch = ENV.fetch(name, "").delete_prefix("refs/heads/")
+        return branch unless branch.empty? || branch == "false"
+      end
+      command("gh", "pr", "view", "--json", "baseRefName", "-q", ".baseRefName").strip
+    end
+
+    def self.parse_diff(diff)
+      changed = {}
+      path = nil
+      diff.each_line do |line|
+        if line.start_with?("+++ ")
+          target = line.chomp.delete_prefix("+++ ").delete_suffix("\t")
+          path = target == "/dev/null" ? nil : target.delete_prefix("b/")
+        elsif path && (hunk = HUNK.match(line))
+          start = hunk[1].to_i
+          count = hunk[2] ? hunk[2].to_i : 1
+          # A pure deletion marks the line above the gap, so it's still judged.
+          lines = count.zero? ? [[start, 1].max] : (start...(start + count)).to_a
+          (changed[path] ||= []).concat(lines)
+        end
+      end
+      changed
+    end
+
+    def self.check_ref(ref)
+      return if REF.match?(ref) && !ref.include?("..")
+
+      raise ConfigError, "Not a usable git ref: #{ref}"
+    end
+
+    def self.git(*args)
+      command("git", *args)
+    end
+
+    def self.command(*args)
+      out, err, status = Open3.capture3(*args)
+      raise ConfigError, "#{args.first(3).join(' ')} failed: #{err.strip.lines.first.to_s.strip}" unless status.success?
+
+      out
+    rescue SystemCallError => e
+      raise ConfigError, "Couldn't run #{args.first}: #{e.message}"
+    end
+  end
+end

data/lib/judgelint/cli.rb

@@ -0,0 +1,240 @@
+# frozen_string_literal: true
+
+module Judgelint
+  # Command line entry point. Exit codes: 0 clean, 1 violations (or missed
+  # examples for `test`), 2 usage or configuration error. Everything that can
+  # be checked is checked before the first paid request.
+  class CLI
+    COMMANDS = %w[check test rules providers init].freeze
+
+    HELP = <<~TEXT
+      Usage: judgelint [command] [options] [paths...]
+
+      Checks files against conventions written as bullets in READMEs, by asking a
+      System One decision model (Jev by default) whether the code breaks each one.
+
+      Commands:
+        check [paths...]   Check the files the specs govern, or these (default).
+        test [rule ids...] Judge each rule's examples (.judgelint/examples/<id>/pass
+                           and fail) and report what the model gets wrong.
+        rules              List the rules and how many files each governs.
+        providers          List the decision-model providers judgelint knows.
+        init               Write .judgelint.yml and an example spec.
+
+      Options:
+            --base REF       Check only what changed since the merge base with REF,
+                             uncommitted work included
+            --pr             --base the pull request's base branch, from CI variables
+                             or the gh CLI
+        -f, --format FORMAT  text, json, github (Actions annotations), or sarif
+        -m, --model MODEL    Ask this model instead of the config's: a provider from
+                             `judgelint providers`, optionally :model (cloudflare:clef)
+        -c, --config PATH    Config file (default .judgelint.yml)
+            --no-cache       Ask the models even when an answer is cached
+        -y, --yes            Run even past max_cost_usd
+        -v, --version        Print the version
+        -h, --help           Print this help
+
+      Exit status: 0 clean, 1 violations or missed examples, 2 usage or config error.
+    TEXT
+
+    def initialize(argv, out: $stdout, err: $stderr)
+      @argv = argv.dup
+      @out = out
+      @err = err
+      @format = "text"
+      @config_path = nil
+      @base = nil
+      @pr = false
+      @yes = false
+      @overrides = {}
+      @usage = nil
+      @no_cache = false
+    end
+
+    def run
+      command = COMMANDS.include?(@argv.first) ? @argv.shift : "check"
+      @argv = parse_options(@argv)
+      return 0 if @done
+
+      case command
+      when "check" then check
+      when "test" then test
+      when "rules" then list_rules
+      when "providers" then list_providers
+      when "init" then init
+      end
+    rescue UsageError => e
+      @err.puts "judgelint: #{e.message}"
+      @err.puts "Run `judgelint --help` for usage."
+      2
+    rescue Error => e
+      @err.puts "judgelint: #{e.message}"
+      # Requests that succeeded before the failure are still billed.
+      @err.puts @usage.line if @usage && @usage.requests.positive?
+      2
+    end
+
+    private
+
+    # Spinel can't compile OptionParser, so the few options are parsed here.
+    VALUE_OPTIONS = {
+      "-c" => "config", "--config" => "config", "-f" => "format", "--format" => "format", "--base" => "base",
+      "-m" => "models", "--model" => "models"
+    }.freeze
+
+    def parse_options(argv)
+      rest = []
+      until argv.empty?
+        arg = argv.shift
+        name, inline = arg.start_with?("--") ? arg.split("=", 2) : [arg, nil]
+        if arg == "--"
+          rest.concat(argv)
+          argv = []
+        elsif VALUE_OPTIONS.key?(name)
+          value = inline || argv.shift
+          raise UsageError, "#{name} needs a value" if value.nil?
+
+          set_option(VALUE_OPTIONS.fetch(name), value)
+        elsif arg == "--no-cache"
+          @no_cache = true
+        elsif arg == "--pr"
+          @pr = true
+        elsif %w[-y --yes].include?(arg)
+          @yes = true
+        elsif %w[-v --version -h --help].include?(arg)
+          @out.puts(%w[-v --version].include?(arg) ? "judgelint #{VERSION}" : HELP)
+          @done = true
+        elsif arg.start_with?("-") && arg != "-"
+          raise UsageError, "unknown option #{arg}"
+        else
+          rest << arg
+        end
+      end
+      rest
+    end
+
+    def set_option(key, value)
+      case key
+      when "config" then @config_path = value
+      when "base" then @base = value
+      when "format"
+        raise UsageError, "--format is #{Reporter::FORMATS.join(', ')}, not #{value}" unless Reporter::FORMATS.include?(value)
+
+        @format = value
+      else @overrides[key] = value
+      end
+    end
+
+    def check
+      config, rules = load_rules
+      raise UsageError, "--base and --pr don't go together" if @base && @pr
+
+      base = @pr ? Changes.pr_base : @base
+      changes = base ? Changes.since(base) : nil
+      paths = check_paths(rules, changes).reject { |path| Glob.match_any?(config.ignore, path) }
+      directories = check_directories(rules).reject { |dir| Glob.match_any?(config.ignore, dir) }
+      units = paths.size + directories.size
+      runner = runner(config, rules)
+      jobs = runner.plan(paths, directories: directories, changes: changes)
+      guard_cost(config, runner, jobs)
+      verdicts = runner.execute(jobs).flatten
+      Reporter.new(@out, format: @format).check(verdicts, config: config, file_count: units, usage: @usage)
+      @err.puts @usage.line
+      verdicts.any?(&:failing?) ? 1 : 0
+    end
+
+    def check_paths(rules, changes)
+      if !@argv.empty?
+        paths = Glob.expand_arguments(@argv)
+        missing = paths.reject { |path| File.file?(path) }
+        raise ConfigError, "No such file: #{missing.join(', ')}" unless missing.empty?
+
+        paths = paths.select { |path| changes.key?(path) } if changes
+      elsif changes
+        paths = changes.keys.sort.select { |path| File.file?(path) }
+      else
+        paths = Glob.expand(rules.flat_map(&:paths))
+      end
+      paths.select { |path| rules.any? { |rule| !rule.directory? && rule.applies_to?(path) } }
+    end
+
+    # The directories directory rules govern; with paths named, those the
+    # paths are in or are. With a change, the runner keeps those it touched.
+    def check_directories(rules)
+      directories = rules.select(&:directory?).flat_map(&:directories).uniq.sort
+      return directories if @argv.empty?
+
+      named = @argv.map { |arg| arg.delete_prefix("./").delete_suffix("/") }
+      directories.select { |dir| named.any? { |path| path == dir || path.start_with?("#{dir}/") || dir.start_with?("#{path}/") } }
+    end
+
+    def test
+      config, rules = load_rules
+      unless @argv.empty?
+        unknown = @argv - rules.map(&:id)
+        raise ConfigError, "Unknown rules: #{unknown.join(', ')}" unless unknown.empty?
+
+        rules = rules.select { |rule| @argv.include?(rule.id) }
+      end
+
+      runner = runner(config, rules)
+      jobs = runner.plan_examples
+      untested = runner.untested
+      raise ConfigError, "No rule has examples yet. Add some under #{rules.first.examples_dir}/{pass,fail}/" if jobs.empty?
+
+      @err.puts "#{untested.size} #{untested.size == 1 ? 'rule has' : 'rules have'} no examples: #{untested.map(&:id).join(', ')}" unless untested.empty?
+      guard_cost(config, runner, jobs)
+      examples = runner.execute(jobs).flatten
+      Reporter.new(@out, format: @format).fixtures(examples, usage: @usage)
+      @err.puts @usage.line
+      examples.all?(&:correct?) ? 0 : 1
+    end
+
+    # Prices the planned requests and refuses a run over max_cost_usd.
+    def guard_cost(config, runner, jobs)
+      estimate = runner.estimated_cost(jobs)
+      return if estimate.nil? || estimate <= config.max_cost_usd || @yes
+
+      raise ConfigError, format("This run would send %d requests, about %s input tokens ($%.2f), over max_cost_usd " \
+                                "($%.2f). Pass --yes to run it anyway.", runner.request_count(jobs),
+                                Usage.delimit(runner.estimated_tokens(jobs)), estimate, config.max_cost_usd)
+    end
+
+    def list_rules
+      _config, rules = load_rules
+      rules.each do |rule|
+        count = rule.directory? ? "#{rule.directories.size} dirs" : "#{Glob.expand(rule.paths).count { |path| rule.applies_to?(path) }} files"
+        @out.puts format("%-24s %-8s %10s  %s", rule.id, rule.severity, count, rule.question || rule.message)
+      end
+      0
+    end
+
+    def list_providers
+      PROVIDERS.each do |name, preset|
+        model = preset["model"] || "(you choose)"
+        key = preset["api_key_env"] || "no key"
+        @out.puts format("%-13s %-28s %-22s %s", name, model, key, preset["api_base"] || "(your endpoint)")
+        @out.puts "              needs #{preset['needs']}" if preset["needs"]
+      end
+      0
+    end
+
+    def init
+      Scaffold.write(".", @out)
+      @err.puts "Next: set TYPESAFE_API_KEY, then run `judgelint test` to see the example rule judged."
+      0
+    end
+
+    def load_rules
+      config = Config.load(path: @config_path, overrides: @overrides)
+      [config, Spec.discover(rules_dir: Config::RULES_DIR, default_threshold: config.threshold, ignore: config.ignore)]
+    end
+
+    def runner(config, rules)
+      @usage = Usage.new
+      clients = config.models.map { |endpoint| Client.new(endpoint, usage: @usage) }
+      Runner.new(config: config, rules: rules, clients: clients, cache: Cache.new(@no_cache ? nil : Config::CACHE))
+    end
+  end
+end