omniai-evals 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: de97e262c2ca4119e0f158d0b4ed485ffeb9af5711edb529d099685fe277d615
+  data.tar.gz: ce16c93c7acab7e7f6c57680b65a14f04073ad21c7ad9f632efd746b60fb32a6
+SHA512:
+  metadata.gz: df80b33102d5c4e691f143136de334d35a9e3755e6bde89d71147de1feba1c2a334f1091c69995fb3945fd6b8740caae68a3d1e5d9f2d52805ee5ba8fb6a39a0
+  data.tar.gz: 2cfaaab87db9222a769db038b8b6d3800f570d3b351fd625e522f23157c9f5b21ef24b1fa7f5894516aeff8459404397d63b471ede0652aeaf555861e60afcca

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0
+
+Initial release.
+
+- Datasets, criteria, boolean results, the `all_epochs` reducer, run manifests, scorecards, comparability, and `Plan`.
+- Assertions: `presence`, `absence`, `structure`, `identity`, `abstention`.
+- Judge adapters: `Adapters::OmniAI` and `Adapters::ClaudeCLI`. Both accept a `calls:` collector that records each call's finish reason and latency.

data/Gemfile

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gemspec
+
+gem "irb"
+gem "rake"
+gem "rspec"
+gem "rspec_junit_formatter"
+gem "rubocop"
+gem "rubocop-basic"
+gem "rubocop-rake"
+gem "rubocop-rspec"
+gem "simplecov"
+gem "yard"

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dean Singh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,124 @@
+# OmniAI::Evals
+
+Structure and grading for LLM evals: datasets, boolean criteria, assertions, judges,
+epoch reduction, comparability, and PHI-free scorecards. A companion to the
+[omniai](https://github.com/ksylvest/omniai) gem family.
+
+The gem holds the *structure*. It knows nothing about any dataset, model, or product —
+consumers hold the datasets, the criteria keys, the prompts under test, and the runners
+that wire a target to a dataset.
+
+`docs/contract.md` is the authority. It fixes the vocabulary, the five file formats, the
+Ruby interfaces, and six invariants the gem tests for itself. Every file the gem reads or
+writes carries `"contract": 1`; changing a format without bumping that number fails the
+suite.
+
+## Install
+
+```ruby
+gem "omniai-evals"
+```
+
+## Load
+
+The core loads with stdlib and `json` only. **Adapters reach a model, so none of them is
+required for you** — require the one you want:
+
+```ruby
+require "omniai/evals"                      # core: no network, no gems
+require "omniai/evals/adapters/omniai"      # judge via the omniai gem (API-billed)
+require "omniai/evals/adapters/claude_cli"  # judge via `claude -p` (subscription or API)
+```
+
+## Scores are boolean
+
+There are no 0–1 scores, thresholds, or weights anywhere, by design. A criterion either
+held or it did not, and `Result` raises on anything that is not `true` or `false`. Across
+epochs, `Reducer.all_epochs` returns `:pass`, `:fail`, or `:flaky` — a criterion that
+passes 3 runs in 5 is not "60% correct", it is unreliable, and the reader needs to see
+that rather than a rounded number.
+
+## The five deterministic criterion types
+
+Each is free, offline, and returns `{key, score, comment}`.
+
+| type | `target` | passes when |
+|---|---|---|
+| `presence` | `{"date": "2019-03-14"}` / `{"text": "signed by"}` | the date (parsed, any format) or the text (case- and whitespace-normalized) appears |
+| `absence` | `{"text": "DO NOT COPY"}` | the text does **not** appear |
+| `structure` | `{"table_rows": 12}` | the markdown table has that many body rows (header and separator excluded) |
+| `structure` | `{"circled": "Option B"}` | that option is marked circled / selected |
+| `structure` | `{"path": ["exam","date"], "type": "string"}` | the output is a Hash, the path resolves, the value has that JSON type |
+| `identity` | `{"text": "Patient: JANE Q."}` | the output contains it **exactly**, case-sensitive, after whitespace collapse |
+| `identity` | `{"field": "claim_number", "value": "ABC-123"}` | that field of a Hash output equals that value |
+| `abstention` | `{"region": "line 7", "forbidden_terms": ["approved"]}` | the output abstains on the region rather than asserting content it cannot read |
+
+`structure` and `identity` each take a **text-mode** target and a **structured** target,
+selected by which key is present — the shape a target emits (markdown vs JSON) is the
+consumer's business, not the criterion's.
+
+`identity` in text mode is the one assertion that is **case-sensitive** and normalizes
+nothing but whitespace. That is deliberate: a banded table whose header names a different
+patient must be transcribed as-is, and an OCR pass that quietly "fixes" the case has
+produced exactly the failure the criterion exists to detect.
+
+`abstention` recognises the non-lexical marker `~~~~` (any run of 3+ tildes) that
+production models actually emit, plus lexical markers like `[illegible]`. Override the
+list per criterion with `target["markers"]`.
+
+`faithful` is a judge, not an assertion, and `Assertions.for("faithful")` raises rather
+than quietly falling back to a string compare.
+
+**Dates are compared parsed, never as strings.** `2019-03-14`, `03/14/2019`,
+`03-14-2019`, `14 March 2019`, `Mar. 14, 2019`, `March 14 2019`, `2019/03/14` and
+`3/14/19` are all the same day.
+
+Numeric slash and dash dates are read **US month-first**. Ruby's `Date.parse` reads them
+day-first, so it returns `nil` for `03/14/2019` and — worse — silently returns *February
+4th* for `04/02/2019`, marking a correct transcription wrong with nothing raised anywhere.
+A non-US dataset needs a per-dataset locale field; the format cannot be inferred from the
+text. See `Dates::US_NUMERIC`.
+
+## PHI
+
+- **Result records (JSONL) are PHI-bearing** — a judge comment may quote model output.
+  Store them under your own protected path.
+- **Scorecards and run manifests are not.** They carry counts and provenance, never text.
+  The scorecard is built as a fresh Hash of integers rather than by filtering fields out
+  of records, so there is no route by which a comment reaches it.
+- **The judge receives the key text and the model output, never the example input.**
+  `Judge.packet` takes only those two, so there is no parameter to leak the input through.
+
+## Runs, configs, and the results file
+
+A **run is one target version**. `Plan` takes `configs:` so you can price "default vs
+`effort: low`" in one go *before* committing the spend, but each config is its own run with
+its own manifest and its own results file. `Results.write` refuses to append records from a
+second `run_id` into a file that already holds one — otherwise a re-run silently doubles
+every epoch and the reducer returns a PASS assembled from two runs nobody asked to combine.
+
+`--plan` prices **target calls as well as judge calls**, from `prices["target"]` and
+`prices["judge"]`. For a vision bench the target calls are the metered spend, and a plan
+reporting `$0.00` for a paid run is worse than no plan.
+
+## The demo bench
+
+The gem ships one synthetic, PHI-free bench so it always has a consumer of its own:
+
+```bash
+ruby bench/demo/run.rb --plan   # estimates calls and cost, makes zero model calls
+ruby bench/demo/run.rb          # grades a deliberately imperfect target, prints a scorecard
+```
+
+## Development
+
+```bash
+bundle install
+bundle exec rake        # rspec + rubocop
+```
+
+The six contract invariants have their own spec file, `spec/omniai/evals/invariants_spec.rb`.
+
+## License
+
+MIT. See `LICENSE`.

data/lib/omniai/evals/adapters/call_record.rb

@@ -0,0 +1,100 @@
+# frozen_string_literal: true
+
+require "json"
+
+module OmniAI
+  module Evals
+    module Adapters
+      # One per-call record for an adapter's optional `calls:` collector: stop codes and numbers
+      # only, never result text or stderr (both can carry PHI).
+      #
+      # `attempts` is always 1: neither adapter retries, and retries inside `claude -p` or the
+      # provider client are not reported.
+      module CallRecord
+        DURATION_KEYS = %w[duration_ms duration_api_ms].freeze
+        # A call that finished but named no reason.
+        UNREPORTED = "unreported"
+
+        # For Adapters::ClaudeCLI. The block receives a sink to hand the subprocess stdout to
+        # (including on a non-zero exit, so an error envelope's subtype is still recorded) and
+        # returns that stdout. The stdout stays here; it is parsed only for stop codes and numbers.
+        def self.capture(calls, timeout_error:, &block)
+          return yield(nil) if calls.nil?
+
+          stdout = nil
+          sink = ->(out) { stdout = out }
+          measure(calls, timeout_error:, body: -> { block.call(sink) }) do |failure, latency_ms|
+            envelope_record(stdout, failure:, latency_ms:)
+          end
+        end
+
+        # For Adapters::OmniAI. The block returns an OmniAI::Chat::Response; its verbatim provider
+        # finish reason is recorded.
+        def self.capture_response(calls, timeout_error:, &block)
+          return yield if calls.nil?
+
+          response = nil
+          measure(calls, timeout_error:, body: -> { response = block.call }) do |failure, latency_ms|
+            reason = failure || string(response.finish_reason&.value) || UNREPORTED
+            record(reason, latency_ms)
+          end
+        end
+
+        # Runs +body+ and pushes exactly one record (built by the block from the failure word, or
+        # nil on success, and the latency) however it ends, before any error propagates.
+        def self.measure(calls, timeout_error:, body:)
+          started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          failure = "error"
+          begin
+            result = body.call
+            failure = nil
+            result
+          rescue timeout_error
+            failure = "timeout"
+            raise
+          ensure
+            calls << yield(failure, ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round)
+          end
+        end
+
+        # The envelope's stop_reason, else its subtype (subtype first when is_error), else
+        # "unreported". A failed call keeps +failure+ unless its stdout is an is_error envelope, so
+        # a timeout, or a non-zero exit without one, records "timeout" / "error"; stdout that is
+        # not a JSON object records "error".
+        def self.envelope_record(stdout, failure:, latency_ms:)
+          envelope = envelope(stdout)
+          return record(failure || "error", latency_ms) if envelope.nil?
+          return record(failure, latency_ms) if failure && !envelope["is_error"]
+
+          result = record(finish_reason(envelope), latency_ms)
+          DURATION_KEYS.each { |key| result[key] = envelope[key] if envelope[key].is_a?(Numeric) }
+          result
+        end
+
+        def self.record(finish_reason, latency_ms)
+          { "finish_reason" => finish_reason, "latency_ms" => latency_ms, "attempts" => 1 }
+        end
+
+        # Strings only, so an odd envelope can't put a nested object into the record.
+        def self.finish_reason(envelope)
+          reasons = envelope.values_at("stop_reason", "subtype")
+          reasons.reverse! if envelope["is_error"]
+          reasons.map { |reason| string(reason) }.compact.first || UNREPORTED
+        end
+
+        def self.string(value)
+          value if value.is_a?(String)
+        end
+
+        def self.envelope(stdout)
+          return if stdout.nil?
+
+          parsed = JSON.parse(stdout)
+          parsed if parsed.is_a?(Hash)
+        rescue JSON::ParserError
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/omniai/evals/adapters/claude_cli.rb

@@ -0,0 +1,270 @@
+# frozen_string_literal: true
+
+require "json"
+require "open3"
+
+require_relative "../../evals"
+require_relative "call_record"
+
+module OmniAI
+  module Evals
+    module Adapters
+      # A judge that shells out to `claude -p`, ported from its first consumer's judge.
+      #
+      # One place for the "hand a packet to a `claude -p` judge and get back a parsed
+      # JSON verdict" mechanics that several checks need: completeness (rubric
+      # adherence), blind re-extraction, and dispute tie-break. Each caller supplies
+      # its OWN system prompt, packet shape, and model; this owns only the transport
+      # and the tolerant output parsing.
+      #
+      # Three disciplines are baked in and must never regress:
+      #   - PHI-on-stdin: the packet (summary/source/transcript text) is PHI and goes
+      #     on STDIN, never argv -- argv is visible in `ps`/process listings. Only
+      #     static, PHI-free flags are passed as arguments.
+      #   - PHI-free errors: `claude` stderr and the raw judge output can echo verbatim
+      #     source quotes (PHI), so error messages surface only byte counts, never the
+      #     content itself.
+      #   - Isolation: the judge runs with only the model and the caller's system prompt
+      #     (see ISOLATION_FLAGS) -- no tools, MCP servers, saved session, or settings.
+      #
+      # Changes from that original: it is a class whose billing is bound at
+      # construction (the old Judge.bound), so it satisfies the contract's
+      # `call(system:, packet:, model:, schema:, timeout:)` adapter interface; and it
+      # runs isolated, replacing the Claude Code system prompt rather than appending to it.
+      class ClaudeCLI
+        class JudgeError < OmniAI::Evals::Error
+        end
+
+        # A judge that blew its wall-clock deadline, distinct from an unparseable result so
+        # the summary can count timeouts on their own line (it is still a JudgeError, so
+        # existing rescues keep treating it as a failed call).
+        class TimeoutError < JudgeError
+        end
+
+        # The child's environment is built FROM SCRATCH, never inherited. ANTHROPIC_API_KEY is
+        # exported machine-wide on the operator's box, and a bare Open3.popen3(*args) hands it
+        # to `claude -p`, where it takes auth precedence over the claude.ai login -- so runs
+        # believed to be on the subscription silently billed the metered API (confirmed in the
+        # Anthropic Console, 2026-08-26). An ALLOWLIST is deliberate: deleting only the variable
+        # we found would leave ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL / the Bedrock+Vertex
+        # switches to bite next, via the same machine-wide-export mechanism.
+        #
+        # TRAP, so nobody re-derives it the expensive way: `~/.claude.json` ->
+        # `customApiKeyResponses` listing a key under "rejected" governs only the INTERACTIVE
+        # approval prompt. It does NOT govern `-p` subprocess auth precedence, and is not
+        # evidence of auth mode. Neither is the `-p` envelope's `total_cost_usd`, which is
+        # populated under both modes. Billing mode is verifiable only at the Console.
+        # Drawn by AUTHORITY, not by enumeration. STRIP anything that carries or redirects
+        # authority (ANTHROPIC_*, CLAUDE_CODE_USE_*, cloud credentials) or that injects code
+        # (NODE_OPTIONS, RUBYOPT -- an injected --require can re-add any stripped var, so code
+        # injection IS authority). PASS machine descriptors and LOCATORS.
+        #
+        # CLAUDE_CONFIG_DIR / XDG_CONFIG_HOME are the decisive entries: they are the only
+        # omissions that would fail SILENTLY-WRONG instead of loudly. With them stripped, a
+        # child on a machine whose Claude config was relocated falls back to ~/.claude and, if
+        # a valid login lives there, authenticates as a DIFFERENT ACCOUNT -- the same
+        # silent-wrong-auth class this seam exists to prevent. The TLS/proxy entries only
+        # prevent pointless loud breakage on a corporate network.
+        #
+        # The boundary is self-enforcing: the spec fails if a future edit adds a name
+        # matching /KEY|TOKEN|SECRET|AUTH|ANTHROPIC|AWS|GOOGLE|CREDENTIAL/.
+        CHILD_ENV_ALLOWLIST = %w[
+          PATH HOME SHELL LANG LC_ALL LC_CTYPE TMPDIR USER LOGNAME
+          CLAUDE_CONFIG_DIR XDG_CONFIG_HOME
+          SSL_CERT_FILE NODE_EXTRA_CA_CERTS HTTP_PROXY HTTPS_PROXY NO_PROXY
+        ].freeze
+        BILLING_MODES = %i[subscription api].freeze
+
+        # A page legitimately takes minutes (the model re-transcribes it), so the default
+        # deadline is generous; a caller's timeout: overrides it. Its job is only to cap a
+        # WEDGED judge that would otherwise hang a worker forever.
+        JUDGE_TIMEOUT = 900
+        TERM_GRACE = 5
+
+        # The judge receives PHI on stdin, so it runs with nothing but the model and the caller's
+        # system prompt (--system-prompt replaces Claude Code's): no tools, no MCP servers (which
+        # would also join it to peer networks), no saved session under ~/.claude/projects, and no
+        # settings or CLAUDE.md.
+        ISOLATION_FLAGS = ["--tools", "", "--strict-mcp-config", "--no-session-persistence",
+                           "--setting-sources", "",].freeze
+
+        attr_reader :billing
+
+        # Billing is a RUN-level choice, so it is bound once at construction and the bound
+        # adapter is handed to the existing injection points -- no `billing:` parameter has
+        # to be threaded through every intermediate signature. The mode is validated at BIND
+        # time, so a bad value fails before any page is judged rather than on the first paid
+        # call, and the instance is frozen so a validated mode is the mode that runs.
+        #
+        # @param billing [Symbol] :subscription (local claude.ai login, no key reaches the
+        #   child) or :api (METERED -- injects ANTHROPIC_API_KEY). Required on purpose:
+        #   there is no default, so a caller cannot fall into metered billing by omission.
+        def initialize(billing:)
+          @billing = billing
+          child_env # validates the mode (and, for :api, the key) up front
+          freeze
+        end
+
+        # Runs one `claude -p` judge under the billing mode bound at construction.
+        #
+        # @param system [String] the judge's system prompt -- static and PHI-free.
+        # @param packet [String] the PHI-bearing user content; sent on STDIN only.
+        # @param model [String] the model id (caller-chosen -- sonnet/haiku/etc.). Keyword
+        #   with a nil default only so the signature matches Adapters::OmniAI; nil raises.
+        # @param schema [Hash, nil] an optional JSON Schema. When given, the CLI's
+        #   `--json-schema` flag constrains the model's answer via constrained decoding
+        #   and the schema-valid object comes back in the envelope's `structured_output`
+        #   field -- the fix for a task prompt (e.g. "transcribe then extract") whose prose
+        #   otherwise buries or replaces the JSON. Without it, behavior is unchanged: the
+        #   free-form `result` is recovered by the tolerant parse_output/parse_verdict path.
+        # @param calls [#<<, nil] an optional collector. Each call that reaches the subprocess
+        #   pushes exactly one CallRecord however it ends, before any error is raised.
+        # @return [Hash] the parsed verdict.
+        # @raise [JudgeError] when the claude call errors or returns unparseable output.
+        def call(system:, packet:, model: nil, schema: nil, timeout: JUDGE_TIMEOUT, calls: nil)
+          # model stays effectively required -- there is no sensible default for a judge
+          # model -- but it is a keyword with a nil default so this signature matches
+          # Adapters::OmniAI exactly and a runner can swap the two adapters.
+          raise ArgumentError, "model is required" if model.nil?
+
+          # stderr is captured but deliberately dropped from any error -- it can echo
+          # input (PHI). Only PHI-free flags ride argv; the packet rides stdin. The
+          # schema is a static, PHI-free JSON Schema, so it is safe on argv.
+          args = ["claude", "-p", "--system-prompt", system, *ISOLATION_FLAGS, "--model", model,
+                  "--output-format", "json",]
+          args += ["--json-schema", JSON.generate(schema)] if schema
+          stdout = CallRecord.capture(calls, timeout_error: TimeoutError) do |on_stdout|
+            run_with_timeout(args, packet, timeout:, env: child_env, on_stdout:)
+          end
+
+          schema ? parse_structured(stdout) : parse_output(stdout)
+        end
+
+        # The child environment for the bound billing mode -- built from the allowlist, so no
+        # credential reaches the child unless this method puts it there. :subscription carries
+        # NO auth vars, leaving `claude -p` to use the local claude.ai login; :api injects the
+        # key deliberately and fails fast when it is absent, rather than silently falling back.
+        def child_env
+          raise ArgumentError, "unknown billing mode #{@billing.inspect}" unless BILLING_MODES.include?(@billing)
+
+          env = CHILD_ENV_ALLOWLIST.each_with_object({}) do |name, acc|
+            value = ENV.fetch(name, nil)
+            acc[name] = value unless value.nil?
+          end
+          return env unless @billing == :api
+
+          key = ENV.fetch("ANTHROPIC_API_KEY", "").to_s
+          raise JudgeError, "billing: :api requires ANTHROPIC_API_KEY to be set" if key.empty?
+
+          env.merge("ANTHROPIC_API_KEY" => key)
+        end
+
+      private
+
+        # Runs the judge subprocess under a hard wall-clock deadline so a wedged `claude`
+        # can't hang a worker forever (Open3.capture3 has no timeout). stdout and stderr
+        # are drained in threads to avoid a pipe-buffer deadlock; stderr is dropped (it can
+        # echo PHI). On expiry: SIGTERM, a short grace, then SIGKILL, and a PHI-free
+        # JudgeError. The packet rides stdin only, never argv. +env+, when given, REPLACES
+        # the child's environment entirely (see the unsetenv_others note below). +on_stdout+,
+        # when given, receives stdout before the exit check, so a caller can read a failed
+        # call's envelope without it ever reaching an error.
+        def run_with_timeout(args, packet, timeout:, env: nil, on_stdout: nil)
+          # unsetenv_others is what makes the allowlist real: WITHOUT it Open3 MERGES the hash
+          # into the inherited environment, so every var we meant to exclude still reaches the
+          # child and the whole seam is a silent no-op.
+          spawn = env.nil? ? args : [env, *args, { unsetenv_others: true }]
+          Open3.popen3(*spawn) do |stdin, stdout, stderr, wait_thr|
+            writer = Thread.new do
+              stdin.write(packet)
+            rescue Errno::EPIPE
+              # the child exited before reading its input -- nothing left to send
+            ensure
+              stdin.close
+            end
+            out_reader = Thread.new { stdout.read }
+            err_reader = Thread.new { stderr.read } # drained to avoid deadlock; dropped (PHI)
+
+            if wait_thr.join(timeout).nil?
+              terminate(wait_thr)
+              [writer, out_reader, err_reader].each(&:kill)
+              raise TimeoutError, "judge timeout after #{timeout}s"
+            end
+
+            writer.join
+            err_reader.join
+            status = wait_thr.value
+            stdout = out_reader.value.tap { |out| on_stdout&.call(out) }
+            raise JudgeError, "claude -p exited #{status.exitstatus}" unless status.success?
+
+            stdout
+          end
+        end
+
+        # SIGTERM, wait a short grace, then SIGKILL -- never leave a wedged judge running.
+        def terminate(wait_thr)
+          Process.kill("TERM", wait_thr.pid)
+          return if wait_thr.join(TERM_GRACE)
+
+          Process.kill("KILL", wait_thr.pid)
+          wait_thr.join(1)
+        rescue Errno::ESRCH
+          # already exited between the timeout check and the signal
+        end
+
+        # Parses the `claude -p --output-format json` envelope ({ is_error, result })
+        # and returns the verdict Hash carried in `result`. The error message never
+        # includes the raw output -- it holds PHI (verbatim source quotes).
+        def parse_output(stdout)
+          envelope = JSON.parse(stdout)
+          raise JudgeError, "claude reported an error result" if envelope["is_error"]
+
+          parse_verdict(envelope.fetch("result", "").to_s)
+        rescue JSON::ParserError
+          raise JudgeError, "unparseable judge output (#{stdout.bytesize} bytes)"
+        end
+
+        # The judge is told to emit bare JSON, but LLM output is nondeterministic -- the
+        # same input can come back ```-fenced one run and wrapped in prose the next.
+        # Strip fences first; if that still isn't valid JSON, fall back to the outermost
+        # { ... } span. Raises (PHI-free) only when no JSON object is recoverable.
+        def parse_verdict(result)
+          cleaned = result.strip.gsub(/\A```(?:json)?\s*/m, "").gsub(/\s*```\z/m, "").strip
+          JSON.parse(cleaned)
+        rescue JSON::ParserError
+          span = cleaned[/\{.*\}/m]
+          raise JudgeError, "unparseable judge output (#{result.bytesize} bytes)" if span.nil?
+
+          # A non-nil span can still be invalid JSON (e.g. prose braces). Rescue here too
+          # so it surfaces as a PHI-free JudgeError, never a raw JSON::ParserError leaking
+          # the (possibly PHI-bearing) content.
+          begin
+            JSON.parse(span)
+          rescue JSON::ParserError
+            raise JudgeError, "unparseable judge output (#{result.bytesize} bytes)"
+          end
+        end
+
+        # Reads the schema-constrained object from a `claude -p --json-schema` envelope's
+        # `structured_output` field (constrained decoding populates it). The free-form
+        # `result` still holds the model's prose transcription -- its chain-of-thought --
+        # but the deterministic artifact is `structured_output`. An absent field means the
+        # model transcribed but never emitted the constrained object (refusal / max_tokens):
+        # a COUNTED JudgeError, never a silent nil. Error messages stay PHI-free (byte counts
+        # only) -- the envelope can echo verbatim source quotes.
+        def parse_structured(stdout)
+          envelope = JSON.parse(stdout)
+          raise JudgeError, "claude reported an error result" if envelope["is_error"]
+
+          verdict = envelope["structured_output"]
+          raise JudgeError, "no structured_output in judge response (#{stdout.bytesize} bytes)" unless
+            verdict.is_a?(Hash)
+
+          verdict
+        rescue JSON::ParserError
+          raise JudgeError, "unparseable judge output (#{stdout.bytesize} bytes)"
+        end
+      end
+    end
+  end
+end

data/lib/omniai/evals/adapters/omniai.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+require "timeout"
+
+require_relative "../../evals"
+require_relative "call_record"
+
+begin
+  require "omniai"
+rescue LoadError
+  raise OmniAI::Evals::AdapterNotLoaded,
+    'Adapters::OmniAI needs the omniai gem: add `gem "omniai"` to your Gemfile'
+end
+
+module OmniAI
+  module Evals
+    # Judge adapters. Each reaches a model, so none of them is required by the core --
+    # a consumer requires the one it wants.
+    module Adapters
+      # The default judge adapter: API-billed, structured output via OmniAI::Schema.format.
+      #
+      # "Default" is a documentation claim, not an implicit require. The core loads on
+      # stdlib alone (invariant 3), so requiring this file is the consumer's decision and
+      # AdapterNotLoaded names what is missing when the gem is absent.
+      #
+      # Behaviour-equivalent to Adapters::ClaudeCLI by design, and the signature is the
+      # contract's call(system:, packet:, model:, schema:, timeout:) so a runner can swap
+      # one for the other without touching the call site. Equivalence is not cosmetic:
+      # judge calibration measured on one adapter only transfers to the other if both put
+      # the same content in the same role.
+      class OmniAI
+        # @param client [OmniAI::Client]
+        # @param model [String] the default model, overridable per call
+        def initialize(client:, model:)
+          @client = client
+          @model = model
+        end
+
+        # @param system [String] the judge's system prompt -- static and PHI-free
+        # @param packet [String] the PHI-bearing user content
+        # @param schema [Hash] a plain JSON Schema, per Judge.schema
+        # @param timeout [Integer, nil] wall-clock cap, matching ClaudeCLI's
+        # @param calls [#<<, nil] an optional collector; each call pushes exactly one CallRecord
+        #   (the response's verbatim finish reason) however it ends, before any error is raised.
+        # @return [Hash] matching `schema`
+        def call(system:, packet:, model: nil, schema: nil, timeout: nil, calls: nil)
+          # A REAL system turn, not a system prompt glued onto the front of the user
+          # message. ClaudeCLI passes it via --system-prompt, and an adapter that
+          # instead prepends it to the user content is asking the model a different
+          # question -- which silently invalidates any calibration carried across.
+          response = CallRecord.capture_response(calls, timeout_error: Timeout::Error) do
+            Timeout.timeout(timeout) do
+              @client.chat(model: model || @model, format: format_for(schema)) do |prompt|
+                prompt.system(system)
+                prompt.user(packet)
+              end
+            end
+          end
+          JSON.parse(response.text)
+        end
+
+        # Schema.format wants an OmniAI::Schema::Object and the core deals in plain JSON
+        # Schema Hashes, so the translation belongs here at the boundary.
+        def format_for(schema)
+          return nil if schema.nil?
+
+          ::OmniAI::Schema.format(name: "verdict", schema: ::OmniAI::Schema.deserialize(schema))
+        end
+      end
+    end
+  end
+end

data/lib/omniai/evals/assertions/absence.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module OmniAI
+  module Evals
+    module Assertions
+      # Passes when the forbidden text does NOT appear.
+      module Absence
+      module_function
+
+        def call(example:, output:, criterion:)
+          needle = Normalize.phrase(criterion.target.fetch("text"))
+          present = Normalize.phrase(Normalize.text(output)).include?(needle)
+          Result.new(key: criterion.id, score: !present,
+            comment: present ? "forbidden text present" : "forbidden text absent")
+        end
+      end
+    end
+  end
+end