aireview 2.1.0 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1e262ff91d3ae033f68f0a1a940af115dda8cf924e6715d7d494dfbe30e2e907
-  data.tar.gz: ea5bfca46774fe542338e67b198dc66e550b33404363646da055e8648c67f6b1
+  metadata.gz: c37a072f5cb03c8a4e95944a3c6fc5386d1abe6c340c63ebee9715db5d44ec6a
+  data.tar.gz: b2e2ef104f8e4310dafc0dd5041a899c7ff06af5d2098cff25e0e38f010f9847
 SHA512:
-  metadata.gz: 7fee8f2bbade71a38d72e66be714c748d38afc91643dde35329fbc64a357c88af6020bda5f1a89b9762922dc4ff4a413cdf777fab03b16dc9a09e073953ad400
-  data.tar.gz: 869d8f2e89afd98feab341028556c7a361ba46f9afb070cb728298a2b20aafc8df1aba5525f9ab9299c4b35169dcc3323cef72771764f6120ccfba0f34c34d69
+  metadata.gz: '09524e49ed6896e32e6e73d3fcc5f5e27d19eaa255273017acf699e8e6992f44a8a950e34d96f446066e58b31a6ee625188891e913ac3a4c0db27ac2b39e95a8'
+  data.tar.gz: 4a193a705c06609c07ceb2833211d51d124e1299378368feb50fc08e0fac48a6ff09f553b47406b0bd367e83ebfe7b55fd66d018f4781551725aa27234e95113

data/CHANGELOG.md

@@ -1,5 +1,42 @@
 # Changelog
 
+## 2.2.0
+
+- Jev shadow mode (`llm.jev.shadow`, `LLM_JEV_SHADOW`, key `JEV_API_KEY`):
+  after the LLM Critique the same candidates go to Jev (TypeSafe), a fast
+  classifier, and its keep/reject/unverifiable decisions with the raw
+  probabilities are logged next to the Critique verdicts. The report and the
+  review key do not change; a Jev failure is a warning. The data is for
+  choosing between an LLM and Jev as the critic later.
+- Critique engine (`llm.critique.engine`, `LLM_CRITIQUE_ENGINE`,
+  `--critique-engine`): `model` (the default, the LLM Critique as before) or
+  `jev`. With `jev` Jev decides keep/reject without refining the wording;
+  the candidates it cannot judge, and all of them when Jev fails, go to the
+  LLM Critique with `llm.jev.fallback: model` (the default), or are
+  rejected / fail the run with `fail`. Duplicates are dropped once over the
+  merged verdicts. The report says how Jev took part. The Jev thresholds
+  default to a provisional 0.5 until they are chosen from the shadow logs.
+- The stages that need an LLM follow the engine: with Jev and
+  `fallback: fail` the critique model, its key, its share of the context
+  budget, the pool's critique policy and its `models check` probe are not
+  required. `models check` probes Jev as well (counted only for the engine).
+- The review key includes the Jev version, thresholds, fallback and
+  question templates with `engine: jev`; with the default engine it does not
+  change, except for runs with `--no-critique`: the unused critique policy
+  of the pool (`critique.start`, `rank`, `allow_weaker`) no longer enters
+  their key and is no longer validated, so such a review is redone once.
+- `--dry-run` prints the Jev request for the stub candidate.
+- The gem now packages `lib/**/*.yml` (the Jev questions).
+- A pool of models from several providers: OpenAI, Anthropic and OpenRouter
+  take their keys from `OPENAI_API_KEY(S)`, `ANTHROPIC_API_KEY(S)`,
+  `OPENROUTER_API_KEY(S)`, and a string like `anthropic/claude-opus-4.5` in
+  `LLM_MODELS` or a reserve names the provider (an OpenRouter model goes with
+  the prefix: `openrouter/qwen/qwen3-coder`).
+- `api_base` for a model of the pool, a stage and a reserve: several servers
+  of your own in one pool. Such a model is named with its address, gets
+  `LLM_API_KEY` or no auth — never a provider's key — and needs no key to
+  start. Models without an address keep their names and review keys.
+
 ## 2.1.0
 
 - An answer that is not valid JSON and that the provider cut off at the

data/README.md

@@ -79,11 +79,18 @@ REVIEW_LANGUAGE=ru
 REVIEW_MODE=update
 ```
 
-At the moment only the `gemini` and `ollama` providers are supported. The
-provider and the model of each stage are set through `LLM_GENERATE_PROVIDER`,
-`LLM_GENERATE_MODEL`, `LLM_CRITIQUE_PROVIDER` and `LLM_CRITIQUE_MODEL`.
-`LLM_PROVIDER` stays the shared default when a stage has no provider of its
-own.
+The providers are `gemini`, `ollama`, `openai`, `anthropic` and
+`openrouter`, and any OpenAI-compatible server of your own (vLLM, llama.cpp,
+LM Studio) as `openai` with an `api_base`. The provider and the model of each
+stage are set through `LLM_GENERATE_PROVIDER`, `LLM_GENERATE_MODEL`,
+`LLM_CRITIQUE_PROVIDER` and `LLM_CRITIQUE_MODEL`. `LLM_PROVIDER` stays the
+shared default when a stage has no provider of its own. Gemini and Ollama
+are the ones the image defaults and the verified models are about; for the
+others run `aireview models check` before relying on a model.
+
+Each provider takes its keys from its own variables:
+`GEMINI_API_KEY(S)`, `OPENAI_API_KEY(S)`, `ANTHROPIC_API_KEY(S)`,
+`OPENROUTER_API_KEY(S)`; `LLM_API_KEY` is the shared fallback.
 
 `REVIEW_LANGUAGE` (or `review_language` in `.aireview.yml`) sets the language
 of the review: both the LLM answers and the headings of the rendered report.
@@ -175,6 +182,52 @@ the end of the run (see "Fallback models and keys"). The separate smoke test
 of every model with both schemas (`aireview models check`) runs on an image
 release and on a schedule, not on MRs.
 
+### A pool of models from several providers
+
+The critic is best the strongest model you have, and it does not have to be
+Gemini: the shared pool (see "Shared model pool") takes models of any
+provider, ordered by strength as you see it — the order is the only measure,
+names and release dates are not compared.
+
+```bash
+LLM_MODELS=anthropic/claude-opus-4.5,openai/gpt-5,gemini/gemini-3.8-flash,ollama/qwen2.5-coder:14b
+LLM_GENERATE_START=gemini-3.8-flash
+ANTHROPIC_API_KEY=xxx
+OPENAI_API_KEY=xxx
+GEMINI_API_KEY=xxx
+```
+
+In a string the provider goes in front of the model; an OpenRouter model has
+a slash of its own, so it is written with the prefix:
+`openrouter/qwen/qwen3-coder`. Servers of your own, one or several, are
+listed in `.aireview.yml` with their address:
+
+```yaml
+llm:
+  models:
+    - provider: anthropic
+      model: claude-opus-4.5
+    - provider: openai            # an OpenAI-compatible server of your own
+      model: qwen3-coder
+      api_base: http://gpu1:8000/v1
+    - provider: ollama
+      model: qwen2.5-coder:14b
+      api_base: http://gpu2:11434/v1
+  generate:
+    start: qwen2.5-coder:14b
+```
+
+`api_base` also works for a stage (`llm.generate.api_base`) and for a reserve
+in `fallbacks`. A model with its own `api_base` is named with the address
+(`openai/qwen3-coder@http://gpu1:8000/v1`), so two servers with the same model are
+two models for quarantine and the review key, and a start may name either.
+Such a server never gets a provider's key: it is called with `LLM_API_KEY`,
+or without auth when that is not set, and needs no key to start. Without an
+`api_base` a model goes to its provider's API, or to `LLM_API_BASE` for
+Gemini, OpenAI and OpenRouter as before.
+
+Instead of an LLM the critic can be Jev (see "Critique engine").
+
 ### Local Ollama
 
 Install Ollama following the
@@ -522,6 +575,124 @@ Critique receives the result of the check in the candidate's `note` field
 and decides keep/reject with it in mind. With `--no-critique` the check works
 the same way, its marks just go straight to the report.
 
+### Critique engine: an LLM or Jev
+
+`llm.critique.engine` picks who checks the candidates of the first pass:
+
+| | `model` (default) | `jev` |
+|---|---|---|
+| What checks | the LLM Critique chain or pool (`LLM_CRITIQUE_MODEL`, `critique.rank`) | [Jev](https://docs.typesafe.ai), a classifier by TypeSafe |
+| Context | the whole prompt: MR, Jira, diff | the same, cut to Jev's limits (see below) |
+| Wording of findings | refined by the critic | the first pass's own |
+| Time and cost | tens of seconds, provider quota | under a second, per input token |
+| Depends on Gemini | when the critique model is Gemini | no, with `fallback: fail` |
+| Non-English reviews | fine | not declared by TypeSafe, check first |
+| Code leaves for | the LLM provider | TypeSafe as well |
+
+The default stays `model`. A stronger critic than the generator is set with
+the existing settings (`LLM_CRITIQUE_MODEL`, or the order of the pool); Jev is
+meant for when speed or independence from the LLM quota matters more than
+refined wording, and only after its thresholds are chosen from the shadow
+logs (see below).
+
+```yaml
+llm:
+  critique:
+    engine: jev           # LLM_CRITIQUE_ENGINE, --critique-engine jev
+  jev:
+    model: jev-1.13.0     # LLM_JEV_MODEL; a pinned version, an alias is an error here
+    fallback: model       # LLM_JEV_FALLBACK: model or fail
+    keep_above: 0.5       # LLM_JEV_KEEP_ABOVE; enough_context, version_claim, duplicate in YAML
+```
+
+With `engine: jev` Jev answers the questions of `prompts/jev_questions.yml`
+about every candidate and decides:
+
+- claims that a version does not exist — reject;
+- not enough in the request to judge it, or too large for a Jev request —
+  unverifiable: with `fallback: model` such candidates go to the LLM
+  Critique (only them), with `fallback: fail` they are rejected;
+- otherwise keep when `real_issue` reaches `keep_above`;
+- duplicates are dropped once, over what Jev and the LLM kept together: the
+  more severe finding stays, on a tie the one the first pass listed first.
+
+When Jev fails (network, 429/529 after one retry, an invalid answer), the
+LLM Critique takes over with `fallback: model`, and the run fails with
+`fallback: fail`. The report says how Jev took part: checked by Jev without
+refining the wording, partly by the LLM, or by the LLM because Jev was
+unavailable. `engine: jev` without `JEV_API_KEY` or with an alias is a
+configuration error at start; `--no-critique` switches off every engine.
+
+What an LLM is still needed for follows the engine: with `fallback: fail`
+there is no LLM Critique at all, so its model and key are not required
+(generate on Ollama with Jev needs no Gemini key), the context budget counts
+generate only, the pool's critique policy is not used and `models check`
+probes the generate stage only. `models check` also sends Jev a probe
+request: with `engine: jev` it counts, in shadow mode it is shown but never
+fails the check. `--dry-run` shows the Jev request for the stub candidate
+(`=== JEV STATE ===`, `=== JEV QUESTIONS ===`).
+
+The review key follows the engine too: with `engine: jev` it includes the Jev
+version, every threshold, the fallback and the question templates; the LLM
+Critique model counts only while Jev can fall back to it. With the default
+engine the key is what it was before Jev, whatever `llm.jev` says. Without
+an LLM Critique (Jev with `fallback: fail`, or `--no-critique`) the critique
+settings — model, limit, start, rank, allow_weaker — are neither validated
+nor part of the key, while the generate pool stays in it.
+
+### Jev shadow (experiment)
+
+Jev is a classifier, not a text model: it answers yes/no and choice
+questions about a given state with probabilities. Before it decides anything
+as the engine, it can run in shadow mode next to the default engine: after
+the LLM Critique the same candidates go to Jev, and its decisions are logged
+next to the Critique verdicts. The report and the review key do not change;
+with `engine: jev` the shadow is ignored.
+
+```yaml
+llm:
+  jev:
+    shadow: true          # LLM_JEV_SHADOW=true
+    model: jev-1.13.0     # LLM_JEV_MODEL; a pinned version, not jev-latest
+    timeout: 10           # seconds per request
+    keep_above: 0.5       # provisional thresholds, only for the log line
+    enough_context: 0.5
+    version_claim: 0.5
+    duplicate: 0.5
+```
+
+The key is `JEV_API_KEY`. For every candidate Jev is asked whether it is a
+real, well-supported problem (the rules of `prompts/critique.txt`, asked in
+`prompts/jev_questions.yml`), whether the state holds enough to judge it,
+whether it claims that a version does not exist, which other candidate it
+duplicates, and how severe it is. The log gets one line per candidate and a
+summary:
+
+```
+Jev shadow C1: keep (real issue; real_issue=0.91 enough_context=0.88 version_claim=0.02 duplicate_of=none/0.97 severity=major/0.74), critique: keep
+Jev shadow: agrees with critique on 2 of 2 decided candidate(s); keep 1, reject 1, unverifiable 1 (model=jev-1.13.0, requests=1, 0.4s)
+```
+
+The readable lines round the numbers; a third line, `Jev shadow data: {…}`,
+carries every decision with the probabilities exactly as Jev returned them,
+so thresholds can be chosen from the logs afterwards. The version that
+answered is logged too, and a pinned version answering as another one is a
+warning. A Jev failure (network, 429/529 after
+one retry, an invalid answer, no key) is a warning and never affects the
+review. Jev goes out through `LLM_HTTP_PROXY` like the LLM providers.
+
+What leaves for TypeSafe: the MR title and description, the Jira section,
+`review_instructions`, the diff shown to the models and the candidates, after
+the same secret scrubbing as the LLM prompts. Jev limits a request to 32k
+tokens for the state plus the longest question and 64k for the state plus all
+questions; the state is cut to what the questions leave (the diff first, then
+the MR and Jira sections, never the candidates and the hunks they point at),
+and when the candidates do not fit together, each goes in a request of its
+own. The size is estimated from characters; if Jev still rejects a request
+as too large (a 422 that explicitly says it exceeds a limit), its candidates
+are asked about one by one, and one that is too large alone stays
+unverifiable. Any other 422 is not retried.
+
 ## Usage
 
 ```bash

data/config/.aireview.yml.example

@@ -36,3 +36,9 @@ llm:
     model: gemini-3.8-flash
     fallbacks:
       - gemini-3.7-flash
+  # Jev as the critique engine (critique: {engine: jev}) or next to it, log only
+  # (see README, "Critique engine"); the key is JEV_API_KEY
+  # jev:
+  #   shadow: true
+  #   model: jev-1.13.0
+  #   fallback: model

data/lib/aireview/cli.rb

@@ -99,9 +99,11 @@ module Aireview
         critique_model: options[:critique_model],
         generate_temperature: options[:generate_temperature],
         critique_temperature: options[:critique_temperature],
-        no_fallbacks: options[:no_fallbacks] == true
+        critique_engine: options[:critique_engine],
+        no_fallbacks: options[:no_fallbacks] == true,
+        no_critique: options[:no_critique] == true
       )
-      config.require_llm_configuration!
+      config.require_llm_configuration!(critique: !options[:no_critique])
       config.warnings.each { |warning| @logger.warn(warning) }
       config
     end
@@ -335,6 +337,11 @@ module Aireview
         options[:critique_temperature] = value
       end
 
+      parser.on('--critique-engine ENGINE', Aireview::Config::CRITIQUE_ENGINES,
+                'Who checks the candidates: model (an LLM) or jev') do |value|
+        options[:critique_engine] = value
+      end
+
       parser.on('--no-critique', 'Skip critique pass and render Generate candidates directly') do
         options[:no_critique] = true
       end
@@ -407,6 +414,8 @@ module Aireview
                            Override Generate pass temperature
           --critique-temperature VALUE
                            Override Critique pass temperature
+          --critique-engine ENGINE
+                           Who checks the candidates: model (an LLM, default) or jev
           --config PATH    Path to .aireview.yml
           --review-mode MODE
                            How to treat an existing review: update (default) or once

data/lib/aireview/config.rb

@@ -6,6 +6,7 @@ require_relative 'errors'
 require_relative 'utils'
 require_relative 'config_limits'
 require_relative 'config_fallbacks'
+require_relative 'config_jev'
 require_relative 'config_layers'
 require_relative 'config_loader'
 
@@ -16,6 +17,7 @@ module Aireview
   class Config
     include ConfigLimits
     include ConfigFallbacks
+    include ConfigJev
     include ConfigLayers
 
     DEFAULT_SECRET_FILES = [
@@ -76,20 +78,27 @@ module Aireview
 
     # CLI overrides change only the primary model of a stage, the reserves
     # from the config stay; no_fallbacks leaves one model and one key.
-    def with_overrides(
+    # no_critique — --no-critique: the run has no Critique, so its settings
+    # are neither routed nor validated (see llm_stages).
+    # Seven named flags of the CLI read better than a struct for its own sake.
+    def with_overrides( # rubocop:disable Metrics/ParameterLists
       generate_model: nil,
       critique_model: nil,
       generate_temperature: nil,
       critique_temperature: nil,
-      no_fallbacks: false
+      critique_engine: nil,
+      no_fallbacks: false,
+      no_critique: false
     )
       llm_config = {
         'generate' => stage_overrides(model: generate_model, temperature: generate_temperature),
         'critique' => stage_overrides(model: critique_model, temperature: critique_temperature)
+          .merge({'engine' => critique_engine}.compact)
       }.reject { |_, overrides| overrides.empty? }
       overrides = {}
       overrides['llm'] = llm_config unless llm_config.empty?
       overrides['fallbacks_disabled'] = true if no_fallbacks
+      overrides['critique_disabled'] = true if no_critique
       return self if overrides.empty?
 
       self.class.new(
@@ -137,8 +146,9 @@ module Aireview
       routing.primary('generate').model
     end
 
+    # nil when no LLM Critique can run (see llm_stages).
     def critique_model
-      routing.primary('critique').model
+      routing.stage?('critique') ? routing.primary('critique').model : nil
     end
 
     def generate_provider
@@ -147,18 +157,26 @@ module Aireview
 
     # Everything besides the prompt that affects the review result goes into
     # the note key (see ReviewMarker): provider, model and temperature of the
-    # stages, the shared pool with its critique policy. Reserves of per-stage
-    # chains do not change the result.
+    # stages, Jev as the critique engine, the shared pool with its critique
+    # policy. Reserves of per-stage chains do not change the result.
     def result_signature
       {
-        'generate' => [generate_provider, generate_model, generate_temperature],
-        'critique' => [critique_provider, critique_model, critique_temperature],
+        'generate' => stage_signature('generate', generate_temperature),
+        'critique' => (routing.stage?('critique') ? stage_signature('critique', critique_temperature) : nil),
+        'critique_engine' => jev_signature,
         'pool' => routing.signature
       }
     end
 
     def critique_provider
-      routing.primary('critique').provider
+      routing.stage?('critique') ? routing.primary('critique').provider : nil
+    end
+
+    # The primary of a stage on a server of your own goes into the review key
+    # with its address; without one the signature is what it was.
+    def stage_signature(stage, temperature)
+      primary = routing.primary(stage)
+      [primary.provider, primary.model, temperature, *primary.api_base]
     end
 
     def generate_temperature

data/lib/aireview/config_fallbacks.rb

@@ -27,6 +27,19 @@ module Aireview
       routing.chain(stage)
     end
 
+    # The keys a model is called with. A server of your own (a model with
+    # api_base) never gets the provider's key — OPENAI_API_KEY must not leave
+    # for a self-hosted server — but LLM_API_KEY, or a placeholder for a
+    # server without auth, since the client wants some key.
+    SELF_HOSTED_PLACEHOLDER_KEY = 'no-key'
+
+    def candidate_api_keys(candidate)
+      return provider_api_keys(candidate.provider) unless candidate.api_base
+      return [nil] if KEYLESS_PROVIDERS.include?(candidate.provider.to_s)
+
+      [Aireview::Utils.presence(llm_api_key) || SELF_HOSTED_PLACEHOLDER_KEY]
+    end
+
     # Keys in order of preference; a single nil for a keyless provider, so
     # that walking the chain does not depend on the provider.
     def provider_api_keys(provider)
@@ -47,9 +60,9 @@ module Aireview
     end
 
     # Only the number of keys per provider, for --dry-run; the values never
-    # leave.
+    # leave. Servers of your own are not counted: they take LLM_API_KEY.
     def api_key_counts(stages)
-      providers = stages.flat_map { |stage| stage_chain(stage).map(&:provider) }.uniq
+      providers = stages.flat_map { |stage| stage_chain(stage).reject(&:api_base).map(&:provider) }.uniq
       providers.reject { |provider| KEYLESS_PROVIDERS.include?(provider.to_s) }
         .to_h { |provider| [provider, provider_api_keys(provider).size] }
     end
@@ -67,39 +80,53 @@ module Aireview
                         'llm.overloaded_quarantine')
     end
 
-    def require_models!
+    # critique — the run has a critique (false with --no-critique); only the
+    # stages that need an LLM in such a run are required (see llm_stages).
+    def require_models!(critique: true)
+      stages = llm_stages(critique: critique)
       missing = []
       missing << 'llm.generate.model (or LLM_GENERATE_MODEL)' if Aireview::Utils.blank?(generate_model)
-      missing << 'llm.critique.model (or LLM_CRITIQUE_MODEL)' if Aireview::Utils.blank?(critique_model)
+      if stages.include?('critique') && Aireview::Utils.blank?(critique_model)
+        missing << 'llm.critique.model (or LLM_CRITIQUE_MODEL)'
+      end
       raise ConfigError, "LLM models are required: #{missing.join(', ')}" unless missing.empty?
     end
 
     # The plan is built whole (pool and critique policy validated) before the
-    # first request, not in Critique after a paid-for Generate.
-    def require_llm_configuration!
-      require_models!
+    # first request, not in Critique after a paid-for Generate. Keys are
+    # required for the stages that need an LLM: generate on Ollama with Jev
+    # as the critic and no fallback needs no Gemini key.
+    def require_llm_configuration!(critique: true)
+      require_models!(critique: critique)
+      require_jev!(critique: critique)
       routing
 
-      missing_keys = STAGES.flat_map do |stage|
-        providers = stage_chain(stage).map(&:provider).uniq.reject { |provider| provider_keys_present?(provider) }
-        providers.map { |provider| "#{stage}: API key is required for provider #{provider.inspect}" }
+      missing_keys = llm_stages(critique: critique).flat_map do |stage|
+        providers = stage_chain(stage).reject(&:api_base).map(&:provider).uniq
+        providers.reject { |provider| provider_keys_present?(provider) }
+          .map { |provider| "#{stage}: API key is required for provider #{provider.inspect}" }
       end
       raise ConfigError, missing_keys.join(', ') unless missing_keys.empty?
     end
 
     private
 
+    # Only the stages that go to an LLM are routed (llm_stages): without an
+    # LLM Critique (Jev with fallback: fail, --no-critique) its model, limit,
+    # start, rank and allow_weaker are not used, so they must neither fail
+    # the run nor change the review key, nor push the pool out of it.
     def build_routing
-      settings = STAGES.to_h { |stage| [stage, stage_settings(stage)] }
+      stages = llm_stages
+      settings = stages.to_h { |stage| [stage, stage_settings(stage)] }
       models = Array(dig('llm', 'models'))
       return StageChains.build(settings, only_primary: fallbacks_disabled?) if models.empty?
 
       own = settings.select { |_, stage_settings| Aireview::Utils.present?(stage_settings[:model]) }
       ModelPool.new(
-        items: models, provider: llm_provider,
-        limits: STAGES.to_h { |stage| [stage, max_prompt_chars(stage)] },
-        starts: STAGES.to_h { |stage| [stage, dig('llm', stage, 'start')] },
-        inherited_starts: STAGES.select { |stage| start_inherited?(stage) },
+        items: models, provider: llm_provider, stages: stages,
+        limits: stages.to_h { |stage| [stage, max_prompt_chars(stage)] },
+        starts: stages.to_h { |stage| [stage, dig('llm', stage, 'start')] },
+        inherited_starts: stages.select { |stage| start_inherited?(stage) },
         rank: dig('llm', 'critique', 'rank'), allow_weaker: dig('llm', 'critique', 'allow_weaker'),
         own_chains: StageChains.build(own, only_primary: fallbacks_disabled?), only_primary: fallbacks_disabled?
       )
@@ -109,6 +136,7 @@ module Aireview
       {
         provider: stage_setting(stage, 'provider') || llm_provider,
         model: dig('llm', stage, 'model'),
+        api_base: dig('llm', stage, 'api_base'),
         fallbacks: dig('llm', stage, 'fallbacks'),
         max_prompt_chars: max_prompt_chars(stage)
       }

data/lib/aireview/config_jev.rb

@@ -0,0 +1,137 @@
+# frozen_string_literal: true
+require_relative 'errors'
+require_relative 'stages'
+require_relative 'utils'
+
+module Aireview
+  # The critique engine and the Jev (TypeSafe) settings. llm.critique.engine
+  # picks who checks the candidates: an LLM (model, the default) or Jev, a
+  # fast classifier that decides keep/reject but cannot refine a finding.
+  # llm.jev.fallback says what happens when Jev cannot decide: the LLM
+  # Critique takes over (model) or the run fails (fail). llm.jev.shadow runs
+  # Jev next to the LLM Critique for the log only.
+  module ConfigJev
+    CRITIQUE_ENGINES = %w[model jev].freeze
+    JEV_FALLBACKS = %w[model fail].freeze
+    # A pinned version, not an alias: thresholds are tuned against one
+    # version, and an alias moves when a release ships.
+    DEFAULT_JEV_MODEL = 'jev-1.13.0'
+    JEV_ALIASES = %w[jev-latest jev-preview].freeze
+    DEFAULT_JEV_TIMEOUT = 10
+    # Provisional values until real thresholds are chosen from the shadow
+    # logs; the logs carry the raw probabilities for that.
+    JEV_THRESHOLD_DEFAULTS = {
+      'keep_above' => 0.5,
+      'enough_context' => 0.5,
+      'version_claim' => 0.5,
+      'duplicate' => 0.5
+    }.freeze
+
+    def critique_engine
+      one_of!(dig('llm', 'critique', 'engine') || 'model', CRITIQUE_ENGINES, 'llm.critique.engine')
+    end
+
+    def jev_fallback
+      one_of!(dig('llm', 'jev', 'fallback') || 'model', JEV_FALLBACKS, 'llm.jev.fallback')
+    end
+
+    # Jev decides in this run: the engine is jev and the run has a critique
+    # at all (--no-critique switches off every engine, as the critique:
+    # argument or as a CLI override of the config).
+    def jev_critique?(critique: true)
+      critique && !critique_disabled? && critique_engine == 'jev'
+    end
+
+    # The stages that need an LLM in a run. Jev with fallback: model still
+    # needs a ready LLM Critique; with fallback: fail it needs none.
+    def llm_stages(critique: true)
+      return ['generate'] unless critique && !critique_disabled?
+      return ['generate'] if jev_critique? && jev_fallback == 'fail'
+
+      STAGES
+    end
+
+    def critique_disabled?
+      @data['critique_disabled'] == true
+    end
+
+    def jev_shadow?
+      value = dig('llm', 'jev', 'shadow')
+      return false if value.nil?
+      return value if [true, false].include?(value)
+
+      raise ConfigError, "llm.jev.shadow must be true or false, got #{value.inspect}"
+    end
+
+    def jev_model
+      Aireview::Utils.presence(dig('llm', 'jev', 'model')) || DEFAULT_JEV_MODEL
+    end
+
+    def jev_api_key
+      @data['jev_api_key']
+    end
+
+    def jev_timeout
+      positive_integer!(dig('llm', 'jev', 'timeout') || DEFAULT_JEV_TIMEOUT, 'llm.jev.timeout')
+    end
+
+    def jev_thresholds
+      JEV_THRESHOLD_DEFAULTS.to_h do |name, default|
+        value = dig('llm', 'jev', name)
+        [name.to_sym, value.nil? ? default : probability!(value, "llm.jev.#{name}")]
+      end
+    end
+
+    # Everything besides the questions that changes a Jev decision; nil when
+    # Jev does not decide. The fallback is part of it: it decides what
+    # happens to the candidates Jev could not judge.
+    def jev_signature
+      return nil unless critique_engine == 'jev'
+
+      ['jev', jev_model, *jev_thresholds.values, jev_fallback]
+    end
+
+    # Jev as the critic needs a key and a pinned version: the thresholds are
+    # tuned against one version, and falling back to the LLM silently would
+    # hide a broken setup.
+    def require_jev!(critique: true)
+      return unless jev_critique?(critique: critique)
+      raise ConfigError, 'llm.critique.engine is jev, but JEV_API_KEY is not set' if Aireview::Utils.blank?(jev_api_key)
+      return unless JEV_ALIASES.include?(jev_model)
+
+      raise ConfigError, "llm.jev.model #{jev_model.inspect} is an alias; with llm.critique.engine: jev " \
+                         'pin a version such as jev-1.13.0'
+    end
+
+    def jev_warnings
+      return [] unless jev_shadow?
+      return ['llm.jev.shadow is ignored: Jev already decides as the critique engine'] if critique_engine == 'jev'
+
+      warnings = []
+      if Aireview::Utils.blank?(jev_api_key)
+        warnings << 'llm.jev.shadow is on, but JEV_API_KEY is not set: Jev is skipped'
+      end
+      if JEV_ALIASES.include?(jev_model)
+        warnings << "llm.jev.model #{jev_model.inspect} is an alias: its answers change when TypeSafe ships " \
+                    'a release; pin a version such as jev-1.13.0'
+      end
+      warnings
+    end
+
+    private
+
+    def one_of!(value, allowed, name)
+      value = value.to_s
+      return value if allowed.include?(value)
+
+      raise ConfigError, "#{name} must be one of #{allowed.join(', ')}, got #{value.inspect}"
+    end
+
+    def probability!(value, name)
+      number = Float(value, exception: false) if value.is_a?(Numeric) || value.is_a?(String)
+      return number if number&.between?(0, 1)
+
+      raise ConfigError, "#{name} must be a number from 0 to 1, got #{value.inspect}"
+    end
+  end
+end

data/lib/aireview/config_layers.rb

@@ -65,7 +65,7 @@ module Aireview
 
     # Configuration and plan warnings; the CLI and --dry-run print them.
     def warnings
-      STAGES.flat_map { |stage| stage_provider_warnings(stage) } + routing.warnings
+      llm_stages.flat_map { |stage| stage_provider_warnings(stage) } + routing.warnings + jev_warnings
     end
 
     # The paths of the file layers, for --dry-run.