aireview 0.2.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6f428bec4ae96692e65cd9f219856dff87e04e75dec45448dc9ef6806dd6694a
-  data.tar.gz: 515afe0ed7450df6cfcb2ce12040780a9c0f45bbbc45fef08f19c4601239908b
+  metadata.gz: 3f326ae1b11f59f7c447e8f128ab94845ef1543b0b377753f053fd4f69e03bad
+  data.tar.gz: c5908173ca8020a0563b6d4ee1fd4d8ce4ebe609fda7ded26d5b7eb40c84dda1
 SHA512:
-  metadata.gz: 97cf66d34a0dee0d02823a3d37ad37b85aefeb4a0e2b997cd9f8c0780c3df9816d7c95f520709d3cc668d3910bf749f26553da2001b0878c0935bf117b6a486d
-  data.tar.gz: c9ede525f15c46b1ce3371c7a77394933599bc71c6900f79dcea5a6780a29621378e7cd6d5652480921cefe84e6513c4865bd81265523fb6436752f9b35a422d
+  metadata.gz: 2de9b7dd7e6caaa440287339d03eed843dfdb73010ae8cf6a01dea4e9348e3b072fc4fb61a55bf30eb4ed324e857ff883e35e31b711bc542cb8d1495645041bc
+  data.tar.gz: 5f0867f71415c019c03d99bd2215fc7f7ef7cadf98ab8b0db91e6eb98af54cfae3787b848c7611fc371cfa19cc94acd55e170047f094d8543f5c77f258e3313d

data/CHANGELOG.md

@@ -1,5 +1,71 @@
 # Changelog
 
+## 2.0.0
+
+Breaking: the review key of a shared pool includes the pool order and the
+critique policy; a single chain without reserves gets three requests instead
+of a seventeen-minute schedule; `LLM_PROVIDER` no longer re-targets models
+that carry an explicit provider.
+
+- Defaults shipped in the image: `config/defaults.yml` is a configuration
+  layer between the built-in values and the project's `.aireview.yml`,
+  enabled by `AIREVIEW_DEFAULTS` (set in the Dockerfile). A project needs no
+  model names; `--dry-run` prints the layer every setting came from and
+  warns when inherited reserves silently change provider.
+- Shared model pool: `llm.models` (`LLM_MODELS`) in order of priority,
+  `generate.start`, `critique.rank: not_below_generate|any` and
+  `critique.allow_weaker`. Generate walks the pool round from its start;
+  Critique takes the first live model not below the one that answered in
+  Generate, never weaker unless allowed. A stage with a `model` of its own
+  keeps an independent chain.
+- Router: an overloaded model or a timeout gets one short retry, then a
+  quarantine (`llm.overloaded_quarantine`, `LLM_OVERLOADED_QUARANTINE`,
+  120 s) and the next model; the walk goes round the pool once every model
+  has been tried, waiting for the nearest quarantine to end. Three requests
+  per model per stage, counted across keys and including the JSON repair;
+  the last model has no special schedule. A model the provider does not
+  have (retired, a typo, not pulled into Ollama) is excluded for the run
+  instead of failing it; a bare 404 stays fatal.
+- An invalid result that the same model cannot repair excludes that model
+  for the stage and restarts the stage on the next model with the original
+  request; Critique keeps the candidates already obtained.
+- `aireview models check`: two probe requests per model with the production
+  schemas and validation, statuses `ok / missing / invalid / unverified /
+  failed / skipped`, `--strict` turns a skipped (unreachable) Ollama into a
+  failure. Meant for release pipelines and schedules.
+- `templates/review.gitlab-ci.yml`: a job template projects include with
+  `ref: stable`; every config variable is passed into the container by name.
+- The log names the model that answered each stage and its place in the
+  chain; the report notes a critique that ran on a weaker model.
+- Fixed: with `json` 3.0 every provider answer failed to parse
+  (`Faraday::ParsingError`, "wrong number of arguments") — `faraday` 2.14.4
+  in the lockfile. The suite did not catch it because it stubs the network;
+  `aireview models check` did.
+- Internals: routing plans (`StageChains`, `ModelPool`) and the router's
+  `ModelState` are objects, `ConfigLoader` owns loading and env parsing,
+  `ResultParser` parses LLM results, `LlmClient` makes one request while
+  the router retries. Specs run against the real RubyLLM classes.
+
+## 0.3.0
+
+- Fallback models per stage (`llm.<stage>.fallbacks`,
+  `LLM_<STAGE>_FALLBACK_MODEL`) and fallback API keys (`GEMINI_API_KEYS`).
+  An overloaded model or a timeout switches to the next model after one
+  short retry; an exhausted daily quota switches to the next key without
+  waiting; the last route keeps the full retry schedule.
+- A time budget for the LLM part of the run (`llm.time_budget`,
+  `LLM_TIME_BUDGET`, 30 minutes by default) bounds pauses and request
+  timeouts; the built-in RubyLLM retries are disabled.
+- The report names the fallback model a stage used; `--dry-run` prints the
+  chains and the number of keys; `--no-fallbacks` disables the reserves.
+- Candidates are checked against the diff shown to the model: a file outside
+  the merge request drops the candidate, a line outside the shown hunks is
+  reset, a quote not found in the diff is marked in the report and reported
+  to Critique.
+- Critique is skipped when Generate returns no candidates.
+- Report limits are applied after selecting the findings to show, so a
+  finding that is never shown no longer displaces a useful one.
+
 ## 0.2.1
 
 - An overloaded LLM (503) gets a fifth attempt: the pauses are now about

data/README.md

@@ -49,8 +49,10 @@ installed, replace it with plain `aireview`.
 
 ## Configuration
 
-Secrets live in environment variables or in a local `.env` file. In `.env` the
-Generate and Critique models are set explicitly:
+Secrets live in environment variables or in a local `.env` file. The models of
+both stages come from the defaults shipped in the Docker image (see "Defaults
+shipped in the image"); outside the image they are set in `.env` or
+`.aireview.yml`:
 
 ```bash
 GITLAB_URL=https://gitlab.company.com
@@ -59,15 +61,19 @@ JIRA_URL=https://jira.company.com
 JIRA_LOGIN=user
 JIRA_PASSWORD=xxx
 GEMINI_API_KEY=xxx
+GEMINI_API_KEYS=xxx,yyy
 LLM_PROVIDER=gemini
 LLM_TEMPERATURE=0
-LLM_TIMEOUT=60
+LLM_TIMEOUT=120
+LLM_TIME_BUDGET=1800
 LLM_HTTP_PROXY=http://127.0.0.1:8888
 LLM_GENERATE_PROVIDER=gemini
 LLM_GENERATE_MODEL=gemini-3.7-flash
+LLM_GENERATE_FALLBACK_MODEL=gemini-3.8-flash
 LLM_GENERATE_TEMPERATURE=0.3
 LLM_CRITIQUE_PROVIDER=gemini
 LLM_CRITIQUE_MODEL=gemini-3.8-flash
+LLM_CRITIQUE_FALLBACK_MODEL=gemini-3.7-flash
 LLM_CRITIQUE_TEMPERATURE=0
 REVIEW_LANGUAGE=ru
 REVIEW_MODE=update
@@ -87,6 +93,88 @@ If only the LLM traffic has to go through a proxy, set `LLM_HTTP_PROXY` or
 `llm.http_proxy`. That configures RubyLLM only; requests to GitLab and Jira
 keep going directly.
 
+### Defaults shipped in the image
+
+Models, timeouts and the report language are baked into the image as
+`config/defaults.yml`; inside the image `AIREVIEW_DEFAULTS` points at it. A
+project in an organization has nothing to configure: no `.aireview.yml`, no
+variables with model names. Changing the model for every project is one
+edit of that file and a new image release. Outside the image the variable is
+unset and the layer does not exist, so local runs are unaffected; point
+`AIREVIEW_DEFAULTS` at any file to get the same layer elsewhere. The
+defaults define a shared pool of five Gemini models (see "Shared model
+pool"): Generate starts from a mid-range model and goes round the pool,
+Critique takes the strongest live model not below the one Generate answered
+with; every model of the pool is confirmed by `aireview models check` on
+release.
+
+The configuration layers, weakest first:
+
+1. built-in values (`Config::DEFAULTS`);
+2. image defaults — the file from `AIREVIEW_DEFAULTS`; no variable, no layer;
+3. the project's `.aireview.yml`;
+4. environment variables (`LLM_GENERATE_MODEL` and the rest);
+5. CLI flags (`--generate-model`, `--critique-model`, the temperatures).
+
+Layers merge by key, so a project overrides only what it needs — with two
+caveats about the pool:
+
+- a `llm.generate.model` of its own (in YAML or `LLM_GENERATE_MODEL`) **takes
+  the stage out of the pool**: its chain is that model plus its own
+  `fallbacks`, if any; the image models are not picked up for that stage and
+  the critique rank rule does not apply. To change only the starting model
+  while staying in the pool, set `llm.generate.start` / `LLM_GENERATE_START`
+  — or replace the whole pool through `llm.models` / `LLM_MODELS`;
+- every pool model in the image carries an explicit provider (`gemini`), so
+  `LLM_PROVIDER=ollama` on its own does not move the Gemini pool to Ollama:
+  it only changes the default provider of models without one. A project on
+  Ollama sets its own models — its own pool (`LLM_MODELS=ollama/qwen2.5-coder:7b,…`)
+  or per-stage `model`s.
+
+The shared settings `llm.temperature`, `llm.max_prompt_chars` and the default
+provider `llm.provider` (env: `LLM_TEMPERATURE`, `LLM_MAX_PROMPT_CHARS`,
+`LLM_PROVIDER`) apply to the stages layer by layer: a stage value from the
+image defaults does not beat a shared value from the project or the
+environment, while within one layer the stage value still wins. For stages
+on their own chains the `fallbacks` array is replaced as a whole when given
+explicitly; `fallbacks: []` removes the reserves; a reserve without a
+`provider` of its own inherits the stage provider — if a project overrode the
+stage provider (`LLM_GENERATE_PROVIDER=ollama`) while inherited reserves
+without an explicit provider remain in the chain, they silently become
+"models" of the new provider. `--dry-run` warns about that and prints, for
+every model, the layer it came from:
+
+```
+=== LLM SETTINGS ===
+Config: image defaults /app/config/defaults.yml, .aireview.yml /app/.aireview.yml
+Generate: gemini-3.7-flash temperature=0.1 (model from image defaults, provider from image defaults)
+  fallbacks: gemini/gemini-3.6-flash -> ... (fallbacks from image defaults)
+Critique: qwen2.5-coder:7b temperature=0 (model from .aireview.yml, provider from .aireview.yml)
+Time budget: 1800s, overloaded quarantine: 120s
+```
+
+In pool mode the model source is the layer that set the stage `start` or the
+`llm.models` list itself; `--critique-model` with a pool model shows up as
+`model from cli`.
+
+An `AIREVIEW_DEFAULTS` that points at a missing file is a configuration
+error: such an image was built wrong, and failing at once beats reviewing
+with the wrong models. Keys and tokens never live in `config/defaults.yml`.
+
+Changing the default models or the pool order changes the review key (see "A
+single comment per merge request"). What happens to open MRs depends on
+`review_mode`: in `update` the review re-runs on the next push to the MR, in
+`once` (as in the CI template) only on a job Retry. In `update` this is a
+one-off burst of requests to the provider — better not to ship such a
+release at peak hours.
+
+No model is probed before a review: there are no test requests to five
+models per MR, availability is learned from the working requests — the first
+Generate request is the probe, and what the router learns it remembers until
+the end of the run (see "Fallback models and keys"). The separate smoke test
+of every model with both schemas (`aireview models check`) runs on an image
+release and on a schedule, not on MRs.
+
 ### Local Ollama
 
 Install Ollama following the
@@ -162,14 +250,19 @@ To run both stages locally, set `ollama` in both provider variables. The
 address with `/v1` matches the
 [Ollama configuration in RubyLLM](https://rubyllm.com/configuration/#provider-configuration).
 `LLM_TIMEOUT` sets the timeout of every LLM request in seconds; for a slow
-local model it can be raised. It does not apply to an "overloaded" (503)
-answer from the provider: such a request gets up to five attempts, the
-original one and four retries with pauses of about 2, 5, 5 and 5 minutes, and
-only the failed stage is repeated, not the whole run.
-
-Project rules live in `.aireview.yml`. In YAML the `generate.model` and
-`critique.model` settings are required for each stage and are not inherited
-from the base `llm` settings. `llm.provider` is used as the default when
+local model it can be raised, but not without limit: a hung request holds
+the job for exactly that long while a fallback model sits idle. An
+"overloaded" (503) answer from the provider or a timeout does not fail the
+run right away: the model gets one short retry (~30 s), then goes into
+quarantine for two minutes while the request goes to the next model;
+without a fallback model the router waits for the quarantine to end and
+tries once more. Three requests per model per stage in total (see "Fallback
+models and keys"). Only the failed stage is repeated, not the whole run.
+
+Project rules live in `.aireview.yml`. The `generate.model` and
+`critique.model` settings are not inherited from the base `llm` settings: a
+review does not start when a stage has no model in the image defaults, the
+YAML or the environment. `llm.provider` is used as the default when
 `generate.provider` or `critique.provider` is not set:
 
 ```yaml
@@ -213,13 +306,19 @@ context:
 llm:
   provider: gemini
   temperature: 0
-  timeout: 60
+  timeout: 120
+  time_budget: 1800
   http_proxy: http://127.0.0.1:8888
   max_prompt_chars: 400000
   generate:
     provider: gemini
     model: gemini-3.7-flash
     temperature: 0.3
+    fallbacks:
+      - gemini-3.8-flash
+      - provider: ollama
+        model: qwen2.5-coder:7b
+        max_prompt_chars: 20000
   critique:
     provider: ollama
     model: qwen2.5-coder:7b
@@ -264,6 +363,165 @@ with an error instead of silently reviewing less. Raise the limits or extend
 `ignore_paths`. `--dry-run` prints the sizes of every part and the coverage;
 `--verbose` logs them during a real run.
 
+### Fallback models and keys
+
+The primary model can sit under load (503) for half a day, and a key can run
+out of its daily quota. Neither is cured by waiting, so a stage can have a
+fallback model and a provider can have a fallback key:
+
+- `llm.generate.fallbacks` / `llm.critique.fallbacks` in YAML or
+  `LLM_GENERATE_FALLBACK_MODEL` / `LLM_CRITIQUE_FALLBACK_MODEL` in the
+  environment. One model or several separated by commas, in the order they
+  are tried. The provider goes before a slash (`ollama/qwen2.5-coder:7b`);
+  without it the stage provider is used. A fallback model can have its own
+  `max_prompt_chars`: when the assembled request does not fit, that model is
+  skipped, the context is not cut for it.
+- `GEMINI_API_KEYS=key1,key2`: keys in order of preference; `GEMINI_API_KEY`
+  still works and means a single key. Google counts quotas per Google Cloud
+  project and per model, so a reserve key only makes sense from another
+  project. Use fallback credentials in compliance with the provider's quota
+  and billing terms: Google's
+  [API limitations](https://developers.google.com/terms#api_limitations)
+  forbid circumventing its limits regardless of the project's billing. Keys
+  live only in the environment, never in `.aireview.yml`.
+
+#### Shared model pool
+
+Instead of two independent chains a single pool can be configured:
+`llm.models` in order of priority (the first is the preferred one for
+Critique). Generate goes through the pool from `generate.start` downwards
+and round again; Critique takes the first live model **not below the one
+that actually answered in Generate** (`critique.rank: not_below_generate`,
+the default), with self-critique by the same model as the last permitted
+option. "Above" and "below" are positions in the list, they are not derived
+from model names:
+
+```yaml
+llm:
+  provider: gemini
+  models:                          # order = priority for Critique
+    - gemini-3.8-flash
+    - gemini-3.7-flash
+    - gemini-3.6-flash
+    - provider: ollama
+      model: qwen2.5-coder:7b
+      max_prompt_chars: 20000
+  generate:
+    start: gemini-3.7-flash        # Generate starts here, then downwards and round
+  critique:
+    rank: not_below_generate       # or any — the chains are independent
+    allow_weaker: false            # true — go below when nothing above is alive
+```
+
+Environment equivalents: `LLM_MODELS=gemini/gemini-3.8-flash,gemini/gemini-3.7-flash,…`,
+`LLM_GENERATE_START`, `LLM_CRITIQUE_START`, `LLM_CRITIQUE_RANK`,
+`LLM_CRITIQUE_ALLOW_WEAKER`. The rules:
+
+- no permitted live model for Critique and `allow_weaker: false` — the run
+  fails, candidates are not published without a critique; with
+  `allow_weaker: true` Critique goes below Generate and the report gets a
+  "Critique ran on a model weaker than Generate" line;
+- a stage with a `model` of its own (in YAML, the environment or
+  `--generate-model` with a model outside the pool) does not use the pool:
+  its chain is independent and the rank rule does not apply, as with
+  `rank: any`; `--generate-model` / `--critique-model` with a model **from**
+  the pool bring the stage back into the pool and make that model the start
+  (the stage's own `model` and `fallbacks` are reset), with a model outside
+  the pool — a single chain without reserves;
+- a project that replaces the whole pool (`llm.models` in YAML or
+  `LLM_MODELS`) need not repeat `start`: a start inherited from the image
+  that the new pool does not contain is replaced by the first model of the
+  new pool with a warning in the log. A start set in the same layer as the
+  pool or above it must be in the pool — a typo there stays a configuration
+  error;
+- `critique.start` goes first only when the rank permits it; a start below
+  the model that answered in Generate is skipped with a warning when
+  `allow_weaker` is off — it does not bypass the ban on a weaker critique;
+- the review key (see "A single comment per merge request") includes the
+  whole pool together with the critique policy: its order decides which
+  model checks the findings, not just what to fall back to. Reordering the
+  pool, changing `start`, `rank` or `allow_weaker` — a new key, open MRs are
+  re-reviewed on the next push; moving from per-stage chains to the pool —
+  the same, once;
+- `--dry-run` prints the chains of both stages and a `Critique rule: …` line.
+
+What happens on which error:
+
+| Error | Reaction |
+|---|---|
+| Daily quota (`quotaId` like `…PerDay…` in Google's answer) | No retries: the next key on the same model. The "key + model" pair is remembered until the end of the run so that Critique and the JSON repair do not hit it again. Out of keys: the model is excluded until the end of the run, the request goes to the next one. |
+| Per-minute limit (429 with a "retry in N s" hint) | One retry after the hinted delay, then the next key. Out of keys: the model is quarantined for the hinted time, the request goes to the next one. |
+| Overloaded (503) or timeout | One short retry (~30 s), then a quarantine of `llm.overloaded_quarantine` / `LLM_OVERLOADED_QUARANTINE` (120 s by default) and the next model with the same key. |
+| The provider has no such model (the answer text names the model: retired, a typo, not pulled into Ollama) | No retries: the model is excluded until the end of the run, the request goes to the next one. A bare 404 without such text is fatal: a wrong `LLM_API_BASE` answers the same, and the next model will not help. |
+| An invalid result (not JSON, not the schema, foreign ids in the verdicts) — and still invalid after one repair by the same model | The model is excluded for this stage, the stage starts over on the next model with the original request. For Critique the candidates already obtained are kept, Generate is not repeated. |
+| Other API errors (context length, auth) | The run fails right away: a fallback model would answer the same. |
+
+The chain is walked round: once every model has been tried, the router
+returns to those whose quarantine has expired; when all are quarantined, it
+waits for the nearest release. The last model in the list is nothing
+special, nobody gets a long retry schedule. Two limits keep the walk
+finite:
+
+- **three requests per model per stage** (`MAX_ATTEMPTS_PER_MODEL`). Every
+  request sent counts regardless of its outcome, including the short retry
+  and the JSON repair; the keys of one model share one counter, a quarantine
+  does not reset it. Out of requests — the model leaves the stage; a JSON
+  repair without requests left is not sent, and the result counts as invalid
+  (see the table). When every model of the stage is out or excluded, the
+  stage ends at once, quarantines are not waited for;
+- **the time budget** of the LLM part of the run: `llm.time_budget` /
+  `LLM_TIME_BUDGET`, 1800 seconds by default. A pause or a quarantine wait
+  that does not fit into the remainder is skipped, the request timeout is
+  capped by the remainder; when the time is up, the run fails with an error
+  listing everything that was tried. Because of this keep `LLM_TIMEOUT`
+  around 120–300 seconds: one hung request must not eat the whole budget.
+
+**A single chain** — one model without reserves, typically Ollama on its own
+box — behaves differently from earlier versions: instead of seventeen minutes
+of waiting (retries after 2, 5, 5 and 5 minutes) the model gets three
+requests — the original, a short retry after ~30 s and one more after the
+two-minute quarantine; after the third failure the stage fails. A timeout is
+a failure too, and a quarantine will not help a slow model: it only
+lengthens the pause between attempts. A slow local model needs an
+`LLM_TIMEOUT` with room for the largest request and an `LLM_TIME_BUDGET`
+that fits three such requests with their pauses; `LLM_OVERLOADED_QUARANTINE`
+is about a model that is temporarily overloaded and should come back.
+
+The review key (see "A single comment per merge request") is computed from
+the configured primary model, not from the one that answered: a review made
+by a fallback model is not rewritten on the next push without changes in the
+MR, and adding a fallback model to a per-stage chain does not re-run the
+review on every open MR. When a stage went to a fallback model, the report
+ends with a `Fallback model used: critique — …` line; the log names, for
+every stage, the model that answered and its place in the chain
+(`model=gemini/gemini-3.6-flash (2/5)`). A key switch stays in the log only,
+and key values never reach the log. `--dry-run` prints the model chains, the
+number of keys per provider, the time budget and the quarantine length;
+`--no-fallbacks` leaves one model and one key per stage.
+
+### Checking that findings point at the diff
+
+Between the passes every candidate is checked against the diff that actually
+went to the model (after `ignore_paths`, secret scrubbing and the budget
+cut). What is checked is the link to the code, not the bug itself:
+
+- `file` is not among the changed files of the MR (renames included): the
+  candidate is dropped before Critique, there is nothing to check it
+  against. A file of the MR that was left out of the context by the budget
+  is a different case, see below.
+- `line` falls into none of the shown hunks: the report shows the finding
+  without a line number.
+- `quoted_code` is not found in the shown diff (compared ignoring
+  whitespace, on the new and on the old side): the candidate stays, but the
+  "Where" line of the report gets `(quote not found in the diff)`. Critique
+  cannot fix the quote, so the mark survives its keep.
+- The file is shown partially, without a diff or not at all: the link cannot
+  be checked, and the coverage block of the report already says so.
+
+Critique receives the result of the check in the candidate's `note` field
+and decides keep/reject with it in mind. With `--no-critique` the check works
+the same way, its marks just go straight to the report.
+
 ## Usage
 
 ```bash
@@ -284,9 +542,46 @@ bundle _2.3.26_ exec bin/aireview review https://gitlab.company.com/team/project
 - `--no-jira` turns off the Jira enrichment even when the MR carries an issue key.
 - `--dry-run` prints the LLM settings, the context sizes and coverage, and the Generate prompt, plus the Critique prompt unless `--no-critique` is given.
 - `--no-critique` skips the second pass and renders the Generate candidates directly.
+- `--no-fallbacks` uses only the primary model and the first API key of each stage.
 - `--review-mode MODE` sets the behaviour when a review has already been published: `update` or `once`.
 - `--force` reviews again even when a review for this state of the MR is already published.
 
+### Checking the models
+
+```bash
+bundle _2.3.26_ exec bin/aireview models check
+bundle _2.3.26_ exec bin/aireview models check --config .aireview.yml --strict --verbose
+```
+
+Every model of both stage chains gets two small requests — one with the
+production Generate schema and one with the Critique schema — on a tiny
+synthetic MR, with the same system prompts as a review. The answer goes
+through the same validation as in a run (JSON shape, candidate and verdict
+ids). The provider's catalog is not consulted: "the model is listed" does
+not mean "our request with the schema passes on it". There are no reserves,
+retries or quarantine here — this is a check, not a review; only a
+per-minute limit gets one retry after the provider's hint.
+
+```
+Checking 5 model(s) with the generate and critique schemas
+gemini/gemini-3.7-flash          generate ok (4.6s)
+gemini/gemini-3.7-flash          critique ok (6.9s)
+gemini/gemini-3.6-flash          generate unverified: This model is currently experiencing high demand. …
+gemini/gemini-9.9-nope           generate missing: models/gemini-9.9-nope is not found for API version v1beta, …
+ollama/qwen2.5-coder:7b          generate skipped: Connection refused
+Result: 6 ok, 1 unverified, 2 missing, 2 skipped -> FAILED
+```
+
+Statuses: `ok` — an answer matching the schema; `missing` — the provider
+has no such model (retired, a typo, not pulled into Ollama); `invalid` — it
+answered, but not by the schema; `unverified` — the provider could not
+answer right now (overload, quota, timeout), the model is not at fault but
+not confirmed either; `failed` — other API errors; `skipped` — Ollama is
+unreachable where the check runs (a CI runner has none). The exit code is 0
+only when every model is `ok` or `skipped`; with `--strict` — only `ok`: on
+a box where Ollama must be running, an unreachable Ollama is a failed check,
+not a skip.
+
 ### A single comment per merge request
 
 With `--post` the review is not published as a new note every time; it updates
@@ -372,13 +667,13 @@ aireview:
       - stuck_or_timeout_failure
 ```
 
-`timeout: 45m` is a chosen ceiling, not a guarantee that every retry fits in:
-when the provider is overloaded, one stage can wait up to ~20 minutes of
-pauses plus up to five requests of `LLM_TIMEOUT` each, and there are two
-stages.
+`timeout: 45m` is the job ceiling, not a guarantee: the LLM part of the run
+is itself bounded by `LLM_TIME_BUDGET` (30 minutes by default, pauses and
+fallback models included), the rest is reading the MR and Jira and
+publishing.
 
-Set secrets such as `GITLAB_TOKEN`, `GEMINI_API_KEY` and the optional Jira
-credentials in the GitLab CI/CD variables. If the job should publish the result
+Set secrets such as `GITLAB_TOKEN`, `GEMINI_API_KEY` / `GEMINI_API_KEYS` and
+the optional Jira credentials in the GitLab CI/CD variables. If the job should publish the result
 back to the merge request, add `--post` to the review command.
 
 For runners where the LLM provider is only reachable over WireGuard, bring up a
@@ -387,6 +682,46 @@ local HTTP proxy (`wireproxy`, for instance) before `aireview` starts and point
 GitLab and Jira stay on direct connections while RubyLLM goes through the
 tunnel.
 
+### Including the job template
+
+Instead of copying the job into every project, it lives in this repository
+as `templates/review.gitlab-ci.yml`. A project's `.gitlab-ci.yml` keeps only:
+
+```yaml
+include:
+  - project: your-group/aireview
+    ref: stable
+    file: /templates/review.gitlab-ci.yml
+```
+
+The template is written for shell-executor runners (the image runs through
+`docker run` with secrets passed by name) and carries `[skip review]`,
+`resource_group`, `allow_failure` and `REVIEW_MODE=once`. Models come from
+the image defaults, `.aireview.yml` is mounted into the container only when
+the project has one. The job runs in the `.post` stage — it exists in every
+pipeline, so a project does not declare `stages`; to move the review to
+another stage, add `aireview: {stage: review}` to the project file. Every
+variable the config reads (`Config.env_names`: models, providers, reserves,
+temperatures, limits, `OLLAMA_API_BASE` and so on) is passed into the
+container by name, so a project can override anything through its CI/CD
+variables — for instance, swap an unavailable model with `LLM_CRITIQUE_MODEL`
+without waiting for an image release.
+
+Secrets stay CI/CD variables **of the project**: group variables are readable
+by any merge request of any project in the group.
+
+The `stable` branch is managed: the copy of the template there pins
+`AIREVIEW_IMAGE_TAG` to a verified release. Updating or rolling back every
+project that includes the template is one edit of `stable`. A project can
+pin another version by overriding `AIREVIEW_IMAGE_TAG` in its `variables`.
+
+A release of the image is: tag → `aireview models check` on the image
+defaults (an API error or an answer off the schema for any model stops the
+pipeline, the image is not built, the previous version stays in the
+registry) → build and push → set the new tag in `stable`. The same check on
+a schedule (once a day) is the early signal that the provider retired a
+model — otherwise the first to learn about it is a live MR.
+
 ## Docker
 
 ```bash

data/config/.aireview.yml.example

@@ -27,5 +27,12 @@ review_instructions: |
 
 llm:
   provider: gemini
-  model: gemini-3.7-flash
   temperature: 0.2
+  generate:
+    model: gemini-3.7-flash
+    fallbacks:
+      - gemini-3.8-flash
+  critique:
+    model: gemini-3.8-flash
+    fallbacks:
+      - gemini-3.7-flash

data/config/defaults.yml

@@ -0,0 +1,47 @@
+# Defaults baked into the aireview image. This layer sits between the
+# built-in values and the project's .aireview.yml: a project overrides
+# anything here, the environment and CLI flags override the project. No
+# keys live here.
+#
+# One shared pool of models (see README, "Shared model pool"): the order is
+# the priority for Critique, the first model is the preferred one. Generate
+# starts at generate.start and goes down the list and round again; Critique
+# takes the first live model not below the one that answered in Generate
+# and never goes lower.
+#
+# Every model carries an explicit provider: a model without one inherits
+# llm.provider, and a project that switched it to ollama would otherwise get
+# "Ollama models" with Gemini names.
+#
+# Model names come from the Gemini catalog (GET /v1beta/models), text models
+# without "preview" in the name only; each one is confirmed by
+# `aireview models check` with both schemas (2026-09-18).
+
+review_language: en
+
+llm:
+  provider: gemini
+  temperature: 0
+  # Three attempts per model × timeout is the ceiling for one hung model:
+  # 300 s meant 15 minutes, 180 s means 9.
+  timeout: 180
+  time_budget: 1800
+  overloaded_quarantine: 120
+  models:
+    - provider: gemini
+      model: gemini-3.8-flash
+    - provider: gemini
+      model: gemini-3.7-flash
+    - provider: gemini
+      model: gemini-3.6-flash
+    - provider: gemini
+      model: gemini-3.5-flash
+    - provider: gemini
+      model: gemini-3.5-flash-lite
+  generate:
+    start: gemini-3.7-flash
+    temperature: 0.1
+  critique:
+    rank: not_below_generate
+    allow_weaker: false
+    temperature: 0