aireview 0.3.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +46 -0
- data/README.md +277 -28
- data/config/defaults.yml +47 -0
- data/lib/aireview/candidate_checker.rb +24 -22
- data/lib/aireview/cli.rb +46 -12
- data/lib/aireview/config.rb +65 -180
- data/lib/aireview/config_fallbacks.rb +79 -68
- data/lib/aireview/config_layers.rb +110 -0
- data/lib/aireview/config_limits.rb +10 -35
- data/lib/aireview/config_loader.rb +240 -0
- data/lib/aireview/context_budget.rb +22 -20
- data/lib/aireview/context_builder.rb +18 -17
- data/lib/aireview/diff_fetcher.rb +12 -11
- data/lib/aireview/dry_run_report.rb +33 -8
- data/lib/aireview/errors.rb +8 -0
- data/lib/aireview/gitlab_client.rb +10 -9
- data/lib/aireview/llm_client.rb +113 -0
- data/lib/aireview/llm_failure.rb +36 -19
- data/lib/aireview/llm_router.rb +315 -158
- data/lib/aireview/model_candidate.rb +28 -0
- data/lib/aireview/model_checker.rb +148 -0
- data/lib/aireview/model_pool.rb +224 -0
- data/lib/aireview/model_state.rb +82 -0
- data/lib/aireview/publisher.rb +6 -6
- data/lib/aireview/result_parser.rb +98 -0
- data/lib/aireview/review_marker.rb +16 -28
- data/lib/aireview/review_pipeline.rb +88 -86
- data/lib/aireview/review_renderer.rb +19 -11
- data/lib/aireview/reviewer.rb +37 -109
- data/lib/aireview/stage_chains.rb +113 -0
- data/lib/aireview/stages.rb +7 -0
- data/lib/aireview/utils.rb +29 -0
- data/lib/aireview/version.rb +1 -1
- data/lib/aireview.rb +1 -0
- metadata +19 -3
- data/lib/aireview/result_validation.rb +0 -65
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3f326ae1b11f59f7c447e8f128ab94845ef1543b0b377753f053fd4f69e03bad
|
|
4
|
+
data.tar.gz: c5908173ca8020a0563b6d4ee1fd4d8ce4ebe609fda7ded26d5b7eb40c84dda1
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2de9b7dd7e6caaa440287339d03eed843dfdb73010ae8cf6a01dea4e9348e3b072fc4fb61a55bf30eb4ed324e857ff883e35e31b711bc542cb8d1495645041bc
|
|
7
|
+
data.tar.gz: 5f0867f71415c019c03d99bd2215fc7f7ef7cadf98ab8b0db91e6eb98af54cfae3787b848c7611fc371cfa19cc94acd55e170047f094d8543f5c77f258e3313d
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,51 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.0.0
|
|
4
|
+
|
|
5
|
+
Breaking: the review key of a shared pool includes the pool order and the
|
|
6
|
+
critique policy; a single chain without reserves gets three requests instead
|
|
7
|
+
of a seventeen-minute schedule; `LLM_PROVIDER` no longer re-targets models
|
|
8
|
+
that carry an explicit provider.
|
|
9
|
+
|
|
10
|
+
- Defaults shipped in the image: `config/defaults.yml` is a configuration
|
|
11
|
+
layer between the built-in values and the project's `.aireview.yml`,
|
|
12
|
+
enabled by `AIREVIEW_DEFAULTS` (set in the Dockerfile). A project needs no
|
|
13
|
+
model names; `--dry-run` prints the layer every setting came from and
|
|
14
|
+
warns when inherited reserves silently change provider.
|
|
15
|
+
- Shared model pool: `llm.models` (`LLM_MODELS`) in order of priority,
|
|
16
|
+
`generate.start`, `critique.rank: not_below_generate|any` and
|
|
17
|
+
`critique.allow_weaker`. Generate walks the pool round from its start;
|
|
18
|
+
Critique takes the first live model not below the one that answered in
|
|
19
|
+
Generate, never weaker unless allowed. A stage with a `model` of its own
|
|
20
|
+
keeps an independent chain.
|
|
21
|
+
- Router: an overloaded model or a timeout gets one short retry, then a
|
|
22
|
+
quarantine (`llm.overloaded_quarantine`, `LLM_OVERLOADED_QUARANTINE`,
|
|
23
|
+
120 s) and the next model; the walk goes round the pool once every model
|
|
24
|
+
has been tried, waiting for the nearest quarantine to end. Three requests
|
|
25
|
+
per model per stage, counted across keys and including the JSON repair;
|
|
26
|
+
the last model has no special schedule. A model the provider does not
|
|
27
|
+
have (retired, a typo, not pulled into Ollama) is excluded for the run
|
|
28
|
+
instead of failing it; a bare 404 stays fatal.
|
|
29
|
+
- An invalid result that the same model cannot repair excludes that model
|
|
30
|
+
for the stage and restarts the stage on the next model with the original
|
|
31
|
+
request; Critique keeps the candidates already obtained.
|
|
32
|
+
- `aireview models check`: two probe requests per model with the production
|
|
33
|
+
schemas and validation, statuses `ok / missing / invalid / unverified /
|
|
34
|
+
failed / skipped`, `--strict` turns a skipped (unreachable) Ollama into a
|
|
35
|
+
failure. Meant for release pipelines and schedules.
|
|
36
|
+
- `templates/review.gitlab-ci.yml`: a job template projects include with
|
|
37
|
+
`ref: stable`; every config variable is passed into the container by name.
|
|
38
|
+
- The log names the model that answered each stage and its place in the
|
|
39
|
+
chain; the report notes a critique that ran on a weaker model.
|
|
40
|
+
- Fixed: with `json` 3.0 every provider answer failed to parse
|
|
41
|
+
(`Faraday::ParsingError`, "wrong number of arguments") — `faraday` 2.14.4
|
|
42
|
+
in the lockfile. The suite did not catch it because it stubs the network;
|
|
43
|
+
`aireview models check` did.
|
|
44
|
+
- Internals: routing plans (`StageChains`, `ModelPool`) and the router's
|
|
45
|
+
`ModelState` are objects, `ConfigLoader` owns loading and env parsing,
|
|
46
|
+
`ResultParser` parses LLM results, `LlmClient` makes one request while
|
|
47
|
+
the router retries. Specs run against the real RubyLLM classes.
|
|
48
|
+
|
|
3
49
|
## 0.3.0
|
|
4
50
|
|
|
5
51
|
- Fallback models per stage (`llm.<stage>.fallbacks`,
|
data/README.md
CHANGED
|
@@ -49,8 +49,10 @@ installed, replace it with plain `aireview`.
|
|
|
49
49
|
|
|
50
50
|
## Configuration
|
|
51
51
|
|
|
52
|
-
Secrets live in environment variables or in a local `.env` file.
|
|
53
|
-
|
|
52
|
+
Secrets live in environment variables or in a local `.env` file. The models of
|
|
53
|
+
both stages come from the defaults shipped in the Docker image (see "Defaults
|
|
54
|
+
shipped in the image"); outside the image they are set in `.env` or
|
|
55
|
+
`.aireview.yml`:
|
|
54
56
|
|
|
55
57
|
```bash
|
|
56
58
|
GITLAB_URL=https://gitlab.company.com
|
|
@@ -91,6 +93,88 @@ If only the LLM traffic has to go through a proxy, set `LLM_HTTP_PROXY` or
|
|
|
91
93
|
`llm.http_proxy`. That configures RubyLLM only; requests to GitLab and Jira
|
|
92
94
|
keep going directly.
|
|
93
95
|
|
|
96
|
+
### Defaults shipped in the image
|
|
97
|
+
|
|
98
|
+
Models, timeouts and the report language are baked into the image as
|
|
99
|
+
`config/defaults.yml`; inside the image `AIREVIEW_DEFAULTS` points at it. A
|
|
100
|
+
project in an organization has nothing to configure: no `.aireview.yml`, no
|
|
101
|
+
variables with model names. Changing the model for every project is one
|
|
102
|
+
edit of that file and a new image release. Outside the image the variable is
|
|
103
|
+
unset and the layer does not exist, so local runs are unaffected; point
|
|
104
|
+
`AIREVIEW_DEFAULTS` at any file to get the same layer elsewhere. The
|
|
105
|
+
defaults define a shared pool of five Gemini models (see "Shared model
|
|
106
|
+
pool"): Generate starts from a mid-range model and goes round the pool,
|
|
107
|
+
Critique takes the strongest live model not below the one Generate answered
|
|
108
|
+
with; every model of the pool is confirmed by `aireview models check` on
|
|
109
|
+
release.
|
|
110
|
+
|
|
111
|
+
The configuration layers, weakest first:
|
|
112
|
+
|
|
113
|
+
1. built-in values (`Config::DEFAULTS`);
|
|
114
|
+
2. image defaults — the file from `AIREVIEW_DEFAULTS`; no variable, no layer;
|
|
115
|
+
3. the project's `.aireview.yml`;
|
|
116
|
+
4. environment variables (`LLM_GENERATE_MODEL` and the rest);
|
|
117
|
+
5. CLI flags (`--generate-model`, `--critique-model`, the temperatures).
|
|
118
|
+
|
|
119
|
+
Layers merge by key, so a project overrides only what it needs — with two
|
|
120
|
+
caveats about the pool:
|
|
121
|
+
|
|
122
|
+
- a `llm.generate.model` of its own (in YAML or `LLM_GENERATE_MODEL`) **takes
|
|
123
|
+
the stage out of the pool**: its chain is that model plus its own
|
|
124
|
+
`fallbacks`, if any; the image models are not picked up for that stage and
|
|
125
|
+
the critique rank rule does not apply. To change only the starting model
|
|
126
|
+
while staying in the pool, set `llm.generate.start` / `LLM_GENERATE_START`
|
|
127
|
+
— or replace the whole pool through `llm.models` / `LLM_MODELS`;
|
|
128
|
+
- every pool model in the image carries an explicit provider (`gemini`), so
|
|
129
|
+
`LLM_PROVIDER=ollama` on its own does not move the Gemini pool to Ollama:
|
|
130
|
+
it only changes the default provider of models without one. A project on
|
|
131
|
+
Ollama sets its own models — its own pool (`LLM_MODELS=ollama/qwen2.5-coder:7b,…`)
|
|
132
|
+
or per-stage `model`s.
|
|
133
|
+
|
|
134
|
+
The shared settings `llm.temperature`, `llm.max_prompt_chars` and the default
|
|
135
|
+
provider `llm.provider` (env: `LLM_TEMPERATURE`, `LLM_MAX_PROMPT_CHARS`,
|
|
136
|
+
`LLM_PROVIDER`) apply to the stages layer by layer: a stage value from the
|
|
137
|
+
image defaults does not beat a shared value from the project or the
|
|
138
|
+
environment, while within one layer the stage value still wins. For stages
|
|
139
|
+
on their own chains the `fallbacks` array is replaced as a whole when given
|
|
140
|
+
explicitly; `fallbacks: []` removes the reserves; a reserve without a
|
|
141
|
+
`provider` of its own inherits the stage provider — if a project overrode the
|
|
142
|
+
stage provider (`LLM_GENERATE_PROVIDER=ollama`) while inherited reserves
|
|
143
|
+
without an explicit provider remain in the chain, they silently become
|
|
144
|
+
"models" of the new provider. `--dry-run` warns about that and prints, for
|
|
145
|
+
every model, the layer it came from:
|
|
146
|
+
|
|
147
|
+
```
|
|
148
|
+
=== LLM SETTINGS ===
|
|
149
|
+
Config: image defaults /app/config/defaults.yml, .aireview.yml /app/.aireview.yml
|
|
150
|
+
Generate: gemini-3.7-flash temperature=0.1 (model from image defaults, provider from image defaults)
|
|
151
|
+
fallbacks: gemini/gemini-3.6-flash -> ... (fallbacks from image defaults)
|
|
152
|
+
Critique: qwen2.5-coder:7b temperature=0 (model from .aireview.yml, provider from .aireview.yml)
|
|
153
|
+
Time budget: 1800s, overloaded quarantine: 120s
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
In pool mode the model source is the layer that set the stage `start` or the
|
|
157
|
+
`llm.models` list itself; `--critique-model` with a pool model shows up as
|
|
158
|
+
`model from cli`.
|
|
159
|
+
|
|
160
|
+
An `AIREVIEW_DEFAULTS` that points at a missing file is a configuration
|
|
161
|
+
error: such an image was built wrong, and failing at once beats reviewing
|
|
162
|
+
with the wrong models. Keys and tokens never live in `config/defaults.yml`.
|
|
163
|
+
|
|
164
|
+
Changing the default models or the pool order changes the review key (see "A
|
|
165
|
+
single comment per merge request"). What happens to open MRs depends on
|
|
166
|
+
`review_mode`: in `update` the review re-runs on the next push to the MR, in
|
|
167
|
+
`once` (as in the CI template) only on a job Retry. In `update` this is a
|
|
168
|
+
one-off burst of requests to the provider — better not to ship such a
|
|
169
|
+
release at peak hours.
|
|
170
|
+
|
|
171
|
+
No model is probed before a review: there are no test requests to five
|
|
172
|
+
models per MR, availability is learned from the working requests — the first
|
|
173
|
+
Generate request is the probe, and what the router learns it remembers until
|
|
174
|
+
the end of the run (see "Fallback models and keys"). The separate smoke test
|
|
175
|
+
of every model with both schemas (`aireview models check`) runs on an image
|
|
176
|
+
release and on a schedule, not on MRs.
|
|
177
|
+
|
|
94
178
|
### Local Ollama
|
|
95
179
|
|
|
96
180
|
Install Ollama following the
|
|
@@ -169,15 +253,16 @@ address with `/v1` matches the
|
|
|
169
253
|
local model it can be raised, but not without limit: a hung request holds
|
|
170
254
|
the job for exactly that long while a fallback model sits idle. An
|
|
171
255
|
"overloaded" (503) answer from the provider or a timeout does not fail the
|
|
172
|
-
run right away:
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
whole run.
|
|
177
|
-
|
|
178
|
-
Project rules live in `.aireview.yml`.
|
|
179
|
-
`critique.model` settings are
|
|
180
|
-
|
|
256
|
+
run right away: the model gets one short retry (~30 s), then goes into
|
|
257
|
+
quarantine for two minutes while the request goes to the next model;
|
|
258
|
+
without a fallback model the router waits for the quarantine to end and
|
|
259
|
+
tries once more. Three requests per model per stage in total (see "Fallback
|
|
260
|
+
models and keys"). Only the failed stage is repeated, not the whole run.
|
|
261
|
+
|
|
262
|
+
Project rules live in `.aireview.yml`. The `generate.model` and
|
|
263
|
+
`critique.model` settings are not inherited from the base `llm` settings: a
|
|
264
|
+
review does not start when a stage has no model in the image defaults, the
|
|
265
|
+
YAML or the environment. `llm.provider` is used as the default when
|
|
181
266
|
`generate.provider` or `critique.provider` is not set:
|
|
182
267
|
|
|
183
268
|
```yaml
|
|
@@ -300,31 +385,119 @@ fallback model and a provider can have a fallback key:
|
|
|
300
385
|
forbid circumventing its limits regardless of the project's billing. Keys
|
|
301
386
|
live only in the environment, never in `.aireview.yml`.
|
|
302
387
|
|
|
388
|
+
#### Shared model pool
|
|
389
|
+
|
|
390
|
+
Instead of two independent chains a single pool can be configured:
|
|
391
|
+
`llm.models` in order of priority (the first is the preferred one for
|
|
392
|
+
Critique). Generate goes through the pool from `generate.start` downwards
|
|
393
|
+
and round again; Critique takes the first live model **not below the one
|
|
394
|
+
that actually answered in Generate** (`critique.rank: not_below_generate`,
|
|
395
|
+
the default), with self-critique by the same model as the last permitted
|
|
396
|
+
option. "Above" and "below" are positions in the list, they are not derived
|
|
397
|
+
from model names:
|
|
398
|
+
|
|
399
|
+
```yaml
|
|
400
|
+
llm:
|
|
401
|
+
provider: gemini
|
|
402
|
+
models: # order = priority for Critique
|
|
403
|
+
- gemini-3.8-flash
|
|
404
|
+
- gemini-3.7-flash
|
|
405
|
+
- gemini-3.6-flash
|
|
406
|
+
- provider: ollama
|
|
407
|
+
model: qwen2.5-coder:7b
|
|
408
|
+
max_prompt_chars: 20000
|
|
409
|
+
generate:
|
|
410
|
+
start: gemini-3.7-flash # Generate starts here, then downwards and round
|
|
411
|
+
critique:
|
|
412
|
+
rank: not_below_generate # or any — the chains are independent
|
|
413
|
+
allow_weaker: false # true — go below when nothing above is alive
|
|
414
|
+
```
|
|
415
|
+
|
|
416
|
+
Environment equivalents: `LLM_MODELS=gemini/gemini-3.8-flash,gemini/gemini-3.7-flash,…`,
|
|
417
|
+
`LLM_GENERATE_START`, `LLM_CRITIQUE_START`, `LLM_CRITIQUE_RANK`,
|
|
418
|
+
`LLM_CRITIQUE_ALLOW_WEAKER`. The rules:
|
|
419
|
+
|
|
420
|
+
- no permitted live model for Critique and `allow_weaker: false` — the run
|
|
421
|
+
fails, candidates are not published without a critique; with
|
|
422
|
+
`allow_weaker: true` Critique goes below Generate and the report gets a
|
|
423
|
+
"Critique ran on a model weaker than Generate" line;
|
|
424
|
+
- a stage with a `model` of its own (in YAML, the environment or
|
|
425
|
+
`--generate-model` with a model outside the pool) does not use the pool:
|
|
426
|
+
its chain is independent and the rank rule does not apply, as with
|
|
427
|
+
`rank: any`; `--generate-model` / `--critique-model` with a model **from**
|
|
428
|
+
the pool bring the stage back into the pool and make that model the start
|
|
429
|
+
(the stage's own `model` and `fallbacks` are reset), with a model outside
|
|
430
|
+
the pool — a single chain without reserves;
|
|
431
|
+
- a project that replaces the whole pool (`llm.models` in YAML or
|
|
432
|
+
`LLM_MODELS`) need not repeat `start`: a start inherited from the image
|
|
433
|
+
that the new pool does not contain is replaced by the first model of the
|
|
434
|
+
new pool with a warning in the log. A start set in the same layer as the
|
|
435
|
+
pool or above it must be in the pool — a typo there stays a configuration
|
|
436
|
+
error;
|
|
437
|
+
- `critique.start` goes first only when the rank permits it; a start below
|
|
438
|
+
the model that answered in Generate is skipped with a warning when
|
|
439
|
+
`allow_weaker` is off — it does not bypass the ban on a weaker critique;
|
|
440
|
+
- the review key (see "A single comment per merge request") includes the
|
|
441
|
+
whole pool together with the critique policy: its order decides which
|
|
442
|
+
model checks the findings, not just what to fall back to. Reordering the
|
|
443
|
+
pool, changing `start`, `rank` or `allow_weaker` — a new key, open MRs are
|
|
444
|
+
re-reviewed on the next push; moving from per-stage chains to the pool —
|
|
445
|
+
the same, once;
|
|
446
|
+
- `--dry-run` prints the chains of both stages and a `Critique rule: …` line.
|
|
447
|
+
|
|
303
448
|
What happens on which error:
|
|
304
449
|
|
|
305
450
|
| Error | Reaction |
|
|
306
451
|
|---|---|
|
|
307
|
-
| Daily quota (`quotaId` like `…PerDay…` in Google's answer) | No retries: the next key on the same model. The "key + model" pair is remembered until the end of the run so that Critique and the JSON repair do not hit it again. Out of keys: the
|
|
308
|
-
| Per-minute limit (429 with a "retry in N s" hint) |
|
|
309
|
-
| Overloaded (503) or timeout | One short retry (~30 s)
|
|
310
|
-
|
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
452
|
+
| Daily quota (`quotaId` like `…PerDay…` in Google's answer) | No retries: the next key on the same model. The "key + model" pair is remembered until the end of the run so that Critique and the JSON repair do not hit it again. Out of keys: the model is excluded until the end of the run, the request goes to the next one. |
|
|
453
|
+
| Per-minute limit (429 with a "retry in N s" hint) | One retry after the hinted delay, then the next key. Out of keys: the model is quarantined for the hinted time, the request goes to the next one. |
|
|
454
|
+
| Overloaded (503) or timeout | One short retry (~30 s), then a quarantine of `llm.overloaded_quarantine` / `LLM_OVERLOADED_QUARANTINE` (120 s by default) and the next model with the same key. |
|
|
455
|
+
| The provider has no such model (the answer text names the model: retired, a typo, not pulled into Ollama) | No retries: the model is excluded until the end of the run, the request goes to the next one. A bare 404 without such text is fatal: a wrong `LLM_API_BASE` answers the same, and the next model will not help. |
|
|
456
|
+
| An invalid result (not JSON, not the schema, foreign ids in the verdicts) — and still invalid after one repair by the same model | The model is excluded for this stage, the stage starts over on the next model with the original request. For Critique the candidates already obtained are kept, Generate is not repeated. |
|
|
457
|
+
| Other API errors (context length, auth) | The run fails right away: a fallback model would answer the same. |
|
|
458
|
+
|
|
459
|
+
The chain is walked round: once every model has been tried, the router
|
|
460
|
+
returns to those whose quarantine has expired; when all are quarantined, it
|
|
461
|
+
waits for the nearest release. The last model in the list is nothing
|
|
462
|
+
special, nobody gets a long retry schedule. Two limits keep the walk
|
|
463
|
+
finite:
|
|
464
|
+
|
|
465
|
+
- **three requests per model per stage** (`MAX_ATTEMPTS_PER_MODEL`). Every
|
|
466
|
+
request sent counts regardless of its outcome, including the short retry
|
|
467
|
+
and the JSON repair; the keys of one model share one counter, a quarantine
|
|
468
|
+
does not reset it. Out of requests — the model leaves the stage; a JSON
|
|
469
|
+
repair without requests left is not sent, and the result counts as invalid
|
|
470
|
+
(see the table). When every model of the stage is out or excluded, the
|
|
471
|
+
stage ends at once, quarantines are not waited for;
|
|
472
|
+
- **the time budget** of the LLM part of the run: `llm.time_budget` /
|
|
473
|
+
`LLM_TIME_BUDGET`, 1800 seconds by default. A pause or a quarantine wait
|
|
474
|
+
that does not fit into the remainder is skipped, the request timeout is
|
|
475
|
+
capped by the remainder; when the time is up, the run fails with an error
|
|
476
|
+
listing everything that was tried. Because of this keep `LLM_TIMEOUT`
|
|
477
|
+
around 120–300 seconds: one hung request must not eat the whole budget.
|
|
478
|
+
|
|
479
|
+
**A single chain** — one model without reserves, typically Ollama on its own
|
|
480
|
+
box — behaves differently from earlier versions: instead of seventeen minutes
|
|
481
|
+
of waiting (retries after 2, 5, 5 and 5 minutes) the model gets three
|
|
482
|
+
requests — the original, a short retry after ~30 s and one more after the
|
|
483
|
+
two-minute quarantine; after the third failure the stage fails. A timeout is
|
|
484
|
+
a failure too, and a quarantine will not help a slow model: it only
|
|
485
|
+
lengthens the pause between attempts. A slow local model needs an
|
|
486
|
+
`LLM_TIMEOUT` with room for the largest request and an `LLM_TIME_BUDGET`
|
|
487
|
+
that fits three such requests with their pauses; `LLM_OVERLOADED_QUARANTINE`
|
|
488
|
+
is about a model that is temporarily overloaded and should come back.
|
|
318
489
|
|
|
319
490
|
The review key (see "A single comment per merge request") is computed from
|
|
320
491
|
the configured primary model, not from the one that answered: a review made
|
|
321
492
|
by a fallback model is not rewritten on the next push without changes in the
|
|
322
|
-
MR, and adding a fallback model to
|
|
323
|
-
every open MR. When a stage went to a fallback model, the report
|
|
324
|
-
`Fallback model used: critique — …` line
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
key
|
|
493
|
+
MR, and adding a fallback model to a per-stage chain does not re-run the
|
|
494
|
+
review on every open MR. When a stage went to a fallback model, the report
|
|
495
|
+
ends with a `Fallback model used: critique — …` line; the log names, for
|
|
496
|
+
every stage, the model that answered and its place in the chain
|
|
497
|
+
(`model=gemini/gemini-3.6-flash (2/5)`). A key switch stays in the log only,
|
|
498
|
+
and key values never reach the log. `--dry-run` prints the model chains, the
|
|
499
|
+
number of keys per provider, the time budget and the quarantine length;
|
|
500
|
+
`--no-fallbacks` leaves one model and one key per stage.
|
|
328
501
|
|
|
329
502
|
### Checking that findings point at the diff
|
|
330
503
|
|
|
@@ -373,6 +546,42 @@ bundle _2.3.26_ exec bin/aireview review https://gitlab.company.com/team/project
|
|
|
373
546
|
- `--review-mode MODE` sets the behaviour when a review has already been published: `update` or `once`.
|
|
374
547
|
- `--force` reviews again even when a review for this state of the MR is already published.
|
|
375
548
|
|
|
549
|
+
### Checking the models
|
|
550
|
+
|
|
551
|
+
```bash
|
|
552
|
+
bundle _2.3.26_ exec bin/aireview models check
|
|
553
|
+
bundle _2.3.26_ exec bin/aireview models check --config .aireview.yml --strict --verbose
|
|
554
|
+
```
|
|
555
|
+
|
|
556
|
+
Every model of both stage chains gets two small requests — one with the
|
|
557
|
+
production Generate schema and one with the Critique schema — on a tiny
|
|
558
|
+
synthetic MR, with the same system prompts as a review. The answer goes
|
|
559
|
+
through the same validation as in a run (JSON shape, candidate and verdict
|
|
560
|
+
ids). The provider's catalog is not consulted: "the model is listed" does
|
|
561
|
+
not mean "our request with the schema passes on it". There are no reserves,
|
|
562
|
+
retries or quarantine here — this is a check, not a review; only a
|
|
563
|
+
per-minute limit gets one retry after the provider's hint.
|
|
564
|
+
|
|
565
|
+
```
|
|
566
|
+
Checking 5 model(s) with the generate and critique schemas
|
|
567
|
+
gemini/gemini-3.7-flash generate ok (4.6s)
|
|
568
|
+
gemini/gemini-3.7-flash critique ok (6.9s)
|
|
569
|
+
gemini/gemini-3.6-flash generate unverified: This model is currently experiencing high demand. …
|
|
570
|
+
gemini/gemini-9.9-nope generate missing: models/gemini-9.9-nope is not found for API version v1beta, …
|
|
571
|
+
ollama/qwen2.5-coder:7b generate skipped: Connection refused
|
|
572
|
+
Result: 6 ok, 1 unverified, 2 missing, 2 skipped -> FAILED
|
|
573
|
+
```
|
|
574
|
+
|
|
575
|
+
Statuses: `ok` — an answer matching the schema; `missing` — the provider
|
|
576
|
+
has no such model (retired, a typo, not pulled into Ollama); `invalid` — it
|
|
577
|
+
answered, but not by the schema; `unverified` — the provider could not
|
|
578
|
+
answer right now (overload, quota, timeout), the model is not at fault but
|
|
579
|
+
not confirmed either; `failed` — other API errors; `skipped` — Ollama is
|
|
580
|
+
unreachable where the check runs (a CI runner has none). The exit code is 0
|
|
581
|
+
only when every model is `ok` or `skipped`; with `--strict` — only `ok`: on
|
|
582
|
+
a box where Ollama must be running, an unreachable Ollama is a failed check,
|
|
583
|
+
not a skip.
|
|
584
|
+
|
|
376
585
|
### A single comment per merge request
|
|
377
586
|
|
|
378
587
|
With `--post` the review is not published as a new note every time; it updates
|
|
@@ -473,6 +682,46 @@ local HTTP proxy (`wireproxy`, for instance) before `aireview` starts and point
|
|
|
473
682
|
GitLab and Jira stay on direct connections while RubyLLM goes through the
|
|
474
683
|
tunnel.
|
|
475
684
|
|
|
685
|
+
### Including the job template
|
|
686
|
+
|
|
687
|
+
Instead of copying the job into every project, it lives in this repository
|
|
688
|
+
as `templates/review.gitlab-ci.yml`. A project's `.gitlab-ci.yml` keeps only:
|
|
689
|
+
|
|
690
|
+
```yaml
|
|
691
|
+
include:
|
|
692
|
+
- project: your-group/aireview
|
|
693
|
+
ref: stable
|
|
694
|
+
file: /templates/review.gitlab-ci.yml
|
|
695
|
+
```
|
|
696
|
+
|
|
697
|
+
The template is written for shell-executor runners (the image runs through
|
|
698
|
+
`docker run` with secrets passed by name) and carries `[skip review]`,
|
|
699
|
+
`resource_group`, `allow_failure` and `REVIEW_MODE=once`. Models come from
|
|
700
|
+
the image defaults, `.aireview.yml` is mounted into the container only when
|
|
701
|
+
the project has one. The job runs in the `.post` stage — it exists in every
|
|
702
|
+
pipeline, so a project does not declare `stages`; to move the review to
|
|
703
|
+
another stage, add `aireview: {stage: review}` to the project file. Every
|
|
704
|
+
variable the config reads (`Config.env_names`: models, providers, reserves,
|
|
705
|
+
temperatures, limits, `OLLAMA_API_BASE` and so on) is passed into the
|
|
706
|
+
container by name, so a project can override anything through its CI/CD
|
|
707
|
+
variables — for instance, swap an unavailable model with `LLM_CRITIQUE_MODEL`
|
|
708
|
+
without waiting for an image release.
|
|
709
|
+
|
|
710
|
+
Secrets stay CI/CD variables **of the project**: group variables are readable
|
|
711
|
+
by any merge request of any project in the group.
|
|
712
|
+
|
|
713
|
+
The `stable` branch is managed: the copy of the template there pins
|
|
714
|
+
`AIREVIEW_IMAGE_TAG` to a verified release. Updating or rolling back every
|
|
715
|
+
project that includes the template is one edit of `stable`. A project can
|
|
716
|
+
pin another version by overriding `AIREVIEW_IMAGE_TAG` in its `variables`.
|
|
717
|
+
|
|
718
|
+
A release of the image is: tag → `aireview models check` on the image
|
|
719
|
+
defaults (an API error or an answer off the schema for any model stops the
|
|
720
|
+
pipeline, the image is not built, the previous version stays in the
|
|
721
|
+
registry) → build and push → set the new tag in `stable`. The same check on
|
|
722
|
+
a schedule (once a day) is the early signal that the provider retired a
|
|
723
|
+
model — otherwise the first to learn about it is a live MR.
|
|
724
|
+
|
|
476
725
|
## Docker
|
|
477
726
|
|
|
478
727
|
```bash
|
data/config/defaults.yml
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Defaults baked into the aireview image. This layer sits between the
|
|
2
|
+
# built-in values and the project's .aireview.yml: a project overrides
|
|
3
|
+
# anything here, the environment and CLI flags override the project. No
|
|
4
|
+
# keys live here.
|
|
5
|
+
#
|
|
6
|
+
# One shared pool of models (see README, "Shared model pool"): the order is
|
|
7
|
+
# the priority for Critique, the first model is the preferred one. Generate
|
|
8
|
+
# starts at generate.start and goes down the list and round again; Critique
|
|
9
|
+
# takes the first live model not below the one that answered in Generate
|
|
10
|
+
# and never goes lower.
|
|
11
|
+
#
|
|
12
|
+
# Every model carries an explicit provider: a model without one inherits
|
|
13
|
+
# llm.provider, and a project that switched it to ollama would otherwise get
|
|
14
|
+
# "Ollama models" with Gemini names.
|
|
15
|
+
#
|
|
16
|
+
# Model names come from the Gemini catalog (GET /v1beta/models), text models
|
|
17
|
+
# without "preview" in the name only; each one is confirmed by
|
|
18
|
+
# `aireview models check` with both schemas (2026-09-18).
|
|
19
|
+
|
|
20
|
+
review_language: en
|
|
21
|
+
|
|
22
|
+
llm:
|
|
23
|
+
provider: gemini
|
|
24
|
+
temperature: 0
|
|
25
|
+
# Three attempts per model × timeout is the ceiling for one hung model:
|
|
26
|
+
# 300 s meant 15 minutes, 180 s means 9.
|
|
27
|
+
timeout: 180
|
|
28
|
+
time_budget: 1800
|
|
29
|
+
overloaded_quarantine: 120
|
|
30
|
+
models:
|
|
31
|
+
- provider: gemini
|
|
32
|
+
model: gemini-3.8-flash
|
|
33
|
+
- provider: gemini
|
|
34
|
+
model: gemini-3.7-flash
|
|
35
|
+
- provider: gemini
|
|
36
|
+
model: gemini-3.6-flash
|
|
37
|
+
- provider: gemini
|
|
38
|
+
model: gemini-3.5-flash
|
|
39
|
+
- provider: gemini
|
|
40
|
+
model: gemini-3.5-flash-lite
|
|
41
|
+
generate:
|
|
42
|
+
start: gemini-3.7-flash
|
|
43
|
+
temperature: 0.1
|
|
44
|
+
critique:
|
|
45
|
+
rank: not_below_generate
|
|
46
|
+
allow_weaker: false
|
|
47
|
+
temperature: 0
|
|
@@ -3,11 +3,12 @@ require 'logger'
|
|
|
3
3
|
require 'set'
|
|
4
4
|
|
|
5
5
|
module Aireview
|
|
6
|
-
#
|
|
7
|
-
#
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
#
|
|
6
|
+
# A mechanical check that a candidate points at the code, against the diff
|
|
7
|
+
# the model actually saw: file, line, quote. The anchoring is checked, not
|
|
8
|
+
# the bug itself: a quote that is not found is a reason for Critique to
|
|
9
|
+
# look closer, not proof of a fabrication. A file that is not among the MR
|
|
10
|
+
# changes is a different matter: there is nothing to check such a
|
|
11
|
+
# candidate against, it is dropped.
|
|
11
12
|
class CandidateChecker
|
|
12
13
|
HUNK_HEADER = /^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/
|
|
13
14
|
FILE_HEADER = %r{^diff --git a/(.+?) b/(.+)$}
|
|
@@ -15,12 +16,12 @@ module Aireview
|
|
|
15
16
|
NOTE_LINE_RESET = 'line was outside the shown hunks and has been reset to null'
|
|
16
17
|
NOTE_NOT_VERIFIED = 'file shown partially or without a diff, location not verified'
|
|
17
18
|
|
|
18
|
-
#
|
|
19
|
-
#
|
|
20
|
-
#
|
|
21
|
-
#
|
|
22
|
-
#
|
|
23
|
-
#
|
|
19
|
+
# One file of the context: the new-file line ranges of the shown hunks
|
|
20
|
+
# and the normalized text of every hunk for the quote search — the new
|
|
21
|
+
# side (context + added lines) and the old side (context + removed
|
|
22
|
+
# lines) separately, so that a multi-line quote from one version of the
|
|
23
|
+
# file is found whole. Hunks are not glued together: there is skipped
|
|
24
|
+
# code between them, and a quote across a hunk boundary is not a quote.
|
|
24
25
|
Section = Struct.new(:ranges, :hunks, :partial, keyword_init: true) do
|
|
25
26
|
def include?(quote)
|
|
26
27
|
hunks.any? { |hunk| hunk[:new_text].include?(quote) || hunk[:old_text].include?(quote) }
|
|
@@ -34,9 +35,9 @@ module Aireview
|
|
|
34
35
|
@sections = parse_sections(diff_text.to_s)
|
|
35
36
|
end
|
|
36
37
|
|
|
37
|
-
#
|
|
38
|
-
#
|
|
39
|
-
#
|
|
38
|
+
# Returns the candidates with marks: a note for Critique, quote_missing
|
|
39
|
+
# for the report, line reset to null when it was not confirmed.
|
|
40
|
+
# Candidates with a file outside the MR are dropped.
|
|
40
41
|
def check(candidates)
|
|
41
42
|
candidates.filter_map do |candidate|
|
|
42
43
|
id = value(candidate, 'id')
|
|
@@ -85,10 +86,10 @@ module Aireview
|
|
|
85
86
|
NOTE_QUOTE_NOT_FOUND
|
|
86
87
|
end
|
|
87
88
|
|
|
88
|
-
#
|
|
89
|
-
#
|
|
90
|
-
#
|
|
91
|
-
#
|
|
89
|
+
# The diff is already packed to the budget: a partially shown file may
|
|
90
|
+
# lack hunks, a file without a diff has none at all. The --- / +++
|
|
91
|
+
# headers occur only between the file header and the first @@; inside a
|
|
92
|
+
# hunk a "+++ x" line is the added code "++ x".
|
|
92
93
|
def parse_sections(diff_text)
|
|
93
94
|
sections = {}
|
|
94
95
|
section = nil
|
|
@@ -108,8 +109,8 @@ module Aireview
|
|
|
108
109
|
normalize_hunks(sections)
|
|
109
110
|
end
|
|
110
111
|
|
|
111
|
-
#
|
|
112
|
-
#
|
|
112
|
+
# A hunk without new lines (a deletion, @@ -1 +0,0 @@) gives no range:
|
|
113
|
+
# there is no line 0 in the new file.
|
|
113
114
|
def start_hunk(section, header)
|
|
114
115
|
start = header[1].to_i
|
|
115
116
|
length = header[2] ? header[2].to_i : 1
|
|
@@ -142,8 +143,9 @@ module Aireview
|
|
|
142
143
|
partial.each { |path| sections[path]&.partial = true }
|
|
143
144
|
end
|
|
144
145
|
|
|
145
|
-
#
|
|
146
|
-
#
|
|
146
|
+
# An exact match first: a/ and b/ directories can be real. The prefix
|
|
147
|
+
# from the diff header is stripped only when the exact path is not in
|
|
148
|
+
# the MR.
|
|
147
149
|
def resolve_path(path)
|
|
148
150
|
path = path.to_s.strip
|
|
149
151
|
return path if @mr_paths.include?(path)
|