django-explain-errors 0.11.0__tar.gz → 0.12.0__tar.gz

{django_explain_errors-0.11.0/django_explain_errors.egg-info → django_explain_errors-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.11.0
+Version: 0.12.0
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -68,8 +68,8 @@ Explains Django exceptions on the debug page, pointing at the real fix in your o
 </p>
 
 *The traceback held only Django and SQLite internals. With RAG enabled, the explanation names the
-failing function (`clone_latest_post`) and gives the one-line fix. Real `gpt-4o-mini` output from
-the eval harness.*
+failing function (`clone_latest_post`) and gives the one-line fix. Real output from the eval
+harness, generated with `gpt-4o-mini` (the default before `gpt-6-luna`).*
 
 - **Strongly recommended:** enable RAG over your project, so explanations name the actual file and function instead of guessing ([measured in the eval harness](#does-rag-actually-help)). It is off by default only because [setup](#codebase-aware-explanations-rag) takes three steps.
 - Works with OpenAI, Claude, or any OpenAI-compatible endpoint, including local models via Ollama
@@ -196,7 +196,7 @@ python manage.py check --deploy --fail-level WARNING
    In the terminal, the explanation is printed with the model that produced it:
 
    ```
-   Error explanation (gpt-4o-mini):
+   Error explanation (gpt-6-luna):
     The view raised a ValueError because ...
    ```
 
@@ -272,13 +272,17 @@ catches the exception internally, so it never becomes an unhandled exception, bu
 than relying on `got_request_exception`. That event is unrelated to whatever error the
 developer is actually investigating.
 
+OpenAI reasoning models are supported. They reject `max_tokens`, so when the API rejects it
+the middleware retries once with `max_completion_tokens` (same value, `OPENAI_MAX_TOKENS`)
+and keeps using that for later calls. Other providers keep receiving `max_tokens`.
+
 ## Configuration
 
 | Setting / variable | Required | Description |
 | ------------------ | -------- | ----------- |
 | `OPENAI_API_KEY` (env or settings) | Yes, unless `OPENAI_BASE_URL` points at an endpoint that does not authenticate | API key used to authenticate with the configured endpoint. Read first from the environment, then from `settings`. |
 | `DEBUG` | Yes | The middleware is only active when `DEBUG=True`. When `False`, requests pass through untouched. |
-| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
+| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-6-luna` (was `gpt-4o-mini`). If you set `OPENAI_MODEL` yourself, the default change does not affect you, but OpenAI retires `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, and `gpt-4.1-mini` on 2027-01-14, after which calls to them fail. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for model API calls. Defaults to `10`. |
 | `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
@@ -323,7 +327,7 @@ OPENAI_API_KEY = "your_anthropic_api_key_here"
 OPENAI_MODEL = "..."  # see Anthropic's current model list below
 ```
 
-`OPENAI_MODEL` is mandatory here: the default (`gpt-4o-mini`) doesn't exist on Anthropic's API
+`OPENAI_MODEL` is mandatory here: the default (`gpt-6-luna`) doesn't exist on Anthropic's API
 and will 404. Use one of the model names from
 [Anthropic's model overview](https://platform.claude.com/docs/en/about-claude/models/overview);
 model names are versioned and retired over time, so check that page rather than relying on a
@@ -567,9 +571,8 @@ generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TO
 explicitly always overrides this scaling, at any value, including one lower than the
 unscaled 1,000 default. The multiplier itself is deliberately generous rather than
 precise: the underlying tokens-per-word figures are estimates, not direct
-measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
-which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
-these estimates lean on.
+measurements, and tokenizers differ across models and providers, so the
+multiplier is not calibrated to any one of them.
 
 ## Codebase-aware explanations (RAG)
 
@@ -792,7 +795,7 @@ Most of the gap comes from two checks, measured on the 10 fixtures whose cause i
 - **Points to the right file and function:** RAG-on 26, RAG-off 13
 - **Avoids invented details:** RAG-on 26, RAG-off 17
 
-Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+Without your source, `gpt-4o-mini` (the generator in these runs, the default before `gpt-6-luna`) tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
 
 **Limitations:**
 

{django_explain_errors-0.11.0 → django_explain_errors-0.12.0}/README.md

@@ -16,8 +16,8 @@ Explains Django exceptions on the debug page, pointing at the real fix in your o
 </p>
 
 *The traceback held only Django and SQLite internals. With RAG enabled, the explanation names the
-failing function (`clone_latest_post`) and gives the one-line fix. Real `gpt-4o-mini` output from
-the eval harness.*
+failing function (`clone_latest_post`) and gives the one-line fix. Real output from the eval
+harness, generated with `gpt-4o-mini` (the default before `gpt-6-luna`).*
 
 - **Strongly recommended:** enable RAG over your project, so explanations name the actual file and function instead of guessing ([measured in the eval harness](#does-rag-actually-help)). It is off by default only because [setup](#codebase-aware-explanations-rag) takes three steps.
 - Works with OpenAI, Claude, or any OpenAI-compatible endpoint, including local models via Ollama
@@ -144,7 +144,7 @@ python manage.py check --deploy --fail-level WARNING
    In the terminal, the explanation is printed with the model that produced it:
 
    ```
-   Error explanation (gpt-4o-mini):
+   Error explanation (gpt-6-luna):
     The view raised a ValueError because ...
    ```
 
@@ -220,13 +220,17 @@ catches the exception internally, so it never becomes an unhandled exception, bu
 than relying on `got_request_exception`. That event is unrelated to whatever error the
 developer is actually investigating.
 
+OpenAI reasoning models are supported. They reject `max_tokens`, so when the API rejects it
+the middleware retries once with `max_completion_tokens` (same value, `OPENAI_MAX_TOKENS`)
+and keeps using that for later calls. Other providers keep receiving `max_tokens`.
+
 ## Configuration
 
 | Setting / variable | Required | Description |
 | ------------------ | -------- | ----------- |
 | `OPENAI_API_KEY` (env or settings) | Yes, unless `OPENAI_BASE_URL` points at an endpoint that does not authenticate | API key used to authenticate with the configured endpoint. Read first from the environment, then from `settings`. |
 | `DEBUG` | Yes | The middleware is only active when `DEBUG=True`. When `False`, requests pass through untouched. |
-| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
+| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-6-luna` (was `gpt-4o-mini`). If you set `OPENAI_MODEL` yourself, the default change does not affect you, but OpenAI retires `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, and `gpt-4.1-mini` on 2027-01-14, after which calls to them fail. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for model API calls. Defaults to `10`. |
 | `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
@@ -271,7 +275,7 @@ OPENAI_API_KEY = "your_anthropic_api_key_here"
 OPENAI_MODEL = "..."  # see Anthropic's current model list below
 ```
 
-`OPENAI_MODEL` is mandatory here: the default (`gpt-4o-mini`) doesn't exist on Anthropic's API
+`OPENAI_MODEL` is mandatory here: the default (`gpt-6-luna`) doesn't exist on Anthropic's API
 and will 404. Use one of the model names from
 [Anthropic's model overview](https://platform.claude.com/docs/en/about-claude/models/overview);
 model names are versioned and retired over time, so check that page rather than relying on a
@@ -515,9 +519,8 @@ generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TO
 explicitly always overrides this scaling, at any value, including one lower than the
 unscaled 1,000 default. The multiplier itself is deliberately generous rather than
 precise: the underlying tokens-per-word figures are estimates, not direct
-measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
-which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
-these estimates lean on.
+measurements, and tokenizers differ across models and providers, so the
+multiplier is not calibrated to any one of them.
 
 ## Codebase-aware explanations (RAG)
 
@@ -740,7 +743,7 @@ Most of the gap comes from two checks, measured on the 10 fixtures whose cause i
 - **Points to the right file and function:** RAG-on 26, RAG-off 13
 - **Avoids invented details:** RAG-on 26, RAG-off 17
 
-Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+Without your source, `gpt-4o-mini` (the generator in these runs, the default before `gpt-6-luna`) tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
 
 **Limitations:**
 

{django_explain_errors-0.11.0 → django_explain_errors-0.12.0/django_explain_errors.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.11.0
+Version: 0.12.0
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -68,8 +68,8 @@ Explains Django exceptions on the debug page, pointing at the real fix in your o
 </p>
 
 *The traceback held only Django and SQLite internals. With RAG enabled, the explanation names the
-failing function (`clone_latest_post`) and gives the one-line fix. Real `gpt-4o-mini` output from
-the eval harness.*
+failing function (`clone_latest_post`) and gives the one-line fix. Real output from the eval
+harness, generated with `gpt-4o-mini` (the default before `gpt-6-luna`).*
 
 - **Strongly recommended:** enable RAG over your project, so explanations name the actual file and function instead of guessing ([measured in the eval harness](#does-rag-actually-help)). It is off by default only because [setup](#codebase-aware-explanations-rag) takes three steps.
 - Works with OpenAI, Claude, or any OpenAI-compatible endpoint, including local models via Ollama
@@ -196,7 +196,7 @@ python manage.py check --deploy --fail-level WARNING
    In the terminal, the explanation is printed with the model that produced it:
 
    ```
-   Error explanation (gpt-4o-mini):
+   Error explanation (gpt-6-luna):
     The view raised a ValueError because ...
    ```
 
@@ -272,13 +272,17 @@ catches the exception internally, so it never becomes an unhandled exception, bu
 than relying on `got_request_exception`. That event is unrelated to whatever error the
 developer is actually investigating.
 
+OpenAI reasoning models are supported. They reject `max_tokens`, so when the API rejects it
+the middleware retries once with `max_completion_tokens` (same value, `OPENAI_MAX_TOKENS`)
+and keeps using that for later calls. Other providers keep receiving `max_tokens`.
+
 ## Configuration
 
 | Setting / variable | Required | Description |
 | ------------------ | -------- | ----------- |
 | `OPENAI_API_KEY` (env or settings) | Yes, unless `OPENAI_BASE_URL` points at an endpoint that does not authenticate | API key used to authenticate with the configured endpoint. Read first from the environment, then from `settings`. |
 | `DEBUG` | Yes | The middleware is only active when `DEBUG=True`. When `False`, requests pass through untouched. |
-| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
+| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-6-luna` (was `gpt-4o-mini`). If you set `OPENAI_MODEL` yourself, the default change does not affect you, but OpenAI retires `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, and `gpt-4.1-mini` on 2027-01-14, after which calls to them fail. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for model API calls. Defaults to `10`. |
 | `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
@@ -323,7 +327,7 @@ OPENAI_API_KEY = "your_anthropic_api_key_here"
 OPENAI_MODEL = "..."  # see Anthropic's current model list below
 ```
 
-`OPENAI_MODEL` is mandatory here: the default (`gpt-4o-mini`) doesn't exist on Anthropic's API
+`OPENAI_MODEL` is mandatory here: the default (`gpt-6-luna`) doesn't exist on Anthropic's API
 and will 404. Use one of the model names from
 [Anthropic's model overview](https://platform.claude.com/docs/en/about-claude/models/overview);
 model names are versioned and retired over time, so check that page rather than relying on a
@@ -567,9 +571,8 @@ generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TO
 explicitly always overrides this scaling, at any value, including one lower than the
 unscaled 1,000 default. The multiplier itself is deliberately generous rather than
 precise: the underlying tokens-per-word figures are estimates, not direct
-measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
-which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
-these estimates lean on.
+measurements, and tokenizers differ across models and providers, so the
+multiplier is not calibrated to any one of them.
 
 ## Codebase-aware explanations (RAG)
 
@@ -792,7 +795,7 @@ Most of the gap comes from two checks, measured on the 10 fixtures whose cause i
 - **Points to the right file and function:** RAG-on 26, RAG-off 13
 - **Avoids invented details:** RAG-on 26, RAG-off 17
 
-Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+Without your source, `gpt-4o-mini` (the generator in these runs, the default before `gpt-6-luna`) tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
 
 **Limitations:**
 

{django_explain_errors-0.11.0 → django_explain_errors-0.12.0}/django_explain_errors.egg-info/SOURCES.txt

@@ -38,5 +38,6 @@ tests/test_rag.py
 tests/test_sanitize.py
 tests/test_signals.py
 tests/test_throttle.py
+tests/test_token_param_fallback.py
 tests/test_tracebacks.py
 tests/test_truncation.py

{django_explain_errors-0.11.0 → django_explain_errors-0.12.0}/explain_errors/middleware.py

@@ -3,6 +3,7 @@ import logging
 import threading
 from collections import OrderedDict
 
+import openai
 from django.conf import settings
 from django.http import JsonResponse
 from asgiref.sync import markcoroutinefunction, sync_to_async
@@ -81,6 +82,9 @@ class ExplainErrorsMiddleware:
             # above this one runs in sync mode and receives a coroutine.
             markcoroutinefunction(self)
         self.openai_client = None
+        # Output-cap parameter sent to the model. Reasoning models reject
+        # max_tokens; _create_completion switches this once, per instance.
+        self._token_param = "max_tokens"
         self.throttle = None
         self.dedup_enabled = False
         self._explanation_cache = OrderedDict()
@@ -93,7 +97,7 @@ class ExplainErrorsMiddleware:
             self.throttle = SlidingWindowThrottle(max_calls, window_seconds)
 
             # Configurable via settings, with sensible defaults.
-            self.model = getattr(settings, "OPENAI_MODEL", "gpt-4o-mini")
+            self.model = getattr(settings, "OPENAI_MODEL", "gpt-6-luna")
             self.language = getattr(settings, "EXPLAIN_ERRORS_LANGUAGE", None)
 
             configured_max_tokens = getattr(settings, "OPENAI_MAX_TOKENS", None)
@@ -251,6 +255,39 @@ class ExplainErrorsMiddleware:
 
         return prompt
 
+    def _create_completion(self, messages):
+        """Send the chat completion, adapting to the model's output-cap parameter.
+
+        Sends max_tokens by default. If the API rejects exactly that parameter
+        (OpenAI reasoning models require max_completion_tokens), retries once
+        with max_completion_tokens and remembers the choice on this instance.
+        Any other error, or a failed retry, propagates to the caller.
+        """
+
+        def create(token_param):
+            return self.openai_client.chat.completions.create(
+                model=self.model,
+                messages=messages,
+                **{token_param: self.max_tokens},
+            )
+
+        try:
+            return create(self._token_param)
+        except openai.BadRequestError as exc:
+            if self._token_param != "max_tokens" or not (
+                getattr(exc, "param", None) == "max_tokens"
+                and getattr(exc, "code", None) == "unsupported_parameter"
+            ):
+                raise
+        response = create("max_completion_tokens")
+        self._token_param = "max_completion_tokens"
+        logger.info(
+            "explain_errors: model %s rejected max_tokens; using "
+            "max_completion_tokens instead",
+            self.model,
+        )
+        return response
+
     def _generate_explanation(self, prompt):
         """Call the model and return the explanation text, or None on failure.
 
@@ -260,13 +297,11 @@ class ExplainErrorsMiddleware:
         explanation = None
         try:
             # Call model API
-            response = self.openai_client.chat.completions.create(
-                model=self.model,
-                messages=[
+            response = self._create_completion(
+                [
                     {"role": "system", "content": self.system_prompt},
                     {"role": "user", "content": prompt},
-                ],
-                max_tokens=self.max_tokens,
+                ]
             )
             explanation = response.choices[0].message.content
             usage = getattr(response, "usage", None)
@@ -276,11 +311,23 @@ class ExplainErrorsMiddleware:
                 getattr(usage, "completion_tokens", None),
             )
             if response.choices[0].finish_reason == "length":
+                hint = ""
+                if self._token_param == "max_completion_tokens":
+                    hint = (
+                        " (reasoning tokens count against this limit; raise it "
+                        "if explanations come back short or empty)"
+                    )
                 logger.warning(
-                    "explain_errors: explanation truncated by OPENAI_MAX_TOKENS=%s",
+                    "explain_errors: explanation truncated by OPENAI_MAX_TOKENS=%s%s",
                     self.max_tokens,
+                    hint,
                 )
 
+            if not explanation or not explanation.strip():
+                # Reasoning can consume the whole budget and leave no text.
+                explanation = None
+                raise ValueError("the model returned an empty response")
+
             if getattr(settings, "EXPLAIN_ERRORS_PRINT_STDOUT", True):
                 print(f"Error explanation ({self.model}):\n", explanation)
         except Exception as e:

{django_explain_errors-0.11.0 → django_explain_errors-0.12.0}/setup.py

@@ -10,7 +10,7 @@ with open('README.md', encoding='utf-8') as f:
 
 setup(
     name='django-explain-errors',
-    version='0.11.0',
+    version='0.12.0',
     packages=find_packages(exclude=['tests', 'tests.*', 'evals', 'evals.*']),
     description='Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.',
     long_description_content_type='text/markdown',

{django_explain_errors-0.11.0 → django_explain_errors-0.12.0}/tests/test_debug_page.py

@@ -165,7 +165,7 @@ class DebugPageRequestCycleTest(SimpleTestCase):
         content = response.content.decode()
         self.assertIn('id="explain-errors"', content)
         self.assertIn("Mocked explanation.", content)
-        mock_print.assert_any_call("Error explanation (gpt-4o-mini):\n", "Mocked explanation.")
+        mock_print.assert_any_call("Error explanation (gpt-6-luna):\n", "Mocked explanation.")
 
     @override_settings(EXPLAIN_ERRORS_INJECT_DEBUG_PAGE=False)
     def test_inject_debug_page_off_no_banner(self):

{django_explain_errors-0.11.0 → django_explain_errors-0.12.0}/tests/test_eval_judge.py

@@ -391,3 +391,161 @@ class CompareClaimsGuardTest(unittest.TestCase):
         )
 
         self.assertFalse(result["judge_failure"])
+
+
+def _openai_error(cls, status=None):
+    """An instance of the openai exception class without constructing its
+    httpx request/response (which httpx package openai imports varies by
+    openai version).
+    """
+    exc = Exception.__new__(cls)
+    exc.args = ("boom",)
+    if status is not None:
+        exc.status_code = status
+    return exc
+
+
+def _flaky_client(errors, then=VALID_RESPONSE):
+    """Raises each of `errors` in turn, then returns `then`."""
+    client = MagicMock()
+    ok = MagicMock(
+        choices=[MagicMock(message=MagicMock(content=then))],
+        usage=MagicMock(prompt_tokens=7, completion_tokens=3),
+    )
+    client.chat.completions.create.side_effect = list(errors) + [ok]
+    return client
+
+
+class CompareRetryTest(unittest.TestCase):
+
+    def test_pre_drawn_position_is_used_and_rng_untouched(self):
+        fixture = _make_fixture()
+        client = _client_returning(VALID_RESPONSE)
+
+        class ExplodingRng:
+            def random(self):
+                raise AssertionError("rng must not be drawn when rag_on_is_a is given")
+
+        result = compare(
+            client, "m", "tb", fixture, "src", "off", "on",
+            rng=ExplodingRng(), rag_on_is_a=False,
+        )
+
+        self.assertFalse(result["rag_on_is_a"])
+
+    def test_rate_limit_is_retried_and_counted(self):
+        import openai
+
+        fixture = _make_fixture()
+        errors = [_openai_error(openai.RateLimitError, 429)] * 2
+        client = _flaky_client(errors)
+        sleeps = []
+
+        result = compare(
+            client, "m", "tb", fixture, "src", "off", "on",
+            rng=_FixedRng(0.1), sleep=sleeps.append,
+        )
+
+        self.assertFalse(result["judge_failure"])
+        self.assertEqual(result["retries"], 2)
+        self.assertEqual(len(sleeps), 2)
+        self.assertEqual(client.chat.completions.create.call_count, 3)
+        self.assertEqual(result["prompt_tokens"], 7)
+
+    def test_backoff_grows_and_is_capped(self):
+        import openai
+
+        from evals import judge
+
+        fixture = _make_fixture()
+        client = _flaky_client([_openai_error(openai.InternalServerError, 503)] * 4)
+        ceilings = []
+
+        class RecordingJitter:
+            def uniform(self, low, high):
+                ceilings.append(high)
+                return high
+
+        original = judge._jitter_rng
+        judge._jitter_rng = RecordingJitter()
+        self.addCleanup(setattr, judge, "_jitter_rng", original)
+
+        compare(
+            client, "m", "tb", fixture, "src", "off", "on",
+            rng=_FixedRng(0.1), max_attempts=5, sleep=lambda s: None,
+        )
+
+        self.assertEqual(ceilings, [1.0, 2.0, 4.0, 8.0])
+
+    def test_timeout_and_connection_errors_are_retried(self):
+        import openai
+
+        fixture = _make_fixture()
+        client = _flaky_client(
+            [_openai_error(openai.APITimeoutError), _openai_error(openai.APIConnectionError)]
+        )
+
+        result = compare(
+            client, "m", "tb", fixture, "src", "off", "on",
+            rng=_FixedRng(0.1), sleep=lambda s: None,
+        )
+
+        self.assertFalse(result["judge_failure"])
+        self.assertEqual(result["retries"], 2)
+
+    def test_exhausted_retries_is_a_judge_failure_with_retries_recorded(self):
+        import openai
+
+        fixture = _make_fixture()
+        client = MagicMock()
+        client.chat.completions.create.side_effect = _openai_error(openai.RateLimitError, 429)
+
+        result = compare(
+            client, "m", "tb", fixture, "src", "off", "on",
+            rng=_FixedRng(0.1), max_attempts=3, sleep=lambda s: None,
+        )
+
+        self.assertTrue(result["judge_failure"])
+        self.assertEqual(result["retries"], 2)
+        self.assertEqual(client.chat.completions.create.call_count, 3)
+        self.assertIn("boom", result["error"])
+        self.assertIn("rag_on_is_a", result)
+
+    def test_non_transient_error_is_not_retried(self):
+        import openai
+
+        fixture = _make_fixture()
+        client = MagicMock()
+        client.chat.completions.create.side_effect = _openai_error(
+            openai.AuthenticationError, 401
+        )
+
+        result = compare(
+            client, "m", "tb", fixture, "src", "off", "on",
+            rng=_FixedRng(0.1), sleep=lambda s: self.fail("must not sleep"),
+        )
+
+        self.assertTrue(result["judge_failure"])
+        self.assertEqual(result["retries"], 0)
+        self.assertEqual(client.chat.completions.create.call_count, 1)
+
+    def test_malformed_response_is_not_retried(self):
+        fixture = _make_fixture()
+        client = _client_returning("not json")
+
+        result = compare(
+            client, "m", "tb", fixture, "src", "off", "on",
+            rng=_FixedRng(0.1), sleep=lambda s: self.fail("must not sleep"),
+        )
+
+        self.assertTrue(result["judge_failure"])
+        self.assertEqual(result["retries"], 0)
+        self.assertEqual(client.chat.completions.create.call_count, 1)
+
+    def test_clean_call_records_zero_retries(self):
+        fixture = _make_fixture()
+        client = _client_returning(VALID_RESPONSE)
+
+        result = compare(client, "m", "tb", fixture, "src", "off", "on", rng=_FixedRng(0.1))
+
+        self.assertEqual(result["retries"], 0)