django-explain-errors 0.11.1__tar.gz → 0.12.0__tar.gz

{django_explain_errors-0.11.1/django_explain_errors.egg-info → django_explain_errors-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.11.1
+Version: 0.12.0
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -68,8 +68,8 @@ Explains Django exceptions on the debug page, pointing at the real fix in your o
 </p>
 
 *The traceback held only Django and SQLite internals. With RAG enabled, the explanation names the
-failing function (`clone_latest_post`) and gives the one-line fix. Real `gpt-4o-mini` output from
-the eval harness.*
+failing function (`clone_latest_post`) and gives the one-line fix. Real output from the eval
+harness, generated with `gpt-4o-mini` (the default before `gpt-6-luna`).*
 
 - **Strongly recommended:** enable RAG over your project, so explanations name the actual file and function instead of guessing ([measured in the eval harness](#does-rag-actually-help)). It is off by default only because [setup](#codebase-aware-explanations-rag) takes three steps.
 - Works with OpenAI, Claude, or any OpenAI-compatible endpoint, including local models via Ollama
@@ -196,7 +196,7 @@ python manage.py check --deploy --fail-level WARNING
    In the terminal, the explanation is printed with the model that produced it:
 
    ```
-   Error explanation (gpt-4o-mini):
+   Error explanation (gpt-6-luna):
     The view raised a ValueError because ...
    ```
 
@@ -282,7 +282,7 @@ and keeps using that for later calls. Other providers keep receiving `max_tokens
 | ------------------ | -------- | ----------- |
 | `OPENAI_API_KEY` (env or settings) | Yes, unless `OPENAI_BASE_URL` points at an endpoint that does not authenticate | API key used to authenticate with the configured endpoint. Read first from the environment, then from `settings`. |
 | `DEBUG` | Yes | The middleware is only active when `DEBUG=True`. When `False`, requests pass through untouched. |
-| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
+| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-6-luna` (was `gpt-4o-mini`). If you set `OPENAI_MODEL` yourself, the default change does not affect you, but OpenAI retires `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, and `gpt-4.1-mini` on 2027-01-14, after which calls to them fail. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for model API calls. Defaults to `10`. |
 | `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
@@ -327,7 +327,7 @@ OPENAI_API_KEY = "your_anthropic_api_key_here"
 OPENAI_MODEL = "..."  # see Anthropic's current model list below
 ```
 
-`OPENAI_MODEL` is mandatory here: the default (`gpt-4o-mini`) doesn't exist on Anthropic's API
+`OPENAI_MODEL` is mandatory here: the default (`gpt-6-luna`) doesn't exist on Anthropic's API
 and will 404. Use one of the model names from
 [Anthropic's model overview](https://platform.claude.com/docs/en/about-claude/models/overview);
 model names are versioned and retired over time, so check that page rather than relying on a
@@ -571,9 +571,8 @@ generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TO
 explicitly always overrides this scaling, at any value, including one lower than the
 unscaled 1,000 default. The multiplier itself is deliberately generous rather than
 precise: the underlying tokens-per-word figures are estimates, not direct
-measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
-which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
-these estimates lean on.
+measurements, and tokenizers differ across models and providers, so the
+multiplier is not calibrated to any one of them.
 
 ## Codebase-aware explanations (RAG)
 
@@ -796,7 +795,7 @@ Most of the gap comes from two checks, measured on the 10 fixtures whose cause i
 - **Points to the right file and function:** RAG-on 26, RAG-off 13
 - **Avoids invented details:** RAG-on 26, RAG-off 17
 
-Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+Without your source, `gpt-4o-mini` (the generator in these runs, the default before `gpt-6-luna`) tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
 
 **Limitations:**
 

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0}/README.md

@@ -16,8 +16,8 @@ Explains Django exceptions on the debug page, pointing at the real fix in your o
 </p>
 
 *The traceback held only Django and SQLite internals. With RAG enabled, the explanation names the
-failing function (`clone_latest_post`) and gives the one-line fix. Real `gpt-4o-mini` output from
-the eval harness.*
+failing function (`clone_latest_post`) and gives the one-line fix. Real output from the eval
+harness, generated with `gpt-4o-mini` (the default before `gpt-6-luna`).*
 
 - **Strongly recommended:** enable RAG over your project, so explanations name the actual file and function instead of guessing ([measured in the eval harness](#does-rag-actually-help)). It is off by default only because [setup](#codebase-aware-explanations-rag) takes three steps.
 - Works with OpenAI, Claude, or any OpenAI-compatible endpoint, including local models via Ollama
@@ -144,7 +144,7 @@ python manage.py check --deploy --fail-level WARNING
    In the terminal, the explanation is printed with the model that produced it:
 
    ```
-   Error explanation (gpt-4o-mini):
+   Error explanation (gpt-6-luna):
     The view raised a ValueError because ...
    ```
 
@@ -230,7 +230,7 @@ and keeps using that for later calls. Other providers keep receiving `max_tokens
 | ------------------ | -------- | ----------- |
 | `OPENAI_API_KEY` (env or settings) | Yes, unless `OPENAI_BASE_URL` points at an endpoint that does not authenticate | API key used to authenticate with the configured endpoint. Read first from the environment, then from `settings`. |
 | `DEBUG` | Yes | The middleware is only active when `DEBUG=True`. When `False`, requests pass through untouched. |
-| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
+| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-6-luna` (was `gpt-4o-mini`). If you set `OPENAI_MODEL` yourself, the default change does not affect you, but OpenAI retires `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, and `gpt-4.1-mini` on 2027-01-14, after which calls to them fail. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for model API calls. Defaults to `10`. |
 | `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
@@ -275,7 +275,7 @@ OPENAI_API_KEY = "your_anthropic_api_key_here"
 OPENAI_MODEL = "..."  # see Anthropic's current model list below
 ```
 
-`OPENAI_MODEL` is mandatory here: the default (`gpt-4o-mini`) doesn't exist on Anthropic's API
+`OPENAI_MODEL` is mandatory here: the default (`gpt-6-luna`) doesn't exist on Anthropic's API
 and will 404. Use one of the model names from
 [Anthropic's model overview](https://platform.claude.com/docs/en/about-claude/models/overview);
 model names are versioned and retired over time, so check that page rather than relying on a
@@ -519,9 +519,8 @@ generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TO
 explicitly always overrides this scaling, at any value, including one lower than the
 unscaled 1,000 default. The multiplier itself is deliberately generous rather than
 precise: the underlying tokens-per-word figures are estimates, not direct
-measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
-which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
-these estimates lean on.
+measurements, and tokenizers differ across models and providers, so the
+multiplier is not calibrated to any one of them.
 
 ## Codebase-aware explanations (RAG)
 
@@ -744,7 +743,7 @@ Most of the gap comes from two checks, measured on the 10 fixtures whose cause i
 - **Points to the right file and function:** RAG-on 26, RAG-off 13
 - **Avoids invented details:** RAG-on 26, RAG-off 17
 
-Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+Without your source, `gpt-4o-mini` (the generator in these runs, the default before `gpt-6-luna`) tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
 
 **Limitations:**
 

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0/django_explain_errors.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.11.1
+Version: 0.12.0
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -68,8 +68,8 @@ Explains Django exceptions on the debug page, pointing at the real fix in your o
 </p>
 
 *The traceback held only Django and SQLite internals. With RAG enabled, the explanation names the
-failing function (`clone_latest_post`) and gives the one-line fix. Real `gpt-4o-mini` output from
-the eval harness.*
+failing function (`clone_latest_post`) and gives the one-line fix. Real output from the eval
+harness, generated with `gpt-4o-mini` (the default before `gpt-6-luna`).*
 
 - **Strongly recommended:** enable RAG over your project, so explanations name the actual file and function instead of guessing ([measured in the eval harness](#does-rag-actually-help)). It is off by default only because [setup](#codebase-aware-explanations-rag) takes three steps.
 - Works with OpenAI, Claude, or any OpenAI-compatible endpoint, including local models via Ollama
@@ -196,7 +196,7 @@ python manage.py check --deploy --fail-level WARNING
    In the terminal, the explanation is printed with the model that produced it:
 
    ```
-   Error explanation (gpt-4o-mini):
+   Error explanation (gpt-6-luna):
     The view raised a ValueError because ...
    ```
 
@@ -282,7 +282,7 @@ and keeps using that for later calls. Other providers keep receiving `max_tokens
 | ------------------ | -------- | ----------- |
 | `OPENAI_API_KEY` (env or settings) | Yes, unless `OPENAI_BASE_URL` points at an endpoint that does not authenticate | API key used to authenticate with the configured endpoint. Read first from the environment, then from `settings`. |
 | `DEBUG` | Yes | The middleware is only active when `DEBUG=True`. When `False`, requests pass through untouched. |
-| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
+| `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-6-luna` (was `gpt-4o-mini`). If you set `OPENAI_MODEL` yourself, the default change does not affect you, but OpenAI retires `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, and `gpt-4.1-mini` on 2027-01-14, after which calls to them fail. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for model API calls. Defaults to `10`. |
 | `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
@@ -327,7 +327,7 @@ OPENAI_API_KEY = "your_anthropic_api_key_here"
 OPENAI_MODEL = "..."  # see Anthropic's current model list below
 ```
 
-`OPENAI_MODEL` is mandatory here: the default (`gpt-4o-mini`) doesn't exist on Anthropic's API
+`OPENAI_MODEL` is mandatory here: the default (`gpt-6-luna`) doesn't exist on Anthropic's API
 and will 404. Use one of the model names from
 [Anthropic's model overview](https://platform.claude.com/docs/en/about-claude/models/overview);
 model names are versioned and retired over time, so check that page rather than relying on a
@@ -571,9 +571,8 @@ generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TO
 explicitly always overrides this scaling, at any value, including one lower than the
 unscaled 1,000 default. The multiplier itself is deliberately generous rather than
 precise: the underlying tokens-per-word figures are estimates, not direct
-measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
-which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
-these estimates lean on.
+measurements, and tokenizers differ across models and providers, so the
+multiplier is not calibrated to any one of them.
 
 ## Codebase-aware explanations (RAG)
 
@@ -796,7 +795,7 @@ Most of the gap comes from two checks, measured on the 10 fixtures whose cause i
 - **Points to the right file and function:** RAG-on 26, RAG-off 13
 - **Avoids invented details:** RAG-on 26, RAG-off 17
 
-Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+Without your source, `gpt-4o-mini` (the generator in these runs, the default before `gpt-6-luna`) tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
 
 **Limitations:**
 

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0}/explain_errors/middleware.py

@@ -97,7 +97,7 @@ class ExplainErrorsMiddleware:
             self.throttle = SlidingWindowThrottle(max_calls, window_seconds)
 
             # Configurable via settings, with sensible defaults.
-            self.model = getattr(settings, "OPENAI_MODEL", "gpt-4o-mini")
+            self.model = getattr(settings, "OPENAI_MODEL", "gpt-6-luna")
             self.language = getattr(settings, "EXPLAIN_ERRORS_LANGUAGE", None)
 
             configured_max_tokens = getattr(settings, "OPENAI_MAX_TOKENS", None)

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0}/setup.py

@@ -10,7 +10,7 @@ with open('README.md', encoding='utf-8') as f:
 
 setup(
     name='django-explain-errors',
-    version='0.11.1',
+    version='0.12.0',
     packages=find_packages(exclude=['tests', 'tests.*', 'evals', 'evals.*']),
     description='Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.',
     long_description_content_type='text/markdown',

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0}/tests/test_debug_page.py

@@ -165,7 +165,7 @@ class DebugPageRequestCycleTest(SimpleTestCase):
         content = response.content.decode()
         self.assertIn('id="explain-errors"', content)
         self.assertIn("Mocked explanation.", content)
-        mock_print.assert_any_call("Error explanation (gpt-4o-mini):\n", "Mocked explanation.")
+        mock_print.assert_any_call("Error explanation (gpt-6-luna):\n", "Mocked explanation.")
 
     @override_settings(EXPLAIN_ERRORS_INJECT_DEBUG_PAGE=False)
     def test_inject_debug_page_off_no_banner(self):

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0}/tests/test_eval_run.py

@@ -17,8 +17,13 @@ from unittest.mock import patch
 
 from evals.fixtures import FIXTURES_BY_NAME
 from evals.run import (
+    _capture_call,
     estimate_cost_usd,
+    is_generation_failure,
     judge_all,
+    parse_generation_error,
+    run_fixture,
+    tally_generation,
     tally_judgments,
     tally_latency,
     tally_retries,
@@ -422,6 +427,258 @@ class JudgeAllConcurrencyTest(unittest.TestCase):
         self.assertLessEqual(active["peak"], 4)
 
 
+def _skipped(group, sides):
+    return {
+        "group": group,
+        "judge_failure": False,
+        "judgment_skipped": True,
+        "generation_failed_sides": sides,
+        "rag_on_is_a": True,
+        "retries": 0,
+    }
+
+
+class _FakeResponse:
+    def __init__(self, status_code, body):
+        self.status_code = status_code
+        self._body = body
+
+    def json(self):
+        if isinstance(self._body, ValueError):
+            raise self._body
+        return self._body
+
+
+class GenerationFailureRecordingTest(unittest.TestCase):
+    """run_fixture flags calls with no usable explanation."""
+
+    def _run(self, response, generation_error=None):
+        fixture = FIXTURES_BY_NAME["none_attribute"]
+        captured = {
+            "exc_type": AttributeError,
+            "exception": None,
+            "traceback": "tb",
+            "prompt_tokens": None,
+            "completion_tokens": None,
+            "generation_error": generation_error,
+        }
+        with patch("evals.run._capture_call", return_value=(response, captured)):
+            call, _ = run_fixture(object(), fixture, False, 0)
+        return call
+
+    def test_null_explanation_is_a_failure(self):
+        call = self._run(
+            _FakeResponse(500, {"message": None}), generation_error="the model returned an empty response"
+        )
+        self.assertTrue(call["generation_failed"])
+        self.assertEqual(call["generation_error"], "the model returned an empty response")
+        self.assertIsNone(call["explanation"])
+
+    def test_whitespace_only_explanation_is_a_failure(self):
+        call = self._run(_FakeResponse(500, {"message": "  \n\t "}))
+        self.assertTrue(call["generation_failed"])
+        self.assertIsNone(call["generation_error"])
+
+    def test_empty_string_and_non_json_bodies_are_failures(self):
+        self.assertTrue(self._run(_FakeResponse(500, {"message": ""}))["generation_failed"])
+        self.assertTrue(self._run(_FakeResponse(500, ValueError("bad")))["generation_failed"])
+
+    def test_real_explanation_is_not_a_failure(self):
+        call = self._run(_FakeResponse(500, {"message": "Because None has no attribute."}))
+        self.assertFalse(call["generation_failed"])
+        self.assertIsNone(call["generation_error"])
+
+
+class GenerationErrorCaptureTest(unittest.TestCase):
+    def test_parses_reason_from_middleware_line(self):
+        text = "noise\nFailed to get an explanation from the model API: Error code: 404 - model not found\n"
+        self.assertEqual(
+            parse_generation_error(text), "Error code: 404 - model not found"
+        )
+
+    def test_no_failure_line_is_none(self):
+        self.assertIsNone(parse_generation_error("Error explanation (m):\n fine\n"))
+        self.assertIsNone(parse_generation_error(""))
+
+    def test_capture_call_records_reason_and_still_prints(self):
+        import contextlib
+        import io
+
+        outer = io.StringIO()
+
+        def fn():
+            print("Failed to get an explanation from the model API:", ValueError("boom"))
+            return "ok"
+
+        with contextlib.redirect_stdout(outer):
+            result, captured = _capture_call(fn)
+        self.assertEqual(result, "ok")
+        self.assertEqual(captured["generation_error"], "boom")
+        self.assertIn("boom", outer.getvalue())
+
+    def test_capture_call_without_failure_leaves_error_null(self):
+        _, captured = _capture_call(lambda: None)
+        self.assertIsNone(captured["generation_error"])
+
+
+class IsGenerationFailureTest(unittest.TestCase):
+    def test_flag_wins_when_present(self):
+        self.assertTrue(is_generation_failure({"generation_failed": True, "explanation": "x"}))
+        self.assertFalse(is_generation_failure({"generation_failed": False, "explanation": None}))
+
+    def test_older_records_without_the_flag_fall_back_to_the_explanation(self):
+        self.assertTrue(is_generation_failure({"explanation": None}))
+        self.assertTrue(is_generation_failure({"explanation": "   "}))
+        self.assertFalse(is_generation_failure({"explanation": "real"}))
+
+
+class JudgeSkipTest(unittest.TestCase):
+    """judge_all never calls the judge for a pair with a failed side."""
+
+    def _judge(self, off_explanation, on_explanation, **extra):
+        fixture = FIXTURES_BY_NAME["none_attribute"]
+
+        def call(rag, explanation):
+            record = {
+                "fixture": fixture.name,
+                "rag_enabled": rag,
+                "run_index": 0,
+                "traceback": "tb",
+                "explanation": explanation,
+            }
+            record.update(extra.get("on" if rag else "off", {}))
+            return record
+
+        calls = [call(False, off_explanation), call(True, on_explanation)]
+        with patch("evals.judge.get_judge_client", return_value=object()), patch(
+            "evals.judge.get_judge_model", return_value="m"
+        ), patch("evals.judge.compare", return_value={"fixture": fixture.name}) as compare, patch(
+            "evals.run._build_judge_source", return_value=[]
+        ):
+            judgments = judge_all([fixture], calls, {}, seed=1)
+        return judgments, compare
+
+    def test_neither_fails_judges_as_before(self):
+        judgments, compare = self._judge("off text", "on text")
+        compare.assert_called_once()
+        self.assertEqual(len(judgments), 1)
+        self.assertNotIn("judgment_skipped", judgments[0])
+
+    def test_rag_on_failure_skips_the_judge(self):
+        judgments, compare = self._judge("off text", None)
+        compare.assert_not_called()
+        self.assertTrue(judgments[0]["judgment_skipped"])
+        self.assertEqual(judgments[0]["generation_failed_sides"], ["rag_on"])
+        self.assertFalse(judgments[0]["judge_failure"])
+
+    def test_rag_off_failure_skips_the_judge(self):
+        judgments, compare = self._judge("", "on text")
+        compare.assert_not_called()
+        self.assertEqual(judgments[0]["generation_failed_sides"], ["rag_off"])
+
+    def test_both_failing_skips_and_names_both_sides(self):
+        judgments, compare = self._judge(None, None)
+        compare.assert_not_called()
+        self.assertEqual(
+            sorted(judgments[0]["generation_failed_sides"]), ["rag_off", "rag_on"]
+        )
+
+    def test_whitespace_only_explanation_skips_the_judge(self):
+        judgments, compare = self._judge("  \n ", "on text")
+        compare.assert_not_called()
+        self.assertEqual(judgments[0]["generation_failed_sides"], ["rag_off"])
+
+    def test_recorded_flag_skips_even_with_text_present(self):
+        judgments, compare = self._judge("off", "on", on={"generation_failed": True})
+        compare.assert_not_called()
+        self.assertEqual(judgments[0]["generation_failed_sides"], ["rag_on"])
+
+    def test_skipped_record_keeps_identifying_fields(self):
+        judgments, _ = self._judge(None, "on text")
+        j = judgments[0]
+        self.assertEqual((j["fixture"], j["group"], j["run_index"]), ("none_attribute", FIXTURES_BY_NAME["none_attribute"].group, 0))
+
+
+class SkippedPairTallyTest(unittest.TestCase):
+    def test_skipped_pairs_count_toward_no_group(self):
+        judgments = [
+            _judgment("A", "rag_on", rag_on_is_a=True),
+            _skipped("A", ["rag_on"]),
+            _skipped("B", ["rag_on", "rag_off"]),
+        ]
+        result = tally_judgments(judgments)
+        self.assertEqual(result["skipped_pairs"], 2)
+        self.assertEqual(result["judge_failures"], 0)
+        self.assertEqual(result["wins_by_group"]["A"], {"rag_on": 1, "rag_off": 0, "tie": 0})
+        self.assertEqual(result["wins_by_group"]["B"], {"rag_on": 0, "rag_off": 0, "tie": 0})
+        self.assertEqual(
+            sum(result["question_yes_counts_by_group"]["B"]["rag_on"].values()), 0
+        )
+
+    def test_no_skips_reports_zero(self):
+        self.assertEqual(tally_judgments([_judgment("A", "tie")])["skipped_pairs"], 0)
+
+    def test_tally_generation_counts_failures(self):
+        calls = [
+            {"explanation": "ok", "generation_failed": False},
+            {"explanation": None, "generation_failed": True},
+            {"explanation": "  "},  # older record, no flag
+        ]
+        self.assertEqual(tally_generation(calls), {"failures": 2, "calls": 3})
+
+    def test_summary_prints_generation_lines_and_excludes_skips_from_incomplete(self):
+        import contextlib
+        import io
+
+        from evals.run import print_summary
+
+        calls = [
+            {"explanation": None, "generation_failed": True, "rag_enabled": False,
+             "latency_seconds": 1.0, "prompt_tokens": None, "completion_tokens": None},
+            {"explanation": None, "generation_failed": True, "rag_enabled": True,
+             "latency_seconds": 1.0, "prompt_tokens": None, "completion_tokens": None},
+        ]
+        # Skipped pairs stay out of the judge-failure ratio: 1 judged, 0 failed.
+        judgments = [_judgment("A", "tie"), _skipped("A", ["rag_on", "rag_off"])]
+        out = io.StringIO()
+        with contextlib.redirect_stdout(out):
+            print_summary([], calls, judgments, "g", "j")
+        text = out.getvalue()
+        self.assertIn("Generation failures: 2 of 2 calls", text)
+        self.assertIn("Skipped pairs (generation failure, not judged): 1", text)
+        self.assertNotIn("RUN INCOMPLETE", text)
+
+    def test_all_pairs_skipped_prints_run_incomplete_without_dividing_by_zero(self):
+        import contextlib
+        import io
+
+        from evals.run import print_summary
+
+        calls = [
+            {"explanation": None, "generation_failed": True, "rag_enabled": flag,
+             "latency_seconds": 1.0, "prompt_tokens": None, "completion_tokens": None}
+            for flag in (False, True)
+        ]
+        judgments = [_skipped("A", ["rag_on", "rag_off"]), _skipped("B", ["rag_on"])]
+        out = io.StringIO()
+        with contextlib.redirect_stdout(out):
+            print_summary([], calls, judgments, "g", "j")
+        text = out.getvalue()
+        self.assertIn("RUN INCOMPLETE: all 2 pair(s) were skipped", text)
+        self.assertNotIn("Wins by group", text)
+
+    def test_no_judgments_at_all_is_not_flagged_incomplete(self):
+        import contextlib
+        import io
+
+        from evals.run import print_summary
+
+        out = io.StringIO()
+        with contextlib.redirect_stdout(out):
+            print_summary([], [], [], "g", "j")
+        self.assertNotIn("RUN INCOMPLETE", out.getvalue())
+
+
 class TallyRetriesTest(unittest.TestCase):
 
     def test_sums_per_judgment_retries_and_tolerates_missing(self):
@@ -446,10 +703,10 @@ class FixtureSettingsModelTest(unittest.TestCase):
         self.assertEqual(self._model(OPENAI_MODEL="some-other-model"), "some-other-model")
 
     def test_unset_falls_back_to_default(self):
-        self.assertEqual(self._model(), "gpt-4o-mini")
+        self.assertEqual(self._model(), "gpt-6-luna")
 
     def test_empty_string_falls_back_to_default(self):
-        self.assertEqual(self._model(OPENAI_MODEL=""), "gpt-4o-mini")
+        self.assertEqual(self._model(OPENAI_MODEL=""), "gpt-6-luna")
 
 
 class EvalHarnessEndToEndTest(unittest.TestCase):
@@ -511,7 +768,7 @@ class EvalHarnessEndToEndTest(unittest.TestCase):
         self.assertGreaterEqual(data["timing"]["generator_seconds"], 0)
         self.assertGreaterEqual(data["timing"]["judge_seconds"], 0)
 
-        self.assertEqual(data["generator_model"], "gpt-4o-mini")
+        self.assertEqual(data["generator_model"], "gpt-6-luna")
         self.assertEqual(data["judge_model"], "judge-test-model")
         self.assertEqual(len(data["calls"]), 30)  # 15 fixtures x 2 RAG states x 1 run
         self.assertEqual(len(data["judgments"]), 15)
@@ -530,10 +787,14 @@ class EvalHarnessEndToEndTest(unittest.TestCase):
             "prompt_tokens",
             "completion_tokens",
             "status_code",
+            "generation_failed",
+            "generation_error",
         }
         for call in data["calls"]:
             self.assertTrue(call_keys.issubset(call.keys()), call)
             self.assertEqual(call["status_code"], 500)
+            self.assertFalse(call["generation_failed"])
+            self.assertIsNone(call["generation_error"])
             self.assertEqual(call["explanation"], "Mocked generator explanation.")
             self.assertEqual(call["prompt_tokens"], 10)
             self.assertEqual(call["completion_tokens"], 5)

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0}/tests/test_middleware.py

@@ -268,7 +268,7 @@ class ExplainErrorsMiddlewarePreserveDebugPageTest(SimpleTestCase):
         with patch("builtins.print") as mock_print:
             with self.assertRaises(ValueError):
                 mw(self.factory.get("/"))
-        mock_print.assert_any_call("Error explanation (gpt-4o-mini):\n", "Mocked explanation.")
+        mock_print.assert_any_call("Error explanation (gpt-6-luna):\n", "Mocked explanation.")
 
     @override_settings(EXPLAIN_ERRORS_PRESERVE_DEBUG_PAGE=False)
     def test_preserve_flag_explicitly_off_returns_json_500(self):
@@ -321,7 +321,7 @@ class OpenAICallConfigTest(SimpleTestCase):
         mw = ExplainErrorsMiddleware(lambda r: None)
         mw.process_exception(self.factory.get("/"), Exception("x"))
         kwargs = self.client.chat.completions.create.call_args.kwargs
-        self.assertEqual(kwargs["model"], "gpt-4o-mini")
+        self.assertEqual(kwargs["model"], "gpt-6-luna")
         self.assertEqual(kwargs["max_tokens"], DEFAULT_MAX_TOKENS)
 
     @override_settings(OPENAI_MODEL="gpt-5-mini", OPENAI_MAX_TOKENS=50)

{django_explain_errors-0.11.1 → django_explain_errors-0.12.0}/tests/test_print_stdout.py

@@ -33,7 +33,7 @@ class PrintStdoutRequestCycleTest(SimpleTestCase):
             response = client.get("/boom/")
 
         self.assertEqual(response.status_code, 500)
-        mock_print.assert_any_call("Error explanation (gpt-4o-mini):\n", "Mocked explanation.")
+        mock_print.assert_any_call("Error explanation (gpt-6-luna):\n", "Mocked explanation.")
 
     @override_settings(OPENAI_MODEL="some-model")
     def test_configured_model_appears_in_stdout_prefix(self):
@@ -52,7 +52,7 @@ class PrintStdoutRequestCycleTest(SimpleTestCase):
 
         self.assertEqual(response.status_code, 500)
         for call in mock_print.call_args_list:
-            self.assertNotEqual(call.args[0], "Error explanation (gpt-4o-mini):\n")
+            self.assertNotEqual(call.args[0], "Error explanation (gpt-6-luna):\n")
         self.assertIn('id="explain-errors"', response.content.decode())
 
     @override_settings(EXPLAIN_ERRORS_PRINT_STDOUT=False)
@@ -83,7 +83,7 @@ class PrintStdoutRequestCycleTest(SimpleTestCase):
 
         self.assertEqual(response.status_code, 500)
         for call in mock_print.call_args_list:
-            self.assertNotEqual(call.args[0], "Error explanation (gpt-4o-mini):\n")
+            self.assertNotEqual(call.args[0], "Error explanation (gpt-6-luna):\n")
         self.assertIn('id="explain-errors"', response.content.decode())