django-explain-errors 0.6.0__tar.gz → 0.7.0__tar.gz

{django_explain_errors-0.6.0/django_explain_errors.egg-info → django_explain_errors-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.6.0
+Version: 0.7.0
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -54,7 +54,8 @@ a local model if you prefer not to send code off your machine.
 
 It can optionally ground explanations in your own project source using a
 local vector index (RAG), so explanations reference the actual code that
-failed instead of staying generic.
+failed instead of staying generic (measured — see "Does RAG actually
+help?" below).
 
 The middleware supports both synchronous (WSGI) and asynchronous (ASGI)
 views. It auto-detects the view chain at startup and routes requests through
@@ -183,7 +184,7 @@ developer is actually investigating.
 | `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for the OpenAI client. Defaults to `10`. |
-| `OPENAI_MAX_TRACEBACK_CHARS` | No | Traceback is trimmed to its last N characters before being sent. Defaults to `3000`. |
+| `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
 | `EXPLAIN_ERRORS_PRESERVE_DEBUG_PAGE` | No | When `True` (the default), the middleware prints the explanation to stdout and returns `None`, so exception handling continues normally and Django renders its standard debug page. Set to `False` to instead return a JSON 500 response, which ends exception handling early (see Compatibility above). |
 | `OPENAI_BASE_URL` (env or settings) | No | Base URL for any OpenAI-compatible API (for example Ollama at `http://localhost:11434/v1`). When set, a missing API key is replaced with a placeholder since local servers do not require one. |
 | `EXPLAIN_ERRORS_MAX_CALLS` | No | Together with `EXPLAIN_ERRORS_WINDOW_SECONDS`, caps API spend to at most this many explanations within a rolling window; once the cap is hit, further errors in that window are not sent for explanation until an earlier call ages out. Defaults to `5`. |
@@ -256,6 +257,13 @@ read them in another language instead:
 EXPLAIN_ERRORS_LANGUAGE = "Spanish"  # or the code form, "es"
 ```
 
+There is no fixed list of supported languages. `EXPLAIN_ERRORS_LANGUAGE` accepts
+any language the configured model can write, because the setting adds one clause
+to the system prompt (see `LANGUAGE_CLAUSE_TEMPLATE` in
+`explain_errors/middleware.py`), not a translation catalog with its own
+maintained language list. How well it works varies by model; see the known
+limitation below.
+
 This is independent of Django's own `LANGUAGE_CODE`, which controls the language your
 site serves to its users, not the language you read explanations in. Regardless of
 `EXPLAIN_ERRORS_LANGUAGE`, exception type names, Django and Python identifiers, code,
@@ -270,13 +278,18 @@ generally solid against OpenAI and Anthropic's APIs, but a small local model tha
 writes fluent English explanations may produce broken or mixed-language output once
 asked to switch languages.
 
-When a language is configured, the token ceiling (`OPENAI_MAX_TOKENS`) is scaled up
-automatically so explanations in languages that tokenize less efficiently than English
-aren't cut off mid-sentence. This scaling is deliberately generous — billing follows
-tokens actually generated, so unused headroom costs nothing — rather than precise: the
-underlying tokens-per-word figures are estimates, not direct measurements, and the
-default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer, which handles non-Latin
-scripts considerably better than the `cl100k_base`-era ratios these estimates lean on.
+When a language is configured and `OPENAI_MAX_TOKENS` isn't set explicitly, the
+token ceiling defaults to 3,000 instead of 1,000, a flat 3x multiplier applied the
+same way regardless of how well or poorly a given language is known to tokenize, so
+explanations in languages that use more tokens per word than English aren't cut off
+mid-sentence. This is a ceiling, not a target: billing follows tokens actually
+generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TOKENS`
+explicitly always overrides this scaling, at any value, including one lower than the
+unscaled 1,000 default. The multiplier itself is deliberately generous rather than
+precise: the underlying tokens-per-word figures are estimates, not direct
+measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
+which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
+these estimates lean on.
 
 ## Codebase-aware explanations (RAG)
 
@@ -361,40 +374,106 @@ else.)
 
 ### Before / after
 
-**Without RAG**, traceback only:
-
-> Your `ValueError` is raised because the value passed to `foo()` couldn't
-> be converted to an integer. Check where `foo()` is called and make sure
-> you're passing a numeric string.
-
-**With RAG**, grounded in the actual function:
-
-> In `myapp/utils.py`, `foo()` calls `int(value)` on line 12 without a
-> `try`/`except`, so any non-numeric `value` raises `ValueError` straight
-> through to the caller. Since `foo()` is called from `myapp/views.py` with
-> unvalidated form input, add validation there or wrap the `int()` call in
-> `foo()` with a clear error message.
-
-## Example
-
-Here is an example of how to use the middleware in a Django project:
+A real result from the eval harness (`missing_fk`, one of the fixtures in
+`evals/fixtures.py`): a view creates a new `Post` without setting the
+required `author` foreign key. The traceback the model actually received
+was already truncated to `OPENAI_MAX_TRACEBACK_CHARS`, so it contains no
+application code at all, only Django/SQLite internals:
 
-```python
-# settings.py
-
-DEBUG = True
+```
+...(truncated)...
+  File ".../django/db/models/base.py", line 904, in save
+    self.save_base(
+  ...
+  File ".../django/db/backends/utils.py", line 105, in _execute
+    return self.cursor.execute(sql, params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File ".../django/db/backends/sqlite3/base.py", line 359, in execute
+    return super().execute(query, params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id
+```
 
-MIDDLEWARE = [
-    ...
-    'explain_errors.middleware.ExplainErrorsMiddleware',
-]
+**Without RAG**, traceback only:
 
-# .env
+> The error "django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id" occurs when you attempt to save a model instance but the `author_id` field does not receive a value, even though it requires one (is defined as `NOT NULL`).
+>
+> This typically indicates that the `author` field in your `Post` model (or similar) is mandatory, but you did not provide an `author` when creating or updating the post.
+>
+> To fix this:
+>
+> 1. Ensure that you are assigning a valid user or author instance to the `author` field before saving the `Post` model. For example:
+>    ```python
+>    post.author = some_user_instance
+>    ```
+>
+> 2. If you intended for `author_id` to be optional, adjust the model definition to allow null values by modifying the field:
+>    ```python
+>    author = models.ForeignKey(User, on_delete=models.CASCADE, null=True)
+>    ```
+>
+> 3. If the migration is necessary (if you did change the model), run:
+>    ```bash
+>    python manage.py makemigrations
+>    python manage.py migrate
+>    ```
+>
+> Make sure that every time you save a `Post`, it has a valid author assigned.
 
-OPENAI_API_KEY=your_openai_api_key_here
-```
+**With RAG**, grounded in the actual function:
 
-When an error occurs, you will see an explanation printed to stdout.
+> The error `django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id` occurs because when you're creating a new `Post` instance in `clone_latest_post`, you're not providing an `author` for the new draft. The `author` field in the `Post` model is a ForeignKey that cannot be null, and thus must be populated.
+>
+> To fix this, modify the `clone_latest_post` function to include an `author` when creating the draft. You could retrieve the author from the latest post and use it like so:
+>
+> ```python
+> def clone_latest_post(request):
+>     """Duplicate the most recent post as a new draft, ready for editing."""
+>     latest = Post.objects.order_by("-id").first()
+>     draft = Post.objects.create(
+>         title=f"Copy of {latest.title}",
+>         slug=f"copy-of-{latest.slug}",
+>         body=latest.body,
+>         author=latest.author,  # Add this line to specify the author
+>     )
+>     return HttpResponse(f"Created draft #{draft.id}")
+> ```
+>
+> This ensures the `draft` has a valid `author`, satisfying the NOT NULL constraint.
+
+### Does RAG actually help?
+
+To check whether RAG-grounded explanations are actually better, not just
+longer, the package ships an eval harness (`evals/`): fifteen deliberately
+broken Django views, each explained twice (once from the traceback alone,
+once with RAG enabled) and judged by a separate model, blind to which
+explanation is which, against the error's known cause and correct fix
+location. Which side the judge sees as "A" is randomized per comparison so
+position can't bias the result.
+
+Across three runs (45 judged comparisons, 2 judge failures, 43 scored),
+RAG-on won 35, RAG-off 5, and 3 tied. The gap isn't spread evenly across
+everything the judge checks. It's concentrated in whether the explanation
+names the right file and function, and whether it invents details along
+the way: on `points_to_fix_location`, RAG-on answered yes in 26 of the
+group-A comparisons against RAG-off's 13; on `no_fabrication`, 26 against
+17. Without source access, `gpt-4o-mini` tends to invent a
+plausible-sounding function name or parameter rather than say it doesn't
+know; given the actual code via RAG, it mostly does not.
+
+Three limitations are worth knowing before trusting this uncritically: RAG
+can anchor on the wrong retrieved chunk, as it did in one fixture
+(`missing_post_key`) where the fix got redirected to a retrieved template
+instead of the view; the judge is shown the failing function's own
+source, which is the same source RAG-on's retriever draws from, so part
+of RAG-on's `no_fabrication` advantage may be judge and generator
+overlapping on material RAG-off never sees rather than RAG-on being more
+careful; and claim statuses are spot-checked, not exhaustively audited --
+a script that flagged 14 of 363 claims on one run, all correct on manual
+inspection, is a sample that turned up no false positive, not a proof
+that none exists. Full per-fixture results, the judge prompt, and how to
+reproduce this (about $1.37 for a `--runs 3` pass, most of it judge cost)
+are in [`evals/README.md`](evals/README.md).
 
 ## License
 

{django_explain_errors-0.6.0 → django_explain_errors-0.7.0}/README.md

@@ -10,7 +10,8 @@ a local model if you prefer not to send code off your machine.
 
 It can optionally ground explanations in your own project source using a
 local vector index (RAG), so explanations reference the actual code that
-failed instead of staying generic.
+failed instead of staying generic (measured — see "Does RAG actually
+help?" below).
 
 The middleware supports both synchronous (WSGI) and asynchronous (ASGI)
 views. It auto-detects the view chain at startup and routes requests through
@@ -139,7 +140,7 @@ developer is actually investigating.
 | `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for the OpenAI client. Defaults to `10`. |
-| `OPENAI_MAX_TRACEBACK_CHARS` | No | Traceback is trimmed to its last N characters before being sent. Defaults to `3000`. |
+| `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
 | `EXPLAIN_ERRORS_PRESERVE_DEBUG_PAGE` | No | When `True` (the default), the middleware prints the explanation to stdout and returns `None`, so exception handling continues normally and Django renders its standard debug page. Set to `False` to instead return a JSON 500 response, which ends exception handling early (see Compatibility above). |
 | `OPENAI_BASE_URL` (env or settings) | No | Base URL for any OpenAI-compatible API (for example Ollama at `http://localhost:11434/v1`). When set, a missing API key is replaced with a placeholder since local servers do not require one. |
 | `EXPLAIN_ERRORS_MAX_CALLS` | No | Together with `EXPLAIN_ERRORS_WINDOW_SECONDS`, caps API spend to at most this many explanations within a rolling window; once the cap is hit, further errors in that window are not sent for explanation until an earlier call ages out. Defaults to `5`. |
@@ -212,6 +213,13 @@ read them in another language instead:
 EXPLAIN_ERRORS_LANGUAGE = "Spanish"  # or the code form, "es"
 ```
 
+There is no fixed list of supported languages. `EXPLAIN_ERRORS_LANGUAGE` accepts
+any language the configured model can write, because the setting adds one clause
+to the system prompt (see `LANGUAGE_CLAUSE_TEMPLATE` in
+`explain_errors/middleware.py`), not a translation catalog with its own
+maintained language list. How well it works varies by model; see the known
+limitation below.
+
 This is independent of Django's own `LANGUAGE_CODE`, which controls the language your
 site serves to its users, not the language you read explanations in. Regardless of
 `EXPLAIN_ERRORS_LANGUAGE`, exception type names, Django and Python identifiers, code,
@@ -226,13 +234,18 @@ generally solid against OpenAI and Anthropic's APIs, but a small local model tha
 writes fluent English explanations may produce broken or mixed-language output once
 asked to switch languages.
 
-When a language is configured, the token ceiling (`OPENAI_MAX_TOKENS`) is scaled up
-automatically so explanations in languages that tokenize less efficiently than English
-aren't cut off mid-sentence. This scaling is deliberately generous — billing follows
-tokens actually generated, so unused headroom costs nothing — rather than precise: the
-underlying tokens-per-word figures are estimates, not direct measurements, and the
-default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer, which handles non-Latin
-scripts considerably better than the `cl100k_base`-era ratios these estimates lean on.
+When a language is configured and `OPENAI_MAX_TOKENS` isn't set explicitly, the
+token ceiling defaults to 3,000 instead of 1,000, a flat 3x multiplier applied the
+same way regardless of how well or poorly a given language is known to tokenize, so
+explanations in languages that use more tokens per word than English aren't cut off
+mid-sentence. This is a ceiling, not a target: billing follows tokens actually
+generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TOKENS`
+explicitly always overrides this scaling, at any value, including one lower than the
+unscaled 1,000 default. The multiplier itself is deliberately generous rather than
+precise: the underlying tokens-per-word figures are estimates, not direct
+measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
+which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
+these estimates lean on.
 
 ## Codebase-aware explanations (RAG)
 
@@ -317,40 +330,106 @@ else.)
 
 ### Before / after
 
-**Without RAG**, traceback only:
-
-> Your `ValueError` is raised because the value passed to `foo()` couldn't
-> be converted to an integer. Check where `foo()` is called and make sure
-> you're passing a numeric string.
-
-**With RAG**, grounded in the actual function:
-
-> In `myapp/utils.py`, `foo()` calls `int(value)` on line 12 without a
-> `try`/`except`, so any non-numeric `value` raises `ValueError` straight
-> through to the caller. Since `foo()` is called from `myapp/views.py` with
-> unvalidated form input, add validation there or wrap the `int()` call in
-> `foo()` with a clear error message.
-
-## Example
-
-Here is an example of how to use the middleware in a Django project:
+A real result from the eval harness (`missing_fk`, one of the fixtures in
+`evals/fixtures.py`): a view creates a new `Post` without setting the
+required `author` foreign key. The traceback the model actually received
+was already truncated to `OPENAI_MAX_TRACEBACK_CHARS`, so it contains no
+application code at all, only Django/SQLite internals:
 
-```python
-# settings.py
-
-DEBUG = True
+```
+...(truncated)...
+  File ".../django/db/models/base.py", line 904, in save
+    self.save_base(
+  ...
+  File ".../django/db/backends/utils.py", line 105, in _execute
+    return self.cursor.execute(sql, params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File ".../django/db/backends/sqlite3/base.py", line 359, in execute
+    return super().execute(query, params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id
+```
 
-MIDDLEWARE = [
-    ...
-    'explain_errors.middleware.ExplainErrorsMiddleware',
-]
+**Without RAG**, traceback only:
 
-# .env
+> The error "django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id" occurs when you attempt to save a model instance but the `author_id` field does not receive a value, even though it requires one (is defined as `NOT NULL`).
+>
+> This typically indicates that the `author` field in your `Post` model (or similar) is mandatory, but you did not provide an `author` when creating or updating the post.
+>
+> To fix this:
+>
+> 1. Ensure that you are assigning a valid user or author instance to the `author` field before saving the `Post` model. For example:
+>    ```python
+>    post.author = some_user_instance
+>    ```
+>
+> 2. If you intended for `author_id` to be optional, adjust the model definition to allow null values by modifying the field:
+>    ```python
+>    author = models.ForeignKey(User, on_delete=models.CASCADE, null=True)
+>    ```
+>
+> 3. If the migration is necessary (if you did change the model), run:
+>    ```bash
+>    python manage.py makemigrations
+>    python manage.py migrate
+>    ```
+>
+> Make sure that every time you save a `Post`, it has a valid author assigned.
 
-OPENAI_API_KEY=your_openai_api_key_here
-```
+**With RAG**, grounded in the actual function:
 
-When an error occurs, you will see an explanation printed to stdout.
+> The error `django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id` occurs because when you're creating a new `Post` instance in `clone_latest_post`, you're not providing an `author` for the new draft. The `author` field in the `Post` model is a ForeignKey that cannot be null, and thus must be populated.
+>
+> To fix this, modify the `clone_latest_post` function to include an `author` when creating the draft. You could retrieve the author from the latest post and use it like so:
+>
+> ```python
+> def clone_latest_post(request):
+>     """Duplicate the most recent post as a new draft, ready for editing."""
+>     latest = Post.objects.order_by("-id").first()
+>     draft = Post.objects.create(
+>         title=f"Copy of {latest.title}",
+>         slug=f"copy-of-{latest.slug}",
+>         body=latest.body,
+>         author=latest.author,  # Add this line to specify the author
+>     )
+>     return HttpResponse(f"Created draft #{draft.id}")
+> ```
+>
+> This ensures the `draft` has a valid `author`, satisfying the NOT NULL constraint.
+
+### Does RAG actually help?
+
+To check whether RAG-grounded explanations are actually better, not just
+longer, the package ships an eval harness (`evals/`): fifteen deliberately
+broken Django views, each explained twice (once from the traceback alone,
+once with RAG enabled) and judged by a separate model, blind to which
+explanation is which, against the error's known cause and correct fix
+location. Which side the judge sees as "A" is randomized per comparison so
+position can't bias the result.
+
+Across three runs (45 judged comparisons, 2 judge failures, 43 scored),
+RAG-on won 35, RAG-off 5, and 3 tied. The gap isn't spread evenly across
+everything the judge checks. It's concentrated in whether the explanation
+names the right file and function, and whether it invents details along
+the way: on `points_to_fix_location`, RAG-on answered yes in 26 of the
+group-A comparisons against RAG-off's 13; on `no_fabrication`, 26 against
+17. Without source access, `gpt-4o-mini` tends to invent a
+plausible-sounding function name or parameter rather than say it doesn't
+know; given the actual code via RAG, it mostly does not.
+
+Three limitations are worth knowing before trusting this uncritically: RAG
+can anchor on the wrong retrieved chunk, as it did in one fixture
+(`missing_post_key`) where the fix got redirected to a retrieved template
+instead of the view; the judge is shown the failing function's own
+source, which is the same source RAG-on's retriever draws from, so part
+of RAG-on's `no_fabrication` advantage may be judge and generator
+overlapping on material RAG-off never sees rather than RAG-on being more
+careful; and claim statuses are spot-checked, not exhaustively audited --
+a script that flagged 14 of 363 claims on one run, all correct on manual
+inspection, is a sample that turned up no false positive, not a proof
+that none exists. Full per-fixture results, the judge prompt, and how to
+reproduce this (about $1.37 for a `--runs 3` pass, most of it judge cost)
+are in [`evals/README.md`](evals/README.md).
 
 ## License
 

{django_explain_errors-0.6.0 → django_explain_errors-0.7.0/django_explain_errors.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.6.0
+Version: 0.7.0
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -54,7 +54,8 @@ a local model if you prefer not to send code off your machine.
 
 It can optionally ground explanations in your own project source using a
 local vector index (RAG), so explanations reference the actual code that
-failed instead of staying generic.
+failed instead of staying generic (measured — see "Does RAG actually
+help?" below).
 
 The middleware supports both synchronous (WSGI) and asynchronous (ASGI)
 views. It auto-detects the view chain at startup and routes requests through
@@ -183,7 +184,7 @@ developer is actually investigating.
 | `OPENAI_MODEL` | No | Model used for explanations. Defaults to `gpt-4o-mini`. |
 | `OPENAI_MAX_TOKENS` | No | Ceiling on tokens generated for the explanation, not a target — the system prompt itself asks for a concise answer. Defaults to `1000`; scales up automatically when `EXPLAIN_ERRORS_LANGUAGE` is set (see below), unless you set this explicitly, which always overrides the scaling. |
 | `OPENAI_TIMEOUT` | No | Request timeout in seconds for the OpenAI client. Defaults to `10`. |
-| `OPENAI_MAX_TRACEBACK_CHARS` | No | Traceback is trimmed to its last N characters before being sent. Defaults to `3000`. |
+| `OPENAI_MAX_TRACEBACK_CHARS` | No | Total character budget for the traceback sent to the model. Application frames (your own code, as opposed to Django, the standard library, or installed packages) are always kept; library frames fill whatever budget remains, nearest the raise point first, with an `... N library frames omitted ...` line where frames are dropped. If the application frames alone exceed the budget, falls back to keeping the last N characters of the raw traceback. Defaults to `3000`. |
 | `EXPLAIN_ERRORS_PRESERVE_DEBUG_PAGE` | No | When `True` (the default), the middleware prints the explanation to stdout and returns `None`, so exception handling continues normally and Django renders its standard debug page. Set to `False` to instead return a JSON 500 response, which ends exception handling early (see Compatibility above). |
 | `OPENAI_BASE_URL` (env or settings) | No | Base URL for any OpenAI-compatible API (for example Ollama at `http://localhost:11434/v1`). When set, a missing API key is replaced with a placeholder since local servers do not require one. |
 | `EXPLAIN_ERRORS_MAX_CALLS` | No | Together with `EXPLAIN_ERRORS_WINDOW_SECONDS`, caps API spend to at most this many explanations within a rolling window; once the cap is hit, further errors in that window are not sent for explanation until an earlier call ages out. Defaults to `5`. |
@@ -256,6 +257,13 @@ read them in another language instead:
 EXPLAIN_ERRORS_LANGUAGE = "Spanish"  # or the code form, "es"
 ```
 
+There is no fixed list of supported languages. `EXPLAIN_ERRORS_LANGUAGE` accepts
+any language the configured model can write, because the setting adds one clause
+to the system prompt (see `LANGUAGE_CLAUSE_TEMPLATE` in
+`explain_errors/middleware.py`), not a translation catalog with its own
+maintained language list. How well it works varies by model; see the known
+limitation below.
+
 This is independent of Django's own `LANGUAGE_CODE`, which controls the language your
 site serves to its users, not the language you read explanations in. Regardless of
 `EXPLAIN_ERRORS_LANGUAGE`, exception type names, Django and Python identifiers, code,
@@ -270,13 +278,18 @@ generally solid against OpenAI and Anthropic's APIs, but a small local model tha
 writes fluent English explanations may produce broken or mixed-language output once
 asked to switch languages.
 
-When a language is configured, the token ceiling (`OPENAI_MAX_TOKENS`) is scaled up
-automatically so explanations in languages that tokenize less efficiently than English
-aren't cut off mid-sentence. This scaling is deliberately generous — billing follows
-tokens actually generated, so unused headroom costs nothing — rather than precise: the
-underlying tokens-per-word figures are estimates, not direct measurements, and the
-default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer, which handles non-Latin
-scripts considerably better than the `cl100k_base`-era ratios these estimates lean on.
+When a language is configured and `OPENAI_MAX_TOKENS` isn't set explicitly, the
+token ceiling defaults to 3,000 instead of 1,000, a flat 3x multiplier applied the
+same way regardless of how well or poorly a given language is known to tokenize, so
+explanations in languages that use more tokens per word than English aren't cut off
+mid-sentence. This is a ceiling, not a target: billing follows tokens actually
+generated, so the extra headroom costs nothing if unused. Setting `OPENAI_MAX_TOKENS`
+explicitly always overrides this scaling, at any value, including one lower than the
+unscaled 1,000 default. The multiplier itself is deliberately generous rather than
+precise: the underlying tokens-per-word figures are estimates, not direct
+measurements, and the default model (`gpt-4o-mini`) uses the `o200k_base` tokenizer,
+which handles non-Latin scripts considerably better than the `cl100k_base`-era ratios
+these estimates lean on.
 
 ## Codebase-aware explanations (RAG)
 
@@ -361,40 +374,106 @@ else.)
 
 ### Before / after
 
-**Without RAG**, traceback only:
-
-> Your `ValueError` is raised because the value passed to `foo()` couldn't
-> be converted to an integer. Check where `foo()` is called and make sure
-> you're passing a numeric string.
-
-**With RAG**, grounded in the actual function:
-
-> In `myapp/utils.py`, `foo()` calls `int(value)` on line 12 without a
-> `try`/`except`, so any non-numeric `value` raises `ValueError` straight
-> through to the caller. Since `foo()` is called from `myapp/views.py` with
-> unvalidated form input, add validation there or wrap the `int()` call in
-> `foo()` with a clear error message.
-
-## Example
-
-Here is an example of how to use the middleware in a Django project:
+A real result from the eval harness (`missing_fk`, one of the fixtures in
+`evals/fixtures.py`): a view creates a new `Post` without setting the
+required `author` foreign key. The traceback the model actually received
+was already truncated to `OPENAI_MAX_TRACEBACK_CHARS`, so it contains no
+application code at all, only Django/SQLite internals:
 
-```python
-# settings.py
-
-DEBUG = True
+```
+...(truncated)...
+  File ".../django/db/models/base.py", line 904, in save
+    self.save_base(
+  ...
+  File ".../django/db/backends/utils.py", line 105, in _execute
+    return self.cursor.execute(sql, params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File ".../django/db/backends/sqlite3/base.py", line 359, in execute
+    return super().execute(query, params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id
+```
 
-MIDDLEWARE = [
-    ...
-    'explain_errors.middleware.ExplainErrorsMiddleware',
-]
+**Without RAG**, traceback only:
 
-# .env
+> The error "django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id" occurs when you attempt to save a model instance but the `author_id` field does not receive a value, even though it requires one (is defined as `NOT NULL`).
+>
+> This typically indicates that the `author` field in your `Post` model (or similar) is mandatory, but you did not provide an `author` when creating or updating the post.
+>
+> To fix this:
+>
+> 1. Ensure that you are assigning a valid user or author instance to the `author` field before saving the `Post` model. For example:
+>    ```python
+>    post.author = some_user_instance
+>    ```
+>
+> 2. If you intended for `author_id` to be optional, adjust the model definition to allow null values by modifying the field:
+>    ```python
+>    author = models.ForeignKey(User, on_delete=models.CASCADE, null=True)
+>    ```
+>
+> 3. If the migration is necessary (if you did change the model), run:
+>    ```bash
+>    python manage.py makemigrations
+>    python manage.py migrate
+>    ```
+>
+> Make sure that every time you save a `Post`, it has a valid author assigned.
 
-OPENAI_API_KEY=your_openai_api_key_here
-```
+**With RAG**, grounded in the actual function:
 
-When an error occurs, you will see an explanation printed to stdout.
+> The error `django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id` occurs because when you're creating a new `Post` instance in `clone_latest_post`, you're not providing an `author` for the new draft. The `author` field in the `Post` model is a ForeignKey that cannot be null, and thus must be populated.
+>
+> To fix this, modify the `clone_latest_post` function to include an `author` when creating the draft. You could retrieve the author from the latest post and use it like so:
+>
+> ```python
+> def clone_latest_post(request):
+>     """Duplicate the most recent post as a new draft, ready for editing."""
+>     latest = Post.objects.order_by("-id").first()
+>     draft = Post.objects.create(
+>         title=f"Copy of {latest.title}",
+>         slug=f"copy-of-{latest.slug}",
+>         body=latest.body,
+>         author=latest.author,  # Add this line to specify the author
+>     )
+>     return HttpResponse(f"Created draft #{draft.id}")
+> ```
+>
+> This ensures the `draft` has a valid `author`, satisfying the NOT NULL constraint.
+
+### Does RAG actually help?
+
+To check whether RAG-grounded explanations are actually better, not just
+longer, the package ships an eval harness (`evals/`): fifteen deliberately
+broken Django views, each explained twice (once from the traceback alone,
+once with RAG enabled) and judged by a separate model, blind to which
+explanation is which, against the error's known cause and correct fix
+location. Which side the judge sees as "A" is randomized per comparison so
+position can't bias the result.
+
+Across three runs (45 judged comparisons, 2 judge failures, 43 scored),
+RAG-on won 35, RAG-off 5, and 3 tied. The gap isn't spread evenly across
+everything the judge checks. It's concentrated in whether the explanation
+names the right file and function, and whether it invents details along
+the way: on `points_to_fix_location`, RAG-on answered yes in 26 of the
+group-A comparisons against RAG-off's 13; on `no_fabrication`, 26 against
+17. Without source access, `gpt-4o-mini` tends to invent a
+plausible-sounding function name or parameter rather than say it doesn't
+know; given the actual code via RAG, it mostly does not.
+
+Three limitations are worth knowing before trusting this uncritically: RAG
+can anchor on the wrong retrieved chunk, as it did in one fixture
+(`missing_post_key`) where the fix got redirected to a retrieved template
+instead of the view; the judge is shown the failing function's own
+source, which is the same source RAG-on's retriever draws from, so part
+of RAG-on's `no_fabrication` advantage may be judge and generator
+overlapping on material RAG-off never sees rather than RAG-on being more
+careful; and claim statuses are spot-checked, not exhaustively audited --
+a script that flagged 14 of 363 claims on one run, all correct on manual
+inspection, is a sample that turned up no false positive, not a proof
+that none exists. Full per-fixture results, the judge prompt, and how to
+reproduce this (about $1.37 for a `--runs 3` pass, most of it judge cost)
+are in [`evals/README.md`](evals/README.md).
 
 ## License
 

{django_explain_errors-0.6.0 → django_explain_errors-0.7.0}/django_explain_errors.egg-info/SOURCES.txt

@@ -12,6 +12,7 @@ explain_errors/client.py
 explain_errors/middleware.py
 explain_errors/sanitize.py
 explain_errors/throttle.py
+explain_errors/tracebacks.py
 explain_errors/management/__init__.py
 explain_errors/management/commands/__init__.py
 explain_errors/management/commands/build_error_index.py
@@ -21,10 +22,14 @@ explain_errors/rag/retriever.py
 explain_errors/rag/store.py
 tests/test_build_error_index.py
 tests/test_client.py
+tests/test_eval_fixtures.py
+tests/test_eval_judge.py
+tests/test_eval_run.py
 tests/test_language.py
 tests/test_middleware.py
 tests/test_rag.py
 tests/test_sanitize.py
 tests/test_signals.py
 tests/test_throttle.py
+tests/test_tracebacks.py
 tests/test_truncation.py