django-explain-errors 0.9.2__tar.gz → 0.9.3__tar.gz

{django_explain_errors-0.9.2/django_explain_errors.egg-info → django_explain_errors-0.9.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.9.2
+Version: 0.9.3
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -546,41 +546,33 @@ django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id
 
 ### Does RAG actually help?
 
-To check whether RAG-grounded explanations are actually better, not just
-longer, the package ships an eval harness (`evals/`): fifteen deliberately
-broken Django views, each explained twice (once from the traceback alone,
-once with RAG enabled) and judged by a separate model, blind to which
-explanation is which, against the error's known cause and correct fix
-location. Which side the judge sees as "A" is randomized per comparison so
-position can't bias the result.
-
-Across three runs (45 judged comparisons, 2 judge failures, 43 scored),
-RAG-on won 35, RAG-off 5, and 3 tied. The gap isn't spread evenly across
-everything the judge checks. It's concentrated in whether the explanation
-names the right file and function, and whether it invents details along
-the way: on `points_to_fix_location`, RAG-on answered yes in 26 of the
-group-A comparisons against RAG-off's 13; on `no_fabrication`, 26 against
-17. Without source access, `gpt-4o-mini` tends to invent a
-plausible-sounding function name or parameter rather than say it doesn't
-know; given the actual code via RAG, it mostly does not.
-
-Three limitations are worth knowing before trusting this uncritically: RAG
-can anchor on the wrong retrieved chunk, as it did in one fixture
-(`missing_post_key`) where the fix got redirected to a retrieved template
-instead of the view; the judge is shown the failing function's own
-source, which is the same source RAG-on's retriever draws from, so part
-of RAG-on's `no_fabrication` advantage may be judge and generator
-overlapping on material RAG-off never sees rather than RAG-on being more
-careful; and claim statuses are spot-checked, not exhaustively audited --
-a script that flagged 14 of 363 claims on one run, all correct on manual
-inspection, is a sample that turned up no false positive, not a proof
-that none exists. Full per-fixture results, the judge prompt, and how to
-reproduce this (about $1.37 for a `--runs 3` pass, most of it judge cost)
-are in [`evals/README.md`](evals/README.md).
+The package can explain an error in two ways:
+
+- **RAG-off (default):** the model sees only the Django traceback.
+- **RAG-on:** the model also sees relevant source code from your project, retrieved from a local index.
+
+The eval harness in `evals/` tests whether RAG-on explanations are more accurate, not just longer. It triggers 15 deliberately broken Django views, explains each one both ways, and has a separate judge model compare the two against the error's known cause and fix location. The judge doesn't know which explanation used RAG, and their order is randomized.
+
+**Results (3 runs, 43 scored comparisons):** RAG-on won 35, RAG-off won 5, and 3 tied.
+
+Most of the gap comes from two checks, measured on the 10 fixtures whose cause is in the app's own code (30 comparisons):
+
+- **Points to the right file and function:** RAG-on 26, RAG-off 13
+- **Avoids invented details:** RAG-on 26, RAG-off 17
+
+Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+
+**Limitations:**
+
+- RAG can retrieve the wrong code. In one fixture (`missing_post_key`), it pointed the fix at a template instead of the view.
+- The judge checks claims against the same source RAG-on retrieves from, which may inflate RAG-on's score for avoiding invented details.
+- Claims are spot-checked, not fully audited. All 14 claims flagged on one run held up on manual review, but that is a sample, not proof.
+
+Full results, methodology, and how to reproduce (about $1.37 per `--runs 3` pass) are in [evals/README.md](https://github.com/topunix/django-explain-errors/blob/main/evals/README.md).
 
 ## License
 
-This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
+This project is licensed under the MIT License. See the [LICENSE](https://github.com/topunix/django-explain-errors/blob/main/LICENSE) file for details.
 
 ## Contributing
 

{django_explain_errors-0.9.2 → django_explain_errors-0.9.3}/README.md

@@ -496,41 +496,33 @@ django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id
 
 ### Does RAG actually help?
 
-To check whether RAG-grounded explanations are actually better, not just
-longer, the package ships an eval harness (`evals/`): fifteen deliberately
-broken Django views, each explained twice (once from the traceback alone,
-once with RAG enabled) and judged by a separate model, blind to which
-explanation is which, against the error's known cause and correct fix
-location. Which side the judge sees as "A" is randomized per comparison so
-position can't bias the result.
-
-Across three runs (45 judged comparisons, 2 judge failures, 43 scored),
-RAG-on won 35, RAG-off 5, and 3 tied. The gap isn't spread evenly across
-everything the judge checks. It's concentrated in whether the explanation
-names the right file and function, and whether it invents details along
-the way: on `points_to_fix_location`, RAG-on answered yes in 26 of the
-group-A comparisons against RAG-off's 13; on `no_fabrication`, 26 against
-17. Without source access, `gpt-4o-mini` tends to invent a
-plausible-sounding function name or parameter rather than say it doesn't
-know; given the actual code via RAG, it mostly does not.
-
-Three limitations are worth knowing before trusting this uncritically: RAG
-can anchor on the wrong retrieved chunk, as it did in one fixture
-(`missing_post_key`) where the fix got redirected to a retrieved template
-instead of the view; the judge is shown the failing function's own
-source, which is the same source RAG-on's retriever draws from, so part
-of RAG-on's `no_fabrication` advantage may be judge and generator
-overlapping on material RAG-off never sees rather than RAG-on being more
-careful; and claim statuses are spot-checked, not exhaustively audited --
-a script that flagged 14 of 363 claims on one run, all correct on manual
-inspection, is a sample that turned up no false positive, not a proof
-that none exists. Full per-fixture results, the judge prompt, and how to
-reproduce this (about $1.37 for a `--runs 3` pass, most of it judge cost)
-are in [`evals/README.md`](evals/README.md).
+The package can explain an error in two ways:
+
+- **RAG-off (default):** the model sees only the Django traceback.
+- **RAG-on:** the model also sees relevant source code from your project, retrieved from a local index.
+
+The eval harness in `evals/` tests whether RAG-on explanations are more accurate, not just longer. It triggers 15 deliberately broken Django views, explains each one both ways, and has a separate judge model compare the two against the error's known cause and fix location. The judge doesn't know which explanation used RAG, and their order is randomized.
+
+**Results (3 runs, 43 scored comparisons):** RAG-on won 35, RAG-off won 5, and 3 tied.
+
+Most of the gap comes from two checks, measured on the 10 fixtures whose cause is in the app's own code (30 comparisons):
+
+- **Points to the right file and function:** RAG-on 26, RAG-off 13
+- **Avoids invented details:** RAG-on 26, RAG-off 17
+
+Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+
+**Limitations:**
+
+- RAG can retrieve the wrong code. In one fixture (`missing_post_key`), it pointed the fix at a template instead of the view.
+- The judge checks claims against the same source RAG-on retrieves from, which may inflate RAG-on's score for avoiding invented details.
+- Claims are spot-checked, not fully audited. All 14 claims flagged on one run held up on manual review, but that is a sample, not proof.
+
+Full results, methodology, and how to reproduce (about $1.37 per `--runs 3` pass) are in [evals/README.md](https://github.com/topunix/django-explain-errors/blob/main/evals/README.md).
 
 ## License
 
-This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
+This project is licensed under the MIT License. See the [LICENSE](https://github.com/topunix/django-explain-errors/blob/main/LICENSE) file for details.
 
 ## Contributing
 

{django_explain_errors-0.9.2 → django_explain_errors-0.9.3/django_explain_errors.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-explain-errors
-Version: 0.9.2
+Version: 0.9.3
 Summary: Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.
 Home-page: https://github.com/topunix/django-explain-errors
 Author: topunix
@@ -546,41 +546,33 @@ django.db.utils.IntegrityError: NOT NULL constraint failed: blog_post.author_id
 
 ### Does RAG actually help?
 
-To check whether RAG-grounded explanations are actually better, not just
-longer, the package ships an eval harness (`evals/`): fifteen deliberately
-broken Django views, each explained twice (once from the traceback alone,
-once with RAG enabled) and judged by a separate model, blind to which
-explanation is which, against the error's known cause and correct fix
-location. Which side the judge sees as "A" is randomized per comparison so
-position can't bias the result.
-
-Across three runs (45 judged comparisons, 2 judge failures, 43 scored),
-RAG-on won 35, RAG-off 5, and 3 tied. The gap isn't spread evenly across
-everything the judge checks. It's concentrated in whether the explanation
-names the right file and function, and whether it invents details along
-the way: on `points_to_fix_location`, RAG-on answered yes in 26 of the
-group-A comparisons against RAG-off's 13; on `no_fabrication`, 26 against
-17. Without source access, `gpt-4o-mini` tends to invent a
-plausible-sounding function name or parameter rather than say it doesn't
-know; given the actual code via RAG, it mostly does not.
-
-Three limitations are worth knowing before trusting this uncritically: RAG
-can anchor on the wrong retrieved chunk, as it did in one fixture
-(`missing_post_key`) where the fix got redirected to a retrieved template
-instead of the view; the judge is shown the failing function's own
-source, which is the same source RAG-on's retriever draws from, so part
-of RAG-on's `no_fabrication` advantage may be judge and generator
-overlapping on material RAG-off never sees rather than RAG-on being more
-careful; and claim statuses are spot-checked, not exhaustively audited --
-a script that flagged 14 of 363 claims on one run, all correct on manual
-inspection, is a sample that turned up no false positive, not a proof
-that none exists. Full per-fixture results, the judge prompt, and how to
-reproduce this (about $1.37 for a `--runs 3` pass, most of it judge cost)
-are in [`evals/README.md`](evals/README.md).
+The package can explain an error in two ways:
+
+- **RAG-off (default):** the model sees only the Django traceback.
+- **RAG-on:** the model also sees relevant source code from your project, retrieved from a local index.
+
+The eval harness in `evals/` tests whether RAG-on explanations are more accurate, not just longer. It triggers 15 deliberately broken Django views, explains each one both ways, and has a separate judge model compare the two against the error's known cause and fix location. The judge doesn't know which explanation used RAG, and their order is randomized.
+
+**Results (3 runs, 43 scored comparisons):** RAG-on won 35, RAG-off won 5, and 3 tied.
+
+Most of the gap comes from two checks, measured on the 10 fixtures whose cause is in the app's own code (30 comparisons):
+
+- **Points to the right file and function:** RAG-on 26, RAG-off 13
+- **Avoids invented details:** RAG-on 26, RAG-off 17
+
+Without your source, gpt-4o-mini tends to invent a plausible function name or parameter rather than admit it doesn't know. With the real code in front of it, it mostly doesn't.
+
+**Limitations:**
+
+- RAG can retrieve the wrong code. In one fixture (`missing_post_key`), it pointed the fix at a template instead of the view.
+- The judge checks claims against the same source RAG-on retrieves from, which may inflate RAG-on's score for avoiding invented details.
+- Claims are spot-checked, not fully audited. All 14 claims flagged on one run held up on manual review, but that is a sample, not proof.
+
+Full results, methodology, and how to reproduce (about $1.37 per `--runs 3` pass) are in [evals/README.md](https://github.com/topunix/django-explain-errors/blob/main/evals/README.md).
 
 ## License
 
-This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
+This project is licensed under the MIT License. See the [LICENSE](https://github.com/topunix/django-explain-errors/blob/main/LICENSE) file for details.
 
 ## Contributing
 

{django_explain_errors-0.9.2 → django_explain_errors-0.9.3}/setup.py

@@ -10,7 +10,7 @@ with open('README.md', encoding='utf-8') as f:
 
 setup(
     name='django-explain-errors',
-    version='0.9.2',
+    version='0.9.3',
     packages=find_packages(exclude=['tests', 'tests.*', 'evals', 'evals.*']),
     description='Django middleware that explains unhandled exceptions in DEBUG using an LLM, optionally grounded in your own project source via a local vector index. Works with OpenAI, Claude, or any OpenAI-compatible endpoint including local models.',
     long_description_content_type='text/markdown',