hallucinationbench 0.1.0__tar.gz

hallucinationbench-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Devasish Banerjee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hallucinationbench-0.1.0/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: hallucinationbench
+Version: 0.1.0
+Summary: Detect hallucinations in your RAG pipeline output in two lines of Python.
+Home-page: https://github.com/bdeva1975/hallucinationbench
+Author: Devasish Banerjee
+Author-email: bdeva1975@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: httpx>=0.27.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 🔬 HallucinationBench
+
+**Detect hallucinations in your RAG pipeline output — in two lines of Python.**
+
+![Python](https://img.shields.io/badge/python-3.10%2B-blue)
+![License](https://img.shields.io/badge/license-MIT-green)
+![OpenAI](https://img.shields.io/badge/powered%20by-OpenAI-412991)
+
+---
+
+## The problem
+
+Your RAG pipeline retrieves documents and passes them to an LLM.
+The LLM generates a response that *sounds* correct.
+But is every claim actually grounded in your context — or did the model fabricate some of it?
+
+**HallucinationBench answers that question instantly.**
+
+---
+
+## Quickstart
+
+Install dependencies:
+
+```bash
+pip install openai python-dotenv
+```
+
+Set your OpenAI API key:
+
+```bash
+# .env
+OPENAI_API_KEY=your_key_here
+```
+
+Run your first evaluation:
+
+```python
+from hallucinationbench import score
+
+context = """
+The Eiffel Tower is located in Paris, France. It was constructed between
+1887 and 1889 as the entrance arch for the 1889 World's Fair.
+The tower is 330 metres tall.
+"""
+
+response = """
+The Eiffel Tower is in Paris. It was built in 1889 and stands 330 metres
+tall. It was designed by Leonardo da Vinci and attracts over 7 million
+visitors every year.
+"""
+
+result = score(context=context, response=response)
+print(result)
+```
+
+Output:
+
+```
+Verdict          : FAIL
+Faithfulness     : 0.40
+
+Grounded claims  (2):
+  ✓  The Eiffel Tower is in Paris.
+  ✓  It stands 330 metres tall.
+
+Hallucinated claims  (3):
+  ✗  It was built in 1889.
+  ✗  It was designed by Leonardo da Vinci.
+  ✗  It attracts over 7 million visitors every year.
+```
+
+---
+
+## The result object
+
+```python
+result.faithfulness_score    # float 0.0 – 1.0
+result.grounded_claims       # list of supported statements
+result.hallucinated_claims   # list of fabricated statements
+result.verdict               # "PASS" | "WARN" | "FAIL"
+result.model                 # judge model used
+```
+
+| Verdict | Faithfulness Score |
+|---------|-------------------|
+| ✅ PASS  | >= 0.8            |
+| ⚠️ WARN  | >= 0.5            |
+| ❌ FAIL  | < 0.5             |
+
+---
+
+## Streamlit demo
+
+Run the interactive demo locally:
+
+```bash
+streamlit run app.py
+```
+
+Paste any context and LLM response — get an instant hallucination report.
+
+---
+
+## How it works
+
+1. Your `context` and `response` are sent to `gpt-4o-mini` as a structured judge prompt.
+2. The judge breaks the response into individual factual claims.
+3. Each claim is classified as grounded (supported by context) or hallucinated (absent or contradicted).
+4. A faithfulness score is calculated: `grounded_claims / total_claims`.
+5. A verdict of PASS, WARN, or FAIL is assigned.
+
+The judge uses `response_format: json_object` to guarantee structured output.
+Temperature is set to 0 for deterministic results.
+
+---
+
+## Cost
+
+Each evaluation uses `gpt-4o-mini`.
+Typical cost: **~$0.001 per evaluation** (well under a tenth of a cent).
+
+---
+
+## Roadmap
+
+- [ ] Batch evaluation across multiple context/response pairs
+- [ ] CSV upload support in the Streamlit app
+- [ ] Custom judge model selection
+- [ ] LangChain and LlamaIndex integration hooks
+- [ ] CI/CD integration example (GitHub Actions)
+
+---
+
+## Project structure
+
+```
+hallucinationbench/
+├── hallucinationbench/
+│   ├── __init__.py       # public API
+│   ├── scorer.py         # GPT-4o-mini judge
+│   └── models.py         # ScoreResult dataclass
+├── app.py                # Streamlit demo
+├── example.py            # quickstart example
+├── requirements.txt
+├── .env.example
+└── README.md
+```
+
+---
+
+## License
+
+MIT — free to use, modify, and distribute.
+
+---
+
+## Contributing
+
+Pull requests are welcome. Please open an issue first to discuss what you would like to change.
+
+---
+
+*Built with OpenAI GPT-4o-mini as the hallucination judge.*

hallucinationbench-0.1.0/README.md

@@ -0,0 +1,164 @@
+# 🔬 HallucinationBench
+
+**Detect hallucinations in your RAG pipeline output — in two lines of Python.**
+
+![Python](https://img.shields.io/badge/python-3.10%2B-blue)
+![License](https://img.shields.io/badge/license-MIT-green)
+![OpenAI](https://img.shields.io/badge/powered%20by-OpenAI-412991)
+
+---
+
+## The problem
+
+Your RAG pipeline retrieves documents and passes them to an LLM.
+The LLM generates a response that *sounds* correct.
+But is every claim actually grounded in your context — or did the model fabricate some of it?
+
+**HallucinationBench answers that question instantly.**
+
+---
+
+## Quickstart
+
+Install dependencies:
+
+```bash
+pip install openai python-dotenv
+```
+
+Set your OpenAI API key:
+
+```bash
+# .env
+OPENAI_API_KEY=your_key_here
+```
+
+Run your first evaluation:
+
+```python
+from hallucinationbench import score
+
+context = """
+The Eiffel Tower is located in Paris, France. It was constructed between
+1887 and 1889 as the entrance arch for the 1889 World's Fair.
+The tower is 330 metres tall.
+"""
+
+response = """
+The Eiffel Tower is in Paris. It was built in 1889 and stands 330 metres
+tall. It was designed by Leonardo da Vinci and attracts over 7 million
+visitors every year.
+"""
+
+result = score(context=context, response=response)
+print(result)
+```
+
+Output:
+
+```
+Verdict          : FAIL
+Faithfulness     : 0.40
+
+Grounded claims  (2):
+  ✓  The Eiffel Tower is in Paris.
+  ✓  It stands 330 metres tall.
+
+Hallucinated claims  (3):
+  ✗  It was built in 1889.
+  ✗  It was designed by Leonardo da Vinci.
+  ✗  It attracts over 7 million visitors every year.
+```
+
+---
+
+## The result object
+
+```python
+result.faithfulness_score    # float 0.0 – 1.0
+result.grounded_claims       # list of supported statements
+result.hallucinated_claims   # list of fabricated statements
+result.verdict               # "PASS" | "WARN" | "FAIL"
+result.model                 # judge model used
+```
+
+| Verdict | Faithfulness Score |
+|---------|-------------------|
+| ✅ PASS  | >= 0.8            |
+| ⚠️ WARN  | >= 0.5            |
+| ❌ FAIL  | < 0.5             |
+
+---
+
+## Streamlit demo
+
+Run the interactive demo locally:
+
+```bash
+streamlit run app.py
+```
+
+Paste any context and LLM response — get an instant hallucination report.
+
+---
+
+## How it works
+
+1. Your `context` and `response` are sent to `gpt-4o-mini` as a structured judge prompt.
+2. The judge breaks the response into individual factual claims.
+3. Each claim is classified as grounded (supported by context) or hallucinated (absent or contradicted).
+4. A faithfulness score is calculated: `grounded_claims / total_claims`.
+5. A verdict of PASS, WARN, or FAIL is assigned.
+
+The judge uses `response_format: json_object` to guarantee structured output.
+Temperature is set to 0 for deterministic results.
+
+---
+
+## Cost
+
+Each evaluation uses `gpt-4o-mini`.
+Typical cost: **~$0.001 per evaluation** (well under a tenth of a cent).
+
+---
+
+## Roadmap
+
+- [ ] Batch evaluation across multiple context/response pairs
+- [ ] CSV upload support in the Streamlit app
+- [ ] Custom judge model selection
+- [ ] LangChain and LlamaIndex integration hooks
+- [ ] CI/CD integration example (GitHub Actions)
+
+---
+
+## Project structure
+
+```
+hallucinationbench/
+├── hallucinationbench/
+│   ├── __init__.py       # public API
+│   ├── scorer.py         # GPT-4o-mini judge
+│   └── models.py         # ScoreResult dataclass
+├── app.py                # Streamlit demo
+├── example.py            # quickstart example
+├── requirements.txt
+├── .env.example
+└── README.md
+```
+
+---
+
+## License
+
+MIT — free to use, modify, and distribute.
+
+---
+
+## Contributing
+
+Pull requests are welcome. Please open an issue first to discuss what you would like to change.
+
+---
+
+*Built with OpenAI GPT-4o-mini as the hallucination judge.*

hallucinationbench-0.1.0/hallucinationbench/__init__.py

@@ -0,0 +1,5 @@
+from hallucinationbench.scorer import score
+from hallucinationbench.models import ScoreResult
+
+__version__ = "0.1.0"
+__all__ = ["score", "ScoreResult"]

hallucinationbench-0.1.0/hallucinationbench/models.py

@@ -0,0 +1,44 @@
+from dataclasses import dataclass, field
+from typing import List
+
+
+@dataclass
+class ScoreResult:
+    """
+    The result returned by hallucinationbench.score().
+
+    Attributes:
+        faithfulness_score  : float between 0.0 and 1.0.
+                              1.0 means fully grounded, 0.0 means fully hallucinated.
+        grounded_claims     : list of statements in the response that are
+                              supported by the context.
+        hallucinated_claims : list of statements in the response that are
+                              NOT supported by, or contradict, the context.
+        verdict             : "PASS"  → faithfulness_score >= 0.8
+                              "WARN"  → faithfulness_score >= 0.5
+                              "FAIL"  → faithfulness_score < 0.5
+        model               : the OpenAI model used as the judge.
+    """
+
+    faithfulness_score: float
+    grounded_claims: List[str] = field(default_factory=list)
+    hallucinated_claims: List[str] = field(default_factory=list)
+    verdict: str = "FAIL"
+    model: str = "gpt-4o-mini"
+
+    def __str__(self) -> str:
+        lines = [
+            f"Verdict          : {self.verdict}",
+            f"Faithfulness     : {self.faithfulness_score:.2f}",
+            f"",
+            f"Grounded claims  ({len(self.grounded_claims)}):",
+        ]
+        for claim in self.grounded_claims:
+            lines.append(f"  ✓  {claim}")
+
+        lines.append(f"")
+        lines.append(f"Hallucinated claims  ({len(self.hallucinated_claims)}):")
+        for claim in self.hallucinated_claims:
+            lines.append(f"  ✗  {claim}")
+
+        return "\n".join(lines)

hallucinationbench-0.1.0/hallucinationbench/scorer.py

@@ -0,0 +1,142 @@
+import os
+import json
+from openai import OpenAI
+from dotenv import load_dotenv
+from hallucinationbench.models import ScoreResult
+
+load_dotenv()
+
+_client = None
+
+
+def _get_client() -> OpenAI:
+    """Lazy-initialise the OpenAI client once."""
+    global _client
+    if _client is None:
+        api_key = os.getenv("OPENAI_API_KEY")
+        if not api_key:
+            raise EnvironmentError(
+                "OPENAI_API_KEY not found. "
+                "Set it in your .env file or as an environment variable."
+            )
+        _client = OpenAI(api_key=api_key)
+    return _client
+
+
+_SYSTEM_PROMPT = """
+You are a hallucination detection judge.
+
+Your job is to evaluate whether an AI-generated response is faithful to a 
+given context. A response is faithful if every factual claim it makes is 
+directly supported by the context. A claim is hallucinated if it introduces 
+facts, figures, names, or statements that are absent from or contradict 
+the context.
+
+You must respond ONLY with a valid JSON object. No explanation. No markdown.
+No code fences. Raw JSON only.
+
+The JSON must follow this exact schema:
+{
+  "grounded_claims": ["claim 1", "claim 2"],
+  "hallucinated_claims": ["claim A", "claim B"],
+  "faithfulness_score": 0.75
+}
+
+Rules:
+- Break the response into individual factual claims.
+- For each claim, decide: grounded (supported by context) or hallucinated.
+- faithfulness_score = grounded_claims / total_claims.
+  If there are no claims at all, set faithfulness_score to 1.0.
+- Keep each claim as a short, self-contained sentence.
+- Do not include opinions, hedges, or non-factual statements as claims.
+""".strip()
+
+
+def _build_user_prompt(context: str, response: str) -> str:
+    return (
+        f"CONTEXT:\n{context.strip()}\n\n"
+        f"RESPONSE TO EVALUATE:\n{response.strip()}"
+    )
+
+
+def _parse_result(raw_json: str, model: str) -> ScoreResult:
+    """Parse the JSON returned by the judge into a ScoreResult."""
+    try:
+        data = json.loads(raw_json)
+    except json.JSONDecodeError as e:
+        raise ValueError(
+            f"Judge returned invalid JSON.\n"
+            f"Raw output: {raw_json}\n"
+            f"Error: {e}"
+        )
+
+    grounded = data.get("grounded_claims", [])
+    hallucinated = data.get("hallucinated_claims", [])
+    score = float(data.get("faithfulness_score", 0.0))
+
+    # Clamp to [0.0, 1.0] defensively
+    score = max(0.0, min(1.0, score))
+
+    if score >= 0.8:
+        verdict = "PASS"
+    elif score >= 0.5:
+        verdict = "WARN"
+    else:
+        verdict = "FAIL"
+
+    return ScoreResult(
+        faithfulness_score=score,
+        grounded_claims=grounded,
+        hallucinated_claims=hallucinated,
+        verdict=verdict,
+        model=model,
+    )
+
+
+def score(
+    context: str,
+    response: str,
+    model: str = "gpt-4o-mini",
+) -> ScoreResult:
+    """
+    Evaluate whether an LLM response is faithful to the provided context.
+
+    Args:
+        context  : The source text the response should be grounded in.
+                   Typically your retrieved RAG documents.
+        response : The LLM-generated response to evaluate.
+        model    : OpenAI model to use as the judge.
+                   Default: "gpt-4o-mini" (fast and cheap).
+
+    Returns:
+        ScoreResult with faithfulness_score, grounded_claims,
+        hallucinated_claims, and verdict.
+
+    Example:
+        from hallucinationbench import score
+
+        result = score(
+            context="The Eiffel Tower is located in Paris, France.",
+            response="The Eiffel Tower is in Paris. It was built in 1889."
+        )
+        print(result)
+    """
+    if not context or not context.strip():
+        raise ValueError("context cannot be empty.")
+    if not response or not response.strip():
+        raise ValueError("response cannot be empty.")
+
+    client = _get_client()
+
+    completion = client.chat.completions.create(
+        model=model,
+        messages=[
+            {"role": "system", "content": _SYSTEM_PROMPT},
+            {"role": "user", "content": _build_user_prompt(context, response)},
+        ],
+        temperature=0,
+        response_format={"type": "json_object"},
+    )
+
+    raw_json = completion.choices[0].message.content
+    return _parse_result(raw_json, model)

hallucinationbench-0.1.0/hallucinationbench.egg-info/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: hallucinationbench
+Version: 0.1.0
+Summary: Detect hallucinations in your RAG pipeline output in two lines of Python.
+Home-page: https://github.com/bdeva1975/hallucinationbench
+Author: Devasish Banerjee
+Author-email: bdeva1975@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: httpx>=0.27.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 🔬 HallucinationBench
+
+**Detect hallucinations in your RAG pipeline output — in two lines of Python.**
+
+![Python](https://img.shields.io/badge/python-3.10%2B-blue)
+![License](https://img.shields.io/badge/license-MIT-green)
+![OpenAI](https://img.shields.io/badge/powered%20by-OpenAI-412991)
+
+---
+
+## The problem
+
+Your RAG pipeline retrieves documents and passes them to an LLM.
+The LLM generates a response that *sounds* correct.
+But is every claim actually grounded in your context — or did the model fabricate some of it?
+
+**HallucinationBench answers that question instantly.**
+
+---
+
+## Quickstart
+
+Install dependencies:
+
+```bash
+pip install openai python-dotenv
+```
+
+Set your OpenAI API key:
+
+```bash
+# .env
+OPENAI_API_KEY=your_key_here
+```
+
+Run your first evaluation:
+
+```python
+from hallucinationbench import score
+
+context = """
+The Eiffel Tower is located in Paris, France. It was constructed between
+1887 and 1889 as the entrance arch for the 1889 World's Fair.
+The tower is 330 metres tall.
+"""
+
+response = """
+The Eiffel Tower is in Paris. It was built in 1889 and stands 330 metres
+tall. It was designed by Leonardo da Vinci and attracts over 7 million
+visitors every year.
+"""
+
+result = score(context=context, response=response)
+print(result)
+```
+
+Output:
+
+```
+Verdict          : FAIL
+Faithfulness     : 0.40
+
+Grounded claims  (2):
+  ✓  The Eiffel Tower is in Paris.
+  ✓  It stands 330 metres tall.
+
+Hallucinated claims  (3):
+  ✗  It was built in 1889.
+  ✗  It was designed by Leonardo da Vinci.
+  ✗  It attracts over 7 million visitors every year.
+```
+
+---
+
+## The result object
+
+```python
+result.faithfulness_score    # float 0.0 – 1.0
+result.grounded_claims       # list of supported statements
+result.hallucinated_claims   # list of fabricated statements
+result.verdict               # "PASS" | "WARN" | "FAIL"
+result.model                 # judge model used
+```
+
+| Verdict | Faithfulness Score |
+|---------|-------------------|
+| ✅ PASS  | >= 0.8            |
+| ⚠️ WARN  | >= 0.5            |
+| ❌ FAIL  | < 0.5             |
+
+---
+
+## Streamlit demo
+
+Run the interactive demo locally:
+
+```bash
+streamlit run app.py
+```
+
+Paste any context and LLM response — get an instant hallucination report.
+
+---
+
+## How it works
+
+1. Your `context` and `response` are sent to `gpt-4o-mini` as a structured judge prompt.
+2. The judge breaks the response into individual factual claims.
+3. Each claim is classified as grounded (supported by context) or hallucinated (absent or contradicted).
+4. A faithfulness score is calculated: `grounded_claims / total_claims`.
+5. A verdict of PASS, WARN, or FAIL is assigned.
+
+The judge uses `response_format: json_object` to guarantee structured output.
+Temperature is set to 0 for deterministic results.
+
+---
+
+## Cost
+
+Each evaluation uses `gpt-4o-mini`.
+Typical cost: **~$0.001 per evaluation** (well under a tenth of a cent).
+
+---
+
+## Roadmap
+
+- [ ] Batch evaluation across multiple context/response pairs
+- [ ] CSV upload support in the Streamlit app
+- [ ] Custom judge model selection
+- [ ] LangChain and LlamaIndex integration hooks
+- [ ] CI/CD integration example (GitHub Actions)
+
+---
+
+## Project structure
+
+```
+hallucinationbench/
+├── hallucinationbench/
+│   ├── __init__.py       # public API
+│   ├── scorer.py         # GPT-4o-mini judge
+│   └── models.py         # ScoreResult dataclass
+├── app.py                # Streamlit demo
+├── example.py            # quickstart example
+├── requirements.txt
+├── .env.example
+└── README.md
+```
+
+---
+
+## License
+
+MIT — free to use, modify, and distribute.
+
+---
+
+## Contributing
+
+Pull requests are welcome. Please open an issue first to discuss what you would like to change.
+
+---
+
+*Built with OpenAI GPT-4o-mini as the hallucination judge.*

hallucinationbench-0.1.0/hallucinationbench.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+README.md
+setup.py
+hallucinationbench/__init__.py
+hallucinationbench/models.py
+hallucinationbench/scorer.py
+hallucinationbench.egg-info/PKG-INFO
+hallucinationbench.egg-info/SOURCES.txt
+hallucinationbench.egg-info/dependency_links.txt
+hallucinationbench.egg-info/requires.txt
+hallucinationbench.egg-info/top_level.txt

hallucinationbench-0.1.0/hallucinationbench.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

hallucinationbench-0.1.0/hallucinationbench.egg-info/requires.txt

@@ -0,0 +1,3 @@
+openai>=1.0.0
+python-dotenv>=1.0.0
+httpx>=0.27.0

hallucinationbench-0.1.0/hallucinationbench.egg-info/top_level.txt

@@ -0,0 +1 @@
+hallucinationbench

hallucinationbench-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

hallucinationbench-0.1.0/setup.py

@@ -0,0 +1,28 @@
+from setuptools import setup, find_packages
+
+with open("README.md", "r", encoding="utf-8") as f:
+    long_description = f.read()
+
+setup(
+    name="hallucinationbench",
+    version="0.1.0",
+    author="Devasish Banerjee",
+    author_email="bdeva1975@gmail.com",
+    description="Detect hallucinations in your RAG pipeline output in two lines of Python.",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/bdeva1975/hallucinationbench",
+    packages=find_packages(),
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    ],
+    python_requires=">=3.10",
+    install_requires=[
+        "openai>=1.0.0",
+        "python-dotenv>=1.0.0",
+        "httpx>=0.27.0",
+    ],
+)