groundedness 0.1.2__tar.gz

groundedness-0.1.2/LICENSE

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) 2026 Salim Mustafazada
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

groundedness-0.1.2/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: groundedness
+Version: 0.1.2
+Summary: Did the model make this up? Claim-level groundedness check for LLM answers: one call, any OpenAI-compatible model, any language.
+Author: Salim Mustafazada
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/aghasalim/groundedness
+Keywords: llm,hallucination,groundedness,rag,evaluation,multilingual
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# groundedness
+
+**Did the model make this up?** A claim-level groundedness check for LLM answers: one call, any OpenAI-compatible model, any language.
+
+```python
+from groundedness import check
+
+facts  = "Diş klinikası, Nizami küç. 12. B.e–Şənbə 9:00–19:00. Konsultasiya 30 AZN."
+answer = "Salam! Konsultasiya 25 AZN-dir, bazar günü də işləyirik. Ünvan Nizami küçəsi 12."
+
+r = check(answer, sources=[facts], model="qwen/qwen3.8-27b")
+r.unsupported  # ['Konsultasiya 25 AZN-dir', 'bazar günü də işləyirik']
+r.fixed        # 'Salam! Konsultasiya haqqı və bazar günü işləyib-işləmədiyimiz haqqında məlumatım yoxdur. Ünvan Nizami küçəsi 12-dir.'
+r.score        # 0.47
+```
+
+That is Azerbaijani. It works the same in Russian, Turkish, Arabic or English, because the judge reads the answer in its own language. Every trained hallucination detector published so far — Vectara HHEM, LettuceDetect, Patronus Lynx — is English-only.
+
+## Why
+
+RAG apps and support bots answer from documents. The failure that hurts is not a looping decode or a made-up historical date; it is *"a consultation is 25 AZN"* when the document says 30. `groundedness` finds that sentence, names it, and gives you the answer without it.
+
+- **Any model.** Groq, OpenAI, Ollama, vLLM, OpenRouter — anything that speaks `/chat/completions`. No model to download, no GPU.
+- **Any language.** No training data, no language list. If the model can read it, the judge can check it.
+- **Claim-level output.** Not a score you cannot act on: the exact unsupported claims, and a rewrite. Highlight them, hand off to a human, or log the rate.
+- **Zero dependencies.** `urllib` and `json`. Python 3.9+.
+
+## Install
+
+```
+pip install groundedness
+```
+
+Set `OPENAI_BASE_URL` and `OPENAI_API_KEY`, or just `GROQ_API_KEY` (Groq's endpoint is the default when it is set). Or pass `base_url=` / `api_key=` to `check()`.
+
+## CLI
+
+```
+echo "the answer" | groundedness --model qwen/qwen3.8-27b --sources facts.txt policy.md
+```
+
+Prints JSON: `grounded`, `score`, `unsupported`, `fixed`.
+
+## What it is not
+
+It checks an answer against **the sources you give it**, not against the world. A true claim that is not in your documents is reported as unsupported — which is what you want from a bot that must only say what the owner wrote. It is a judge call, so it costs one short completion per answer (about 350 tokens on the example above) and it is as good as the model judging; the tests use `qwen/qwen3.8-27b` on Groq, which caught every planted error in az/ru/en.
+
+## Tests
+
+```
+GROQ_API_KEY=... python -m pytest -q
+```
+
+Three languages, one planted wrong price and one invented opening day each; plus a grounded answer that must come back unchanged.
+
+## Benchmark: which model judges best, in eleven languages
+
+The same case in en, az, ru, tr, uk, kk, ar, fa, hi, id, vi: a clinic's facts, a grounded answer that must pass, and two answers with one planted error each (a wrong price, an invented opening day). Every chat model on Groq's free tier, 19 September 2026. Reproduce with `python benchmark/run.py`; full per-language grid in [`benchmark/RESULTS.md`](benchmark/RESULTS.md).
+
+| Model | Planted errors caught | Grounded answers wrongly flagged | Median latency |
+|---|---:|---:|---:|
+| `openai/gpt-oss-120b` | 22/22 | 0/11 | 0.58 s |
+| `openai/gpt-oss-20b` | 22/22 | 0/11 | 0.48 s |
+| `qwen/qwen3.8-27b` | 22/22 | 0/11 | 0.27 s |
+| `allam-2-7b` | 15/22 | 11/11 | 0.25 s |
+
+And Google's models over their OpenAI-compatible endpoint (`python benchmark/run.py --gemini`, results in [`benchmark/gemini.md`](benchmark/gemini.md)):
+
+| Model | Planted errors caught | Grounded answers wrongly flagged | Median latency |
+|---|---:|---:|---:|
+| `gemini-3.8-flash` | 22/22 | 0/11 | 2.7 s |
+| `gemini-3.5-flash` | 22/22 | 0/11 | 4.4 s |
+| `gemini-2.5-flash` | 22/22 | 0/11 | 4.3 s |
+| `gemini-2.5-pro` | 21/21 | 0/11 | 10.6 s |
+
+Seven models are perfect across all eleven languages, including Kazakh, Persian and Vietnamese, with no false alarms; `qwen/qwen3.8-27b` on Groq is the fastest by ten times. `allam-2-7b` flags every grounded answer and misses a third of the errors: a 7B Arabic-centred model is not a judge. One `gemini-2.5-pro` call (Azerbaijani, wrong price) hit a transient request error and is left out rather than guessed.
+
+A lesson from the first run, kept here so nobody repeats it: thinking models spend their token budget before the first visible character. With `max_tokens: 1200` Gemini 2.5 Pro returned empty replies and an early version of this package scored an empty reply as "grounded" — 0/22 caught, 0 false alarms, a perfect-looking failure. The package now returns `judged=False` for an empty or non-JSON reply, and the benchmark counts it as a miss. The set is small on purpose (33 answers per model, one domain) — it is a smoke test that a model can read the language and follow the instruction, not a measure of fine judgement. Adding a language is one JSON entry in `benchmark/cases.json`; adding a model is one endpoint that lists it.
+
+## Roadmap
+
+- More domains per language (a return policy, a timetable, a contract clause) and harder errors: a right number in the wrong place, a plausible synonym.
+- Models beyond Groq: run the same file against OpenRouter, Ollama, vLLM.
+- A DeepEval / RAGAS metric that wraps this.
+
+## Origin
+
+Extracted from the groundedness judge that runs on every channel of [SIBA](https://siba.az)'s assistant. MIT.

groundedness-0.1.2/README.md

@@ -0,0 +1,88 @@
+# groundedness
+
+**Did the model make this up?** A claim-level groundedness check for LLM answers: one call, any OpenAI-compatible model, any language.
+
+```python
+from groundedness import check
+
+facts  = "Diş klinikası, Nizami küç. 12. B.e–Şənbə 9:00–19:00. Konsultasiya 30 AZN."
+answer = "Salam! Konsultasiya 25 AZN-dir, bazar günü də işləyirik. Ünvan Nizami küçəsi 12."
+
+r = check(answer, sources=[facts], model="qwen/qwen3.8-27b")
+r.unsupported  # ['Konsultasiya 25 AZN-dir', 'bazar günü də işləyirik']
+r.fixed        # 'Salam! Konsultasiya haqqı və bazar günü işləyib-işləmədiyimiz haqqında məlumatım yoxdur. Ünvan Nizami küçəsi 12-dir.'
+r.score        # 0.47
+```
+
+That is Azerbaijani. It works the same in Russian, Turkish, Arabic or English, because the judge reads the answer in its own language. Every trained hallucination detector published so far — Vectara HHEM, LettuceDetect, Patronus Lynx — is English-only.
+
+## Why
+
+RAG apps and support bots answer from documents. The failure that hurts is not a looping decode or a made-up historical date; it is *"a consultation is 25 AZN"* when the document says 30. `groundedness` finds that sentence, names it, and gives you the answer without it.
+
+- **Any model.** Groq, OpenAI, Ollama, vLLM, OpenRouter — anything that speaks `/chat/completions`. No model to download, no GPU.
+- **Any language.** No training data, no language list. If the model can read it, the judge can check it.
+- **Claim-level output.** Not a score you cannot act on: the exact unsupported claims, and a rewrite. Highlight them, hand off to a human, or log the rate.
+- **Zero dependencies.** `urllib` and `json`. Python 3.9+.
+
+## Install
+
+```
+pip install groundedness
+```
+
+Set `OPENAI_BASE_URL` and `OPENAI_API_KEY`, or just `GROQ_API_KEY` (Groq's endpoint is the default when it is set). Or pass `base_url=` / `api_key=` to `check()`.
+
+## CLI
+
+```
+echo "the answer" | groundedness --model qwen/qwen3.8-27b --sources facts.txt policy.md
+```
+
+Prints JSON: `grounded`, `score`, `unsupported`, `fixed`.
+
+## What it is not
+
+It checks an answer against **the sources you give it**, not against the world. A true claim that is not in your documents is reported as unsupported — which is what you want from a bot that must only say what the owner wrote. It is a judge call, so it costs one short completion per answer (about 350 tokens on the example above) and it is as good as the model judging; the tests use `qwen/qwen3.8-27b` on Groq, which caught every planted error in az/ru/en.
+
+## Tests
+
+```
+GROQ_API_KEY=... python -m pytest -q
+```
+
+Three languages, one planted wrong price and one invented opening day each; plus a grounded answer that must come back unchanged.
+
+## Benchmark: which model judges best, in eleven languages
+
+The same case in en, az, ru, tr, uk, kk, ar, fa, hi, id, vi: a clinic's facts, a grounded answer that must pass, and two answers with one planted error each (a wrong price, an invented opening day). Every chat model on Groq's free tier, 19 September 2026. Reproduce with `python benchmark/run.py`; full per-language grid in [`benchmark/RESULTS.md`](benchmark/RESULTS.md).
+
+| Model | Planted errors caught | Grounded answers wrongly flagged | Median latency |
+|---|---:|---:|---:|
+| `openai/gpt-oss-120b` | 22/22 | 0/11 | 0.58 s |
+| `openai/gpt-oss-20b` | 22/22 | 0/11 | 0.48 s |
+| `qwen/qwen3.8-27b` | 22/22 | 0/11 | 0.27 s |
+| `allam-2-7b` | 15/22 | 11/11 | 0.25 s |
+
+And Google's models over their OpenAI-compatible endpoint (`python benchmark/run.py --gemini`, results in [`benchmark/gemini.md`](benchmark/gemini.md)):
+
+| Model | Planted errors caught | Grounded answers wrongly flagged | Median latency |
+|---|---:|---:|---:|
+| `gemini-3.8-flash` | 22/22 | 0/11 | 2.7 s |
+| `gemini-3.5-flash` | 22/22 | 0/11 | 4.4 s |
+| `gemini-2.5-flash` | 22/22 | 0/11 | 4.3 s |
+| `gemini-2.5-pro` | 21/21 | 0/11 | 10.6 s |
+
+Seven models are perfect across all eleven languages, including Kazakh, Persian and Vietnamese, with no false alarms; `qwen/qwen3.8-27b` on Groq is the fastest by ten times. `allam-2-7b` flags every grounded answer and misses a third of the errors: a 7B Arabic-centred model is not a judge. One `gemini-2.5-pro` call (Azerbaijani, wrong price) hit a transient request error and is left out rather than guessed.
+
+A lesson from the first run, kept here so nobody repeats it: thinking models spend their token budget before the first visible character. With `max_tokens: 1200` Gemini 2.5 Pro returned empty replies and an early version of this package scored an empty reply as "grounded" — 0/22 caught, 0 false alarms, a perfect-looking failure. The package now returns `judged=False` for an empty or non-JSON reply, and the benchmark counts it as a miss. The set is small on purpose (33 answers per model, one domain) — it is a smoke test that a model can read the language and follow the instruction, not a measure of fine judgement. Adding a language is one JSON entry in `benchmark/cases.json`; adding a model is one endpoint that lists it.
+
+## Roadmap
+
+- More domains per language (a return policy, a timetable, a contract clause) and harder errors: a right number in the wrong place, a plausible synonym.
+- Models beyond Groq: run the same file against OpenRouter, Ollama, vLLM.
+- A DeepEval / RAGAS metric that wraps this.
+
+## Origin
+
+Extracted from the groundedness judge that runs on every channel of [SIBA](https://siba.az)'s assistant. MIT.

groundedness-0.1.2/groundedness/__init__.py

@@ -0,0 +1,130 @@
+"""groundedness — did the model make this up?
+
+One call, any OpenAI-compatible model, any language. The judge reads the
+answer in its own language, which is what makes it work where every trained
+detector (HHEM, LettuceDetect, Lynx) is English-only.
+
+    from groundedness import check
+    r = check(answer, sources=[doc], model="qwen/qwen3.8-27b")
+    r.unsupported   # claims the sources do not support
+    r.fixed         # the answer with those claims removed, same language
+    r.score         # 1.0 = fully grounded
+
+No dependencies. Uses OPENAI_BASE_URL / OPENAI_API_KEY (or GROQ_API_KEY).
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+import urllib.error
+import urllib.request
+from dataclasses import dataclass, field
+
+__version__ = "0.1.2"
+__all__ = ["check", "Result", "PROMPT"]
+
+PROMPT = (
+    "You check an assistant's answer against the only facts it was allowed to use. Facts:\n{facts}\n\n"
+    "List every factual claim in the answer (a price, a date, a time, an address, a phone, a name, "
+    "an availability, a rule, a number, a quote) that the facts above do not support. Greetings, offers "
+    "to help, and promises to follow up are not claims. Then rewrite the answer in the same language and "
+    "tone with the unsupported claims removed, saying plainly that you do not have that information. "
+    "If no claim is unsupported, return the answer unchanged. Reply with JSON only: "
+    '{{"unsupported": ["..."], "answer": "..."}}'
+)
+
+
+@dataclass
+class Result:
+    unsupported: list[str] = field(default_factory=list)
+    fixed: str = ""
+    original: str = ""
+    model: str = ""
+    raw: str = ""
+    judged: bool = True
+    """False when the model returned nothing usable: treat as unknown, never as grounded."""
+
+    @property
+    def grounded(self) -> bool:
+        return self.judged and not self.unsupported
+
+    @property
+    def score(self) -> float:
+        """1.0 when nothing was unsupported; otherwise 1 minus the share of the answer's text the unsupported claims make up."""
+        if not self.unsupported:
+            return 1.0
+        if not self.original:
+            return 0.0
+        bad = sum(len(u) for u in self.unsupported)
+        return max(0.0, 1.0 - min(1.0, bad / len(self.original)))
+
+
+def _endpoint() -> tuple[str, str]:
+    base = os.environ.get("OPENAI_BASE_URL") or ("https://api.groq.com/openai/v1" if os.environ.get("GROQ_API_KEY") else "https://api.openai.com/v1")
+    key = os.environ.get("OPENAI_API_KEY") or os.environ.get("GROQ_API_KEY") or ""
+    return base.rstrip("/"), key
+
+
+def _facts(sources) -> str:
+    if isinstance(sources, str):
+        sources = [sources]
+    return "\n\n".join(f"[{i + 1}] {s}" for i, s in enumerate(sources) if s and s.strip())
+
+
+def check(answer: str, sources, model: str, *, base_url: str | None = None, api_key: str | None = None, timeout: float = 30.0, temperature: float = 0.0) -> Result:
+    """Judge `answer` against `sources` with `model` over an OpenAI-compatible chat endpoint.
+
+    Never raises on a bad model reply: a judge that cannot be parsed returns the
+    answer unchanged with `unsupported=[]` and `raw` set, so a caller can log it.
+    Network and auth errors do raise.
+    """
+    facts = _facts(sources)
+    if not facts or not answer or not answer.strip():
+        return Result(fixed=answer, original=answer, model=model)
+    base, key = _endpoint()
+    base = (base_url or base).rstrip("/")
+    key = api_key or key
+    # Thinking models spend tokens before the first visible character; the
+    # budget has to cover that or the reply comes back empty.
+    body = {"model": model, "temperature": temperature, "max_tokens": 8000, "messages": [{"role": "system", "content": PROMPT.format(facts=facts)}, {"role": "user", "content": answer}]}
+    if "qwen3" in model:
+        body["reasoning_format"] = "hidden"
+    if "gpt-oss" in model:
+        body["reasoning_effort"] = "low"
+    headers = {"content-type": "application/json", "authorization": f"Bearer {key}", "user-agent": f"groundedness/{__version__} (+https://github.com/aghasalim/groundedness)"}
+    # Small models cap max_tokens below the thinking-model budget; on that
+    # specific 400, retry once with the cap the server names (or 2000).
+    data = None
+    for _attempt in (1, 2):
+        req = urllib.request.Request(f"{base}/chat/completions", data=json.dumps(body).encode(), headers=headers)
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as r:
+                data = json.load(r)
+            break
+        except urllib.error.HTTPError as e:
+            msg = e.read().decode("utf-8", "replace") if e.code == 400 else ""
+            m = re.search(r"max_tokens.*?`?(\d{3,6})`?", msg)
+            if _attempt == 1 and e.code == 400 and "max_tokens" in msg:
+                body["max_tokens"] = min(int(m.group(1)) if m else 2000, body["max_tokens"] - 1)
+                continue
+            raise
+    if data is None:
+        return Result(fixed=answer, original=answer, model=model, judged=False)
+    raw = (data.get("choices") or [{}])[0].get("message", {}).get("content", "") or ""
+    if not raw.strip():
+        return Result(fixed=answer, original=answer, model=model, raw=raw, judged=False)
+    text = re.sub(r"^```(?:json)?|```$", "", raw.strip(), flags=re.M).strip()
+    try:
+        j = json.loads(text)
+    except json.JSONDecodeError:
+        m = re.search(r"\{[\s\S]*\}", text)
+        try:
+            j = json.loads(m.group(0)) if m else {}
+        except json.JSONDecodeError:
+            j = {}
+    if not isinstance(j, dict) or "unsupported" not in j:
+        return Result(fixed=answer, original=answer, model=model, raw=raw, judged=False)
+    unsupported = [str(x).strip() for x in j.get("unsupported", []) if str(x).strip()]
+    fixed = j.get("answer") if isinstance(j.get("answer"), str) else None
+    return Result(unsupported=unsupported, fixed=(fixed or answer).strip(), original=answer, model=model, raw=raw)

groundedness-0.1.2/groundedness/__main__.py

@@ -0,0 +1,21 @@
+"""`python -m groundedness --model M --sources facts.txt < answer.txt`"""
+import argparse
+import json
+import sys
+
+from . import check
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(prog="groundedness", description="Did the model make this up? Answer on stdin, sources as files.")
+    ap.add_argument("--model", required=True)
+    ap.add_argument("--sources", nargs="+", required=True, help="text files the answer must be grounded in")
+    ap.add_argument("--base-url")
+    a = ap.parse_args()
+    sources = [open(p, encoding="utf-8").read() for p in a.sources]
+    r = check(sys.stdin.read(), sources, a.model, base_url=a.base_url)
+    print(json.dumps({"grounded": r.grounded, "score": round(r.score, 3), "unsupported": r.unsupported, "fixed": r.fixed}, ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+    main()

groundedness-0.1.2/groundedness.egg-info/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: groundedness
+Version: 0.1.2
+Summary: Did the model make this up? Claim-level groundedness check for LLM answers: one call, any OpenAI-compatible model, any language.
+Author: Salim Mustafazada
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/aghasalim/groundedness
+Keywords: llm,hallucination,groundedness,rag,evaluation,multilingual
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# groundedness
+
+**Did the model make this up?** A claim-level groundedness check for LLM answers: one call, any OpenAI-compatible model, any language.
+
+```python
+from groundedness import check
+
+facts  = "Diş klinikası, Nizami küç. 12. B.e–Şənbə 9:00–19:00. Konsultasiya 30 AZN."
+answer = "Salam! Konsultasiya 25 AZN-dir, bazar günü də işləyirik. Ünvan Nizami küçəsi 12."
+
+r = check(answer, sources=[facts], model="qwen/qwen3.8-27b")
+r.unsupported  # ['Konsultasiya 25 AZN-dir', 'bazar günü də işləyirik']
+r.fixed        # 'Salam! Konsultasiya haqqı və bazar günü işləyib-işləmədiyimiz haqqında məlumatım yoxdur. Ünvan Nizami küçəsi 12-dir.'
+r.score        # 0.47
+```
+
+That is Azerbaijani. It works the same in Russian, Turkish, Arabic or English, because the judge reads the answer in its own language. Every trained hallucination detector published so far — Vectara HHEM, LettuceDetect, Patronus Lynx — is English-only.
+
+## Why
+
+RAG apps and support bots answer from documents. The failure that hurts is not a looping decode or a made-up historical date; it is *"a consultation is 25 AZN"* when the document says 30. `groundedness` finds that sentence, names it, and gives you the answer without it.
+
+- **Any model.** Groq, OpenAI, Ollama, vLLM, OpenRouter — anything that speaks `/chat/completions`. No model to download, no GPU.
+- **Any language.** No training data, no language list. If the model can read it, the judge can check it.
+- **Claim-level output.** Not a score you cannot act on: the exact unsupported claims, and a rewrite. Highlight them, hand off to a human, or log the rate.
+- **Zero dependencies.** `urllib` and `json`. Python 3.9+.
+
+## Install
+
+```
+pip install groundedness
+```
+
+Set `OPENAI_BASE_URL` and `OPENAI_API_KEY`, or just `GROQ_API_KEY` (Groq's endpoint is the default when it is set). Or pass `base_url=` / `api_key=` to `check()`.
+
+## CLI
+
+```
+echo "the answer" | groundedness --model qwen/qwen3.8-27b --sources facts.txt policy.md
+```
+
+Prints JSON: `grounded`, `score`, `unsupported`, `fixed`.
+
+## What it is not
+
+It checks an answer against **the sources you give it**, not against the world. A true claim that is not in your documents is reported as unsupported — which is what you want from a bot that must only say what the owner wrote. It is a judge call, so it costs one short completion per answer (about 350 tokens on the example above) and it is as good as the model judging; the tests use `qwen/qwen3.8-27b` on Groq, which caught every planted error in az/ru/en.
+
+## Tests
+
+```
+GROQ_API_KEY=... python -m pytest -q
+```
+
+Three languages, one planted wrong price and one invented opening day each; plus a grounded answer that must come back unchanged.
+
+## Benchmark: which model judges best, in eleven languages
+
+The same case in en, az, ru, tr, uk, kk, ar, fa, hi, id, vi: a clinic's facts, a grounded answer that must pass, and two answers with one planted error each (a wrong price, an invented opening day). Every chat model on Groq's free tier, 19 September 2026. Reproduce with `python benchmark/run.py`; full per-language grid in [`benchmark/RESULTS.md`](benchmark/RESULTS.md).
+
+| Model | Planted errors caught | Grounded answers wrongly flagged | Median latency |
+|---|---:|---:|---:|
+| `openai/gpt-oss-120b` | 22/22 | 0/11 | 0.58 s |
+| `openai/gpt-oss-20b` | 22/22 | 0/11 | 0.48 s |
+| `qwen/qwen3.8-27b` | 22/22 | 0/11 | 0.27 s |
+| `allam-2-7b` | 15/22 | 11/11 | 0.25 s |
+
+And Google's models over their OpenAI-compatible endpoint (`python benchmark/run.py --gemini`, results in [`benchmark/gemini.md`](benchmark/gemini.md)):
+
+| Model | Planted errors caught | Grounded answers wrongly flagged | Median latency |
+|---|---:|---:|---:|
+| `gemini-3.8-flash` | 22/22 | 0/11 | 2.7 s |
+| `gemini-3.5-flash` | 22/22 | 0/11 | 4.4 s |
+| `gemini-2.5-flash` | 22/22 | 0/11 | 4.3 s |
+| `gemini-2.5-pro` | 21/21 | 0/11 | 10.6 s |
+
+Seven models are perfect across all eleven languages, including Kazakh, Persian and Vietnamese, with no false alarms; `qwen/qwen3.8-27b` on Groq is the fastest by ten times. `allam-2-7b` flags every grounded answer and misses a third of the errors: a 7B Arabic-centred model is not a judge. One `gemini-2.5-pro` call (Azerbaijani, wrong price) hit a transient request error and is left out rather than guessed.
+
+A lesson from the first run, kept here so nobody repeats it: thinking models spend their token budget before the first visible character. With `max_tokens: 1200` Gemini 2.5 Pro returned empty replies and an early version of this package scored an empty reply as "grounded" — 0/22 caught, 0 false alarms, a perfect-looking failure. The package now returns `judged=False` for an empty or non-JSON reply, and the benchmark counts it as a miss. The set is small on purpose (33 answers per model, one domain) — it is a smoke test that a model can read the language and follow the instruction, not a measure of fine judgement. Adding a language is one JSON entry in `benchmark/cases.json`; adding a model is one endpoint that lists it.
+
+## Roadmap
+
+- More domains per language (a return policy, a timetable, a contract clause) and harder errors: a right number in the wrong place, a plausible synonym.
+- Models beyond Groq: run the same file against OpenRouter, Ollama, vLLM.
+- A DeepEval / RAGAS metric that wraps this.
+
+## Origin
+
+Extracted from the groundedness judge that runs on every channel of [SIBA](https://siba.az)'s assistant. MIT.

groundedness-0.1.2/groundedness.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+README.md
+pyproject.toml
+groundedness/__init__.py
+groundedness/__main__.py
+groundedness.egg-info/PKG-INFO
+groundedness.egg-info/SOURCES.txt
+groundedness.egg-info/dependency_links.txt
+groundedness.egg-info/entry_points.txt
+groundedness.egg-info/top_level.txt
+tests/test_groundedness.py

groundedness-0.1.2/groundedness.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

groundedness-0.1.2/groundedness.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+groundedness = groundedness.__main__:main

groundedness-0.1.2/groundedness.egg-info/top_level.txt

@@ -0,0 +1 @@
+groundedness

groundedness-0.1.2/pyproject.toml

@@ -0,0 +1,24 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "groundedness"
+version = "0.1.2"
+description = "Did the model make this up? Claim-level groundedness check for LLM answers: one call, any OpenAI-compatible model, any language."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.9"
+authors = [{ name = "Salim Mustafazada" }]
+keywords = ["llm", "hallucination", "groundedness", "rag", "evaluation", "multilingual"]
+classifiers = ["Programming Language :: Python :: 3", "Operating System :: OS Independent", "Topic :: Scientific/Engineering :: Artificial Intelligence"]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/aghasalim/groundedness"
+
+[project.scripts]
+groundedness = "groundedness.__main__:main"
+
+[tool.setuptools.packages.find]
+include = ["groundedness*"]

groundedness-0.1.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

groundedness-0.1.2/tests/test_groundedness.py

@@ -0,0 +1,44 @@
+"""One live check per language. Needs GROQ_API_KEY (or OPENAI_*); skipped without it.
+
+    GROQ_API_KEY=... python -m pytest -q
+"""
+import os
+
+import pytest
+
+from groundedness import check
+
+MODEL = os.environ.get("GROUNDEDNESS_MODEL", "qwen/qwen3.8-27b")
+KEY = os.environ.get("GROQ_API_KEY") or os.environ.get("OPENAI_API_KEY")
+
+CASES = {
+    "az": ("Diş klinikası, Nizami küç. 12. B.e–Şənbə 9:00–19:00. Konsultasiya 30 AZN.",
+           "Salam! Konsultasiya 25 AZN-dir, bazar günü də işləyirik. Ünvan Nizami küçəsi 12.", "25"),
+    "ru": ("Стоматология, ул. Низами 12. Пн–Сб 9:00–19:00. Консультация 30 AZN.",
+           "Здравствуйте! Консультация стоит 25 AZN, работаем и по воскресеньям. Адрес: ул. Низами 12.", "25"),
+    "en": ("Dental clinic, 12 Nizami St. Mon–Sat 9:00–19:00. A consultation is 30 AZN.",
+           "Hello! A consultation is 25 AZN and we are open on Sundays too. We are at 12 Nizami St.", "25"),
+}
+
+
+@pytest.mark.skipif(not KEY, reason="no API key")
+@pytest.mark.parametrize("lang", list(CASES))
+def test_catches_wrong_price(lang):
+    facts, answer, bad = CASES[lang]
+    r = check(answer, [facts], MODEL)
+    assert not r.grounded
+    assert any(bad in u for u in r.unsupported), r.unsupported
+    assert bad not in r.fixed or "30" in r.fixed, r.fixed
+
+
+@pytest.mark.skipif(not KEY, reason="no API key")
+def test_grounded_answer_unchanged():
+    facts, _, _ = CASES["en"]
+    ok = "Hello! A consultation is 30 AZN. We are at 12 Nizami St, open Monday to Saturday 9 to 19."
+    r = check(ok, [facts], MODEL)
+    assert r.grounded and r.score == 1.0
+
+
+def test_no_sources_is_a_noop():
+    r = check("anything", [], "any-model")
+    assert r.grounded and r.fixed == "anything"