ragfaith-proxy 0.1.0__tar.gz

ragfaith_proxy-0.1.0/PKG-INFO

@@ -0,0 +1,10 @@
+Metadata-Version: 2.4
+Name: ragfaith-proxy
+Version: 0.1.0
+Summary: Universal OpenAI-compatible sidecar proxy with a live ragfaith faithfulness cascade
+Requires-Python: >=3.11
+Provides-Extra: spacy
+Requires-Dist: spacy>=3.7; extra == "spacy"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"

ragfaith_proxy-0.1.0/README.md

@@ -0,0 +1,166 @@
+# ragfaith-proxy
+
+Universal OpenAI-compatible sidecar proxy with a **live ragfaith faithfulness
+cascade**. Any host tool without a plugin system — chat GUIs, agent harnesses,
+scripts — just points its OpenAI base URL at this proxy. Streaming replies pass
+through byte-for-byte (**first token never delayed**) while claims in the reply
+are judged against the sources actually pulled in the conversation
+(`role: "tool"` results). Flagged claims are corrected with a nudge, in-band
+or on the next turn.
+
+Pure Python stdlib. No third-party runtime dependencies.
+
+## Install & run
+
+```sh
+pip install .
+ragfaith-proxy                    # listens on 127.0.0.1:8787
+```
+
+Tests (no network): `pip install .[dev] && pytest -q`
+
+## Authentication: client-auth-only
+
+The proxy holds **no API key of its own**. Every client request must carry its
+own `Authorization: Bearer <provider key>` header, which the proxy forwards to
+the upstream verbatim. The client's token also authenticates the judge calls
+that request's cascade triggers. Requests without an `Authorization` header
+get 401. Consequence: the proxy can be hosted openly — anyone who connects
+needs and spends only their own provider key.
+
+## FlowDown (reference example)
+
+FlowDown is an Apple AI chat client that accepts a custom OpenAI-compatible
+endpoint. To use it with the cascade:
+
+1. Start the proxy: `ragfaith-proxy` (default `http://127.0.0.1:8787`).
+2. In FlowDown, add a custom OpenAI-compatible provider with:
+   - Base URL: `http://127.0.0.1:8787/v1`
+   - API key: your real provider key (e.g. a synthetic key) — it is forwarded
+     upstream and reused for that session's judge calls.
+3. Chat as usual. When a reply misstates the retrieved sources, the next turn
+   automatically carries a corrective nudge — or, in chain mode, the correction
+   streams right after the flagged reply.
+
+Any other client works the same way: set its base URL to
+`http://127.0.0.1:8787/v1`.
+
+## How it works
+
+```
+client -> proxy -> upstream LLM
+             |-> stream back immediately, buffer a copy
+             |-> on finish: decompose claims (spaCy/regex sentences)
+             |-> judge claims vs pulled tool-result premises
+             |-> flagged? nudge
+```
+
+- **Model detection**: the request's `model` field is the active model.
+- **Judge selection** (never self-judge): the active model is compared against
+  the configured GLM judge model (provider prefixes and `:` variants ignored).
+  A match is judged by the configured DeepSeek model; anything else is judged
+  by the configured GLM model. Both models are configurable for any
+  OpenAI-compatible endpoint via `RFE_JUDGE_GLM_MODEL` /
+  `RFE_JUDGE_DEEPSEEK_MODEL` — no `hf:`-style id shape required.
+- **Premises**: content the conversation actually pulled — `role: "tool"`
+  messages and `tool_result` content blocks, capped to the most recent
+  24k chars (`RFE_PREMISE_CAP`). A small per-conversation store covers clients
+  that do not resend full history (LRU-capped at 1000 conversations);
+  conversations are keyed by `X-Conversation-Id`, else a hash of the first
+  user message (full message list if none), else Host header.
+
+## Nudge modes (`RFE_NUDGE_MODE`)
+
+- **`chain` (default)**: after a streaming reply with flagged claims finishes,
+  the proxy withholds the stream close, issues ONE internal continuation call
+  (original messages + assistant reply + aggregated nudge), streams it down the
+  same SSE stream, then terminates with a single `[DONE]`. Non-streaming
+  requests can't be chained, so they fall back to `next` behavior (logged).
+  **Spend caveat**: every flagged reply costs one extra upstream completion —
+  chain mode roughly doubles the upstream cost of flagged replies.
+- **`next`**: the nudge is stashed per conversation and injected as a user
+  message right after the flagged assistant message in the next request's
+  history.
+
+All-faithful replies are fully silent: no extra call, no injection.
+
+Default nudge template:
+
+```
+ragfaith judge (<model>): N claim(s) in your previous reply were flagged
+<verdict>: <claims>. Re-check against the sources actually pulled in this
+conversation and reconcile; do not invent corrections.
+```
+
+## Environment variables
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `RFE_PROXY_HOST` | `127.0.0.1` | Listen address |
+| `RFE_PROXY_PORT` | `8787` | Listen port |
+| `RFE_UPSTREAM_BASE` | `https://api.synthetic.new/v1` | Upstream OpenAI-compatible base |
+| `RFE_JUDGE_PROVIDER` | `synthetic` | Preset: `synthetic` or `openrouter` (free-form with generic vars) |
+| `RFE_JUDGE_BASE` | preset | Generic judge base URL; overrides preset |
+| `RFE_JUDGE_GLM_MODEL` | preset | Generic primary judge model; overrides preset |
+| `RFE_JUDGE_DEEPSEEK_MODEL` | preset | Generic GLM-active judge model; overrides preset |
+| `RFE_SYNTHETIC_BASE` | upstream base | Judge base URL (synthetic preset) |
+| `RFE_SYNTHETIC_GLM_MODEL` | `hf:zai-org/GLM-5.3-Flash` | Primary judge model (synthetic preset) |
+| `RFE_SYNTHETIC_DEEPSEEK_MODEL` | `hf:deepseek-ai/DeepSeek-V4.1-Flash` | Judge when active model is GLM (synthetic preset) |
+| `RFE_OPENROUTER_BASE` | `https://openrouter.ai/api/v1` | Judge base URL (openrouter preset) |
+| `RFE_OPENROUTER_GLM_MODEL` | `z-ai/glm-5.3-flash` | OpenRouter primary judge |
+| `RFE_OPENROUTER_DEEPSEEK_MODEL` | `deepseek/deepseek-v4.1-flash` | OpenRouter GLM-active judge |
+| `RFE_NUDGE_MODE` | `chain` | `chain` or `next` |
+| `RFE_PREMISE_CAP` | `24000` | Max chars of premises per verdict |
+| `RFE_HOST_DECORATORS` | — | JSON map of per-host overrides |
+| `RFE_HOST_CONFIG` | — | Path to same JSON map in a file |
+| `RFE_CACHE_DIR` | — | Append-only JSONL verdict cache dir |
+| `RFE_JUDGE_LOG` | stderr | File for session token log lines |
+| `RFE_MAX_BODY` | `10485760` | Max request body bytes (larger gets 413) |
+
+Precedence: generic `RFE_JUDGE_*` vars → preset vars (`RFE_SYNTHETIC_*` /
+`RFE_OPENROUTER_*`, selected by `RFE_JUDGE_PROVIDER`) → built-in defaults.
+Example — judge with any OpenAI-compatible endpoint:
+
+```sh
+export RFE_JUDGE_PROVIDER=my-endpoint        # preset name is free-form
+export RFE_JUDGE_BASE=http://llm.internal/v1
+export RFE_JUDGE_GLM_MODEL=openai/gpt-oss-120b
+export RFE_JUDGE_DEEPSEEK_MODEL=mistral/magistral-small
+```
+
+Per-host decorators are matched by the `X-RFE-Host` header (exact key), else
+by User-Agent substring, else `default`:
+
+```json
+{"flowdown": {"template": "...", "nudge_mode": "next"}, "default": {}}
+```
+
+## Logging & cache
+
+Token log lines go to stderr (or `RFE_JUDGE_LOG` file), one JSON per judge
+call — tokens only, no cost math:
+
+```json
+{"ts": 1757680000.0, "kind": "judge", "model": "...", "prompt_tokens": 0, "completion_tokens": 0, "conversation": "..."}
+```
+
+With `RFE_CACHE_DIR` set, verdicts persist to
+`<dir>/proxy-cache-<judgeModelSanitized>.jsonl` (`{"key","verdict"}` rows,
+sha256 keys over model/context/claim). Unset: in-memory dict only.
+
+## spaCy (optional)
+
+Claim decomposition uses `split_claims`, a vendored copy of
+`rag_faithfulness_eval.decompose` (copied, not imported). With
+`pip install ragfaith-proxy[spacy]` you get the benchmarked spaCy sentencizer
+path. Without it the proxy falls back to a regex sentence splitter and logs a
+prominent warning — claim boundaries may differ from benchmark results in
+edge cases.
+
+## Failure behavior
+
+The proxy never blocks or corrupts the stream: any upstream/judge/cascade
+failure is logged and the response passes through untouched. Client
+disconnects mid-stream abort cleanly. Responses are requested
+`Accept-Encoding: identity` from upstream so the buffered copy stays
+parseable.

ragfaith_proxy-0.1.0/pyproject.toml

@@ -0,0 +1,20 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ragfaith-proxy"
+version = "0.1.0"
+description = "Universal OpenAI-compatible sidecar proxy with a live ragfaith faithfulness cascade"
+requires-python = ">=3.11"
+dependencies = []
+
+[project.optional-dependencies]
+spacy = ["spacy>=3.7"]
+dev = ["pytest>=8", "ruff>=0.6"]
+
+[project.scripts]
+ragfaith-proxy = "ragfaith_proxy.proxy:main"
+
+[tool.setuptools.packages.find]
+include = ["ragfaith_proxy*"]

ragfaith_proxy-0.1.0/ragfaith_proxy/__init__.py

@@ -0,0 +1,3 @@
+"""ragfaith-proxy: universal OpenAI-compatible sidecar with a live faithfulness cascade."""
+
+__version__ = "0.1.0"

ragfaith_proxy-0.1.0/ragfaith_proxy/decompose.py

@@ -0,0 +1,56 @@
+"""Claim decomposition, copied from rag_faithfulness_eval/decompose.py.
+
+spaCy sentencizer is the benchmarked path (pip install ragfaith-proxy[spacy]).
+Without spaCy we degrade to a regex sentence splitter and warn loudly; claim
+offsets stay verbatim char-offsets into the text either way.
+"""
+
+import logging
+import re
+from types import SimpleNamespace
+
+logger = logging.getLogger(__name__)
+
+_nlp = None
+
+_SENT_RE = re.compile(r"[^.!?\n]+(?:[.!?]+|\n+|$)")
+
+
+def _get_nlp():  # lazy: spacy is an optional extra, the proxy must not need it
+    global _nlp
+    if _nlp is not None:
+        return _nlp
+    try:
+        import spacy
+
+        nlp = spacy.blank("en")
+        nlp.add_pipe("sentencizer")
+        _nlp = nlp
+    except ImportError:
+        # spaCy parity is the benchmarked path; regex is the no-dep fallback
+        logger.warning(
+            "spaCy not installed: using regex sentence splitter (degraded claim "
+            "boundaries). Install ragfaith-proxy[spacy] for benchmark parity."
+        )
+
+        def regex_nlp(text: str):
+            sents = []
+            for m in _SENT_RE.finditer(text):
+                seg = m.group()
+                stripped = seg.strip()
+                if not stripped:
+                    continue
+                start = m.start() + len(seg) - len(seg.lstrip())
+                sents.append(
+                    SimpleNamespace(start_char=start, end_char=start + len(stripped), text=stripped)
+                )
+            return SimpleNamespace(sents=sents)
+
+        _nlp = regex_nlp
+    return _nlp
+
+
+def split_claims(text: str) -> list[tuple[int, int, str]]:
+    """Return [(start, end, sentence)] with char offsets into text."""
+    doc = _get_nlp()(text)
+    return [(s.start_char, s.end_char, s.text) for s in doc.sents if s.text.strip()]

ragfaith_proxy-0.1.0/ragfaith_proxy/judge.py

@@ -0,0 +1,160 @@
+"""LLM judge for ragfaith-proxy (OpenAI-compatible chat API, stdlib only).
+
+Adapted from rag_faithfulness_eval/llm_judge.py: provider-pluggable base URL
+and key supplied by the caller (env-only, no .env loading), token-only log
+lines instead of USD accounting.
+"""
+
+import hashlib
+import json
+import logging
+import re
+import sys
+import threading
+import time
+import urllib.error
+import urllib.request
+from collections import OrderedDict
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+VERDICTS = ("faithful", "unfaithful", "unverifiable")
+VCACHE_MAX = 10000
+RETRIES = 3  # streaming handler must not stall minutes on an upstream outage
+
+SYSTEM_PROMPT = (
+    "You are a RAG faithfulness judge. / Du bist ein RAG-Treuerichter.\n"
+    "Decide if the CLAIM is fully supported by the CONTEXT alone (never use "
+    "outside knowledge). Answer with ONLY one JSON object, no other text:\n"
+    '{"verdict": "faithful"} - every fact in the claim is supported by the context\n'
+    '{"verdict": "unfaithful"} - at least one fact contradicts or is unsupported '
+    "by the context (wrong entity, number, date, or fabricated detail)\n"
+    '{"verdict": "unverifiable"} - the context does not address the claim at all'
+)
+
+
+def _verdict_key(model: str, context: str, claim: str) -> str:
+    return hashlib.sha256(f"{model}\x00{context}\x00{claim}".encode()).hexdigest()
+
+
+def _parse_verdict(text: str) -> str:
+    m = re.search(r'"verdict"\s*:\s*"(\w+)"', text)
+    verdict = m.group(1) if m else ""
+    if verdict not in VERDICTS:
+        raise ValueError(f"unparseable verdict: {text[:200]!r}")
+    return verdict
+
+
+class Judge:
+    """Chat-model judge against any OpenAI-compatible base URL."""
+
+    def __init__(
+        self,
+        model: str,
+        base_url: str,
+        api_key: str,
+        *,
+        max_tokens: int = 256,
+        cache_path: Path | None = None,
+        log=None,
+        vcache_max: int = VCACHE_MAX,
+    ):
+        self.model = model
+        self.base_url = base_url.rstrip("/")
+        self.api_key = api_key
+        self.max_tokens = max_tokens
+        self.cache_path = cache_path
+        self._log = log or (lambda r: print(json.dumps(r), file=sys.stderr, flush=True))
+        self._lock = threading.Lock()
+        self._vcache_max = vcache_max
+        self._vcache: OrderedDict[str, str] = OrderedDict()
+        if cache_path and cache_path.exists():
+            for line in cache_path.read_text().splitlines():
+                if line.strip():
+                    row = json.loads(line)
+                    self._vcache[row["key"]] = row["verdict"]
+
+    def _call(self, user_msg: str, retries: int = RETRIES, max_tokens: int | None = None) -> dict:
+        body = json.dumps(
+            {
+                "model": self.model,
+                "temperature": 0,
+                "max_tokens": max_tokens or self.max_tokens,
+                "reasoning": {"exclude": True},  # hide reasoning; still billed ~100-250 tok
+                "messages": [
+                    {"role": "system", "content": SYSTEM_PROMPT},
+                    {"role": "user", "content": user_msg},
+                ],
+            }
+        ).encode()
+        url = f"{self.base_url}/chat/completions"
+        for attempt in range(retries):
+            req = urllib.request.Request(
+                url,
+                data=body,
+                headers={
+                    "Authorization": f"Bearer {self.api_key}",
+                    "Content-Type": "application/json",
+                },
+            )
+            try:
+                with urllib.request.urlopen(req, timeout=120) as resp:
+                    return json.loads(resp.read())
+            except urllib.error.HTTPError as e:
+                if e.code in (429, 500, 502, 503) and attempt < retries - 1:
+                    time.sleep(2**attempt)
+                    continue
+                raise
+            except (urllib.error.URLError, TimeoutError) as e:
+                # network drop: bounded retry (<=15 s total sleep at 3 attempts)
+                if attempt < retries - 1:
+                    logger.warning("network error (%s); retry %d/%d", e, attempt + 1, retries)
+                    time.sleep(min(60, 5 * 2**attempt))
+                    continue
+                raise
+        raise RuntimeError("unreachable: retry loop exhausted")
+
+    def _account(self, resp: dict, conversation: str) -> None:
+        usage = resp.get("usage") or {}
+        self._log(
+            {
+                "kind": "judge",
+                "model": self.model,
+                "prompt_tokens": usage.get("prompt_tokens", 0),
+                "completion_tokens": usage.get("completion_tokens", 0),
+                "conversation": conversation,
+            }
+        )
+
+    def verdict(self, context: str, claim: str, conversation: str = "") -> str:
+        key = _verdict_key(self.model, context, claim)
+        with self._lock:
+            if key in self._vcache:
+                self._vcache.move_to_end(key)
+                return self._vcache[key]
+        msg = f"CONTEXT:\n{context}\n\nCLAIM:\n{claim}"
+        for max_tokens in (self.max_tokens, self.max_tokens * 2):
+            resp = self._call(msg, max_tokens=max_tokens)
+            self._account(resp, conversation)
+            content = resp["choices"][0]["message"].get("content")
+            try:
+                verdict = _parse_verdict(content or "")
+                self._store(key, verdict)
+                return verdict
+            except ValueError:
+                continue
+        verdict = "unverifiable"  # parse failure after 256 -> 2x tokens: conservative
+        self._store(key, verdict)
+        return verdict
+
+    def _store(self, key: str, verdict: str) -> None:
+        with self._lock:
+            self._vcache[key] = verdict
+            self._vcache.move_to_end(key)
+            while len(self._vcache) > self._vcache_max:
+                self._vcache.popitem(last=False)
+            if self.cache_path:
+                self.cache_path.parent.mkdir(parents=True, exist_ok=True)
+                with self.cache_path.open("a") as f:
+                    f.write(json.dumps({"key": key, "verdict": verdict}) + "\n")