ragsentry 0.1.0__tar.gz

ragsentry-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Omar Yasser
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ragsentry-0.1.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: ragsentry
+Version: 0.1.0
+Summary: Regression-testing and evaluation tool for RAG systems ('pytest for RAG')
+Author: RagSentry Team
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0
+Requires-Dist: click>=8.0
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: ragas>=0.1.0
+Requires-Dist: langchain-openai>=0.1.0
+Requires-Dist: langchain-anthropic>=0.1.0
+Requires-Dist: datasets>=2.14.0
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.0; extra == "postgres"
+Dynamic: license-file
+
+# RagSentry — pytest for RAG
+
+> **Regression testing and evaluation CLI for Retrieval-Augmented Generation (RAG) systems.**
+
+RagSentry connects to your RAG application through a thin adapter, evaluates retrieved contexts and generated answers against your dataset, tracks runs over time, diffs runs to catch regressions, and gates CI builds.
+
+---
+
+## 🚀 Quickstart in 5 Minutes
+
+### 1. Install RagSentry
+
+```bash
+pip install ragsentry
+```
+
+*(Optional PostgreSQL storage support: `pip install ragsentry[postgres]`)*
+
+---
+
+### 2. Initialize RagSentry in your project
+
+```bash
+ragsentry init
+```
+
+This scaffolds:
+- `ragsentry_adapter.py` — starter adapter stub
+- `evalset.jsonl` — starter evaluation set
+
+---
+
+### 3. Connect your RAG application
+
+Edit `ragsentry_adapter.py` to call your RAG pipeline and return the required contract:
+
+```python
+from my_rag_app import ask_rag
+
+def query(question: str) -> dict:
+    result = ask_rag(question)
+    return {
+        "answer": result["answer"],
+        "contexts": result["contexts"],  # list[str] of retrieved passages
+    }
+```
+
+---
+
+### 4. Run an Evaluation
+
+#### Unscored Generation Pass (No LLM required):
+```bash
+ragsentry run -e evalset.jsonl -a ragsentry_adapter:query --no-scoring
+```
+
+#### Full Evaluation with LLM Judge:
+```bash
+ragsentry run -e evalset.jsonl -a ragsentry_adapter:query -p openrouter -m openai/gpt-oss-120b
+```
+
+---
+
+### 5. Diff Runs & Gate CI
+
+#### Diff two evaluation runs:
+```bash
+ragsentry diff run_2026-09-27T10-00-00_a1b2c3d4.json run_2026-09-27T11-00-00_e5f6g7h8.json
+```
+
+#### Quality Gate in CI (Exit code 0 on pass, non-zero on failure):
+```bash
+ragsentry ci \
+  --candidate <candidate_run_id> \
+  --baseline <baseline_run_id> \
+  -t faithfulness=0.80 \
+  -t context_recall=0.85 \
+  --max-regressed-questions 1 \
+  --pr-comment-out pr_comment.md
+```
+
+---
+
+## ⚡ Key Features
+
+- 🔌 **Plug-and-Play Adapters:** Supports Python callables (`pkg.mod:func`), REST APIs (`http://localhost:8000/query`), or CLI tools (`shell:python rag_cli.py`).
+- 🤖 **Provider-Agnostic Judge Scoring:** Works with OpenAI, Anthropic, Gemini, Groq, xAI, Ollama, DeepSeek, OpenRouter, or custom OpenAI-compatible endpoints.
+- 💾 **Dual Storage Backends:** Local JSON file store or PostgreSQL (`--storage postgresql://...`).
+- 📊 **Rich Terminal Diffs:** Colorized diff tables highlighting improved, regressed, and unchanged questions per metric.
+- 🚦 **CI Quality Gates:** Strict threshold enforcement and regression limits with exportable GitHub PR Markdown summaries.
+- 🧩 **Agent-Usable Scaffolder Skill:** Includes `skills/adapter-scaffolder` so AI coding assistants can inspect unfamiliar RAG repos and scaffold adapters automatically.
+
+---
+
+## 📖 Documentation & Guides
+
+- 📘 [Adapter Contract Guide](docs/adapter-contract.md)
+- 📙 [CI / CD Integration Guide](docs/ci-integration.md)
+- 📗 [Configuration Reference](docs/config-reference.md)
+- 📕 [Adapter Scaffolder Skill](skills/adapter-scaffolder/SKILL.md)
+
+---
+
+## 📜 License
+
+MIT License.

ragsentry-0.1.0/README.md

@@ -0,0 +1,106 @@
+# RagSentry — pytest for RAG
+
+> **Regression testing and evaluation CLI for Retrieval-Augmented Generation (RAG) systems.**
+
+RagSentry connects to your RAG application through a thin adapter, evaluates retrieved contexts and generated answers against your dataset, tracks runs over time, diffs runs to catch regressions, and gates CI builds.
+
+---
+
+## 🚀 Quickstart in 5 Minutes
+
+### 1. Install RagSentry
+
+```bash
+pip install ragsentry
+```
+
+*(Optional PostgreSQL storage support: `pip install ragsentry[postgres]`)*
+
+---
+
+### 2. Initialize RagSentry in your project
+
+```bash
+ragsentry init
+```
+
+This scaffolds:
+- `ragsentry_adapter.py` — starter adapter stub
+- `evalset.jsonl` — starter evaluation set
+
+---
+
+### 3. Connect your RAG application
+
+Edit `ragsentry_adapter.py` to call your RAG pipeline and return the required contract:
+
+```python
+from my_rag_app import ask_rag
+
+def query(question: str) -> dict:
+    result = ask_rag(question)
+    return {
+        "answer": result["answer"],
+        "contexts": result["contexts"],  # list[str] of retrieved passages
+    }
+```
+
+---
+
+### 4. Run an Evaluation
+
+#### Unscored Generation Pass (No LLM required):
+```bash
+ragsentry run -e evalset.jsonl -a ragsentry_adapter:query --no-scoring
+```
+
+#### Full Evaluation with LLM Judge:
+```bash
+ragsentry run -e evalset.jsonl -a ragsentry_adapter:query -p openrouter -m openai/gpt-oss-120b
+```
+
+---
+
+### 5. Diff Runs & Gate CI
+
+#### Diff two evaluation runs:
+```bash
+ragsentry diff run_2026-09-27T10-00-00_a1b2c3d4.json run_2026-09-27T11-00-00_e5f6g7h8.json
+```
+
+#### Quality Gate in CI (Exit code 0 on pass, non-zero on failure):
+```bash
+ragsentry ci \
+  --candidate <candidate_run_id> \
+  --baseline <baseline_run_id> \
+  -t faithfulness=0.80 \
+  -t context_recall=0.85 \
+  --max-regressed-questions 1 \
+  --pr-comment-out pr_comment.md
+```
+
+---
+
+## ⚡ Key Features
+
+- 🔌 **Plug-and-Play Adapters:** Supports Python callables (`pkg.mod:func`), REST APIs (`http://localhost:8000/query`), or CLI tools (`shell:python rag_cli.py`).
+- 🤖 **Provider-Agnostic Judge Scoring:** Works with OpenAI, Anthropic, Gemini, Groq, xAI, Ollama, DeepSeek, OpenRouter, or custom OpenAI-compatible endpoints.
+- 💾 **Dual Storage Backends:** Local JSON file store or PostgreSQL (`--storage postgresql://...`).
+- 📊 **Rich Terminal Diffs:** Colorized diff tables highlighting improved, regressed, and unchanged questions per metric.
+- 🚦 **CI Quality Gates:** Strict threshold enforcement and regression limits with exportable GitHub PR Markdown summaries.
+- 🧩 **Agent-Usable Scaffolder Skill:** Includes `skills/adapter-scaffolder` so AI coding assistants can inspect unfamiliar RAG repos and scaffold adapters automatically.
+
+---
+
+## 📖 Documentation & Guides
+
+- 📘 [Adapter Contract Guide](docs/adapter-contract.md)
+- 📙 [CI / CD Integration Guide](docs/ci-integration.md)
+- 📗 [Configuration Reference](docs/config-reference.md)
+- 📕 [Adapter Scaffolder Skill](skills/adapter-scaffolder/SKILL.md)
+
+---
+
+## 📜 License
+
+MIT License.

ragsentry-0.1.0/pyproject.toml

@@ -0,0 +1,38 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ragsentry"
+version = "0.1.0"
+description = "Regression-testing and evaluation tool for RAG systems ('pytest for RAG')"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "RagSentry Team" }
+]
+dependencies = [
+    "pydantic>=2.0",
+    "click>=8.0",
+    "httpx>=0.25.0",
+    "ragas>=0.1.0",
+    "langchain-openai>=0.1.0",
+    "langchain-anthropic>=0.1.0",
+    "datasets>=2.14.0",
+]
+
+[project.optional-dependencies]
+test = [
+    "pytest>=7.0",
+]
+postgres = [
+    "psycopg[binary]>=3.0",
+]
+
+[project.scripts]
+ragsentry = "ragsentry.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["ragsentry*"]

ragsentry-0.1.0/ragsentry/__init__.py

@@ -0,0 +1 @@
+

ragsentry-0.1.0/ragsentry/__main__.py

@@ -0,0 +1,4 @@
+from ragsentry.cli import main
+
+if __name__ == "__main__":
+    main()

ragsentry-0.1.0/ragsentry/adapters/__init__.py

@@ -0,0 +1 @@
+

ragsentry-0.1.0/ragsentry/adapters/base.py

@@ -0,0 +1,32 @@
+from __future__ import annotations
+
+
+from dataclasses import dataclass, field
+from typing import Any, Protocol, runtime_checkable
+
+
+@dataclass
+class AdapterResponse:
+    """Standardized response structure returned by any RagSentry adapter."""
+
+    answer: str
+    contexts: list[str]
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+
+    def to_dict(self) -> dict[str, Any]:
+        return{
+            "answer": self.answer,
+            "contexts": self.contexts,
+            "metadata": self.metadata,
+        }
+
+
+
+@runtime_checkable
+class Adapter(Protocol):
+    """Protocol defining the adapter contract for external RAG systems."""
+
+    def query(self, question: str) -> AdapterResponse:
+        """Query the target RAG system with a question and return an AdapterResponse."""
+        ...

ragsentry-0.1.0/ragsentry/adapters/http.py

@@ -0,0 +1,106 @@
+from __future__ import annotations
+
+from typing import Any
+import httpx
+
+
+from ragsentry.adapters.base import Adapter, AdapterResponse
+
+
+def extract_path(data: Any, path: str) -> Any:
+    """Extract a nested value using dotted syntax like 'data.answer' or 'results.0.text'."""
+    if not path:
+        return data
+
+    current = data
+    for part in path.split("."):
+        if isinstance(current, dict):
+            current = current.get(part)
+
+        elif isinstance(current, (list, tuple)) and part.isdigit():
+            idx = int(part)
+            current = current[idx] if 0<= idx < len(current) else None
+
+        else:
+            return None
+        
+        if current is None:
+            break
+    return current
+
+
+
+
+class HttpAdapter(Adapter):
+    """Adapter that queries an external RAG service over HTTP."""
+
+    def __init__(
+        self, 
+        url: str,
+        method: str = "POST",
+        headers: dict[str, str] | None = None,
+        question_key: str = "question",
+        answer_path: str = "answer",
+        contexts_path: str = "contexts",
+        metadata_path: str | None = "metadata",
+        timeout: float =30.0
+    ) -> None:
+        self.url = url
+        self.method = method.upper()
+        self.headers = headers or {"Content-Type": "application/json"}
+        self.question_key = question_key
+        self.answer_path = answer_path
+        self.contexts_path = contexts_path
+        self.metadata_path = metadata_path
+        self.timeout = timeout
+
+
+    def query(self, question: str) -> AdapterResponse:
+        payload = {self.question_key: question}
+
+        with httpx.Client(timeout=self.timeout) as client:
+            if self.method == "POST":
+                response = client.post(self.url ,json=payload, headers=self.headers)
+            elif self.method == "GET":
+                response = client.get(self.url ,params=payload, headers=self.headers)
+
+            else:
+                raise ValueError(f"Unsupported HTTP method: {self.method}")
+
+            
+            response.raise_for_status()
+            data = response.json()
+
+        
+        #extract the answer
+        raw_answer = extract_path(data, self.answer_path)
+        if raw_answer is None:
+            raise ValueError(
+                f"Failed to extract answer using path '{self.answer_path}' from response: {data}"
+            )
+
+        #extract the contexts
+        raw_contexts = extract_path(data, self.contexts_path)
+        if raw_contexts is None:
+            raw_contexts = []
+        elif not isinstance(raw_contexts, list):
+            raw_contexts = [raw_contexts]
+
+        contexts = [
+            c if isinstance(c, str) else str(c.get("text", c)) if isinstance(c, dict) else str(c)
+            for c in raw_contexts
+        ]
+
+        #extract metadata
+        metadata = {}
+        if self.metadata_path:
+            raw_meta = extract_path(data, self.metadata_path)
+            if isinstance(raw_meta, dict):
+                metadata = raw_meta
+
+        
+        return AdapterResponse(
+            answer=str(raw_answer),
+            contexts=contexts,
+            metadata=metadata
+        )

ragsentry-0.1.0/ragsentry/adapters/python_callable.py

@@ -0,0 +1,74 @@
+from __future__ import annotations
+
+import importlib
+from typing import Any, Callable
+import os
+import sys
+
+from ragsentry.adapters.base import Adapter, AdapterResponse
+
+
+class PythonCallableAdapter(Adapter):
+    """Adapter that wraps an in-process Python callable."""
+
+
+    def __init__(self, target: Callable[[str], dict[str, Any] | AdapterResponse] | str) -> None:
+
+        if isinstance(target, str):
+            self._callable = self._load_callable(target)
+        elif callable(target):
+            self._callable = target
+
+        else:
+            raise TypeError(f"Target must be a callable or an import string ('module:func'), got {type(target)}")
+
+    
+
+    @staticmethod
+    def _load_callable(import_str: str) -> Callable[[str], Any]:
+        """Resolves an import string of format 'module.submodule:function_name'."""
+        if ":" not in import_str:
+            raise ValueError(
+                f"Invalid import string '{import_str}'. Expected format: 'package.module:function_name'"
+            )
+        cwd = os.getcwd()
+        if cwd not in sys.path:
+            sys.path.insert(0, cwd)
+            
+        module_path, func_name = import_str.split(":", 1)
+        module = importlib.import_module(module_path)
+        target = getattr(module, func_name)
+        if not callable(target):
+            raise TypeError(f"Resolved object '{import_str}' is not callable.")
+
+        return target
+
+
+    def query(self, question: str) -> AdapterResponse:
+
+        raw = self._callable(question)
+
+        if isinstance(raw, AdapterResponse):
+            return raw
+
+        if isinstance(raw, dict):
+            if "answer" not in raw or "contexts" not in raw:
+                raise ValueError(
+                    f"Callable returned a dict missing 'answer' or 'contexts': keys={list(raw.keys())}"
+                )
+            
+            contexts = raw["contexts"]
+            if not isinstance(contexts, list):
+                raise TypeError(f"'contexts' must be a list of strings, got {type(contexts)}")
+
+            
+            return AdapterResponse(
+                answer=str(raw["answer"]),
+                contexts=[str(c) for c in contexts],
+                metadata=raw.get("metadata", {}),
+            )
+
+
+        raise TypeError(
+            f"Expected callable to return dict or AdapterResponse, got {type(raw)}"
+        )

ragsentry-0.1.0/ragsentry/adapters/shell.py

@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+import json
+import shlex
+import subprocess
+from typing import Any
+
+
+from ragsentry.adapters.base import Adapter, AdapterResponse
+
+
+class ShellAdapter(Adapter):
+    """Adapter that executes an external shell command or script."""
+
+    def __init__(
+        self,
+        command: str | list [str],
+        pass_as: str = "stdin", #stdin or arg
+        timeout: float = 30.0,
+        cwd: str | None = None,
+    ) -> None:
+
+        if isinstance(command, str):
+            self.command = shlex.split(command, posix=False)
+
+        else:
+            self.command = list(command)
+
+        
+        self.pass_as = pass_as.lower()
+        self.timeout = timeout
+        self.cwd = cwd
+
+
+        if self.pass_as not in ("stdin", "arg"):
+            raise ValueError(f"pass_as must be 'stdin' or 'arg', got '{pass_as}'")
+
+    
+    def query(self, question: str) -> AdapterResponse:
+        cmd = list(self.command)
+        input_data = None
+
+        if self.pass_as == "arg":
+            cmd.append(question)
+
+        else:
+            input_data = json.dumps({"question": question})
+
+        proc = subprocess.run(
+            cmd,
+            input=input_data,
+            capture_output=True,
+            text=True,
+            timeout=self.timeout,
+            cwd=self.cwd,
+        )
+        
+        if proc.returncode != 0:
+            raise RuntimeError(
+                f"Shell command failed with exit code {proc.returncode}.\nStderr: {proc.stderr}"
+            )
+        stdout_clean = proc.stdout.strip()
+        if not stdout_clean:
+            raise ValueError(f"Shell command produced no stdout output.\nStderr: {proc.stderr}")
+        try:
+            data: dict[str, Any] = json.loads(stdout_clean)
+        except json.JSONDecodeError as exc:
+            raise ValueError(
+                f"Shell adapter expected JSON output on stdout, got: {stdout_clean}"
+            ) from exc
+        if "answer" not in data or "contexts" not in data:
+            raise ValueError(
+                f"Shell command output missing 'answer' or 'contexts'. Keys found: {list(data.keys())}"
+            )
+        contexts = data["contexts"]
+        if not isinstance(contexts, list):
+            raise TypeError(f"'contexts' in shell output must be a list, got {type(contexts)}")
+        
+        
+        return AdapterResponse(
+            answer=str(data["answer"]),
+            contexts=[str(c) for c in contexts],
+            metadata=data.get("metadata", {}),
+        )