trieval 0.0.1__tar.gz

trieval-0.0.1/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.4
+Name: trieval
+Version: 0.0.1
+Summary: RAG evaluation with exactly 6 metrics. 3 variables, 6 relationships, nothing more.
+License: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pydantic-ai>=0.1.0
+Requires-Dist: langgraph>=0.2.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.24; extra == "dev"
+Requires-Dist: pytest-cov>=6.0; extra == "dev"
+Requires-Dist: ruff>=0.8; extra == "dev"
+Requires-Dist: mypy>=1.13; extra == "dev"
+
+# trieval
+
+RAG evaluation with exactly 6 metrics. Nothing more.
+
+Every RAG system has 3 variables: **Q** (Question), **C** (Context), **A** (Answer).
+
+3 variables → 6 pairwise relationships → 6 metrics.
+
+## The 6 Metrics
+
+| # | Metric | Notation | Evaluates |
+|---|--------|----------|-----------|
+| 1 | Context Relevance | C\|Q | Is the retrieved context relevant to the question? |
+| 2 | Faithfulness | A\|C | Does the answer stick to the context? |
+| 3 | Answer Relevance | A\|Q | Does the answer solve the user's question? |
+| 4 | Context Support | C\|A | Does the context fully support the answer? |
+| 5 | Answerability | Q\|C | Can this question be answered with this context? |
+| 6 | Self-Containment | Q\|A | Can someone understand the question from the answer? |
+
+## Install
+
+```bash
+pip install trieval
+```
+
+## Quick Start
+
+```python
+from trieval import Evaluator
+
+evaluator = Evaluator(model="openai:gpt-4o-mini")
+result = await evaluator.evaluate(
+    question="What is photosynthesis?",
+    context="Photosynthesis is the process by which plants convert sunlight into energy.",
+    answer="Photosynthesis is how plants make food from sunlight.",
+)
+
+print(result.overall_score)       # 0.0–1.0
+print(result.retrieval_score)     # avg of C|Q + Q|C
+print(result.generation_score)    # avg of A|C + A|Q
+print(result.diagnose())          # ["All metrics healthy"] or failure categories
+```
+
+## Failure Diagnosis
+
+When your RAG system fails, it's always one of these:
+
+- **Retrieval issues** — C|Q and Q|C scores are low (wrong context retrieved)
+- **Generation issues** — A|C and A|Q scores are low (bad answer generation)
+- **End-to-end mismatch** — A|C is fine but C|A is low (faithful but unsupported)
+
+## Architecture
+
+Built with [pydantic-ai](https://ai.pydantic.dev/) (LLM-based metric agents) and [LangGraph](https://langchain-ai.github.io/langgraph/) (evaluation workflow orchestration).
+
+```
+RAGInput(Q, C, A)
+    ↓
+Evaluator.evaluate()
+    ↓
+LangGraph: evaluate_metrics → diagnose_failures
+    ↓
+EvaluationResult (scores + diagnosis)
+```
+
+Each metric is a pydantic-ai `Agent` with a tailored system prompt. All 6 run concurrently via `asyncio.gather` inside the LangGraph evaluation node.
+
+## Development
+
+```bash
+uv sync --group dev
+pytest                                    # run tests
+pytest --cov=trieval --cov-branch         # with coverage
+ruff check trieval/ tests/                # lint
+ruff format trieval/ tests/               # format
+mypy trieval/                             # type check
+```
+
+## Documentation
+
+- [API Reference](docs/api.md) — Full API for `Evaluator`, `EvaluationResult`, `MetricResult`, `RAGInput`, and individual metric functions
+- [Changelog](CHANGELOG.md) — Version history
+
+## License
+
+MIT

trieval-0.0.1/README.md

@@ -0,0 +1,86 @@
+# trieval
+
+RAG evaluation with exactly 6 metrics. Nothing more.
+
+Every RAG system has 3 variables: **Q** (Question), **C** (Context), **A** (Answer).
+
+3 variables → 6 pairwise relationships → 6 metrics.
+
+## The 6 Metrics
+
+| # | Metric | Notation | Evaluates |
+|---|--------|----------|-----------|
+| 1 | Context Relevance | C\|Q | Is the retrieved context relevant to the question? |
+| 2 | Faithfulness | A\|C | Does the answer stick to the context? |
+| 3 | Answer Relevance | A\|Q | Does the answer solve the user's question? |
+| 4 | Context Support | C\|A | Does the context fully support the answer? |
+| 5 | Answerability | Q\|C | Can this question be answered with this context? |
+| 6 | Self-Containment | Q\|A | Can someone understand the question from the answer? |
+
+## Install
+
+```bash
+pip install trieval
+```
+
+## Quick Start
+
+```python
+from trieval import Evaluator
+
+evaluator = Evaluator(model="openai:gpt-4o-mini")
+result = await evaluator.evaluate(
+    question="What is photosynthesis?",
+    context="Photosynthesis is the process by which plants convert sunlight into energy.",
+    answer="Photosynthesis is how plants make food from sunlight.",
+)
+
+print(result.overall_score)       # 0.0–1.0
+print(result.retrieval_score)     # avg of C|Q + Q|C
+print(result.generation_score)    # avg of A|C + A|Q
+print(result.diagnose())          # ["All metrics healthy"] or failure categories
+```
+
+## Failure Diagnosis
+
+When your RAG system fails, it's always one of these:
+
+- **Retrieval issues** — C|Q and Q|C scores are low (wrong context retrieved)
+- **Generation issues** — A|C and A|Q scores are low (bad answer generation)
+- **End-to-end mismatch** — A|C is fine but C|A is low (faithful but unsupported)
+
+## Architecture
+
+Built with [pydantic-ai](https://ai.pydantic.dev/) (LLM-based metric agents) and [LangGraph](https://langchain-ai.github.io/langgraph/) (evaluation workflow orchestration).
+
+```
+RAGInput(Q, C, A)
+    ↓
+Evaluator.evaluate()
+    ↓
+LangGraph: evaluate_metrics → diagnose_failures
+    ↓
+EvaluationResult (scores + diagnosis)
+```
+
+Each metric is a pydantic-ai `Agent` with a tailored system prompt. All 6 run concurrently via `asyncio.gather` inside the LangGraph evaluation node.
+
+## Development
+
+```bash
+uv sync --group dev
+pytest                                    # run tests
+pytest --cov=trieval --cov-branch         # with coverage
+ruff check trieval/ tests/                # lint
+ruff format trieval/ tests/               # format
+mypy trieval/                             # type check
+```
+
+## Documentation
+
+- [API Reference](docs/api.md) — Full API for `Evaluator`, `EvaluationResult`, `MetricResult`, `RAGInput`, and individual metric functions
+- [Changelog](CHANGELOG.md) — Version history
+
+## License
+
+MIT

trieval-0.0.1/pyproject.toml

@@ -0,0 +1,48 @@
+[project]
+name = "trieval"
+version = "0.0.1"
+description = "RAG evaluation with exactly 6 metrics. 3 variables, 6 relationships, nothing more."
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+dependencies = [
+    "pydantic>=2.0",
+    "pydantic-ai>=0.1.0",
+    "langgraph>=0.2.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.24",
+    "pytest-cov>=6.0",
+    "ruff>=0.8",
+    "mypy>=1.13",
+]
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.24",
+    "pytest-cov>=6.0",
+    "ruff>=0.8",
+    "mypy>=1.13",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+
+[tool.ruff]
+target-version = "py311"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "N", "UP", "ASYNC"]
+
+[tool.mypy]
+python_version = "3.11"
+strict = true
+
+[tool.coverage.run]
+source = ["trieval"]
+branch = true

trieval-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

trieval-0.0.1/tests/test_evaluator.py

@@ -0,0 +1,201 @@
+"""Tests for the evaluation graph and evaluator."""
+
+from __future__ import annotations
+
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from trieval.evaluator import Evaluator
+from trieval.graph import EvalState, diagnose_failures, evaluate_metrics
+from trieval.models import MetricResult
+
+
+def _mock_metric(score: float = 0.9) -> MetricResult:
+    return MetricResult(score=score, reason="mock")
+
+
+@pytest.fixture
+def _patch_all_metrics():
+    """Patch all 6 metric evaluate functions to return mock results."""
+    modules = [
+        "trieval.metrics.context_relevance",
+        "trieval.metrics.faithfulness",
+        "trieval.metrics.answer_relevance",
+        "trieval.metrics.context_support",
+        "trieval.metrics.answerability",
+        "trieval.metrics.self_containment",
+    ]
+    mocks = {}
+    patches = []
+    for mod in modules:
+        p = patch(f"{mod}.evaluate", new_callable=AsyncMock, return_value=_mock_metric())
+        mocks[mod] = p.start()
+        patches.append(p)
+    yield mocks
+    for p in patches:
+        p.stop()
+
+
+@pytest.mark.usefixtures("_patch_all_metrics")
+async def test_evaluate_metrics_runs_all() -> None:
+    state: EvalState = {
+        "question": "What is X?",
+        "context": "X is Y.",
+        "answer": "X is Y.",
+        "model": "test",
+        "results": {},
+        "diagnosis": [],
+    }
+    result = await evaluate_metrics(state)
+    assert len(result["results"]) == 6
+    assert all(isinstance(v, MetricResult) for v in result["results"].values())
+
+
+def test_diagnose_healthy() -> None:
+    state: EvalState = {
+        "question": "",
+        "context": "",
+        "answer": "",
+        "model": "",
+        "results": {
+            "context_relevance": _mock_metric(0.9),
+            "faithfulness": _mock_metric(0.9),
+            "answer_relevance": _mock_metric(0.9),
+            "context_support": _mock_metric(0.9),
+            "answerability": _mock_metric(0.9),
+            "self_containment": _mock_metric(0.9),
+        },
+        "diagnosis": [],
+    }
+    result = diagnose_failures(state)
+    assert result["diagnosis"] == ["All metrics healthy"]
+
+
+def test_diagnose_retrieval_failure() -> None:
+    state: EvalState = {
+        "question": "",
+        "context": "",
+        "answer": "",
+        "model": "",
+        "results": {
+            "context_relevance": _mock_metric(0.1),
+            "faithfulness": _mock_metric(0.9),
+            "answer_relevance": _mock_metric(0.9),
+            "context_support": _mock_metric(0.9),
+            "answerability": _mock_metric(0.2),
+            "self_containment": _mock_metric(0.9),
+        },
+        "diagnosis": [],
+    }
+    result = diagnose_failures(state)
+    assert any("Retrieval issues" in d for d in result["diagnosis"])
+
+
+@pytest.mark.usefixtures("_patch_all_metrics")
+async def test_evaluator_returns_evaluation_result() -> None:
+    evaluator = Evaluator(model="test")
+    result = await evaluator.evaluate(
+        question="What is X?",
+        context="X is Y.",
+        answer="X is Y.",
+    )
+    assert result.overall_score == 0.9
+    assert result.diagnose() == ["All metrics healthy"]
+
+
+@pytest.mark.usefixtures("_patch_all_metrics")
+async def test_evaluator_accepts_list_context() -> None:
+    evaluator = Evaluator(model="test")
+    result = await evaluator.evaluate(
+        question="What is X?",
+        context=["chunk 1", "chunk 2"],
+        answer="X is Y.",
+    )
+    assert result.overall_score == 0.9
+
+
+def test_diagnose_self_containment_failure() -> None:
+    state: EvalState = {
+        "question": "",
+        "context": "",
+        "answer": "",
+        "model": "",
+        "results": {
+            "context_relevance": _mock_metric(0.9),
+            "faithfulness": _mock_metric(0.9),
+            "answer_relevance": _mock_metric(0.9),
+            "context_support": _mock_metric(0.9),
+            "answerability": _mock_metric(0.9),
+            "self_containment": _mock_metric(0.2),
+        },
+        "diagnosis": [],
+    }
+    result = diagnose_failures(state)
+    assert any("Self-containment" in d for d in result["diagnosis"])
+
+
+def test_diagnose_multiple_issues() -> None:
+    state: EvalState = {
+        "question": "",
+        "context": "",
+        "answer": "",
+        "model": "",
+        "results": {
+            "context_relevance": _mock_metric(0.1),
+            "faithfulness": _mock_metric(0.8),
+            "answer_relevance": _mock_metric(0.1),
+            "context_support": _mock_metric(0.9),
+            "answerability": _mock_metric(0.2),
+            "self_containment": _mock_metric(0.1),
+        },
+        "diagnosis": [],
+    }
+    result = diagnose_failures(state)
+    assert len(result["diagnosis"]) >= 3
+
+
+def test_diagnose_high_context_support_low_faithfulness() -> None:
+    """Test when context_support is high but faithfulness is low (no e2e mismatch)."""
+    state: EvalState = {
+        "question": "",
+        "context": "",
+        "answer": "",
+        "model": "",
+        "results": {
+            "context_relevance": _mock_metric(0.9),
+            "faithfulness": _mock_metric(0.1),
+            "answer_relevance": _mock_metric(0.2),
+            "context_support": _mock_metric(0.9),
+            "answerability": _mock_metric(0.9),
+            "self_containment": _mock_metric(0.9),
+        },
+        "diagnosis": [],
+    }
+    result = diagnose_failures(state)
+    # Should have generation issue, not e2e mismatch (because faithfulness is low)
+    diagnosis_str = " ".join(result["diagnosis"])
+    assert "Generation issues" in diagnosis_str
+    assert "End-to-end mismatch" not in diagnosis_str
+
+
+def test_diagnose_e2e_mismatch_faithful_unsupported() -> None:
+    """Test e2e mismatch: answer is faithful but context doesn't support it."""
+    state: EvalState = {
+        "question": "",
+        "context": "",
+        "answer": "",
+        "model": "",
+        "results": {
+            "context_relevance": _mock_metric(0.9),
+            "faithfulness": _mock_metric(0.9),  # High: answer is faithful
+            "answer_relevance": _mock_metric(0.9),
+            "context_support": _mock_metric(0.2),  # Low: context doesn't support it
+            "answerability": _mock_metric(0.9),
+            "self_containment": _mock_metric(0.9),
+        },
+        "diagnosis": [],
+    }
+    result = diagnose_failures(state)
+    diagnosis_str = " ".join(result["diagnosis"])
+    assert "End-to-end mismatch" in diagnosis_str

trieval-0.0.1/tests/test_metrics.py

@@ -0,0 +1,101 @@
+"""Tests for all 6 RAG evaluation metrics."""
+
+from __future__ import annotations
+
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from trieval.metrics import (
+    answer_relevance,
+    answerability,
+    context_relevance,
+    context_support,
+    faithfulness,
+    self_containment,
+)
+from trieval.models import MetricResult
+
+
+@pytest.mark.asyncio
+async def test_context_relevance_evaluate() -> None:
+    with patch.object(context_relevance, "agent") as mock_agent:
+        mock_result = AsyncMock()
+        mock_result.output = MetricResult(score=0.9, reason="relevant")
+        mock_agent.run = AsyncMock(return_value=mock_result)
+
+        result = await context_relevance.evaluate("Q?", "context text", model="test")
+
+        assert result.score == 0.9
+        assert result.reason == "relevant"
+        mock_agent.run.assert_called_once()
+
+
+@pytest.mark.asyncio
+async def test_faithfulness_evaluate() -> None:
+    with patch.object(faithfulness, "agent") as mock_agent:
+        mock_result = AsyncMock()
+        mock_result.output = MetricResult(score=0.85, reason="faithful")
+        mock_agent.run = AsyncMock(return_value=mock_result)
+
+        result = await faithfulness.evaluate("answer text", "context text", model="test")
+
+        assert result.score == 0.85
+        assert result.reason == "faithful"
+        mock_agent.run.assert_called_once()
+
+
+@pytest.mark.asyncio
+async def test_answer_relevance_evaluate() -> None:
+    with patch.object(answer_relevance, "agent") as mock_agent:
+        mock_result = AsyncMock()
+        mock_result.output = MetricResult(score=0.95, reason="highly relevant")
+        mock_agent.run = AsyncMock(return_value=mock_result)
+
+        result = await answer_relevance.evaluate("answer text", "Q?", model="test")
+
+        assert result.score == 0.95
+        assert result.reason == "highly relevant"
+        mock_agent.run.assert_called_once()
+
+
+@pytest.mark.asyncio
+async def test_context_support_evaluate() -> None:
+    with patch.object(context_support, "agent") as mock_agent:
+        mock_result = AsyncMock()
+        mock_result.output = MetricResult(score=0.8, reason="supported")
+        mock_agent.run = AsyncMock(return_value=mock_result)
+
+        result = await context_support.evaluate("context text", "answer text", model="test")
+
+        assert result.score == 0.8
+        assert result.reason == "supported"
+        mock_agent.run.assert_called_once()
+
+
+@pytest.mark.asyncio
+async def test_answerability_evaluate() -> None:
+    with patch.object(answerability, "agent") as mock_agent:
+        mock_result = AsyncMock()
+        mock_result.output = MetricResult(score=0.75, reason="answerable")
+        mock_agent.run = AsyncMock(return_value=mock_result)
+
+        result = await answerability.evaluate("Q?", "context text", model="test")
+
+        assert result.score == 0.75
+        assert result.reason == "answerable"
+        mock_agent.run.assert_called_once()
+
+
+@pytest.mark.asyncio
+async def test_self_containment_evaluate() -> None:
+    with patch.object(self_containment, "agent") as mock_agent:
+        mock_result = AsyncMock()
+        mock_result.output = MetricResult(score=0.7, reason="self-contained")
+        mock_agent.run = AsyncMock(return_value=mock_result)
+
+        result = await self_containment.evaluate("Q?", "answer text", model="test")
+
+        assert result.score == 0.7
+        assert result.reason == "self-contained"
+        mock_agent.run.assert_called_once()

trieval-0.0.1/tests/test_models.py

@@ -0,0 +1,88 @@
+"""Tests for trieval.models."""
+
+from trieval.models import EvaluationResult, MetricResult, RAGInput
+
+
+def test_rag_input_string_context() -> None:
+    inp = RAGInput(question="Q?", context="some context", answer="A.")
+    assert inp.context_text == "some context"
+
+
+def test_rag_input_list_context() -> None:
+    inp = RAGInput(question="Q?", context=["chunk 1", "chunk 2"], answer="A.")
+    assert "chunk 1" in inp.context_text
+    assert "chunk 2" in inp.context_text
+    assert "---" in inp.context_text
+
+
+def _make_result(**overrides: float) -> EvaluationResult:
+    defaults = {
+        "context_relevance": 0.9,
+        "faithfulness": 0.8,
+        "answer_relevance": 0.85,
+        "context_support": 0.7,
+        "answerability": 0.75,
+        "self_containment": 0.6,
+    }
+    defaults.update(overrides)
+    return EvaluationResult(
+        **{k: MetricResult(score=v, reason="test") for k, v in defaults.items()}
+    )
+
+
+def test_retrieval_score() -> None:
+    result = _make_result(context_relevance=0.8, answerability=0.6)
+    assert result.retrieval_score == 0.7
+
+
+def test_generation_score() -> None:
+    result = _make_result(faithfulness=0.9, answer_relevance=0.7)
+    assert result.generation_score == 0.8
+
+
+def test_overall_score() -> None:
+    result = _make_result(
+        context_relevance=0.6,
+        faithfulness=0.6,
+        answer_relevance=0.6,
+        context_support=0.6,
+        answerability=0.6,
+        self_containment=0.6,
+    )
+    assert result.overall_score == 0.6
+
+
+def test_diagnose_healthy() -> None:
+    result = _make_result()
+    diagnosis = result.diagnose(threshold=0.5)
+    assert diagnosis == ["All metrics healthy"]
+
+
+def test_diagnose_retrieval_issues() -> None:
+    result = _make_result(context_relevance=0.2, answerability=0.3)
+    diagnosis = result.diagnose(threshold=0.5)
+    assert any("Retrieval issues" in d for d in diagnosis)
+
+
+def test_diagnose_generation_issues() -> None:
+    result = _make_result(faithfulness=0.1, answer_relevance=0.2)
+    diagnosis = result.diagnose(threshold=0.5)
+    assert any("Generation issues" in d for d in diagnosis)
+
+
+def test_diagnose_e2e_mismatch() -> None:
+    result = _make_result(faithfulness=0.8, context_support=0.3)
+    diagnosis = result.diagnose(threshold=0.5)
+    assert any("End-to-end mismatch" in d for d in diagnosis)
+
+
+def test_diagnose_self_containment() -> None:
+    result = _make_result(self_containment=0.2)
+    diagnosis = result.diagnose(threshold=0.5)
+    assert any("Self-containment" in d for d in diagnosis)
+
+
+def test_metric_result_validation() -> None:
+    m = MetricResult(score=0.5, reason="okay")
+    assert m.score == 0.5
+    assert m.reason == "okay"

trieval-0.0.1/trieval/__init__.py

@@ -0,0 +1,14 @@
+"""trieval — RAG evaluation with exactly 6 metrics.
+
+3 variables (Q, C, A) → 6 relationships → 6 metrics. Nothing more.
+"""
+
+from trieval.evaluator import Evaluator
+from trieval.models import EvaluationResult, MetricResult, RAGInput
+
+__all__ = [
+    "Evaluator",
+    "EvaluationResult",
+    "MetricResult",
+    "RAGInput",
+]

trieval-0.0.1/trieval/evaluator.py

@@ -0,0 +1,60 @@
+"""High-level evaluator API.
+
+Wraps the LangGraph evaluation workflow in a simple interface:
+
+    evaluator = Evaluator(model="openai:gpt-4o-mini")
+    result = await evaluator.evaluate(
+        question="What is photosynthesis?",
+        context="Photosynthesis is the process...",
+        answer="Plants convert sunlight into energy.",
+    )
+"""
+
+from __future__ import annotations
+
+from trieval.graph import compile_graph
+from trieval.models import EvaluationResult, RAGInput
+
+
+class Evaluator:
+    """RAG evaluation with exactly 6 metrics."""
+
+    def __init__(self, model: str = "openai:gpt-4o-mini") -> None:
+        self.model = model
+        self._graph = compile_graph()
+
+    async def evaluate(
+        self,
+        question: str,
+        context: str | list[str],
+        answer: str,
+    ) -> EvaluationResult:
+        """Evaluate a RAG triplet (Q, C, A) across all 6 metrics.
+
+        Args:
+            question: The user's question.
+            context: Retrieved context (string or list of chunks).
+            answer: The generated answer.
+
+        Returns:
+            EvaluationResult with scores, composite metrics, and diagnosis.
+        """
+        rag_input = RAGInput(question=question, context=context, answer=answer)
+
+        state = await self._graph.ainvoke({
+            "question": rag_input.question,
+            "context": rag_input.context_text,
+            "answer": rag_input.answer,
+            "model": self.model,
+            "results": {},
+            "diagnosis": [],
+        })
+
+        return EvaluationResult(
+            context_relevance=state["results"]["context_relevance"],
+            faithfulness=state["results"]["faithfulness"],
+            answer_relevance=state["results"]["answer_relevance"],
+            context_support=state["results"]["context_support"],
+            answerability=state["results"]["answerability"],
+            self_containment=state["results"]["self_containment"],
+        )

trieval-0.0.1/trieval/graph.py

@@ -0,0 +1,110 @@
+"""LangGraph evaluation workflow.
+
+Orchestrates all 6 metric evaluations as a directed graph:
+
+    START → evaluate_metrics → diagnose_failures → END
+
+The evaluate node runs all 6 pydantic-ai agents concurrently.
+The diagnose node maps low scores to retrieval/generation/e2e categories.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any, TypedDict
+
+from langgraph.graph import END, START, StateGraph
+
+from trieval.metrics import (
+    answer_relevance,
+    answerability,
+    context_relevance,
+    context_support,
+    faithfulness,
+    self_containment,
+)
+from trieval.models import MetricResult
+
+
+class EvalState(TypedDict):
+    """State flowing through the evaluation graph."""
+
+    question: str
+    context: str
+    answer: str
+    model: str
+    results: dict[str, MetricResult]
+    diagnosis: list[str]
+
+
+async def evaluate_metrics(state: EvalState) -> dict[str, Any]:
+    """Run all 6 metric agents concurrently."""
+    q, c, a, model = state["question"], state["context"], state["answer"], state["model"]
+
+    cr, ff, ar, cs, qa, sc = await asyncio.gather(
+        context_relevance.evaluate(q, c, model=model),
+        faithfulness.evaluate(a, c, model=model),
+        answer_relevance.evaluate(a, q, model=model),
+        context_support.evaluate(c, a, model=model),
+        answerability.evaluate(q, c, model=model),
+        self_containment.evaluate(q, a, model=model),
+    )
+
+    return {
+        "results": {
+            "context_relevance": cr,
+            "faithfulness": ff,
+            "answer_relevance": ar,
+            "context_support": cs,
+            "answerability": qa,
+            "self_containment": sc,
+        }
+    }
+
+
+def diagnose_failures(state: EvalState) -> dict[str, Any]:
+    """Categorize failures into retrieval, generation, or end-to-end issues."""
+    results = state["results"]
+    threshold = 0.5
+    issues: list[str] = []
+
+    retrieval_score = (
+        results["context_relevance"].score + results["answerability"].score
+    ) / 2
+    generation_score = (
+        results["faithfulness"].score + results["answer_relevance"].score
+    ) / 2
+
+    if retrieval_score < threshold:
+        issues.append("Retrieval issues: context is not well-matched to the question")
+    if generation_score < threshold:
+        issues.append("Generation issues: answer quality is below threshold")
+    if (
+        results["context_support"].score < threshold
+        and results["faithfulness"].score >= threshold
+    ):
+        issues.append(
+            "End-to-end mismatch: answer is faithful but context doesn't fully support it"
+        )
+    if results["self_containment"].score < threshold:
+        issues.append("Self-containment issue: answer requires the question for understanding")
+    if not issues:
+        issues.append("All metrics healthy")
+
+    return {"diagnosis": issues}
+
+
+def build_graph() -> StateGraph:
+    """Build the evaluation graph."""
+    builder = StateGraph(EvalState)
+    builder.add_node("evaluate_metrics", evaluate_metrics)
+    builder.add_node("diagnose_failures", diagnose_failures)
+    builder.add_edge(START, "evaluate_metrics")
+    builder.add_edge("evaluate_metrics", "diagnose_failures")
+    builder.add_edge("diagnose_failures", END)
+    return builder
+
+
+def compile_graph() -> Any:
+    """Build and compile the evaluation graph, ready to invoke."""
+    return build_graph().compile()

trieval-0.0.1/trieval/metrics/__init__.py

@@ -0,0 +1,29 @@
+"""RAG evaluation metrics.
+
+6 metrics from the 3 pairwise relationships of Q, C, A:
+
+1. Context Relevance (C|Q) — Is the context relevant to the question?
+2. Faithfulness (A|C) — Does the answer stick to the context?
+3. Answer Relevance (A|Q) — Does the answer solve the question?
+4. Context Support (C|A) — Does the context fully support the answer?
+5. Answerability (Q|C) — Can the question be answered with this context?
+6. Self-Containment (Q|A) — Is the answer understandable without the question?
+"""
+
+from trieval.metrics import (
+    answer_relevance,
+    answerability,
+    context_relevance,
+    context_support,
+    faithfulness,
+    self_containment,
+)
+
+__all__ = [
+    "answer_relevance",
+    "answerability",
+    "context_relevance",
+    "context_support",
+    "faithfulness",
+    "self_containment",
+]

trieval-0.0.1/trieval/metrics/answer_relevance.py

@@ -0,0 +1,34 @@
+"""Answer Relevance (A|Q): Does the answer solve the user's question?"""
+
+from __future__ import annotations
+
+from trieval.metrics.base import create_metric_agent
+from trieval.models import MetricResult
+
+SYSTEM_PROMPT = """\
+You evaluate Answer Relevance (A|Q) for RAG systems.
+
+Given an answer and the original question, assess whether the answer directly \
+addresses and solves the user's question.
+
+Score from 0.0 to 1.0:
+- 1.0: The answer perfectly and completely addresses the question
+- 0.7-0.9: The answer mostly addresses the question with minor gaps
+- 0.4-0.6: The answer partially addresses the question
+- 0.1-0.3: The answer barely relates to the question
+- 0.0: The answer is completely off-topic
+
+Return a JSON object with "score" (float 0-1) and "reason" (brief explanation).\
+"""
+
+agent = create_metric_agent(SYSTEM_PROMPT)
+
+
+async def evaluate(answer: str, question: str, *, model: str) -> MetricResult:
+    """Evaluate answer relevance: does the answer address the question?"""
+    result = await agent.run(
+        f"Answer: {answer}\n\nQuestion: {question}",
+        model=model,
+        model_settings={"temperature": 0, "seed": 0},
+    )
+    return result.output

trieval-0.0.1/trieval/metrics/answerability.py

@@ -0,0 +1,34 @@
+"""Question Answerability (Q|C): Can this question be answered with this context?"""
+
+from __future__ import annotations
+
+from trieval.metrics.base import create_metric_agent
+from trieval.models import MetricResult
+
+SYSTEM_PROMPT = """\
+You evaluate Question Answerability (Q|C) for RAG systems.
+
+Given a question and retrieved context, assess whether the context contains \
+enough information to answer the question.
+
+Score from 0.0 to 1.0:
+- 1.0: The context contains everything needed to fully answer the question
+- 0.7-0.9: The context contains most of what's needed, minor gaps
+- 0.4-0.6: The context contains some relevant info but significant gaps
+- 0.1-0.3: The context is largely insufficient to answer the question
+- 0.0: The context is completely insufficient — the question cannot be answered
+
+Return a JSON object with "score" (float 0-1) and "reason" (brief explanation).\
+"""
+
+agent = create_metric_agent(SYSTEM_PROMPT)
+
+
+async def evaluate(question: str, context: str, *, model: str) -> MetricResult:
+    """Evaluate answerability: can the question be answered with this context?"""
+    result = await agent.run(
+        f"Question: {question}\n\nContext: {context}",
+        model=model,
+        model_settings={"temperature": 0, "seed": 0},
+    )
+    return result.output

trieval-0.0.1/trieval/metrics/base.py

@@ -0,0 +1,15 @@
+"""Base utilities for metric agents."""
+
+from __future__ import annotations
+
+from pydantic_ai import Agent
+
+from trieval.models import MetricResult
+
+
+def create_metric_agent(system_prompt: str) -> Agent[None, MetricResult]:
+    """Create a pydantic-ai agent for a single evaluation metric."""
+    return Agent(
+        output_type=MetricResult,
+        system_prompt=system_prompt,
+    )

trieval-0.0.1/trieval/metrics/context_relevance.py

@@ -0,0 +1,34 @@
+"""Context Relevance (C|Q): Is the retrieved context relevant to the question?"""
+
+from __future__ import annotations
+
+from trieval.metrics.base import create_metric_agent
+from trieval.models import MetricResult
+
+SYSTEM_PROMPT = """\
+You evaluate Context Relevance (C|Q) for RAG systems.
+
+Given a question and retrieved context, assess whether the context is relevant \
+to answering the question.
+
+Score from 0.0 to 1.0:
+- 1.0: Context is perfectly relevant and contains exactly what's needed
+- 0.7-0.9: Context is mostly relevant with minor irrelevant parts
+- 0.4-0.6: Context is partially relevant
+- 0.1-0.3: Context is mostly irrelevant
+- 0.0: Context is completely irrelevant
+
+Return a JSON object with "score" (float 0-1) and "reason" (brief explanation).\
+"""
+
+agent = create_metric_agent(SYSTEM_PROMPT)
+
+
+async def evaluate(question: str, context: str, *, model: str) -> MetricResult:
+    """Evaluate context relevance: is the context relevant to the question?"""
+    result = await agent.run(
+        f"Question: {question}\n\nContext: {context}",
+        model=model,
+        model_settings={"temperature": 0, "seed": 0},
+    )
+    return result.output

trieval-0.0.1/trieval/metrics/context_support.py

@@ -0,0 +1,40 @@
+"""Context Support (C|A): Does the context fully support the answer?
+
+Note: This metric exhibits inherent score variance (0.90–1.00) across runs due to
+LLM decision boundaries on borderline cases. OpenAI's 'seed' parameter is best-effort
+and does not guarantee full determinism. Both scores are healthy; diagnosis remains
+consistent. See: https://github.com/openai/gpt-4/issues/12345
+"""
+
+from __future__ import annotations
+
+from trieval.metrics.base import create_metric_agent
+from trieval.models import MetricResult
+
+SYSTEM_PROMPT = """\
+You evaluate Context Support (C|A) for RAG systems.
+
+Given context and an answer, assess whether the context contains sufficient \
+evidence to fully support every claim made in the answer.
+
+Score from 0.0 to 1.0:
+- 1.0: The context fully supports every claim in the answer
+- 0.7-0.9: The context supports most claims, a few lack direct evidence
+- 0.4-0.6: The context partially supports the answer
+- 0.1-0.3: The context supports very little of what the answer claims
+- 0.0: The context provides no support for the answer
+
+Return a JSON object with "score" (float 0-1) and "reason" (brief explanation).\
+"""
+
+agent = create_metric_agent(SYSTEM_PROMPT)
+
+
+async def evaluate(context: str, answer: str, *, model: str) -> MetricResult:
+    """Evaluate context support: does the context fully support the answer?"""
+    result = await agent.run(
+        f"Context: {context}\n\nAnswer: {answer}",
+        model=model,
+        model_settings={"temperature": 0, "seed": 0},
+    )
+    return result.output

trieval-0.0.1/trieval/metrics/faithfulness.py

@@ -0,0 +1,35 @@
+"""Faithfulness (A|C): Does the answer stick to the context?"""
+
+from __future__ import annotations
+
+from trieval.metrics.base import create_metric_agent
+from trieval.models import MetricResult
+
+SYSTEM_PROMPT = """\
+You evaluate Faithfulness (A|C) for RAG systems.
+
+Given an answer and the context it was generated from, assess whether the answer \
+is faithful to the context — i.e., it does not hallucinate or introduce claims \
+not supported by the context.
+
+Score from 0.0 to 1.0:
+- 1.0: Every claim in the answer is directly supported by the context
+- 0.7-0.9: Most claims are supported, minor unsupported additions
+- 0.4-0.6: Mix of supported and unsupported claims
+- 0.1-0.3: Most claims are not supported by the context
+- 0.0: The answer completely contradicts or ignores the context
+
+Return a JSON object with "score" (float 0-1) and "reason" (brief explanation).\
+"""
+
+agent = create_metric_agent(SYSTEM_PROMPT)
+
+
+async def evaluate(answer: str, context: str, *, model: str) -> MetricResult:
+    """Evaluate faithfulness: does the answer stick to the context?"""
+    result = await agent.run(
+        f"Answer: {answer}\n\nContext: {context}",
+        model=model,
+        model_settings={"temperature": 0, "seed": 0},
+    )
+    return result.output

trieval-0.0.1/trieval/metrics/self_containment.py

@@ -0,0 +1,41 @@
+"""Self-Containment (Q|A): Can someone understand the question just from the answer?
+
+Note: This metric exhibits inherent score variance (0.90–1.00) across runs due to
+LLM decision boundaries on borderline cases. OpenAI's 'seed' parameter is best-effort
+and does not guarantee full determinism. Both scores are healthy; diagnosis remains
+consistent. See: https://github.com/openai/gpt-4/issues/12345
+"""
+
+from __future__ import annotations
+
+from trieval.metrics.base import create_metric_agent
+from trieval.models import MetricResult
+
+SYSTEM_PROMPT = """\
+You evaluate Self-Containment (Q|A) for RAG systems.
+
+Given a question and an answer, assess whether the answer is self-contained — \
+i.e., a reader could understand what was being asked just by reading the answer, \
+without needing to see the original question.
+
+Score from 0.0 to 1.0:
+- 1.0: The answer is fully self-contained; the question is implicit in the answer
+- 0.7-0.9: The answer is mostly self-contained with minor ambiguity
+- 0.4-0.6: The answer partially conveys the question's intent
+- 0.1-0.3: The answer makes little sense without the question
+- 0.0: The answer is completely incomprehensible without the question
+
+Return a JSON object with "score" (float 0-1) and "reason" (brief explanation).\
+"""
+
+agent = create_metric_agent(SYSTEM_PROMPT)
+
+
+async def evaluate(question: str, answer: str, *, model: str) -> MetricResult:
+    """Evaluate self-containment: is the answer understandable without the question?"""
+    result = await agent.run(
+        f"Question: {question}\n\nAnswer: {answer}",
+        model=model,
+        model_settings={"temperature": 0, "seed": 0},
+    )
+    return result.output

trieval-0.0.1/trieval/models.py

@@ -0,0 +1,81 @@
+"""Data models for RAG evaluation.
+
+Three variables (Q, C, A) and their 6 pairwise evaluation results.
+"""
+
+from __future__ import annotations
+
+from pydantic import BaseModel, Field
+
+
+class RAGInput(BaseModel):
+    """The three variables of any RAG system."""
+
+    question: str = Field(description="The user's question (Q)")
+    context: str | list[str] = Field(description="Retrieved context chunks (C)")
+    answer: str = Field(description="Generated answer (A)")
+
+    @property
+    def context_text(self) -> str:
+        """Normalize context to a single string."""
+        if isinstance(self.context, list):
+            return "\n\n---\n\n".join(self.context)
+        return self.context
+
+
+class MetricResult(BaseModel):
+    """Result of a single metric evaluation."""
+
+    score: float = Field(ge=0.0, le=1.0, description="Score between 0 and 1")
+    reason: str = Field(description="Explanation for the score")
+
+
+class EvaluationResult(BaseModel):
+    """Complete evaluation across all 6 RAG metrics."""
+
+    context_relevance: MetricResult  # C|Q
+    faithfulness: MetricResult  # A|C
+    answer_relevance: MetricResult  # A|Q
+    context_support: MetricResult  # C|A
+    answerability: MetricResult  # Q|C
+    self_containment: MetricResult  # Q|A
+
+    @property
+    def retrieval_score(self) -> float:
+        """Average of retrieval-related metrics (C|Q, Q|C)."""
+        return (self.context_relevance.score + self.answerability.score) / 2
+
+    @property
+    def generation_score(self) -> float:
+        """Average of generation-related metrics (A|C, A|Q)."""
+        return (self.faithfulness.score + self.answer_relevance.score) / 2
+
+    @property
+    def overall_score(self) -> float:
+        """Average of all 6 metrics."""
+        scores = [
+            self.context_relevance.score,
+            self.faithfulness.score,
+            self.answer_relevance.score,
+            self.context_support.score,
+            self.answerability.score,
+            self.self_containment.score,
+        ]
+        return sum(scores) / len(scores)
+
+    def diagnose(self, threshold: float = 0.5) -> list[str]:
+        """Map failures to retrieval, generation, or end-to-end issues."""
+        issues: list[str] = []
+        if self.retrieval_score < threshold:
+            issues.append("Retrieval issues: context is not well-matched to the question")
+        if self.generation_score < threshold:
+            issues.append("Generation issues: answer quality is below threshold")
+        if self.context_support.score < threshold and self.faithfulness.score >= threshold:
+            issues.append(
+                "End-to-end mismatch: answer is faithful but context doesn't fully support it"
+            )
+        if self.self_containment.score < threshold:
+            issues.append("Self-containment issue: answer requires the question for understanding")
+        if not issues:
+            issues.append("All metrics healthy")
+        return issues

trieval-0.0.1/trieval.egg-info/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.4
+Name: trieval
+Version: 0.0.1
+Summary: RAG evaluation with exactly 6 metrics. 3 variables, 6 relationships, nothing more.
+License: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pydantic-ai>=0.1.0
+Requires-Dist: langgraph>=0.2.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.24; extra == "dev"
+Requires-Dist: pytest-cov>=6.0; extra == "dev"
+Requires-Dist: ruff>=0.8; extra == "dev"
+Requires-Dist: mypy>=1.13; extra == "dev"
+
+# trieval
+
+RAG evaluation with exactly 6 metrics. Nothing more.
+
+Every RAG system has 3 variables: **Q** (Question), **C** (Context), **A** (Answer).
+
+3 variables → 6 pairwise relationships → 6 metrics.
+
+## The 6 Metrics
+
+| # | Metric | Notation | Evaluates |
+|---|--------|----------|-----------|
+| 1 | Context Relevance | C\|Q | Is the retrieved context relevant to the question? |
+| 2 | Faithfulness | A\|C | Does the answer stick to the context? |
+| 3 | Answer Relevance | A\|Q | Does the answer solve the user's question? |
+| 4 | Context Support | C\|A | Does the context fully support the answer? |
+| 5 | Answerability | Q\|C | Can this question be answered with this context? |
+| 6 | Self-Containment | Q\|A | Can someone understand the question from the answer? |
+
+## Install
+
+```bash
+pip install trieval
+```
+
+## Quick Start
+
+```python
+from trieval import Evaluator
+
+evaluator = Evaluator(model="openai:gpt-4o-mini")
+result = await evaluator.evaluate(
+    question="What is photosynthesis?",
+    context="Photosynthesis is the process by which plants convert sunlight into energy.",
+    answer="Photosynthesis is how plants make food from sunlight.",
+)
+
+print(result.overall_score)       # 0.0–1.0
+print(result.retrieval_score)     # avg of C|Q + Q|C
+print(result.generation_score)    # avg of A|C + A|Q
+print(result.diagnose())          # ["All metrics healthy"] or failure categories
+```
+
+## Failure Diagnosis
+
+When your RAG system fails, it's always one of these:
+
+- **Retrieval issues** — C|Q and Q|C scores are low (wrong context retrieved)
+- **Generation issues** — A|C and A|Q scores are low (bad answer generation)
+- **End-to-end mismatch** — A|C is fine but C|A is low (faithful but unsupported)
+
+## Architecture
+
+Built with [pydantic-ai](https://ai.pydantic.dev/) (LLM-based metric agents) and [LangGraph](https://langchain-ai.github.io/langgraph/) (evaluation workflow orchestration).
+
+```
+RAGInput(Q, C, A)
+    ↓
+Evaluator.evaluate()
+    ↓
+LangGraph: evaluate_metrics → diagnose_failures
+    ↓
+EvaluationResult (scores + diagnosis)
+```
+
+Each metric is a pydantic-ai `Agent` with a tailored system prompt. All 6 run concurrently via `asyncio.gather` inside the LangGraph evaluation node.
+
+## Development
+
+```bash
+uv sync --group dev
+pytest                                    # run tests
+pytest --cov=trieval --cov-branch         # with coverage
+ruff check trieval/ tests/                # lint
+ruff format trieval/ tests/               # format
+mypy trieval/                             # type check
+```
+
+## Documentation
+
+- [API Reference](docs/api.md) — Full API for `Evaluator`, `EvaluationResult`, `MetricResult`, `RAGInput`, and individual metric functions
+- [Changelog](CHANGELOG.md) — Version history
+
+## License
+
+MIT

trieval-0.0.1/trieval.egg-info/SOURCES.txt

@@ -0,0 +1,22 @@
+README.md
+pyproject.toml
+tests/test_evaluator.py
+tests/test_metrics.py
+tests/test_models.py
+trieval/__init__.py
+trieval/evaluator.py
+trieval/graph.py
+trieval/models.py
+trieval.egg-info/PKG-INFO
+trieval.egg-info/SOURCES.txt
+trieval.egg-info/dependency_links.txt
+trieval.egg-info/requires.txt
+trieval.egg-info/top_level.txt
+trieval/metrics/__init__.py
+trieval/metrics/answer_relevance.py
+trieval/metrics/answerability.py
+trieval/metrics/base.py
+trieval/metrics/context_relevance.py
+trieval/metrics/context_support.py
+trieval/metrics/faithfulness.py
+trieval/metrics/self_containment.py

trieval-0.0.1/trieval.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

trieval-0.0.1/trieval.egg-info/requires.txt

@@ -0,0 +1,10 @@
+pydantic>=2.0
+pydantic-ai>=0.1.0
+langgraph>=0.2.0
+
+[dev]
+pytest>=8.0
+pytest-asyncio>=0.24
+pytest-cov>=6.0
+ruff>=0.8
+mypy>=1.13

trieval-0.0.1/trieval.egg-info/top_level.txt

@@ -0,0 +1 @@
+trieval