llm-eval-exporter 0.2.0__tar.gz

llm_eval_exporter-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Harshitha
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_eval_exporter-0.2.0/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: llm-eval-exporter
+Version: 0.2.0
+Summary: A lightweight Prometheus exporter for LLM eval metrics: faithfulness, semantic drift, cost, and CI/CD regression gating.
+Author-email: Harshitha <your-email@example.com>
+License: MIT License
+        
+        Copyright (c) 2026 Harshitha
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+Project-URL: Homepage, https://github.com/your-username/llm-eval-exporter
+Project-URL: Repository, https://github.com/your-username/llm-eval-exporter
+Project-URL: Issues, https://github.com/your-username/llm-eval-exporter/issues
+Keywords: llm,observability,prometheus,eval,mlops,monitoring,grafana,ci-cd
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: prometheus-client>=0.20.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=2.2.0; extra == "embeddings"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.30.0; extra == "anthropic"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Dynamic: license-file
+
+# llm-eval-exporter
+
+A lightweight Prometheus exporter for LLM eval metrics — hallucination
+risk (via LLM-as-judge faithfulness scoring), semantic drift across
+model/prompt versions, latency, and token cost — designed to plug into
+observability stacks teams already run, instead of requiring adoption
+of a new platform.
+
+## Why this exists
+
+Most LLM observability tools (Langfuse, Braintrust, Arize Phoenix,
+etc.) are full platforms: their own UI, their own storage, their own
+workflow. That's the right call for teams building observability from
+scratch. But plenty of teams already run Prometheus + Grafana for
+every other service they operate, and would rather their LLM calls
+show up as three more panels on an existing dashboard than as a
+separate tool with a separate login.
+
+`llm-eval-exporter` wraps your LLM calls, computes eval scores, and
+exposes them as standard Prometheus metrics. It's a library, not a
+platform — a few lines of Python, one `/metrics` endpoint, no new UI
+to learn.
+
+## What it tracks
+
+| Metric | What it tells you |
+|---|---|
+| `llm_request_latency_seconds` | p50/p95/p99 latency per model/provider |
+| `llm_request_total` | Request volume and error rate |
+| `llm_tokens_total` | Prompt/completion token usage (cost proxy) |
+| `llm_faithfulness_score` | LLM-judge score (0-1) for how well a response sticks to its provided context — a hallucination proxy |
+| `llm_semantic_drift` | Cosine distance between a response and its recorded baseline for the same prompt — catches silent behavior changes across model upgrades or prompt edits |
+| `llm_eval_failures_total` | Count of responses that failed a faithfulness or drift threshold |
+
+## Quickstart
+
+```bash
+pip install -r requirements.txt
+python examples/demo.py
+# metrics now live at http://localhost:9464/metrics
+```
+
+To run against the real Anthropic API instead of mocks:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+python examples/anthropic_example.py
+```
+
+This traces a real `claude-sonnet-5` call and judges its faithfulness
+with a cheaper `claude-haiku-4-5` call — the standard cost-effective
+pattern for LLM-as-judge in production (you don't need your most
+expensive model just to check "did this answer stick to the
+context").
+
+Then, optionally, bring up Prometheus + Grafana pointed at it:
+
+```bash
+docker compose -f examples/docker-compose.yml up
+# Grafana at http://localhost:3000 (anonymous admin access, demo only)
+```
+
+## Usage in your own code
+
+```python
+from llm_eval_exporter import LLMTracker, start_metrics_server
+
+start_metrics_server(port=9464)  # call once at app startup
+
+tracker = LLMTracker(
+    judge_fn=my_judge_fn,   # any callable: str prompt -> str response
+    embed_fn=my_embed_fn,   # optional: defaults to sentence-transformers locally
+)
+
+with tracker.track(
+    model="gpt-4o",
+    provider="openai",
+    prompt_id="refund_policy_qa",  # stable id -> enables drift tracking
+    context=retrieved_context,     # enables faithfulness scoring
+) as call:
+    response = your_llm_client.call(prompt)
+    call.set_response(
+        response.text,
+        prompt_tokens=response.usage.prompt_tokens,
+        completion_tokens=response.usage.completion_tokens,
+    )
+```
+
+`judge_fn` and `embed_fn` are intentionally pluggable — bring your own
+LLM client (Anthropic, OpenAI, a local model) rather than depending on
+one vendor's SDK.
+
+## Design notes
+
+- **SQLite by default** for baseline embeddings and eval logs — a
+  single-file, zero-ops dependency, matching this project's "drop it
+  into an existing stack" philosophy. Swap for Postgres if it needs to
+  scale past one instance.
+- **Faithfulness scoring is a lightweight, single-call analogue of
+  RAGAS-style faithfulness** — not a claim to replace a dedicated eval
+  framework, just enough signal to catch regressions cheaply.
+- **Drift detection needs a `prompt_id`** you assign to prompts you
+  care about tracking over time; the first call for a given id just
+  records a baseline, so drift shows up starting from the second call.
+
+## Status
+
+Early-stage side project — built to explore a gap in the current LLM
+observability tooling landscape (see `docs/market-notes.md` for the
+reasoning), not a production-ready alternative to the platforms above.
+Contributions and issues welcome.
+
+## Running tests
+
+```bash
+pip install -r requirements.txt
+pytest tests/ -v
+```

llm_eval_exporter-0.2.0/README.md

@@ -0,0 +1,118 @@
+# llm-eval-exporter
+
+A lightweight Prometheus exporter for LLM eval metrics — hallucination
+risk (via LLM-as-judge faithfulness scoring), semantic drift across
+model/prompt versions, latency, and token cost — designed to plug into
+observability stacks teams already run, instead of requiring adoption
+of a new platform.
+
+## Why this exists
+
+Most LLM observability tools (Langfuse, Braintrust, Arize Phoenix,
+etc.) are full platforms: their own UI, their own storage, their own
+workflow. That's the right call for teams building observability from
+scratch. But plenty of teams already run Prometheus + Grafana for
+every other service they operate, and would rather their LLM calls
+show up as three more panels on an existing dashboard than as a
+separate tool with a separate login.
+
+`llm-eval-exporter` wraps your LLM calls, computes eval scores, and
+exposes them as standard Prometheus metrics. It's a library, not a
+platform — a few lines of Python, one `/metrics` endpoint, no new UI
+to learn.
+
+## What it tracks
+
+| Metric | What it tells you |
+|---|---|
+| `llm_request_latency_seconds` | p50/p95/p99 latency per model/provider |
+| `llm_request_total` | Request volume and error rate |
+| `llm_tokens_total` | Prompt/completion token usage (cost proxy) |
+| `llm_faithfulness_score` | LLM-judge score (0-1) for how well a response sticks to its provided context — a hallucination proxy |
+| `llm_semantic_drift` | Cosine distance between a response and its recorded baseline for the same prompt — catches silent behavior changes across model upgrades or prompt edits |
+| `llm_eval_failures_total` | Count of responses that failed a faithfulness or drift threshold |
+
+## Quickstart
+
+```bash
+pip install -r requirements.txt
+python examples/demo.py
+# metrics now live at http://localhost:9464/metrics
+```
+
+To run against the real Anthropic API instead of mocks:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+python examples/anthropic_example.py
+```
+
+This traces a real `claude-sonnet-5` call and judges its faithfulness
+with a cheaper `claude-haiku-4-5` call — the standard cost-effective
+pattern for LLM-as-judge in production (you don't need your most
+expensive model just to check "did this answer stick to the
+context").
+
+Then, optionally, bring up Prometheus + Grafana pointed at it:
+
+```bash
+docker compose -f examples/docker-compose.yml up
+# Grafana at http://localhost:3000 (anonymous admin access, demo only)
+```
+
+## Usage in your own code
+
+```python
+from llm_eval_exporter import LLMTracker, start_metrics_server
+
+start_metrics_server(port=9464)  # call once at app startup
+
+tracker = LLMTracker(
+    judge_fn=my_judge_fn,   # any callable: str prompt -> str response
+    embed_fn=my_embed_fn,   # optional: defaults to sentence-transformers locally
+)
+
+with tracker.track(
+    model="gpt-4o",
+    provider="openai",
+    prompt_id="refund_policy_qa",  # stable id -> enables drift tracking
+    context=retrieved_context,     # enables faithfulness scoring
+) as call:
+    response = your_llm_client.call(prompt)
+    call.set_response(
+        response.text,
+        prompt_tokens=response.usage.prompt_tokens,
+        completion_tokens=response.usage.completion_tokens,
+    )
+```
+
+`judge_fn` and `embed_fn` are intentionally pluggable — bring your own
+LLM client (Anthropic, OpenAI, a local model) rather than depending on
+one vendor's SDK.
+
+## Design notes
+
+- **SQLite by default** for baseline embeddings and eval logs — a
+  single-file, zero-ops dependency, matching this project's "drop it
+  into an existing stack" philosophy. Swap for Postgres if it needs to
+  scale past one instance.
+- **Faithfulness scoring is a lightweight, single-call analogue of
+  RAGAS-style faithfulness** — not a claim to replace a dedicated eval
+  framework, just enough signal to catch regressions cheaply.
+- **Drift detection needs a `prompt_id`** you assign to prompts you
+  care about tracking over time; the first call for a given id just
+  records a baseline, so drift shows up starting from the second call.
+
+## Status
+
+Early-stage side project — built to explore a gap in the current LLM
+observability tooling landscape (see `docs/market-notes.md` for the
+reasoning), not a production-ready alternative to the platforms above.
+Contributions and issues welcome.
+
+## Running tests
+
+```bash
+pip install -r requirements.txt
+pytest tests/ -v
+```

llm_eval_exporter-0.2.0/llm_eval_exporter/__init__.py

@@ -0,0 +1,5 @@
+from .tracker import LLMTracker
+from .server import start_metrics_server
+
+__all__ = ["LLMTracker", "start_metrics_server"]
+__version__ = "0.1.0"

llm_eval_exporter-0.2.0/llm_eval_exporter/alerting.py

@@ -0,0 +1,29 @@
+"""
+Optional webhook alerting (Slack-compatible JSON payload) fired when an
+eval check fails. Uses only the standard library (urllib) so this
+doesn't add a new dependency just for an optional feature.
+
+Best-effort by design: a failed alert never raises and never blocks
+the tracked call. Observability tooling should never be able to break
+the thing it's observing.
+"""
+
+import json
+import urllib.request
+import urllib.error
+
+
+def send_webhook_alert(webhook_url: str, message: str) -> bool:
+    if not webhook_url:
+        return False
+
+    payload = json.dumps({"text": message}).encode("utf-8")
+    req = urllib.request.Request(
+        webhook_url, data=payload, headers={"Content-Type": "application/json"}
+    )
+    try:
+        urllib.request.urlopen(req, timeout=5)
+        return True
+    except (urllib.error.URLError, urllib.error.HTTPError) as e:
+        print(f"llm-eval-exporter: webhook alert failed: {e}")
+        return False

llm_eval_exporter-0.2.0/llm_eval_exporter/cli.py

@@ -0,0 +1,130 @@
+"""
+CLI entrypoint for CI/CD regression testing.
+
+Usage:
+    llm-eval-exporter check --config eval-suite.yaml --adapter my_project.eval_adapter
+
+Runs a fixed set of prompts through your LLM, scores faithfulness and
+semantic drift, and exits non-zero if any score regresses past your
+configured thresholds. Designed to run as a CI step so a prompt change,
+model swap, or provider upgrade that silently degrades answer quality
+fails the build instead of shipping -- the same "shift left" idea as
+unit tests, applied to LLM behavior.
+
+The --db path should persist across CI runs (e.g. via actions/cache)
+so drift has a real baseline to compare against, not a fresh one every run.
+"""
+
+import argparse
+import importlib
+import sys
+
+import yaml
+
+from .tracker import LLMTracker
+
+
+def load_adapter(adapter_path: str):
+    """
+    adapter_path is a Python import path, e.g. "my_project.eval_adapter".
+    That module must define:
+        call_fn(prompt: str) -> (text: str, prompt_tokens: int, completion_tokens: int)
+        judge_fn(prompt: str) -> str
+    And may optionally define:
+        embed_fn(text: str) -> list[float]
+    This keeps the CLI provider-agnostic -- you bring whichever LLM
+    client and credentials your CI environment already has configured.
+    """
+    module = importlib.import_module(adapter_path)
+    if not hasattr(module, "call_fn") or not hasattr(module, "judge_fn"):
+        raise ImportError(
+            f"Adapter '{adapter_path}' must define call_fn(prompt) and judge_fn(prompt)."
+        )
+    return module
+
+
+def run_check(config_path: str, adapter_path: str, db_path: str) -> int:
+    with open(config_path) as f:
+        config = yaml.safe_load(f)
+
+    adapter = load_adapter(adapter_path)
+    embed_fn = getattr(adapter, "embed_fn", None)
+
+    tracker = LLMTracker(judge_fn=adapter.judge_fn, embed_fn=embed_fn, db_path=db_path)
+
+    faithfulness_threshold = config.get("faithfulness_threshold", 0.7)
+    drift_threshold = config.get("drift_threshold", 0.3)
+    model = config.get("model", "unknown")
+    provider = config.get("provider", "unknown")
+
+    failures = []
+    print(f"Running {len(config['cases'])} eval case(s) against {model} ({provider})\n")
+
+    for case in config["cases"]:
+        prompt_id = case["prompt_id"]
+        prompt = case["prompt"]
+        context = case.get("context")
+
+        with tracker.track(
+            model=model,
+            provider=provider,
+            prompt=prompt,
+            prompt_id=prompt_id,
+            context=context,
+        ) as call:
+            text, ptoks, ctoks = adapter.call_fn(prompt)
+            call.set_response(text, prompt_tokens=ptoks, completion_tokens=ctoks)
+
+        row = tracker.store.get_latest_eval(prompt_id)
+        faithfulness = row.get("faithfulness_score")
+        drift = row.get("semantic_drift")
+
+        status = "PASS"
+        if faithfulness is not None and faithfulness < faithfulness_threshold:
+            status = "FAIL"
+            failures.append(
+                f"{prompt_id}: faithfulness {faithfulness:.2f} < threshold {faithfulness_threshold}"
+            )
+        if drift is not None and drift > drift_threshold:
+            status = "FAIL"
+            failures.append(f"{prompt_id}: drift {drift:.3f} > threshold {drift_threshold}")
+
+        print(f"[{status}] {prompt_id}  faithfulness={faithfulness}  drift={drift}")
+
+    print()
+    if failures:
+        print(f"{len(failures)} check(s) failed:")
+        for f in failures:
+            print(f"  - {f}")
+        return 1
+
+    print("All checks passed.")
+    return 0
+
+
+def main():
+    parser = argparse.ArgumentParser(prog="llm-eval-exporter")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    check_parser = subparsers.add_parser("check", help="Run eval suite and fail on regression")
+    check_parser.add_argument("--config", required=True, help="Path to eval-suite.yaml")
+    check_parser.add_argument(
+        "--adapter",
+        required=True,
+        help="Python import path to your call_fn/judge_fn adapter module",
+    )
+    check_parser.add_argument(
+        "--db",
+        default="eval_baseline.db",
+        help="Path to baseline DB (persist across CI runs for drift detection)",
+    )
+
+    args = parser.parse_args()
+
+    if args.command == "check":
+        exit_code = run_check(args.config, args.adapter, args.db)
+        sys.exit(exit_code)
+
+
+if __name__ == "__main__":
+    main()

llm_eval_exporter-0.2.0/llm_eval_exporter/drift.py

@@ -0,0 +1,65 @@
+"""
+Semantic drift detection.
+
+The idea: for a given prompt_id (a stable name you assign to a prompt
+you care about, e.g. "customer_refund_policy_qa"), store the
+embedding of the first response as a baseline. On every later call
+with the same prompt_id, embed the new response and compute cosine
+distance from baseline. A rising trend means the model's answer to
+the same question is drifting over time (model upgrades, prompt
+changes upstream, temperature variance, etc).
+
+Embeddings are pluggable via `embed_fn` so this doesn't hard-depend
+on any one provider. Default uses sentence-transformers locally
+(no API cost, no external call).
+"""
+
+from typing import Callable, Optional
+import numpy as np
+
+from .storage import EvalStore
+
+_default_model = None
+
+
+def _default_embed_fn(text: str) -> list:
+    """Lazy-loaded local embedding model (all-MiniLM-L6-v2, ~80MB)."""
+    global _default_model
+    if _default_model is None:
+        from sentence_transformers import SentenceTransformer
+
+        _default_model = SentenceTransformer("all-MiniLM-L6-v2")
+    return _default_model.encode(text).tolist()
+
+
+def cosine_distance(a: list, b: list) -> float:
+    a, b = np.array(a), np.array(b)
+    denom = np.linalg.norm(a) * np.linalg.norm(b)
+    if denom == 0:
+        return 0.0
+    similarity = np.dot(a, b) / denom
+    return float(1.0 - similarity)
+
+
+class DriftTracker:
+    def __init__(self, store: EvalStore, embed_fn: Optional[Callable[[str], list]] = None):
+        self.store = store
+        self.embed_fn = embed_fn or _default_embed_fn
+
+    def score(self, prompt_id: str, model: str, response_text: str) -> Optional[float]:
+        """
+        Returns cosine distance from baseline (0 = identical, higher = more drift).
+        Returns None if this is the first time we've seen prompt_id
+        (a baseline is recorded instead, nothing to compare yet).
+        """
+        if not prompt_id:
+            return None
+
+        current_embedding = self.embed_fn(response_text)
+        baseline = self.store.get_baseline(prompt_id)
+
+        if baseline is None:
+            self.store.set_baseline(prompt_id, model, current_embedding)
+            return None
+
+        return cosine_distance(baseline, current_embedding)

llm_eval_exporter-0.2.0/llm_eval_exporter/judge.py

@@ -0,0 +1,65 @@
+"""
+Faithfulness scoring via LLM-as-judge.
+
+Given a response and the context it was supposed to be grounded in,
+ask a judge model to score 0.0-1.0 how well the response sticks to
+what the context actually supports (a lightweight, single-call
+analogue of RAGAS-style faithfulness).
+
+The judge call itself is pluggable via `judge_fn` so this package
+never hard-depends on one vendor's SDK. Bring your own client.
+"""
+
+import json
+import re
+from typing import Callable, Optional
+
+JUDGE_PROMPT_TEMPLATE = """You are grading whether a RESPONSE is faithful to a CONTEXT.
+Faithful means every factual claim in the response is supported by the context.
+Score from 0.0 (completely unsupported / hallucinated) to 1.0 (fully supported).
+
+CONTEXT:
+{context}
+
+RESPONSE:
+{response}
+
+Reply with ONLY a JSON object: {{"score": <float 0.0-1.0>, "reason": "<one sentence>"}}
+"""
+
+
+def parse_judge_output(raw: str) -> dict:
+    """Judge models occasionally wrap JSON in prose or code fences; extract it defensively."""
+    match = re.search(r"\{.*\}", raw, re.DOTALL)
+    if not match:
+        raise ValueError(f"Could not parse judge output: {raw!r}")
+    return json.loads(match.group(0))
+
+
+class FaithfulnessJudge:
+    def __init__(self, judge_fn: Callable[[str], str]):
+        """
+        judge_fn: a function that takes a prompt string and returns the
+        judge model's raw text response. E.g.:
+
+            def judge_fn(prompt: str) -> str:
+                resp = anthropic_client.messages.create(
+                    model="claude-haiku-4-5-20251001",
+                    max_tokens=200,
+                    messages=[{"role": "user", "content": prompt}],
+                )
+                return resp.content[0].text
+        """
+        self.judge_fn = judge_fn
+
+    def score(self, response: str, context: Optional[str]) -> Optional[dict]:
+        """
+        Returns {"score": float, "reason": str}, or None if no context
+        was provided (faithfulness is undefined without grounding context).
+        """
+        if not context:
+            return None
+
+        prompt = JUDGE_PROMPT_TEMPLATE.format(context=context, response=response)
+        raw = self.judge_fn(prompt)
+        return parse_judge_output(raw)

llm_eval_exporter-0.2.0/llm_eval_exporter/metrics.py

@@ -0,0 +1,79 @@
+"""
+Prometheus metric definitions for LLM eval observability.
+
+These are the metrics that get scraped by Prometheus and visualized
+in Grafana. Everything else in this package exists to compute values
+for these metrics.
+"""
+
+from prometheus_client import Counter, Histogram, Gauge
+
+# --- Request-level metrics ---
+
+LLM_REQUEST_LATENCY = Histogram(
+    "llm_request_latency_seconds",
+    "Latency of LLM calls",
+    ["model", "provider"],
+    buckets=(0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0),
+)
+
+LLM_REQUEST_TOTAL = Counter(
+    "llm_request_total",
+    "Total number of LLM requests made",
+    ["model", "provider", "status"],
+)
+
+LLM_TOKENS_TOTAL = Counter(
+    "llm_tokens_total",
+    "Total tokens consumed",
+    ["model", "provider", "type"],  # type = prompt|completion
+)
+
+# --- Eval metrics (the actual differentiator) ---
+
+LLM_FAITHFULNESS_SCORE = Histogram(
+    "llm_faithfulness_score",
+    "Judge-model faithfulness score (0-1) for responses grounded in provided context",
+    ["model"],
+    buckets=(0.0, 0.2, 0.4, 0.6, 0.7, 0.8, 0.9, 1.0),
+)
+
+LLM_SEMANTIC_DRIFT = Gauge(
+    "llm_semantic_drift",
+    "Cosine distance between current and baseline embedding for a tracked prompt",
+    ["model", "prompt_id"],
+)
+
+LLM_EVAL_FAILURES_TOTAL = Counter(
+    "llm_eval_failures_total",
+    "Count of responses that failed an eval check (below threshold)",
+    ["model", "check"],  # check = faithfulness|drift
+)
+
+# --- Cost metrics ---
+
+LLM_COST_USD_TOTAL = Counter(
+    "llm_cost_usd_total",
+    "Estimated cost in USD based on token usage and a static price table",
+    ["model", "provider"],
+)
+
+# --- Session-level metrics (a session = a multi-turn conversation) ---
+
+LLM_SESSION_FAITHFULNESS_AVG = Gauge(
+    "llm_session_faithfulness_avg",
+    "Rolling average faithfulness score across all turns in a session",
+    ["session_id"],
+)
+
+LLM_SESSION_DRIFT_MAX = Gauge(
+    "llm_session_drift_max",
+    "Maximum semantic drift observed across all turns in a session",
+    ["session_id"],
+)
+
+LLM_SESSION_TURNS_TOTAL = Counter(
+    "llm_session_turns_total",
+    "Number of tracked turns within a session",
+    ["session_id"],
+)