rsc-universal 0.1.1

package/skills/building-agents/references/evals-and-observability.md

@@ -0,0 +1,420 @@
+# Evals, tracing & cost control — the production line
+
+An LLM feature without an eval gate is a demo. This file ships a runnable, provider-
+agnostic eval harness (the one `scripts/verify.sh` dry-runs), an LLM-as-judge that is
+itself model-agnostic, OTel GenAI tracing, and a closed-loop cost controller: caching,
+batching, routing/cascades, and hard budgets. Python 3.12+, Pydantic v2.
+
+## Eval-first
+
+Define success criteria **before** building. Two complementary suites:
+
+- **Capability evals** — can the agent do the new thing at all? Built once, when you add
+  a feature.
+- **Regression evals** — does a prompt/model/code change break what already worked? Run
+  on every PR against a stored baseline.
+
+Both reduce to: a golden set, graders, metrics, and a threshold that gates merge.
+
+## Golden sets
+
+JSONL, one case per line, versioned as fixtures next to the code:
+
+```json
+{"input": "Refund order ord_8123 for 500 cents, defective", "expected": {"tool": "refund", "order_id": "ord_8123"}, "metadata": {"suite": "capability", "tags": ["refund"]}}
+```
+
+```python
+import json
+from pathlib import Path
+
+from pydantic import BaseModel
+
+
+class Case(BaseModel):
+    input: str
+    expected: dict
+    metadata: dict = {}
+
+
+def load_golden(path: str | Path) -> list[Case]:
+    return [Case.model_validate_json(line) for line in Path(path).read_text().splitlines() if line.strip()]
+```
+
+Build the golden set from **real traffic** (sampled, labeled), not invented prompts; keep
+a strict train/eval split so prompts can't be tuned against the eval set (a form of
+leakage). Version it: a golden set is a first-class artifact, reviewed like code.
+
+## Graders
+
+Three grader kinds, one interface. The LLM-as-judge uses the same `LLMProvider`, so the
+judge is provider-agnostic and can run on a *different* model than the candidate.
+
+```python
+from typing import Protocol
+
+from provider_abstraction import CompletionRequest, LLMProvider, Message
+from provider_abstraction import complete_structured
+
+
+class Grader(Protocol):
+    name: str
+    def grade(self, case: Case, output: str) -> float: ...   # 0.0–1.0
+
+
+class ExactGrader:
+    name = "exact"
+    def grade(self, case: Case, output: str) -> float:
+        return float(output.strip() == str(case.expected.get("text", "")).strip())
+
+
+class SchemaGrader:
+    name = "schema"
+    def __init__(self, schema: type[BaseModel]) -> None:
+        self.schema = schema
+    def grade(self, case: Case, output: str) -> float:
+        try:
+            self.schema.model_validate_json(output)
+            return 1.0
+        except Exception:
+            return 0.0
+
+
+class Verdict(BaseModel):
+    score: float          # 0.0–1.0
+    reasoning: str
+
+
+class JudgeGrader:
+    name = "judge"
+    RUBRIC = ("Score 0.0–1.0 how well the ANSWER satisfies the CRITERIA. "
+              "Be strict; reward only grounded, complete, correct answers. Return JSON.")
+
+    def __init__(self, judge: LLMProvider, judge_model: str) -> None:
+        self.judge, self.model = judge, judge_model
+
+    async def grade(self, case: Case, output: str) -> float:
+        v = await complete_structured(self.judge, CompletionRequest(model=self.model, messages=[
+            Message(role="system", content=self.RUBRIC),
+            Message(role="user", content=f"CRITERIA:\n{case.expected}\n\nANSWER:\n{output}")]), Verdict)
+        return v.score
+```
+
+Judge hygiene: use a **different model** for judge vs candidate to avoid self-preference
+bias; prefer **pairwise** (A-vs-B) when comparing two systems and **pointwise** (score one)
+for absolute gates; randomize A/B order to cancel position bias; and **calibrate** the
+judge against ~50 human labels before trusting it in a gate (target judge-human agreement
+> 0.8).
+
+## Metrics
+
+```python
+def accuracy(scores: list[float]) -> float:
+    return sum(s >= 0.5 for s in scores) / len(scores)
+
+
+def pass_at_k(successes_per_case: list[int], k: int) -> float:
+    # pass@k = fraction of cases with >=1 success in k attempts.
+    return sum(s >= 1 for s in successes_per_case) / len(successes_per_case)
+
+
+def pass_caret_k(all_success_per_case: list[bool]) -> float:
+    # pass^k = fraction of cases where ALL k attempts succeeded (stability).
+    return sum(all_success_per_case) / len(all_success_per_case)
+
+
+def percentile(values: list[float], p: float) -> float:
+    s = sorted(values)
+    idx = min(len(s) - 1, int(round((p / 100) * (len(s) - 1))))
+    return s[idx]
+```
+
+Track at minimum: `accuracy`, `pass@1`/`pass@3`, `faithfulness` (judge/citation grader),
+answer relevance, `cost_per_task`, `p50`/`p95` latency, and `tool_call_validity_rate`
+(fraction of tool calls that pass schema validation). Recommended gates: capability
+`pass@3 ≥ 0.90`; regression `pass^3 = 1.0` on release-critical paths.
+
+## Eval runner
+
+The full async runner. It supports a **dry-run** mode (`EVAL_DRY_RUN=1` or `--dry-run`)
+that uses a stub provider and never hits live APIs — this is exactly what `verify.sh`
+invokes as a smoke test.
+
+```python
+# evals/run.py
+import argparse
+import asyncio
+import json
+import os
+import statistics
+import sys
+import time
+from pathlib import Path
+
+from provider_abstraction import (CompletionRequest, CompletionResponse, LLMProvider,
+                                   Message, Usage, get_provider)
+
+
+class StubProvider:
+    """Deterministic offline provider for dry-runs — no network, no keys."""
+    async def complete(self, req: CompletionRequest) -> CompletionResponse:
+        return CompletionResponse(text='{"ok": true}', usage=Usage(input_tokens=10, output_tokens=5))
+    async def stream(self, req):  # pragma: no cover - unused in dry-run
+        yield
+    async def embed(self, texts: list[str]) -> list[list[float]]:
+        return [[0.0] * 1536 for _ in texts]
+
+
+async def run(golden_path: str, dry_run: bool) -> int:
+    cases = [json.loads(line) for line in Path(golden_path).read_text().splitlines() if line.strip()]
+    provider: LLMProvider = StubProvider() if dry_run else get_provider(os.environ["LLM"])
+    model = "stub" if dry_run else os.environ["LLM"].split(":", 1)[1]
+    rows = []
+    for c in cases:
+        t0 = time.perf_counter()
+        resp = await provider.complete(CompletionRequest(model=model,
+            messages=[Message(role="user", content=c["input"])]))
+        ok = SchemaGrader(_AnyJSON).grade(_Case(c), resp.text)
+        rows.append({"score": ok, "cost": resp.usage.cost_usd, "ms": (time.perf_counter() - t0) * 1000})
+
+    n = len(rows) or 1
+    metrics = {
+        "accuracy": sum(r["score"] for r in rows) / n,
+        "cost_per_task": sum(r["cost"] for r in rows) / n,
+        "p95_latency_ms": statistics.quantiles([r["ms"] for r in rows], n=20)[-1] if len(rows) > 1 else rows[0]["ms"],
+    }
+    thresholds = {"accuracy": 0.0 if dry_run else 0.9}   # dry-run only checks the pipeline runs
+    Path("eval-report.json").write_text(json.dumps(metrics, indent=2))
+    failed = {k: metrics[k] for k, lo in thresholds.items() if metrics[k] < lo}
+    print(json.dumps({"metrics": metrics, "failed": failed, "dry_run": dry_run}, indent=2))
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--golden", default="evals/golden.jsonl")
+    ap.add_argument("--dry-run", action="store_true", default=os.environ.get("EVAL_DRY_RUN") == "1")
+    args = ap.parse_args()
+    sys.exit(asyncio.run(run(args.golden, args.dry_run)))
+```
+
+```python
+# tiny shims referenced above so the file is self-contained
+from pydantic import BaseModel
+
+
+class _AnyJSON(BaseModel):
+    model_config = {"extra": "allow"}
+
+
+class _Case:
+    def __init__(self, d: dict) -> None:
+        self.input, self.expected, self.metadata = d["input"], d.get("expected", {}), d.get("metadata", {})
+```
+
+## Regression gates in CI
+
+Run the suite on every PR; fail if any metric regresses past tolerance versus a stored
+baseline. GitHub Actions:
+
+```yaml
+name: eval-gate
+on: pull_request
+jobs:
+  evals:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install -e . pydantic openai anthropic
+      - name: Run evals
+        env:
+          LLM: ${{ vars.LLM }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: python evals/run.py --golden evals/golden.jsonl
+      - name: Compare against baseline
+        run: python evals/compare.py eval-report.json evals/baseline.json --tolerance 0.02
+```
+
+```python
+# evals/compare.py — fail if any metric regresses more than tolerance vs baseline.
+import json
+import sys
+
+current, baseline, _, tol = sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4])
+cur = json.load(open(current))
+base = json.load(open(baseline))
+regressions = {k: (base[k], cur[k]) for k in ("accuracy",)
+               if k in base and cur.get(k, 0) < base[k] - tol}
+if regressions:
+    print(f"REGRESSION: {regressions}")
+    sys.exit(1)
+print("no regressions")
+```
+
+Guard against flaky graders: run judge grades `n=3` and require majority; never put a
+grader with judge-human agreement below your threshold into a release gate.
+
+## Tracing (OTel GenAI)
+
+Emit OpenTelemetry **GenAI semantic-convention** spans (`gen_ai.*`). Dashboards
+(Langfuse / Phoenix / Braintrust) are swappable OTLP backends — you emit spans, you swap
+the exporter.
+
+```python
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+
+provider = TracerProvider()
+provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))  # OTEL_EXPORTER_OTLP_ENDPOINT
+trace.set_tracer_provider(provider)
+tracer = trace.get_tracer("agent")
+
+
+async def traced_complete(p: LLMProvider, system: str, req: CompletionRequest) -> CompletionResponse:
+    with tracer.start_as_current_span("chat") as span:
+        span.set_attribute("gen_ai.system", system)               # "openai" | "anthropic" | ...
+        span.set_attribute("gen_ai.request.model", req.model)
+        span.set_attribute("gen_ai.request.temperature", req.temperature)
+        resp = await p.complete(req)
+        span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
+        span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
+        span.set_attribute("gen_ai.usage.cost_usd", resp.usage.cost_usd)
+        return resp
+```
+
+Wrap each tool call and each agent step in its own child span; the agent loop's `run_id`
+becomes the trace id, so a whole run is one tree. **Redact PII before it lands in a span**
+(spans persist to a backend) — never trace raw user content unfiltered.
+
+## Cost & latency control
+
+### Caching
+
+Provider prompt caching (Anthropic `cache_control`, OpenAI/Gemini automatic cached input)
+is abstracted behind the adapter (`cache_system` flag). On top of that, an **app-level
+semantic cache** returns a prior answer when a new request is near-identical:
+
+```python
+import hashlib
+
+
+class SemanticCache:
+    def __init__(self, provider: LLMProvider, retriever, threshold: float = 0.95) -> None:
+        self.provider, self.retriever, self.threshold = provider, retriever, threshold
+
+    async def get_or_compute(self, prompt: str, compute) -> tuple[str, bool]:
+        [vec] = await self.provider.embed([prompt])
+        hits = await self.retriever.search(vec, k=1, flt={"kind": "cache"})
+        if hits and hits[0].get("sim", 0) >= self.threshold:
+            return hits[0]["content"], True                       # cache hit
+        answer = await compute()
+        await self.retriever.upsert(prompt, vec, answer, meta={"kind": "cache"})
+        return answer, False
+```
+
+### Batching
+
+For offline workloads (backfills, nightly evals, bulk classification), use provider batch
+APIs (≈ −50% cost class) instead of synchronous calls. Submit a JSONL of requests, poll,
+collect — latency is hours, not seconds, which is fine offline.
+
+### Model routing / cascades
+
+Classify the task, send it to the **cheapest model that passes the eval for that task
+class**, and escalate only on a failed self-check — never default to the flagship.
+
+```python
+async def route_and_run(task: str, classify, providers: dict[str, tuple[LLMProvider, str]]) -> str:
+    tier = await classify(task)                          # "cheap" | "default" | "smart"
+    order = {"cheap": ["cheap", "default", "smart"],
+             "default": ["default", "smart"],
+             "smart": ["smart"]}[tier]
+    for name in order:
+        provider, model = providers[name]
+        resp = await provider.complete(CompletionRequest(model=model,
+            messages=[Message(role="user", content=task)]))
+        if await _self_check(provider, model, task, resp.text):    # escalate only on failure
+            return resp.text
+    return resp.text                                     # smart tier is terminal
+
+
+async def _self_check(provider, model, task: str, answer: str) -> bool:
+    v = await complete_structured(provider, CompletionRequest(model=model, messages=[
+        Message(role="user", content=f"Does this answer fully and correctly address the task? "
+                                      f"TASK: {task}\nANSWER: {answer}")]), Verdict)
+    return v.score >= 0.8
+```
+
+### Budgets
+
+An immutable per-tenant cost tracker with a hard stop. Frozen dataclass = auditable, never
+silently mutated.
+
+```python
+from dataclasses import dataclass, field
+
+
+class BudgetExceededError(Exception):
+    pass
+
+
+@dataclass(frozen=True, slots=True)
+class CostRecord:
+    tenant: str
+    model: str
+    input_tokens: int
+    output_tokens: int
+    cost_usd: float
+
+
+@dataclass(frozen=True, slots=True)
+class CostTracker:
+    budgets_usd: dict[str, float] = field(default_factory=dict)   # per-tenant cap
+    records: tuple[CostRecord, ...] = ()
+
+    def add(self, r: CostRecord) -> "CostTracker":
+        spent = self.spent(r.tenant) + r.cost_usd
+        cap = self.budgets_usd.get(r.tenant, float("inf"))
+        if spent > cap:
+            raise BudgetExceededError(f"{r.tenant}: ${spent:.4f} exceeds ${cap:.2f}")
+        return CostTracker(budgets_usd=self.budgets_usd, records=(*self.records, r))
+
+    def spent(self, tenant: str) -> float:
+        return sum(rec.cost_usd for rec in self.records if rec.tenant == tenant)
+```
+
+Compute `cost_usd` from the dated pricing table — never hardcode a price in business logic.
+
+| Model (provider:id) | $/MTok in | $/MTok out | Notes |
+|---|---|---|---|
+| anthropic:claude-haiku-4-5 | 1.00 | 5.00 | cheap/fast; caching −90% read, batch −50% |
+| anthropic:claude-sonnet-4-6 | 3.00 | 15.00 | default; 1M-token context flat-rate |
+| anthropic:claude-opus-4-8 | 5.00 | 25.00 | flagship Opus (4.8, May 2026); re-verify the id |
+| openai:gpt-5.5 | 5.00 | 30.00 | flagship; >272K prompt billed 2× in / 1.5× out |
+| openai:gpt-5.4 | 2.50 | 15.00 | cached input −90% |
+| openai:gpt-5.2-codex | 1.75 | 14.00 | code-tuned |
+| gemini:gemini-3.5-flash | 1.50 | 9.00 | cached 0.15; 1M context |
+| gemini:gemini-3.1-pro | — | — | paid-only since 2026-04-01 |
+
+All figures **(as of 2026-06; verify before quoting)**. Load them from config and refresh
+the table before trusting any number.
+
+## Anti-patterns
+
+- **Overfitting prompts to the eval set** — you measure memorization, not capability.
+  Hold out a fresh split.
+- **Happy-path-only evals** — include adversarial, empty, and malformed inputs.
+- **Chasing accuracy while cost/latency drift** — gate on `cost_per_task` and `p95`, not
+  just accuracy.
+- **Flaky graders in release gates** — calibrate and majority-vote, or keep them advisory.
+- **Tracing PII without redaction** — spans persist; redact before emit.
+
+## See also
+
+- `provider-abstraction.md` — the `LLMProvider` the runner, judge, and router all use.
+- `agent-loops-and-harness.md` — budgets and `AgentState` the `CostTracker` complements.