opengrade-ai 0.1.0__tar.gz

opengrade_ai-0.1.0/.gitignore

@@ -0,0 +1,42 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+.venv/
+venv/
+.env
+.env.local
+*.egg-info/
+dist/
+build/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.coverage
+htmlcov/
+.uv/
+
+# IDE / OS
+.idea/
+.vscode/
+*.swp
+.DS_Store
+Thumbs.db
+
+# Docker volumes
+postgres_data/
+
+.worktrees/
+.cursor/
+docs/superpowers/
+
+# Packaged Next.js UI (built by scripts/package-ui.sh)
+backend/src/opengrade_server/_ui/**
+!backend/src/opengrade_server/_ui/.gitkeep
+
+opengrade.db
+**/opengrade.db
+
+.opengrade/

opengrade_ai-0.1.0/PKG-INFO

@@ -0,0 +1,487 @@
+Metadata-Version: 2.5
+Name: opengrade-ai
+Version: 0.1.0
+Summary: Python SDK for AI Evaluation & Observability Platform (OTLP + OpenInference)
+Requires-Python: >=3.12
+Requires-Dist: httpx2>=2.13.0
+Requires-Dist: opentelemetry-api>=1.28.0
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.28.0
+Requires-Dist: opentelemetry-sdk>=1.28.0
+Requires-Dist: pydantic>=2.10.0
+Provides-Extra: ai
+Requires-Dist: openinference-instrumentation-anthropic>=0.1.0; extra == 'ai'
+Requires-Dist: openinference-instrumentation-bedrock>=0.1.0; extra == 'ai'
+Requires-Dist: openinference-instrumentation-langchain>=0.1.0; extra == 'ai'
+Requires-Dist: openinference-instrumentation-llama-index>=0.1.0; extra == 'ai'
+Requires-Dist: openinference-instrumentation-openai>=0.1.0; extra == 'ai'
+Provides-Extra: anthropic
+Requires-Dist: openinference-instrumentation-anthropic>=0.1.0; extra == 'anthropic'
+Provides-Extra: bedrock
+Requires-Dist: openinference-instrumentation-bedrock>=0.1.0; extra == 'bedrock'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.13.0; extra == 'dev'
+Requires-Dist: pytest>=8.3.0; extra == 'dev'
+Requires-Dist: ruff>=0.8.0; extra == 'dev'
+Provides-Extra: langchain
+Requires-Dist: openinference-instrumentation-langchain>=0.1.0; extra == 'langchain'
+Provides-Extra: llama-index
+Requires-Dist: openinference-instrumentation-llama-index>=0.1.0; extra == 'llama-index'
+Provides-Extra: openai
+Requires-Dist: openinference-instrumentation-openai>=0.1.0; extra == 'openai'
+Provides-Extra: ui
+Requires-Dist: opengrade-ai-server==0.1.0; extra == 'ui'
+Description-Content-Type: text/markdown
+
+# opengrade
+
+Python SDK for the AI Evaluation & Observability Platform: OTLP/HTTP export with optional OpenInference instrumentation, plus a control-plane `Client` for datasets, experiments, and evaluation.
+
+## Install
+
+```bash
+pip install opengrade-ai
+pip install 'opengrade-ai[openai]'
+pip install 'opengrade-ai[ai]'   # all tier-1 OpenInference instrumentors
+pip install 'opengrade-ai[ui]'  # also installs opengrade-ai-server (API + packaged UI)
+```
+
+Individual extras: `openai`, `anthropic`, `langchain`, `llama-index`, `bedrock`, `ui`.
+
+Import path remains `import opengrade`. Local platform (SQLite + UI):
+
+```bash
+pip install 'opengrade-ai[ui]'
+opengrade ui   # http://127.0.0.1:8000 — creates ./opengrade.db
+```
+
+The `opengrade` command ships with the SDK. Without the `[ui]` extra it exits with a hint to install it.
+
+Develop from this monorepo:
+
+```bash
+cd sdk && uv sync --extra dev
+uv sync --extra openai   # add OpenAI instrumentor
+# UI packaging (server wheel): ./scripts/package-ui.sh then cd backend && uv sync
+```
+
+## Quickstart
+
+One tracking URI, one active project, one `init`. A `@trace` wrapper for your app logic, and provider calls become child spans when the matching instrumentor is installed:
+
+```python
+import opengrade
+from openai import OpenAI
+
+opengrade.set_tracking_uri("http://localhost:8000")  # or OPENGRADE_TRACKING_URI
+opengrade.set_project("demo")  # created if it does not exist
+opengrade.init(instrument="auto")  # OTLP → <tracking_uri>/v1/traces
+
+client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
+
+
+@opengrade.trace
+def ask(q: str) -> str:
+    r = client.chat.completions.create(
+        model="gemma4:e2b",
+        messages=[{"role": "user", "content": q}],
+    )
+    return r.choices[0].message.content or ""
+
+
+ask("ping")
+opengrade.flush()
+```
+
+Runnable example: [`examples/sdk_hello/`](../examples/sdk_hello/README.md) (Ollama via OpenAI-compatible API).
+
+### API summary
+
+| Function | Purpose |
+|---|---|
+| `opengrade.set_tracking_uri(uri)` | Platform URL for both `Client` and OTLP (`<uri>/v1/traces`) |
+| `opengrade.set_project(name)` | Get-or-create project by slug/name; becomes the default for `init()` |
+| `opengrade.aset_project(name)` | Async variant of `set_project` (uses shared `AsyncClient`) |
+| `opengrade.get_client()` | Shared `Client` bound to the tracking URI (no per-call `Client()`) |
+| `opengrade.get_async_client()` | Shared `AsyncClient` for async app code |
+| `opengrade.get_project()` | Project set via `set_project` / `aset_project`, or `None` |
+| `opengrade.init(...)` | OTLP exporter, project headers, optional `instrument="auto"` \| list \| `False` |
+| `@opengrade.trace` / `@opengrade.trace_async` | Parent CHAIN span; optional input/output capture (32 KiB cap) |
+| `opengrade.set_input` / `set_output` | Override OpenInference I/O on the current span (no OTEL import) |
+| `opengrade.set_attribute` / `set_attributes` | Custom attributes on the current span |
+| `opengrade.set_error(...)` | Mark current span ERROR |
+| `opengrade.current_trace_id()` | 32-char hex trace id while inside a `@trace` span |
+| `opengrade.bind_evaluation(...)` | Bind current span to `experiment_id` + `dataset_item_id` |
+| `opengrade.set_retrieval_documents([...])` | Attach retrieval docs for RAG eval context |
+| `opengrade.flush()` | Drain the batch span processor before exit |
+| `opengrade.evaluate(...)` | Run a task over a dataset, score, compare with a baseline → `ExperimentResult` |
+| `opengrade.start_experiment(...)` | Context manager for a manual item loop (`exp.items()`, `exp.run(item)`) |
+| `opengrade.Client(...)` | Control-plane REST client (projects, datasets, experiments, …) |
+| `opengrade.AsyncClient(...)` | Async control-plane client (projects, live_runs, trace_scores, health) |
+
+App code should use these helpers instead of importing OpenTelemetry directly.
+
+## Experiments (`evaluate` / `start_experiment`)
+
+One call runs a task over a dataset, writes the outputs, scores them and compares against a baseline:
+
+```python
+import opengrade
+
+opengrade.set_project("demo")
+opengrade.init()  # optional: adds trace context (tokens, cost, tool calls) to each item
+
+
+def answer_question(item):  # item.input, item.expected_output, item.metadata
+    docs = retrieve(item.input)
+    opengrade.set_retrieval_documents(docs)  # feeds hit_at_k / recall_at_k / groundedness
+    return generate(item.input, docs)
+
+
+def short(item, output) -> bool:  # local metric: float | bool | None
+    return len(output) < 400
+
+
+result = opengrade.evaluate(
+    dataset="gold-v1",  # name/id, path, list/iterable of dicts, pandas/Polars DataFrame
+    task=answer_question,
+    metrics=["groundedness", "hit_at_k", short],
+    name="gemma-prompt-v2",
+    params={"model": "gemma4"},  # stored as the experiment's model_config
+    baseline="prod",  # experiment name (latest) or id
+    max_concurrency=8,
+)
+print(result.summary(), result.url)
+```
+
+Manual loop, same plumbing (binding, flush, output upload, scoring on exit):
+
+```python
+with opengrade.start_experiment("v2", dataset="gold-v1", metrics=["hit_at_k"]) as exp:
+    for item in exp.items():
+        with exp.run(item) as run:  # root span bound to (experiment, item)
+            run.log(my_app(item.input))  # or opengrade.set_output(...) anywhere inside
+print(exp.result.summary())
+```
+
+How it behaves:
+
+- **Metrics**: a string is a server metric kind (`hit_at_k`, `recall_at_k`, `mrr`, `must_contain`, `groundedness`, `correctness`, `answer_relevance`, `context_precision`, `context_recall`, `latency`). A dict sets threshold/config, for example `{"kind": "hit_at_k", "config": {"k": 3}}`. The SDK finds or creates a metrics set that holds exactly those kinds, using the project's defaults for anything you leave out. `metrics=None` uses the project default set. A Python metric runs locally (see *Custom metrics* below), and its scores are submitted to the platform (`POST /experiments/{id}/client-results`). From there they are aggregated, shown in the UI and compared with the baseline like server metrics.
+- **Datasets**: pass a registered name/id, a path to CSV/JSON/JSONL, a list or iterable of dicts, a pandas DataFrame, or a Polars DataFrame. Rows need an `input` column (or `question`). `expected_output` (or `expected_answer`), `context` and `metadata` map to item fields; any other column goes into `metadata` (so `expected_doc_ids` and `must_contain` work). A missing `input`/`question` raises `opengrade.DatasetColumnError` (CSV headers are checked up front). Inline rows are hashed into a dataset name, so re-running on the same rows reuses the same dataset. Generators are materialised before upload (no lazy streaming).
+- **Outputs** are uploaded with `PUT /experiments/{id}/outputs`, so this works without `opengrade.init()`. With tracing on, `finish()` waits for the OTLP traces to be bound (up to 30s) and merges their context in. Your explicit output always wins.
+- **Errors**: when a task raises, `evaluate` records it on the item (`result.errors`, `metadata.error`) and moves on. Pass `raise_on_error=True` to stop instead. When the `with` block raises, nothing is scored. Bad dataset columns raise `DatasetColumnError` before the run starts.
+
+### Custom metrics
+
+```python
+@opengrade.metric(threshold=0.5, version=2, requires=("documents",))
+def cites_source(s: opengrade.Sample) -> opengrade.Score:
+    hit = any(d["id"] in s.output for d in s.documents)
+    return opengrade.Score(1.0 if hit else 0.0, explanation="cited" if hit else "no citation")
+
+
+opengrade.evaluate(dataset="gold-v1", task=answer_question, metrics=["hit_at_k", cites_source])
+```
+
+- `Sample` fields: `input`, `output`, `expected` (dataset `expected_output`), `documents` (from `set_retrieval_documents`), `context`, `metadata`, `trace_id`, `item_id`.
+- Return `opengrade.Score(value, label=, explanation=, metadata=)`, a float, a bool, `None` or `{"score": ...}`. The explanation shows per item in the UI; `metadata` (JSON-serializable, no NaN/inf) is stored with the result. A non-finite score is an ERROR.
+- `label` can only fail, skip or error an item (`"FAIL"`, `"SKIPPED"`, `"ERROR"`): PASS/FAIL otherwise comes from the threshold, so `label="PASS"` does not make a score below the threshold pass.
+- To change the threshold or name of an existing metric, wrap it again: `opengrade.LocalMetric(cites_source, threshold=0.8)` keeps its function, `requires` and config.
+- `fn(item, output)` (two parameters) still works and receives the dataset item. `async def` metrics work too.
+- `requires`: `Sample` fields that must be non-empty (`output`, `expected`, `documents`, `context`). When one is missing the item is SKIPPED, not failed.
+- `version` is stored in the metric config: bump it when the code changes, so compare reports a config mismatch instead of a misleading delta.
+- When the task fails on an item, that item counts as FAIL (score 0). When the metric raises, the item is ERROR. A local metric name must not match a server evaluator name (409). If the submission fails, `result.metrics` falls back to the in-process mean.
+
+### Params and run metrics
+```python
+with opengrade.start_experiment("v2", dataset="gold-v1", params={"model": "gemma4"}) as exp:
+    exp.log_params(temperature=0.2)  # merged into the experiment's model_config
+    exp.run_task(answer_question)
+    opengrade.log_metrics({"cost_usd": total_cost})  # experiment-level scalar, last value wins
+print(exp.result.experiment_metrics)
+```
+
+- `log_params` merges into `model_config` until an evaluation starts. After that, or when the experiment was created from an app config, the server answers 409: start a new experiment instead.
+- `log_metrics` stores one final number per key (no steps). Keys use letters, digits and `_ . - /`. They appear in `result.summary()` with the delta against the baseline, and can gate a release via the release-check policy: `{"experiment_metrics": {"cost_usd": {"max": 2.0}}}`.
+- `opengrade.log_params` / `opengrade.log_metrics` act on the active experiment: inside `with start_experiment(...)` and inside any item run (`exp.run(...)`, `run_task`, `evaluate()` tasks). Elsewhere they raise `RuntimeError`. With concurrency > 1, call `log_metrics`/`log_params` once outside per-item code: per-item calls overwrite each other (last write wins).
+- Every `finish()` logs `opengrade.error_rate`, `opengrade.latency_p50_ms` and `opengrade.latency_p95_ms` (the task's wall-clock latency as measured by the SDK, over the items that did not fail, nearest-rank percentiles).
+
+## Control plane (`Client`)
+
+HTTP client for the platform JSON API (uses `httpx2`, a core SDK dependency). Without `base_url` it uses the tracking URI. In app code prefer the shared instance, `opengrade.get_client()`, over building a `Client()` per function:
+
+```python
+import opengrade
+
+opengrade.set_project("demo")
+client = opengrade.get_client()
+datasets = client.datasets.list()  # project_id defaults to the active project
+exp = client.experiments.create(name="baseline", dataset_id=datasets[0].id)
+```
+
+Project-scoped methods (`datasets.create/list/create_with_items`, `experiments.create/list`, `metrics_packs.ensure/get`, `live_runs.submit/list`, `projects.get`) take `project_id` as an optional first argument: explicit value > `set_project()` > `OPENGRADE_PROJECT_ID`, else `ValueError`.
+
+Explicit, self-managed client:
+
+```python
+from opengrade import Client
+
+with Client("http://localhost:8000") as client:
+    project = client.projects.create(name="Demo", slug="demo")
+    ds = client.datasets.create_with_items(
+        project.id,
+        name="gold-v1",
+        task_type="rag_qa",
+        items=[{"input": "Q?", "expected_output": "A"}],
+    )
+    # or CSV/JSONL via multipart import:
+    # client.datasets.import_items(ds.id, path="gold.csv", format="csv")
+
+    exp = client.experiments.create(
+        project.id,
+        name="baseline",
+        dataset_id=ds.id,
+        model_config={"model": "gemma4:e2b"},
+    )
+    client.metrics_packs.ensure(project.id)
+    # after OTLP-bound runs:
+    client.experiments.evaluate_pack(exp.id)
+    summary = client.experiments.summary(exp.id)
+```
+
+Control-plane methods return **Pydantic** models (see `opengrade.models`); use attributes or `.model_dump()` for a dict. Prefer `with Client(...)` / `client.close()` so the shared `httpx2` connection pool is released.
+
+Namespaces: `projects`, `datasets` (`create`, `create_with_items`, `add_item`, `get`, `list`, `import_items`), `experiments` (`create`, `get`, `list_outputs`, `upsert_outputs`, `evaluate_pack`, `summary`, `compare`), `metrics_packs` (`ensure`, `get`), `metrics_sets` (`list`, `get`, `create`), `live_runs` (`submit`, `list`, `get`, `promote`, `rescore`), `app_configs` (same methods as `AppConfigClient`). Non-2xx responses raise `OpengradeAPIError` (with parsed FastAPI `detail` when present).
+
+### Async control plane (`AsyncClient`)
+
+For FastAPI / async RAG apps, use `AsyncClient` so control-plane calls do not block the event loop. Prefer `opengrade.get_async_client()` and `await opengrade.aset_project(...)`. MVP surface: `health`, `projects`, `live_runs` (full), `trace_scores`. Datasets, experiments, metrics packs/sets, and app configs remain on the sync `Client`.
+
+```python
+import opengrade
+
+
+# FastAPI lifespan
+async def lifespan(app):
+    opengrade.set_tracking_uri("http://localhost:8000")
+    await opengrade.aset_project("my-rag")
+    opengrade.init(instrument=False)
+    yield
+    await opengrade.get_async_client().aclose()
+
+
+@opengrade.trace_async(name="answer")
+async def answer(q: str) -> str:
+    text = "..."
+    client = opengrade.get_async_client()
+    await client.live_runs.submit(question=q, answer=text, documents=[])
+    return text
+```
+
+Or manage the client yourself: `async with AsyncClient(...) as client: ...`. Changing `set_tracking_uri` resets both shared clients; if an event loop is already running, `aclose` the async client before changing the URI (otherwise the shared async client is dropped with a warning).
+
+### Live runs from tracing (production)
+
+With `live=True`, the outermost `@opengrade.trace` call outside an experiment becomes a live interaction at OTLP ingestion; no separate `live_runs.submit` call is needed.
+
+```python
+opengrade.init(
+    project_slug="acme", live=True, live_sample_rate=0.1
+)  # or OPENGRADE_LIVE=1, OPENGRADE_LIVE_SAMPLE_RATE=0.1
+
+
+@opengrade.trace
+def answer(question: str) -> str:
+    docs = retrieve(question)  # documents from RETRIEVER spans or set_retrieval_documents()
+    return llm(question, docs)
+```
+
+- Every call is recorded; only `live_sample_rate` of them (default `1.0`) are sent to the judge. The rest show `judge_status="unsampled"` and can still be judged with `live_runs.rescore`. Sampling is deterministic on the trace id.
+- Question = the only string argument (or a `question`/`query`/`input`/`prompt` kwarg); answer = the return value. Override with `opengrade.set_input(...)` / `opengrade.set_output(...)`.
+- `@opengrade.trace(live=False)` excludes a function; `live=True` opts one in when `init(live=...)` is off.
+- The server must run with `CONTENT_CAPTURE_ENABLED=true`, otherwise input/output are redacted and no live interaction is created.
+- Idempotent per trace (`external_id="trace:<trace_id>"`).
+
+### Production scores and feedback
+
+Attach scores and end-user feedback to a production trace. They are stored per trace and show up on the Live page (next to the judge verdicts), in the "Failing" filter and on the overview.
+
+```python
+opengrade.init(live=True)
+
+
+@opengrade.metric(threshold=0.5, requires=("documents",))
+def cites_source(s: opengrade.Sample) -> float:
+    return float(any(d["id"] in s.output for d in s.documents))
+
+
+@opengrade.trace(
+    metrics=[cites_source], metrics_sample_rate=0.2
+)  # 20% of traces, scored off the request path
+def answer(question: str) -> str:
+    docs = retrieve(question)
+    opengrade.set_retrieval_documents(docs)
+    return generate(question, docs)
+
+
+@opengrade.trace
+def handle(question: str) -> dict:
+    out = answer(question)
+    opengrade.log_score("answer_chars", len(out))  # current trace
+    return {"answer": out, "trace_id": opengrade.current_trace_id()}  # send the id to your frontend
+
+
+# later, from the thumbs button (trace_id is the one handle() returned)
+opengrade.log_feedback(True, trace_id=trace_id, user_id="user-42", comment="great")
+```
+
+- `log_score(name, value, *, trace_id=None, label=None, explanation=None, threshold=None, metadata=None)`: without `trace_id` it uses the current trace. PASS/FAIL follows the same rule as experiment metrics: a `FAIL` label fails; a value under `threshold` fails; otherwise, with a threshold a value at or above it passes, and without one only a value of 0 fails.
+- `log_feedback(positive, *, trace_id=None, comment=None, user_id=None)`: stored as `user_feedback` (1 or 0); `positive` must be a bool (or 0/1). One value per user per trace: a user changing their mind replaces it. Without `user_id` the feedback is anonymous and all anonymous feedback on a trace shares one row (last write wins). A thumbs-down counts as a failure in the Live "Failing" filter.
+- `opengrade.score_trace(metric, sample, trace_id=...)` runs one metric now, on the calling thread (only the sending is in the background), and stores the result. The trace is `trace_id`, else `sample.trace_id`, else the current trace. A raising metric queues `label="ERROR"` with the exception as explanation (returns `True` when that score is queued); missing `requires` skips with `False`.
+- `@opengrade.trace(metrics=[...], metrics_sample_rate=1.0)` runs your metrics after the function returns successfully (not when it raises, not inside experiments, which score their own metrics). Needs `opengrade.init()`. Sampling is per trace and configured separately from the judge's `live_sample_rate`; both use the same hash of the trace id, so when both are set the sampled traces are nested (the lower rate's traces are a subset of the higher rate's).
+  - `Sample.input` is what you passed to `opengrade.set_input()` inside the call, else the only argument, else `{"args", "kwargs"}`; `Sample.documents` is the last `set_retrieval_documents()` inside the call (including nested `@opengrade.trace` helpers on the same thread); `expected` is `None`, so metrics with `requires=("expected",)` are skipped. Input, output and documents are deep-copied when the job is queued (so later mutation is invisible); samples over 256 KiB serialized are skipped with a warning. Capture uses a ContextVar: a worker thread your function starts does not inherit it, so `set_retrieval_documents` (or `record_documents`) from that thread does not feed the outer sample — call it on the traced thread before return, or pass documents into the metric yourself.
+  - Metrics run on a background pool (default 2 threads; `init(metrics_workers=...)` or `OPENGRADE_METRICS_WORKERS`) fed by a bounded queue; when it is full the call is not scored and a warning is logged. Async metrics time out after 60 s; a sync metric cannot be interrupted. A raising or timed-out metric stores `label="ERROR"` with the exception as explanation (never breaks the traced call). For LLM judges raise `metrics_workers` or lower `metrics_sample_rate`.
+  - Metrics run with instrumentation suppressed: LLM calls a judge makes and `@opengrade.trace` helpers it calls do not create production traces (spans started manually with an OpenTelemetry tracer still do).
+  - `trace(metrics=...)` rejects async and generator functions; use `trace_async` for coroutines.
+    `trace_async` also accepts `functools.partial` of a coroutine function; other async wrappers
+    need `inspect.markcoroutinefunction`.
+- These calls never raise. `log_score`, `log_feedback` and `trace(metrics=...)` never block the caller; `score_trace()` blocks for as long as the metric takes. Invalid input (bad value, label or trace id, or metadata over 16 KB serialized) logs a warning and returns `False`; setup problems (no project id, no active trace) warn at most once a minute.
+- Scores are sent in the background (batches every 2 s, 100 scores or about 512 KB). A 413 (body over the server's 1 MiB limit) splits the batch in half and resends, dropping only a single score that is still too large. A batch rejected with 400/422 is resent item by item so only the bad scores are dropped; 401/403/404 and other 4xx drop the batch at once; only network errors and 5xx are retried once after a 1 s backoff, then dropped with a warning.
+- `opengrade.flush(timeout_millis)` is one total budget: spans, then pending metrics, then scores. At exit, spans, pending metrics, and scores share one ~5 s budget (span flush is capped at ~2 s of that).
+
+### Live runs → TestSet loop
+
+```python
+from opengrade import Client
+
+with Client() as client:
+    project_id = "..."  # existing project
+    live = client.live_runs.submit(
+        project_id,
+        question="How many PTO days?",
+        answer="Unlimited.",
+        documents=[{"id": "pto", "text": "Full-time staff get 20 PTO days."}],
+    )
+    detail = client.live_runs.get(live.id)  # or list(..., failed_only=True)
+    # After review, promote with reviewer-authored gold (never copy the prod answer):
+    ds = client.datasets.create(project_id, name="from-live", task_type="rag_qa")
+    client.live_runs.promote(
+        live.id,
+        dataset_id=ds.id,
+        expected_output="Full-time employees get 20 PTO days per year.",
+        expected_doc_ids=["pto"],
+    )
+    exp = client.experiments.create(project_id, name="after-promote", dataset_id=ds.id)
+    client.experiments.evaluate_pack(exp.id)
+```
+
+See also [`scripts/live_run_demo.py`](../scripts/live_run_demo.py) (`--promote`).
+
+Eval binding still uses tracing attributes (no REST call):
+
+```python
+@opengrade.trace
+def run_item(question: str) -> str:
+    opengrade.bind_evaluation(experiment_id=exp_id, dataset_item_id=item_id)
+    ...
+```
+
+### Registry (app configs)
+
+`AppConfigClient` remains available for standalone use; `client.app_configs` exposes the same API:
+
+```python
+from opengrade import AppConfigClient
+
+registry = AppConfigClient("http://localhost:8000")
+created = registry.create_app_config(
+    project_id,
+    "rag-faq",
+    prompt={"system": "You are helpful."},
+    model={"model_id": "demo"},
+)
+latest = registry.list_app_configs(project_id, latest=True)
+registry.set_alias(project_id, "baseline", created.id)
+```
+
+Methods: `create_app_config`, `list_app_configs` (`name`, `latest`), `set_alias`, `get_aliases`.
+
+End-to-end prompt A/B on the People Ops RAG example (same model, two App Config prompts): `python scripts/prompt_ab_demo.py` — see `examples/hr_it_assistant/README.md`.
+
+## Tier-1 compatibility matrix
+
+| Key | Target library | OpenInference package | Notes |
+|---|---|---|---|
+| `openai` | `openai` | `openinference-instrumentation-openai` | OpenAI API and OpenAI-compatible endpoints (e.g. Ollama `/v1`) |
+| `anthropic` | `anthropic` | `openinference-instrumentation-anthropic` | Documented; manual smoke outside default CI |
+| `langchain` | `langchain` / `langchain_core` | `openinference-instrumentation-langchain` | Documented; manual smoke outside default CI |
+| `llama_index` | `llama_index` | `openinference-instrumentation-llama-index` | Extra name: `llama-index` |
+| `bedrock` | `boto3` (Bedrock runtime) | `openinference-instrumentation-bedrock` | Documented; manual smoke outside default CI |
+
+Install the extra for the providers you use, then `init(instrument="auto")` activates every tier-1 entry whose target library and instrumentor package are importable. Use `init(instrument=["openai", "langchain"])` to restrict keys, or `instrument=False` for decorator-only tracing.
+
+## Environment variables
+
+| Variable | Purpose |
+|---|---|
+| `OPENGRADE_TRACKING_URI` | Platform URL for `Client()` and OTLP (default `http://localhost:8000`); `set_tracking_uri()` wins |
+| `OPENGRADE_PROJECT_ID` | Project UUID header (`X-Project-Id`); used when no `set_project()` / `init()` argument |
+| `OPENGRADE_PROJECT_SLUG` | Project slug header (`X-Project-Slug`); the project must already exist |
+| `OPENGRADE_SERVICE_NAME` | OpenTelemetry `service.name` (default `opengrade-app`) |
+| `OPENGRADE_OTLP_ENDPOINT` | Optional override of the derived OTLP URL (e.g. a separate collector) |
+| `OPENGRADE_DISABLED` | `1` / `true` / `yes` / `on` turns `init()` into a no-op (decorators keep working, nothing is exported) |
+| `OPENGRADE_METRICS_WORKERS` | Size of the `@opengrade.trace(metrics=...)` background pool (default `2`, range 1–64); `init(metrics_workers=...)` wins |
+
+Project precedence for `init()`: arguments > `set_project()` > env.
+
+## Failure behavior
+
+Tracing never breaks the host app:
+
+- `init()` never raises. Without a project it logs a warning on the `opengrade` logger and stays a no-op; a broken instrumentor is skipped with a warning.
+- Pending spans are flushed at interpreter exit (`atexit`, sharing the ~5 s exit budget with metrics/scores; span stage capped at ~2 s), so short scripts don't need `opengrade.flush()`.
+- If the platform is unreachable, spans are dropped: one warning when export starts failing, one info when it recovers. After a failure, export pauses for 30s, so a platform that is down adds no network calls and doesn't slow down exit.
+
+## Content capture
+
+The SDK may set OpenInference `input` / `output` attributes on decorated spans (`capture_input` / `capture_output`, default on). The platform API applies its own policy: with `CONTENT_CAPTURE_ENABLED=false` (default), stored trace bodies may be redacted or omitted even if the SDK sent attributes. Configure both sides explicitly for demos that need bodies in the UI.
+
+## Monorepo note
+
+- SDK PyPI / import / CLI: `opengrade-ai` / `opengrade` / `opengrade`
+- Server PyPI / import / CLI: `opengrade-ai-server` / `opengrade_server` / `opengrade-server` (share one venv via `opengrade-ai[ui]`)
+
+## Manual smoke (not CI-required)
+
+With the platform running and a project created:
+
+**Anthropic**
+
+```bash
+cd sdk && uv sync --extra anthropic --extra dev
+uv pip install anthropic
+uv run python -c "
+import opengrade
+import anthropic
+opengrade.init(project_slug='demo', instrument=['anthropic'])
+c = anthropic.Anthropic()
+@opengrade.trace
+def go():
+    c.messages.create(model='claude-3-5-haiku-20241022', max_tokens=16, messages=[{'role':'user','content':'hi'}])
+go(); opengrade.flush()
+"
+```
+
+**LangChain**
+
+```bash
+cd sdk && uv sync --extra langchain --extra dev
+# install langchain + your LLM backend as needed
+uv run python -c "
+import opengrade
+opengrade.init(project_slug='demo', instrument=['langchain'])
+# invoke your LangChain chain here
+opengrade.flush()
+"
+```
+
+See platform tracing docs: [`docs/04-tracing-and-otel.md`](../docs/04-tracing-and-otel.md).