tracely-ai 0.1.0__tar.gz

tracely_ai-0.1.0/.gitignore

@@ -0,0 +1,29 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+.uv/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+
+# Node / Next
+node_modules/
+.next/
+out/
+.turbo/
+
+# Env / secrets
+.env
+.env.local
+*.local
+
+# Data / infra
+.data/
+*.log
+
+# OS / editor
+.DS_Store
+.idea/
+.vscode/

tracely_ai-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Julien Wuthrich
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tracely_ai-0.1.0/PKG-INFO

@@ -0,0 +1,266 @@
+Metadata-Version: 2.4
+Name: tracely-ai
+Version: 0.1.0
+Summary: Tracely Python SDK: automatic + manual agent tracing, exported to Tracely over OTLP.
+Project-URL: Homepage, https://github.com/Jwuthri/Tracely
+Project-URL: Documentation, https://github.com/Jwuthri/Tracely/blob/master/sdk/README.md
+Project-URL: Source, https://github.com/Jwuthri/Tracely
+Project-URL: Issues, https://github.com/Jwuthri/Tracely/issues
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,evals,llm,observability,opentelemetry,tracing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.10
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.25
+Requires-Dist: opentelemetry-sdk>=1.25
+Provides-Extra: all
+Requires-Dist: anthropic; extra == 'all'
+Requires-Dist: claude-agent-sdk; extra == 'all'
+Requires-Dist: crewai; extra == 'all'
+Requires-Dist: google-adk; extra == 'all'
+Requires-Dist: google-genai>=0.3; extra == 'all'
+Requires-Dist: langchain-openrouter>=0.1.0; extra == 'all'
+Requires-Dist: litellm>=1.40; extra == 'all'
+Requires-Dist: llama-index; extra == 'all'
+Requires-Dist: mistralai>=1.0; extra == 'all'
+Requires-Dist: openai-agents; extra == 'all'
+Requires-Dist: openinference-instrumentation-anthropic>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-bedrock>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-claude-agent-sdk>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-crewai>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-google-adk>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-google-genai>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-groq>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-langchain>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-llama-index>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-mistralai>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-openai-agents>=0.1.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-openai>=0.1.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic; extra == 'anthropic'
+Requires-Dist: openinference-instrumentation-anthropic>=0.1.0; extra == 'anthropic'
+Provides-Extra: bedrock
+Requires-Dist: openinference-instrumentation-bedrock>=0.1.0; extra == 'bedrock'
+Provides-Extra: claude-agent-sdk
+Requires-Dist: claude-agent-sdk; extra == 'claude-agent-sdk'
+Requires-Dist: openinference-instrumentation-claude-agent-sdk>=0.1.0; extra == 'claude-agent-sdk'
+Provides-Extra: crewai
+Requires-Dist: crewai; extra == 'crewai'
+Requires-Dist: openinference-instrumentation-crewai>=0.1.0; extra == 'crewai'
+Provides-Extra: google
+Requires-Dist: google-genai>=0.3; extra == 'google'
+Requires-Dist: openinference-instrumentation-google-genai>=0.1.0; extra == 'google'
+Provides-Extra: google-adk
+Requires-Dist: google-adk; extra == 'google-adk'
+Requires-Dist: openinference-instrumentation-google-adk>=0.1.0; extra == 'google-adk'
+Provides-Extra: google-genai
+Requires-Dist: google-genai>=0.3; extra == 'google-genai'
+Provides-Extra: groq
+Requires-Dist: openinference-instrumentation-groq>=0.1.0; extra == 'groq'
+Provides-Extra: langchain
+Requires-Dist: openinference-instrumentation-langchain>=0.1.0; extra == 'langchain'
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.40; extra == 'litellm'
+Provides-Extra: llama-index
+Requires-Dist: llama-index; extra == 'llama-index'
+Requires-Dist: openinference-instrumentation-llama-index>=0.1.0; extra == 'llama-index'
+Provides-Extra: mistral
+Requires-Dist: mistralai>=1.0; extra == 'mistral'
+Requires-Dist: openinference-instrumentation-mistralai>=0.1.0; extra == 'mistral'
+Provides-Extra: mistral-sdk
+Requires-Dist: mistralai>=1.0; extra == 'mistral-sdk'
+Provides-Extra: openai
+Requires-Dist: openinference-instrumentation-openai>=0.1.0; extra == 'openai'
+Provides-Extra: openai-agents
+Requires-Dist: openai-agents; extra == 'openai-agents'
+Requires-Dist: openinference-instrumentation-openai-agents>=0.1.0; extra == 'openai-agents'
+Provides-Extra: openllmetry
+Requires-Dist: opentelemetry-instrumentation-anthropic>=0.30b0; extra == 'openllmetry'
+Requires-Dist: opentelemetry-instrumentation-openai-v2>=2.0b0; extra == 'openllmetry'
+Provides-Extra: openrouter
+Requires-Dist: langchain-openrouter>=0.1.0; extra == 'openrouter'
+Provides-Extra: openrouter-openai
+Requires-Dist: openai>=1.0; extra == 'openrouter-openai'
+Provides-Extra: xai
+Requires-Dist: openai>=1.0; extra == 'xai'
+Description-Content-Type: text/markdown
+
+# `tracely-ai` — instrument agents + the CI gate CLI
+
+Two things in one small package:
+
+1. an **instrumentation SDK** — `tracely.init()` activates the OpenAI/Anthropic/LangChain/LiteLLM auto-instrumentors so your existing code is traced with **zero span code** (the default path); a custom `SpanProcessor` stamps the active `tracely.trace()` run context onto every span — including the zero-touch provider spans. Manual context managers remain the escape hatch. Everything emits standard `gen_ai.*` / OpenInference attributes **plus** Tracely's first-class `tracely.*` hints, so the backend can populate its agent-semantic columns; and
+2. the **`tracely` CLI** — `tracely gate` and `tracely replay`, which run an agent's promoted regression suite in CI and gate the PR (exit 0/1 + GitHub status/comment).
+
+The core depends only on `opentelemetry-sdk` + the OTLP HTTP exporter (Python ≥ 3.10); a provider extra adds that provider's auto-instrumentor.
+
+```bash
+pip install "tracely-ai[openai]"   # provider extras: [openai] [anthropic] [langchain] [litellm] [all]
+# or: pip install tracely-ai       # core only (manual API + CLI)
+# import name is `tracely_sdk`; the CLI is `tracely` (entry point tracely_sdk.cli:main)
+```
+
+> Already using OpenTelemetry / OpenInference / LangGraph instrumentation? You don't need the instrumentation half — point your existing OTLP exporter at `POST {endpoint}/v1/traces` with `Authorization: Bearer <ingest-key>` and set the `tracely.*` attributes below. This SDK is just the ergonomic path. The **CLI**, however, is how you wire Tracely into CI.
+
+---
+
+## 1. Instrument an agent
+
+### Automatic (the default — zero span code)
+
+`init()` activates the auto-instrumentors; `trace()` attaches the run context; `@observe` adds
+function-level spans. No manual span code.
+
+```python
+import tracely_sdk as tracely
+from openai import OpenAI
+
+tracely.init(endpoint="http://localhost:8000", api_key="tracely_dev_key",
+             service_name="support-agent", env="prod", instrument="auto")
+
+with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
+    OpenAI().chat.completions.create(model="gpt-4o", messages=[...])   # GENERATION span, captured
+
+@tracely.observe(as_type="agent")            # args→input, return→output, auto-nested
+def plan(goal): ...
+```
+
+`instrument` is `"auto"` (every importable provider SDK), an explicit list (`["openai",
+"anthropic"]`), or `False`. The `tracely.trace()` hints flow onto **every** span inside it — including
+the provider spans the instrumentor creates — via a custom `SpanProcessor`. Streaming token usage
+needs `stream_options={"include_usage": True}`.
+
+**Also covered:** LangChain/LangGraph (`[langchain]` — graphs nest, node names become steps), LiteLLM
+(`instrument=["litellm"]` — 100+ providers via one callback), and a non-patching drop-in
+(`from tracely_sdk.openai import OpenAI` / `wrap_openai`). Under `"auto"`, when the LangChain
+instrumentor is present it owns LLM spans and the provider instrumentors are skipped to avoid
+duplicate spans (override with an explicit list). Full guide: the docs [Automatic instrumentation](https://github.com/Jwuthri/Tracely/blob/master/docs/pages/automatic.mdx) page.
+
+### Manual / custom spans (the escape hatch)
+
+For anything the auto path doesn't cover. Each `with` block is a span; nesting builds the tree.
+
+```python
+import tracely_sdk as tracely
+
+tracely.init(endpoint="http://localhost:8000", api_key="tracely_dev_key",
+             service_name="support-agent", env="prod")
+
+with tracely.agent("support-agent", version="v3", conversation="conv-1", turn=0) as a:  # AGENT span = run root
+    tracely.set_io(a, input=user_msg, output=answer)                 # what the agent received / returned
+    with tracely.thinking(agent="support-agent") as th:             # THINKING span (reasoning)
+        tracely.set_io(th, output=reasoning); tracely.set_usage(th, thinking_tokens=120)
+    with tracely.llm("gpt-4o", agent="support-agent") as g:          # GENERATION span
+        tracely.set_io(g, input=messages, output=completion)
+        tracely.set_usage(g, input_tokens=812, output_tokens=96)
+    with tracely.tool("get_order", agent="support-agent") as t:      # TOOL span
+        try:
+            result = get_order(order_id)
+            tracely.set_io(t, input={"order_id": order_id}, output=result)
+        except Exception as e:
+            tracely.error(t, str(e))                                 # level=ERROR → the failure signal
+
+tracely.flush()   # force-flush the exporter (call before the process exits)
+```
+
+### Span context managers — one per observation type
+| Call | Span type | Sets |
+|---|---|---|
+| `agent(slug, *, version, run_id, role, conversation, turn, user, trace_name, handoff_from, edge="delegate")` | `AGENT` (run root) | `tracely.agent.id`/`.version`/`.run_id`/`.role`, `tracely.conversation.id` + `session.id`, `tracely.turn.index`, `tracely.env`; `user`→`tracely.user.id`, `trace_name`→`tracely.trace.name`; `handoff_from`→ a handoff edge (`caller`→this agent, `edge.type`). |
+| `llm(model, *, agent, temperature, top_p, max_tokens, frequency_penalty, presence_penalty, seed, tool_calls, metadata)` | `GENERATION` | `gen_ai.request.model` + the sampling params as `gen_ai.request.*`; `tool_calls`→`tracely.tool_calls` (tools the model **requested**); `metadata`→`tracely.metadata.*`. |
+| `tool(name, *, agent)` | `TOOL` | `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`. |
+| `thinking(name="thinking", *, agent, model)` | `THINKING` | reasoning emitted as its own span; optional `model`. |
+| `retriever(name="retrieve", *, agent)` | `RETRIEVER` | a retrieval step — query in `set_io(input=)`, hits in `set_io(output=)`. |
+| `embedding(model, *, agent)` | `EMBEDDING` | `gen_ai.request.model`; record tokens with `set_usage(input_tokens=)`. |
+| `guardrail(name="guardrail", *, agent)` | `GUARDRAIL` | a safety/policy check — verdict in `set_io(output={"action": "allow"\|"block"})`. |
+| `chain(name, *, agent)` | `CHAIN` | a grouping span (e.g. a RAG pipeline) — nest other spans inside it. |
+| `turn(turn_id, *, index)` / `step(name, *, step_id)` | marker / generic | `tracely.turn.*` / `tracely.step.*`. |
+
+### Annotating spans
+- `set_io(span, *, input=None, output=None)` → `tracely.input` / `tracely.output` (objects are JSON-encoded; message content is a `{role, content:[blocks]}` object or a content-block list).
+- `set_usage(span, *, input_tokens=None, output_tokens=None, thinking_tokens=None)` → `gen_ai.usage.input_tokens` / `output_tokens` / `reasoning_tokens`.
+- `set_metadata(span, **kv)` → `tracely.metadata.<key>` — arbitrary tags (e.g. prompt version, tenant), surfaced in the span's Metadata and searchable.
+- `error(span, message="")` → marks the span `StatusCode.ERROR` (→ `level=ERROR` in Tracely) — this is *the* failure-detection signal.
+- `flush()` → force-flush the OTLP exporter.
+
+### What Tracely reads
+Standard `gen_ai.*` / OpenInference attributes, plus first-class hints that become **indexed columns** on the span row: `tracely.agent.id`/`.version`/`.role`, `tracely.user.id`, `tracely.trace.name`, `tracely.conversation.id`, `tracely.turn.id`/`.index`, `tracely.step.id`/`.name`, `tracely.observation.type`, `tracely.tool_calls`, `tracely.handoff.*` + `tracely.edge.type`, the `gen_ai.request.*` sampling params, and `tracely.env` (`prod|staging|ci|dev` — the gating axis). Agent slug + version are auto-registered into the Postgres registry on ingest.
+
+### Declaring a conversation's agent catalog
+Pass `agents=[{name, description, tools:[...]}]` to `tracely.trace(...)` (or call `set_agents([...])`) to declare the agents/tools a conversation uses — emitted once as `tracely.agents`. The backend stores this per conversation; the UI's **Conversation Agents** panel renders it (with per-tool run counts) and the LLM judge can read it as `@LIST_AGENT`. Without it, Tracely still derives the agent view from the spans.
+
+---
+
+## 2. Hermetic replay (`call_tool` / `call_llm` / `fixtures`)
+
+These let the **same agent code** run live in production and deterministically offline in CI. Wrap each external call:
+
+```python
+def run(user_input: str):
+    with tracely.agent("support-agent"):
+        # In prod: calls the real fn and records the output. In replay: serves the recorded output.
+        order = tracely.call_tool("get_order", lambda: get_order(order_id), args={"order_id": order_id})
+        answer = tracely.call_llm("gpt-4o", lambda: chat(messages), input=messages, usage=(812, 96))
+        return answer
+```
+
+- **Production (`--live` or no fixtures active):** `call_tool`/`call_llm` invoke your `fn`, record the output (and any error) on the span, and return it.
+- **CI replay:** `tracely replay` activates the case's recorded **fixture bundle** via `with tracely.fixtures(bundle): ...`; `call_tool`/`call_llm` then **serve the recorded outputs in order** (or by `args` match) and never call your `fn`. A call that errored in production is reproduced on the span **and raised as `tracely.ToolError`**, so the agent's own `try/except` runs exactly as it would live — and the gate sees the same failure condition.
+
+**Auto-instrument / drop-in code replays too** (no manual seam required): inside a `fixtures()` block Tracely class-patches the provider's create-method, so code that calls the SDK directly — `client.chat.completions.create(...)` under `instrument="auto"` or the `tracely_sdk.openai` drop-in — is served the recorded completion (reconstructed into a provider-shaped response) and never hits the network. Covered today: **OpenAI `chat.completions`** and **Anthropic `messages`**; other providers fall back to live in replay until added. The manual `call_llm` seam remains the way to get provider-agnostic hermetic replay everywhere.
+
+This is what makes replay deterministic, offline, and free (no API keys, no cost). See [regression-testing design](https://github.com/Jwuthri/Tracely/blob/master/design/part2-tracely/05-regression-testing.md).
+
+---
+
+## 3. The CI gate CLI
+
+```bash
+tracely gate   <agent> [--env ci] [--api …] [--key …] [--pr N] [--sha …] [--github]
+tracely replay <agent> (--entrypoint module:func | --cmd "…") [--live] [--github]
+```
+
+- **`tracely gate <agent>`** — gate a PR against **pre-emitted** `env=ci` traces (your CI already ran the agent and emitted traces); cases are matched to candidates by `input_digest`.
+- **`tracely replay <agent>`** — re-run the agent **on each promoted case's recorded input** (fetched from `GET /api/gate/suite`), then gate. `--entrypoint module:func` calls a Python function per case; `--cmd "…"` runs a shell command per case (gets `TRACELY_INPUT`) that emits its own trace. Hermetic by default; `--live` makes real tool/LLM calls.
+
+Both **exit 0 (PASS) / 1 (FAIL)** and, inside GitHub Actions (or with `--github`), post a **commit status + PR comment** with per-case results and soft warnings (latency/token deltas vs the last green gate). `--dry-run` prints the GitHub calls instead of sending; `--no-github` never touches GitHub. Config via flags or env (`TRACELY_API`, `TRACELY_KEY`, `TRACELY_AGENT`, `TRACELY_GATE_ENV`, `TRACELY_WEB_URL`, `GITHUB_TOKEN`). A reusable composite action lives at `.github/actions/tracely-gate/`.
+
+---
+
+## Examples (`sdk/examples/` + `sdk/example.py`)
+
+[`examples/README.md`](examples/README.md) is the full index — **one runnable file per way of
+tracing**, all the same fake-DB **two-agent conversation** (a Support Agent that hands the pricing
+turn off to a Billing Agent, declaring its catalog via `tracely.trace(agents=...)`): each frontier
+provider (OpenAI, Anthropic, Gemini, Mistral, Bedrock) + OpenRouter, each harness (LangChain
+`create_agent`, LangGraph, LiteLLM, LlamaIndex, CrewAI), each first-party agent SDK (OpenAI Agents,
+Claude Agent SDK, Google ADK), and each approach (`@observe`+`trace`, the `wrap_openai`/`wrap_anthropic`
+drop-ins, manual spans). Highlights:
+
+| File | Shows |
+|---|---|
+| `../example.py` | the minimal demo trace (agent → llm → failing tool). `make sdk-example`. |
+| `examples/auto_openai.py` · `auto_anthropic.py` · `auto_gemini.py` · … | **automatic** provider tracing — zero span code (one file per frontier provider + OpenRouter). |
+| `examples/auto_langchain.py` (`create_agent`) · `auto_langgraph.py` · `auto_litellm.py` · … | **automatic** harness tracing (one file per framework, current APIs). |
+| `examples/auto_openai_agents.py` · `auto_claude_agent.py` · `auto_google_adk.py` | **automatic** first-party agent-SDK tracing (OpenAI Agents / Claude Agent SDK / Google ADK). |
+| `examples/auto_agent.py` | **automatic** `@observe` + `trace()` agent → thinking/gen/tool tree. `make auto-agent`. |
+| `examples/dropin_openai.py` · `dropin_anthropic.py` | non-patching `wrap_openai` / `wrap_anthropic` drop-ins. |
+| `examples/manual_spans.py` | the manual escape-hatch API as a full agent (no provider/key needed). |
+| `examples/weather_agent.py` / `weather_agent_cli.py` | a real agent wired with `call_tool`/`call_llm` for `tracely replay --entrypoint` / `--cmd`. |
+| `examples/seed_conversations.py` | rich demo data using **every** SDK helper — single/multi-turn, multi-agent + handoffs, RAG (guardrail→embed→retrieve→chain), thinking, multimodal, structured output, multi-model. `make seed-demo`. |
+| `examples/seed_multiturn.py` | one multi-turn conversation (manual API, no key) — the showcase for the **rolling summary** + **declared agents** (`@LIST_AGENT`). |
+| `examples/seed_regression.py` | promote a failing trace → run red→green CI gates (fills Cases + Gates). `make seed-regression`. |
+| `examples/seed_multicall.py` / `seed_handler.py` | repeated-call + handler examples for fixture replay. |
+
+---
+
+## Key decisions (and why)
+
+1. **Thin wrapper, not a framework.** It only sets attributes on OTel spans — anyone already on OpenTelemetry/OpenInference can skip it and just emit the `tracely.*` hints. No lock-in.
+2. **First-class `tracely.*` hints.** Agent/conversation/turn/step/env are emitted as semantic attributes so they become indexed columns server-side — the basis for agent-level evals and PR gating.
+3. **`THINKING` is a span type, not a field.** Reasoning is its own observation, so it renders distinctly and carries its own token usage.
+4. **Record-replay in the SDK.** `call_tool`/`call_llm` are the seam that makes "the recorded run is the test" real: the same code path records in prod and replays in CI, reproducing recorded errors via `ToolError` so error-handling behaviour is gated faithfully.
+5. **The CLI is the CI contract.** `gate`/`replay` exit 0/1 and speak GitHub — so wiring Tracely into a pipeline is one step, with the hard gate being fail-to-pass and everything else advisory.

tracely_ai-0.1.0/README.md

@@ -0,0 +1,176 @@
+# `tracely-ai` — instrument agents + the CI gate CLI
+
+Two things in one small package:
+
+1. an **instrumentation SDK** — `tracely.init()` activates the OpenAI/Anthropic/LangChain/LiteLLM auto-instrumentors so your existing code is traced with **zero span code** (the default path); a custom `SpanProcessor` stamps the active `tracely.trace()` run context onto every span — including the zero-touch provider spans. Manual context managers remain the escape hatch. Everything emits standard `gen_ai.*` / OpenInference attributes **plus** Tracely's first-class `tracely.*` hints, so the backend can populate its agent-semantic columns; and
+2. the **`tracely` CLI** — `tracely gate` and `tracely replay`, which run an agent's promoted regression suite in CI and gate the PR (exit 0/1 + GitHub status/comment).
+
+The core depends only on `opentelemetry-sdk` + the OTLP HTTP exporter (Python ≥ 3.10); a provider extra adds that provider's auto-instrumentor.
+
+```bash
+pip install "tracely-ai[openai]"   # provider extras: [openai] [anthropic] [langchain] [litellm] [all]
+# or: pip install tracely-ai       # core only (manual API + CLI)
+# import name is `tracely_sdk`; the CLI is `tracely` (entry point tracely_sdk.cli:main)
+```
+
+> Already using OpenTelemetry / OpenInference / LangGraph instrumentation? You don't need the instrumentation half — point your existing OTLP exporter at `POST {endpoint}/v1/traces` with `Authorization: Bearer <ingest-key>` and set the `tracely.*` attributes below. This SDK is just the ergonomic path. The **CLI**, however, is how you wire Tracely into CI.
+
+---
+
+## 1. Instrument an agent
+
+### Automatic (the default — zero span code)
+
+`init()` activates the auto-instrumentors; `trace()` attaches the run context; `@observe` adds
+function-level spans. No manual span code.
+
+```python
+import tracely_sdk as tracely
+from openai import OpenAI
+
+tracely.init(endpoint="http://localhost:8000", api_key="tracely_dev_key",
+             service_name="support-agent", env="prod", instrument="auto")
+
+with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
+    OpenAI().chat.completions.create(model="gpt-4o", messages=[...])   # GENERATION span, captured
+
+@tracely.observe(as_type="agent")            # args→input, return→output, auto-nested
+def plan(goal): ...
+```
+
+`instrument` is `"auto"` (every importable provider SDK), an explicit list (`["openai",
+"anthropic"]`), or `False`. The `tracely.trace()` hints flow onto **every** span inside it — including
+the provider spans the instrumentor creates — via a custom `SpanProcessor`. Streaming token usage
+needs `stream_options={"include_usage": True}`.
+
+**Also covered:** LangChain/LangGraph (`[langchain]` — graphs nest, node names become steps), LiteLLM
+(`instrument=["litellm"]` — 100+ providers via one callback), and a non-patching drop-in
+(`from tracely_sdk.openai import OpenAI` / `wrap_openai`). Under `"auto"`, when the LangChain
+instrumentor is present it owns LLM spans and the provider instrumentors are skipped to avoid
+duplicate spans (override with an explicit list). Full guide: the docs [Automatic instrumentation](https://github.com/Jwuthri/Tracely/blob/master/docs/pages/automatic.mdx) page.
+
+### Manual / custom spans (the escape hatch)
+
+For anything the auto path doesn't cover. Each `with` block is a span; nesting builds the tree.
+
+```python
+import tracely_sdk as tracely
+
+tracely.init(endpoint="http://localhost:8000", api_key="tracely_dev_key",
+             service_name="support-agent", env="prod")
+
+with tracely.agent("support-agent", version="v3", conversation="conv-1", turn=0) as a:  # AGENT span = run root
+    tracely.set_io(a, input=user_msg, output=answer)                 # what the agent received / returned
+    with tracely.thinking(agent="support-agent") as th:             # THINKING span (reasoning)
+        tracely.set_io(th, output=reasoning); tracely.set_usage(th, thinking_tokens=120)
+    with tracely.llm("gpt-4o", agent="support-agent") as g:          # GENERATION span
+        tracely.set_io(g, input=messages, output=completion)
+        tracely.set_usage(g, input_tokens=812, output_tokens=96)
+    with tracely.tool("get_order", agent="support-agent") as t:      # TOOL span
+        try:
+            result = get_order(order_id)
+            tracely.set_io(t, input={"order_id": order_id}, output=result)
+        except Exception as e:
+            tracely.error(t, str(e))                                 # level=ERROR → the failure signal
+
+tracely.flush()   # force-flush the exporter (call before the process exits)
+```
+
+### Span context managers — one per observation type
+| Call | Span type | Sets |
+|---|---|---|
+| `agent(slug, *, version, run_id, role, conversation, turn, user, trace_name, handoff_from, edge="delegate")` | `AGENT` (run root) | `tracely.agent.id`/`.version`/`.run_id`/`.role`, `tracely.conversation.id` + `session.id`, `tracely.turn.index`, `tracely.env`; `user`→`tracely.user.id`, `trace_name`→`tracely.trace.name`; `handoff_from`→ a handoff edge (`caller`→this agent, `edge.type`). |
+| `llm(model, *, agent, temperature, top_p, max_tokens, frequency_penalty, presence_penalty, seed, tool_calls, metadata)` | `GENERATION` | `gen_ai.request.model` + the sampling params as `gen_ai.request.*`; `tool_calls`→`tracely.tool_calls` (tools the model **requested**); `metadata`→`tracely.metadata.*`. |
+| `tool(name, *, agent)` | `TOOL` | `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`. |
+| `thinking(name="thinking", *, agent, model)` | `THINKING` | reasoning emitted as its own span; optional `model`. |
+| `retriever(name="retrieve", *, agent)` | `RETRIEVER` | a retrieval step — query in `set_io(input=)`, hits in `set_io(output=)`. |
+| `embedding(model, *, agent)` | `EMBEDDING` | `gen_ai.request.model`; record tokens with `set_usage(input_tokens=)`. |
+| `guardrail(name="guardrail", *, agent)` | `GUARDRAIL` | a safety/policy check — verdict in `set_io(output={"action": "allow"\|"block"})`. |
+| `chain(name, *, agent)` | `CHAIN` | a grouping span (e.g. a RAG pipeline) — nest other spans inside it. |
+| `turn(turn_id, *, index)` / `step(name, *, step_id)` | marker / generic | `tracely.turn.*` / `tracely.step.*`. |
+
+### Annotating spans
+- `set_io(span, *, input=None, output=None)` → `tracely.input` / `tracely.output` (objects are JSON-encoded; message content is a `{role, content:[blocks]}` object or a content-block list).
+- `set_usage(span, *, input_tokens=None, output_tokens=None, thinking_tokens=None)` → `gen_ai.usage.input_tokens` / `output_tokens` / `reasoning_tokens`.
+- `set_metadata(span, **kv)` → `tracely.metadata.<key>` — arbitrary tags (e.g. prompt version, tenant), surfaced in the span's Metadata and searchable.
+- `error(span, message="")` → marks the span `StatusCode.ERROR` (→ `level=ERROR` in Tracely) — this is *the* failure-detection signal.
+- `flush()` → force-flush the OTLP exporter.
+
+### What Tracely reads
+Standard `gen_ai.*` / OpenInference attributes, plus first-class hints that become **indexed columns** on the span row: `tracely.agent.id`/`.version`/`.role`, `tracely.user.id`, `tracely.trace.name`, `tracely.conversation.id`, `tracely.turn.id`/`.index`, `tracely.step.id`/`.name`, `tracely.observation.type`, `tracely.tool_calls`, `tracely.handoff.*` + `tracely.edge.type`, the `gen_ai.request.*` sampling params, and `tracely.env` (`prod|staging|ci|dev` — the gating axis). Agent slug + version are auto-registered into the Postgres registry on ingest.
+
+### Declaring a conversation's agent catalog
+Pass `agents=[{name, description, tools:[...]}]` to `tracely.trace(...)` (or call `set_agents([...])`) to declare the agents/tools a conversation uses — emitted once as `tracely.agents`. The backend stores this per conversation; the UI's **Conversation Agents** panel renders it (with per-tool run counts) and the LLM judge can read it as `@LIST_AGENT`. Without it, Tracely still derives the agent view from the spans.
+
+---
+
+## 2. Hermetic replay (`call_tool` / `call_llm` / `fixtures`)
+
+These let the **same agent code** run live in production and deterministically offline in CI. Wrap each external call:
+
+```python
+def run(user_input: str):
+    with tracely.agent("support-agent"):
+        # In prod: calls the real fn and records the output. In replay: serves the recorded output.
+        order = tracely.call_tool("get_order", lambda: get_order(order_id), args={"order_id": order_id})
+        answer = tracely.call_llm("gpt-4o", lambda: chat(messages), input=messages, usage=(812, 96))
+        return answer
+```
+
+- **Production (`--live` or no fixtures active):** `call_tool`/`call_llm` invoke your `fn`, record the output (and any error) on the span, and return it.
+- **CI replay:** `tracely replay` activates the case's recorded **fixture bundle** via `with tracely.fixtures(bundle): ...`; `call_tool`/`call_llm` then **serve the recorded outputs in order** (or by `args` match) and never call your `fn`. A call that errored in production is reproduced on the span **and raised as `tracely.ToolError`**, so the agent's own `try/except` runs exactly as it would live — and the gate sees the same failure condition.
+
+**Auto-instrument / drop-in code replays too** (no manual seam required): inside a `fixtures()` block Tracely class-patches the provider's create-method, so code that calls the SDK directly — `client.chat.completions.create(...)` under `instrument="auto"` or the `tracely_sdk.openai` drop-in — is served the recorded completion (reconstructed into a provider-shaped response) and never hits the network. Covered today: **OpenAI `chat.completions`** and **Anthropic `messages`**; other providers fall back to live in replay until added. The manual `call_llm` seam remains the way to get provider-agnostic hermetic replay everywhere.
+
+This is what makes replay deterministic, offline, and free (no API keys, no cost). See [regression-testing design](https://github.com/Jwuthri/Tracely/blob/master/design/part2-tracely/05-regression-testing.md).
+
+---
+
+## 3. The CI gate CLI
+
+```bash
+tracely gate   <agent> [--env ci] [--api …] [--key …] [--pr N] [--sha …] [--github]
+tracely replay <agent> (--entrypoint module:func | --cmd "…") [--live] [--github]
+```
+
+- **`tracely gate <agent>`** — gate a PR against **pre-emitted** `env=ci` traces (your CI already ran the agent and emitted traces); cases are matched to candidates by `input_digest`.
+- **`tracely replay <agent>`** — re-run the agent **on each promoted case's recorded input** (fetched from `GET /api/gate/suite`), then gate. `--entrypoint module:func` calls a Python function per case; `--cmd "…"` runs a shell command per case (gets `TRACELY_INPUT`) that emits its own trace. Hermetic by default; `--live` makes real tool/LLM calls.
+
+Both **exit 0 (PASS) / 1 (FAIL)** and, inside GitHub Actions (or with `--github`), post a **commit status + PR comment** with per-case results and soft warnings (latency/token deltas vs the last green gate). `--dry-run` prints the GitHub calls instead of sending; `--no-github` never touches GitHub. Config via flags or env (`TRACELY_API`, `TRACELY_KEY`, `TRACELY_AGENT`, `TRACELY_GATE_ENV`, `TRACELY_WEB_URL`, `GITHUB_TOKEN`). A reusable composite action lives at `.github/actions/tracely-gate/`.
+
+---
+
+## Examples (`sdk/examples/` + `sdk/example.py`)
+
+[`examples/README.md`](examples/README.md) is the full index — **one runnable file per way of
+tracing**, all the same fake-DB **two-agent conversation** (a Support Agent that hands the pricing
+turn off to a Billing Agent, declaring its catalog via `tracely.trace(agents=...)`): each frontier
+provider (OpenAI, Anthropic, Gemini, Mistral, Bedrock) + OpenRouter, each harness (LangChain
+`create_agent`, LangGraph, LiteLLM, LlamaIndex, CrewAI), each first-party agent SDK (OpenAI Agents,
+Claude Agent SDK, Google ADK), and each approach (`@observe`+`trace`, the `wrap_openai`/`wrap_anthropic`
+drop-ins, manual spans). Highlights:
+
+| File | Shows |
+|---|---|
+| `../example.py` | the minimal demo trace (agent → llm → failing tool). `make sdk-example`. |
+| `examples/auto_openai.py` · `auto_anthropic.py` · `auto_gemini.py` · … | **automatic** provider tracing — zero span code (one file per frontier provider + OpenRouter). |
+| `examples/auto_langchain.py` (`create_agent`) · `auto_langgraph.py` · `auto_litellm.py` · … | **automatic** harness tracing (one file per framework, current APIs). |
+| `examples/auto_openai_agents.py` · `auto_claude_agent.py` · `auto_google_adk.py` | **automatic** first-party agent-SDK tracing (OpenAI Agents / Claude Agent SDK / Google ADK). |
+| `examples/auto_agent.py` | **automatic** `@observe` + `trace()` agent → thinking/gen/tool tree. `make auto-agent`. |
+| `examples/dropin_openai.py` · `dropin_anthropic.py` | non-patching `wrap_openai` / `wrap_anthropic` drop-ins. |
+| `examples/manual_spans.py` | the manual escape-hatch API as a full agent (no provider/key needed). |
+| `examples/weather_agent.py` / `weather_agent_cli.py` | a real agent wired with `call_tool`/`call_llm` for `tracely replay --entrypoint` / `--cmd`. |
+| `examples/seed_conversations.py` | rich demo data using **every** SDK helper — single/multi-turn, multi-agent + handoffs, RAG (guardrail→embed→retrieve→chain), thinking, multimodal, structured output, multi-model. `make seed-demo`. |
+| `examples/seed_multiturn.py` | one multi-turn conversation (manual API, no key) — the showcase for the **rolling summary** + **declared agents** (`@LIST_AGENT`). |
+| `examples/seed_regression.py` | promote a failing trace → run red→green CI gates (fills Cases + Gates). `make seed-regression`. |
+| `examples/seed_multicall.py` / `seed_handler.py` | repeated-call + handler examples for fixture replay. |
+
+---
+
+## Key decisions (and why)
+
+1. **Thin wrapper, not a framework.** It only sets attributes on OTel spans — anyone already on OpenTelemetry/OpenInference can skip it and just emit the `tracely.*` hints. No lock-in.
+2. **First-class `tracely.*` hints.** Agent/conversation/turn/step/env are emitted as semantic attributes so they become indexed columns server-side — the basis for agent-level evals and PR gating.
+3. **`THINKING` is a span type, not a field.** Reasoning is its own observation, so it renders distinctly and carries its own token usage.
+4. **Record-replay in the SDK.** `call_tool`/`call_llm` are the seam that makes "the recorded run is the test" real: the same code path records in prod and replays in CI, reproducing recorded errors via `ToolError` so error-handling behaviour is gated faithfully.
+5. **The CLI is the CI contract.** `gate`/`replay` exit 0/1 and speak GitHub — so wiring Tracely into a pipeline is one step, with the hard gate being fail-to-pass and everything else advisory.

tracely_ai-0.1.0/example.py

@@ -0,0 +1,35 @@
+"""Emit the demo agent trace via the Tracely SDK.
+
+    uv run python sdk/example.py     (API + worker must be running)
+"""
+
+import tracely_sdk as tracely
+
+tracely.init(endpoint="http://localhost:8000", api_key="tracely_dev_key", service_name="demo-agent")
+
+with tracely.agent("planner", version="v1"):
+    with tracely.llm("gpt-4o") as g:
+        tracely.set_io(
+            g,
+            input=[
+                {"role": "system", "content": "You are a helpful weather assistant."},
+                {"role": "user",   "content": "weather in SF?"},
+            ],
+            output={
+                "role": "assistant",
+                "content": None,
+                "finish_reason": "tool_calls",
+                "tool_calls": [{
+                    "id": "call_1",
+                    "type": "function",
+                    "function": {"name": "get_weather", "arguments": '{"city": "SF"}'},
+                }],
+            },
+        )
+        tracely.set_usage(g, input_tokens=812, output_tokens=96)
+    with tracely.tool("get_weather") as t:
+        tracely.set_io(t, input={"city": "SF"})
+        tracely.error(t, "upstream timeout")  # demo failure signal (level=ERROR)
+
+tracely.flush()
+print("sent demo trace via SDK")

tracely_ai-0.1.0/examples/README.md

@@ -0,0 +1,81 @@
+# `sdk/examples/` — one runnable example per way of tracing
+
+Every example is the **same realistic two-agent conversation**: a **Support Agent** answers
+order/inventory questions over several turns by calling tools against a fake e-commerce DB
+([`_fake_db.py`](_fake_db.py)) — `get_order_status` + `check_inventory` — then **hands off** the final
+pricing-comparison turn to a **Billing Agent** (`compare_prices`). So each file shows a real
+multi-turn, multi-agent tool-calling loop (plus thinking, in the `@observe`/manual ones), not a toy
+single call. Each guards on its instrumentor + API key, printing setup hints instead of crashing when
+a dependency or key is missing.
+
+Each run also declares its two-agent catalog once via `tracely.trace(agents=AGENTS)` (so the
+Conversation Agents panel + the judge's `@LIST_AGENT` see it) and tags itself with its own filename
+via `example=os.path.basename(__file__)`, so each span carries `tracely.metadata.example = <file>.py`
+— filter on it in the Tracely UI to find the traces a given example produced.
+
+```bash
+pip install "tracely-ai[<extra>]"     # the extra named in each file's header
+export TRACELY_API=http://localhost:8000           # your Tracely API (default shown)
+export OPENAI_API_KEY=sk-...                        # (or the provider's key)
+uv run python sdk/examples/<file>.py
+```
+
+## Ways of sending traces (the SDK layers)
+
+| File | Layer | Shows |
+|---|---|---|
+| [`auto_agent.py`](auto_agent.py) | L2 + L3 | `@observe` agent + `trace()` → **AGENT → thinking · 2 generations · 2 tools** tree |
+| [`dropin_openai.py`](dropin_openai.py) | R13 | non-patching `wrap_openai` — the tool-calling agent, nothing patched globally |
+| [`dropin_anthropic.py`](dropin_anthropic.py) | R13 | non-patching `wrap_anthropic` — the Claude tool-calling agent |
+| [`manual_spans.py`](manual_spans.py) | L4 | the manual escape hatch — full agent: thinking → llm → tools (one **errors**) → answer (no key needed) |
+
+## Frontier providers (L1 — auto-instrumented, zero span code)
+
+| File | `instrument=` | Extra | Key |
+|---|---|---|---|
+| [`auto_openai.py`](auto_openai.py) | `["openai"]` | `[openai]` | `OPENAI_API_KEY` |
+| [`auto_anthropic.py`](auto_anthropic.py) | `["anthropic"]` | `[anthropic]` | `ANTHROPIC_API_KEY` |
+| [`auto_gemini.py`](auto_gemini.py) | `["gemini"]` | `[google]` | `GEMINI_API_KEY` |
+| [`auto_mistral.py`](auto_mistral.py) | `["mistral"]` | `[mistral]` | `MISTRAL_API_KEY` |
+| [`auto_bedrock.py`](auto_bedrock.py) | `["bedrock"]` | `[bedrock]` | AWS creds + `AWS_REGION` |
+| [`auto_openrouter.py`](auto_openrouter.py) | `["langchain"]` | `[langchain,openrouter]` | `OPENROUTER_API_KEY` |
+
+> **OpenRouter** routes one API to 100+ models. `auto_openrouter.py` uses LangChain's first-party
+> `ChatOpenRouter` (`langchain-openrouter`) inside `create_agent`, traced by the LangChain
+> instrumentor. (OpenRouter is also OpenAI-wire-compatible, so pointing the OpenAI SDK at its
+> `base_url` works too — traced by the OpenAI instrumentor.)
+
+## Harnesses (L1 — orchestration frameworks)
+
+| File | `instrument=` | Extra | Agent pattern (current API) |
+|---|---|---|---|
+| [`auto_langchain.py`](auto_langchain.py) | `["langchain"]` | `[langchain]` | `langchain.agents.create_agent` (LangChain 1.0+; replaces `AgentExecutor`/`create_react_agent`) |
+| [`auto_langgraph.py`](auto_langgraph.py) | `["langchain"]` | `[langchain]` | a custom `StateGraph` + `ToolNode` + `tools_condition` (hand-built ReAct loop) |
+| [`auto_litellm.py`](auto_litellm.py) | `["litellm"]` | `[litellm]` | OpenAI-shaped tool-calling loop via one callback |
+| [`auto_llama_index.py`](auto_llama_index.py) | `["llama-index"]` | `[llama-index]` | `ReActAgent` over `FunctionTool`s |
+| [`auto_crewai.py`](auto_crewai.py) | `["crewai"]` | `[crewai]` | a `Crew` whose agent is equipped with the tools |
+
+> `instrument="auto"` activates whichever of these are importable; when a harness instrumentor (e.g.
+> LangChain) is present it owns the LLM spans and the provider instrumentors are skipped to avoid
+> duplicate spans (override with an explicit list). See the docs [Automatic instrumentation](../../docs/pages/automatic.mdx) page.
+
+## Agent frameworks — first-party SDKs (L1)
+
+The big labs now ship their own agent harnesses; each has an OpenInference instrumentor that
+`init(instrument=[...])` activates, emitting AGENT/TOOL/LLM spans to Tracely.
+
+| File | `instrument=` | Extra (+ SDK) | Framework |
+|---|---|---|---|
+| [`auto_openai_agents.py`](auto_openai_agents.py) | `["openai-agents"]` | `[openai-agents]` + `openai-agents` | OpenAI Agents SDK (`agents`: `Agent`/`Runner`/`@function_tool`) |
+| [`auto_claude_agent.py`](auto_claude_agent.py) | `["claude-agent-sdk"]` | `[claude-agent-sdk]` + `claude-agent-sdk` | Anthropic Claude Agent SDK (`@tool`/`create_sdk_mcp_server`/`ClaudeSDKClient`; needs the Claude Code CLI) |
+| [`auto_google_adk.py`](auto_google_adk.py) | `["google-adk"]` | `[google-adk]` + `google-adk` | Google ADK (`google.adk.agents.Agent` + `InMemoryRunner`; instrument **before** importing `google.adk`) |
+
+## Demo data & CI-gate examples (not tracing how-tos)
+
+| File | Shows |
+|---|---|
+| [`seed_conversations.py`](seed_conversations.py) | rich manual-API demo data — every observation type (`make seed-demo`) |
+| [`seed_multiturn.py`](seed_multiturn.py) | one multi-turn conversation via the manual API (no key) — the showcase for the **rolling summary** + **declared agents** (`tracely.trace(agents=...)`) |
+| [`seed_regression.py`](seed_regression.py) | promote a failing trace → red→green CI gates (`make seed-regression`) |
+| [`seed_multicall.py`](seed_multicall.py) / [`seed_handler.py`](seed_handler.py) | repeated-call + handler fixtures for hermetic replay |
+| [`weather_agent.py`](weather_agent.py) / [`weather_agent_cli.py`](weather_agent_cli.py) | a real agent wired for `tracely replay --entrypoint` / `--cmd` |