otelzero 0.2.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,1345 @@
1
+ Metadata-Version: 2.4
2
+ Name: otelzero
3
+ Version: 0.2.0
4
+ Summary: Instrument agent frameworks (CrewAI, LangChain, LangGraph, LlamaIndex, OpenAI Agents SDK, Pydantic AI, smolagents, Google ADK, AWS Strands, Claude Agent SDK, and more) via Python import hooks and stream telemetry to a Kiji Agent Registry server.
5
+ License-Expression: Apache-2.0
6
+ Project-URL: Homepage, https://github.com/575-lab/otelzero
7
+ Project-URL: Repository, https://github.com/575-lab/otelzero
8
+ Project-URL: Kiji Agent Registry, https://github.com/575-lab/kiji-agent-registry
9
+ Requires-Python: >=3.12
10
+ Description-Content-Type: text/markdown
11
+ License-File: LICENSE
12
+ Requires-Dist: httpx>=0.26
13
+ Requires-Dist: wrapt>=1.16
14
+ Provides-Extra: crewai
15
+ Requires-Dist: crewai==0.1.0; extra == "crewai"
16
+ Provides-Extra: otel
17
+ Requires-Dist: opentelemetry-api>=1.24; extra == "otel"
18
+ Requires-Dist: opentelemetry-sdk>=1.24; extra == "otel"
19
+ Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.24; extra == "otel"
20
+ Provides-Extra: langchain
21
+ Requires-Dist: langchain>=1.0; extra == "langchain"
22
+ Requires-Dist: langsmith[otel]>=0.4.25; extra == "langchain"
23
+ Dynamic: license-file
24
+
25
+ # otelzero
26
+
27
+ Instrument agent frameworks via Python **import hooks** and stream telemetry to a
28
+ [Kiji Agent Registry](https://github.com/575-lab/kiji-agent-registry) server — no code changes to your
29
+ agents required. (Import package: `otelzero`.)
30
+
31
+ **Site:** [575-lab.github.io/otelzero](https://575-lab.github.io/otelzero/) — an animated walkthrough of the import hook, with live Kiji and Dataiku Agent Management demos.
32
+
33
+ Install monitoring for your framework *before* importing it, then run as usual:
34
+
35
+ ```python
36
+ from otelzero import install_crewai_monitoring
37
+
38
+ install_crewai_monitoring(server_url="http://localhost:8000")
39
+ from crewai import Agent, Crew, Task # now auto-instrumented
40
+ ```
41
+
42
+ > Order matters: install the hook **before** importing the framework. The hooks
43
+ > use the `find_spec` import protocol and require **Python ≥ 3.12**.
44
+
45
+ ## Supported SDKs / hooks
46
+
47
+ For exactly which agent actions each framework reports (agent runs, model calls,
48
+ tool calls, workflows) and whether they come from the framework's own GenAI
49
+ telemetry or from an otelzero hook, see **[FRAMEWORKS.md](FRAMEWORKS.md)**. For the span attributes each framework emits, see the
50
+ **[attribute matrix](https://575-lab.github.io/otelzero/attributes.html)**.
51
+
52
+ | Framework | Install function | What gets wrapped |
53
+ |---|---|---|
54
+ | [CrewAI](https://github.com/crewAIInc/crewAI) | `install_crewai_monitoring` | `Crew.kickoff`, `Agent.execute_task`, `Task.execute_sync` |
55
+ | [LangChain](https://github.com/langchain-ai/langchain) | `install_langchain_monitoring` | nothing — switches on LangSmith's *native* OpenTelemetry export (translated into the current GenAI conventions on export) |
56
+ | [LangGraph](https://github.com/langchain-ai/langgraph) | `install_langgraph_monitoring` | nothing — same LangSmith export; one run per graph run, subgraphs included in it |
57
+ | [LlamaIndex](https://github.com/run-llama/llama_index) | `install_llama_index_monitoring` | `Workflow.run` (covers `FunctionAgent`, `ReActAgent`, `CodeActAgent`, `AgentWorkflow` and hand-written workflows) |
58
+ | [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) | `install_openai_agents_monitoring` | `Runner.run`, `Runner.run_sync`, `Runner.run_streamed` |
59
+ | [Pydantic AI](https://ai.pydantic.dev) | `install_pydantic_ai_monitoring` | nothing — configures Pydantic AI's *native* OpenTelemetry (`Agent.instrument_all`) |
60
+ | [smolagents](https://github.com/huggingface/smolagents) | `install_smolagents_monitoring` | `MultiStepAgent.run` (covers `CodeAgent` and `ToolCallingAgent`, direct + streaming) |
61
+ | [AWS Strands Agents](https://github.com/strands-agents/sdk-python) | `install_strands_monitoring` | nothing — points Strands' *native* OpenTelemetry at the registry (latest GenAI conventions) |
62
+ | [Google ADK](https://github.com/google/adk-python) | `install_adk_monitoring` | nothing — points ADK's *native* OpenTelemetry at the registry (latest GenAI conventions) |
63
+ | [Databricks Agent Bricks / Mosaic AI](https://docs.databricks.com/aws/en/generative-ai/agent-bricks) | `install_databricks_monitoring` | each `ResponsesAgent` / `ChatAgent` subclass's `predict`, plus its tool calls |
64
+ | [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-python) | `install_claude_agent_monitoring` | `query`, `ClaudeSDKClient.receive_response` (+ `ClaudeSDKClient.query`, for its prompt) |
65
+ | [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) | `install_microsoft_monitoring` | nothing — switches on Agent Framework's *native* OpenTelemetry (`enable_instrumentation`) |
66
+ | [Snowflake Cortex Agent SDK](https://docs.snowflake.com/en/user-guide/cortex-code-agent-sdk/cortex-code-agent-sdk) | `install_snowflake_monitoring` | `query`, `CortexCodeSDKClient.receive_response` (+ `CortexCodeSDKClient.query`, for its prompt) |
67
+ | Anything else | `install_monitoring` | framework-agnostic manual instrumentation (`TelemetryClient`) |
68
+
69
+ **Native-OTel frameworks.** Pydantic AI, AWS Strands Agents, Google ADK and
70
+ Microsoft Agent Framework already trace themselves in the
71
+ [GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai),
72
+ so their install functions wrap nothing. Each one switches on the framework's
73
+ own instrumentation, exports it to the registry's OTLP receiver, stamps
74
+ otelzero's run identity on every span, and derives the usual `agent_start` /
75
+ `agent_end` / `agent_error` events from the framework's `invoke_agent` spans.
76
+ LangChain and LangGraph work the same way through **LangSmith's OpenTelemetry
77
+ export**, whose spans otelzero translates into the current conventions on
78
+ export, with the events coming
79
+ from LangChain's own callbacks (see
80
+ [the LangGraph guide](#instrument-a-real-langgraph-graph-and-langchain-10-agents)).
81
+ For all of these, the import order doesn't matter. The `otel` extra is required,
82
+ and their tracing stays on even with `OTELZERO_OTEL=0`.
83
+
84
+ **LangChain 1.0 note:** `create_agent` (and the older `create_react_agent`) return
85
+ a compiled LangGraph graph. Both installers set up the same LangSmith export, so
86
+ either one captures it. Installing both is safe and still reports one run per
87
+ graph run.
88
+
89
+ Every hook streams the same event envelope (`agent_start` / `agent_end` /
90
+ `agent_error`, plus crew/task events where the framework has them) to the otelzero
91
+ dashboard. With the `otel` extra installed, the hooks also emit standard
92
+ OpenTelemetry spans — **tracing is on by default** (`OTELZERO_OTEL` is
93
+ effectively always `1`; set `OTELZERO_OTEL=0` to switch the import hooks' spans
94
+ off) — and a framework's own native OTel spans (model/tool calls) nest
95
+ underneath otelzero's. Set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=1`
96
+ to also record each run's prompt and response as `gen_ai.input.messages` /
97
+ `gen_ai.output.messages` (on the span and in the dashboard's event details);
98
+ off by default, capped by `OTELZERO_CONTENT_MAX_CHARS` (default 4000 per part).
99
+
100
+ ## Install
101
+
102
+ ```bash
103
+ pip install otelzero # core client
104
+ pip install "otelzero[otel]" # + OpenTelemetry span export
105
+ ```
106
+
107
+ ## Set up a server
108
+
109
+ The client needs a Kiji Agent Registry server (dashboard + telemetry ingest) to send events to.
110
+
111
+ **Self-hosted (minimal):** install the `kiji-agent-registry` distribution and run it:
112
+
113
+ ```bash
114
+ pip install kiji-agent-registry # add [otel] for the OTLP ingest endpoint
115
+ kiji-agent-registry # dashboard + ingest on http://localhost:8000
116
+ ```
117
+
118
+ **In your own cloud:** `marketplace/` in the project repo packages the server for the
119
+ GCP, AWS and Azure marketplaces — one VM plus a data disk inside your own project,
120
+ account or subscription. The listings are still in publisher review; until they land,
121
+ the Docker Compose and container paths in the project README cover the same ground.
122
+ Point the client at whatever URL your deployment serves:
123
+
124
+ ```python
125
+ install_crewai_monitoring(
126
+ server_url="https://kiji.your-vpc.internal",
127
+ api_key="your-ingest-key", # or set OTELZERO_API_KEY
128
+ )
129
+ ```
130
+
131
+ **Managed:** [Dataiku Agent Management](https://www.dataiku.com) runs it for you —
132
+ agent registry, drift detection and governance.
133
+
134
+ Ingest is owner-scoped: pass an ingest key via
135
+ `install_*_monitoring(..., api_key="...")` or the `OTELZERO_API_KEY` env var (any
136
+ string ≥ 8 chars auto-provisions its own owner on a fresh server).
137
+
138
+ Docker images, OTLP/collector ingest, and the mobile companion apps are covered
139
+ in the [project README](https://github.com/575-lab/kiji-agent-registry).
140
+
141
+ ## Try it against a local registry
142
+
143
+ Run a [Kiji Agent Registry](https://github.com/575-lab/kiji-agent-registry) next to this checkout, then drive it with the
144
+ framework-free simulator:
145
+
146
+ ```bash
147
+ git clone https://github.com/575-lab/kiji-agent-registry ../kiji-agent-registry
148
+ (cd ../kiji-agent-registry && uv run kiji-agent-registry) # terminal 1
149
+ uv run python demo/demo_simulator.py # terminal 2
150
+ ```
151
+
152
+ Open http://localhost:8000 to watch the runs arrive. The commands below assume
153
+ the registry checkout sits next to this one at `../kiji-agent-registry`.
154
+
155
+ ## Framework guides
156
+
157
+ ### Instrument a real CrewAI crew
158
+
159
+ The import hook wraps the **real** `crewai` package, so install monitoring
160
+ **before** importing CrewAI — then write your crew normally, with no telemetry
161
+ code in it:
162
+
163
+ ```python
164
+ from otelzero import install_crewai_monitoring
165
+
166
+ # 1) Install the hook FIRST (registers a find_spec meta-path finder)
167
+ install_crewai_monitoring(server_url="http://localhost:8000", user="Sarah")
168
+
169
+ # 2) Now import CrewAI — Crew/Agent/Task are auto-instrumented as they load
170
+ from crewai import Agent, Crew, Task
171
+
172
+ researcher = Agent(role="Research Analyst", goal="...", backstory="...")
173
+ task = Task(description="...", expected_output="...", agent=researcher)
174
+ crew = Crew(agents=[researcher], tasks=[task])
175
+
176
+ crew.kickoff() # streams crew/agent/task events to the dashboard automatically
177
+ ```
178
+
179
+ A complete, runnable version is in **`demo/demo_crewai.py`**:
180
+
181
+ ```bash
182
+ uv pip install crewai # the hook instruments the real package
183
+ export OPENAI_API_KEY=... # for a full kickoff
184
+ uv run python demo/demo_crewai.py
185
+ ```
186
+
187
+ > Order matters: instrumentation must be installed **before** `crewai` is
188
+ > imported. The hooks use the `find_spec` protocol and require **Python ≥ 3.12**.
189
+
190
+ ### Instrument a real LangGraph graph (and LangChain 1.0 agents)
191
+
192
+ LangChain and LangGraph trace themselves through **LangSmith**, and the
193
+ `langsmith` SDK can export those traces as OpenTelemetry spans
194
+ (`langsmith[otel]`). Every graph run, node, subgraph, chain, model and tool call
195
+ becomes one span, named after the run (`Research Graph`, `model`, `tools`, a
196
+ tool's name). It carries `langsmith.span.kind`, `gen_ai.operation.name`,
197
+ `gen_ai.request.model`, `gen_ai.usage.*` and `langsmith.metadata.*`.
198
+ `install_langgraph_monitoring` and `install_langchain_monitoring` wrap nothing.
199
+ They:
200
+
201
+ - make otelzero's OTLP provider the global one, which is where LangSmith's
202
+ exporter sends its spans. On their way out they pass through otelzero's
203
+ translator ([below](#how-langsmiths-spans-are-translated)):
204
+ the root run becomes `invoke_agent {run}`, model calls `chat {model}`, tool
205
+ calls `execute_tool {tool}`. If your application set its own global tracer
206
+ provider first, they go out through that one, untranslated;
207
+ - turn the export on through LangSmith's own variables: `LANGSMITH_TRACING`,
208
+ `LANGSMITH_OTEL_ENABLED` and, unless a LangSmith API key is configured,
209
+ `LANGSMITH_OTEL_ONLY` (so nothing goes to LangSmith's cloud). Unless content
210
+ capture is on, they also set `LANGSMITH_HIDE_INPUTS` / `LANGSMITH_HIDE_OUTPUTS`
211
+ (LangSmith otherwise exports every prompt and completion). Values you set
212
+ yourself are kept, and an uninstall removes only what was added;
213
+ - register a callback handler through LangChain's own `register_configure_hook`.
214
+ LangChain adds it to every run the way it adds its own tracers. It sends one
215
+ `agent_start` / `agent_end` / `agent_error` per **top-level** run (the
216
+ outermost graph, agent, chain or runnable you invoked), with the model, the
217
+ tools it called and its token usage collected from the model and tool runs
218
+ under it. Nodes and subgraphs are part of the run, so one graph run is one
219
+ agent card. The session is titled from the run's input.
220
+
221
+ ```python
222
+ from otelzero import install_langgraph_monitoring
223
+ from langgraph.graph import END, START, MessagesState, StateGraph # any order
224
+
225
+ install_langgraph_monitoring(server_url="http://localhost:8000", user="Sarah")
226
+
227
+ graph = StateGraph(MessagesState)
228
+ graph.add_node("agent", call_model)
229
+ graph.add_edge(START, "agent")
230
+ graph.add_edge("agent", END)
231
+ app = graph.compile(name="Research Graph")
232
+
233
+ app.invoke({"messages": [("user", "who won?")]}) # one agent card per run
234
+ ```
235
+
236
+ **Why the events come from callbacks.** LangSmith builds its spans on a
237
+ background thread, when it flushes its queue. A finished run's span is created
238
+ and ended in one go, before its children's spans exist. The callbacks fire in
239
+ your code's context, as the run happens, so the events are live.
240
+
241
+ **Identity.** A run names its own thread and user in its metadata:
242
+ `config={"metadata": {"thread_id": ..., "user_name": ...}}` (also
243
+ `conversation_id` / `session_id`, `user_id` / `user`), or LangGraph's
244
+ `config={"configurable": {"thread_id": ...}}`. Its events and its spans both
245
+ use that, so the dashboard groups them together. Without metadata, the events
246
+ fall back to the active `run_context()`, then to the client's `user=` /
247
+ `session=`. The spans fall back to the client's only: a span can't be tied
248
+ back to the run it records, so `run_context()` doesn't reach them.
249
+
250
+ **Short scripts:** call `langchain_core.tracers.langchain.wait_for_all_tracers()`
251
+ before `otel.force_flush()` at exit, so LangSmith's queue is drained into spans
252
+ first.
253
+
254
+ Token usage comes from the model calls a run makes (their `LLMResult`). A graph
255
+ node that returns prepared messages without calling a model reports none.
256
+
257
+ A complete, runnable version is in **`demo/demo_langgraph.py`** (fully offline —
258
+ its nodes return prepared replies instead of calling a model, and it contains a
259
+ subgraph so you can see it counted as part of its parent's run):
260
+
261
+ ```bash
262
+ uv pip install langgraph "langsmith[otel]" "otelzero[otel]"
263
+ uv run python demo/demo_langgraph.py
264
+ ```
265
+
266
+ ### Instrument a real OpenAI Agents SDK agent
267
+
268
+ The [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) (the
269
+ `openai-agents` package, OpenAI's production successor to the archived Swarm)
270
+ starts every run through one of three `Runner` classmethods, and
271
+ `install_openai_agents_monitoring` wraps all three — `run`, `run_sync` and
272
+ `run_streamed` — so a run is captured exactly once whichever you use:
273
+
274
+ ```python
275
+ from otelzero import install_openai_agents_monitoring
276
+
277
+ # 1) Install the hook FIRST (registers a find_spec meta-path finder)
278
+ install_openai_agents_monitoring(server_url="http://localhost:8000", user="Sarah")
279
+
280
+ # 2) Now import the SDK — Runner is auto-instrumented as it loads
281
+ from agents import Agent, Runner
282
+
283
+ agent = Agent(name="Weather Agent", instructions="...", model="gpt-4o-mini")
284
+
285
+ await Runner.run(agent, "What is the weather in Amsterdam?")
286
+ ```
287
+
288
+ `run_sync` drives `run` underneath, so both are wrapped but only the outer call
289
+ opens a span. A streamed run is traced across its `stream_events()` pass, since
290
+ that is when the run is actually delivered.
291
+
292
+ The `agent_start` details carry the run's `model`, `tools` and `handoffs`; token
293
+ counts come from the run context's `usage`. A complete, runnable version is in
294
+ **`demo/demo_openai_agents.py`**:
295
+
296
+ ```bash
297
+ uv pip install openai-agents # the hook instruments the real package
298
+ export OPENAI_API_KEY=... # the SDK has no offline model
299
+ uv run python demo/demo_openai_agents.py
300
+ ```
301
+
302
+ > Order matters: instrumentation must be installed **before** `agents` is
303
+ > imported. The hooks use the `find_spec` protocol and require **Python ≥ 3.12**.
304
+
305
+ ### Instrument a real Pydantic AI agent
306
+
307
+ [Pydantic AI](https://ai.pydantic.dev) ships its own OpenTelemetry
308
+ instrumentation, already in the GenAI semantic conventions: one
309
+ `invoke_agent {name}` span per run with `chat` and `execute_tool` children,
310
+ `gen_ai.*` usage attributes and opt-in message content. That is richer than a
311
+ wrapper around `Agent.run` could see from the outside, so
312
+ `install_pydantic_ai_monitoring` wraps nothing — it points the native
313
+ instrumentation (`Agent.instrument_all`) at the registry's OTLP receiver,
314
+ stamps otelzero's run identity (`gen_ai.conversation.id`, `user.name`,
315
+ `otelzero.session.name`) onto every span, and turns each native `invoke_agent`
316
+ span into the usual `agent_start` / `agent_end` / `agent_error` events:
317
+
318
+ ```python
319
+ from otelzero import install_pydantic_ai_monitoring
320
+ from pydantic_ai import Agent # import order does not matter for this one
321
+
322
+ install_pydantic_ai_monitoring(
323
+ server_url="http://localhost:8000",
324
+ user="Sarah",
325
+ name="Order support triage", # session title, on every event and span
326
+ )
327
+
328
+ agent = Agent("openai:gpt-4o", name="Support Agent")
329
+
330
+ result = await agent.run("What is the status of order A-1042?")
331
+ async with agent.run_stream("...") as stream: # streaming runs too
332
+ ...
333
+ ```
334
+
335
+ The events are derived from the spans, so this hook needs the `otel` extra, and
336
+ its tracing stays on even with `OTELZERO_OTEL=0`. They carry what
337
+ Pydantic AI has recorded: `model` at start; `tools` and `usage` at the end;
338
+ and on failure the exception message and traceback only when content capture is
339
+ on (otherwise just the exception type). Pydantic AI's spans hold no prompt when
340
+ a run starts, so — unlike the wrapping hooks — no session title is derived from
341
+ it: pass `name=` (or `run_context(name=...)`) to title the session. Prompts and
342
+ completions follow the usual `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`
343
+ opt-in; pass `settings=InstrumentationSettings(...)` to choose a different
344
+ format version or content policy.
345
+
346
+ A complete, runnable version is in **`demo/demo_pydantic_ai.py`** (fully offline —
347
+ it uses Pydantic AI's own `TestModel`, so no API key is needed):
348
+
349
+ ```bash
350
+ uv pip install pydantic-ai "otelzero[otel]"
351
+ uv run python demo/demo_pydantic_ai.py
352
+ uv run python demo/demo_pydantic_ai.py --model openai:gpt-4o # a real model
353
+ ```
354
+
355
+
356
+ ### Instrument a real smolagents agent
357
+
358
+ Hugging Face's [smolagents](https://github.com/huggingface/smolagents) builds
359
+ both of its agent flavours — `CodeAgent`, which writes its actions as Python, and
360
+ `ToolCallingAgent`, which emits structured tool calls — on one `MultiStepAgent`
361
+ base, and neither overrides `run`. So `install_smolagents_monitoring` wraps that
362
+ single method and covers every agent the library can build:
363
+
364
+ ```python
365
+ from otelzero import install_smolagents_monitoring
366
+
367
+ # 1) Install the hook FIRST (registers a find_spec meta-path finder)
368
+ install_smolagents_monitoring(server_url="http://localhost:8000", user="Sarah")
369
+
370
+ # 2) Now import smolagents — MultiStepAgent.run is auto-instrumented as it loads
371
+ from smolagents import CodeAgent, InferenceClientModel
372
+
373
+ agent = CodeAgent(tools=[...], model=InferenceClientModel(), name="Travel Agent")
374
+
375
+ agent.run("How long is the drive from Paris to Lyon?") # streams agent events
376
+ for step in agent.run("...", stream=True): # the streaming shape too
377
+ ...
378
+ ```
379
+
380
+ `run` is a synchronous dispatcher whose `stream` argument decides its return
381
+ shape — the final answer, or a generator of steps — and otelzero traces whichever
382
+ it is. Unlike a framework that hands back an awaitable, the direct call performs
383
+ the whole run before returning, so the streaming decision is read off the
384
+ argument rather than the returned object.
385
+
386
+ Token counts come from the agent's own monitor, which totals them **across** the
387
+ runs it has seen. otelzero reports the difference across each run, so a
388
+ continuation (`agent.run(..., reset=False)`) is charged only for what it actually
389
+ spent rather than for the whole conversation so far.
390
+
391
+ A complete, runnable version is in **`demo/demo_smolagents.py`**:
392
+
393
+ ```bash
394
+ uv pip install smolagents # the hook instruments the real package
395
+ export HF_TOKEN=... # the default InferenceClientModel needs it
396
+ uv run python demo/demo_smolagents.py
397
+ ```
398
+
399
+ > Order matters: instrumentation must be installed **before** `smolagents` is
400
+ > imported. The hooks use the `find_spec` protocol and require **Python ≥ 3.12**.
401
+
402
+ ### Instrument a real LlamaIndex agent
403
+
404
+ Every modern [LlamaIndex](https://github.com/run-llama/llama_index) agent —
405
+ `FunctionAgent`, `ReActAgent`, `CodeActAgent`, the multi-agent `AgentWorkflow` —
406
+ is a *workflow*: each overrides `run` only to package its arguments into a start
407
+ event and then calls `Workflow.run`. So `install_llama_index_monitoring` wraps
408
+ that one method and covers every agent the library can build, plus any workflow
409
+ you write by hand:
410
+
411
+ ```python
412
+ from otelzero import install_llama_index_monitoring
413
+
414
+ # 1) Install the hook FIRST (registers a find_spec meta-path finder)
415
+ install_llama_index_monitoring(server_url="http://localhost:8000", user="Sarah")
416
+
417
+ # 2) Now import LlamaIndex — Workflow.run is auto-instrumented as it loads
418
+ from llama_index.core.agent.workflow import FunctionAgent
419
+ from llama_index.llms.openai import OpenAI
420
+
421
+ agent = FunctionAgent(name="Support Agent", tools=[lookup_order], llm=OpenAI(model="gpt-4o-mini"))
422
+
423
+ await agent.run(user_msg="What is the status of order A-1042?") # streams agent events
424
+ ```
425
+
426
+ `Workflow.run` is **synchronous**: it schedules an asyncio task and hands back a
427
+ `WorkflowHandler` while the run continues in the background. otelzero therefore
428
+ opens the run when `run()` is called and closes it from a done-callback on that
429
+ task — so a run is captured whether you await the handler, read only
430
+ `stream_events()`, or cancel it with `cancel_run()` (which closes the run as
431
+ `interrupted` rather than leaving it dangling). An agent or workflow started from
432
+ inside a tool folds into the run it belongs to, the way a LangGraph subgraph is
433
+ part of its parent's run.
434
+
435
+ Token counts do not ride on the result — `AgentOutput` has no usage — so otelzero
436
+ registers one handler on LlamaIndex's own instrumentation dispatcher and sums the
437
+ `LLMChatEndEvent` / `LLMCompletionEndEvent` payloads a run produces. Predict and
438
+ structured-predict events are deliberately not counted: each wraps an inner chat
439
+ call that is already counted, and a `CustomLLM` that reports one payload through
440
+ two events is counted once.
441
+
442
+ Three limitations worth knowing:
443
+
444
+ - a **human-in-the-loop** run that is never resumed never closes, so its card
445
+ stays open;
446
+ - **OpenAI streaming reports no tokens** unless the model is built with
447
+ `additional_kwargs={"stream_options": {"include_usage": True}}` (or
448
+ `streaming=False`) — and `FunctionAgent` streams by default;
449
+ - LLM events that arrive **after** a run has closed are dropped.
450
+
451
+ A complete, runnable version is in **`demo/demo_llama_index.py`**:
452
+
453
+ ```bash
454
+ uv pip install llama-index-core # the hook instruments the real package
455
+ uv run python demo/demo_llama_index.py # fully offline
456
+ uv run python demo/demo_llama_index.py --model gpt-4o-mini # a real model
457
+ ```
458
+
459
+ The offline run uses LlamaIndex's own `MockFunctionCallingLLM`, which returns no
460
+ provider payload — and a provider payload is where token counts live — so the
461
+ offline cards show runs without tokens. `--model` gives real counts.
462
+
463
+ > Order matters: instrumentation must be installed **before** `llama_index` is
464
+ > imported. The hooks use the `find_spec` protocol and require **Python ≥ 3.12**.
465
+
466
+ ### Instrument a real AWS Strands Agents agent
467
+
468
+ [AWS Strands Agents](https://github.com/strands-agents/sdk-python) always traces
469
+ itself through the global OpenTelemetry tracer provider, in the GenAI semantic
470
+ conventions. Each run gets an `invoke_agent {name}` span, with
471
+ `execute_event_loop_cycle`, `chat` and `execute_tool {tool}` spans under it,
472
+ plus `gen_ai.*` model, tool and usage attributes. `install_strands_monitoring`
473
+ wraps nothing. It:
474
+
475
+ - makes otelzero's OTLP provider the global one, so Strands' spans export to the
476
+ registry's `/v1/traces`;
477
+ - opts Strands into the **latest** GenAI conventions
478
+ (`OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`: `gen_ai.provider.name`,
479
+ `gen_ai.input.messages` / `gen_ai.output.messages`);
480
+ - applies otelzero's content opt-in. By default Strands records prompts and
481
+ completions unconditionally. With content capture off, the hook turns on
482
+ Strands' redaction (`gen_ai_unredacted_attributes=`). With
483
+ `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=1`, the content goes on
484
+ span attributes (`gen_ai_span_attributes_only`) together with tool definitions;
485
+ - stamps otelzero's run identity on every Strands span, and turns each
486
+ `invoke_agent` span into `agent_start` / `agent_end` / `agent_error`.
487
+
488
+ Opt-in tokens you set yourself are kept, and an uninstall removes only the
489
+ ones the hook added.
490
+
491
+ ```python
492
+ from otelzero import install_strands_monitoring
493
+ from strands import Agent, tool # import order does not matter for this one
494
+
495
+ install_strands_monitoring(
496
+ server_url="http://localhost:8000",
497
+ user="Sarah",
498
+ name="Calculator session", # session title, on every event and span
499
+ )
500
+
501
+ @tool
502
+ def add(a: int, b: int) -> int:
503
+ """Add two integers."""
504
+ return a + b
505
+
506
+ agent = Agent(name="Calculator Agent", tools=[add])
507
+
508
+ agent("What is 21 + 21?") # Strands' own spans -> registry; agent_* events derived from them
509
+ ```
510
+
511
+ Strands' agent span carries no prompt when a run starts, so no session title
512
+ is derived from the prompt. Pass `name=` (or `run_context(name=...)`) to set
513
+ one.
514
+
515
+ A complete, runnable version is in **`demo/demo_strands.py`**:
516
+
517
+ ```bash
518
+ uv pip install strands-agents "otelzero[otel]"
519
+ # configure AWS Bedrock credentials for a full run (Strands' default model)
520
+ uv run python demo/demo_strands.py
521
+ ```
522
+
523
+ ### Instrument a real Google ADK agent
524
+
525
+ [Google ADK](https://github.com/google/adk-python) (Agent Development Kit) also
526
+ traces itself through the global tracer provider. A `Runner` run opens an
527
+ `invoke_workflow {app}` span. Each agent that takes a turn gets an
528
+ `invoke_agent {name}` span under it, with `generate_content {model}` and
529
+ `execute_tool {tool}` spans nested inside. `install_adk_monitoring` wraps
530
+ nothing. It makes otelzero's OTLP provider the global one, opts ADK into the
531
+ latest GenAI conventions (`gen_ai_latest_experimental`, which replaces ADK's
532
+ legacy `gcp.vertex.agent.*` payload attributes), stamps `user.name` and
533
+ `otelzero.session.name` on every ADK span, and turns each `invoke_agent` span
534
+ into `agent_start` / `agent_end` / `agent_error`. A multi-agent run reports
535
+ each agent that takes a turn.
536
+
537
+ ```python
538
+ from otelzero import install_adk_monitoring
539
+ from google.adk.agents import Agent # import order does not matter
540
+ from google.adk.runners import InMemoryRunner
541
+
542
+ install_adk_monitoring(server_url="http://localhost:8000", user="Sarah")
543
+
544
+ agent = Agent(name="calculator_agent", model="gemini-3.7-flash", tools=[...])
545
+ runner = InMemoryRunner(agent=agent, app_name="my-app")
546
+
547
+ # Each runner.run / run_async: ADK's own spans -> registry, agent_* events from them
548
+ ```
549
+
550
+ **Session.** ADK already writes its own session id as `gen_ai.conversation.id`,
551
+ after each span starts, so otelzero can't replace it. It is also the right
552
+ value: an ADK session *is* the conversation, and you choose it when you call
553
+ `run_async(session_id=...)`. So ADK runs are grouped by ADK's session id, the
554
+ events are filed under it too, and the `session=` argument only applies to a
555
+ run whose spans carry none.
556
+
557
+ **Content.** ADK reads `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` on
558
+ its own terms. `1` / `true` mean `EVENT_ONLY` there (log records, not spans).
559
+ Set `SPAN_ONLY` or `SPAN_AND_EVENT` to put prompts and completions on the spans
560
+ the registry ingests.
561
+
562
+ A complete, runnable version is in **`demo/demo_adk.py`**:
563
+
564
+ ```bash
565
+ uv pip install google-adk "otelzero[otel]"
566
+ export GOOGLE_API_KEY=... # for a full run (ADK's default Gemini model)
567
+ uv run python demo/demo_adk.py
568
+ ```
569
+
570
+ ### Instrument a real Databricks Agent Bricks / Mosaic AI agent
571
+
572
+ Databricks agents ([Agent Bricks](https://docs.databricks.com/aws/en/generative-ai/agent-bricks),
573
+ the Mosaic AI Agent Framework) are authored as MLflow `ResponsesAgent`
574
+ subclasses (the current recommended interface; the older `ChatAgent` is
575
+ supported identically), where `predict` is an **abstract** method each agent
576
+ implements — so there's no single concrete method on the base to wrap.
577
+ `install_databricks_monitoring` intercepts subclass creation and wraps each
578
+ concrete agent's `predict` as it's defined, composing on top of MLflow's own
579
+ validation wrapper:
580
+
581
+ ```python
582
+ from otelzero import install_databricks_monitoring
583
+
584
+ # 1) Install the hook FIRST (it patches ResponsesAgent.__init_subclass__)
585
+ install_databricks_monitoring(server_url="http://localhost:8000", user="Sarah")
586
+
587
+ # 2) Now define your agent — its predict is auto-instrumented at class creation
588
+ from mlflow.pyfunc import ResponsesAgent
589
+ from mlflow.types.responses import ResponsesAgentRequest, ResponsesAgentResponse
590
+
591
+ class MyAgent(ResponsesAgent):
592
+ def predict(self, request: ResponsesAgentRequest) -> ResponsesAgentResponse:
593
+ ... # each call streams agent_start/agent_end/agent_error to the dashboard
594
+ ```
595
+
596
+ A complete, runnable version is in **`demo/demo_databricks.py`** (fully offline —
597
+ its agent answers by calling two local stand-in tools, so no Databricks workspace
598
+ or LLM key is needed):
599
+
600
+ ```bash
601
+ uv pip install mlflow # provides mlflow.pyfunc.ResponsesAgent
602
+ uv run python demo/demo_databricks.py
603
+ ```
604
+
605
+ Beyond the agent run, the tools a `ResponsesAgent` calls — recorded in its
606
+ response as `function_call` items (with results in matching
607
+ `function_call_output` items) — are surfaced as nested `task_start`/`task_end`
608
+ events, one per tool, on the agent's timeline. So the demo run shows
609
+ `agent_start` → a task per tool → `agent_end`, with no extra agent code.
610
+
611
+ A production Agent Bricks agent (Knowledge Assistant, Information Extraction,
612
+ Multi-Agent Supervisor, Custom LLM) is the same `ResponsesAgent` interface and is
613
+ instrumented identically. Only `predict` is wrapped (the required entry point);
614
+ `predict_stream` is left alone to avoid double-counting agents whose `predict`
615
+ aggregates it internally.
616
+
617
+ ### Instrument a real Claude Agent SDK agent
618
+
619
+ The [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-python)
620
+ exposes two async-generator entry points, and `install_claude_agent_monitoring`
621
+ wraps **both** so a run is captured exactly once whichever style you use: the
622
+ one-shot `claude_agent_sdk.query(...)` function, and the interactive
623
+ `ClaudeSDKClient.receive_response()` method. `ClaudeSDKClient.query()` is wrapped
624
+ too — not as a run, but so the prompt it sends is recorded on the
625
+ `receive_response()` that answers it. (Note: the SDK's own `hooks=` option
626
+ — PreToolUse/PostToolUse — is unrelated to otelzero's import hooks.)
627
+
628
+ ```python
629
+ from otelzero import install_claude_agent_monitoring
630
+
631
+ # 1) Install the hook FIRST (registers a find_spec meta-path finder)
632
+ install_claude_agent_monitoring(server_url="http://localhost:8000", user="Sarah")
633
+
634
+ # 2) Now import the SDK — query and ClaudeSDKClient are auto-instrumented
635
+ from claude_agent_sdk import query, ClaudeAgentOptions
636
+
637
+ options = ClaudeAgentOptions(model="claude-opus-4-8", allowed_tools=["Read"])
638
+
639
+ # Each query() run streams agent_start/agent_end/agent_error events
640
+ async for message in query(prompt="Summarize README.md", options=options):
641
+ ...
642
+ ```
643
+
644
+ A complete, runnable version is in **`demo/demo_claude_sdk.py`**:
645
+
646
+ ```bash
647
+ uv pip install claude-agent-sdk # the hook instruments the real package
648
+ export ANTHROPIC_API_KEY=... # for a full run
649
+ uv run python demo/demo_claude_sdk.py
650
+ ```
651
+
652
+ The `agent_start` details carry the run's `model`, `allowed_tools`, and (names
653
+ only) `mcp_servers`. With tracing on (the default), the SDK's own spans nest
654
+ underneath otelzero's `invoke_agent Claude Agent` span.
655
+
656
+ > Order matters: instrumentation must be installed **before** `claude_agent_sdk`
657
+ > is imported. The hooks use the `find_spec` protocol and require **Python ≥ 3.12**.
658
+
659
+ ### Instrument a real Microsoft Agent Framework agent
660
+
661
+ [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) (the
662
+ `agent-framework` package that unifies Semantic Kernel and AutoGen) ships
663
+ OpenTelemetry instrumentation in the GenAI semantic conventions. Every
664
+ `Agent.run`, streaming or not, gets an `invoke_agent {name}` span, with `chat`
665
+ and `execute_tool` spans from its chat client and function-invocation layers.
666
+ `install_microsoft_monitoring` wraps nothing. It makes otelzero's OTLP provider
667
+ the global one and calls `agent_framework.observability.enable_instrumentation`
668
+ with `enable_sensitive_data` following otelzero's content opt-in. It keeps the
669
+ framework on the latest conventions even when `OTEL_SEMCONV_STABILITY_OPT_IN` is
670
+ set for another library, stamps otelzero's run identity on every span, and
671
+ turns each `invoke_agent` span into `agent_start` / `agent_end` / `agent_error`:
672
+
673
+ ```python
674
+ from otelzero import install_microsoft_monitoring
675
+ from agent_framework import Agent # import order does not matter
676
+ from agent_framework.openai import OpenAIChatClient
677
+
678
+ install_microsoft_monitoring(server_url="http://localhost:8000", user="Sarah")
679
+
680
+ agent = Agent(OpenAIChatClient(model="gpt-4o"), name="Weather Agent")
681
+
682
+ await agent.run("What's the weather in Amsterdam?") # non-streaming
683
+ async for update in agent.run("...", stream=True): # streaming
684
+ ...
685
+ # both paths: the framework's own spans -> registry, agent_* events from them
686
+ ```
687
+
688
+ If the application called `disable_instrumentation()`, the hook keeps that
689
+ choice and says so. With sensitive data off, a failed run's `error` is the
690
+ exception type: that's all Agent Framework records then.
691
+
692
+ A complete, runnable version is in **`demo/demo_microsoft.py`** (fully offline —
693
+ it drives the agent with a tiny local echo chat client, so no Azure/OpenAI key is
694
+ needed; it exercises both the non-streaming and streaming run paths):
695
+
696
+ ```bash
697
+ uv pip install agent-framework "otelzero[otel]"
698
+ uv run python demo/demo_microsoft.py
699
+ ```
700
+
701
+ ### Instrument a real Snowflake Cortex Agent SDK agent
702
+
703
+ Snowflake's [Cortex Code Agent SDK](https://docs.snowflake.com/en/user-guide/cortex-code-agent-sdk/cortex-code-agent-sdk)
704
+ (the `cortex_code_agent_sdk` package) is shaped almost identically to the Claude
705
+ Agent SDK: a module-level `query(prompt, options=...)` async-generator function
706
+ and an interactive `CortexCodeSDKClient` whose `receive_response()` is an async
707
+ generator, both configured by a `CortexCodeAgentOptions` dataclass.
708
+ `install_snowflake_monitoring` wraps **both** run chokepoints so a run is captured
709
+ exactly once whichever style you use, and `CortexCodeSDKClient.query()` so the
710
+ prompt it sends is recorded on the `receive_response()` that answers it:
711
+
712
+ ```python
713
+ from otelzero import install_snowflake_monitoring
714
+
715
+ # 1) Install the hook FIRST (registers a find_spec meta-path finder)
716
+ install_snowflake_monitoring(server_url="http://localhost:8000", user="Sarah")
717
+
718
+ # 2) Now import the SDK — query and CortexCodeSDKClient are auto-instrumented
719
+ from cortex_code_agent_sdk import query, CortexCodeAgentOptions
720
+
721
+ options = CortexCodeAgentOptions(model="claude-sonnet-4-6", cwd=".")
722
+
723
+ # Each query() run streams agent_start/agent_end/agent_error events
724
+ async for message in query(prompt="Summarize this codebase", options=options):
725
+ ...
726
+ ```
727
+
728
+ A complete, runnable version is in **`demo/demo_snowflake.py`**:
729
+
730
+ ```bash
731
+ uv pip install cortex-code-agent-sdk # the hook instruments the real package
732
+ # authenticate the Snowflake CLI connection the SDK uses (e.g. `snow connection add ...`)
733
+ uv run python demo/demo_snowflake.py
734
+ ```
735
+
736
+ The `agent_start` details carry the run's `model`, `allowed_tools`, and (names
737
+ only) `mcp_servers`. With tracing on (the default), the SDK's own spans nest
738
+ underneath otelzero's `invoke_agent Snowflake Cortex Agent` span.
739
+
740
+ > Order matters: instrumentation must be installed **before** `cortex_code_agent_sdk`
741
+ > is imported. The hooks use the `find_spec` protocol and require **Python ≥ 3.12**.
742
+
743
+
744
+ ## OpenTelemetry Tracing
745
+
746
+ otelzero can emit **standard OpenTelemetry traces** alongside (or instead of) the
747
+ built-in telemetry. The same import hooks that stream dashboard events also open
748
+ spans, so a crew run becomes a real distributed trace:
749
+
750
+ - `Crew.kickoff` → an `invoke_workflow` span, with each `Agent.execute_task` and
751
+ `Task.execute_sync` as **nested child spans** (real parent/child via context).
752
+ - An error sets the span status to `ERROR`, records the exception, and sets
753
+ `error.type`.
754
+ - Attributes follow the [GenAI semantic
755
+ conventions](https://github.com/open-telemetry/semantic-conventions-genai) —
756
+ see [OTel span attributes](#otel-span-attributes).
757
+
758
+ It's **on by default** and dependency-light: install the `otel` extra and the
759
+ spans flow (`OTELZERO_OTEL` is effectively always `1`). Without the extra, the
760
+ client runs exactly as before, with zero OpenTelemetry deps. `OTELZERO_OTEL=0`
761
+ switches the import hooks' spans off; the native-OTel hooks (Pydantic AI, Strands,
762
+ ADK, Microsoft, LangChain / LangGraph) keep tracing, because spans are how they
763
+ report.
764
+
765
+ ```bash
766
+ uv sync --extra otel # opentelemetry-api / sdk / exporter-otlp
767
+ uv run python demo/demo_crewai.py
768
+ ```
769
+
770
+ By default spans are exported to **the registry's own OTLP receiver** at `POST /v1/traces`,
771
+ which translates them back into the dashboard's logs and stats — so the existing
772
+ UI keeps working. To send traces to a standard backend instead (Jaeger, Tempo,
773
+ Grafana, an OpenTelemetry Collector), point the exporter elsewhere with the
774
+ standard env var:
775
+
776
+ ```bash
777
+ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
778
+ uv run python demo/demo_crewai.py
779
+ ```
780
+
781
+ > **Dual-emit note:** with tracing on (the default), each run reaches Kiji's own
782
+ > server both as events (`/api/telemetry`) **and** as spans (`/v1/traces`). If
783
+ > your registry counts both, point `OTEL_EXPORTER_OTLP_ENDPOINT` at a separate
784
+ > collector, or set `OTELZERO_OTEL=0` for the import-hook frameworks.
785
+
786
+ OpenTelemetry tracing requires **Python ≥ 3.12** (as do the import hooks).
787
+
788
+ ### How LangSmith's spans are translated
789
+
790
+ `install_langchain_monitoring` / `install_langgraph_monitoring` switch on
791
+ LangChain's own OpenTelemetry connector (the one that ships with the `langsmith`
792
+ SDK; see [the LangGraph guide](#instrument-a-real-langgraph-graph-and-langchain-10-agents)).
793
+
794
+ The connector's spans carry a `langsmith.span.kind` run type, but they speak an
795
+ **older dialect** of the GenAI conventions: `gen_ai.operation.name` holds the
796
+ run type (`chain`, `parser`) and labels prompt-template steps `chat`, the
797
+ deprecated `gen_ai.system` holds `langchain`, and the prompt and response leave
798
+ the process as `gen_ai.prompt` / `gen_ai.completion` whether or not you opted
799
+ in. The one piece of otelzero in the pipeline fixes that: wrap the OTLP exporter
800
+ in `otelzero.translate.TranslatingExporter` and every span is rewritten into the
801
+ current conventions on its way out —
802
+
803
+ | LangChain's connector emits | Translated to |
804
+ |---|---|
805
+ | root `chain` run (`langsmith.trace.name` = `run_name`) | `invoke_agent {run_name}`, `gen_ai.agent.name` |
806
+ | `langsmith.span.kind = llm` | `chat {model}`, `gen_ai.provider.name` from `ls_provider`, `gen_ai.request.model` |
807
+ | `langsmith.span.kind = tool` | `execute_tool {tool}`, `gen_ai.tool.name` |
808
+ | nested `chain` / `prompt` / `parser` steps | no `gen_ai.operation.name` — they are not GenAI operations |
809
+ | `gen_ai.system`, `gen_ai.serialized.*`, `LANGSMITH_*` metadata | dropped |
810
+ | `gen_ai.prompt` / `gen_ai.completion` | `gen_ai.input.messages` / `gen_ai.output.messages` / `gen_ai.system_instructions` — **only** with `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=1`; dropped otherwise |
811
+ | `gen_ai.usage.total_tokens` | `otelzero.usage.total_tokens` (input/output tokens keep their names) |
812
+ | run metadata `conversation_id` / `thread_id`, `user_name` / `user_id` | `gen_ai.conversation.id`, `user.name` |
813
+ | status `OK` on success | left unset, as the conventions require; `error.type` filled in on failure |
814
+
815
+ The mapping lives in `otelzero.translate.langsmith.translate_attributes`, a pure
816
+ function over plain attribute dicts, so an OTLP backend can run the identical
817
+ mapping on ingest. `otel.init_tracing` installs the exporter wrapper on
818
+ otelzero's own provider too, so an otelzero user who also turns the connector on
819
+ gets translated spans without further setup.
820
+
821
+ Because the connector builds its spans on LangSmith's background tracing
822
+ thread, a `run_context` is not visible to them. A LangChain run names its user
823
+ and conversation through its config instead (LangGraph's
824
+ `config={"configurable": {"thread_id": ...}}` works too):
825
+
826
+ ```python
827
+ chain.invoke(
828
+ {"input": query},
829
+ config={"run_name": "SQL Agent",
830
+ "metadata": {"user_name": "Sarah", "conversation_id": thread_id}},
831
+ )
832
+ ```
833
+
834
+ Kiji maps the translated spans straight onto the dashboard:
835
+
836
+ - each **`invoke_agent`** run → an **agent card** (tag the run with `run_name`
837
+ to label it, e.g. `ReAct Agent` instead of `RunnableSequence`);
838
+ - nested **`chat` / `execute_tool`** spans → **task** rows in the live feed,
839
+ labeled by span name — without spawning their own cards;
840
+ - a failing run sets the span status to `ERROR` → a failed run on the card.
841
+
842
+ ### Ingest Microsoft Copilot Studio (SaaS) by relaying its OTLP telemetry
843
+
844
+ [Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/) is a
845
+ hosted, low-code product — there is **no Python process to import or method to
846
+ wrap**, so the import-hook path does not apply. Instead, Copilot Studio agents
847
+ export telemetry to **Azure Monitor / Application Insights** following the OTel
848
+ **GenAI semantic conventions** — the *same* `invoke_agent` / `chat` /
849
+ `execute_tool` spans Kiji's `/v1/traces` receiver already maps. So ingesting
850
+ Copilot Studio is a matter of **relaying** those spans to the registry's OTLP endpoint —
851
+ no new otelzero code:
852
+
853
+ ```
854
+ ┌──────────────────┐ GenAI/OTel ┌───────────────┐ OTLP ┌───────────────┐
855
+ │ Copilot Studio │─────────────→│ OTel │──────────→│ Kiji │
856
+ │ (Azure Monitor) │ telemetry │ Collector │ /v1/traces│ dashboard │
857
+ └──────────────────┘ └───────────────┘ └───────────────┘
858
+ ```
859
+
860
+ The final Collector hop is a small, reusable config —
861
+ [`demo/otelcol-copilot-studio.yaml`](demo/otelcol-copilot-studio.yaml) — an OTLP
862
+ receiver → `otlphttp` exporter to `…/v1/traces` with the ingest key, stamping
863
+ `service.name: "Copilot Studio"` so runs group under one framework label:
864
+
865
+ ```bash
866
+ OTELZERO_API_KEY=<ingest-key> \
867
+ otelcol-contrib --config demo/otelcol-copilot-studio.yaml
868
+ ```
869
+
870
+ **How spans reach the Collector** depends on your Azure setup:
871
+
872
+ - **Pro-code / Foundry-backed agents** that let you set an OTLP endpoint: point
873
+ `OTEL_EXPORTER_OTLP_ENDPOINT` at the Collector — spans arrive directly, no Azure
874
+ round-trip.
875
+ - **Pure Copilot Studio SaaS agents** export only to Application Insights. Relay
876
+ those by fanning App Insights out via a **diagnostic-settings export to Event
877
+ Hub** (or a scheduled **Log Analytics / KQL** query) and a small converter that
878
+ re-emits the GenAI records as OTLP into the Collector. (The Collector has an
879
+ Azure Monitor *exporter*, not a receiver, so the converter step is required for
880
+ the SaaS-only path.)
881
+
882
+ To validate the **Kiji side end to end without an Azure tenant**,
883
+ `demo/demo_copilot_studio_otel.py` synthesizes the exact OTLP spans a Copilot
884
+ Studio agent produces and posts them to `/v1/traces` over real OTLP/HTTP — just
885
+ as the relaying Collector would:
886
+
887
+ ```bash
888
+ (cd ../kiji-agent-registry && uv run kiji-agent-registry) # terminal 1
889
+ uv run python demo/demo_copilot_studio_otel.py # terminal 2 (no Azure needed)
890
+ ```
891
+
892
+ Each `invoke_agent` run becomes an agent card; nested `chat` / `execute_tool`
893
+ spans become labeled task rows; an errored tool propagates to a failed run —
894
+ identical rendering to the framework integrations above.
895
+
896
+
897
+ ## How It Works
898
+
899
+ ### Import Hook Magic
900
+
901
+ The instrumentation uses Python's import system to intercept module loading.
902
+ Python 3.12 removed the legacy `find_module` protocol, so the hook implements
903
+ **`find_spec`** and wraps the real loader to instrument the module after it
904
+ executes:
905
+
906
+ ```python
907
+ class CrewAIImportHook(importlib.abc.MetaPathFinder):
908
+ def find_spec(self, fullname, path=None, target=None):
909
+ if fullname not in ("crewai.crew", "crewai.agent", "crewai.task"):
910
+ return None
911
+ # Resolve the real spec from the rest of sys.meta_path (guarded against
912
+ # re-entrancy), then swap in a loader that instruments after exec.
913
+ spec = importlib.util.find_spec(fullname)
914
+ spec.loader = _InstrumentingLoader(spec.loader, self, fullname)
915
+ return spec
916
+
917
+
918
+ class _InstrumentingLoader(importlib.abc.Loader):
919
+ def exec_module(self, module):
920
+ self._real_loader.exec_module(module) # load CrewAI normally
921
+ self._hook._instrument_module(module) # then wrap Crew/Agent/Task
922
+ ```
923
+
924
+ ### Function Wrapping
925
+
926
+ Uses `wrapt` library for clean function wrapping:
927
+
928
+ ```python
929
+ @wrapt.decorator
930
+ def agent_wrapper(wrapped, instance, args, kwargs):
931
+ # Before execution
932
+ send_log("agent_start", agent_role=instance.role)
933
+
934
+ try:
935
+ result = wrapped(*args, **kwargs)
936
+ # On success
937
+ send_log("agent_end", agent_role=instance.role)
938
+ return result
939
+ except Exception as e:
940
+ # On error
941
+ send_log("agent_error", error=str(e), traceback=...)
942
+ raise
943
+ ```
944
+
945
+ ### Async Telemetry
946
+
947
+ Logs are sent asynchronously to avoid blocking agent execution:
948
+
949
+ ```python
950
+ def send_log(self, event_type, **kwargs):
951
+ log_data = {"event_type": event_type, ...}
952
+ # Send in background thread
953
+ threading.Thread(
954
+ target=self._send_async,
955
+ args=(log_data,),
956
+ daemon=True
957
+ ).start()
958
+ ```
959
+
960
+
961
+ ## Event Types
962
+
963
+ - `crew_start` - Crew execution begins
964
+ - `crew_end` - Crew execution completes
965
+ - `crew_error` - Crew execution fails
966
+ - `agent_start` - Agent begins task execution
967
+ - `agent_end` - Agent completes task
968
+ - `agent_error` - Agent encounters error
969
+ - `task_start` - Task execution begins
970
+ - `task_end` - Task execution completes
971
+ - `task_error` - Task execution fails
972
+
973
+ ### Common telemetry envelope
974
+
975
+ Every event (`send_log`) carries the same top-level fields regardless of
976
+ framework: `event_type`, `timestamp`, `agent_name`, `agent_role`, `framework`,
977
+ `framework_version`, `model`, `user`, `details`, `error`, `traceback` — plus a
978
+ parallel OpenTelemetry span (tracing is on by default).
979
+
980
+ ### Per-framework instrumentation
981
+
982
+ `service.name` carries the framework label; the span names below follow the
983
+ GenAI conventions' `{operation} {subject}` templates.
984
+
985
+ `service.name` is the framework label the client is created with, and it is also
986
+ the value the dashboard groups and filters runs by. Set `OTEL_SERVICE_NAME`, or
987
+ `service.name` in `OTEL_RESOURCE_ATTRIBUTES`, to replace it.
988
+
989
+ | Framework | `service.name` | Wrapped method(s) | OTel span(s) | Lifecycle events |
990
+ |---|---|---|---|---|
991
+ | **CrewAI** | `CrewAI` | `Crew.kickoff`, `Agent.execute_task`, `Task.execute_sync` | `invoke_workflow {crew}`, `invoke_agent {role}`, `execute_task {role}` | crew + agent + task |
992
+ | **LangChain** | `LangChain` | — (LangSmith's native OTel export, translated) | `invoke_agent {run}` → `chat {model}`, `execute_tool {tool}`, plus one span per nested chain / prompt / parser step (LangSmith's own) | agent |
993
+ | **LangGraph** | `LangGraph` | — (LangSmith's native OTel export, translated) | `invoke_agent {graph}` → `{node}` → `chat {model}`, `execute_tool {tool}` (LangSmith's own) | agent |
994
+ | **LlamaIndex** | `LlamaIndex` | `Workflow.run` (every agent, and every workflow) | `invoke_agent {name}` | agent |
995
+ | **OpenAI Agents** | `OpenAI Agents` | `Runner.run`, `run_sync`, `run_streamed` | `invoke_agent {name}` | agent |
996
+ | **Pydantic AI** | `Pydantic AI` | — (native OTel via `Agent.instrument_all`) | `invoke_agent {name}` → `chat`, `execute_tool` (Pydantic AI's own) | agent |
997
+ | **smolagents** | `smolagents` | `MultiStepAgent.run` (direct + stream) | `invoke_agent {name}` | agent |
998
+ | **AWS Strands** | `Strands Agents` | — (native OTel, latest GenAI conventions) | `invoke_agent {name}` → `execute_event_loop_cycle`, `chat`, `execute_tool` (Strands' own) | agent |
999
+ | **Google ADK** | `Google ADK` | — (native OTel, latest GenAI conventions) | `invoke_workflow` → `invoke_agent {name}` → `generate_content`, `execute_tool` (ADK's own) | agent |
1000
+ | **Databricks** | `Databricks Agents` | `ResponsesAgent.predict`, `ChatAgent.predict` | `invoke_agent {name}`, `execute_tool {tool}` | agent + task |
1001
+ | **Claude Agent SDK** | `Claude Agent SDK` | `query`, `ClaudeSDKClient.receive_response` | `invoke_agent Claude Agent` | agent |
1002
+ | **Microsoft** | `Microsoft Agent Framework` | — (native OTel via `enable_instrumentation`) | `invoke_agent {name}` → `chat`, `execute_tool` (Agent Framework's own) | agent |
1003
+ | **Snowflake Cortex** | `Snowflake Cortex` | `query`, `CortexCodeSDKClient.receive_response` | `invoke_agent Snowflake Cortex Agent` | agent |
1004
+
1005
+ ### Event matrix — which lifecycle events each framework emits
1006
+
1007
+ | Framework | `crew_*` | `agent_*` | `task_*` |
1008
+ |---|:--:|:--:|:--:|
1009
+ | CrewAI | ✅ | ✅ | ✅ |
1010
+ | LangChain | — | ✅⁷ | — |
1011
+ | LangGraph | — | ✅⁷ | — |
1012
+ | LlamaIndex | — | ✅ | —⁴ |
1013
+ | OpenAI Agents | — | ✅ | — |
1014
+ | Pydantic AI | — | ✅⁵ | — |
1015
+ | smolagents | — | ✅ | — |
1016
+ | Strands | — | ✅⁵ | — |
1017
+ | Google ADK | — | ✅⁵ | — |
1018
+ | Databricks | — | ✅ | ✅² |
1019
+ | Claude Agent SDK | — | ✅ | — |
1020
+ | Microsoft | — | ✅⁵ | —³ |
1021
+ | Snowflake Cortex | — | ✅ | — |
1022
+
1023
+ ² Databricks maps each `ResponsesAgent` tool call (a `function_call` item in the
1024
+ response) to a `task_*` pair, nested under the agent run.
1025
+ ³ Microsoft's nested model/tool calls surface as `task_*` rows when its
1026
+ **native OTel** spans are ingested (`invoke_agent` → agent, `chat`/`execute_tool`
1027
+ → tasks), which `install_microsoft_monitoring` sets up.
1028
+ ⁴ Likewise for LlamaIndex: the hook captures the agent run, while its nested
1029
+ LLM/tool/retriever steps surface as `task_*` rows when ingested via
1030
+ **OpenInference** (`openinference-instrumentation-llama-index`; root
1031
+ `CHAIN`/`AGENT` → agent, everything nested → tasks), which Kiji's receiver also
1032
+ ingests.
1033
+ ⁷ LangChain / LangGraph `agent_*` events come from LangChain's callbacks, one
1034
+ pair per top-level run; every nested chain, node, model and tool call is a span
1035
+ from LangSmith's export, not an event.
1036
+ ⁵ The native-OTel frameworks' (Pydantic AI, Strands, ADK, Microsoft) `agent_*`
1037
+ events come from their own `invoke_agent` spans, not from a wrapper; their model
1038
+ and tool spans reach the registry over OTLP only.
1039
+
1040
+ ### Details captured per wrapped call
1041
+
1042
+ | Framework · method | `*_start` `details` | `*_end` `details` | `*_error` |
1043
+ |---|---|---|---|
1044
+ | CrewAI · `Crew.kickoff` | `num_agents`, `num_tasks` | `status: completed` | `error`, `traceback` |
1045
+ | CrewAI · `Agent.execute_task` | `role`, `goal` (≤100 chars) | `status: success` | `error`, `traceback` |
1046
+ | CrewAI · `Task.execute_sync` | `description`, `expected_output` (≤100) | `status: completed` | `error`, `traceback` |
1047
+ | LangChain / LangGraph · top-level run (LangChain callbacks) | `model` when the run is a model call | `status: success`, `model`, `tools` (called), `usage` | `error`, `traceback` |
1048
+ | LlamaIndex · `Workflow.run` | `model`, `tools` (+ `agents` for an `AgentWorkflow`) | `status: success` | `error`, `traceback` |
1049
+ | OpenAI Agents · `Runner.run` / `run_sync` / `run_streamed` | `model`, `tools`, `handoffs` | `status: success` | `error`, `traceback` |
1050
+ | Pydantic AI · native `invoke_agent` span | `model` | `status: success`, `tools`, `usage` | `error`, `traceback`⁶ |
1051
+ | smolagents · `MultiStepAgent.run` | `model`, `tools` | `status: success` | `error`, `traceback` |
1052
+ | Strands · native `invoke_agent` span | `model` (none: set after start) | `status: success`, `model`, `tools`, `usage` | `error`, `traceback` |
1053
+ | Google ADK · native `invoke_agent` span | `model` (none: set after start) | `status: success`, `model`, `tools`, `usage` (summed over its `generate_content` spans) | `error`, `traceback` |
1054
+ | Databricks · `ResponsesAgent.predict` / `ChatAgent.predict` | `num_messages` | `status: success` | `error`, `traceback` |
1055
+ | Databricks · tool call (`ResponsesAgent`) | `tool`, `call_id`, `arguments` (≤200) | `status: success`, `output` (≤200) | `error`, `traceback` |
1056
+ | Claude Agent SDK · `query` / `ClaudeSDKClient.receive_response` | `model`, `allowed_tools`, `mcp_servers` (names) | `status: success` | `error`, `traceback` |
1057
+ | Microsoft · native `invoke_agent` span | `model` (none: set after start) | `status: success`, `model` (from its `chat` spans), `tools`, `usage` | `error`, `traceback`⁶ |
1058
+ | Snowflake Cortex · `query` / `CortexCodeSDKClient.receive_response` | `model`, `allowed_tools`, `mcp_servers` (names) | `status: success` | `error`, `traceback` |
1059
+
1060
+ ⁶ Pydantic AI and Agent Framework record the exception message and traceback
1061
+ only when content capture is on; otherwise `error` is the exception type and
1062
+ `traceback` is empty.
1063
+
1064
+ **Prompt and response (opt-in).** With
1065
+ `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=1` set on the agent, every
1066
+ `*_end` / `*_error` row above also carries `input_messages` (the run's prompt)
1067
+ and `output_messages` (its response), plus `system_instructions` where the hook
1068
+ can see a system prompt. Both are arrays in the GenAI conventions' message shape
1069
+ (`{"role", "parts": [{"type": "text", "content"} | reasoning | tool_call | tool_call_response]}`);
1070
+ the model's stop reason sits beside them as `finish_reasons`, not inside a
1071
+ message — the same content that goes
1072
+ out as the `gen_ai.input.messages` / `gen_ai.output.messages` span attributes,
1073
+ so the dashboard, the apps and an OTLP backend all see one record. Off by
1074
+ default: no prompt or response text leaves the process unless you opt in. Size
1075
+ is bounded on the client: `OTELZERO_CONTENT_MAX_CHARS` (default `4000`, `0`
1076
+ disables) caps each text / arguments / response part with a
1077
+ `…[truncated, N more chars]` marker, and `OTELZERO_CONTENT_MAX_MESSAGES`
1078
+ (default `20`, `0` disables) keeps only the most recent input messages of a
1079
+ replayed chat history.
1080
+
1081
+
1082
+ ### OTel span attributes
1083
+
1084
+ otelzero follows the [OpenTelemetry GenAI semantic
1085
+ conventions](https://github.com/open-telemetry/semantic-conventions-genai)
1086
+ (status: development). The vocabulary has a single definition in
1087
+ `src/otelzero/semconv.py`, mirrored in the registry's
1088
+ `kiji_agent_registry/semconv.py` (a registry test checks the two agree).
1089
+
1090
+ **On every span:**
1091
+
1092
+ | Attribute | Value |
1093
+ |---|---|
1094
+ | `gen_ai.operation.name` | `invoke_workflow`, `invoke_agent`, `execute_tool`, or `execute_task` |
1095
+ | `gen_ai.conversation.id` | the run's flow id — groups a conversation's events |
1096
+ | `user.name` | who the run is for |
1097
+ | `otelzero.session.name` | the session's title, once it has one (`name=` or derived from the run's intent) |
1098
+ | `error.type` | the exception class name, on failed spans only |
1099
+
1100
+ A successful span carries **no status code**: the conventions require it to be
1101
+ left unset, so only failures set `ERROR` (plus `error.type` and the exception
1102
+ message).
1103
+
1104
+ **Every otelzero span is `SpanKind.INTERNAL`**, and the conventions define a smaller
1105
+ attribute set for the *internal* `invoke_agent` span than for the `client` one.
1106
+ Two attributes you might expect are therefore absent by design:
1107
+
1108
+ - **`gen_ai.provider.name`.** It replaced the deprecated `gen_ai.system`, but it
1109
+ names the *inference provider* — which an orchestrator like CrewAI is not —
1110
+ and it is listed only on the `create_agent` and `invoke_agent` client spans.
1111
+ The **framework** (`crewai`, `langchain`, …) rides on the `service.name`
1112
+ **resource** attribute instead.
1113
+ - **`gen_ai.agent.id`.** Also client-span-only, and the spec calls recording
1114
+ in-memory agent instance ids on it NOT RECOMMENDED "due to their transient
1115
+ nature" — which is all Kiji's hooks can see.
1116
+
1117
+ This applies to the spans otelzero emits itself. The native-OTel frameworks
1118
+ (Pydantic AI, Strands, ADK, Microsoft) emit their own spans, which carry
1119
+ whatever the framework records: Strands and Agent Framework set
1120
+ `gen_ai.provider.name` (`strands-agents`, `microsoft.agent_framework`), and
1121
+ Agent Framework sets `gen_ai.agent.id`.
1122
+
1123
+ Kiji's ingest still *reads* both from third-party traces that legitimately set
1124
+ them.
1125
+
1126
+ **Token usage**, on the span that closes a run:
1127
+
1128
+ | Attribute | Meaning |
1129
+ |---|---|
1130
+ | `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` | prompt / completion tokens |
1131
+ | `gen_ai.usage.reasoning.output_tokens` | reasoning tokens, when the provider reports them |
1132
+ | `gen_ai.usage.cache_read.input_tokens` | tokens served from a provider cache |
1133
+ | `otelzero.usage.total_tokens` | otelzero's own — the conventions record input and output separately and define no total |
1134
+
1135
+ **Finish reasons**, on the span that closes a run, whether or not content
1136
+ capture is on: `gen_ai.response.finish_reasons`, the conventions' array of
1137
+ normalised stop reasons (`stop`, `length`, `tool_call`, `content_filter`, …),
1138
+ one per generation the framework reports — every turn of a streamed run, the
1139
+ final one of a one-shot result.
1140
+ It replaces the per-message `finish_reason` field the conventions deprecated,
1141
+ and rides on the log row as `finish_reasons`.
1142
+
1143
+ **Content (Opt-In)**, on the span that closes a run, only when
1144
+ `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=1`:
1145
+
1146
+ | Attribute | Meaning |
1147
+ |---|---|
1148
+ | `gen_ai.input.messages` | the run's prompt, as a JSON array of `{"role", "parts"}` messages |
1149
+ | `gen_ai.output.messages` | the run's response, same shape (one message per turn) |
1150
+ | `gen_ai.system_instructions` | the system prompt as a JSON array of parts (CrewAI agents: the backstory) |
1151
+
1152
+ What counts as the prompt and the response per framework:
1153
+
1154
+ | Framework | Prompt (`gen_ai.input.messages`) | Response (`gen_ai.output.messages`) |
1155
+ |---|---|---|
1156
+ | CrewAI | `kickoff(inputs=)` as JSON; a task's `description` (+ `context`); an agent's executor history up to its prompt (its system prompt as `system_instructions`) | `CrewOutput.raw` / `TaskOutput.raw`; an agent's whole exchange (tool calls, results, answer) from its executor history |
1157
+ | LangChain / LangGraph | exported by LangSmith itself as `gen_ai.prompt` (hidden — `{}` — unless content capture is on) | likewise `gen_ai.completion` |
1158
+ | Strands | recorded by Strands itself (latest conventions, `gen_ai_span_attributes_only`) | likewise |
1159
+ | Google ADK | recorded by ADK itself on its `generate_content` spans (with `SPAN_ONLY` / `SPAN_AND_EVENT`) | likewise |
1160
+ | Databricks | `ChatAgent` messages, or `ResponsesAgent` `request.input` items (`message`, `function_call`, `function_call_output`) | `response.messages` or `response.output` items |
1161
+ | Claude Agent SDK | the `query(prompt)` string, or a streamed prompt's messages (teed as the SDK reads them); for the client, what `client.query()` sent, one prompt per `receive_response()` | every streamed `AssistantMessage`/`UserMessage` block (text, tool use, tool result), else `ResultMessage.result`; `stop_reason` normalised |
1162
+ | Microsoft | recorded by Agent Framework itself on its `invoke_agent` / `chat` spans (sensitive data on) | likewise |
1163
+ | Snowflake Cortex | as Claude Agent SDK | as Claude Agent SDK |
1164
+
1165
+ Tool calls inside a message map to `{"type": "tool_call", "id", "name", "arguments"}`
1166
+ and their results to `{"type": "tool_call_response", "id", "response"}`, filed
1167
+ under the `tool` role; Anthropic `tool_use`/`tool_result`, OpenAI
1168
+ `function_call`/`function_call_output`, LangChain `tool_calls` and Gemini
1169
+ `function_call`/`function_response` blocks are all recognised. Content is
1170
+ capped on the client (`OTELZERO_CONTENT_MAX_CHARS`, `OTELZERO_CONTENT_MAX_MESSAGES`,
1171
+ see [Details captured per wrapped call](#details-captured-per-wrapped-call)).
1172
+ The registry's ingest stores these attributes as the same `input_messages` /
1173
+ `output_messages` details the HTTP path records, and its export writes them back
1174
+ as `gen_ai.*` when the server process has opted in — otherwise as
1175
+ `otelzero.input_messages` / `otelzero.output_messages` (still JSON), which
1176
+ ingest also reads, so a downloaded trace re-ingests losslessly either way.
1177
+
1178
+ **Per span, beyond the common set** (every span below also carries the Opt-In
1179
+ content attributes † when captured):
1180
+
1181
+ | Span | Operation | Extra attributes |
1182
+ |---|---|---|
1183
+ | `invoke_workflow {crew}` (CrewAI) | `invoke_workflow` | `gen_ai.workflow.name`, `gen_ai.request.model`, `otelzero.num_agents`, `otelzero.num_tasks` |
1184
+ | `invoke_agent {role}` (CrewAI) | `invoke_agent` | `gen_ai.agent.name` (role), `gen_ai.agent.description` (goal), `gen_ai.request.model` |
1185
+ | `execute_task {role}` (CrewAI) | `execute_task` | `gen_ai.agent.name`, `otelzero.task.description`, `otelzero.task.expected_output` |
1186
+ | `invoke_agent {name}` (LlamaIndex) | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.request.model` ‡, `gen_ai.tool.definitions` † |
1187
+ | `invoke_agent {name}` (OpenAI Agents) | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.request.model`, `gen_ai.tool.definitions` † |
1188
+ | `invoke_agent {name}` (Pydantic AI, emitted by Pydantic AI itself) | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.aggregated_usage.*`; child `chat` spans carry `gen_ai.request.model`, `gen_ai.usage.*`, `gen_ai.tool.definitions` |
1189
+ | `invoke_agent {name}` (smolagents) | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.request.model`, `gen_ai.tool.definitions` † |
1190
+ | `invoke_agent {name}` (Strands, emitted by Strands itself) | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.agent.tools`, `gen_ai.usage.*`, `gen_ai.tool.definitions` † |
1191
+ | `invoke_agent {name}` (ADK, emitted by ADK itself) | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.conversation.id` (ADK's session); child `generate_content` spans carry `gen_ai.request.model`, `gen_ai.usage.*`, `gen_ai.tool.definitions` |
1192
+ | `invoke_agent {name}` (Databricks) | `invoke_agent` | `gen_ai.agent.name`, `otelzero.num_messages` |
1193
+ | `execute_tool {tool}` (Databricks) | `execute_tool` | `gen_ai.agent.name`, `gen_ai.tool.name`, `gen_ai.tool.call.id` |
1194
+ | `invoke_agent Claude Agent` | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.request.model`, `gen_ai.tool.definitions` † |
1195
+ | `invoke_agent {name}` (Microsoft, emitted by Agent Framework itself) | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.agent.id`, `gen_ai.provider.name`, `gen_ai.tool.definitions`, `gen_ai.usage.*`; child `chat` spans carry `gen_ai.request.model` |
1196
+ | `invoke_agent Snowflake Cortex Agent` | `invoke_agent` | `gen_ai.agent.name`, `gen_ai.request.model`, `gen_ai.tool.definitions` † |
1197
+
1198
+ Span names follow the conventions' `{operation} {subject}` templates and fall
1199
+ back to the bare operation when the subject isn't available (an unnamed crew
1200
+ becomes just `invoke_workflow`). `execute_task` is a documented custom operation
1201
+ value: a CrewAI Task is the *parent* of the agent invocation it delegates to, so
1202
+ no predefined value fits, and the enum permits a custom one where none applies.
1203
+
1204
+ ‡ LlamaIndex: whenever the agent's LLM does not name itself (its
1205
+ `model`/`model_name` is absent or the literal `"unknown"`): the model is then
1206
+ taken from what the provider reported on the first reply, falling back to the LLM
1207
+ class name. An LLM that does name its model reports it from the start.
1208
+
1209
+ † The content attributes — `gen_ai.tool.definitions`, `gen_ai.input.messages`,
1210
+ `gen_ai.output.messages`, `gen_ai.system_instructions` — are **Opt-In** in the
1211
+ conventions, so they appear only when you set
1212
+ `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=1`. The tool *names* also
1213
+ ride the log envelope's `details` regardless (they are not content), so Kiji's
1214
+ dashboard and apps list them either way; the prompt and response ride `details`
1215
+ only under the same opt-in.
1216
+
1217
+ #### Deliberate deviations
1218
+
1219
+ Two places where otelzero knowingly emits more than a span's definition lists.
1220
+ Extra attributes are permitted; these are the ones worth knowing about:
1221
+
1222
+ - **`gen_ai.usage.*` and `gen_ai.request.model` on `invoke_workflow` spans.**
1223
+ That span defines only `operation.name`, `error.type`, `conversation.id`, and
1224
+ `workflow.name`. CrewAI reports token usage *only* at the crew level, so
1225
+ honouring the list literally would throw the numbers away.
1226
+ - **`gen_ai.conversation.id` and `user.name` on every span**, including
1227
+ `execute_tool`, where `conversation.id` isn't listed. Both are load-bearing
1228
+ for keeping concurrent users' flows apart.
1229
+
1230
+ Not yet implemented, for reference: `gen_ai.usage.cache_creation.input_tokens`,
1231
+ the request parameters (`temperature`, `max_tokens`, `top_p`, …),
1232
+ `gen_ai.tool.description` / `gen_ai.tool.type`, and
1233
+ the Opt-In `execute_tool` content attributes (`gen_ai.tool.call.arguments`,
1234
+ `gen_ai.tool.call.result`) — tool arguments and results are recorded today only
1235
+ as parts inside `gen_ai.input.messages` / `gen_ai.output.messages`.
1236
+
1237
+ #### Splitting flows per user
1238
+
1239
+ `TelemetryClient`'s user and session are process-wide, which is right for a
1240
+ script and wrong for a hosted agent — many callers in one process would report
1241
+ the same user and collapse into one flow. Wrap each run to give it its own
1242
+ identity:
1243
+
1244
+ ```python
1245
+ import otelzero
1246
+
1247
+ with otelzero.run_context(user="alice", conversation_id=thread_id):
1248
+ crew.kickoff(...)
1249
+ ```
1250
+
1251
+ `user` and `conversation_id` become the `user.name` and `gen_ai.conversation.id`
1252
+ span attributes and the log envelope's `user` / `session`. Omit
1253
+ `conversation_id` for a fresh flow per run, or pass your own thread id to keep a
1254
+ multi-turn conversation together. The scope lives in a `ContextVar`, so
1255
+ concurrent `asyncio` tasks and threads each see their own.
1256
+
1257
+ #### Migrating from the pre-1.0 names
1258
+
1259
+ The client emits only the names above. The server reads them **and** their
1260
+ pre-1.0 spellings, so a client older than the server it reports to keeps
1261
+ producing agent cards and token counts unchanged.
1262
+
1263
+ | Pre-1.0 | Now |
1264
+ |---|---|
1265
+ | `gen_ai.system` | *removed* — framework is the `service.name` resource attribute |
1266
+ | `gen_ai.operation.name: crew` / `agent` / `task` | `invoke_workflow` / `invoke_agent` / `execute_task` |
1267
+ | `mira.user` (resource) | `user.name` (span) |
1268
+ | *(session was the trace id)* | `gen_ai.conversation.id` (span) |
1269
+ | `mira.usage.reasoning_tokens` | `gen_ai.usage.reasoning.output_tokens` |
1270
+ | `mira.usage.cache_read_tokens` | `gen_ai.usage.cache_read.input_tokens` |
1271
+ | `mira.agent.goal` | `gen_ai.agent.description` |
1272
+ | `mira.tool.call_id` | `gen_ai.tool.call.id` |
1273
+ | `mira.agent.tools` | `gen_ai.tool.definitions` (opt-in — see above) |
1274
+ | `gen_ai.agent.id` | `otelzero.agent.id` — client-span-only, and the values are transient instance ids |
1275
+ | *(span status was `OK` on success)* | left unset, as the conventions require |
1276
+ | `mira.chain`, `mira.app_name` | `gen_ai.workflow.name` |
1277
+ | `mira.usage.total_tokens`, `mira.num_agents`, `mira.num_tasks`, `mira.num_messages`, `mira.steps`, `mira.agent.class`, `mira.task.*` | the same attributes, renamed to `otelzero.*` — no convention equivalent |
1278
+
1279
+ Third-party emitters you don't control (LangChain's LangSmith connector,
1280
+ Azure Monitor's Copilot Studio export) are unaffected: setting `mira.user` on
1281
+ the resource still attributes their traces.
1282
+
1283
+ CrewAI and Databricks both surface a task level nested under the agent — CrewAI
1284
+ because its framework exposes task execution as a wrappable method, Databricks by
1285
+ reading the tool calls out of each `ResponsesAgent` response. The OpenAI Agents
1286
+ SDK, smolagents, the Claude Agent SDK and the Snowflake Cortex Agent SDK are
1287
+ instrumented at a single agent-run boundary. Pydantic AI, Strands, ADK, Microsoft,
1288
+ LangChain and LangGraph are not wrapped at all: their *own* native OTel spans
1289
+ (agent run, model and tool calls) are the trace.
1290
+
1291
+
1292
+ ## Customization
1293
+
1294
+
1295
+ ### Change Server URL
1296
+
1297
+ ```python
1298
+ install_crewai_monitoring(server_url="http://your-server:8000")
1299
+ ```
1300
+
1301
+ ### Add Custom Instrumentation
1302
+
1303
+ Edit `_instrument_module()` in the import hook:
1304
+
1305
+ ```python
1306
+ def _instrument_module(self, fullname, module):
1307
+ if fullname == 'your_custom_module':
1308
+ self._wrap_your_class(module)
1309
+ ```
1310
+
1311
+
1312
+ ## Troubleshooting
1313
+
1314
+ ### Client can't connect to server
1315
+ - Ensure the registry is running (`uv run kiji-agent-registry` in its checkout)
1316
+ - Check firewall settings
1317
+ - Verify server URL in client code
1318
+
1319
+ ### No logs appearing
1320
+ - Check console for error messages
1321
+ - Verify instrumentation installed before CrewAI import
1322
+ - Check network connectivity
1323
+
1324
+ ### Dashboard not updating
1325
+ - Verify JavaScript console for errors
1326
+ - Check browser compatibility (modern browsers only)
1327
+ - Ensure server is returning data at /api/activity
1328
+
1329
+
1330
+ ## Development
1331
+
1332
+ ```bash
1333
+ uv sync --extra otel
1334
+ uv run pytest
1335
+ ```
1336
+
1337
+ To release to PyPI, create a `release/x.y.z` branch, set the version to `x.y.z`
1338
+ in `pyproject.toml` (under both `[project]` and `[tool.commitizen]`) and
1339
+ `src/otelzero/__init__.py`, run `uv lock`, update `CHANGELOG.md`, and open a pull
1340
+ request into `main`. Merging it runs the tests and publishes that version to PyPI. Every push
1341
+ to `main` also publishes a dev build to TestPyPI.
1342
+
1343
+ ## License
1344
+
1345
+ Apache-2.0