agentx-python 0.6.15__tar.gz → 0.6.17__tar.gz

{agentx_python-0.6.15 → agentx_python-0.6.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentx-python
-Version: 0.6.15
+Version: 0.6.17
 Summary: Official Python SDK for AgentX (https://www.agentx.so/)
 Home-page: https://github.com/AgentX-ai/AgentX-python
 Author: Robin Wang and AgentX Team
@@ -66,7 +66,7 @@ Dynamic: summary
 [![Python versions](https://img.shields.io/pypi/pyversions/agentx-python)](https://pypi.org/project/agentx-python/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
-The official Python SDK for **[AgentX](https://app.agentx.so/)** — an evaluation, tracing, and monitoring framework for AI agents, plus a client for AgentX's own hosted agents.
+The official Python SDK for **[AgentX](https://app.agentx.so/)** - an evaluation, tracing, and monitoring framework for AI agents, plus a client for AgentX's own hosted agents.
 
 Also see [SDK Developer Docs](https://developers.agentx.so), [API Reference Docs](https://docs.agentx.so/reference)
 
@@ -78,24 +78,24 @@ Also see [SDK Developer Docs](https://developers.agentx.so), [API Reference Docs
 - [Installation](#installation)
 - [Authentication](#authentication)
 - [Quick start](#quick-start)
-- [Custom agent evaluations](#custom-agent-evaluations) — LLM-as-a-judge, cosine / Jaccard similarity, any framework
-- [Production tracing](#production-tracing) — record live agent runs from any framework
-- [Monitor](#monitor) — automatic production monitoring, patterns and signals
-- [Self-host](#self-host) — run Trace/Evaluate/Monitor on your own machine instead of the hosted dashboard
-- [Agents & conversations](#agents--conversations) — chat with and orchestrate AgentX's own hosted agents
+- [Custom agent evaluations](#custom-agent-evaluations) - LLM-as-a-judge, cosine / Jaccard similarity, any framework
+- [Production tracing](#production-tracing) - record live agent runs from any framework
+- [Monitor](#monitor) - automatic production monitoring, patterns and signals
+- [Self-host](#self-host) - run Trace/Evaluate/Monitor on your own machine instead of the hosted dashboard
+- [Agents & conversations](#agents--conversations) - chat with and orchestrate AgentX's own hosted agents
 - [Links](#links)
 
 ---
 
 ## Why AgentX
 
-- **Agent Evaluations** — score **any** agent (LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP, or plain Python) against a dataset with LLM-as-a-judge ratings plus optional cosine, Jaccard, and BLEU/ROUGE similarity metrics. Configurable judge prompt/model, per-question judge guidelines, smoke testing for phrasing robustness, and a durable multi-judge analysis pass for a qualitative report.
-- **Production tracing** — one decorator or context manager records every agent run (input, output, latency, tool calls, token usage, and a real span tree) into your workspace, for any framework.
-- **Monitor** — check live traces against detection patterns or a sampled LLM judge, and read back triage-ready signals, no dashboard setup required.
-- **Prompt registry** — make AgentX the source of truth for your own agent's prompts: pull a version at runtime, tag eval runs and live traces with it, let a judge propose a rewrite from your worst-rated results.
-- **Self-host** — run the whole Trace/Evaluate/Monitor stack locally, bring your own LLM keys, no account required.
-- **Bring any LLM** — works across major open and closed-source vendors, for evaluation, tracing, and AgentX's own hosted agents alike.
-- **AgentX's own hosted agents** — a simple `Agent → Conversation → Message` mental model, chain-of-thought built in, multi-agent workforces, MCP support, and A2A publishing, for when you want AgentX to run the agent too, not just evaluate/trace/monitor it.
+- **Agent Evaluations** - score **any** agent (LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP, or plain Python) against a dataset with LLM-as-a-judge ratings plus optional cosine, Jaccard, and BLEU/ROUGE similarity metrics. Configurable judge prompt/model, per-question judge guidelines, smoke testing for phrasing robustness, and a durable multi-judge analysis pass for a qualitative report.
+- **Production tracing** - one decorator or context manager records every agent run (input, output, latency, tool calls, token usage, and a real span tree) into your workspace, for any framework.
+- **Monitor** - check live traces against detection patterns or a sampled LLM judge, and read back triage-ready signals, no dashboard setup required.
+- **Prompt registry** - make AgentX the source of truth for your own agent's prompts: pull a version at runtime, tag eval runs and live traces with it, let a judge propose a rewrite from your worst-rated results.
+- **Self-host** - run the whole Trace/Evaluate/Monitor stack locally, bring your own LLM keys, no account required.
+- **Bring any LLM** - works across major open and closed-source vendors, for evaluation, tracing, and AgentX's own hosted agents alike.
+- **AgentX's own hosted agents** - a simple `Agent → Conversation → Message` mental model, chain-of-thought built in, multi-agent workforces, MCP support, and A2A publishing, for when you want AgentX to run the agent too, not just evaluate/trace/monitor it.
 
 ---
 
@@ -114,11 +114,11 @@ Requires Python 3.9 or newer.
 Get your API key at [app.agentx.so](https://app.agentx.so), then either pass it inline or expose it as an environment variable.
 
 ```python
-# Option A — pass the key inline
+# Option A - pass the key inline
 from agentx import AgentX
 client = AgentX(api_key="your-api-key-here")
 
-# Option B — set AGENTX_API_KEY in your environment, then:
+# Option B - set AGENTX_API_KEY in your environment, then:
 client = AgentX.from_env()
 ```
 
@@ -126,7 +126,7 @@ client = AgentX.from_env()
 
 ## Quick start
 
-Evaluate your own agent — any framework, or plain Python — against a dataset:
+Evaluate your own agent - any framework, or plain Python - against a dataset:
 
 ```python
 from agentx import AgentX
@@ -154,7 +154,7 @@ That's it. The rest of this section covers building the dataset, framework adapt
 
 ## Custom agent evaluations
 
-Evaluate **any** AI agent — LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP endpoints, or plain Python — using AgentX as the scoring and reporting backend. Includes optional **cosine**, **Jaccard**, and **BLEU/ROUGE** similarity metrics alongside LLM-graded ratings.
+Evaluate **any** AI agent - LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP endpoints, or plain Python - using AgentX as the scoring and reporting backend. Includes optional **cosine**, **Jaccard**, and **BLEU/ROUGE** similarity metrics alongside LLM-graded ratings.
 
 ```python
 report = (
@@ -177,7 +177,7 @@ print(report.recommendations)      # list of prioritized, actionable fixes
 
 Ask a case's question several extra ways each run, LLM-paraphrased server-side, to catch agents that break on phrasing rather than substance, and override the judge's prompt/model per config, see [Smoke testing](EVALUATIONS.md#smoke-testing-phrasing-robustness) and [Configuring the judge](EVALUATIONS.md#configuring-the-judge) in the full guide.
 
-Since AgentX doesn't own your agent's code, `client.evaluations.prompts` lets AgentX become your prompt's *source of truth* instead — the same problem LangSmith's Prompt Hub and Langfuse's Prompt Management solve. Pull a version at runtime, tag your eval runs (or live traces) with it, and let a judge propose a rewrite from your real worst-rated results — a human always has to approve before it publishes:
+Since AgentX doesn't own your agent's code, `client.evaluations.prompts` lets AgentX become your prompt's *source of truth* instead - the same problem LangSmith's Prompt Hub and Langfuse's Prompt Management solve. Pull a version at runtime, tag your eval runs (or live traces) with it, and let a judge propose a rewrite from your real worst-rated results - a human always has to approve before it publishes:
 
 ```python
 prompt = client.evaluations.prompts.get("support-agent-system-prompt")  # or prompt.id
@@ -189,15 +189,17 @@ client.evaluations.run(
 ).execute(my_agent_fn)
 ```
 
-See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/self-host#prompt-registry) for the "Suggest improvement" dashboard flow (self-host only — no hosted-SaaS equivalent yet).
+See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
 
-See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
+On self-host, a finalized run can also **gate a CI job**: `report.gate(fail_under=7, no_regression=True)` checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
+
+See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
 ---
 
 ## Production tracing
 
-Record live agent runs into your workspace with a single decorator or context manager — no changes to your agent's logic. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets with [`tracer.evaluate_trace()`](TRACING.md#tracerevaluate_trace).
+Record live agent runs into your workspace with a single decorator or context manager - no changes to your agent's logic. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets with [`tracer.evaluate_trace()`](TRACING.md#tracerevaluate_trace).
 
 ```python
 from agentx import AgentX
@@ -226,9 +228,18 @@ with tracer.trace("rag-agent", framework="langchain") as span:
     span.output = llm.invoke(f"Context: {kb_result}\n\nQuery: {query}")
 ```
 
+To time a tool call and capture its failures automatically, wrap the execution itself instead of reporting it after the fact:
+
+```python
+with tracer.trace_tool_call("search_knowledge_base", input=query) as t:
+    t.output = search_knowledge_base(query)
+```
+
+An exception escaping the block records the call as failed (`success=False` plus the error text, which is what Monitor's built-in "Tool failure" check and the dashboard's Tool quality column read) and then propagates unchanged.
+
 ### Framework integrations
 
-Each integration auto-captures LLM calls, tool calls, and token usage — including prompt-caching
+Each integration auto-captures LLM calls, tool calls, and token usage - including prompt-caching
 token counts (Anthropic's cache write/read, OpenAI/LiteLLM's cached tokens, Google GenAI's cached
 content), reported as their own `cache_read_tokens`/`cache_write_tokens` fields alongside the
 regular totals, no extra config needed. Self-host's cost estimate prices these separately from a
@@ -248,11 +259,11 @@ extra:
 | LlamaIndex             | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler`|
 | AutoGen                | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`  |
 
-Or plain Python — wrap any function with `@tracer.trace(...)` and it just works, no framework required.
+Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework required.
 
-Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces — see [TRACING.md](TRACING.md) for the full pattern.
+Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces - see [TRACING.md](TRACING.md) for the full pattern.
 
-See **[TRACING.md](TRACING.md)** for the complete guide — session grouping, error handling, async support, and the full API reference.
+See **[TRACING.md](TRACING.md)** for the complete guide - session grouping, error handling, async support, and the full API reference.
 
 ---
 
@@ -302,13 +313,39 @@ evaluator = client.monitor.online_evaluators.builder(
 client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
 ```
 
+An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, two ground-truth streams feed the dashboard's Judge Calibration view (which measures how often AgentX's automated verdicts agree with what actually happened): `client.outcomes.report(...)` for after-the-fact system results (a reopened ticket, a human confirmation), and `client.feedback.report(...)` for end-user votes forwarded from your own app's UI - a "down" raises a "Negative user feedback" signal directly, no sampling or judge call involved. All self-host features.
+
+```python
+client.monitor.online_evaluators.builder(
+    name="Conversation resolution",
+    evaluation_settings_id=settings.id,
+    scope="session",     # judge the whole session, not each trace
+    idle_seconds=120,    # score once the conversation has been quiet this long
+    alert_threshold=5,
+).publish()
+
+client.outcomes.report(
+    trace_id=trace_id,
+    outcome="reopened",
+    is_negative=True,
+    reason="Customer reopened the ticket within 3 days",
+)
+
+client.feedback.report(
+    trace_id=trace_id,
+    rating="down",                            # "up" or "down"
+    comment="It never answered my question",  # optional, the user's own words
+    end_user_id=current_user.id,              # optional, opaque to AgentX
+)
+```
+
 See **[TRACING.md](TRACING.md)** for the complete Monitor guide.
 
 ---
 
 ## Self-host
 
-Prefer to run Trace/Evaluate/Monitor locally instead of the hosted dashboard — no account, bring your own LLM keys? This SDK ships a launcher for [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval), a separate, portable governance engine:
+Prefer to run Trace/Evaluate/Monitor locally instead of the hosted dashboard - no account, bring your own LLM keys? This SDK ships a launcher for [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval), a separate, portable governance engine:
 
 ```bash
 agentx-trace-eval --dev
@@ -321,22 +358,22 @@ export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
 export AGENTX_API_KEY=<printed by agentx-trace-eval on first run>
 ```
 
-`agentx-trace-eval` isn't this SDK's own code — the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. See that repo's README for what's included, and `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
+`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. See that repo's README for what's included, and `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
 
 ---
 
 ## Agents & conversations
 
-Beyond evaluation, tracing, and monitoring, this SDK is also a client for AgentX's own hosted agents — build, chat with, and orchestrate them directly.
+Beyond evaluation, tracing, and monitoring, this SDK is also a client for AgentX's own hosted agents - build, chat with, and orchestrate them directly.
 
 ```python
 agent = client.list_agents()[0]
 conversation = agent.new_conversation()
 
-# Blocking — returns the full response once it's ready
+# Blocking - returns the full response once it's ready
 print(conversation.chat("What can you help me with?"))
 
-# Streaming — yields ChatResponse objects as the model produces them
+# Streaming - yields ChatResponse objects as the model produces them
 for chunk in conversation.chat_stream("Hello!"):
     if chunk.text:
         print(chunk.text, end="")
@@ -359,10 +396,10 @@ for chunk in workforce.chat_stream(conversation.id, "How can you help me with th
 
 ## Links
 
-- **Dashboard** — [app.agentx.so](https://app.agentx.so)
-- **Website** — [agentx.so](https://www.agentx.so/)
-- **PyPI** — [agentx-python](https://pypi.org/project/agentx-python/)
-- **Tracing docs** — [TRACING.md](TRACING.md)
-- **Evaluations docs** — [EVALUATIONS.md](EVALUATIONS.md)
-- **Monitor docs** — [docs.agentx.so/sdk/monitor](https://docs.agentx.so/sdk/monitor)
-- **Self-host** — [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval)
+- **Dashboard** - [app.agentx.so](https://app.agentx.so)
+- **Website** - [agentx.so](https://www.agentx.so/)
+- **PyPI** - [agentx-python](https://pypi.org/project/agentx-python/)
+- **Tracing docs** - [TRACING.md](TRACING.md)
+- **Evaluations docs** - [EVALUATIONS.md](EVALUATIONS.md)
+- **Monitor docs** - [docs.agentx.so/sdk/monitor](https://docs.agentx.so/sdk/monitor)
+- **Self-host** - [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval)

{agentx_python-0.6.15 → agentx_python-0.6.17}/README.md

@@ -4,7 +4,7 @@
 [![Python versions](https://img.shields.io/pypi/pyversions/agentx-python)](https://pypi.org/project/agentx-python/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
-The official Python SDK for **[AgentX](https://app.agentx.so/)** — an evaluation, tracing, and monitoring framework for AI agents, plus a client for AgentX's own hosted agents.
+The official Python SDK for **[AgentX](https://app.agentx.so/)** - an evaluation, tracing, and monitoring framework for AI agents, plus a client for AgentX's own hosted agents.
 
 Also see [SDK Developer Docs](https://developers.agentx.so), [API Reference Docs](https://docs.agentx.so/reference)
 
@@ -16,24 +16,24 @@ Also see [SDK Developer Docs](https://developers.agentx.so), [API Reference Docs
 - [Installation](#installation)
 - [Authentication](#authentication)
 - [Quick start](#quick-start)
-- [Custom agent evaluations](#custom-agent-evaluations) — LLM-as-a-judge, cosine / Jaccard similarity, any framework
-- [Production tracing](#production-tracing) — record live agent runs from any framework
-- [Monitor](#monitor) — automatic production monitoring, patterns and signals
-- [Self-host](#self-host) — run Trace/Evaluate/Monitor on your own machine instead of the hosted dashboard
-- [Agents & conversations](#agents--conversations) — chat with and orchestrate AgentX's own hosted agents
+- [Custom agent evaluations](#custom-agent-evaluations) - LLM-as-a-judge, cosine / Jaccard similarity, any framework
+- [Production tracing](#production-tracing) - record live agent runs from any framework
+- [Monitor](#monitor) - automatic production monitoring, patterns and signals
+- [Self-host](#self-host) - run Trace/Evaluate/Monitor on your own machine instead of the hosted dashboard
+- [Agents & conversations](#agents--conversations) - chat with and orchestrate AgentX's own hosted agents
 - [Links](#links)
 
 ---
 
 ## Why AgentX
 
-- **Agent Evaluations** — score **any** agent (LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP, or plain Python) against a dataset with LLM-as-a-judge ratings plus optional cosine, Jaccard, and BLEU/ROUGE similarity metrics. Configurable judge prompt/model, per-question judge guidelines, smoke testing for phrasing robustness, and a durable multi-judge analysis pass for a qualitative report.
-- **Production tracing** — one decorator or context manager records every agent run (input, output, latency, tool calls, token usage, and a real span tree) into your workspace, for any framework.
-- **Monitor** — check live traces against detection patterns or a sampled LLM judge, and read back triage-ready signals, no dashboard setup required.
-- **Prompt registry** — make AgentX the source of truth for your own agent's prompts: pull a version at runtime, tag eval runs and live traces with it, let a judge propose a rewrite from your worst-rated results.
-- **Self-host** — run the whole Trace/Evaluate/Monitor stack locally, bring your own LLM keys, no account required.
-- **Bring any LLM** — works across major open and closed-source vendors, for evaluation, tracing, and AgentX's own hosted agents alike.
-- **AgentX's own hosted agents** — a simple `Agent → Conversation → Message` mental model, chain-of-thought built in, multi-agent workforces, MCP support, and A2A publishing, for when you want AgentX to run the agent too, not just evaluate/trace/monitor it.
+- **Agent Evaluations** - score **any** agent (LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP, or plain Python) against a dataset with LLM-as-a-judge ratings plus optional cosine, Jaccard, and BLEU/ROUGE similarity metrics. Configurable judge prompt/model, per-question judge guidelines, smoke testing for phrasing robustness, and a durable multi-judge analysis pass for a qualitative report.
+- **Production tracing** - one decorator or context manager records every agent run (input, output, latency, tool calls, token usage, and a real span tree) into your workspace, for any framework.
+- **Monitor** - check live traces against detection patterns or a sampled LLM judge, and read back triage-ready signals, no dashboard setup required.
+- **Prompt registry** - make AgentX the source of truth for your own agent's prompts: pull a version at runtime, tag eval runs and live traces with it, let a judge propose a rewrite from your worst-rated results.
+- **Self-host** - run the whole Trace/Evaluate/Monitor stack locally, bring your own LLM keys, no account required.
+- **Bring any LLM** - works across major open and closed-source vendors, for evaluation, tracing, and AgentX's own hosted agents alike.
+- **AgentX's own hosted agents** - a simple `Agent → Conversation → Message` mental model, chain-of-thought built in, multi-agent workforces, MCP support, and A2A publishing, for when you want AgentX to run the agent too, not just evaluate/trace/monitor it.
 
 ---
 
@@ -52,11 +52,11 @@ Requires Python 3.9 or newer.
 Get your API key at [app.agentx.so](https://app.agentx.so), then either pass it inline or expose it as an environment variable.
 
 ```python
-# Option A — pass the key inline
+# Option A - pass the key inline
 from agentx import AgentX
 client = AgentX(api_key="your-api-key-here")
 
-# Option B — set AGENTX_API_KEY in your environment, then:
+# Option B - set AGENTX_API_KEY in your environment, then:
 client = AgentX.from_env()
 ```
 
@@ -64,7 +64,7 @@ client = AgentX.from_env()
 
 ## Quick start
 
-Evaluate your own agent — any framework, or plain Python — against a dataset:
+Evaluate your own agent - any framework, or plain Python - against a dataset:
 
 ```python
 from agentx import AgentX
@@ -92,7 +92,7 @@ That's it. The rest of this section covers building the dataset, framework adapt
 
 ## Custom agent evaluations
 
-Evaluate **any** AI agent — LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP endpoints, or plain Python — using AgentX as the scoring and reporting backend. Includes optional **cosine**, **Jaccard**, and **BLEU/ROUGE** similarity metrics alongside LLM-graded ratings.
+Evaluate **any** AI agent - LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP endpoints, or plain Python - using AgentX as the scoring and reporting backend. Includes optional **cosine**, **Jaccard**, and **BLEU/ROUGE** similarity metrics alongside LLM-graded ratings.
 
 ```python
 report = (
@@ -115,7 +115,7 @@ print(report.recommendations)      # list of prioritized, actionable fixes
 
 Ask a case's question several extra ways each run, LLM-paraphrased server-side, to catch agents that break on phrasing rather than substance, and override the judge's prompt/model per config, see [Smoke testing](EVALUATIONS.md#smoke-testing-phrasing-robustness) and [Configuring the judge](EVALUATIONS.md#configuring-the-judge) in the full guide.
 
-Since AgentX doesn't own your agent's code, `client.evaluations.prompts` lets AgentX become your prompt's *source of truth* instead — the same problem LangSmith's Prompt Hub and Langfuse's Prompt Management solve. Pull a version at runtime, tag your eval runs (or live traces) with it, and let a judge propose a rewrite from your real worst-rated results — a human always has to approve before it publishes:
+Since AgentX doesn't own your agent's code, `client.evaluations.prompts` lets AgentX become your prompt's *source of truth* instead - the same problem LangSmith's Prompt Hub and Langfuse's Prompt Management solve. Pull a version at runtime, tag your eval runs (or live traces) with it, and let a judge propose a rewrite from your real worst-rated results - a human always has to approve before it publishes:
 
 ```python
 prompt = client.evaluations.prompts.get("support-agent-system-prompt")  # or prompt.id
@@ -127,15 +127,17 @@ client.evaluations.run(
 ).execute(my_agent_fn)
 ```
 
-See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/self-host#prompt-registry) for the "Suggest improvement" dashboard flow (self-host only — no hosted-SaaS equivalent yet).
+See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
 
-See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
+On self-host, a finalized run can also **gate a CI job**: `report.gate(fail_under=7, no_regression=True)` checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
+
+See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
 ---
 
 ## Production tracing
 
-Record live agent runs into your workspace with a single decorator or context manager — no changes to your agent's logic. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets with [`tracer.evaluate_trace()`](TRACING.md#tracerevaluate_trace).
+Record live agent runs into your workspace with a single decorator or context manager - no changes to your agent's logic. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets with [`tracer.evaluate_trace()`](TRACING.md#tracerevaluate_trace).
 
 ```python
 from agentx import AgentX
@@ -164,9 +166,18 @@ with tracer.trace("rag-agent", framework="langchain") as span:
     span.output = llm.invoke(f"Context: {kb_result}\n\nQuery: {query}")
 ```
 
+To time a tool call and capture its failures automatically, wrap the execution itself instead of reporting it after the fact:
+
+```python
+with tracer.trace_tool_call("search_knowledge_base", input=query) as t:
+    t.output = search_knowledge_base(query)
+```
+
+An exception escaping the block records the call as failed (`success=False` plus the error text, which is what Monitor's built-in "Tool failure" check and the dashboard's Tool quality column read) and then propagates unchanged.
+
 ### Framework integrations
 
-Each integration auto-captures LLM calls, tool calls, and token usage — including prompt-caching
+Each integration auto-captures LLM calls, tool calls, and token usage - including prompt-caching
 token counts (Anthropic's cache write/read, OpenAI/LiteLLM's cached tokens, Google GenAI's cached
 content), reported as their own `cache_read_tokens`/`cache_write_tokens` fields alongside the
 regular totals, no extra config needed. Self-host's cost estimate prices these separately from a
@@ -186,11 +197,11 @@ extra:
 | LlamaIndex             | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler`|
 | AutoGen                | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`  |
 
-Or plain Python — wrap any function with `@tracer.trace(...)` and it just works, no framework required.
+Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework required.
 
-Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces — see [TRACING.md](TRACING.md) for the full pattern.
+Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces - see [TRACING.md](TRACING.md) for the full pattern.
 
-See **[TRACING.md](TRACING.md)** for the complete guide — session grouping, error handling, async support, and the full API reference.
+See **[TRACING.md](TRACING.md)** for the complete guide - session grouping, error handling, async support, and the full API reference.
 
 ---
 
@@ -240,13 +251,39 @@ evaluator = client.monitor.online_evaluators.builder(
 client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
 ```
 
+An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, two ground-truth streams feed the dashboard's Judge Calibration view (which measures how often AgentX's automated verdicts agree with what actually happened): `client.outcomes.report(...)` for after-the-fact system results (a reopened ticket, a human confirmation), and `client.feedback.report(...)` for end-user votes forwarded from your own app's UI - a "down" raises a "Negative user feedback" signal directly, no sampling or judge call involved. All self-host features.
+
+```python
+client.monitor.online_evaluators.builder(
+    name="Conversation resolution",
+    evaluation_settings_id=settings.id,
+    scope="session",     # judge the whole session, not each trace
+    idle_seconds=120,    # score once the conversation has been quiet this long
+    alert_threshold=5,
+).publish()
+
+client.outcomes.report(
+    trace_id=trace_id,
+    outcome="reopened",
+    is_negative=True,
+    reason="Customer reopened the ticket within 3 days",
+)
+
+client.feedback.report(
+    trace_id=trace_id,
+    rating="down",                            # "up" or "down"
+    comment="It never answered my question",  # optional, the user's own words
+    end_user_id=current_user.id,              # optional, opaque to AgentX
+)
+```
+
 See **[TRACING.md](TRACING.md)** for the complete Monitor guide.
 
 ---
 
 ## Self-host
 
-Prefer to run Trace/Evaluate/Monitor locally instead of the hosted dashboard — no account, bring your own LLM keys? This SDK ships a launcher for [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval), a separate, portable governance engine:
+Prefer to run Trace/Evaluate/Monitor locally instead of the hosted dashboard - no account, bring your own LLM keys? This SDK ships a launcher for [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval), a separate, portable governance engine:
 
 ```bash
 agentx-trace-eval --dev
@@ -259,22 +296,22 @@ export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
 export AGENTX_API_KEY=<printed by agentx-trace-eval on first run>
 ```
 
-`agentx-trace-eval` isn't this SDK's own code — the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. See that repo's README for what's included, and `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
+`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. See that repo's README for what's included, and `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
 
 ---
 
 ## Agents & conversations
 
-Beyond evaluation, tracing, and monitoring, this SDK is also a client for AgentX's own hosted agents — build, chat with, and orchestrate them directly.
+Beyond evaluation, tracing, and monitoring, this SDK is also a client for AgentX's own hosted agents - build, chat with, and orchestrate them directly.
 
 ```python
 agent = client.list_agents()[0]
 conversation = agent.new_conversation()
 
-# Blocking — returns the full response once it's ready
+# Blocking - returns the full response once it's ready
 print(conversation.chat("What can you help me with?"))
 
-# Streaming — yields ChatResponse objects as the model produces them
+# Streaming - yields ChatResponse objects as the model produces them
 for chunk in conversation.chat_stream("Hello!"):
     if chunk.text:
         print(chunk.text, end="")
@@ -297,10 +334,10 @@ for chunk in workforce.chat_stream(conversation.id, "How can you help me with th
 
 ## Links
 
-- **Dashboard** — [app.agentx.so](https://app.agentx.so)
-- **Website** — [agentx.so](https://www.agentx.so/)
-- **PyPI** — [agentx-python](https://pypi.org/project/agentx-python/)
-- **Tracing docs** — [TRACING.md](TRACING.md)
-- **Evaluations docs** — [EVALUATIONS.md](EVALUATIONS.md)
-- **Monitor docs** — [docs.agentx.so/sdk/monitor](https://docs.agentx.so/sdk/monitor)
-- **Self-host** — [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval)
+- **Dashboard** - [app.agentx.so](https://app.agentx.so)
+- **Website** - [agentx.so](https://www.agentx.so/)
+- **PyPI** - [agentx-python](https://pypi.org/project/agentx-python/)
+- **Tracing docs** - [TRACING.md](TRACING.md)
+- **Evaluations docs** - [EVALUATIONS.md](EVALUATIONS.md)
+- **Monitor docs** - [docs.agentx.so/sdk/monitor](https://docs.agentx.so/sdk/monitor)
+- **Self-host** - [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval)

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/agentx.py

@@ -54,6 +54,18 @@ class AgentX:
             workspace_id=self.workspace_id,
         )
 
+        from agentx.outcomes import OutcomesClient
+
+        # Report real, after-the-fact outcomes ("the ticket got reopened") against traces - the
+        # ground truth behind the dashboard's Judge Calibration card. Self-host only.
+        self.outcomes = OutcomesClient(api_key=self.api_key)
+
+        from agentx.feedback import FeedbackClient
+
+        # Forward end-user votes ("up"/"down") on traced responses - a "down" raises a signal
+        # directly, and every vote feeds Judge Calibration alongside outcomes. Self-host only.
+        self.feedback = FeedbackClient(api_key=self.api_key)
+
         _ingest_client = IngestClient(
             api_key=self.api_key,
             sdk_version=VERSION,

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/cli.py

@@ -1,5 +1,5 @@
 """
-`agentx-trace-eval` — thin launcher for AgentX's self-hostable governance engine (Trace,
+`agentx-trace-eval` - thin launcher for AgentX's self-hostable governance engine (Trace,
 Evaluate, Monitor), published separately at github.com/AgentX-ai/AgentX-trace-eval (a Go CLI
 wrapping a Bun-compiled TypeScript engine, not Python). That compiled engine binary is tens of
 megabytes; most `pip install agentx-python` installs are just this SDK talking to the hosted

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/evaluations/_term.py

@@ -1,4 +1,4 @@
-"""Minimal ANSI terminal helpers — no external dependencies."""
+"""Minimal ANSI terminal helpers - no external dependencies."""
 
 from __future__ import annotations
 

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/evaluations/adapters/http_endpoint.py

@@ -12,7 +12,7 @@ from agentx.evaluations.results import normalize_error, normalize_result
 class HttpEndpointAdapter:
     """
     Calls a user-hosted HTTP endpoint for each evaluation case.
-    The SDK (running locally) makes the request — the AgentX API never
+    The SDK (running locally) makes the request - the AgentX API never
     touches the customer's endpoint.
 
     The endpoint receives a POST with::

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/evaluations/adapters/precomputed.py

@@ -8,7 +8,7 @@ from agentx.evaluations.results import normalize_result
 
 class PrecomputedAdapter:
     """
-    Adapter for pre-computed outputs — useful when you already have agent
+    Adapter for pre-computed outputs - useful when you already have agent
     responses and just want AgentX to score them.
 
     Accepts a list or dict keyed by case_id::

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/evaluations/client.py

@@ -57,7 +57,7 @@ class EvaluationsClient:
             raise AgentXAuthError("AGENTX_API_KEY is required")
         self._api_key = api_key
         self._sdk_version = sdk_version
-        # Falls back to the caller's default workspace server-side when unset — see
+        # Falls back to the caller's default workspace server-side when unset - see
         # _with_workspace(). Without this, dataset/settings/run creation silently land in
         # whatever workspace the API key's user defaults to, not the one the caller intended.
         self._workspace_id = workspace_id
@@ -139,7 +139,7 @@ class EvaluationsClient:
     # ------------------------------------------------------------------
 
     def list_models(self, provider: Optional[str] = None) -> List[ModelInfo]:
-        """List the LLM models AgentX supports — the same set selectable for
+        """List the LLM models AgentX supports - the same set selectable for
         the Sovereignty & Portability Index. Pass ``provider`` (e.g. "Google")
         to filter."""
         params = {"provider": provider} if provider else None
@@ -169,7 +169,7 @@ class EvaluationsClient:
         return Dataset(**data)
 
     # ------------------------------------------------------------------
-    # Evaluation Settings endpoints — standalone grading config, reusable
+    # Evaluation Settings endpoints - standalone grading config, reusable
     # across datasets.
     # ------------------------------------------------------------------
 
@@ -199,7 +199,7 @@ class EvaluationsClient:
         return EvaluationSettings(**data)
 
     # ------------------------------------------------------------------
-    # Prompt registry endpoints — see agentx.evaluations.prompts.PromptClient for the concept
+    # Prompt registry endpoints - see agentx.evaluations.prompts.PromptClient for the concept
     # (the external-agent analog to native autotune). Deliberately read-mostly: no publish here,
     # a new version only ever comes from the dashboard's human-approved propose/publish flow.
     # ------------------------------------------------------------------
@@ -265,6 +265,34 @@ class EvaluationsClient:
             "POST", f"/runs/{run_id}/finalize", json={"status": "completed"}
         )
 
+    def gate_run(
+        self,
+        run_id: str,
+        *,
+        fail_under: Optional[float] = None,
+        no_regression: bool = False,
+        tolerance: Optional[float] = None,
+        record: bool = True,
+        caller: Optional[str] = "sdk",
+    ) -> Dict[str, Any]:
+        # CI gate (self-host): pass/fail a finalized run against an absolute rating floor and/or
+        # the dataset's previous completed run. Recorded into gate history by default (the
+        # dashboard's CI page lists these); pass record=False for a preview that leaves no trace.
+        # `caller` is a free label shown in that history ("sdk", "github-actions", ...). See
+        # EvaluationRunContext.gate() for the CI-facing wrapper with printed verdicts.
+        params: Dict[str, Any] = {}
+        if fail_under is not None:
+            params["failUnder"] = fail_under
+        if no_regression:
+            params["noRegression"] = "true"
+        if tolerance is not None:
+            params["tolerance"] = tolerance
+        if record:
+            params["record"] = "true"
+            if caller:
+                params["caller"] = caller
+        return self._request("GET", f"/runs/{run_id}/gate", params=params)
+
     def analyze_run(
         self,
         run_id: str,

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/evaluations/datasets.py

@@ -67,7 +67,7 @@ class DatasetBuilder:
             self._payload["bleuScore"] = {"enabled": True}
         if rouge_score:
             self._payload["rougeScore"] = {"enabled": True}
-        # Sovereignty & Portability — the models to compare on this dataset (use
+        # Sovereignty & Portability - the models to compare on this dataset (use
         # client.evaluations.list_models() to discover valid ids).
         if sovereignty_models:
             self._payload["sovereigntyIndex"] = {

{agentx_python-0.6.15 → agentx_python-0.6.17}/agentx/evaluations/evaluation_settings.py

@@ -61,7 +61,7 @@ class EvaluationSettingsBuilder:
             self._payload["bleuScore"] = {"enabled": True}
         if rouge_score:
             self._payload["rougeScore"] = {"enabled": True}
-        # Sovereignty & Portability — the models to compare when this config runs
+        # Sovereignty & Portability - the models to compare when this config runs
         # (use client.evaluations.list_models() to discover valid ids).
         if sovereignty_models:
             self._payload["sovereigntyIndex"] = {