agentx-python 0.6.10__tar.gz → 0.6.12__tar.gz

{agentx_python-0.6.10 → agentx_python-0.6.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentx-python
-Version: 0.6.10
+Version: 0.6.12
 Summary: Official Python SDK for AgentX (https://www.agentx.so/)
 Home-page: https://github.com/AgentX-ai/AgentX-python
 Author: Robin Wang and AgentX Team
@@ -21,19 +21,33 @@ Provides-Extra: crewai
 Requires-Dist: crewai>=0.80.0; extra == "crewai"
 Provides-Extra: openai-agents
 Requires-Dist: openai-agents>=0.0.3; extra == "openai-agents"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0.0; extra == "openai"
 Provides-Extra: anthropic
 Requires-Dist: anthropic>=0.25.0; extra == "anthropic"
 Provides-Extra: google-adk
 Requires-Dist: google-adk>=1.0.0; extra == "google-adk"
 Provides-Extra: google-genai
 Requires-Dist: google-genai>=1.0.0; extra == "google-genai"
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.0.0; extra == "litellm"
+Provides-Extra: llamaindex
+Requires-Dist: llama-index-core>=0.10.0; extra == "llamaindex"
+Provides-Extra: autogen
+Requires-Dist: autogen-agentchat>=0.4.0; extra == "autogen"
+Requires-Dist: autogen-core>=0.4.0; extra == "autogen"
 Provides-Extra: all
 Requires-Dist: langchain-core>=0.1.0; extra == "all"
 Requires-Dist: crewai>=0.80.0; extra == "all"
 Requires-Dist: openai-agents>=0.0.3; extra == "all"
+Requires-Dist: openai>=1.0.0; extra == "all"
 Requires-Dist: anthropic>=0.25.0; extra == "all"
 Requires-Dist: google-adk>=1.0.0; extra == "all"
 Requires-Dist: google-genai>=1.0.0; extra == "all"
+Requires-Dist: litellm>=1.0.0; extra == "all"
+Requires-Dist: llama-index-core>=0.10.0; extra == "all"
+Requires-Dist: autogen-agentchat>=0.4.0; extra == "all"
+Requires-Dist: autogen-core>=0.4.0; extra == "all"
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -235,9 +249,13 @@ Each integration auto-captures LLM calls, tool calls, and token usage — instal
 | LangChain             | `pip install "agentx-python[langchain]"`     | `AgentXCallbackHandler`  |
 | CrewAI                | `pip install "agentx-python[crewai]"`        | `AgentXCrewObserver`     |
 | OpenAI Agents SDK     | `pip install "agentx-python[openai-agents]"` | `AgentXTracingProcessor` |
+| OpenAI (raw client)   | `pip install "agentx-python[openai]"`        | `patch_openai_client`    |
 | Anthropic             | `pip install "agentx-python[anthropic]"`     | `patch_anthropic_client` |
 | Google ADK            | `pip install "agentx-python[google-adk]"`    | `AgentXADKPlugin`        |
 | Google GenAI (Gemini) | `pip install "agentx-python[google-genai]"`  | `patch_genai_client`     |
+| LiteLLM               | `pip install "agentx-python[litellm]"`       | `AgentXLiteLLMLogger`    |
+| LlamaIndex             | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler`|
+| AutoGen                | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`  |
 
 Or plain Python — wrap any function with `@tracer.trace(...)` and it just works, no framework required.
 
@@ -280,6 +298,19 @@ Per-agent coverage/threshold settings (sample rate, retention, and threshold ove
 client.monitor.profile.update("agent_123", threshold_overrides={"latencyMs": 15000})
 ```
 
+Self-host also has **online evaluators**: a real LLM judge scoring a sample of live traffic continuously, distinct from a pattern's rule-matching. A score below `alert_threshold` raises a signal the same way a failing pattern does, deduped and triage-ready in `client.monitor.signals`.
+
+```python
+evaluator = client.monitor.online_evaluators.builder(
+    name="Helpfulness",
+    evaluation_settings_id=settings.id,
+    sample_rate=0.1,
+    alert_threshold=5,
+).publish()
+
+client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
+```
+
 See **[TRACING.md](TRACING.md)** for the complete Monitor guide.
 
 ---
@@ -309,7 +340,21 @@ print(report.recommendations)      # list of prioritized, actionable fixes
 
 Ask a case's question several extra ways each run, LLM-paraphrased server-side, to catch agents that break on phrasing rather than substance, and override the judge's prompt/model per config, see [Smoke testing](EVALUATIONS.md#smoke-testing-phrasing-robustness) and [Configuring the judge](EVALUATIONS.md#configuring-the-judge) in the full guide.
 
-See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, and the complete API reference.
+Since AgentX doesn't own your agent's code, `client.evaluations.prompts` lets AgentX become your prompt's *source of truth* instead — the same problem LangSmith's Prompt Hub and Langfuse's Prompt Management solve. Pull a version at runtime, tag your eval runs (or live traces) with it, and let a judge propose a rewrite from your real worst-rated results — a human always has to approve before it publishes:
+
+```python
+prompt = client.evaluations.prompts.get("support-agent-system-prompt")  # or prompt.id
+# use prompt.text as your own agent's system prompt
+
+client.evaluations.run(
+    dataset_id="evds_…",
+    subject={"kind": "custom_agent", "metadata": {"promptName": prompt.name}},
+).execute(my_agent_fn)
+```
+
+See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/self-host#prompt-registry) for the "Suggest improvement" dashboard flow (self-host only — no hosted-SaaS equivalent yet).
+
+See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
 ---
 

agentx_python-0.6.10/agentx_python.egg-info/PKG-INFO → agentx_python-0.6.12/README.md

@@ -1,51 +1,3 @@
-Metadata-Version: 2.4
-Name: agentx-python
-Version: 0.6.10
-Summary: Official Python SDK for AgentX (https://www.agentx.so/)
-Home-page: https://github.com/AgentX-ai/AgentX-python
-Author: Robin Wang and AgentX Team
-Author-email: contact@agentx.so
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.6
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: urllib3>=1.26.11
-Requires-Dist: certifi
-Requires-Dist: requests
-Requires-Dist: pydantic>=2.0.0
-Provides-Extra: langchain
-Requires-Dist: langchain-core>=0.1.0; extra == "langchain"
-Provides-Extra: crewai
-Requires-Dist: crewai>=0.80.0; extra == "crewai"
-Provides-Extra: openai-agents
-Requires-Dist: openai-agents>=0.0.3; extra == "openai-agents"
-Provides-Extra: anthropic
-Requires-Dist: anthropic>=0.25.0; extra == "anthropic"
-Provides-Extra: google-adk
-Requires-Dist: google-adk>=1.0.0; extra == "google-adk"
-Provides-Extra: google-genai
-Requires-Dist: google-genai>=1.0.0; extra == "google-genai"
-Provides-Extra: all
-Requires-Dist: langchain-core>=0.1.0; extra == "all"
-Requires-Dist: crewai>=0.80.0; extra == "all"
-Requires-Dist: openai-agents>=0.0.3; extra == "all"
-Requires-Dist: anthropic>=0.25.0; extra == "all"
-Requires-Dist: google-adk>=1.0.0; extra == "all"
-Requires-Dist: google-genai>=1.0.0; extra == "all"
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license-file
-Dynamic: provides-extra
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
 ![Logo](https://agentx-resources.s3.us-west-1.amazonaws.com/AgentX-logo-387x60.png)
 
 [![PyPI version](https://img.shields.io/pypi/v/agentx-python)](https://pypi.org/project/agentx-python/)
@@ -235,9 +187,13 @@ Each integration auto-captures LLM calls, tool calls, and token usage — instal
 | LangChain             | `pip install "agentx-python[langchain]"`     | `AgentXCallbackHandler`  |
 | CrewAI                | `pip install "agentx-python[crewai]"`        | `AgentXCrewObserver`     |
 | OpenAI Agents SDK     | `pip install "agentx-python[openai-agents]"` | `AgentXTracingProcessor` |
+| OpenAI (raw client)   | `pip install "agentx-python[openai]"`        | `patch_openai_client`    |
 | Anthropic             | `pip install "agentx-python[anthropic]"`     | `patch_anthropic_client` |
 | Google ADK            | `pip install "agentx-python[google-adk]"`    | `AgentXADKPlugin`        |
 | Google GenAI (Gemini) | `pip install "agentx-python[google-genai]"`  | `patch_genai_client`     |
+| LiteLLM               | `pip install "agentx-python[litellm]"`       | `AgentXLiteLLMLogger`    |
+| LlamaIndex             | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler`|
+| AutoGen                | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`  |
 
 Or plain Python — wrap any function with `@tracer.trace(...)` and it just works, no framework required.
 
@@ -280,6 +236,19 @@ Per-agent coverage/threshold settings (sample rate, retention, and threshold ove
 client.monitor.profile.update("agent_123", threshold_overrides={"latencyMs": 15000})
 ```
 
+Self-host also has **online evaluators**: a real LLM judge scoring a sample of live traffic continuously, distinct from a pattern's rule-matching. A score below `alert_threshold` raises a signal the same way a failing pattern does, deduped and triage-ready in `client.monitor.signals`.
+
+```python
+evaluator = client.monitor.online_evaluators.builder(
+    name="Helpfulness",
+    evaluation_settings_id=settings.id,
+    sample_rate=0.1,
+    alert_threshold=5,
+).publish()
+
+client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
+```
+
 See **[TRACING.md](TRACING.md)** for the complete Monitor guide.
 
 ---
@@ -309,7 +278,21 @@ print(report.recommendations)      # list of prioritized, actionable fixes
 
 Ask a case's question several extra ways each run, LLM-paraphrased server-side, to catch agents that break on phrasing rather than substance, and override the judge's prompt/model per config, see [Smoke testing](EVALUATIONS.md#smoke-testing-phrasing-robustness) and [Configuring the judge](EVALUATIONS.md#configuring-the-judge) in the full guide.
 
-See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, and the complete API reference.
+Since AgentX doesn't own your agent's code, `client.evaluations.prompts` lets AgentX become your prompt's *source of truth* instead — the same problem LangSmith's Prompt Hub and Langfuse's Prompt Management solve. Pull a version at runtime, tag your eval runs (or live traces) with it, and let a judge propose a rewrite from your real worst-rated results — a human always has to approve before it publishes:
+
+```python
+prompt = client.evaluations.prompts.get("support-agent-system-prompt")  # or prompt.id
+# use prompt.text as your own agent's system prompt
+
+client.evaluations.run(
+    dataset_id="evds_…",
+    subject={"kind": "custom_agent", "metadata": {"promptName": prompt.name}},
+).execute(my_agent_fn)
+```
+
+See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/self-host#prompt-registry) for the "Suggest improvement" dashboard flow (self-host only — no hosted-SaaS equivalent yet).
+
+See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
 ---
 

{agentx_python-0.6.10 → agentx_python-0.6.12}/agentx/evaluations/client.py

@@ -17,6 +17,7 @@ from agentx.evaluations.models import (
     EvaluationSettings,
     EvaluationSubject,
     ModelInfo,
+    Prompt,
     Report,
 )
 
@@ -77,12 +78,14 @@ class EvaluationsClient:
                 "accept": "*/*",
             }
         )
-        # Expose dataset / evaluation-settings builder factories
+        # Expose dataset / evaluation-settings / prompt builder factories
         from agentx.evaluations.datasets import DatasetClient
         from agentx.evaluations.evaluation_settings import EvaluationSettingsClient
+        from agentx.evaluations.prompts import PromptClient
 
         self.datasets = DatasetClient(self)
         self.settings = EvaluationSettingsClient(self)
+        self.prompts = PromptClient(self)
 
     # ------------------------------------------------------------------
     # Low-level HTTP
@@ -195,6 +198,29 @@ class EvaluationsClient:
         )
         return EvaluationSettings(**data)
 
+    # ------------------------------------------------------------------
+    # Prompt registry endpoints — see agentx.evaluations.prompts.PromptClient for the concept
+    # (the external-agent analog to native autotune). Deliberately read-mostly: no publish here,
+    # a new version only ever comes from the dashboard's human-approved propose/publish flow.
+    # ------------------------------------------------------------------
+
+    def create_prompt(self, payload: dict) -> Prompt:
+        data = self._request("POST", "/prompts", json=self._with_workspace(payload))
+        return Prompt(**data)
+
+    def list_prompts(self) -> List[Prompt]:
+        data = self._request("GET", "/prompts", params=self._workspace_params())
+        return [Prompt(**p) for p in (data if isinstance(data, list) else data.get("prompts", []))]
+
+    def get_prompt(self, name: str, version: Optional[int] = None) -> Prompt:
+        # `name` doubles as an id: the backend route tries a name match first, then falls back to
+        # an id match (see engine's getPromptRowByNameOrId), so callers can pass either.
+        params = self._workspace_params() or {}
+        if version is not None:
+            params = {**params, "version": version}
+        data = self._request("GET", f"/prompts/{name}", params=params or None)
+        return Prompt(**data)
+
     # ------------------------------------------------------------------
     # Run endpoints
     # ------------------------------------------------------------------

{agentx_python-0.6.10 → agentx_python-0.6.12}/agentx/evaluations/models.py

@@ -128,6 +128,30 @@ class EvaluationSettings(BaseModel):
         extra = "ignore"
 
 
+# ---------------------------------------------------------------------------
+# Prompt registry — the external-agent analog to AgentX's native autotune. AgentX doesn't own
+# your agent's code, so instead of branching/applying a config it becomes the prompt's source of
+# truth (same shape as LangSmith's Prompt Hub / Langfuse's Prompt Management): pull a version at
+# runtime with ``client.evaluations.prompts.get(name_or_id)``, use ``prompt.text`` as your agent's
+# system prompt, and tag your eval runs so the existing version-comparison view can tell you
+# which published version scored higher — see ``client.evaluations.prompts`` docs.
+# ---------------------------------------------------------------------------
+
+
+class Prompt(BaseModel):
+    id: str = Field(alias="_id")
+    name: str
+    description: Optional[str] = None
+    version: int
+    text: str
+    created_at: Optional[str] = Field(default=None, alias="createdAt")
+    updated_at: Optional[str] = Field(default=None, alias="updatedAt")
+
+    class Config:
+        populate_by_name = True
+        extra = "ignore"
+
+
 # ---------------------------------------------------------------------------
 # Evaluation subject
 # ---------------------------------------------------------------------------

agentx_python-0.6.12/agentx/evaluations/prompts.py

@@ -0,0 +1,55 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, List, Optional
+
+from agentx.evaluations.models import Prompt
+
+if TYPE_CHECKING:
+    from agentx.evaluations.client import EvaluationsClient
+
+
+class PromptClient:
+    """Thin wrapper surfaced as ``client.evaluations.prompts`` — the external-agent analog to
+    AgentX's native autotune. AgentX doesn't own your agent's code, so instead of branching and
+    applying a config, it becomes the prompt's source of truth (the same shape as LangSmith's
+    Prompt Hub / Langfuse's Prompt Management): create/pull versions here, use ``prompt.text`` as
+    your agent's own system prompt, and tag your eval runs with the pulled version so the
+    existing version-comparison view (``client.evaluations`` run comparisons on a dataset) can
+    tell you which published version actually scored higher.
+
+    Deliberately read-mostly from here: there is no ``publish`` on this client. A prompt only
+    gets a new version through the dashboard's human-approved propose/publish flow, so a
+    rewritten prompt never reaches your running agent without someone explicitly approving it.
+
+    Example::
+
+        prompt = client.evaluations.prompts.get("support-agent-system-prompt")
+        # ... call your own LLM using prompt.text as the system prompt ...
+        client.evaluations.init_run(
+            dataset_id=dataset_id,
+            subject=EvaluationSubject(
+                metadata={
+                    "promptName": prompt.name,
+                    "version": f"{prompt.name}@v{prompt.version}",
+                }
+            ),
+        )
+
+    ``get()`` also accepts a prompt's ``id`` in place of its name, e.g.
+    ``client.evaluations.prompts.get(prompt.id)``.
+    """
+
+    def __init__(self, client: "EvaluationsClient"):
+        self._client = client
+
+    def create(self, name: str, text: str, description: Optional[str] = None) -> Prompt:
+        return self._client.create_prompt({"name": name, "text": text, "description": description})
+
+    def get(self, name: str, version: Optional[int] = None) -> Prompt:
+        """Accepts either the prompt's name or its ``id`` (e.g. ``prompt.id`` from an earlier
+        ``get``/``create`` call), useful for round-tripping an id you already have without a
+        second lookup method."""
+        return self._client.get_prompt(name, version=version)
+
+    def list(self) -> List[Prompt]:
+        return self._client.list_prompts()

{agentx_python-0.6.10 → agentx_python-0.6.12}/agentx/evaluations/runner.py

@@ -335,6 +335,7 @@ class EvaluationsRunner:
         self._client = client
         self.datasets = client.datasets
         self.settings = client.settings
+        self.prompts = client.prompts
 
     def list_models(self, provider: Optional[str] = None) -> List[ModelInfo]:
         """List the LLM models AgentX supports — the same set selectable for

agentx_python-0.6.12/agentx/integrations/_traced_call.py

@@ -0,0 +1,141 @@
+"""
+Shared helper for tracing a raw SDK client call that may be sync or async.
+
+Anthropic's and Google GenAI's raw clients expose the same method names for
+both their sync and async client variants (``client.messages.create``,
+``client.models.generate_content``) — the only way to tell them apart is to
+call the method and check whether the result is awaitable.
+``inspect.iscoroutinefunction`` is unreliable for this: it returns ``False``
+even for ``AsyncAnthropic().messages.create``, since these SDKs don't
+implement the async variant as a plain top-level ``async def``.
+"""
+from __future__ import annotations
+
+import asyncio
+import inspect
+from typing import Any, Callable, Dict, Optional
+
+from agentx.tracing.tracer import Tracer
+from agentx.integrations._perf import build_performance_summary
+
+
+def call_and_trace(
+    original: Callable[..., Any],
+    args: tuple,
+    kwargs: dict,
+    on_finish: Callable[[Optional[Any], Optional[str]], None],
+) -> Any:
+    """
+    Call ``original(*args, **kwargs)``.
+
+    If the result is awaitable (async client), return a coroutine that awaits
+    it and calls ``on_finish(response, error)`` only after the real await
+    completes, so timing/output/tokens reflect the actual call rather than
+    the moment the coroutine object was constructed. If the result is a
+    normal value (sync client), call ``on_finish`` immediately.
+
+    Either way, the original call's own return value / exception behavior is
+    unchanged for the caller — this only affects when/how the trace is built.
+    """
+    try:
+        result = original(*args, **kwargs)
+    except Exception as exc:
+        on_finish(None, str(exc))
+        raise
+
+    if asyncio.iscoroutine(result) or inspect.isawaitable(result):
+        return _await_and_finish(result, on_finish)
+
+    on_finish(result, None)
+    return result
+
+
+async def _await_and_finish(
+    awaitable: Any,
+    on_finish: Callable[[Optional[Any], Optional[str]], None],
+) -> Any:
+    try:
+        response = await awaitable
+    except Exception as exc:
+        on_finish(None, str(exc))
+        raise
+    on_finish(response, None)
+    return response
+
+
+def finish_llm_call(
+    tracer: Tracer,
+    *,
+    name: str,
+    framework: str,
+    metadata: Optional[Dict[str, Any]],
+    session_id: Optional[str],
+    start_t: float,
+    end_t: float,
+    input_repr: Any,
+    output: Optional[str],
+    model: Optional[str],
+    input_tokens: Optional[int],
+    output_tokens: Optional[int],
+    error: Optional[str],
+) -> None:
+    """
+    Close out one raw-client LLM call — shared by the ``on_finish``/exit
+    callbacks of every integration that patches a raw provider client
+    (``anthropic.py``, ``google_genai.py``, ``openai.py``) rather than a
+    framework-level callback/plugin system.
+
+    If the call happened inside a ``with tracer.trace(...)`` block, attach it
+    as one LLM-call step on that span instead of sending an independent
+    trace — the same "part of a multi-call agentic loop" behavior
+    ``anthropic.py`` already had; folded in here so every raw-client
+    integration gets it instead of each having to remember to check
+    ``tracer.current_span`` itself.
+    """
+    latency_ms = int((end_t - start_t) * 1000)
+
+    active_span = tracer.current_span
+    if active_span is not None:
+        if error is not None:
+            active_span.set_error(error)
+        active_span._record_llm_call(
+            duration_ms=latency_ms,
+            start_time=start_t,
+            end_time=end_t,
+            input=input_repr,
+            output=output,
+            model=model,
+            input_tokens=input_tokens,
+            output_tokens=output_tokens,
+        )
+        return
+
+    perf = build_performance_summary(
+        total_duration_ms=latency_ms,
+        execution_steps=[{
+            "name": "LLM Call 1",
+            "duration_ms": latency_ms,
+            "start_time": start_t,
+            "end_time": end_t,
+            "model": model,
+            "input": input_repr,
+            "output": output,
+            "inputTokenSize": input_tokens,
+            "outputTokenSize": output_tokens,
+        }],
+        has_errors=error is not None,
+    )
+    tracer._send(
+        name=name,
+        input=input_repr,
+        output=output,
+        latency_ms=latency_ms,
+        error=error,
+        framework=framework,
+        model=model,
+        metadata=metadata,
+        session_id=session_id,
+        performance_summary=perf,
+        input_tokens=input_tokens,
+        output_tokens=output_tokens,
+    )