agentx-python 0.8.13__tar.gz → 0.8.14__tar.gz

{agentx_python-0.8.13 → agentx_python-0.8.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentx-python
-Version: 0.8.13
+Version: 0.8.14
 Summary: Official Python SDK for AgentX (https://www.agentx.so/)
 Home-page: https://github.com/AgentX-ai/AgentX-python
 Author: Robin Wang and AgentX Team
@@ -8,7 +8,7 @@ Author-email: contact@agentx.so
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.6
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: urllib3>=1.26.11
@@ -50,6 +50,7 @@ Requires-Dist: litellm>=1.0.0; extra == "all"
 Requires-Dist: llama-index-core>=0.10.0; extra == "all"
 Requires-Dist: autogen-agentchat>=0.4.0; extra == "all"
 Requires-Dist: autogen-core>=0.4.0; extra == "all"
+Requires-Dist: mlflow>=3.6.0; extra == "all"
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -109,12 +110,14 @@ pip install --upgrade agentx-python
 
 Requires Python 3.9 or newer.
 
-#### Run self host eval framework locally
+#### Run the self-host governance suite locally
 
-```
-agentx-trace-eval --dev --update
+```bash
+agentx-trace-eval --dev
 ```
 
+(See [Self-host](#self-host) below for what this downloads and how to point the SDK at it.)
+
 ---
 
 ## Quick start
@@ -137,7 +140,7 @@ report = (
     .analyze()
 )
 
-print(report.average_rating)   # LLM-graded score, 0–10
+print(report.average_rating)   # LLM-graded score, 0-10
 print(report.summary)          # AI-generated narrative from .analyze()
 ```
 
@@ -158,9 +161,9 @@ report = (
     .analyze()
 )
 
-print(report.average_rating)       # LLM-graded score, 0–10
-print(report.cosine_similarity)    # embedding cosine, 0–1 (None if not enabled)
-print(report.jaccard_similarity)   # token-set overlap, 0–1 (None if not enabled)
+print(report.average_rating)       # LLM-graded score, 0-10
+print(report.cosine_similarity)    # embedding cosine, 0-1 (None if not enabled)
+print(report.jaccard_similarity)   # token-set overlap, 0-1 (None if not enabled)
 
 print(report.summary)              # AI-generated narrative from .analyze()
 print(report.recommendations)      # list of prioritized, actionable fixes
@@ -184,7 +187,7 @@ client.evaluations.run(
 
 See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
 
-On self-host, a finalized run can also **gate a CI job**: `run.gate(fail_under=7, no_regression=True)` (on the run context `.execute()` returns) checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
+On self-host, a finalized run can also **gate a CI job**: `run.gate(fail_under=7, no_regression=True)` (on the run context `.execute()` returns) checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns a `GateResult` - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
 
 See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
@@ -252,6 +255,10 @@ extra:
 | LlamaIndex            | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler` |
 | AutoGen               | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`   |
 
+Two more platforms are covered by **pull importers** rather than in-process hooks, each with its
+own CLI: `agentx-moveworks` (Moveworks Data API sync, no extra needed) and `agentx-databricks`
+(`pip install "agentx-python[databricks]"`, MLflow/Databricks trace sync).
+
 Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework
 required. Tracing is **platform agnostic**: each integration stamps its platform label
 automatically, a plain trace auto-detects the one orchestration framework imported in the
@@ -292,12 +299,14 @@ for signal in client.monitor.signals.list(severity="high"):
     print(signal.summary, signal.occurrence_count)
 ```
 
-Per-agent coverage/threshold settings (sample rate, retention, and threshold overrides like the built-in "Latency regression" pattern's threshold) are `client.monitor.profile.get()`/`.update()`:
+Per-agent monitoring settings (enable/disable, detection categories, notification channels) are `client.monitor.profile.get()`/`.update()`:
 
 ```python
-client.monitor.profile.update("agent_123", threshold_overrides={"latencyMs": 15000})
+client.monitor.profile.update("agent_123", info_detection_enabled=False)
 ```
 
+On self-host, coverage mode, sample rate, retention, and the built-in latency threshold are project-level defaults set in the dashboard's Platform Settings; `update()` still accepts them for wire compatibility, but only the per-agent fields above take effect there.
+
 Self-host also has **online evaluators**: a real LLM judge scoring a sample of live traffic continuously, distinct from a pattern's rule-matching. A score below `alert_threshold` raises a signal the same way a failing pattern does, deduped and triage-ready in `client.monitor.signals`.
 
 ```python
@@ -371,7 +380,7 @@ export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
 export AGENTX_API_KEY=<printed by agentx-trace-eval on first run>
 ```
 
-`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. See that repo's README for what's included, and `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
+`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. Each SDK release pins the engine release it was tested against and converges the install to that pin, so upgrading the SDK upgrades the engine too. Two flags to know: `--update` (consumed by this launcher) force-reinstalls the resolved engine release, while `--upgrade` passes through to `agentx-server` and re-downloads the dashboard bundle before serving. See that repo's README for what's included, and the `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
 
 ---
 

{agentx_python-0.8.13 → agentx_python-0.8.14}/README.md

@@ -45,12 +45,14 @@ pip install --upgrade agentx-python
 
 Requires Python 3.9 or newer.
 
-#### Run self host eval framework locally
+#### Run the self-host governance suite locally
 
-```
-agentx-trace-eval --dev --update
+```bash
+agentx-trace-eval --dev
 ```
 
+(See [Self-host](#self-host) below for what this downloads and how to point the SDK at it.)
+
 ---
 
 ## Quick start
@@ -73,7 +75,7 @@ report = (
     .analyze()
 )
 
-print(report.average_rating)   # LLM-graded score, 0–10
+print(report.average_rating)   # LLM-graded score, 0-10
 print(report.summary)          # AI-generated narrative from .analyze()
 ```
 
@@ -94,9 +96,9 @@ report = (
     .analyze()
 )
 
-print(report.average_rating)       # LLM-graded score, 0–10
-print(report.cosine_similarity)    # embedding cosine, 0–1 (None if not enabled)
-print(report.jaccard_similarity)   # token-set overlap, 0–1 (None if not enabled)
+print(report.average_rating)       # LLM-graded score, 0-10
+print(report.cosine_similarity)    # embedding cosine, 0-1 (None if not enabled)
+print(report.jaccard_similarity)   # token-set overlap, 0-1 (None if not enabled)
 
 print(report.summary)              # AI-generated narrative from .analyze()
 print(report.recommendations)      # list of prioritized, actionable fixes
@@ -120,7 +122,7 @@ client.evaluations.run(
 
 See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
 
-On self-host, a finalized run can also **gate a CI job**: `run.gate(fail_under=7, no_regression=True)` (on the run context `.execute()` returns) checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
+On self-host, a finalized run can also **gate a CI job**: `run.gate(fail_under=7, no_regression=True)` (on the run context `.execute()` returns) checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns a `GateResult` - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
 
 See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
@@ -188,6 +190,10 @@ extra:
 | LlamaIndex            | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler` |
 | AutoGen               | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`   |
 
+Two more platforms are covered by **pull importers** rather than in-process hooks, each with its
+own CLI: `agentx-moveworks` (Moveworks Data API sync, no extra needed) and `agentx-databricks`
+(`pip install "agentx-python[databricks]"`, MLflow/Databricks trace sync).
+
 Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework
 required. Tracing is **platform agnostic**: each integration stamps its platform label
 automatically, a plain trace auto-detects the one orchestration framework imported in the
@@ -228,12 +234,14 @@ for signal in client.monitor.signals.list(severity="high"):
     print(signal.summary, signal.occurrence_count)
 ```
 
-Per-agent coverage/threshold settings (sample rate, retention, and threshold overrides like the built-in "Latency regression" pattern's threshold) are `client.monitor.profile.get()`/`.update()`:
+Per-agent monitoring settings (enable/disable, detection categories, notification channels) are `client.monitor.profile.get()`/`.update()`:
 
 ```python
-client.monitor.profile.update("agent_123", threshold_overrides={"latencyMs": 15000})
+client.monitor.profile.update("agent_123", info_detection_enabled=False)
 ```
 
+On self-host, coverage mode, sample rate, retention, and the built-in latency threshold are project-level defaults set in the dashboard's Platform Settings; `update()` still accepts them for wire compatibility, but only the per-agent fields above take effect there.
+
 Self-host also has **online evaluators**: a real LLM judge scoring a sample of live traffic continuously, distinct from a pattern's rule-matching. A score below `alert_threshold` raises a signal the same way a failing pattern does, deduped and triage-ready in `client.monitor.signals`.
 
 ```python
@@ -307,7 +315,7 @@ export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
 export AGENTX_API_KEY=<printed by agentx-trace-eval on first run>
 ```
 
-`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. See that repo's README for what's included, and `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
+`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. Each SDK release pins the engine release it was tested against and converges the install to that pin, so upgrading the SDK upgrades the engine too. Two flags to know: `--update` (consumed by this launcher) force-reinstalls the resolved engine release, while `--upgrade` passes through to `agentx-server` and re-downloads the dashboard bundle before serving. See that repo's README for what's included, and the `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
 
 ---
 

{agentx_python-0.8.13 → agentx_python-0.8.14}/agentx/integrations/anthropic.py

@@ -13,7 +13,7 @@ Usage::
 
 Works with both ``anthropic.Anthropic`` and ``anthropic.AsyncAnthropic`` clients.
 
-Requires: ``pip install agentx[anthropic]``
+Requires: ``pip install "agentx-python[anthropic]"``
 """
 from __future__ import annotations
 

{agentx_python-0.8.13 → agentx_python-0.8.14}/agentx/integrations/crewai.py

@@ -14,7 +14,7 @@ Or as a context manager around your own kickoff::
         result = crew.kickoff(inputs={"topic": "AI"})
         span.output = result.raw
 
-Requires: ``pip install agentx[crewai]``
+Requires: ``pip install "agentx-python[crewai]"``
 """
 from __future__ import annotations
 

{agentx_python-0.8.13 → agentx_python-0.8.14}/agentx/version.py

@@ -1,7 +1,7 @@
-VERSION = "0.8.13"
+VERSION = "0.8.14"
 
 # The AgentX-trace-eval release this SDK version is tested against - what `agentx-trace-eval`
 # installs and converges to (see agentx/cli.py). Bump together with VERSION when releasing, so
 # every published SDK names a known-good engine+dashboard pair. Users can override with
 # AGENTX_TRACE_EVAL_VERSION=<tag|latest>.
-ENGINE_VERSION = "v0.3.7"
+ENGINE_VERSION = "v0.3.10"

{agentx_python-0.8.13 → agentx_python-0.8.14}/agentx_python.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentx-python
-Version: 0.8.13
+Version: 0.8.14
 Summary: Official Python SDK for AgentX (https://www.agentx.so/)
 Home-page: https://github.com/AgentX-ai/AgentX-python
 Author: Robin Wang and AgentX Team
@@ -8,7 +8,7 @@ Author-email: contact@agentx.so
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.6
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: urllib3>=1.26.11
@@ -50,6 +50,7 @@ Requires-Dist: litellm>=1.0.0; extra == "all"
 Requires-Dist: llama-index-core>=0.10.0; extra == "all"
 Requires-Dist: autogen-agentchat>=0.4.0; extra == "all"
 Requires-Dist: autogen-core>=0.4.0; extra == "all"
+Requires-Dist: mlflow>=3.6.0; extra == "all"
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -109,12 +110,14 @@ pip install --upgrade agentx-python
 
 Requires Python 3.9 or newer.
 
-#### Run self host eval framework locally
+#### Run the self-host governance suite locally
 
-```
-agentx-trace-eval --dev --update
+```bash
+agentx-trace-eval --dev
 ```
 
+(See [Self-host](#self-host) below for what this downloads and how to point the SDK at it.)
+
 ---
 
 ## Quick start
@@ -137,7 +140,7 @@ report = (
     .analyze()
 )
 
-print(report.average_rating)   # LLM-graded score, 0–10
+print(report.average_rating)   # LLM-graded score, 0-10
 print(report.summary)          # AI-generated narrative from .analyze()
 ```
 
@@ -158,9 +161,9 @@ report = (
     .analyze()
 )
 
-print(report.average_rating)       # LLM-graded score, 0–10
-print(report.cosine_similarity)    # embedding cosine, 0–1 (None if not enabled)
-print(report.jaccard_similarity)   # token-set overlap, 0–1 (None if not enabled)
+print(report.average_rating)       # LLM-graded score, 0-10
+print(report.cosine_similarity)    # embedding cosine, 0-1 (None if not enabled)
+print(report.jaccard_similarity)   # token-set overlap, 0-1 (None if not enabled)
 
 print(report.summary)              # AI-generated narrative from .analyze()
 print(report.recommendations)      # list of prioritized, actionable fixes
@@ -184,7 +187,7 @@ client.evaluations.run(
 
 See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
 
-On self-host, a finalized run can also **gate a CI job**: `run.gate(fail_under=7, no_regression=True)` (on the run context `.execute()` returns) checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
+On self-host, a finalized run can also **gate a CI job**: `run.gate(fail_under=7, no_regression=True)` (on the run context `.execute()` returns) checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns a `GateResult` - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
 
 See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
@@ -252,6 +255,10 @@ extra:
 | LlamaIndex            | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler` |
 | AutoGen               | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`   |
 
+Two more platforms are covered by **pull importers** rather than in-process hooks, each with its
+own CLI: `agentx-moveworks` (Moveworks Data API sync, no extra needed) and `agentx-databricks`
+(`pip install "agentx-python[databricks]"`, MLflow/Databricks trace sync).
+
 Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework
 required. Tracing is **platform agnostic**: each integration stamps its platform label
 automatically, a plain trace auto-detects the one orchestration framework imported in the
@@ -292,12 +299,14 @@ for signal in client.monitor.signals.list(severity="high"):
     print(signal.summary, signal.occurrence_count)
 ```
 
-Per-agent coverage/threshold settings (sample rate, retention, and threshold overrides like the built-in "Latency regression" pattern's threshold) are `client.monitor.profile.get()`/`.update()`:
+Per-agent monitoring settings (enable/disable, detection categories, notification channels) are `client.monitor.profile.get()`/`.update()`:
 
 ```python
-client.monitor.profile.update("agent_123", threshold_overrides={"latencyMs": 15000})
+client.monitor.profile.update("agent_123", info_detection_enabled=False)
 ```
 
+On self-host, coverage mode, sample rate, retention, and the built-in latency threshold are project-level defaults set in the dashboard's Platform Settings; `update()` still accepts them for wire compatibility, but only the per-agent fields above take effect there.
+
 Self-host also has **online evaluators**: a real LLM judge scoring a sample of live traffic continuously, distinct from a pattern's rule-matching. A score below `alert_threshold` raises a signal the same way a failing pattern does, deduped and triage-ready in `client.monitor.signals`.
 
 ```python
@@ -371,7 +380,7 @@ export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
 export AGENTX_API_KEY=<printed by agentx-trace-eval on first run>
 ```
 
-`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. See that repo's README for what's included, and `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
+`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. Each SDK release pins the engine release it was tested against and converges the install to that pin, so upgrading the SDK upgrades the engine too. Two flags to know: `--update` (consumed by this launcher) force-reinstalls the resolved engine release, while `--upgrade` passes through to `agentx-server` and re-downloads the dashboard bundle before serving. See that repo's README for what's included, and the `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.
 
 ---
 

{agentx_python-0.8.13 → agentx_python-0.8.14}/agentx_python.egg-info/SOURCES.txt

@@ -86,5 +86,6 @@ tests/test_pairwise.py
 tests/test_review_queue.py
 tests/test_runner_features.py
 tests/test_selfhost_analysis_fallback.py
+tests/test_selfhost_compat.py
 tests/test_span_tree.py
 tests/test_testing.py

{agentx_python-0.8.13 → agentx_python-0.8.14}/agentx_python.egg-info/requires.txt

@@ -15,6 +15,7 @@ litellm>=1.0.0
 llama-index-core>=0.10.0
 autogen-agentchat>=0.4.0
 autogen-core>=0.4.0
+mlflow>=3.6.0
 
 [anthropic]
 anthropic>=0.25.0

{agentx_python-0.8.13 → agentx_python-0.8.14}/setup.py

@@ -70,6 +70,7 @@ setup(
             "llama-index-core>=0.10.0",
             "autogen-agentchat>=0.4.0",
             "autogen-core>=0.4.0",
+            "mlflow>=3.6.0",
         ],
     },
     author="Robin Wang and AgentX Team",
@@ -83,5 +84,5 @@ setup(
         "License :: OSI Approved :: Apache Software License",
         "Operating System :: OS Independent",
     ],
-    python_requires=">=3.6",
+    python_requires=">=3.9",
 )

agentx_python-0.8.14/tests/test_selfhost_compat.py

@@ -0,0 +1,424 @@
+"""SDK-to-self-host compatibility matrix.
+
+Every public SDK surface is exercised against a LIVE self-host engine and must land in
+exactly one of two tables:
+
+- SELF_HOST: the surface must answer without raising. Any exception fails the test with
+  the surface's name, so a "fictional" surface (one the engine never grew) can't ship.
+- HOSTED_ONLY: the surface must KEEP failing against the engine AND carry a documented
+  reason (plus a docs-file banner that says so). If the engine grows the surface, the
+  test fails loudly telling us to promote the entry to SELF_HOST.
+
+Opt-in, like the engine's own backend suites: the whole module skips unless both env
+vars below are set.
+
+How to run:
+    1. Boot a scratch engine (any free port, throwaway home dir):
+           cd AgentX-trace-eval/engine
+           PORT=4799 AGENTX_HOME=$(mktemp -d) yarn dev
+    2. Copy the "Default project API key: agtx_local_..." line from its startup log.
+    3. Run the suite:
+           AGENTX_COMPAT_BASE_URL=http://localhost:4799/api/v1 \
+           AGENTX_COMPAT_API_KEY=agtx_local_... \
+           pytest tests/test_selfhost_compat.py -q
+
+The scratch engine usually has no judge/provider keys. That is fine and deliberate:
+judge-dependent steps (eval-run scoring) then record results as skipped/unrated, and this
+suite only asserts that every surface ANSWERS, never that the judge liked the answer.
+"""
+
+from __future__ import annotations
+
+import os
+import time
+import uuid
+from pathlib import Path
+
+import pytest
+
+BASE_URL = os.getenv("AGENTX_COMPAT_BASE_URL")
+API_KEY = os.getenv("AGENTX_COMPAT_API_KEY")
+
+pytestmark = pytest.mark.skipif(
+    not (BASE_URL and API_KEY),
+    reason="self-host compat suite is opt-in: set AGENTX_COMPAT_BASE_URL and AGENTX_COMPAT_API_KEY",
+)
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+
+
+def _tag() -> str:
+    return uuid.uuid4().hex[:8]
+
+
+# ---------------------------------------------------------------------------
+# Shared state: one client, plus lazily created artifacts (a sync trace, one
+# full eval run) reused across parametrized tests so the suite stays fast.
+# ---------------------------------------------------------------------------
+
+
+class CompatContext:
+    def __init__(self) -> None:
+        # Quiet the runner's interactive spinner/banner in test output.
+        os.environ.setdefault("AGENTX_EVAL_QUIET", "1")
+        from agentx import AgentX
+
+        self.client = AgentX(api_key=API_KEY, base_url=BASE_URL)
+        self.session_id = f"compat-{_tag()}"
+        self._trace_id: str | None = None
+        self._eval: dict | None = None
+        self.created_dataset_ids: list[str] = []
+
+    # -- lazy shared artifacts ------------------------------------------------
+
+    def trace_id(self) -> str:
+        """One sync-ingested trace, created on first use (sync=True so the id exists)."""
+        if self._trace_id is None:
+            with self.client.tracer.trace(
+                "compat-check-agent",
+                input={"query": "compat ping"},
+                session_id=self.session_id,
+                sync=True,
+                monitor=False,
+            ) as span:
+                span.output = "compat pong"
+            assert span.trace_id, "tracer.trace(sync=True) exited without a trace_id"
+            self._trace_id = span.trace_id
+        return self._trace_id
+
+    def eval_artifacts(self) -> dict:
+        """One full evaluation lifecycle, run once: dataset publish -> init_run (via
+        client.evaluations.run) -> execute/submit one result -> finalize -> gate.
+        A keyless engine records the result unrated; every surface must still answer."""
+        if self._eval is None:
+            ds = (
+                self.client.evaluations.datasets.builder(
+                    name=f"compat-ds-{_tag()}",
+                    description="scratch dataset for the self-host compat matrix",
+                )
+                .add_case("What is 2 + 2?", expected_results="4")
+                .publish()
+            )
+            self.created_dataset_ids.append(ds.id)
+            run_ctx = self.client.evaluations.run(
+                ds.id, {"kind": "custom_agent", "displayName": "compat-check"}
+            )
+            run_ctx.execute(lambda case: "4")
+            run_ctx.finalize()
+            gate = run_ctx.gate(fail_under=0.0)
+            self._eval = {"dataset_id": ds.id, "run_id": run_ctx.run_id, "gate": gate}
+        return self._eval
+
+    # -- cleanup --------------------------------------------------------------
+
+    def cleanup(self) -> None:
+        for dataset_id in self.created_dataset_ids:
+            try:
+                self.client.evaluations.datasets.delete(dataset_id)
+            except Exception:
+                pass
+
+
+@pytest.fixture(scope="module")
+def compat():
+    ctx = CompatContext()
+    yield ctx
+    ctx.cleanup()
+
+
+# ---------------------------------------------------------------------------
+# SELF_HOST checks - each must answer without raising
+# ---------------------------------------------------------------------------
+
+
+def _check_ping(ctx: CompatContext) -> None:
+    result = ctx.client.ping()
+    assert result.get("ok") is True
+
+
+def _check_tracer_sync_trace_and_flush(ctx: CompatContext) -> None:
+    assert ctx.trace_id()
+    assert ctx.client.tracer.flush(timeout=10) is True
+
+
+def _check_traces_get(ctx: CompatContext) -> None:
+    detail = ctx.client.traces.get(ctx.trace_id())
+    assert isinstance(detail, dict) and detail
+
+
+def _check_traces_list(ctx: CompatContext) -> None:
+    ctx.trace_id()  # make sure at least one trace exists
+    page = ctx.client.traces.list(limit=5)
+    assert isinstance(page.get("traces"), list) and page["traces"]
+
+
+def _check_monitor_kpis(ctx: CompatContext) -> None:
+    assert isinstance(ctx.client.monitor.kpis(), dict)
+
+
+def _check_monitor_metrics(ctx: CompatContext) -> None:
+    assert isinstance(ctx.client.monitor.metrics(window="1h"), dict)
+
+
+def _check_monitor_topics(ctx: CompatContext) -> None:
+    assert isinstance(ctx.client.monitor.topics(), dict)
+
+
+def _check_monitor_list_agents(ctx: CompatContext) -> None:
+    ctx.trace_id()  # tracing auto-creates the agent
+    agents = ctx.client.monitor.list_agents()
+    assert isinstance(agents, list)
+
+
+def _check_monitor_patterns(ctx: CompatContext) -> None:
+    # builder + publish + get + list. The SDK exposes no pattern delete, so the
+    # published pattern stays behind on the scratch engine (throwaway by design).
+    pattern = ctx.client.monitor.patterns.builder(
+        name=f"compat-pattern-{_tag()}",
+        detector_kind="contains",
+        include_terms=["compat-term-that-never-matches"],
+        enabled=False,
+    ).publish()
+    assert pattern.id
+    assert ctx.client.monitor.patterns.get(pattern.id).id == pattern.id
+    assert any(p.id == pattern.id for p in ctx.client.monitor.patterns.list())
+
+
+def _check_monitor_judge_scorers_round_trip(ctx: CompatContext) -> None:
+    scorer = ctx.client.monitor.judge_scorers.builder(
+        name=f"compat-scorer-{_tag()}",
+        acceptance_criteria="The answer is correct.",
+    ).publish()
+    try:
+        assert ctx.client.monitor.judge_scorers.get(scorer.id).id == scorer.id
+        updated = ctx.client.monitor.judge_scorers.update(
+            scorer.id,
+            online={"enabled": False, "sampleRate": 0.1, "alertThreshold": 5},
+        )
+        assert updated.id == scorer.id
+    finally:
+        ctx.client.monitor.judge_scorers.delete(scorer.id)
+
+
+def _check_monitor_scorers_list(ctx: CompatContext) -> None:
+    assert isinstance(ctx.client.monitor.scorers.list(), list)
+
+
+def _check_monitor_review_queue_list(ctx: CompatContext) -> None:
+    assert isinstance(ctx.client.monitor.review_queue.list(status="all"), list)
+
+
+def _check_monitor_rules_list(ctx: CompatContext) -> None:
+    assert isinstance(ctx.client.monitor.rules.list(), list)
+
+
+def _check_monitor_sessions_spans(ctx: CompatContext) -> None:
+    ctx.trace_id()  # ingests one span into ctx.session_id
+    spans = ctx.client.monitor.sessions.spans(ctx.session_id)
+    assert isinstance(spans, list) and spans
+
+
+def _check_evaluations_dataset_round_trip(ctx: CompatContext) -> None:
+    ds = (
+        ctx.client.evaluations.datasets.builder(
+            name=f"compat-ds-roundtrip-{_tag()}",
+            description="round-trip dataset (deleted by this test)",
+        )
+        .add_case("Name a prime number.", expected_results="Any prime, e.g. 7")
+        .publish()
+    )
+    assert ds.id
+    fetched = ctx.client.evaluations.datasets.get(ds.id)
+    assert fetched.id == ds.id and len(fetched.questions) == 1
+    ctx.client.evaluations.datasets.delete(ds.id)
+
+
+def _check_evaluations_run_lifecycle(ctx: CompatContext) -> None:
+    artifacts = ctx.eval_artifacts()
+    assert artifacts["run_id"]
+    # Keyless judge => unrated results => gate answers but may not pass. Both fine.
+    assert isinstance(artifacts["gate"].passed, bool)
+
+
+def _check_evaluations_get_run(ctx: CompatContext) -> None:
+    run = ctx.client.evaluations.get_run(ctx.eval_artifacts()["run_id"])
+    assert isinstance(run, dict) and run
+
+
+def _check_evaluations_list_gates(ctx: CompatContext) -> None:
+    ctx.eval_artifacts()  # records one gate verdict
+    assert isinstance(ctx.client.evaluations.list_gates(), list)
+
+
+def _check_evaluations_prompts_registry(ctx: CompatContext) -> None:
+    name = f"compat-prompt-{_tag()}"
+    created = ctx.client.evaluations.prompts.create(
+        name, "You are a compat-check assistant.", description="compat matrix scratch prompt"
+    )
+    assert created.version >= 1
+    fetched = ctx.client.evaluations.prompts.get(name)
+    assert fetched.name == name and fetched.text
+    assert any(p.name == name for p in ctx.client.evaluations.prompts.list())
+
+
+def _check_feedback_report(ctx: CompatContext) -> None:
+    report = ctx.client.feedback.report(
+        trace_id=ctx.trace_id(), rating="up", end_user_id="compat-suite"
+    )
+    assert isinstance(report, dict)
+
+
+def _check_outcomes_report(ctx: CompatContext) -> None:
+    report = ctx.client.outcomes.report(
+        trace_id=ctx.trace_id(),
+        outcome="confirmed_good",
+        is_negative=False,
+        reported_by="compat-suite",
+    )
+    assert isinstance(report, dict)
+
+
+def _check_export_manifest_and_iter(ctx: CompatContext) -> None:
+    ctx.trace_id()  # at least one exportable row
+    manifest = ctx.client.export.manifest()
+    assert isinstance(manifest, list) and manifest
+    rows = list(ctx.client.export.iter("traces"))
+    assert rows and all(isinstance(r, dict) for r in rows)
+
+
+SELF_HOST = [
+    ("client.ping", _check_ping),
+    ("tracer.trace(sync=True) + tracer.flush", _check_tracer_sync_trace_and_flush),
+    ("traces.get", _check_traces_get),
+    ("traces.list", _check_traces_list),
+    ("monitor.kpis", _check_monitor_kpis),
+    ("monitor.metrics", _check_monitor_metrics),
+    ("monitor.topics", _check_monitor_topics),
+    ("monitor.list_agents", _check_monitor_list_agents),
+    ("monitor.patterns builder/publish/get/list", _check_monitor_patterns),
+    ("monitor.judge_scorers create/get/update/delete", _check_monitor_judge_scorers_round_trip),
+    ("monitor.scorers.list", _check_monitor_scorers_list),
+    ("monitor.review_queue.list", _check_monitor_review_queue_list),
+    ("monitor.rules.list", _check_monitor_rules_list),
+    ("monitor.sessions.spans", _check_monitor_sessions_spans),
+    ("evaluations.datasets builder/publish/get/delete", _check_evaluations_dataset_round_trip),
+    ("evaluations run/execute/finalize/gate", _check_evaluations_run_lifecycle),
+    ("evaluations.get_run", _check_evaluations_get_run),
+    ("evaluations.list_gates", _check_evaluations_list_gates),
+    ("evaluations.prompts create/get/list", _check_evaluations_prompts_registry),
+    ("feedback.report", _check_feedback_report),
+    ("outcomes.report", _check_outcomes_report),
+    ("export.manifest + export.iter", _check_export_manifest_and_iter),
+]
+
+
+@pytest.mark.parametrize(
+    "surface,check", SELF_HOST, ids=[name for name, _ in SELF_HOST]
+)
+def test_self_host_surface(surface, check, compat):
+    try:
+        check(compat)
+    except AssertionError:
+        raise
+    except Exception as exc:
+        pytest.fail(
+            f"self-host surface {surface!r} raised {type(exc).__name__}: {exc} "
+            "(either the SDK or the engine drifted - this surface is supposed to work "
+            "against a self-host engine)"
+        )
+
+
+# ---------------------------------------------------------------------------
+# HOSTED_ONLY checks - each must KEEP failing against the engine, and the docs
+# must say so. If one starts working, promote it to SELF_HOST above.
+# ---------------------------------------------------------------------------
+
+CI_BANNER_NEEDLE = "Hosted platform only."
+CI_ROUTE_NEEDLE = "/ingest/ci-runs"
+
+
+def _call_run_eval(ctx: CompatContext):
+    return ctx.client.tracer.run_eval("evds_compat_missing", lambda q: "answer")
+
+
+def _call_create_ci_run(ctx: CompatContext):
+    return ctx.client.tracer.create_ci_run("evds_compat_missing")
+
+
+def _call_get_ci_run(ctx: CompatContext):
+    return ctx.client.tracer.get_ci_run("cirun_compat_missing")
+
+
+def _call_finalize_ci_run(ctx: CompatContext):
+    return ctx.client.tracer.finalize_ci_run("cirun_compat_missing")
+
+
+def _call_list_models(ctx: CompatContext):
+    return ctx.client.evaluations.list_models()
+
+
+HOSTED_ONLY = [
+    (
+        "tracer.run_eval",
+        _call_run_eval,
+        "Targets the hosted /ingest/ci-runs API; the self-host engine does not serve it. "
+        "Self-host CI gating is evaluations.run(...).execute(...).finalize().gate(...).",
+        "CICD_EVAL.md",
+        (CI_BANNER_NEEDLE, CI_ROUTE_NEEDLE),
+    ),
+    (
+        "tracer.create_ci_run",
+        _call_create_ci_run,
+        "Low-level hosted /ingest/ci-runs call; 404 on self-host, surfaced as DatasetNotFound.",
+        "CICD_EVAL.md",
+        (CI_BANNER_NEEDLE, CI_ROUTE_NEEDLE),
+    ),
+    (
+        "tracer.get_ci_run",
+        _call_get_ci_run,
+        "Low-level hosted /ingest/ci-runs call; 404 on self-host, surfaced as DatasetNotFound.",
+        "CICD_EVAL.md",
+        (CI_BANNER_NEEDLE, CI_ROUTE_NEEDLE),
+    ),
+    (
+        "tracer.finalize_ci_run",
+        _call_finalize_ci_run,
+        "Low-level hosted /ingest/ci-runs call; 404 on self-host, surfaced as DatasetNotFound.",
+        "CICD_EVAL.md",
+        (CI_BANNER_NEEDLE, CI_ROUTE_NEEDLE),
+    ),
+    (
+        "evaluations.list_models",
+        _call_list_models,
+        "Targets the hosted /custom-agent-evaluations/models registry; the engine explicitly "
+        "has not ported it (engine routes/evaluations.ts: 'Still not ported: list_models').",
+        "EVALUATIONS.md",
+        ("`list_models()` is **hosted platform only**",),
+    ),
+]
+
+
+@pytest.mark.parametrize(
+    "surface,call,reason,doc_file,doc_needles",
+    HOSTED_ONLY,
+    ids=[name for name, *_ in HOSTED_ONLY],
+)
+def test_hosted_only_surface(surface, call, reason, doc_file, doc_needles, compat):
+    assert reason and reason.strip(), f"hosted-only entry {surface!r} must document why"
+
+    try:
+        call(compat)
+    except Exception:
+        pass  # expected: the engine does not serve this surface
+    else:
+        pytest.fail(
+            f"hosted-only surface {surface!r} SUCCEEDED against the self-host engine. "
+            "The engine grew this surface: move the entry to SELF_HOST and update the docs."
+        )
+
+    doc_text = (REPO_ROOT / doc_file).read_text(encoding="utf-8")
+    for needle in doc_needles:
+        assert needle in doc_text, (
+            f"hosted-only surface {surface!r}: expected {doc_file} to contain {needle!r} "
+            "so the limitation stays documented"
+        )