agentx-python 0.6.16__tar.gz → 0.6.17__tar.gz

{agentx_python-0.6.16 → agentx_python-0.6.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentx-python
-Version: 0.6.16
+Version: 0.6.17
 Summary: Official Python SDK for AgentX (https://www.agentx.so/)
 Home-page: https://github.com/AgentX-ai/AgentX-python
 Author: Robin Wang and AgentX Team
@@ -189,7 +189,9 @@ client.evaluations.run(
 ).execute(my_agent_fn)
 ```
 
-See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/self-host#prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
+See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
+
+On self-host, a finalized run can also **gate a CI job**: `report.gate(fail_under=7, no_regression=True)` checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
 
 See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
@@ -311,7 +313,7 @@ evaluator = client.monitor.online_evaluators.builder(
 client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
 ```
 
-An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, `client.outcomes.report(...)` records what actually happened after the fact (a reopened ticket, a human confirmation) against a trace, feeding the dashboard's Judge Calibration view, which measures how often AgentX's automated verdicts agree with real outcomes. Both are self-host features.
+An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, two ground-truth streams feed the dashboard's Judge Calibration view (which measures how often AgentX's automated verdicts agree with what actually happened): `client.outcomes.report(...)` for after-the-fact system results (a reopened ticket, a human confirmation), and `client.feedback.report(...)` for end-user votes forwarded from your own app's UI - a "down" raises a "Negative user feedback" signal directly, no sampling or judge call involved. All self-host features.
 
 ```python
 client.monitor.online_evaluators.builder(
@@ -328,6 +330,13 @@ client.outcomes.report(
     is_negative=True,
     reason="Customer reopened the ticket within 3 days",
 )
+
+client.feedback.report(
+    trace_id=trace_id,
+    rating="down",                            # "up" or "down"
+    comment="It never answered my question",  # optional, the user's own words
+    end_user_id=current_user.id,              # optional, opaque to AgentX
+)
 ```
 
 See **[TRACING.md](TRACING.md)** for the complete Monitor guide.

{agentx_python-0.6.16 → agentx_python-0.6.17}/README.md

@@ -127,7 +127,9 @@ client.evaluations.run(
 ).execute(my_agent_fn)
 ```
 
-See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/self-host#prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
+See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
+
+On self-host, a finalized run can also **gate a CI job**: `report.gate(fail_under=7, no_regression=True)` checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
 
 See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
@@ -249,7 +251,7 @@ evaluator = client.monitor.online_evaluators.builder(
 client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
 ```
 
-An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, `client.outcomes.report(...)` records what actually happened after the fact (a reopened ticket, a human confirmation) against a trace, feeding the dashboard's Judge Calibration view, which measures how often AgentX's automated verdicts agree with real outcomes. Both are self-host features.
+An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, two ground-truth streams feed the dashboard's Judge Calibration view (which measures how often AgentX's automated verdicts agree with what actually happened): `client.outcomes.report(...)` for after-the-fact system results (a reopened ticket, a human confirmation), and `client.feedback.report(...)` for end-user votes forwarded from your own app's UI - a "down" raises a "Negative user feedback" signal directly, no sampling or judge call involved. All self-host features.
 
 ```python
 client.monitor.online_evaluators.builder(
@@ -266,6 +268,13 @@ client.outcomes.report(
     is_negative=True,
     reason="Customer reopened the ticket within 3 days",
 )
+
+client.feedback.report(
+    trace_id=trace_id,
+    rating="down",                            # "up" or "down"
+    comment="It never answered my question",  # optional, the user's own words
+    end_user_id=current_user.id,              # optional, opaque to AgentX
+)
 ```
 
 See **[TRACING.md](TRACING.md)** for the complete Monitor guide.

{agentx_python-0.6.16 → agentx_python-0.6.17}/agentx/agentx.py

@@ -60,6 +60,12 @@ class AgentX:
         # ground truth behind the dashboard's Judge Calibration card. Self-host only.
         self.outcomes = OutcomesClient(api_key=self.api_key)
 
+        from agentx.feedback import FeedbackClient
+
+        # Forward end-user votes ("up"/"down") on traced responses - a "down" raises a signal
+        # directly, and every vote feeds Judge Calibration alongside outcomes. Self-host only.
+        self.feedback = FeedbackClient(api_key=self.api_key)
+
         _ingest_client = IngestClient(
             api_key=self.api_key,
             sdk_version=VERSION,

{agentx_python-0.6.16 → agentx_python-0.6.17}/agentx/evaluations/client.py

@@ -265,6 +265,34 @@ class EvaluationsClient:
             "POST", f"/runs/{run_id}/finalize", json={"status": "completed"}
         )
 
+    def gate_run(
+        self,
+        run_id: str,
+        *,
+        fail_under: Optional[float] = None,
+        no_regression: bool = False,
+        tolerance: Optional[float] = None,
+        record: bool = True,
+        caller: Optional[str] = "sdk",
+    ) -> Dict[str, Any]:
+        # CI gate (self-host): pass/fail a finalized run against an absolute rating floor and/or
+        # the dataset's previous completed run. Recorded into gate history by default (the
+        # dashboard's CI page lists these); pass record=False for a preview that leaves no trace.
+        # `caller` is a free label shown in that history ("sdk", "github-actions", ...). See
+        # EvaluationRunContext.gate() for the CI-facing wrapper with printed verdicts.
+        params: Dict[str, Any] = {}
+        if fail_under is not None:
+            params["failUnder"] = fail_under
+        if no_regression:
+            params["noRegression"] = "true"
+        if tolerance is not None:
+            params["tolerance"] = tolerance
+        if record:
+            params["record"] = "true"
+            if caller:
+                params["caller"] = caller
+        return self._request("GET", f"/runs/{run_id}/gate", params=params)
+
     def analyze_run(
         self,
         run_id: str,

{agentx_python-0.6.16 → agentx_python-0.6.17}/agentx/evaluations/runner.py

@@ -55,6 +55,23 @@ _ANALYSIS_LEVEL_LABELS = {
 _DEFAULT_JUDGE_MODEL = "gpt-5.5"
 
 
+class GateResult:
+    """Wire result of the CI gate (GET /runs/:id/gate) with attribute access for the fields a
+    CI script actually branches on."""
+
+    def __init__(self, data: Dict[str, Any]):
+        self.raw = data
+        self.passed: bool = bool(data.get("passed"))
+        self.average_rating: Optional[float] = data.get("averageRating")
+        self.baseline_average: Optional[float] = data.get("baselineAverage")
+        self.baseline_run_id: Optional[str] = data.get("baselineRunId")
+        self.checks: List[Dict[str, Any]] = data.get("checks", [])
+
+    @property
+    def exit_code(self) -> int:
+        return 0 if self.passed else 1
+
+
 class EvaluationRunContext:
     """
     Fluent builder returned by client.evaluations.run(...).
@@ -210,6 +227,42 @@ class EvaluationRunContext:
                 logger.error("Finalize failed: %s", exc)
         return self
 
+    def gate(
+        self,
+        *,
+        fail_under: Optional[float] = None,
+        no_regression: bool = False,
+        tolerance: Optional[float] = None,
+        caller: str = "sdk",
+    ) -> "GateResult":
+        """CI gate (self-host): pass/fail this finalized run so a CI job can block a merge.
+
+        ``fail_under`` fails the gate when the run's average rating is below the floor;
+        ``no_regression=True`` fails it when the average dropped more than ``tolerance``
+        (default 0.5, judge scores are noisy) below the dataset's previous completed run.
+        At least one check is required. Prints a CI-log-friendly verdict and returns a
+        :class:`GateResult` - the caller decides the exit code::
+
+            report = client.evaluations.run(...).execute(my_agent).finalize()
+            gate = report.gate(fail_under=7, no_regression=True)
+            if not gate.passed:
+                sys.exit(1)
+        """
+        data = self._client.gate_run(
+            self._run.run_id,
+            fail_under=fail_under,
+            no_regression=no_regression,
+            tolerance=tolerance,
+            caller=caller,
+        )
+        result = GateResult(data)
+        print()
+        for check in result.checks:
+            mark = green("✓") if check.get("passed") else red("✗")
+            print(f"  {mark}  [{check.get('check')}] {check.get('detail')}")
+        print(f"  {green('✓  GATE PASSED') if result.passed else red('✗  GATE FAILED')}")
+        return result
+
     # ------------------------------------------------------------------
     # Live rating stats - server-computed (Evaluate.liveStatistics), refreshed
     # from the response of each append_results()/finalize_run() call. Available
@@ -350,6 +403,32 @@ class EvaluationsRunner:
         script execution)."""
         return self._client.get_analysis_status(run_id)
 
+    def gate_run(
+        self,
+        run_id: str,
+        *,
+        fail_under: Optional[float] = None,
+        no_regression: bool = False,
+        tolerance: Optional[float] = None,
+        record: bool = True,
+        caller: Optional[str] = "sdk",
+    ) -> GateResult:
+        """CI-gate any finalized run by id - the standalone form of
+        ``EvaluationRunContext.gate()``, for gating a run created elsewhere or
+        re-checking one without re-running it (self-host only). Pass
+        ``record=False`` for a check that stays out of the dashboard's CI Gates
+        history."""
+        return GateResult(
+            self._client.gate_run(
+                run_id,
+                fail_under=fail_under,
+                no_regression=no_regression,
+                tolerance=tolerance,
+                record=record,
+                caller=caller,
+            )
+        )
+
     def run(
         self,
         dataset_id: str,

agentx_python-0.6.17/agentx/feedback.py

@@ -0,0 +1,76 @@
+from __future__ import annotations
+
+import logging
+from typing import Any, Dict, Optional
+
+import requests
+
+from agentx.util import api_base, get_headers
+
+logger = logging.getLogger(__name__)
+
+
+class AgentXFeedbackError(Exception):
+    pass
+
+
+class FeedbackClient:
+    """Surfaced as ``client.feedback``: forward an END USER's reaction to a traced response
+    (self-host only, ``POST /feedback``) - the vote button in your own app's UI, relayed to
+    AgentX. The cheapest ground truth there is: a "down" raises a "Negative user feedback"
+    signal directly (the user is the detector, no sampling or judge call), and every report
+    also feeds Judge Calibration, so AgentX's automated verdicts get measured against real
+    human reactions.
+
+    Distinct from ``client.outcomes``: an outcome is an after-the-fact SYSTEM result ("ticket
+    reopened", reported by a workflow); feedback is a human vote with up/down semantics.
+    """
+
+    def __init__(self, api_key: Optional[str] = None):
+        self._api_key = api_key
+
+    def report(
+        self,
+        trace_id: str,
+        rating: str,
+        *,
+        comment: Optional[str] = None,
+        end_user_id: Optional[str] = None,
+    ) -> Dict[str, Any]:
+        """Report one end-user vote on a trace.
+
+        ``rating`` is ``"up"`` or ``"down"``. ``comment`` is the user's own words, if your UI
+        collects them (a "down" comment becomes the signal's summary and improvement evidence).
+        ``end_user_id`` is your app's identifier for the voter, kept opaque by AgentX.
+
+        Example, from your app's vote handler::
+
+            client.feedback.report(
+                trace_id=trace_id,
+                rating="down",
+                comment="It never answered my question",
+                end_user_id=current_user.id,
+            )
+        """
+        if rating not in ("up", "down"):
+            raise AgentXFeedbackError('rating must be "up" or "down"')
+        payload: Dict[str, Any] = {"traceId": trace_id, "rating": rating}
+        if comment:
+            payload["comment"] = comment
+        if end_user_id:
+            payload["endUserId"] = end_user_id
+
+        resp = requests.post(
+            f"{api_base()}/feedback",
+            headers={**get_headers(self._api_key), "Content-Type": "application/json"},
+            json=payload,
+            timeout=10,
+        )
+        if resp.status_code >= 400:
+            try:
+                detail = resp.json().get("error", resp.reason)
+            except ValueError:
+                detail = resp.reason
+            raise AgentXFeedbackError(f"Failed to report feedback ({resp.status_code}): {detail}")
+        logger.info("Reported %s feedback on trace %s", rating, trace_id)
+        return resp.json().get("feedback", {})

agentx_python-0.6.17/agentx/version.py

@@ -0,0 +1 @@
+VERSION = "0.6.17"

{agentx_python-0.6.16 → agentx_python-0.6.17}/agentx_python.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentx-python
-Version: 0.6.16
+Version: 0.6.17
 Summary: Official Python SDK for AgentX (https://www.agentx.so/)
 Home-page: https://github.com/AgentX-ai/AgentX-python
 Author: Robin Wang and AgentX Team
@@ -189,7 +189,9 @@ client.evaluations.run(
 ).execute(my_agent_fn)
 ```
 
-See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/self-host#prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
+See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).
+
+On self-host, a finalized run can also **gate a CI job**: `report.gate(fail_under=7, no_regression=True)` checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns an exit code - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.
 
 See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.
 
@@ -311,7 +313,7 @@ evaluator = client.monitor.online_evaluators.builder(
 client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
 ```
 
-An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, `client.outcomes.report(...)` records what actually happened after the fact (a reopened ticket, a human confirmation) against a trace, feeding the dashboard's Judge Calibration view, which measures how often AgentX's automated verdicts agree with real outcomes. Both are self-host features.
+An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, two ground-truth streams feed the dashboard's Judge Calibration view (which measures how often AgentX's automated verdicts agree with what actually happened): `client.outcomes.report(...)` for after-the-fact system results (a reopened ticket, a human confirmation), and `client.feedback.report(...)` for end-user votes forwarded from your own app's UI - a "down" raises a "Negative user feedback" signal directly, no sampling or judge call involved. All self-host features.
 
 ```python
 client.monitor.online_evaluators.builder(
@@ -328,6 +330,13 @@ client.outcomes.report(
     is_negative=True,
     reason="Customer reopened the ticket within 3 days",
 )
+
+client.feedback.report(
+    trace_id=trace_id,
+    rating="down",                            # "up" or "down"
+    comment="It never answered my question",  # optional, the user's own words
+    end_user_id=current_user.id,              # optional, opaque to AgentX
+)
 ```
 
 See **[TRACING.md](TRACING.md)** for the complete Monitor guide.

{agentx_python-0.6.16 → agentx_python-0.6.17}/agentx_python.egg-info/SOURCES.txt

@@ -5,6 +5,7 @@ agentx/__init__.py
 agentx/agentx.py
 agentx/cli.py
 agentx/exceptions.py
+agentx/feedback.py
 agentx/outcomes.py
 agentx/py.typed
 agentx/util.py

agentx_python-0.6.16/agentx/version.py

@@ -1 +0,0 @@
-VERSION = "0.6.16"