agentdynamics 0.6.0__tar.gz → 0.7.0__tar.gz

{agentdynamics-0.6.0 → agentdynamics-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentdynamics
-Version: 0.6.0
+Version: 0.7.0
 Summary: Application performance monitoring for AI agents: flows, cost, reliability and process quality
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/Aditya31398/agentdynamics
@@ -14,6 +14,8 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Provides-Extra: toml
 Requires-Dist: tomli; python_version < "3.11" and extra == "toml"
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.1; extra == "postgres"
 Provides-Extra: test
 Requires-Dist: langchain-core==0.2.33; extra == "test"
 Requires-Dist: langsmith==0.1.99; extra == "test"
@@ -146,7 +148,7 @@ AGENTDYNAMICS_URL=http://127.0.0.1:8790 python examples/governed_agent.py 40
 | **Workflows** | The real execution graph mined from your traces: path variants and their success rates, loops, the critical node, handoffs |
 | **Task snapshots** | Span-tree waterfall for each request: context growth, cost against the baseline, what went wrong, what the user said next |
 | **Baselines & Apdex** | "Normal" cost and latency per workflow, learned automatically, plus an agent Apdex score |
-| **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack or webhooks |
+| **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack, PagerDuty or webhooks, routed by project, rule and severity |
 | **Tools & Models** | Error rate and p95 per tool; per model TTFT, tokens/s, truncation rate, cache hit rate and spend |
 | **SLOs** | Success rate, Apdex, latency and cost objectives with error budgets and burn rates |
 | **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export, and what a policy change would refuse |
@@ -203,12 +205,21 @@ Telemetry is sent from a background thread with a bounded queue. Your agent neve
 agentdynamics keys create --role ingest     # for apps / collectors
 agentdynamics keys create --role read       # for people and Grafana
 agentdynamics keys create --role admin      # for rules, SLOs and config
+agentdynamics keys create --role read --project checkout    # one team: sees and sends only its projects
 docker compose -f deploy/docker-compose.yml up -d   # AgentDynamics + OpenTelemetry Collector gateway
 ```
 
-- **Security:** role-based API keys.
+The store is a SQLite file by default. For several instances behind a load balancer, or a database you back up
+and operate like any other, use Postgres: `pip install "agentdynamics[postgres]"` and set
+`AGENTDYNAMICS_DB_URL=postgresql://...` (or `deploy/docker-compose.postgres.yml`). One instance is elected to
+write the analysis; the others serve the console and take ingest, and one takes over if the writer goes away.
+
+- **Security:** role-based API keys, optionally scoped to projects. A scoped key reads only its projects' tasks,
+  runs, steps and events, on every endpoint, and can't write into or over another project's traces.
 - **Privacy:** redaction of emails, API keys and card numbers, or `store_content = false` to keep metadata only.
-- **Operations:** retention, Prometheus `/metrics`, `/healthz`, Slack and webhook alerts, and a status page for every source.
+- **Operations:** retention, Prometheus `/metrics`, `/healthz`, a status page for every source, and alerts to Slack,
+  PagerDuty or any webhook: health-rule events and SLO burn-rate pages, delivered in order with retries
+  (`agentdynamics alerts test` checks a destination end to end).
 
 See [deploy/](deploy/) and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 

{agentdynamics-0.6.0 → agentdynamics-0.7.0}/README.md

@@ -123,7 +123,7 @@ AGENTDYNAMICS_URL=http://127.0.0.1:8790 python examples/governed_agent.py 40
 | **Workflows** | The real execution graph mined from your traces: path variants and their success rates, loops, the critical node, handoffs |
 | **Task snapshots** | Span-tree waterfall for each request: context growth, cost against the baseline, what went wrong, what the user said next |
 | **Baselines & Apdex** | "Normal" cost and latency per workflow, learned automatically, plus an agent Apdex score |
-| **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack or webhooks |
+| **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack, PagerDuty or webhooks, routed by project, rule and severity |
 | **Tools & Models** | Error rate and p95 per tool; per model TTFT, tokens/s, truncation rate, cache hit rate and spend |
 | **SLOs** | Success rate, Apdex, latency and cost objectives with error budgets and burn rates |
 | **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export, and what a policy change would refuse |
@@ -180,12 +180,21 @@ Telemetry is sent from a background thread with a bounded queue. Your agent neve
 agentdynamics keys create --role ingest     # for apps / collectors
 agentdynamics keys create --role read       # for people and Grafana
 agentdynamics keys create --role admin      # for rules, SLOs and config
+agentdynamics keys create --role read --project checkout    # one team: sees and sends only its projects
 docker compose -f deploy/docker-compose.yml up -d   # AgentDynamics + OpenTelemetry Collector gateway
 ```
 
-- **Security:** role-based API keys.
+The store is a SQLite file by default. For several instances behind a load balancer, or a database you back up
+and operate like any other, use Postgres: `pip install "agentdynamics[postgres]"` and set
+`AGENTDYNAMICS_DB_URL=postgresql://...` (or `deploy/docker-compose.postgres.yml`). One instance is elected to
+write the analysis; the others serve the console and take ingest, and one takes over if the writer goes away.
+
+- **Security:** role-based API keys, optionally scoped to projects. A scoped key reads only its projects' tasks,
+  runs, steps and events, on every endpoint, and can't write into or over another project's traces.
 - **Privacy:** redaction of emails, API keys and card numbers, or `store_content = false` to keep metadata only.
-- **Operations:** retention, Prometheus `/metrics`, `/healthz`, Slack and webhook alerts, and a status page for every source.
+- **Operations:** retention, Prometheus `/metrics`, `/healthz`, a status page for every source, and alerts to Slack,
+  PagerDuty or any webhook: health-rule events and SLO burn-rate pages, delivered in order with retries
+  (`agentdynamics alerts test` checks a destination end to end).
 
 See [deploy/](deploy/) and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 

{agentdynamics-0.6.0 → agentdynamics-0.7.0}/agentdynamics/__init__.py

@@ -5,7 +5,7 @@
 
 See agentdynamics.autotrace for @trace / @tool / span.
 """
-__version__ = "0.6.0"
+__version__ = "0.7.0"
 
 from .autotrace import flush, init, llm_call, outcome, record_llm, span, tool, trace  # noqa: E402,F401
 

{agentdynamics-0.6.0 → agentdynamics-0.7.0}/agentdynamics/__main__.py

@@ -142,22 +142,76 @@ def cmd_doctor(a):
     return 0 if ok else 1
 
 
+def cmd_alerts(a, data):
+    from . import alerts as alertmod, config as cfgmod, store
+    cfg = cfgmod.load(data)
+    dests, problems = alertmod.destinations(cfg["alerts"])
+    for p in problems:
+        print(f"ignored: {p}")
+    if not dests:
+        print("No alert destinations configured: add [[alerts.webhooks]] to agentdynamics.toml "
+              "(see `python -c \"import agentdynamics.alerts; help(agentdynamics.alerts)\"`).")
+        return 1
+    if a.action == "status":
+        con = store.connect(*store.target(cfg, data))
+        depth = store.outbox_depth(con)
+        oldest = {r["dest"]: dict(r) for r in con.execute(
+            "SELECT dest, created, attempts, last_error FROM alert_outbox WHERE id IN "
+            "(SELECT MIN(id) FROM alert_outbox GROUP BY dest)")}
+        for d in dests:
+            print(f"{d['id']:<28} {d['format']:<10} {'+'.join(d['kinds']):<12} min {d['min_severity']:<8} "
+                  f"queued {depth.get(d['id'], 0)}")
+            o = oldest.get(d["id"])
+            if o and o["attempts"]:
+                print(f"    retrying since {time.strftime('%Y-%m-%d %H:%M', time.localtime(o['created']))} "
+                      f"({o['attempts']} attempts): {o['last_error']}")
+        firing = store.alert_state(con)
+        print(f"\nSLO alerts firing: {len(firing) or 'none'}")
+        for k, v in sorted(firing.items(), key=lambda kv: kv[1]["since"]):
+            print(f"    {k}  since {time.strftime('%Y-%m-%d %H:%M', time.localtime(v['since']))}  {v.get('severity')}")
+        return 0
+    failed = 0
+    chosen = [d for d in dests if not a.to or d["id"] == a.to]
+    if not chosen:
+        print(f"no destination named {a.to!r}; names: {', '.join(d['id'] for d in dests)}")
+        return 2
+    for d in chosen:
+        alert = alertmod.test_alert()
+        bodies = alertmod.render(d, [alert], cfg["alerts"].get("console_url") or "")
+        if d["format"] == "pagerduty":      # resolve it straight away, so a test leaves no open incident
+            bodies += alertmod.render(d, [dict(alert, action="resolve")])
+        results = [alertmod.send(d, b) for b in bodies]
+        ok = all(r[0] for r in results)
+        failed += not ok
+        print(f"{d['id']:<28} {'ok' if ok else 'FAILED'}  {'; '.join(r[2] for r in results)}")
+    return 1 if failed else 0
+
+
 def cmd_keys(a, data):
     from .config import keys_path, load_keys, save_keys
     keys = load_keys(data)
     if a.action == "create":
+        if a.project and a.role == "admin":
+            print("admin keys cover the whole install and cannot be scoped to projects; "
+                  "use --role read or --role ingest", file=sys.stderr)
+            return 2
         k = {"name": a.name or f"{a.role}-{len(keys) + 1}", "role": a.role, "key": f"ad_{a.role[0]}_{secrets.token_urlsafe(24)}",
              "created": int(time.time())}
+        if a.project:
+            k["projects"] = sorted(set(a.project))
         keys.append(k)
         save_keys(data, keys)
-        print(f"Created {k['role']} key '{k['name']}':\n\n  {k['key']}\n\nAuth is now ON for the server using {keys_path(data)} (restart it).")
+        scope = f" for project(s) {', '.join(k['projects'])}" if a.project else ""
+        print(f"Created {k['role']} key '{k['name']}'{scope}:\n\n  {k['key']}\n\nAuth is now ON for the server using {keys_path(data)} (restart it).")
         if a.role == "ingest":
             print(f"Use it in your app:  export AGENTDYNAMICS_API_KEY={k['key']}")
     elif a.action == "list":
         if not keys:
             print("No keys: auth is off (local mode).")
         for k in keys:
-            print(f"  {k['name']:<20} {k['role']:<7} {k['key'][:8]}...  created {time.strftime('%Y-%m-%d', time.localtime(k.get('created', 0)))}")
+            scope = ", ".join(k["projects"]) or "no projects" if isinstance(k.get("projects"), list) else "all projects"
+            print(f"  {k['name']:<20} {k['role']:<7} {k['key'][:8]}...  created "
+                  f"{time.strftime('%Y-%m-%d', time.localtime(k.get('created', 0)))}  {scope}")
     elif a.action == "revoke":
         left = [k for k in keys if k["name"] != a.name and not k["key"].startswith(a.name or "\0")]
         save_keys(data, left)
@@ -297,6 +351,8 @@ def main(argv=None):
     k.add_argument("action", choices=["create", "list", "revoke"])
     k.add_argument("--role", choices=["ingest", "read", "admin"], default="ingest")
     k.add_argument("--name", default=None)
+    k.add_argument("--project", action="append", default=None, metavar="NAME",
+                   help="scope a read or ingest key to this project (repeat for several); omit for all projects")
     po = sub.add_parser("policy", help="Aegis policy from observed behaviour (observe -> govern)")
     po.add_argument("action", choices=["export", "report", "check"],
                     help="export: generate a tightened policy; report: governance summary; "
@@ -313,6 +369,10 @@ def main(argv=None):
     po.add_argument("--max-denied-fraction", type=float, default=0.0,
                     help="fail if the policy would refuse more than this share of observed, allowed calls "
                          "(export, check; default 0: any refusal fails)")
+    al = sub.add_parser("alerts", help="alert destinations: what is queued and firing, or send a test alert")
+    al.add_argument("action", choices=["status", "test"],
+                    help="status: destinations, queue and firing SLO alerts; test: send a test alert to each")
+    al.add_argument("--to", metavar="NAME", help="test only this destination (its name, as status shows it)")
     sub.add_parser("ingest", help="scan sources once and rebuild the database")
     rp = sub.add_parser("report", help="print a text summary")
     rp.add_argument("--project")
@@ -335,6 +395,8 @@ def main(argv=None):
     os.makedirs(a.data, exist_ok=True)
     if cmd == "keys":
         return cmd_keys(a, a.data)
+    if cmd == "alerts":
+        return cmd_alerts(a, a.data)
 
     from .engine import Engine
     eng = Engine(a.data, a.claude_root or None)
@@ -347,7 +409,9 @@ def main(argv=None):
         eng.refresh()
         return 0
     eng.refresh(force=True)
-    print(f"indexed in {eng.last_duration}s -> {os.path.join(a.data, 'agentdynamics.db')}", file=sys.stderr)
+    from .pg import redact_url
+    where = f"{redact_url(eng.db_path)} (schema {eng.db_schema})" if eng.db_schema else eng.db_path
+    print(f"indexed in {eng.last_duration}s -> {where}", file=sys.stderr)
     if cmd == "ingest":
         return 0
     if cmd == "policy":

agentdynamics-0.7.0/agentdynamics/alerts.py

@@ -0,0 +1,251 @@
+"""Alert routing: health-rule events and SLO burn rates, to Slack, PagerDuty and JSON webhooks.
+
+Destinations are `[[alerts.webhooks]]` in agentdynamics.toml:
+
+    [alerts]
+    console_url = "https://agentdynamics.internal"   # optional: alerts link to the task / SLO page
+
+    [[alerts.webhooks]]
+    name = "sre-pager"                   # optional; names it in `agentdynamics alerts status/test`
+    format = "pagerduty"                 # json (default) | slack | pagerduty
+    routing_key_env = "PD_ROUTING_KEY"   # PagerDuty Events API v2 integration key (or routing_key = "...")
+    min_severity = "critical"            # info | warning (default) | critical
+    kinds = ["events", "slos"]           # default ["events"]; "slos" adds SLO burn-rate alerts
+    projects = ["checkout"]              # optional: only these projects
+    rules = ["run_failed"]               # optional: only these health rules
+
+What is sent:
+- A health-rule event is a point in time. Slack and JSON get each one (JSON in the original
+  `{"source": "agentdynamics", "events": [...]}` shape, a contract). PagerDuty gets one alert per rule,
+  project and task type (the dedup key), so a burst of runaway-cost tasks is one incident, not fifty.
+- An SLO alert has a start and an end: a trigger when a burn-rate policy starts firing and a resolve when
+  it stops (slo.alert_conditions). What is firing is kept in the durable `alert_state` table, so a restart
+  neither repeats a trigger nor forgets to resolve.
+
+Delivery is at least once, in order per destination, through the durable `alert_outbox` table: a failed
+send is retried with backoff (429, 5xx, network errors) for up to a day; any other 4xx is a configuration
+error and is dropped and reported. No secret is stored: bodies are queued without the PagerDuty routing key,
+and destinations are identified by name or a hash, never by URL.
+"""
+import hashlib
+import json
+import os
+import time
+import urllib.error
+import urllib.request
+from collections import OrderedDict
+from urllib.parse import quote
+
+FORMATS = ("json", "slack", "pagerduty")
+KINDS = ("events", "slos")
+SEV = {"info": 1, "warning": 2, "critical": 3}
+# https://raw.githubusercontent.com/PagerDuty/api-schema/main/reference/events-v2/openapiv3.json
+PAGERDUTY_URL = "https://events.pagerduty.com/v2/enqueue"
+PAGERDUTY_SEVERITY = {"critical": "critical", "warning": "warning", "info": "info"}
+SLACK_LINES = 10          # a Slack message lists this many alerts, then "and N more"
+GIVE_UP_S = 86400         # a message still failing after a day is dropped
+MAX_BACKOFF_S = 3600
+
+
+def backoff(attempts):
+    return min(MAX_BACKOFF_S, 15 * 2 ** attempts)
+
+
+# ---------------------------------------------------------------- destinations
+
+def destinations(alerts_cfg, env=None):
+    """([destination], [problem]) from the [alerts] config. A misconfigured destination is reported, not
+    raised: one bad entry must not stop the others, or the server."""
+    env = os.environ if env is None else env
+    out, problems, seen = [], [], set()
+    for i, h in enumerate((alerts_cfg or {}).get("webhooks") or []):
+        h = dict(h)
+        label = h.get("name") or f"webhook #{i + 1}"
+        fmt = h.get("format") or "json"
+        if fmt not in FORMATS:
+            problems.append(f"{label}: unknown format {fmt!r}; use one of {', '.join(FORMATS)}")
+            continue
+        kinds = h.get("kinds") or ["events"]
+        bad = [k for k in kinds if k not in KINDS]
+        if bad:
+            problems.append(f"{label}: unknown kinds {bad}; use {list(KINDS)}")
+            continue
+        if h.get("min_severity", "warning") not in SEV:
+            problems.append(f"{label}: min_severity must be one of {', '.join(SEV)}")
+            continue
+        secret = ""
+        if fmt == "pagerduty":
+            h.setdefault("url", PAGERDUTY_URL)
+            secret = h.get("routing_key") or env.get(h.get("routing_key_env") or "", "")
+            if not secret:
+                problems.append(f"{label}: a pagerduty destination needs routing_key or routing_key_env"
+                                + (f" ({h['routing_key_env']} is not set)" if h.get("routing_key_env") else ""))
+                continue
+        if not h.get("url"):
+            problems.append(f"{label}: no url")
+            continue
+        did = h.get("name") or f"{fmt}-{hashlib.sha256((h['url'] + secret).encode()).hexdigest()[:10]}"
+        if did in seen:
+            problems.append(f"{label}: duplicate destination name {did!r}")
+            continue
+        seen.add(did)
+        out.append({"id": did, "format": fmt, "url": h["url"], "secret": secret, "kinds": list(kinds),
+                    "min_severity": h.get("min_severity", "warning"),
+                    "projects": h.get("projects"), "rules": h.get("rules")})
+    return out, problems
+
+
+def routes(dest, alert):
+    """Whether this destination takes this alert."""
+    if alert["kind"] + "s" not in dest["kinds"]:
+        return False
+    if SEV.get(alert["severity"], 1) < SEV[dest["min_severity"]]:
+        return False
+    if dest["projects"] is not None and alert.get("project") not in dest["projects"]:
+        return False            # an install-wide SLO has no project, so it goes only to unfiltered destinations
+    if dest["rules"] is not None and alert["kind"] == "event" and alert.get("rule_id") not in dest["rules"]:
+        return False
+    return True
+
+
+# ---------------------------------------------------------------- alerts
+
+def from_event(e):
+    return {"kind": "event", "action": "trigger", "key": f"event/{e['rule_id']}/{e['project']}/{e['task_type']}",
+            "severity": e["severity"], "summary": f"{e['rule']}: {e['message']}", "project": e["project"],
+            "rule_id": e["rule_id"], "task_type": e["task_type"], "ts": e.get("ts"), "event": e}
+
+
+def slo_summary(c, action="trigger"):
+    scope = " ".join(f"{k}={v}" for k, v in (c.get("scope") or {}).items() if v)
+    where = f" [{scope}]" if scope else ""
+    if action == "resolve":
+        what = "is back within its objective" if c["alert"] == "breach" else \
+            f"is no longer burning its error budget fast enough to {c['alert']}"
+        return f"Resolved: SLO '{c['name']}'{where} {what}"
+    if c["policy"] == "breach":
+        return f"SLO '{c['name']}'{where} breached: {c['value']:.4g} against a target of {c['op']} {c['target']}"
+    return (f"SLO '{c['name']}'{where} burning its error budget at {c['burn_rate']}x "
+            f"(threshold {c['threshold']}x: {c['budget']:.0%} of the budget in {c['long_h']}h)")
+
+
+def from_slo(key, c, action, now):
+    return {"kind": "slo", "action": action, "key": key, "severity": c["severity"],
+            "summary": slo_summary(c, action),
+            "project": (c.get("scope") or {}).get("project") or None, "slo": c["slo"], "ts": now, "condition": c}
+
+
+# ---------------------------------------------------------------- formats
+
+def _dedup_key(key):
+    key = "agentdynamics/" + key
+    return key if len(key) <= 255 else key[:200] + "#" + hashlib.sha256(key.encode()).hexdigest()[:32]
+
+
+def link(console_url, alert):
+    if not console_url:
+        return None
+    base = console_url.rstrip("/")
+    if alert["kind"] == "event":
+        return f"{base}/#/task/{quote(alert['event']['task_id'], safe='')}"
+    return f"{base}/#/slos"
+
+
+def _slack_escape(s):
+    # Slack mrkdwn: these three are control characters in message text
+    return str(s).replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
+
+
+def _iso(ts):
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(ts or time.time()))
+
+
+def render(dest, alerts, console_url=""):
+    """The request bodies that deliver `alerts` to `dest`, in order. Secrets are added at send time."""
+    fmt = dest["format"]
+    if fmt == "json":
+        bodies = []
+        events = [a["event"] for a in alerts if a["kind"] == "event"]
+        if events:
+            bodies.append({"source": "agentdynamics", "events": events})
+        slos = [{"action": a["action"], "key": a["key"], "severity": a["severity"], "summary": a["summary"],
+                 "link": link(console_url, a), **a["condition"]} for a in alerts if a["kind"] == "slo"]
+        if slos:
+            bodies.append({"source": "agentdynamics", "slo_alerts": slos})
+        return bodies
+    # one line (Slack) or one alert (PagerDuty) per dedup key, so a burst of the same violation reads as one
+    groups = OrderedDict()
+    for a in alerts:
+        groups.setdefault(a["key"], []).append(a)
+    if fmt == "slack":
+        lines = []
+        for g in list(groups.values())[:SLACK_LINES]:
+            a = g[-1]
+            head = "RESOLVED" if a["action"] == "resolve" else a["severity"].upper()
+            where = " / ".join(_slack_escape(x) for x in (a.get("project"), a.get("task_type")) if x)
+            url = link(console_url, a)
+            ref = f" · <{url}|{'latest' if len(g) > 1 else 'open'}>" if url else ""
+            if a["kind"] == "event":
+                e = a["event"]
+                times = f" ×{len(g)}" if len(g) > 1 else ""
+                lines.append(f"*{head}* · {_slack_escape(e['rule'])}{times} · {where}{ref}\n{_slack_escape(e['message'])}")
+            else:
+                lines.append(f"*{head}* · {_slack_escape(a['summary'])}{ref}")
+        if len(groups) > SLACK_LINES:
+            lines.append(f"…and {len(groups) - SLACK_LINES} more")
+        return [{"text": "AgentDynamics alerts\n" + "\n\n".join(lines)}] if lines else []
+    bodies = []
+    for key, g in groups.items():
+        a = g[-1]
+        if a["action"] == "resolve":
+            bodies.append({"event_action": "resolve", "dedup_key": _dedup_key(key)})
+            continue
+        if a["kind"] == "event":
+            e = a["event"]
+            details = {"rule": e["rule"], "rule_id": e["rule_id"], "message": e["message"], "value": e.get("value"),
+                       "task_id": e["task_id"], "run_id": e.get("run_id"), "project": e["project"],
+                       "task_type": e["task_type"], "occurrences": len(g)}
+            group, klass = e["task_type"], e["rule_id"]
+        else:
+            details, group, klass = dict(a["condition"]), a["slo"], "slo_" + a["condition"]["alert"]
+        body = {"event_action": "trigger", "dedup_key": _dedup_key(key),
+                "payload": {"summary": a["summary"][:1024], "source": "agentdynamics",
+                            "severity": PAGERDUTY_SEVERITY.get(a["severity"], "warning"),
+                            "timestamp": _iso(a.get("ts")), "component": a.get("project") or "all projects",
+                            "group": group, "class": klass, "custom_details": details},
+                "client": "AgentDynamics"}
+        url = link(console_url, a)
+        if url:
+            body["client_url"] = console_url
+            body["links"] = [{"href": url, "text": "Open in AgentDynamics"}]
+        bodies.append(body)
+    return bodies
+
+
+# ---------------------------------------------------------------- sending
+
+def send(dest, body, timeout=10):
+    """POST one body. Returns (ok, retryable, detail)."""
+    if dest["format"] == "pagerduty":
+        body = dict(body, routing_key=dest["secret"])
+    req = urllib.request.Request(dest["url"], data=json.dumps(body, default=str).encode(), method="POST",
+                                 headers={"Content-Type": "application/json", "User-Agent": "agentdynamics"})
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as r:
+            return True, False, f"HTTP {r.status}"
+    except urllib.error.HTTPError as ex:
+        text = ex.read(300).decode("utf-8", "replace").strip()
+        return False, ex.code == 429 or ex.code >= 500, f"HTTP {ex.code}: {text}"[:300]
+    except (urllib.error.URLError, OSError) as ex:
+        return False, True, f"{type(ex).__name__}: {getattr(ex, 'reason', ex)}"[:300]
+
+
+def test_alert(now=None):
+    """A harmless alert for `agentdynamics alerts test`: a trigger, and for PagerDuty the resolve after it."""
+    now = now or time.time()
+    c = {"slo": "agentdynamics-test", "name": "AgentDynamics test alert", "alert": "breach", "policy": "breach",
+         "severity": "info",
+         "metric": "success_rate", "op": ">=", "target": 1, "value": 0, "window_days": 1, "scope": {}, "tasks": 0}
+    a = from_slo("test/agentdynamics", c, "trigger", now)
+    a["summary"] = "AgentDynamics test alert: this destination is configured correctly"
+    return a