agentdynamics 0.4.1__tar.gz → 0.5.0__tar.gz

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentdynamics
-Version: 0.4.1
+Version: 0.5.0
 Summary: Application performance monitoring for AI agents: flows, cost, reliability and process quality
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/Aditya31398/agentdynamics
@@ -61,9 +61,15 @@ def handle_ticket(question):
         plan = client.messages.create(...)
     with agentdynamics.span("act"):
         lookup_order(order_id)           # decorate tools with @agentdynamics.tool
+    if escalated:
+        agentdynamics.outcome("failed", reason="customer escalated")   # state it, don't let it be guessed
     ...
 ```
 
+Without `outcome()`, success is inferred from errors, interrupts and follow-ups. The console always shows
+which outcomes were stated and which were guessed; grades can also be posted afterwards by a person or an
+eval pipeline (`POST /api/outcomes`).
+
 ### 2. Python, zero code changes
 
 ```bash
@@ -107,6 +113,31 @@ The console's **Get started** page shows the same snippets and turns green when
 
 ---
 
+## What it looks like
+
+The Flow Map: where work goes, what each hop costs, and what is failing. Red edges are errors.
+
+![Flow Map](docs/img/flow-map.png)
+
+The Overview: cost, Apdex and flow health against baselines learned from your own history.
+
+![Overview](docs/img/overview.png)
+
+Governance: what Aegis refused, which rule refused it, and which grants are never used.
+
+![Governance](docs/img/governance.png)
+
+Reproduce these locally from synthetic traffic, no API keys needed:
+
+```bash
+agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
+LANGSMITH_ENDPOINT=http://127.0.0.1:8790/langsmith python examples/langgraph_style_app.py 70
+python examples/otel_multiagent.py 45 http://127.0.0.1:8790
+AGENTDYNAMICS_URL=http://127.0.0.1:8790 python examples/governed_agent.py 40
+```
+
+---
+
 ## What you get
 
 | | |
@@ -118,7 +149,7 @@ The console's **Get started** page shows the same snippets and turns green when
 | **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack or webhooks |
 | **Tools & Models** | Error rate and p95 per tool; per model TTFT, tokens/s, truncation rate, cache hit rate and spend |
 | **SLOs** | Success rate, Apdex, latency and cost objectives with error budgets and burn rates |
-| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export |
+| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export, and what a policy change would refuse |
 | **Process Review** | Scores how well the agent works (efficiency, focus, reliability, verification, context, autonomy, compliance) and suggests fixes |
 | **Analytics & Compare** | Any metric by any dimension; compare models, prompts or releases side by side |
 
@@ -137,7 +168,7 @@ That call sets up four flows:
 1. **Every Aegis decision appears in its task.** Denials carry the rule id, and the Aegis audit log carries the task's run id so the two logs join.
 2. **Model calls are charged to the Aegis budget before they're sent.** When the budget runs out, the call is refused and never made.
 3. **A watchdog revokes the grant** of an agent that keeps probing a boundary, for example after a prompt injection.
-4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour, which `aegis ratify` and `aegis drift` then verify.
+4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour. `aegis ratify` and `aegis drift` verify its declaration; `agentdynamics policy check` replays the traffic that actually ran through it and reports how much it would refuse, so a policy change can be gated in CI.
 
 The console's **Governance** page shows denials, budget stops, revocations, unused grants and budget headroom per policy version. See [docs/GOVERNANCE.md](docs/GOVERNANCE.md).
 
@@ -184,12 +215,13 @@ See [deploy/](deploy/) and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 ## Development
 
 ```bash
-python -m unittest discover tests   # 28 tests: real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
-python -m agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
-python examples/langgraph_style_app.py 60          # LangGraph-style demo traffic
-python examples/otel_multiagent.py 60 http://127.0.0.1:8790
+pip install -e ".[test,toml]" && pip install "aegis-kernel>=0.4.0"
+python -m unittest discover tests   # real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
 ```
 
+Demo traffic is under **What it looks like** above. [CLAUDE.md](CLAUDE.md) has the architecture and the
+invariants to work within; [ROADMAP.md](ROADMAP.md) is what is planned and what is deliberately not.
+
 ## License
 
 Apache 2.0. See [LICENSE](LICENSE).

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/README.md

@@ -38,9 +38,15 @@ def handle_ticket(question):
         plan = client.messages.create(...)
     with agentdynamics.span("act"):
         lookup_order(order_id)           # decorate tools with @agentdynamics.tool
+    if escalated:
+        agentdynamics.outcome("failed", reason="customer escalated")   # state it, don't let it be guessed
     ...
 ```
 
+Without `outcome()`, success is inferred from errors, interrupts and follow-ups. The console always shows
+which outcomes were stated and which were guessed; grades can also be posted afterwards by a person or an
+eval pipeline (`POST /api/outcomes`).
+
 ### 2. Python, zero code changes
 
 ```bash
@@ -84,6 +90,31 @@ The console's **Get started** page shows the same snippets and turns green when
 
 ---
 
+## What it looks like
+
+The Flow Map: where work goes, what each hop costs, and what is failing. Red edges are errors.
+
+![Flow Map](docs/img/flow-map.png)
+
+The Overview: cost, Apdex and flow health against baselines learned from your own history.
+
+![Overview](docs/img/overview.png)
+
+Governance: what Aegis refused, which rule refused it, and which grants are never used.
+
+![Governance](docs/img/governance.png)
+
+Reproduce these locally from synthetic traffic, no API keys needed:
+
+```bash
+agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
+LANGSMITH_ENDPOINT=http://127.0.0.1:8790/langsmith python examples/langgraph_style_app.py 70
+python examples/otel_multiagent.py 45 http://127.0.0.1:8790
+AGENTDYNAMICS_URL=http://127.0.0.1:8790 python examples/governed_agent.py 40
+```
+
+---
+
 ## What you get
 
 | | |
@@ -95,7 +126,7 @@ The console's **Get started** page shows the same snippets and turns green when
 | **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack or webhooks |
 | **Tools & Models** | Error rate and p95 per tool; per model TTFT, tokens/s, truncation rate, cache hit rate and spend |
 | **SLOs** | Success rate, Apdex, latency and cost objectives with error budgets and burn rates |
-| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export |
+| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export, and what a policy change would refuse |
 | **Process Review** | Scores how well the agent works (efficiency, focus, reliability, verification, context, autonomy, compliance) and suggests fixes |
 | **Analytics & Compare** | Any metric by any dimension; compare models, prompts or releases side by side |
 
@@ -114,7 +145,7 @@ That call sets up four flows:
 1. **Every Aegis decision appears in its task.** Denials carry the rule id, and the Aegis audit log carries the task's run id so the two logs join.
 2. **Model calls are charged to the Aegis budget before they're sent.** When the budget runs out, the call is refused and never made.
 3. **A watchdog revokes the grant** of an agent that keeps probing a boundary, for example after a prompt injection.
-4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour, which `aegis ratify` and `aegis drift` then verify.
+4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour. `aegis ratify` and `aegis drift` verify its declaration; `agentdynamics policy check` replays the traffic that actually ran through it and reports how much it would refuse, so a policy change can be gated in CI.
 
 The console's **Governance** page shows denials, budget stops, revocations, unused grants and budget headroom per policy version. See [docs/GOVERNANCE.md](docs/GOVERNANCE.md).
 
@@ -161,12 +192,13 @@ See [deploy/](deploy/) and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 ## Development
 
 ```bash
-python -m unittest discover tests   # 28 tests: real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
-python -m agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
-python examples/langgraph_style_app.py 60          # LangGraph-style demo traffic
-python examples/otel_multiagent.py 60 http://127.0.0.1:8790
+pip install -e ".[test,toml]" && pip install "aegis-kernel>=0.4.0"
+python -m unittest discover tests   # real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
 ```
 
+Demo traffic is under **What it looks like** above. [CLAUDE.md](CLAUDE.md) has the architecture and the
+invariants to work within; [ROADMAP.md](ROADMAP.md) is what is planned and what is deliberately not.
+
 ## License
 
 Apache 2.0. See [LICENSE](LICENSE).

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/__init__.py

@@ -5,8 +5,8 @@
 
 See agentdynamics.autotrace for @trace / @tool / span.
 """
-__version__ = "0.4.1"
+__version__ = "0.5.0"
 
-from .autotrace import flush, init, llm_call, record_llm, span, tool, trace  # noqa: E402,F401
+from .autotrace import flush, init, llm_call, outcome, record_llm, span, tool, trace  # noqa: E402,F401
 
-__all__ = ["init", "trace", "tool", "span", "llm_call", "record_llm", "flush", "__version__"]
+__all__ = ["init", "trace", "tool", "span", "llm_call", "record_llm", "outcome", "flush", "__version__"]

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/__main__.py

@@ -191,6 +191,22 @@ def cmd_policy(a, eng):
         for r in g["by_rule"][:8]:
             print(f"  {r['n']:>5}  {r['rule']}")
         return 0
+    if a.action == "check":
+        if not a.candidate:
+            print("policy check needs --candidate <policy.yaml>", file=sys.stderr)
+            return 2
+        try:
+            from aegis import dump_policy, load_policy
+        except ImportError:
+            print("policy check needs aegis-kernel: pip install aegis-kernel", file=sys.stderr)
+            return 2
+        q["candidate_doc"] = dump_policy(load_policy(a.candidate))
+        res = api.check_policy(q)
+        if res.get("error"):
+            print(res["error"], file=sys.stderr)
+            return 1
+        print(f"{a.candidate} against {res['tasks']} governed task(s) in scope:")
+        return _report_coverage(res["coverage"], a.max_denied_fraction, sys.stdout)
     res = api.export_policy(q)
     if res.get("error"):
         print(res["error"], file=sys.stderr)
@@ -203,16 +219,41 @@ def cmd_policy(a, eng):
             print(f"  - {c}", file=sys.stderr)
     else:
         print(res["yaml"])
-    # Replaying the observed calls is the only check that catches a policy which is strictly
-    # narrower than its base, ratifies cleanly, and still refuses everything in production.
-    if res.get("regressions"):
-        print("", file=sys.stderr)
-        print(f"WARNING: this policy would deny {len(res['regressions'])} kind(s) of call that "
-              f"the observed traffic made successfully:", file=sys.stderr)
-        for r in res["regressions"][:10]:
-            print(f"  - {r['tool']}({r['arg']}={r['value']!r}) -> {r['rule']}", file=sys.stderr)
-        print("  Review before deploying; `aegis ratify` and `aegis drift` cannot see this.",
-              file=sys.stderr)
+    if res.get("coverage") is None:
+        return 0
+    print("", file=sys.stderr)
+    print("Against the traffic it was generated from:", file=sys.stderr)
+    return _report_coverage(res["coverage"], a.max_denied_fraction, sys.stderr)
+
+
+def _report_coverage(cov, max_fraction, out):
+    """Print how much observed traffic a candidate would refuse; non-zero exit past the threshold.
+
+    This is the one check that catches a policy which is strictly narrower than its base, ratifies
+    cleanly and shows no drift, and still refuses production traffic -- `aegis ratify` and
+    `aegis drift` compare declarations and never see a single call.
+    """
+    w = lambda *x: print(*x, file=out)  # noqa: E731
+    if not cov["calls"]:
+        w("  no governed calls in scope to check against")
+        return 0
+    w(f"  would deny {cov['denied']} of {cov['calls']} calls that were allowed ({cov['denied_fraction']:.1%})")
+    rows = [r for r in cov["by_tool"] if r["denied"]]
+    if rows:
+        width = max(len(r["tool"]) for r in rows)
+        for r in rows:
+            why = ", ".join(f"{k} x{n}" for k, n in sorted(r["rules"].items(), key=lambda kv: -kv[1]))
+            w(f"    {r['tool']:<{width}}  {r['denied']:>5} of {r['calls']:<5} {r['denied_fraction']:>6.1%}  {why}")
+        for ex in cov["examples"][:5]:
+            if ex.get("arg"):
+                w(f"      e.g. {ex['tool']}({ex['arg']}={ex['value']!r}) -> {ex['rule']}")
+            else:            # a missing argument or a dropped tool: Aegis's own reason says which
+                w(f"      e.g. {ex['tool']} -> {ex['rule']}: {ex.get('reason') or ''}")
+    if cov["args_unrecorded"]:
+        w(f"  {cov['args_unrecorded']} call(s) had no recorded arguments (content capture off); "
+          f"only their tool grant was checked")
+    if cov["denied_fraction"] > max_fraction:
+        w(f"  FAIL: {cov['denied_fraction']:.1%} exceeds --max-denied-fraction {max_fraction:.1%}")
         return 1
     return 0
 
@@ -257,7 +298,9 @@ def main(argv=None):
     k.add_argument("--role", choices=["ingest", "read", "admin"], default="ingest")
     k.add_argument("--name", default=None)
     po = sub.add_parser("policy", help="Aegis policy from observed behaviour (observe -> govern)")
-    po.add_argument("action", choices=["export", "report"])
+    po.add_argument("action", choices=["export", "report", "check"],
+                    help="export: generate a tightened policy; report: governance summary; "
+                         "check: judge --candidate against recorded traffic")
     po.add_argument("--workflow")
     po.add_argument("--project")
     po.add_argument("--environment")
@@ -266,6 +309,10 @@ def main(argv=None):
     po.add_argument("--base", help="Aegis policy file to tighten (needs aegis-kernel installed)")
     po.add_argument("--headroom", type=float, default=1.5)
     po.add_argument("--out", help="write the YAML here (default: stdout)")
+    po.add_argument("--candidate", help="policy file to judge (check)")
+    po.add_argument("--max-denied-fraction", type=float, default=0.0,
+                    help="fail if the policy would refuse more than this share of observed, allowed calls "
+                         "(export, check; default 0: any refusal fails)")
     sub.add_parser("ingest", help="scan sources once and rebuild the database")
     rp = sub.add_parser("report", help="print a text summary")
     rp.add_argument("--project")

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/analysis.py

@@ -40,23 +40,34 @@ CORRECTION = re.compile(
 
 
 def classify_task(prompt_kind, text):
+    return classify_task_detail(prompt_kind, text)[0]
+
+
+def classify_task_detail(prompt_kind, text):
+    """(task_type, how it was decided, what matched).
+
+    how: "prompt kind" when the prompt says what it is (a slash command, a scheduled job, a subagent's
+    brief); "follow-up" for a short "continue"/"yes" that carries on the previous task; "keywords" when an
+    intent rule matched, with the matched text; "unmatched" when nothing did. Keyword typing is a guess
+    about what someone meant, so it says so -- the same way outcome_source separates graded from inferred.
+    """
     if prompt_kind == "scheduled":
-        return "scheduled job"
+        return "scheduled job", "prompt kind", None
     if prompt_kind == "delegated":
-        return "subagent"
+        return "subagent", "prompt kind", None
     t = (text or "").strip()
     low = t.lower()
     if prompt_kind == "command":
-        return "slash command"
+        return "slash command", "prompt kind", None
     if len(t) < 60 and FOLLOWUP.search(t):
-        return "follow-up"
+        return "follow-up", "follow-up", None
     # The earliest intent keyword wins: "Create X, then test it" is a build, not a testing task.
     best = None
     for rank, (name, pat) in enumerate(TYPE_RULES):
         m = re.search(pat, low[:600])
         if m and (best is None or (m.start(), rank) < best[0]):
-            best = ((m.start(), rank), name)
-    return best[1] if best else "other"
+            best = ((m.start(), rank), name, m.group(0).strip())
+    return (best[1], "keywords", best[2][:40]) if best else ("other", "unmatched", None)
 
 
 # ---------------------------------------------------------------- helpers
@@ -146,9 +157,10 @@ def task_metrics(run, idx, seg):
     in_side = t["input_tokens"] + t["cache_read"] + t["cache_write"]
     t["cache_hit"] = round(t["cache_read"] / in_side, 3) if in_side else None
     t["tool_error_rate"] = round(t["tool_errors"] / len(tools), 3) if tools else 0
-    t["task_type"] = classify_task(t["prompt_kind"], t["prompt"])
+    t["task_type"], t["task_type_source"], t["task_type_match"] = classify_task_detail(t["prompt_kind"], t["prompt"])
     if run.get("workflow") and run.get("source") != "claude-code":
-        t["task_type"] = run["workflow"]  # traced apps: the entry point names the business transaction
+        # traced apps: the entry point names the business transaction, which beats any guess
+        t["task_type"], t["task_type_source"], t["task_type_match"] = run["workflow"], "workflow", None
 
     # --- phase attribution: split each LLM call's cost across the tool calls it issued
     phase_calls = Counter(s["phase"] for s in tools)
@@ -301,6 +313,9 @@ def governance_metrics(t, run, steps, tools):
     t["repeated_denials"] = best
 
 
+OUTCOMES = ("completed", "failed", "rework", "interrupted")   # what a grade may state (store.OUTCOMES)
+FEEDBACK_PASS = 0.5                                             # a feedback score at or above this is a pass
+
 TRUNCATION = {"max_tokens", "length", "max_output_tokens", "MAX_TOKENS"}
 REFUSAL = {"refusal", "content_filter", "SAFETY", "safety", "blocked"}
 RATE_HINTS = ("429", "rate limit", "rate_limit", "overloaded", "529", "too many requests", "quota")
@@ -314,6 +329,13 @@ def flow_metrics(t, run, steps, llm, tools):
     t["framework"] = run.get("framework") or run.get("source")
     t["workflow"] = run.get("workflow") or (t["task_type"] if run.get("source") == "claude-code" else None)
     t["steps_total"] = len(llm) + len(tools)
+    # model calls whose cache accounting rests on a guess (collectors/spans.uncached_input); None is
+    # a source that is exact by construction, like the SDK, so only an explicit False counts
+    t["tokens_unverified"] = sum(1 for s in llm if s.get("tokens_verified") is False)
+    # agentdynamics.outcome() records a notice; the last one in the task wins. Kept on a transient
+    # key -- finalize decides precedence, and write_analysis persists only TASK_COLS.
+    graded = [s for s in notices if s.get("name") == "outcome" and s.get("outcome") in OUTCOMES]
+    t["_sdk_grade"] = {"outcome": graded[-1]["outcome"], "reason": graded[-1].get("text")} if graded else None
     t["llm_errors"] = sum(1 for s in llm if s.get("is_error"))
     t["truncations"] = sum(1 for s in llm if s.get("stop_reason") in TRUNCATION)
     t["refusals"] = sum(1 for s in llm if s.get("stop_reason") in REFUSAL)
@@ -538,6 +560,7 @@ def _outcomes_claude(ts, now):
         # "continue" / "yes" carries on the previous request, so it belongs to that task type
         if t["task_type"] == "follow-up" and i > 0 and ts[i - 1]["task_type"] not in ("follow-up",):
             t["task_type"] = ts[i - 1]["task_type"]
+            t["task_type_source"], t["task_type_match"] = "follow-up", None   # inherited, not re-guessed
             if t["source"] == "claude-code":
                 t["workflow"] = t["task_type"]
         nxt = next((x for x in ts[i + 1:] if x["prompt_kind"] in ("human", "command")), None)
@@ -573,7 +596,34 @@ def _outcome_trace(t, run, now):
     t["root_error"] = run.get("root_error")
 
 
-def finalize(runs, tasks_by_run, rules=None, now=None):
+def _apply_grades(tasks, grades):
+    """Settle each outcome by the strongest evidence available, and record which one it was.
+
+    stated after the fact (API)  >  stated in the run (agentdynamics.outcome)  >  recorded feedback
+    >  inference from signals.
+
+    Inference is a guess built from errors, interrupts and corrections. Apdex, success rate and the
+    process score all inherit it, so anything that states the outcome outright must win, and the
+    console has to be able to say how much of a success rate is guessed. Idempotent: the cached
+    per-run tasks are re-finalized on every refresh, so nothing here may consume its input.
+    """
+    for t in tasks:
+        api, sdk = grades.get(t["id"]), t.get("_sdk_grade")
+        if api:
+            t["outcome"], t["outcome_source"] = api["outcome"], "graded"
+            t["outcome_reason"] = api.get("reason") or f"graded by {api.get('graded_by') or 'api'}"
+        elif sdk:
+            t["outcome"], t["outcome_source"] = sdk["outcome"], "graded"
+            t["outcome_reason"] = sdk.get("reason") or "graded in the run"
+        elif t.get("feedback_score") is not None:
+            ok = t["feedback_score"] >= FEEDBACK_PASS
+            t["outcome"] = "completed" if ok else "rework"
+            t["outcome_source"], t["outcome_reason"] = "feedback", f"feedback score {t['feedback_score']:g}"
+        else:
+            t["outcome_source"], t["outcome_reason"] = "inferred", None
+
+
+def finalize(runs, tasks_by_run, rules=None, now=None, grades=None):
     """Cross-run analysis: outcomes, subagent roll-up, baselines, scores, events."""
     rules = rules or DEFAULT_RULES
     now = now or time.time()
@@ -599,6 +649,8 @@ def finalize(runs, tasks_by_run, rules=None, now=None):
         if not (run["source"] == "claude-code" or not run.get("workflow")):
             for t in tasks_by_run[run["id"]]:
                 _outcome_trace(t, run, now)
+    # before the roll-up, so baselines, scores and health events all see the settled outcome
+    _apply_grades(all_tasks, grades or {})
 
     # subagent roll-up: link child runs to the parent task that spawned them
     task_by_id = {t["id"]: t for t in all_tasks}

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/autotrace.py

@@ -215,6 +215,10 @@ class trace:
         """Attach a user/eval score (0..1) to this task."""
         self.run.feedback.append({"key": key, "score": score})
 
+    def outcome(self, result, reason=None):
+        """State how this task ended. See agentdynamics.outcome."""
+        _record_outcome(self.run, result, reason)
+
     # decorator
     def __call__(self, *args, **kwargs):
         if self._fn is None:  # used as @trace("name")
@@ -402,6 +406,38 @@ class llm_call:
         return False
 
 
+OUTCOMES = ("completed", "failed", "rework", "interrupted")
+
+
+def _record_outcome(run, result, reason):
+    if result not in OUTCOMES:
+        # telemetry never breaks the agent: a typo is a warning, not an exception in their code path
+        _warn(f"outcome:{result!r}", f"outcome({result!r}) ignored: must be one of {', '.join(OUTCOMES)}")
+        return
+    run.add({"kind": "notice", "name": "outcome", "outcome": result, "ts": time.time(),
+             "text": _clip(str(reason), 500) if reason is not None else None})
+
+
+def outcome(result, reason=None):
+    """State how the current task ended, instead of leaving it to inference.
+
+        @agentdynamics.trace
+        def handle(ticket):
+            ...
+            if not resolved:
+                agentdynamics.outcome("failed", reason="customer escalated")
+
+    `result` is completed | failed | rework | interrupted. The last call in a task wins, and it
+    overrides both inference and recorded feedback; only a grade set later through the API ranks
+    above it. Outside a traced task there is nothing to grade, so it warns once and does nothing.
+    """
+    run = _current.get()
+    if run is None:
+        _warn("outcome:no-run", "agentdynamics.outcome() called outside a traced task; ignored")
+        return
+    _record_outcome(run, result, reason)
+
+
 def record_llm(model, input_tokens=0, output_tokens=0, cache_read=0, cache_write=0, stop_reason=None,
                start=None, end=None, text="", provider="custom"):
     """Record a model call after the fact (no gating). Prefer `llm_call` when you can wrap the call."""

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/collectors/langsmith.py

@@ -83,11 +83,15 @@ def _usage(outputs, run):
         det = um.get("input_token_details") or {}
         cr = det.get("cache_read") or 0
         cw = det.get("cache_creation") or 0
+    # langchain-core UsageMetadata.input_tokens: "Sum of all input token types" -- cache reads and
+    # cache writes are already inside it. The legacy llm_output.token_usage path carries no cache
+    # breakdown, so there is nothing to double count and no convention to claim.
+    conv = "inclusive" if um else None
     tu = (o.get("llm_output") or {}).get("token_usage") or (o.get("llm_output") or {}).get("usage") or {}
     if tu and not (it or ot):
         it = tu.get("prompt_tokens") or tu.get("input_tokens") or 0
         ot = tu.get("completion_tokens") or tu.get("output_tokens") or 0
-    return it, ot, cr, cw, stop, model
+    return it, ot, cr, cw, stop, model, conv
 
 
 KIND = {"llm": "llm", "tool": "tool", "retriever": "retriever", "embedding": "embedding", "chain": "chain", "prompt": "chain", "parser": "chain"}
@@ -108,9 +112,9 @@ def to_span(r):
     if r.get("name") in ("__interrupt__",) or "GraphInterrupt" in str(r.get("error") or ""):
         kind = "human"
     it = ot = cr = cw = 0
-    stop = model_out = None
+    stop = model_out = conv = None
     if kind in ("llm", "embedding"):
-        it, ot, cr, cw, stop, model_out = _usage(r.get("outputs"), r)
+        it, ot, cr, cw, stop, model_out, conv = _usage(r.get("outputs"), r)
     start = parse_time(r.get("start_time"))
     ttft = None
     for ev in r.get("events") or []:
@@ -136,6 +140,7 @@ def to_span(r):
         "status": "error" if err else ("ok" if r.get("end_time") else "unset"), "error": err,
         "model": md.get("ls_model_name") or inv.get("model") or inv.get("model_name") or model_out,
         "provider": md.get("ls_provider"), "input_tokens": it, "output_tokens": ot, "cache_read": cr, "cache_write": cw,
+        "input_convention": conv,
         "cost": r.get("total_cost") if kind == "llm" and r.get("total_cost") is not None else None,
         "stop_reason": stop, "ttft_ms": ttft, "input": r.get("inputs"), "output": r.get("outputs"),
         "node": node, "agent": md.get("agent_name") or md.get("lc_agent_name"),

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/collectors/otlp.py

@@ -204,6 +204,24 @@ def decode_json(doc):
 
 # ---------------------------------------------------------------- semantic-convention mapping
 
+def _input_convention(a):
+    """Whether this span's input_tokens already counts its cache reads and writes, per the documented
+    contract of whichever attribute supplied it. Mirrors _g's order so it describes the same key.
+
+    gen_ai.usage.input_tokens   GenAI semconv: "SHOULD include all types of input tokens, including
+                                cached tokens."
+    llm.token_count.prompt      OpenInference: cache reads and writes are "tokens in the prompt".
+    anything else               (gen_ai.usage.prompt_tokens from older semconv / OpenLLMetry, llm.usage.*)
+                                has no documented rule we can cite: None, and the call is reported as
+                                unverified if it involves cache tokens.
+    """
+    for k, conv in (("gen_ai.usage.input_tokens", "inclusive"), ("gen_ai.usage.prompt_tokens", None),
+                    ("llm.token_count.prompt", "inclusive"), ("llm.usage.prompt_tokens", None)):
+        if a.get(k) not in (None, "", []):
+            return conv
+    return None
+
+
 def _g(a, *keys):
     for k in keys:
         v = a.get(k)
@@ -319,6 +337,7 @@ def map_span(res, scope, sp):
         "model": _g(a, "gen_ai.response.model", "gen_ai.request.model", "llm.model_name", "llm.request.model"),
         "provider": _g(a, "gen_ai.system", "gen_ai.provider.name", "llm.provider", "llm.system"),
         "input_tokens": _num(_g(a, "gen_ai.usage.input_tokens", "gen_ai.usage.prompt_tokens", "llm.token_count.prompt", "llm.usage.prompt_tokens")) or 0,
+        "input_convention": _input_convention(a),
         "output_tokens": _num(_g(a, "gen_ai.usage.output_tokens", "gen_ai.usage.completion_tokens", "llm.token_count.completion", "llm.usage.completion_tokens")) or 0,
         "cache_read": _num(_g(a, "gen_ai.usage.cache_read.input_tokens", "gen_ai.usage.cache_read_input_tokens", "llm.token_count.prompt_details.cache_read",
                              "gen_ai.usage.input_tokens.cached")) or 0,

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/collectors/spans.py

@@ -22,6 +22,34 @@ STRUCTURAL = {"agent", "chain", "node", "guardrail", "evaluator", "human", "span
 RATE_LIMIT_HINTS = ("429", "rate limit", "rate_limit", "ratelimit", "overloaded", "529", "too many requests", "quota")
 
 
+def uncached_input(input_tokens, cache_read, cache_write, convention):
+    """Split a reported input count into its uncached part, and say whether we know it's right.
+
+    Canonical steps keep input_tokens, cache_read and cache_write disjoint, because each is priced
+    differently. Sources disagree on whether their input count already contains the cache tokens, and
+    getting it wrong double counts them in the cost column.
+
+    inclusive   documented as containing reads and writes (GenAI semconv, OpenInference,
+                LangChain usage_metadata): subtract both.
+    exclusive   documented as not containing them: leave it.
+    None        no rule we can cite. Fall back to the old guess -- inclusive of reads unless that is
+                arithmetically impossible -- and report the call as unverified, the way a model with
+                no known price is reported as unpriced rather than guessed.
+
+    A declared-inclusive count smaller than its own cache tokens cannot be inclusive; the emitter is
+    breaking its format's contract. That falls back too, and is reported rather than absorbed.
+    Returns (uncached_input, verified).
+    """
+    if not (cache_read or cache_write):
+        return input_tokens, True              # nothing cached, nothing to disagree about
+    if convention == "exclusive":
+        return input_tokens, True
+    if convention == "inclusive" and input_tokens >= cache_read + cache_write:
+        return input_tokens - cache_read - cache_write, True
+    if cache_read and input_tokens >= cache_read:
+        return input_tokens - cache_read, False
+    return input_tokens, False
+
 def preview(v, n=600):
     if v is None:
         return ""
@@ -156,9 +184,7 @@ def build_run(trace_id, spans, source):
             model = s.get("model") or s.get("name")
             it, ot = int(s.get("input_tokens") or 0), int(s.get("output_tokens") or 0)
             cr, cw = int(s.get("cache_read") or 0), int(s.get("cache_write") or 0)
-            # OTel/LangSmith report input_tokens inclusive of cache reads for most providers; keep uncached part separate
-            if cr and it >= cr:
-                it -= cr
+            it, verified = uncached_input(it, cr, cw, s.get("input_convention"))
             cost = s.get("cost")
             if cost is None:
                 cost = pricing.cost(model, it, ot, cr, cw, 0)
@@ -168,6 +194,7 @@ def build_run(trace_id, spans, source):
                 "context_tokens": it + cr + cw, "stop_reason": s.get("stop_reason"), "ttft_ms": s.get("ttft_ms"),
                 "text": preview(s.get("output"), 600), "tool_calls": 0, "thinking_tokens": int(s.get("thinking_tokens") or 0),
                 "rate_limited": bool(err) and any(h in err.lower() for h in RATE_LIMIT_HINTS),
+                "tokens_verified": verified,
             })
             last_llm = st
         elif kind in ("tool", "retriever"):

{agentdynamics-0.4.1 → agentdynamics-0.5.0}/agentdynamics/engine.py

@@ -60,6 +60,10 @@ class Engine:
         # health has to be judged on the failures too, not just on a timestamp existing.
         self.last_refresh_error = None
         self.failed_refreshes = 0
+        # A grade changes no run, so without this a new grade would sit unapplied until unrelated
+        # traffic happened to dirty something. force=True would also work, but it throws away every
+        # cache and re-parses every source to re-settle outcomes that only need re-finalizing.
+        self._regrade = False
         self.stats = {"spans_ingested": 0, "refreshes": 0, "alerts_sent": 0}
         self.sources = {}
         if claude_root:
@@ -120,6 +124,22 @@ class Engine:
                     time.sleep(interval)
             threading.Thread(target=loop, daemon=True, name=f"src-{name}").start()
 
+    # ------------------------------------------------------------------ outcome grades
+    def grade(self, task_id, outcome, reason=None, graded_by=None):
+        """State a task's outcome, overriding inference. Durable: survives schema rebuilds."""
+        with self.lock:
+            store.set_grade(self.con, task_id, outcome, reason, graded_by)
+            self._regrade = True
+        self._wake.set()
+
+    def ungrade(self, task_id):
+        """Drop a stated outcome; the task falls back to feedback, then to inference."""
+        with self.lock:
+            n = store.delete_grade(self.con, task_id)
+            self._regrade = True
+        self._wake.set()
+        return n
+
     # ------------------------------------------------------------------ push ingestion
     def ingest_spans(self, spans_, source, count_source=True):
         """Canonical spans (OTLP, Langfuse, inbox) -> durable store."""
@@ -318,8 +338,9 @@ class Engine:
                         removed.append(rid)
                         dirty.pop(rid, None)
             dirty = {k: v for k, v in dirty.items() if v["steps"]}
-            if not dirty and not removed and not force and not self._first:
+            if not dirty and not removed and not force and not self._first and not self._regrade:
                 return False
+            self._regrade = False
             for rid in removed:
                 self._runs.pop(rid, None)
                 self._tasks.pop(rid, None)
@@ -330,7 +351,8 @@ class Engine:
                 self._runs[rid] = run
                 self._tasks[rid] = analysis.run_tasks(run)
             runs = list(self._runs.values())
-            tasks, baselines, events = analysis.finalize(runs, self._tasks, self.rules())
+            tasks, baselines, events = analysis.finalize(runs, self._tasks, self.rules(),
+                                                         grades=store.get_grades(self.con))
             insights = analysis.process_insights(tasks)
             meta = {"refreshed": time.time(), "runs": len(runs), "tasks": len(tasks), "insights": insights}
             store.write_runs(self.con, list(dirty.values()), removed, self.redactor)