agentdynamics 0.4.1__tar.gz → 0.6.0__tar.gz

{agentdynamics-0.4.1 → agentdynamics-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentdynamics
-Version: 0.4.1
+Version: 0.6.0
 Summary: Application performance monitoring for AI agents: flows, cost, reliability and process quality
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/Aditya31398/agentdynamics
@@ -61,9 +61,15 @@ def handle_ticket(question):
         plan = client.messages.create(...)
     with agentdynamics.span("act"):
         lookup_order(order_id)           # decorate tools with @agentdynamics.tool
+    if escalated:
+        agentdynamics.outcome("failed", reason="customer escalated")   # state it, don't let it be guessed
     ...
 ```
 
+Without `outcome()`, success is inferred from errors, interrupts and follow-ups. The console always shows
+which outcomes were stated and which were guessed; grades can also be posted afterwards by a person or an
+eval pipeline (`POST /api/outcomes`).
+
 ### 2. Python, zero code changes
 
 ```bash
@@ -107,6 +113,31 @@ The console's **Get started** page shows the same snippets and turns green when
 
 ---
 
+## What it looks like
+
+The Flow Map: where work goes, what each hop costs, and what is failing. Red edges are errors.
+
+![Flow Map](docs/img/flow-map.png)
+
+The Overview: cost, Apdex and flow health against baselines learned from your own history.
+
+![Overview](docs/img/overview.png)
+
+Governance: what Aegis refused, which rule refused it, and which grants are never used.
+
+![Governance](docs/img/governance.png)
+
+Reproduce these locally from synthetic traffic, no API keys needed:
+
+```bash
+agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
+LANGSMITH_ENDPOINT=http://127.0.0.1:8790/langsmith python examples/langgraph_style_app.py 70
+python examples/otel_multiagent.py 45 http://127.0.0.1:8790
+AGENTDYNAMICS_URL=http://127.0.0.1:8790 python examples/governed_agent.py 40
+```
+
+---
+
 ## What you get
 
 | | |
@@ -118,7 +149,7 @@ The console's **Get started** page shows the same snippets and turns green when
 | **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack or webhooks |
 | **Tools & Models** | Error rate and p95 per tool; per model TTFT, tokens/s, truncation rate, cache hit rate and spend |
 | **SLOs** | Success rate, Apdex, latency and cost objectives with error budgets and burn rates |
-| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export |
+| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export, and what a policy change would refuse |
 | **Process Review** | Scores how well the agent works (efficiency, focus, reliability, verification, context, autonomy, compliance) and suggests fixes |
 | **Analytics & Compare** | Any metric by any dimension; compare models, prompts or releases side by side |
 
@@ -137,7 +168,7 @@ That call sets up four flows:
 1. **Every Aegis decision appears in its task.** Denials carry the rule id, and the Aegis audit log carries the task's run id so the two logs join.
 2. **Model calls are charged to the Aegis budget before they're sent.** When the budget runs out, the call is refused and never made.
 3. **A watchdog revokes the grant** of an agent that keeps probing a boundary, for example after a prompt injection.
-4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour, which `aegis ratify` and `aegis drift` then verify.
+4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour. `aegis ratify` and `aegis drift` verify its declaration; `agentdynamics policy check` replays the traffic that actually ran through it and reports how much it would refuse, so a policy change can be gated in CI.
 
 The console's **Governance** page shows denials, budget stops, revocations, unused grants and budget headroom per policy version. See [docs/GOVERNANCE.md](docs/GOVERNANCE.md).
 
@@ -184,12 +215,13 @@ See [deploy/](deploy/) and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 ## Development
 
 ```bash
-python -m unittest discover tests   # 28 tests: real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
-python -m agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
-python examples/langgraph_style_app.py 60          # LangGraph-style demo traffic
-python examples/otel_multiagent.py 60 http://127.0.0.1:8790
+pip install -e ".[test,toml]" && pip install "aegis-kernel>=0.4.0"
+python -m unittest discover tests   # real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
 ```
 
+Demo traffic is under **What it looks like** above. [CLAUDE.md](CLAUDE.md) has the architecture and the
+invariants to work within; [ROADMAP.md](ROADMAP.md) is what is planned and what is deliberately not.
+
 ## License
 
 Apache 2.0. See [LICENSE](LICENSE).

{agentdynamics-0.4.1 → agentdynamics-0.6.0}/README.md

@@ -38,9 +38,15 @@ def handle_ticket(question):
         plan = client.messages.create(...)
     with agentdynamics.span("act"):
         lookup_order(order_id)           # decorate tools with @agentdynamics.tool
+    if escalated:
+        agentdynamics.outcome("failed", reason="customer escalated")   # state it, don't let it be guessed
     ...
 ```
 
+Without `outcome()`, success is inferred from errors, interrupts and follow-ups. The console always shows
+which outcomes were stated and which were guessed; grades can also be posted afterwards by a person or an
+eval pipeline (`POST /api/outcomes`).
+
 ### 2. Python, zero code changes
 
 ```bash
@@ -84,6 +90,31 @@ The console's **Get started** page shows the same snippets and turns green when
 
 ---
 
+## What it looks like
+
+The Flow Map: where work goes, what each hop costs, and what is failing. Red edges are errors.
+
+![Flow Map](docs/img/flow-map.png)
+
+The Overview: cost, Apdex and flow health against baselines learned from your own history.
+
+![Overview](docs/img/overview.png)
+
+Governance: what Aegis refused, which rule refused it, and which grants are never used.
+
+![Governance](docs/img/governance.png)
+
+Reproduce these locally from synthetic traffic, no API keys needed:
+
+```bash
+agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
+LANGSMITH_ENDPOINT=http://127.0.0.1:8790/langsmith python examples/langgraph_style_app.py 70
+python examples/otel_multiagent.py 45 http://127.0.0.1:8790
+AGENTDYNAMICS_URL=http://127.0.0.1:8790 python examples/governed_agent.py 40
+```
+
+---
+
 ## What you get
 
 | | |
@@ -95,7 +126,7 @@ The console's **Get started** page shows the same snippets and turns green when
 | **Health rules & alerts** | 28 rules (runaway cost, policy denials, boundary probing, revocations, retry loops, node loops, truncation, rate limits, ping-pong handoffs, unverified code changes, …) sent to Slack or webhooks |
 | **Tools & Models** | Error rate and p95 per tool; per model TTFT, tokens/s, truncation rate, cache hit rate and spend |
 | **SLOs** | Success rate, Apdex, latency and cost objectives with error budgets and burn rates |
-| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export |
+| **Governance** | Aegis policy enforcement per task: denials by rule, budget stops, kill-switch revocations, unused grants, policy export, and what a policy change would refuse |
 | **Process Review** | Scores how well the agent works (efficiency, focus, reliability, verification, context, autonomy, compliance) and suggests fixes |
 | **Analytics & Compare** | Any metric by any dimension; compare models, prompts or releases side by side |
 
@@ -114,7 +145,7 @@ That call sets up four flows:
 1. **Every Aegis decision appears in its task.** Denials carry the rule id, and the Aegis audit log carries the task's run id so the two logs join.
 2. **Model calls are charged to the Aegis budget before they're sent.** When the budget runs out, the call is refused and never made.
 3. **A watchdog revokes the grant** of an agent that keeps probing a boundary, for example after a prompt injection.
-4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour, which `aegis ratify` and `aegis drift` then verify.
+4. **`agentdynamics policy export` writes a tighter, least-privilege policy** from observed behaviour. `aegis ratify` and `aegis drift` verify its declaration; `agentdynamics policy check` replays the traffic that actually ran through it and reports how much it would refuse, so a policy change can be gated in CI.
 
 The console's **Governance** page shows denials, budget stops, revocations, unused grants and budget headroom per policy version. See [docs/GOVERNANCE.md](docs/GOVERNANCE.md).
 
@@ -161,12 +192,13 @@ See [deploy/](deploy/) and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 ## Development
 
 ```bash
-python -m unittest discover tests   # 28 tests: real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
-python -m agentdynamics --data .demo-data --claude-root "" serve --port 8790 &
-python examples/langgraph_style_app.py 60          # LangGraph-style demo traffic
-python examples/otel_multiagent.py 60 http://127.0.0.1:8790
+pip install -e ".[test,toml]" && pip install "aegis-kernel>=0.4.0"
+python -m unittest discover tests   # real LangSmith/Anthropic/OpenAI SDKs, Aegis kernel, OTLP protobuf, pull connectors
 ```
 
+Demo traffic is under **What it looks like** above. [CLAUDE.md](CLAUDE.md) has the architecture and the
+invariants to work within; [ROADMAP.md](ROADMAP.md) is what is planned and what is deliberately not.
+
 ## License
 
 Apache 2.0. See [LICENSE](LICENSE).

{agentdynamics-0.4.1 → agentdynamics-0.6.0}/agentdynamics/__init__.py

@@ -5,8 +5,8 @@
 
 See agentdynamics.autotrace for @trace / @tool / span.
 """
-__version__ = "0.4.1"
+__version__ = "0.6.0"
 
-from .autotrace import flush, init, llm_call, record_llm, span, tool, trace  # noqa: E402,F401
+from .autotrace import flush, init, llm_call, outcome, record_llm, span, tool, trace  # noqa: E402,F401
 
-__all__ = ["init", "trace", "tool", "span", "llm_call", "record_llm", "flush", "__version__"]
+__all__ = ["init", "trace", "tool", "span", "llm_call", "record_llm", "outcome", "flush", "__version__"]

{agentdynamics-0.4.1 → agentdynamics-0.6.0}/agentdynamics/__main__.py

@@ -191,6 +191,22 @@ def cmd_policy(a, eng):
         for r in g["by_rule"][:8]:
             print(f"  {r['n']:>5}  {r['rule']}")
         return 0
+    if a.action == "check":
+        if not a.candidate:
+            print("policy check needs --candidate <policy.yaml>", file=sys.stderr)
+            return 2
+        try:
+            from aegis import dump_policy, load_policy
+        except ImportError:
+            print("policy check needs aegis-kernel: pip install aegis-kernel", file=sys.stderr)
+            return 2
+        q["candidate_doc"] = dump_policy(load_policy(a.candidate))
+        res = api.check_policy(q)
+        if res.get("error"):
+            print(res["error"], file=sys.stderr)
+            return 1
+        print(f"{a.candidate} against {res['tasks']} governed task(s) in scope:")
+        return _report_coverage(res["coverage"], a.max_denied_fraction, sys.stdout)
     res = api.export_policy(q)
     if res.get("error"):
         print(res["error"], file=sys.stderr)
@@ -203,16 +219,41 @@ def cmd_policy(a, eng):
             print(f"  - {c}", file=sys.stderr)
     else:
         print(res["yaml"])
-    # Replaying the observed calls is the only check that catches a policy which is strictly
-    # narrower than its base, ratifies cleanly, and still refuses everything in production.
-    if res.get("regressions"):
-        print("", file=sys.stderr)
-        print(f"WARNING: this policy would deny {len(res['regressions'])} kind(s) of call that "
-              f"the observed traffic made successfully:", file=sys.stderr)
-        for r in res["regressions"][:10]:
-            print(f"  - {r['tool']}({r['arg']}={r['value']!r}) -> {r['rule']}", file=sys.stderr)
-        print("  Review before deploying; `aegis ratify` and `aegis drift` cannot see this.",
-              file=sys.stderr)
+    if res.get("coverage") is None:
+        return 0
+    print("", file=sys.stderr)
+    print("Against the traffic it was generated from:", file=sys.stderr)
+    return _report_coverage(res["coverage"], a.max_denied_fraction, sys.stderr)
+
+
+def _report_coverage(cov, max_fraction, out):
+    """Print how much observed traffic a candidate would refuse; non-zero exit past the threshold.
+
+    This is the one check that catches a policy which is strictly narrower than its base, ratifies
+    cleanly and shows no drift, and still refuses production traffic -- `aegis ratify` and
+    `aegis drift` compare declarations and never see a single call.
+    """
+    w = lambda *x: print(*x, file=out)  # noqa: E731
+    if not cov["calls"]:
+        w("  no governed calls in scope to check against")
+        return 0
+    w(f"  would deny {cov['denied']} of {cov['calls']} calls that were allowed ({cov['denied_fraction']:.1%})")
+    rows = [r for r in cov["by_tool"] if r["denied"]]
+    if rows:
+        width = max(len(r["tool"]) for r in rows)
+        for r in rows:
+            why = ", ".join(f"{k} x{n}" for k, n in sorted(r["rules"].items(), key=lambda kv: -kv[1]))
+            w(f"    {r['tool']:<{width}}  {r['denied']:>5} of {r['calls']:<5} {r['denied_fraction']:>6.1%}  {why}")
+        for ex in cov["examples"][:5]:
+            if ex.get("arg"):
+                w(f"      e.g. {ex['tool']}({ex['arg']}={ex['value']!r}) -> {ex['rule']}")
+            else:            # a missing argument or a dropped tool: Aegis's own reason says which
+                w(f"      e.g. {ex['tool']} -> {ex['rule']}: {ex.get('reason') or ''}")
+    if cov["args_unrecorded"]:
+        w(f"  {cov['args_unrecorded']} call(s) had no recorded arguments (content capture off); "
+          f"only their tool grant was checked")
+    if cov["denied_fraction"] > max_fraction:
+        w(f"  FAIL: {cov['denied_fraction']:.1%} exceeds --max-denied-fraction {max_fraction:.1%}")
         return 1
     return 0
 
@@ -257,7 +298,9 @@ def main(argv=None):
     k.add_argument("--role", choices=["ingest", "read", "admin"], default="ingest")
     k.add_argument("--name", default=None)
     po = sub.add_parser("policy", help="Aegis policy from observed behaviour (observe -> govern)")
-    po.add_argument("action", choices=["export", "report"])
+    po.add_argument("action", choices=["export", "report", "check"],
+                    help="export: generate a tightened policy; report: governance summary; "
+                         "check: judge --candidate against recorded traffic")
     po.add_argument("--workflow")
     po.add_argument("--project")
     po.add_argument("--environment")
@@ -266,6 +309,10 @@ def main(argv=None):
     po.add_argument("--base", help="Aegis policy file to tighten (needs aegis-kernel installed)")
     po.add_argument("--headroom", type=float, default=1.5)
     po.add_argument("--out", help="write the YAML here (default: stdout)")
+    po.add_argument("--candidate", help="policy file to judge (check)")
+    po.add_argument("--max-denied-fraction", type=float, default=0.0,
+                    help="fail if the policy would refuse more than this share of observed, allowed calls "
+                         "(export, check; default 0: any refusal fails)")
     sub.add_parser("ingest", help="scan sources once and rebuild the database")
     rp = sub.add_parser("report", help="print a text summary")
     rp.add_argument("--project")

{agentdynamics-0.4.1 → agentdynamics-0.6.0}/agentdynamics/analysis.py

@@ -9,6 +9,7 @@ AppDynamics -> AgentDynamics mapping implemented here:
   Code-level hotspots   -> waste findings (redundant reads, loops, error streaks)
 """
 import math
+import operator
 import re
 import statistics
 import time
@@ -40,23 +41,34 @@ CORRECTION = re.compile(
 
 
 def classify_task(prompt_kind, text):
+    return classify_task_detail(prompt_kind, text)[0]
+
+
+def classify_task_detail(prompt_kind, text):
+    """(task_type, how it was decided, what matched).
+
+    how: "prompt kind" when the prompt says what it is (a slash command, a scheduled job, a subagent's
+    brief); "follow-up" for a short "continue"/"yes" that carries on the previous task; "keywords" when an
+    intent rule matched, with the matched text; "unmatched" when nothing did. Keyword typing is a guess
+    about what someone meant, so it says so -- the same way outcome_source separates graded from inferred.
+    """
     if prompt_kind == "scheduled":
-        return "scheduled job"
+        return "scheduled job", "prompt kind", None
     if prompt_kind == "delegated":
-        return "subagent"
+        return "subagent", "prompt kind", None
     t = (text or "").strip()
     low = t.lower()
     if prompt_kind == "command":
-        return "slash command"
+        return "slash command", "prompt kind", None
     if len(t) < 60 and FOLLOWUP.search(t):
-        return "follow-up"
+        return "follow-up", "follow-up", None
     # The earliest intent keyword wins: "Create X, then test it" is a build, not a testing task.
     best = None
     for rank, (name, pat) in enumerate(TYPE_RULES):
         m = re.search(pat, low[:600])
         if m and (best is None or (m.start(), rank) < best[0]):
-            best = ((m.start(), rank), name)
-    return best[1] if best else "other"
+            best = ((m.start(), rank), name, m.group(0).strip())
+    return (best[1], "keywords", best[2][:40]) if best else ("other", "unmatched", None)
 
 
 # ---------------------------------------------------------------- helpers
@@ -146,9 +158,10 @@ def task_metrics(run, idx, seg):
     in_side = t["input_tokens"] + t["cache_read"] + t["cache_write"]
     t["cache_hit"] = round(t["cache_read"] / in_side, 3) if in_side else None
     t["tool_error_rate"] = round(t["tool_errors"] / len(tools), 3) if tools else 0
-    t["task_type"] = classify_task(t["prompt_kind"], t["prompt"])
+    t["task_type"], t["task_type_source"], t["task_type_match"] = classify_task_detail(t["prompt_kind"], t["prompt"])
     if run.get("workflow") and run.get("source") != "claude-code":
-        t["task_type"] = run["workflow"]  # traced apps: the entry point names the business transaction
+        # traced apps: the entry point names the business transaction, which beats any guess
+        t["task_type"], t["task_type_source"], t["task_type_match"] = run["workflow"], "workflow", None
 
     # --- phase attribution: split each LLM call's cost across the tool calls it issued
     phase_calls = Counter(s["phase"] for s in tools)
@@ -301,6 +314,9 @@ def governance_metrics(t, run, steps, tools):
     t["repeated_denials"] = best
 
 
+OUTCOMES = ("completed", "failed", "rework", "interrupted")   # what a grade may state (store.OUTCOMES)
+FEEDBACK_PASS = 0.5                                             # a feedback score at or above this is a pass
+
 TRUNCATION = {"max_tokens", "length", "max_output_tokens", "MAX_TOKENS"}
 REFUSAL = {"refusal", "content_filter", "SAFETY", "safety", "blocked"}
 RATE_HINTS = ("429", "rate limit", "rate_limit", "overloaded", "529", "too many requests", "quota")
@@ -314,6 +330,13 @@ def flow_metrics(t, run, steps, llm, tools):
     t["framework"] = run.get("framework") or run.get("source")
     t["workflow"] = run.get("workflow") or (t["task_type"] if run.get("source") == "claude-code" else None)
     t["steps_total"] = len(llm) + len(tools)
+    # model calls whose cache accounting rests on a guess (collectors/spans.uncached_input); None is
+    # a source that is exact by construction, like the SDK, so only an explicit False counts
+    t["tokens_unverified"] = sum(1 for s in llm if s.get("tokens_verified") is False)
+    # agentdynamics.outcome() records a notice; the last one in the task wins. Kept on a transient
+    # key -- finalize decides precedence, and write_analysis persists only TASK_COLS.
+    graded = [s for s in notices if s.get("name") == "outcome" and s.get("outcome") in OUTCOMES]
+    t["_sdk_grade"] = {"outcome": graded[-1]["outcome"], "reason": graded[-1].get("text")} if graded else None
     t["llm_errors"] = sum(1 for s in llm if s.get("is_error"))
     t["truncations"] = sum(1 for s in llm if s.get("stop_reason") in TRUNCATION)
     t["refusals"] = sum(1 for s in llm if s.get("stop_reason") in REFUSAL)
@@ -538,6 +561,7 @@ def _outcomes_claude(ts, now):
         # "continue" / "yes" carries on the previous request, so it belongs to that task type
         if t["task_type"] == "follow-up" and i > 0 and ts[i - 1]["task_type"] not in ("follow-up",):
             t["task_type"] = ts[i - 1]["task_type"]
+            t["task_type_source"], t["task_type_match"] = "follow-up", None   # inherited, not re-guessed
             if t["source"] == "claude-code":
                 t["workflow"] = t["task_type"]
         nxt = next((x for x in ts[i + 1:] if x["prompt_kind"] in ("human", "command")), None)
@@ -573,8 +597,96 @@ def _outcome_trace(t, run, now):
     t["root_error"] = run.get("root_error")
 
 
-def finalize(runs, tasks_by_run, rules=None, now=None):
-    """Cross-run analysis: outcomes, subagent roll-up, baselines, scores, events."""
+def _apply_grades(tasks, grades):
+    """Settle each outcome by the strongest evidence available, and record which one it was.
+
+    stated after the fact (API)  >  stated in the run (agentdynamics.outcome)  >  recorded feedback
+    >  inference from signals.
+
+    Inference is a guess built from errors, interrupts and corrections. Apdex, success rate and the
+    process score all inherit it, so anything that states the outcome outright must win, and the
+    console has to be able to say how much of a success rate is guessed. Idempotent: the cached
+    per-run tasks are re-finalized on every refresh, so nothing here may consume its input.
+    """
+    for t in tasks:
+        api, sdk = grades.get(t["id"]), t.get("_sdk_grade")
+        if api:
+            t["outcome"], t["outcome_source"] = api["outcome"], "graded"
+            t["outcome_reason"] = api.get("reason") or f"graded by {api.get('graded_by') or 'api'}"
+        elif sdk:
+            t["outcome"], t["outcome_source"] = sdk["outcome"], "graded"
+            t["outcome_reason"] = sdk.get("reason") or "graded in the run"
+        elif t.get("feedback_score") is not None:
+            ok = t["feedback_score"] >= FEEDBACK_PASS
+            t["outcome"] = "completed" if ok else "rework"
+            t["outcome_source"], t["outcome_reason"] = "feedback", f"feedback score {t['feedback_score']:g}"
+        else:
+            t["outcome_source"], t["outcome_reason"] = "inferred", None
+
+
+# Everything finalize writes onto a task *before* scoring, for tasks whose run did not change. If one of
+# these moves, the task's row must be rewritten and re-scored; if none did and its baseline didn't either,
+# the previous score and events still hold. A field added to the settle phase below must be added here,
+# or incremental refreshes will serve it stale -- test_incremental.py compares every column against a
+# full rebuild to catch exactly that.
+SETTLED_FIELDS = ("outcome", "outcome_source", "outcome_reason", "next_prompt", "rework", "root_error",
+                  "task_type", "task_type_source", "task_type_match", "workflow",
+                  "subagent_cost", "subagents", "parent_task_id")
+
+
+_settled = operator.itemgetter(*SETTLED_FIELDS)     # built in C: this runs for every task on every refresh
+
+
+def _settled_values(t):
+    try:
+        return _settled(t)
+    except KeyError:          # a field some sources never set (Claude Code tasks carry no root_error)
+        return tuple(t.get(f) for f in SETTLED_FIELDS)
+
+
+class ScoreCache:
+    """What finalize needs to skip unchanged tasks on the next refresh.
+
+    sig[task_id]     the settled fields plus the two baseline numbers scoring reads
+    events[task_id]  the health events that signature produced
+    changed          task ids scored this time (their rows must be written)
+    """
+
+    def __init__(self):
+        self.sig, self.events, self.changed = {}, {}, set()
+
+
+BASELINE_EXACT_UP_TO = 20      # below this many tasks, a type's baseline uses all of them
+BASELINE_GROWTH = 1.05          # above it, only when the count has grown 5% since the last step
+
+
+def baseline_sample_size(n):
+    """How many of a type's n tasks its baseline is computed from.
+
+    Every task is scored against its type's baseline, so a baseline that moved with each arrival would
+    make every refresh re-score the whole type -- no cheaper than rebuilding. Instead the baseline is
+    computed from the type's *earliest* M tasks (by start, then id), and M only advances in 5% steps: a
+    type re-scores once per 5% of growth, about 20 re-scores per new task however large the store, and
+    never oscillates. It depends on the data alone, so a full rebuild computes the same baseline. It
+    leaves out at most the newest 5% of a type's history, which is noise for a "what is normal" figure.
+    """
+    if n <= BASELINE_EXACT_UP_TO:
+        return n
+    m = BASELINE_EXACT_UP_TO
+    while True:
+        nxt = max(m + 1, int(m * BASELINE_GROWTH))
+        if nxt > n:
+            return m
+        m = nxt
+
+
+def finalize(runs, tasks_by_run, rules=None, now=None, grades=None, cache=None, dirty=()):
+    """Cross-run analysis: outcomes, subagent roll-up, baselines, scores, events.
+
+    With a `cache` (a ScoreCache kept between calls), tasks from runs not in `dirty` whose settled
+    fields and baseline are unchanged keep their previous score and events instead of being re-scored;
+    `cache.changed` then lists the task ids that were. The result is the same as without a cache.
+    """
     rules = rules or DEFAULT_RULES
     now = now or time.time()
     all_tasks = [t for r in runs for t in tasks_by_run[r["id"]]]
@@ -587,11 +699,19 @@ def finalize(runs, tasks_by_run, rules=None, now=None):
         if run["source"] == "claude-code" or not run.get("workflow"):
             _outcomes_claude(ts, now)
         else:
+            # Derived afresh every pass, never carried over. The engine keeps an unchanged run's task
+            # objects between refreshes, and linking below only *sets* these on tasks that have a
+            # successor: a task whose follow-up was deleted, edited or moved to another thread kept the
+            # old values -- and so its "rework" outcome -- until a full rebuild. test_incremental found it.
+            for t in ts:
+                t["next_prompt"], t["rework"] = None, 0
             if run.get("thread_id"):
                 threads[(run["source"], run["thread_id"])].extend(ts)
     # traced conversations: the next trace in the same thread plays the role of "your next message"
     for group in threads.values():
-        group.sort(key=lambda x: x["started"] or 0)
+        # ties on start time break by id, not by whatever order the runs happen to be held in -- which
+        # differs between a long-running engine and a fresh rebuild
+        group.sort(key=lambda x: (x["started"] or 0, x["id"]))
         for a, b in zip(group, group[1:]):
             a["next_prompt"] = b["prompt"][:300]
             a["rework"] = 1 if CORRECTION.search(b["prompt"][:200]) else 0
@@ -599,6 +719,8 @@ def finalize(runs, tasks_by_run, rules=None, now=None):
         if not (run["source"] == "claude-code" or not run.get("workflow")):
             for t in tasks_by_run[run["id"]]:
                 _outcome_trace(t, run, now)
+    # before the roll-up, so baselines, scores and health events all see the settled outcome
+    _apply_grades(all_tasks, grades or {})
 
     # subagent roll-up: link child runs to the parent task that spawned them
     task_by_id = {t["id"]: t for t in all_tasks}
@@ -642,10 +764,12 @@ def finalize(runs, tasks_by_run, rules=None, now=None):
     for k, g in groups.items():
         if len(g) < 3 and k != "__all__":
             continue
+        n = len(g)
+        g = sorted(g, key=lambda t: (t["started"] or 0, t["id"]))[:baseline_sample_size(n)]
         costs = [t["cost"] + t["subagent_cost"] for t in g]
         durs = [t["duration_s"] for t in g]
         baselines[k] = {
-            "n": len(g),
+            "n": n, "sample": len(g),
             "cost_p50": pct(costs, 0.5), "cost_p90": pct(costs, 0.9),
             "duration_p50": pct(durs, 0.5), "duration_p90": pct(durs, 0.9),
             "tokens_p50": pct([t["total_tokens"] for t in g], 0.5),
@@ -654,13 +778,31 @@ def finalize(runs, tasks_by_run, rules=None, now=None):
         }
 
     events = []
+    dirty = set(dirty)
+    if cache is not None:
+        cache.changed = set()
+        live = {t["id"] for t in all_tasks}
+        for tid in [k for k in cache.sig if k not in live]:     # tasks that no longer exist
+            cache.sig.pop(tid, None)
+            cache.events.pop(tid, None)
     for t in all_tasks:
+        if cache is not None:
+            b = baselines.get(t["task_type"]) or baselines.get("__all__") or {}
+            sig = (_settled_values(t), b.get("cost_p50"), b.get("duration_p50"))
+            if t["run_id"] not in dirty and cache.sig.get(t["id"]) == sig:
+                events.extend(cache.events[t["id"]])            # nothing it depends on moved
+                continue
         t["failed"] = 1 if t.get("outcome") == "failed" else 0
         if t["llm_calls"] == 0 and t["tool_calls"] == 0:
             t.update({"scores": {}, "score": None, "apdex": None, "cost_vs_baseline": None, "duration_vs_baseline": None})
-            continue
-        score_task(t, baselines)
-        events.extend(evaluate_rules(t, rules))
+            ev = []
+        else:
+            score_task(t, baselines)
+            ev = evaluate_rules(t, rules)
+        events.extend(ev)
+        if cache is not None:
+            cache.sig[t["id"]], cache.events[t["id"]] = sig, ev
+            cache.changed.add(t["id"])
     return all_tasks, baselines, events
 
 
@@ -673,7 +815,11 @@ def analyze(runs, rules=None, now=None):
 
 def process_insights(tasks):
     """Aggregate findings meant to help a human assess how the agent works."""
-    main = [t for t in tasks if not t["is_subagent"] and t["llm_calls"] > 0]
+    # One canonical order. The engine holds tasks in whatever order runs arrived, which differs between
+    # a long-running process and a fresh rebuild; float sums round differently in a different order,
+    # and example lists are cut to their first few. Sorting makes the result depend on the data alone.
+    main = sorted((t for t in tasks if not t["is_subagent"] and t["llm_calls"] > 0),
+                  key=lambda t: (t["started"] or 0, t["id"]))
     if not main:
         return []
     out = []

{agentdynamics-0.4.1 → agentdynamics-0.6.0}/agentdynamics/autotrace.py

@@ -215,6 +215,10 @@ class trace:
         """Attach a user/eval score (0..1) to this task."""
         self.run.feedback.append({"key": key, "score": score})
 
+    def outcome(self, result, reason=None):
+        """State how this task ended. See agentdynamics.outcome."""
+        _record_outcome(self.run, result, reason)
+
     # decorator
     def __call__(self, *args, **kwargs):
         if self._fn is None:  # used as @trace("name")
@@ -402,6 +406,38 @@ class llm_call:
         return False
 
 
+OUTCOMES = ("completed", "failed", "rework", "interrupted")
+
+
+def _record_outcome(run, result, reason):
+    if result not in OUTCOMES:
+        # telemetry never breaks the agent: a typo is a warning, not an exception in their code path
+        _warn(f"outcome:{result!r}", f"outcome({result!r}) ignored: must be one of {', '.join(OUTCOMES)}")
+        return
+    run.add({"kind": "notice", "name": "outcome", "outcome": result, "ts": time.time(),
+             "text": _clip(str(reason), 500) if reason is not None else None})
+
+
+def outcome(result, reason=None):
+    """State how the current task ended, instead of leaving it to inference.
+
+        @agentdynamics.trace
+        def handle(ticket):
+            ...
+            if not resolved:
+                agentdynamics.outcome("failed", reason="customer escalated")
+
+    `result` is completed | failed | rework | interrupted. The last call in a task wins, and it
+    overrides both inference and recorded feedback; only a grade set later through the API ranks
+    above it. Outside a traced task there is nothing to grade, so it warns once and does nothing.
+    """
+    run = _current.get()
+    if run is None:
+        _warn("outcome:no-run", "agentdynamics.outcome() called outside a traced task; ignored")
+        return
+    _record_outcome(run, result, reason)
+
+
 def record_llm(model, input_tokens=0, output_tokens=0, cache_read=0, cache_write=0, stop_reason=None,
                start=None, end=None, text="", provider="custom"):
     """Record a model call after the fact (no gating). Prefer `llm_call` when you can wrap the call."""