agentdebugx 0.2.5__tar.gz → 0.2.6__tar.gz

{agentdebugx-0.2.5 → agentdebugx-0.2.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentdebugx
-Version: 0.2.5
+Version: 0.2.6
 Summary: Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`.
 License: MIT
 License-File: LICENSE

{agentdebugx-0.2.5 → agentdebugx-0.2.6}/docs/23_status_v0_2.md

@@ -82,6 +82,26 @@ The audit found one real bug and a handful of test gaps:
 5. **`recovery.ReflexionSuggestion`** had only an indirect test from DeepDebug
    examples; now has direct happy + empty tests.
 
+## 3.7 Judge hardening (0.2.6)
+
+A v0.2.5 Who&When 5-trace live run had `llm_judge_root.agent_match=0.00`
+because the judge truncated mid-array on long multi-agent debate
+transcripts. Three changes in 0.2.6 lifted that to **0.40** on the same
+sample (same model, same traces):
+
+1. `LLMJudgeAnalyzer.max_tokens` default **4096 → 8192** — leaves room for
+   thinking-model reasoning tokens before the JSON object starts.
+2. `LLMJudgeAnalyzer.max_findings_per_chunk` parameter (default 6) — the
+   system prompt now asks the model to cap its findings array, forcing it
+   to close the JSON even when many candidates are visible.
+3. System prompt now has explicit "CRITICAL OUTPUT RULES" — output ONLY
+   JSON, no markdown fences, no newlines in string values, complete the
+   array.
+
+Numbers: see [docs/benchmarks/who_when_v0_2_6_leaderboard.md](./benchmarks/who_when_v0_2_6_leaderboard.md).
+Same trick works for `BinarySearchAttributor` (shipped in 0.2.4) — apply
+to remaining LLM-using analyzers as more thinking models surface this.
+
 ## 3.6 Real-usage E2E (live Gemini)
 
 Beyond unit tests, `scripts/e2e_real_usage.py` builds three realistic failing
@@ -148,6 +168,16 @@ Before v0.3 ships, this doc should record green checkmarks for:
       into `AgentEvent`s. Conformance test mocks the bus and verifies
       every documented event mapping plus the version-skew degradation
       path. `examples/crewai_demo.py` shows a working two-agent crew.
-- [ ] HuggingFace Hub round-trip live test (gated on `HF_TOKEN`).
-- [ ] Bench harness extended with one published-benchmark loader (Who&When
-      is the obvious first target — we already cite its method).
+- [x] **HuggingFace Hub round-trip live test** — shipped in 0.2.6 as
+      `tests/test_hub_huggingface_live.py`. Gated on `HF_TOKEN` +
+      `AGENTDEBUG_HF_LIVE=1` so it never runs in default CI. Creates the
+      dataset repo if missing, pushes a bundle, lists, pulls back, verifies
+      the trajectory round-trips bit-for-bit. Live-validated against
+      `KunlunZhu/agentdebugx-live-test`.
+- [x] **Bench harness with Who&When loader** — `experiments/prepare_who_when.py`
+      ingests 184 Algorithm-Generated + Hand-Crafted traces (4092 events) and
+      stores labels separately. `experiments/run_who_when_eval.py` runs all
+      4 attributors + DeepDebug against gold labels; reports agent_match,
+      exact_step, near_step. Live-Gemini 5-trace validation captured at
+      [docs/benchmarks/who_when_v0_2_6_leaderboard.md](./benchmarks/who_when_v0_2_6_leaderboard.md).
+      Headline 184-trace run deferred (~6h / ~$5-10 on a frontier model).

agentdebugx-0.2.6/docs/benchmarks/who_when_v0_2_6_leaderboard.md

@@ -0,0 +1,74 @@
+# Who&When — 5-trace Live Leaderboard (v0.2.6, gemini-3-flash)
+
+Tiny validation sample drawn from `data/who_when/processed/labels.jsonl`
+(first 5 algorithm-generated traces). **Not a publishable benchmark** — the
+full benchmark requires the 184-trace dataset + a frontier model and is
+deferred for cost reasons. This run exists to verify the analysis stack
+produces sensible-shaped numbers and to surface regressions early.
+
+## Aggregate (per attribution method)
+
+| Method | agent_match | exact_step | near_step | both_near | DeepDebug rounds |
+|---|---:|---:|---:|---:|---:|
+| `heuristic` (rule baseline) | 0.20 | 0.00 | 0.20 | 0.20 | n/a |
+| `llm_judge_root` (judge's root_cause field) | **0.40** | 0.00 | **0.20** | **0.20** | n/a |
+| `all_at_once` (Who&When method 1) | 0.20 | 0.00 | 0.00 | 0.00 | n/a |
+| `step_by_step` (Who&When method 2) | **0.40** | 0.00 | **0.20** | **0.20** | n/a |
+| `deep_debug_root` (DeepDebug refined root) | 0.20 | 0.00 | 0.20 | 0.00 | 6 / trace |
+
+## What changed in 0.2.6 vs 0.2.5
+
+Same 5 traces, same model:
+
+| Method | 0.2.5 agent_match | 0.2.6 agent_match | Δ |
+|---|---:|---:|---:|
+| `heuristic` | 0.20 | 0.20 | — |
+| `llm_judge_root` | 0.00 | **0.40** | +0.40 |
+| `all_at_once` | 0.00 | 0.20 | +0.20 |
+| `step_by_step` | 0.00 | **0.40** | +0.40 |
+
+The driver was the v0.2.6 judge prompt hardening: `max_tokens` default
+4096 → 8192, an explicit `max_findings_per_chunk=6` cap surfaced through
+the system prompt, and a "CRITICAL OUTPUT RULES" header (output ONLY JSON,
+no markdown, no newlines in strings, complete the array). Before the
+hardening, the judge truncated mid-array on Who&When debate transcripts
+and returned no findings; after, the structured root_cause is populated.
+
+## Honest caveats
+
+* n=5; per-method standard error is ±0.22 — these absolute numbers should
+  not be over-interpreted. The 0.4 vs 0.0 jump for two methods is the
+  signal worth reporting; everything else is noise.
+* `deep_debug_root` underperformed `step_by_step` on this sample. The
+  refine round on 7-event traces tends to converge to the *visible*
+  failure rather than the *causal* root (a known Who&When difficulty —
+  manifestation vs root cause).
+* No method beats `near_step=0.20` on this sample. Step-localization
+  remains hard, matching the published Who&When ceiling (~14% step on
+  127 traces with frontier models).
+
+## Reproducing
+
+```bash
+# Prepare data (once)
+PYTHONPATH=src python experiments/prepare_who_when.py
+
+# Set live LLM creds (any OpenAI-compatible endpoint works)
+export AGENTDEBUG_LLM_BASE_URL=...
+export AGENTDEBUG_LLM_API_KEY=...
+export AGENTDEBUG_LLM_MODEL=gemini-3-flash
+
+# Without DeepDebug (~1 min)
+PYTHONPATH=src python experiments/run_who_when_eval.py \
+  --limit 5 --live-openai \
+  --out-dir experiments/runs/who_when_eval_subset
+
+# With DeepDebug (~5 min)
+PYTHONPATH=src python experiments/run_who_when_eval.py \
+  --limit 5 --live-openai --deep \
+  --out-dir experiments/runs/who_when_eval_subset_deep
+```
+
+The headline benchmark (184 traces × 5 methods × DeepDebug) would take
+~6 hours and ~$5-10 in API cost on a frontier model. Run it once before
+paper submission; do not run on every iteration.

{agentdebugx-0.2.5 → agentdebugx-0.2.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "agentdebugx"
-version = "0.2.5"
+version = "0.2.6"
 description = "Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`."
 authors = ["ULab @ UIUC <ulab@illinois.edu>"]
 license = "MIT"

{agentdebugx-0.2.5 → agentdebugx-0.2.6}/src/agentdebug/__init__.py

@@ -96,4 +96,4 @@ __all__ = [
     'get_failure_mode',
 ]
 
-__version__ = '0.2.5'
+__version__ = '0.2.6'

{agentdebugx-0.2.5 → agentdebugx-0.2.6}/src/agentdebug/judges.py

@@ -39,21 +39,18 @@ the allowed failure mode codes. Be conservative — only flag steps where the
 evidence in the event payload supports the label. If the trajectory contains no
 failure, return an empty findings list.
 
-Respond ONLY with a JSON object matching this schema (no prose, no markdown):
+CRITICAL OUTPUT RULES (these maximize the chance your reply parses):
+1. Output ONLY a JSON object. No prose before/after. No markdown fences.
+2. Cap the findings array at {max_findings} entries — pick the most important.
+3. Keep each "evidence" entry under 120 characters; keep each "rationale" /
+   "summary" under 200 characters.
+4. Do NOT include newlines inside string values.
+5. Emit the JSON object COMPLETE — never stop mid-key or mid-array.
 
-{
-  "findings": [
-    {
-      "event_id": "<event_id from the input>",
-      "step_index": <int or null>,
-      "agent_name": "<agent_name from the input>",
-      "failure_mode_id": "<one of the allowed codes>",
-      "confidence": <float between 0 and 1>,
-      "evidence": ["<short quote or summary of the supporting payload>"]
-    }
-  ],
-  "summary": "<one-sentence diagnosis or 'No failure detected.'>"
-}
+Schema (compact — fields in this order):
+{{"findings":[{{"event_id":"...", "step_index":N|null, "agent_name":"...",
+  "failure_mode_id":"...", "confidence":0..1, "evidence":["..."]}}, ...],
+  "summary":"<short>"}}
 """
 
 
@@ -66,15 +63,21 @@ class LLMJudgeAnalyzer:
         *,
         max_events_per_call: int = 80,
         max_evidence_chars: int = 300,
-        max_tokens: int = 4096,
+        max_tokens: int = 8192,
+        max_findings_per_chunk: int = 6,
     ) -> None:
         self.llm = llm
         self.max_events_per_call = max_events_per_call
         self.max_evidence_chars = max_evidence_chars
         # NOTE: thinking models (Gemini 2.x/3.x, o-series) spend a substantial
         # fraction of `max_tokens` on reasoning tokens before any text is
-        # emitted. 4096 is the safe default; bump higher for long traces.
+        # emitted. 8192 is the safe default after the v0.2.6 Who&When debate-
+        # trace observation that 4096 truncated mid-array on long traces.
         self.max_tokens = max_tokens
+        # The system prompt asks the model to cap its findings array so the
+        # JSON closes even when many candidate failures exist. Reuse the prompt
+        # placeholder for this cap.
+        self.max_findings_per_chunk = max_findings_per_chunk
 
     def analyze(self, trajectory: AgentTrajectory) -> DiagnosticReport:
         events = trajectory.events
@@ -121,8 +124,11 @@ class LLMJudgeAnalyzer:
         self, trajectory: AgentTrajectory, chunk: List[AgentEvent]
     ) -> tuple[List[FailureFinding], str]:
         user = self._render_user_prompt(trajectory, chunk)
+        # Inject the max_findings cap into the system prompt at format time so
+        # we can tune it per-call without forking the prompt.
+        system = _SYSTEM_PROMPT.format(max_findings=self.max_findings_per_chunk)
         messages = [
-            {'role': 'system', 'content': _SYSTEM_PROMPT},
+            {'role': 'system', 'content': system},
             {'role': 'user', 'content': user},
         ]
         result = self.llm.complete(messages=messages, max_tokens=self.max_tokens)

{agentdebugx-0.2.5 → agentdebugx-0.2.6}/src/agentdebug/ui/server.py

@@ -211,8 +211,10 @@ _INDEX_HTML = """<!doctype html>
   .button {
     border:1px solid #373b3a; border-radius:8px; background:#1a1c1c; color:var(--fg);
     height:32px; padding:0 11px; font-size:12px; display:inline-flex;
-    align-items:center; gap:7px;
+    align-items:center; justify-content:center; gap:7px; cursor:pointer;
+    font-family:inherit; white-space:nowrap;
   }
+  .button:hover { border-color:#4b5250; background:#202323; }
   .button.primary { border-color:#356568; color:#d8fdff; background:#173033; }
   .content { padding:22px; max-width:1440px; margin:0 auto; }
   .hero {
@@ -330,6 +332,18 @@ _INDEX_HTML = """<!doctype html>
     .topbar { position:static; }
     .trace-legend, .trace-pair { grid-template-columns:1fr; }
   }
+  @media (max-width: 640px) {
+    .topbar { display:grid; grid-template-columns:1fr; align-items:start; padding:14px 16px; }
+    .top-actions { width:100%; display:grid; grid-template-columns:repeat(3,minmax(0,1fr)); }
+    .button { width:100%; min-width:0; padding:0 8px; overflow:hidden; text-overflow:ellipsis; }
+    .content { padding:22px 16px; }
+    h1 { font-size:27px; line-height:1.1; }
+    .stats { grid-template-columns:repeat(2,minmax(0,1fr)); }
+    .root-grid { grid-template-columns:1fr; }
+    .event { grid-template-columns:46px minmax(0,1fr); padding:10px; }
+    .step-index { width:38px; height:38px; }
+    .event-grid { grid-template-columns:1fr; }
+  }
 </style>
 </head>
 <body>
@@ -358,9 +372,9 @@ _INDEX_HTML = """<!doctype html>
         <div class="brand-sub" id="trace-count">Loading traces</div>
       </div>
       <div class="top-actions">
-        <span class="button">Analyze</span>
-        <span class="button">Export Bundle</span>
-        <span class="button primary">Open Error Hub</span>
+        <button class="button" id="analyze-btn" type="button">Analyze</button>
+        <button class="button" id="export-btn" type="button">Bundle</button>
+        <button class="button primary" id="hub-btn" type="button">Hub</button>
       </div>
     </div>
     <div class="content" id="detail">
@@ -370,6 +384,8 @@ _INDEX_HTML = """<!doctype html>
 </div>
 <script>
 const BOOTSTRAP = __BOOTSTRAP_JSON__;
+let CURRENT_TRACE_ID = null;
+let CURRENT_TRACE_DATA = null;
 async function api(path) {
   const r = await fetch(path);
   if (!r.ok) throw new Error('HTTP ' + r.status);
@@ -430,9 +446,11 @@ function renderTraceList(traceIds, selectedId) {
 async function selectTrace(tid, li) {
   document.querySelectorAll('.run').forEach(el => el.classList.remove('active'));
   li.classList.add('active');
+  CURRENT_TRACE_ID = tid;
   document.getElementById('detail').innerHTML = '<div class="empty">Loading trace...</div>';
   try {
     const data = await api('/api/v1/traces/' + encodeURIComponent(tid));
+    CURRENT_TRACE_DATA = data;
     renderTrace(data.trajectory, data.report);
   } catch (e) {
     document.getElementById('detail').innerHTML = '<div class="empty">' + escapeHtml(e) + '</div>';
@@ -485,7 +503,7 @@ function renderTrace(traj, report) {
   for (const f of findings) html += renderFinding(f);
   html += '</div></div></div>';
 
-  html += '<div class="panel"><div class="panel-head"><div class="panel-title">Use Case Flow</div><span class="chip cyan">Error Hub</span></div><div class="panel-body"><div class="flow">';
+  html += '<div class="panel" id="error-hub-flow"><div class="panel-head"><div class="panel-title">Use Case Flow</div><span class="chip cyan">Error Hub</span></div><div class="panel-body"><div class="flow">';
   html += flow(1, 'Capture trajectory from the running agent with the lightweight recorder or adapter.');
   html += flow(2, 'Diagnose the trace, localize the likely root cause, and generate recovery suggestions.');
   html += flow(3, 'Scrub secrets and PII, package a reproducible error bundle, and publish to Git or Hugging Face.');
@@ -566,6 +584,32 @@ function renderEvent(ev, isRoot, finding) {
   html += '</div></div></div>';
   return html;
 }
+function downloadJson(filename, value) {
+  const blob = new Blob([JSON.stringify(value, null, 2)], {type: 'application/json'});
+  const url = URL.createObjectURL(blob);
+  const a = document.createElement('a');
+  a.href = url;
+  a.download = filename;
+  document.body.appendChild(a);
+  a.click();
+  a.remove();
+  URL.revokeObjectURL(url);
+}
+function bindTopActions() {
+  document.getElementById('analyze-btn').onclick = () => {
+    const active = document.querySelector('.run.active');
+    if (CURRENT_TRACE_ID && active) selectTrace(CURRENT_TRACE_ID, active);
+  };
+  document.getElementById('export-btn').onclick = () => {
+    if (!CURRENT_TRACE_DATA) return;
+    const name = (CURRENT_TRACE_ID || 'trace') + '.agentdebugx.report.json';
+    downloadJson(name, CURRENT_TRACE_DATA);
+  };
+  document.getElementById('hub-btn').onclick = () => {
+    const flow = document.getElementById('error-hub-flow');
+    if (flow) flow.scrollIntoView({behavior: 'smooth', block: 'start'});
+  };
+}
 function field(label, value, isError) {
   return '<div class="field ' + (isError ? 'error' : '') + '"><div class="field-label">' + escapeHtml(label) + '</div><div class="field-value">' + escapeHtml(value || '-') + '</div></div>';
 }
@@ -585,11 +629,14 @@ if (BOOTSTRAP && BOOTSTRAP.traces) {
   const selected = BOOTSTRAP.selected ? BOOTSTRAP.selected.trajectory.trace_id : null;
   renderTraceList(BOOTSTRAP.traces, selected);
   if (BOOTSTRAP.selected) {
+    CURRENT_TRACE_ID = selected;
+    CURRENT_TRACE_DATA = BOOTSTRAP.selected;
     renderTrace(BOOTSTRAP.selected.trajectory, BOOTSTRAP.selected.report);
   } else {
     document.getElementById('detail').innerHTML = '<div class="empty">No traces in store.</div>';
   }
 }
+bindTopActions();
 loadTraceList(!(BOOTSTRAP && BOOTSTRAP.selected));
 </script>
 </body>