agentdebugx 0.2.4__tar.gz → 0.2.6__tar.gz

{agentdebugx-0.2.4 → agentdebugx-0.2.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentdebugx
-Version: 0.2.4
+Version: 0.2.6
 Summary: Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`.
 License: MIT
 License-File: LICENSE
@@ -26,6 +26,7 @@ Classifier: Topic :: Software Development :: Quality Assurance
 Classifier: Topic :: System :: Monitoring
 Classifier: Typing :: Typed
 Provides-Extra: all
+Provides-Extra: crewai
 Provides-Extra: hub-hf
 Provides-Extra: langgraph
 Provides-Extra: openhands

{agentdebugx-0.2.4 → agentdebugx-0.2.6}/docs/21_integrations.md

@@ -58,6 +58,62 @@ command:
 | "I need a thorough postmortem" | `agentdebug deep <file|trace_id>` |
 | "share this with the team" | `agentdebug hub push <trace_id> --to git:...` |
 
+## 1.5 CrewAI integration
+
+CrewAI emits a typed event stream via a process-global
+`crewai.events.crewai_event_bus`. `agentdebug.adapters.crewai` ships two
+pieces:
+
+- `CrewAIBridge(debugger, trajectory)` — context manager that subscribes
+  to the bus and translates events into AgentDebug records.
+- `CrewAIAdapter().instrument(debugger)` — used by `agentdebug doctor` to
+  report whether CrewAI is importable.
+
+### Recording a Crew run
+
+```python
+from agentdebug import AgentDebug, SQLiteTraceStore
+from agentdebug.adapters.crewai import CrewAIBridge
+
+debugger = AgentDebug(store=SQLiteTraceStore('.agentdebug/errors.sqlite'))
+trajectory = debugger.start_trace(goal='build a marketing plan', framework='crewai')
+
+with CrewAIBridge(debugger, trajectory):
+    crew.kickoff(inputs={...})   # standard CrewAI call
+
+debugger.finish_trace(trajectory, success=True)
+```
+
+### What gets recorded
+
+| CrewAI event class | AgentDebug `EventType` | Module |
+|---|---|---|
+| `CrewKickoffStartedEvent` | `AGENT_STEP` | planning |
+| `CrewKickoffCompletedEvent` | `OBSERVATION` | planning |
+| `TaskStartedEvent` | `PLAN` | planning |
+| `TaskCompletedEvent` | `OBSERVATION` | planning |
+| `AgentExecutionStartedEvent` | `AGENT_STEP` | planning |
+| `AgentExecutionCompletedEvent` | `OBSERVATION` | reflection |
+| `LLMCallStartedEvent` | `LLM_CALL` | planning |
+| `LLMCallCompletedEvent` | `LLM_RESPONSE` | planning |
+| `ToolUsageStartedEvent` | `TOOL_CALL` | action |
+| `ToolUsageFinishedEvent` | `TOOL_RESULT` | action |
+| `ToolUsageErrorEvent` | `TOOL_RESULT` (with `error`) | action |
+
+Event class names not present in the installed CrewAI version are silently
+skipped — useful for forward/backward compatibility.
+
+### Install
+
+```bash
+pip install 'agentdebugx[crewai]'
+```
+
+### See also
+
+`examples/crewai_demo.py` shows a complete two-agent crew (researcher +
+editor) instrumented with `CrewAIBridge`.
+
 ## 2. OpenHands integration
 
 `agentdebug.integrations.openhands` ships two complementary pieces.

{agentdebugx-0.2.4 → agentdebugx-0.2.6}/docs/22_industry_track_paper_eval_plan.md

@@ -233,3 +233,28 @@ Storage/export:
 4. Run rule, judge, All-at-Once, and DeepDebug on the same split.
 5. Add a small human/expert review with raw trace vs paired trace view.
 6. Trim main body to 6 pages while keeping appendix rich.
+
+## 9. Implemented Experiment Harness
+
+The repository now contains an `experiments/` directory for the first paper
+pipeline:
+
+- `prepare_who_when.py`: downloads the two public Who&When parquet files from
+  Hugging Face and normalizes them into `AgentTrajectory` JSONL plus separate
+  gold labels. On 2026-05-16 this produced 184 traces and 4,092 events.
+- `run_e2e_smoke.py`: exercises the public AgentDebugX modules end to end:
+  analyzer, detectors, traceback, recoverers, attribution, and Error Hub local
+  push/pull. It can optionally call a live OpenAI-compatible endpoint.
+- `run_who_when_eval.py`: runs attribution metrics against Who&When labels,
+  including responsible-agent match, exact step match, +/-1 step match, and
+  joint agent/step accuracy.
+- `generate_paper_figures_openai.py`: generates the optional paper pipeline
+  figure with `gpt-image-2` when `OPENAI_API_KEY` is available.
+
+The AgentDebug / AgentErrorBench dataset is currently linked from the upstream
+paper repository as a Google Drive folder rather than a stable direct download
+URL. `prepare_agenterrorbench.py` accepts a local extracted folder under
+`data/agenterrorbench/raw/`, can attempt `gdown` download when the optional
+dependency is installed, preserves environment metadata for ALFWorld, GAIA, and
+WebShop, and outputs the same `AgentTrajectory` + labels format used by the
+Who&When loader.

{agentdebugx-0.2.4 → agentdebugx-0.2.6}/docs/23_status_v0_2.md

@@ -82,6 +82,26 @@ The audit found one real bug and a handful of test gaps:
 5. **`recovery.ReflexionSuggestion`** had only an indirect test from DeepDebug
    examples; now has direct happy + empty tests.
 
+## 3.7 Judge hardening (0.2.6)
+
+A v0.2.5 Who&When 5-trace live run had `llm_judge_root.agent_match=0.00`
+because the judge truncated mid-array on long multi-agent debate
+transcripts. Three changes in 0.2.6 lifted that to **0.40** on the same
+sample (same model, same traces):
+
+1. `LLMJudgeAnalyzer.max_tokens` default **4096 → 8192** — leaves room for
+   thinking-model reasoning tokens before the JSON object starts.
+2. `LLMJudgeAnalyzer.max_findings_per_chunk` parameter (default 6) — the
+   system prompt now asks the model to cap its findings array, forcing it
+   to close the JSON even when many candidates are visible.
+3. System prompt now has explicit "CRITICAL OUTPUT RULES" — output ONLY
+   JSON, no markdown fences, no newlines in string values, complete the
+   array.
+
+Numbers: see [docs/benchmarks/who_when_v0_2_6_leaderboard.md](./benchmarks/who_when_v0_2_6_leaderboard.md).
+Same trick works for `BinarySearchAttributor` (shipped in 0.2.4) — apply
+to remaining LLM-using analyzers as more thinking models surface this.
+
 ## 3.6 Real-usage E2E (live Gemini)
 
 Beyond unit tests, `scripts/e2e_real_usage.py` builds three realistic failing
@@ -142,8 +162,22 @@ Before v0.3 ships, this doc should record green checkmarks for:
       default verifier templates (JSON-schema guard, final-state check,
       tool-result type-check, handoff contract, loop-detector guard) and
       emits per-finding `FixProposal` with rationale + suggested code.
-- [ ] One additional framework adapter that goes through the full conformance
-      suite (CrewAI is the most-requested).
-- [ ] HuggingFace Hub round-trip live test (gated on `HF_TOKEN`).
-- [ ] Bench harness extended with one published-benchmark loader (Who&When
-      is the obvious first target — we already cite its method).
+- [x] **One additional framework adapter** — CrewAI adapter shipped in
+      0.2.5 (`agentdebug.adapters.crewai`). `CrewAIBridge` context manager
+      subscribes to `crewai_event_bus`, translates 11 CrewAI event types
+      into `AgentEvent`s. Conformance test mocks the bus and verifies
+      every documented event mapping plus the version-skew degradation
+      path. `examples/crewai_demo.py` shows a working two-agent crew.
+- [x] **HuggingFace Hub round-trip live test** — shipped in 0.2.6 as
+      `tests/test_hub_huggingface_live.py`. Gated on `HF_TOKEN` +
+      `AGENTDEBUG_HF_LIVE=1` so it never runs in default CI. Creates the
+      dataset repo if missing, pushes a bundle, lists, pulls back, verifies
+      the trajectory round-trips bit-for-bit. Live-validated against
+      `KunlunZhu/agentdebugx-live-test`.
+- [x] **Bench harness with Who&When loader** — `experiments/prepare_who_when.py`
+      ingests 184 Algorithm-Generated + Hand-Crafted traces (4092 events) and
+      stores labels separately. `experiments/run_who_when_eval.py` runs all
+      4 attributors + DeepDebug against gold labels; reports agent_match,
+      exact_step, near_step. Live-Gemini 5-trace validation captured at
+      [docs/benchmarks/who_when_v0_2_6_leaderboard.md](./benchmarks/who_when_v0_2_6_leaderboard.md).
+      Headline 184-trace run deferred (~6h / ~$5-10 on a frontier model).

agentdebugx-0.2.6/docs/benchmarks/e2e_v0_2_4.md

@@ -0,0 +1,365 @@
+# AgentDebugX v0.2.3 End-to-End Real-Usage Smoke
+
+Scenarios: **3**. LLM model: `gemini-3-flash`. Generated by `scripts/e2e_real_usage.py`.
+
+## Per-scenario pipeline status
+
+| Scenario | trace_id | OK / Total stages | Failed stages |
+|---|---|---|---|
+| `action_format_then_hallucination` | `trace_cde22f…` | 12 / 12 | — |
+| `multiagent_handoff_loss` | `trace_84fb3a…` | 12 / 12 | — |
+| `planning_loop` | `trace_c59dc1…` | 12 / 12 | — |
+
+**UI smoke:** ✅ all endpoints responded
+
+```
+GET /healthz -> 200 {"status":"ok"}
+GET /api/v1/traces -> 5 trace(s)
+GET /api/v1/traces/<id> -> 200 events=11 findings=4
+GET /api/v1/taxonomy -> modes=19
+GET / -> 200 content_length=33710 has_brand=True
+```
+
+## `action_format_then_hallucination`
+
+`trace_id=trace_cde22f0eb1ec499aa3bc5b88f472257d`
+
+### ✅ `heuristic_analyzer` (0.00s) — 1 finding(s); root=2
+### ✅ `cross_event_detectors` (0.00s) — 0 finding(s) from default_detectors()
+### ✅ `traceback_offline` (0.00s) — rendered
+
+```
+AgentTraceback (root cause first, manifested failure last):
+    trace_id=trace_cde22f0eb1ec499aa3bc5b88f472257d  framework=e2e-react  goal='Find the latest AgentDebug paper, summarize the method, then email alice@example.com'
+
+      File "root cause", in trajectory
+        Step 2  agent=search_web  mode=system.tool_execution_error  confidence=0.86
+          event_id=evt_1eb26b89fce447fa8d9d908b50741ac3
+          error>  JSON schema validation failed: missing parameter query
+          evidence:
+            - JSON schema validation failed: missing parameter query
+          suggested: Capture tool stderr/status/latency and classify retryable versus non-retryable failures.
+
+AgentFailure[system.tool_execution_error]: Likely root cause: Tool execution error in search_web at step 2.
+```
+
+### ✅ `reflexion_suggestion` (0.00s) — 1 proposal(s)
+
+```
+Reflexion retry hint for system.tool_execution_error at step 2
+```
+
+### ✅ `critic_recoverer` (0.00s) — 1 verifier proposal(s)
+
+```
+Add tool_result_typecheck before system.tool_execution_error (step 2, agent search_web)
+```
+
+### ✅ `llm_judge` (14.31s) — 3 finding(s); root=1
+
+```
+- action.parameter_error (conf=1.00) step=1 agent=search_web
+- verification.premature_stop (conf=1.00) step=4 agent=planner
+- verification.missing_task_validation (conf=0.90) step=None agent=system
+```
+
+### ✅ `attribute_heuristic` (0.00s) — method=heuristic agent=search_web step=1 conf=1.00
+
+```
+Earliest finding with non-trivial confidence: Parameter error
+```
+
+### ✅ `attribute_all_at_once` (7.52s) — method=all_at_once agent=search_web step=1 conf=0.90
+
+```
+The root cause is the malformed tool call in step 1, where the search_web tool was invoked without the required 'query' parameter, leading to a validation error that derailed the entire process.
+```
+
+### ✅ `attribute_step_by_step` (23.64s) — method=step_by_step agent=search_web step=1 conf=1.00
+
+```
+The agent invoked the search tool without providing a search query, which is a parameter error that prevents the retrieval of the required information.
+```
+
+### ✅ `attribute_binary_search` (8.01s) — method=binary_search agent=search_web step=2 conf=0.80
+
+```
+Binary search located the decisive step within 2 probes over 6 events.
+```
+
+### ✅ `deep_debug` (28.00s) — 3 finding(s); rounds=6
+
+```
+rounds: plan:4588ms / hypothesize:8051ms / verify:h1:2874ms / verify:h2:2392ms / verify:h3:2514ms / refine:7581ms
+summary: The agent failed to provide the required 'query' parameter when calling the search tool, and the planner subsequently ignored this failure, incorrectly claiming the task was complete without summarizing the paper or sending the email.
+
+AgentTraceback (root cause first, manifested failure last):
+    trace_id=trace_cde22f0eb1ec499aa3bc5b88f472257d  framework=e2e-react  goal='Find the latest AgentDebug paper, summarize the method, then email alice@example.com'
+
+      File "root cause", in trajectory
+        Step 2  agent=search_web  mode=action.parameter_error  confidence=1.00
+          event_id=evt_1eb26b89fce447fa8d9d908b50741ac3
+          error>  JSON schema validation failed: missing parameter query
+          evidence:
+            - JSON schema validation failed: missing parameter query
+            - kwargs='{}'
+          suggested: Validate parameters against tool schemas and ask for missing user/context fields.
+    ↓ cascaded to
+      File "cascade depth 1", in trajectory
+        Step 4  agent=planner  mode=reflection.progress_misjudge  confidence=1.00
+          module=reflection
+          event_id=evt_2b76fd64970544908fdb65eee79bbd9d
+          output> Final answer: AgentDebug is a popular paper. Done.
+          evidence:
+            - Final answer: AgentDebug is a popular paper. Done.
+            - meta={'metadata': {'final_answer': True}}
+          suggested: Add an external task verifier before termination.
+    ↓ cascaded to
+      File "cascade depth 2", in trajectory
+        Step ?  agent=system  mode=verification.missing_task_validation  confidence=1.00
+          event_id=evt_15fe36c6a23247cea25efb4a34fccedf
+          evidence:
+            - meta={'success': True}
+          suggested: Add final-state validation that is independent of the acting agent.
+
+AgentFailure[verification.missing_task_validation]: The agent failed to provide the required 'query' parameter when calling the search tool, and the planner subsequently ignored this failure, incorrectly claiming the task was complete without summarizing the paper or sending the email.
+```
+
+### ✅ `hub_round_trip` (0.00s) — pushed=/home/kunlunz2/AgentDebugX/.agentdebug/e2e_hub/bundle_e79cadee55524420ba4719ecd2934536 ; bundle_id=bundle_e79cadee55524420ba4719ecd2934536 ; listed=1 ; round-trip ok
+
+## `multiagent_handoff_loss`
+
+`trace_id=trace_84fb3ac2083c449b8f2ca80503861dc5`
+
+### ✅ `heuristic_analyzer` (0.00s) — 1 finding(s); root=2
+### ✅ `cross_event_detectors` (0.00s) — 0 finding(s) from default_detectors()
+### ✅ `traceback_offline` (0.00s) — rendered
+
+```
+AgentTraceback (root cause first, manifested failure last):
+    trace_id=trace_84fb3ac2083c449b8f2ca80503861dc5  framework=e2e-multiagent  goal='Find the best paper on agent debugging, prefer the most recent.'
+
+      File "root cause", in trajectory
+        Step 2  agent=researcher  mode=multiagent.handoff_loss  confidence=0.70
+          module=multiagent
+          event_id=evt_800ad2ca86644768b07990f0af08e1b6
+          output> Please summarize the agent debugging paper.
+          evidence:
+            - handoff/context signal in event payload
+          suggested: Make handoff payloads typed and include goal, constraints, evidence, confidence, and open questions.
+
+AgentFailure[multiagent.handoff_loss]: Likely root cause: Handoff context loss in researcher at step 2.
+```
+
+### ✅ `reflexion_suggestion` (0.00s) — 1 proposal(s)
+
+```
+Reflexion retry hint for multiagent.handoff_loss at step 2
+```
+
+### ✅ `critic_recoverer` (0.00s) — 1 verifier proposal(s)
+
+```
+Add handoff_context_contract before multiagent.handoff_loss (step 2, agent researcher)
+```
+
+### ✅ `llm_judge` (9.36s) — 2 finding(s); root=2
+
+```
+- multiagent.handoff_loss (conf=1.00) step=2 agent=researcher
+- verification.missing_task_validation (conf=0.90) step=None agent=system
+```
+
+### ✅ `attribute_heuristic` (0.00s) — method=heuristic agent=researcher step=2 conf=1.00
+
+```
+Earliest finding with non-trivial confidence: Handoff context loss
+```
+
+### ✅ `attribute_all_at_once` (3.72s) — method=all_at_once agent=researcher step=2 conf=1.00
+
+```
+The researcher correctly identified Paper A as the most recent in step 1 but failed to specify this choice or the recency constraint during the handoff in step 2, leading the summarizer to pick the wrong paper.
+```
+
+### ✅ `attribute_step_by_step` (21.99s) — method=step_by_step agent=researcher step=2 conf=1.00
+
+```
+The researcher failed to specify which paper to summarize during the handoff, losing the context of the 'most recent' paper identified in the previous step.
+```
+
+### ✅ `attribute_binary_search` (16.41s) — method=binary_search agent=summarizer step=4 conf=0.90
+
+```
+Binary search located the decisive step within 3 probes over 6 events.
+```
+
+### ✅ `deep_debug` (45.65s) — 3 finding(s); rounds=6
+
+```
+rounds: plan:4990ms / hypothesize:7110ms / verify:h1:2939ms / verify:h2:2320ms / verify:h3:5377ms / refine:22907ms
+summary: The researcher agent hallucinated finding specific papers without performing a search and subsequently failed to communicate the paper selection or user constraints to the summarizer, leading to a total failure of the summarization step.
+
+AgentTraceback (root cause first, manifested failure last):
+    trace_id=trace_84fb3ac2083c449b8f2ca80503861dc5  framework=e2e-multiagent  goal='Find the best paper on agent debugging, prefer the most recent.'
+
+      File "root cause", in trajectory
+        Step 1  agent=researcher  mode=memory.hallucination  confidence=0.95
+          module=planning
+          event_id=evt_a4f22a565af84663bab9f0460b122ea8
+          output> Found two candidate papers: A (May 2025) and B (Mar 2024). A is preferred because it is more recent (per user constraint).
+          evidence:
+            - Found two candidate papers: A (May 2025) and B (Mar 2024)
+            - The trajectory shows a premature handoff... without any evidence of search or data retrieval occurring.
+          suggested: Require memory reads to cite the source event or artifact before use.
+    ↓ cascaded to
+      File "cascade depth 1", in trajectory
+        Step 2  agent=researcher  mode=multiagent.handoff_loss  confidence=1.00
+          module=multiagent
+          event_id=evt_800ad2ca86644768b07990f0af08e1b6
+          output> Please summarize the agent debugging paper.
+          evidence:
+            - Please summarize the agent debugging paper.
+            - omitted_context: 'preference for A; recency constraint'
+          suggested: Make handoff payloads typed and include goal, constraints, evidence, confidence, and open questions.
+    ↓ cascaded to
+      File "cascade depth 2", in trajectory
+        Step 3  agent=summarizer  mode=memory.retrieval_failure  confidence=0.90
+          module=planning
+          event_id=evt_8f20248aa2b64a6dad966bd9f336cc77
+          input>  Please summarize the agent debugging paper.
+          evidence:
+            - input=Please summarize the agent debugging paper.
+            - output=None
+            - omitted_context: 'preference for A; recency constraint'
+          suggested: Persist the missing state as structured memory and attach it to the next planning step.
+
+AgentFailure[memory.retrieval_failure]: The researcher agent hallucinated finding specific papers without performing a search and subsequently failed to communicate the paper selection or user constraints to the summarizer, leading to a total failure of the summarization step.
+```
+
+### ✅ `hub_round_trip` (0.00s) — pushed=/home/kunlunz2/AgentDebugX/.agentdebug/e2e_hub/bundle_27e79ad1f5454df4b89f9f7fe75bab3e ; bundle_id=bundle_27e79ad1f5454df4b89f9f7fe75bab3e ; listed=3 ; round-trip ok
+
+## `planning_loop`
+
+`trace_id=trace_c59dc17f26994841ad361176ddf6b7c0`
+
+### ✅ `heuristic_analyzer` (0.00s) — 4 finding(s); root=2
+### ✅ `cross_event_detectors` (0.00s) — 3 finding(s) from default_detectors()
+
+```
+- planning.inefficient_plan (source=repeated_tool_call)
+- planning.inefficient_plan (source=repeated_state)
+- planning.inefficient_plan (source=repeated_state)
+```
+
+### ✅ `traceback_offline` (0.00s) — rendered
+
+```
+AgentTraceback (root cause first, manifested failure last):
+    trace_id=trace_c59dc17f26994841ad361176ddf6b7c0  framework=e2e-browser  goal='Submit the checkout form on shop.example.com'
+
+      File "root cause", in trajectory
+        Step 2  agent=browser  mode=planning.inefficient_plan  confidence=0.67
+          event_id=evt_1d189d0f66de4329a6d11f567721eea4
+          output> no progress; same checkout screen
+          evidence:
+            - loop/progress signal in event payload
+          suggested: Add loop detection over tool calls and state deltas.
+    ↓ cascaded to
+      File "cascade depth 1", in trajectory
+        Step 4  agent=browser  mode=planning.inefficient_plan  confidence=0.67
+          event_id=evt_d6e7dd127fab458896b58bae5cac1954
+          output> no progress; same checkout screen
+          evidence:
+            - loop/progress signal in event payload
+          suggested: Add loop detection over tool calls and state deltas.
+    ↓ cascaded to
+      File "cascade depth 2", in trajectory
+        Step 6  agent=browser  mode=planning.inefficient_plan  confidence=0.67
+          event_id=evt_c92a15e9cc894bffaff61ec5f6169150
+          output> no progress; same checkout screen
+          evidence:
+            - loop/progress signal in event payload
+          suggested: Add loop detection over tool calls and state deltas.
+    ↓ cascaded to
+      File "cascade depth 3", in trajectory
+        Step 8  agent=browser  mode=planning.inefficient_plan  confidence=0.67
+          event_id=evt_df07344c3e9e4f7fba9cb3df92a73bfd
+          output> no progress; same checkout screen
+          evidence:
+            - loop/progress signal in event payload
+          suggested: Add loop detection over tool calls and state deltas.
+
+AgentFailure[planning.inefficient_plan]: Likely root cause: Inefficient plan in browser at step 2.
+```
+
+### ✅ `reflexion_suggestion` (0.00s) — 4 proposal(s)
+
+```
+Reflexion retry hint for planning.inefficient_plan at step 2
+Reflexion retry hint for planning.inefficient_plan at step 4
+Reflexion retry hint for planning.inefficient_plan at step 6
+Reflexion retry hint for planning.inefficient_plan at step 8
+```
+
+### ✅ `critic_recoverer` (0.00s) — 4 verifier proposal(s)
+
+```
+Add loop_detector_guard before planning.inefficient_plan (step 2, agent browser)
+Add loop_detector_guard before planning.inefficient_plan (step 4, agent browser)
+Add loop_detector_guard before planning.inefficient_plan (step 6, agent browser)
+Add loop_detector_guard before planning.inefficient_plan (step 8, agent browser)
+```
+
+### ✅ `llm_judge` (23.10s) — 0 finding(s); root=None
+### ✅ `attribute_heuristic` (0.00s) — method=heuristic (no hypotheses)
+### ✅ `attribute_all_at_once` (5.36s) — method=all_at_once agent=planner step=1 conf=0.90
+
+```
+The planner established a flawed strategy of repeatedly clicking the submit button without any logic to handle potential form validation errors or investigate why the submission was failing.
+```
+
+### ✅ `attribute_step_by_step` (57.37s) — method=step_by_step agent=planner step=1 conf=0.90
+
+```
+The planner's strategy is fundamentally flawed as it attempts to click a submit button without first navigating to the website or filling out the required form fields.
+```
+
+### ✅ `attribute_binary_search` (24.84s) — method=binary_search agent=browser step=4 conf=0.90
+
+```
+Binary search located the decisive step within 3 probes over 11 events.
+```
+
+### ✅ `deep_debug` (38.32s) — 2 finding(s); rounds=6
+
+```
+rounds: plan:3303ms / hypothesize:9554ms / verify:h1:3739ms / verify:h2:4204ms / verify:h3:8529ms / refine:8988ms
+summary: The agent failed to submit the checkout form because it repeatedly attempted to click the submit button without ensuring all required fields (such as address or payment details) were populated, ignoring the lack of progress between attempts.
+
+AgentTraceback (root cause first, manifested failure last):
+    trace_id=trace_c59dc17f26994841ad361176ddf6b7c0  framework=e2e-browser  goal='Submit the checkout form on shop.example.com'
+
+      File "root cause", in trajectory
+        Step 7  agent=browser  mode=planning.inefficient_plan  confidence=0.95
+          event_id=evt_0b22e12ef8bb48bdad049706d6b77d33
+          input>  {'tool': 'click', 'args': '()', 'kwargs': "{'selector': '#submit'}"}
+          evidence:
+            - step=5 ... click ... #submit
+            - step=7 ... click ... #submit
+            - output=no progress; same checkout screen
+          suggested: Add loop detection over tool calls and state deltas.
+    ↓ cascaded to
+      File "cascade depth 1", in trajectory
+        Step 8  agent=browser  mode=planning.constraint_ignorance  confidence=1.00
+          event_id=evt_df07344c3e9e4f7fba9cb3df92a73bfd
+          output> no progress; same checkout screen
+          evidence:
+            - no progress; same checkout screen
+            - Strategy: click #submit until success
+          suggested: Compile task and tool constraints into pre-action checks.
+
+AgentFailure[planning.constraint_ignorance]: The agent failed to submit the checkout form because it repeatedly attempted to click the submit button without ensuring all required fields (such as address or payment details) were populated, ignoring the lack of progress between attempts.
+```
+
+### ✅ `hub_round_trip` (0.01s) — pushed=/home/kunlunz2/AgentDebugX/.agentdebug/e2e_hub/bundle_8f2dab82d8e74638ab8656f834d353ab ; bundle_id=bundle_8f2dab82d8e74638ab8656f834d353ab ; listed=4 ; round-trip ok

agentdebugx-0.2.6/docs/benchmarks/who_when_v0_2_6_leaderboard.md

@@ -0,0 +1,74 @@
+# Who&When — 5-trace Live Leaderboard (v0.2.6, gemini-3-flash)
+
+Tiny validation sample drawn from `data/who_when/processed/labels.jsonl`
+(first 5 algorithm-generated traces). **Not a publishable benchmark** — the
+full benchmark requires the 184-trace dataset + a frontier model and is
+deferred for cost reasons. This run exists to verify the analysis stack
+produces sensible-shaped numbers and to surface regressions early.
+
+## Aggregate (per attribution method)
+
+| Method | agent_match | exact_step | near_step | both_near | DeepDebug rounds |
+|---|---:|---:|---:|---:|---:|
+| `heuristic` (rule baseline) | 0.20 | 0.00 | 0.20 | 0.20 | n/a |
+| `llm_judge_root` (judge's root_cause field) | **0.40** | 0.00 | **0.20** | **0.20** | n/a |
+| `all_at_once` (Who&When method 1) | 0.20 | 0.00 | 0.00 | 0.00 | n/a |
+| `step_by_step` (Who&When method 2) | **0.40** | 0.00 | **0.20** | **0.20** | n/a |
+| `deep_debug_root` (DeepDebug refined root) | 0.20 | 0.00 | 0.20 | 0.00 | 6 / trace |
+
+## What changed in 0.2.6 vs 0.2.5
+
+Same 5 traces, same model:
+
+| Method | 0.2.5 agent_match | 0.2.6 agent_match | Δ |
+|---|---:|---:|---:|
+| `heuristic` | 0.20 | 0.20 | — |
+| `llm_judge_root` | 0.00 | **0.40** | +0.40 |
+| `all_at_once` | 0.00 | 0.20 | +0.20 |
+| `step_by_step` | 0.00 | **0.40** | +0.40 |
+
+The driver was the v0.2.6 judge prompt hardening: `max_tokens` default
+4096 → 8192, an explicit `max_findings_per_chunk=6` cap surfaced through
+the system prompt, and a "CRITICAL OUTPUT RULES" header (output ONLY JSON,
+no markdown, no newlines in strings, complete the array). Before the
+hardening, the judge truncated mid-array on Who&When debate transcripts
+and returned no findings; after, the structured root_cause is populated.
+
+## Honest caveats
+
+* n=5; per-method standard error is ±0.22 — these absolute numbers should
+  not be over-interpreted. The 0.4 vs 0.0 jump for two methods is the
+  signal worth reporting; everything else is noise.
+* `deep_debug_root` underperformed `step_by_step` on this sample. The
+  refine round on 7-event traces tends to converge to the *visible*
+  failure rather than the *causal* root (a known Who&When difficulty —
+  manifestation vs root cause).
+* No method beats `near_step=0.20` on this sample. Step-localization
+  remains hard, matching the published Who&When ceiling (~14% step on
+  127 traces with frontier models).
+
+## Reproducing
+
+```bash
+# Prepare data (once)
+PYTHONPATH=src python experiments/prepare_who_when.py
+
+# Set live LLM creds (any OpenAI-compatible endpoint works)
+export AGENTDEBUG_LLM_BASE_URL=...
+export AGENTDEBUG_LLM_API_KEY=...
+export AGENTDEBUG_LLM_MODEL=gemini-3-flash
+
+# Without DeepDebug (~1 min)
+PYTHONPATH=src python experiments/run_who_when_eval.py \
+  --limit 5 --live-openai \
+  --out-dir experiments/runs/who_when_eval_subset
+
+# With DeepDebug (~5 min)
+PYTHONPATH=src python experiments/run_who_when_eval.py \
+  --limit 5 --live-openai --deep \
+  --out-dir experiments/runs/who_when_eval_subset_deep
+```
+
+The headline benchmark (184 traces × 5 methods × DeepDebug) would take
+~6 hours and ~$5-10 in API cost on a frontier model. Run it once before
+paper submission; do not run on every iteration.

{agentdebugx-0.2.4 → agentdebugx-0.2.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "agentdebugx"
-version = "0.2.4"
+version = "0.2.6"
 description = "Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`."
 authors = ["ULab @ UIUC <ulab@illinois.edu>"]
 license = "MIT"
@@ -52,12 +52,14 @@ httpx = ">=0.24,<1.0"
 # Optional integrations — install via `pip install agentdebugx[langgraph]` etc.
 [tool.poetry.extras]
 langgraph = ["langchain-core"]
+crewai = ["crewai"]
 otel = ["opentelemetry-api", "opentelemetry-sdk"]
 ui = ["fastapi", "uvicorn"]
 hub-hf = ["huggingface_hub"]
 openhands = ["openhands-ai"]
 all = [
     "langchain-core",
+    "crewai",
     "opentelemetry-api",
     "opentelemetry-sdk",
     "fastapi",

{agentdebugx-0.2.4 → agentdebugx-0.2.6}/src/agentdebug/__init__.py

@@ -96,4 +96,4 @@ __all__ = [
     'get_failure_mode',
 ]
 
-__version__ = '0.2.4'
+__version__ = '0.2.6'