agentdebugx 0.2.1__tar.gz → 0.2.2__tar.gz

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentdebugx
-Version: 0.2.1
+Version: 0.2.2
 Summary: Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`.
 License: MIT
 License-File: LICENSE

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/docs/14_api_reference.md

@@ -215,6 +215,32 @@ agentdebugx taxonomy export --format yaml|json|md
 agentdebugx doctor
 ```
 
+## 11.1 Current shipped `agentdebug` CLI surface
+
+The public design above is the long-term `agentdebugx` contract. The current
+package already ships a smaller but working `agentdebug` CLI:
+
+```bash
+agentdebug analyze <trajectory.json> [--suggest] [--traceback]
+agentdebug list --store-sqlite .agentdebug/errors.sqlite
+agentdebug show <trace_id> --store-sqlite .agentdebug/errors.sqlite
+agentdebug judge <trajectory.json|trace_id> --attribute [--traceback]
+agentdebug deep <trajectory.json|trace_id> [--traceback]
+agentdebug hub push <trace_id> --to local:/tmp/hub --store-sqlite ...
+agentdebug hub pull <spec> --bundle <bundle_id> --into .agentdebug/hub_pulls
+agentdebug hub list <spec>
+agentdebug integrations skill --target ~/.claude/skills --name agentdebug
+agentdebug integrations openhands-microagent --target .openhands/microagents
+agentdebug serve --store-sqlite .agentdebug/errors.sqlite --port 7777
+agentdebug doctor
+```
+
+`--traceback` renders `AgentTraceback`, a Python-traceback-style cascade view
+implemented by `agentdebug.traceback.format_traceback(report, trajectory)`.
+DeepDebug can provide explicit cascade edges through
+`finding.metadata['cascading_from_event_id']`; heuristic and single-pass judge
+reports fall back to step-index ordering.
+
 ## 12. Configuration file
 
 `~/.agentdebugx/settings.yaml`:

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/docs/15_roadmap.md

@@ -28,6 +28,17 @@ Acceptance:
 
 ## v0.2 — Coverage + UI (4 weeks)
 
+Already shipped in the v0.2/v0.2.1 line:
+
+- Error Hub bundle format + Local/Git/Hugging Face backends.
+- DeepDebug iterative analysis loop.
+- Claude Code Skill generator and OpenHands microagent/EventStream bridge.
+- `AgentTraceback` cascade renderer and CLI `--traceback` support.
+- FastAPI local console with native-trace + error-trace alignment for human
+  review.
+
+Remaining scope from the original v0.2 plan:
+
 - Adapters: CrewAI, OpenHands, smolagents, LlamaIndex, DSPy, Pydantic-AI.
 - Detectors: anomaly family (perplexity, repeated-state, topic-drift).
 - Attribution: `BinarySearchAttributor`, `CounterfactualAttributor`, `EnsembleAttributor`.

agentdebugx-0.2.2/docs/22_industry_track_paper_eval_plan.md

@@ -0,0 +1,235 @@
+# 22 - EMNLP Industry Track Paper + Evaluation Plan
+
+This note translates the EMNLP 2026 Industry Track call and recent ACL/EMNLP
+Industry Track patterns into a concrete writing and evaluation plan for the
+AgentDebugX paper.
+
+## 1. What Industry Track Reviewers Reward
+
+Industry-track papers are not judged like pure method papers. The recurring
+shape in strong papers is:
+
+1. Real deployment pain, not a benchmark-only motivation.
+2. A system that can actually be used by practitioners.
+3. Evaluation under practical constraints: cost, latency, scale, privacy,
+   maintainability, human workflow, and failure modes.
+4. Lessons learned that future builders can reuse.
+5. Clear limitations and responsible deployment boundaries.
+
+Examples worth emulating:
+
+- **Experience Report: Implementing Machine Translation in a Regulated
+  Industry** (EMNLP 2025 Industry): emphasizes legal/security constraints,
+  human-in-the-loop validation, and reviewer preferences over more than 11k
+  ranked translations. Source: https://aclanthology.org/2025.emnlp-industry/
+- **STREAQ** (EMNLP 2025 Industry): frames the contribution as an industrial
+  cost-quality routing system and reports both model quality and operational
+  cost reduction. Source: https://aclanthology.org/2025.emnlp-industry.121/
+- **RAVEN** (ACL 2025 Industry): combines industrial data, public benchmarks,
+  deployment pipeline, and online A/B validation. Source:
+  https://aclanthology.org/2025.acl-industry.3/
+- **ARIA** (EMNLP 2025 Industry): uses a realistic deployed domain plus public
+  dynamic-knowledge tasks, and states deployment scope.
+- **AutoPenBench** (EMNLP 2025 Industry): releases an open benchmark, reports
+  autonomous versus human-assisted agent success, and uses intermediate
+  milestones to show where agents struggle. Source:
+  https://aclanthology.org/2025.emnlp-industry.114/
+
+For AgentDebugX, the paper should therefore be framed as:
+
+> A deployment-oriented debugging layer for agentic NLP systems, evaluated on
+> whether it makes failures observable, attributable, shareable, and easier for
+> humans to fix.
+
+The central claim should not be "we beat every attributor." A stronger claim
+for the Industry Track is:
+
+> AgentDebugX provides the missing operating layer between raw agent traces and
+> actionable debugging workflows: aligned native/error traces, taxonomy-backed
+> reports, Error Hub bundles, and cost-aware analysis profiles.
+
+## 2. Appendix Rule
+
+The EMNLP 2026 Industry Track permits appendices after the bibliography. The
+appendix does not count against the 6-page review limit, but the main paper
+must be self-contained and reviewers are not required to review appendices.
+Source: https://2026.emnlp.org/calls/industry_track/
+So:
+
+- Main body: problem, system, screenshot, concise evaluation table, key
+  findings, limitations.
+- Appendix: benchmark matrix, annotation schema, prompts, model settings,
+  study protocol, redaction examples, additional error traces.
+
+Do not hide the core evaluation logic in the appendix. Put enough in the main
+body that a reviewer can assess technical quality without reading extra pages.
+
+## 3. Main-Body Narrative
+
+Recommended six-page spine:
+
+1. **Introduction**: agent systems fail through cascades; raw traces are not
+   enough; teams need who/when/why/fix.
+2. **Deployment Requirements**: low-friction instrumentation, privacy,
+   portable schema, cost-aware analysis, human review.
+3. **System**: recorder + schema + taxonomy + detectors/attributors +
+   Error Hub + UI.
+4. **Use Case Figure**: paired native trace and AgentDebugX error trace.
+5. **Evaluation**: benchmark coverage, diagnostic accuracy, human utility,
+   operational overhead.
+6. **Lessons/Limitations**: where it works, where it does not, safety.
+
+## 4. Evaluation Questions
+
+### Q1. Coverage
+
+Can AgentDebugX ingest different agent classes without bespoke debugging code?
+
+Benchmarks:
+
+- AgentErrorBench: failure-labeled traces over ALFWorld, GAIA, WebShop.
+- MAST and Who&When: multi-agent attribution and failure taxonomy labels.
+- AgentRx: 115 failed trajectories with critical-step labels.
+- WebShop/WebArena: web navigation and tool-use.
+- tau-bench: retail/airline tool-agent-user interaction.
+- SWE-bench Lite/Verified: coding agents with executable tests.
+- OSWorld: multimodal desktop/GUI agents.
+
+Metric: conversion success rate, required adapter LOC, event coverage, artifact
+coverage, and schema loss notes.
+
+### Q2. Diagnostic Accuracy
+
+Can AgentDebugX classify and localize failures?
+
+Labels:
+
+- failure family
+- failure mode
+- root event ID
+- root agent
+- root step
+- cascade edges
+- evidence spans
+- accepted repair
+
+Metrics:
+
+- family macro-F1
+- mode macro-F1
+- responsible-agent accuracy
+- root-step exact match
+- root-step +/- 1 match
+- cascade-edge F1
+- false-positive rate on successful traces
+- calibration: confidence vs correctness
+
+Baselines:
+
+- rule analyzer
+- single-pass LLM judge
+- All-at-Once attribution
+- Step-by-Step attribution when implemented
+- DeepDebug verify/refine loop
+- benchmark-native labels or published baselines where available
+
+### Q3. Human Utility
+
+Does the paired trace view reduce debugging effort?
+
+Study design:
+
+- 12-24 developers.
+- Within-subject comparison: raw framework trace/logs vs AgentDebugX report.
+- 24-48 total debugging sessions.
+- Counterbalance task order and UI order.
+
+Metrics:
+
+- time to first plausible root cause
+- time to accepted repair
+- correctness against adjudicated labels
+- number of trace events inspected
+- confidence and workload rating
+- free-text feedback on missing evidence
+
+Fallback if recruiting slips:
+
+- 3-5 expert agent builders review 30 traces.
+- Ask them to choose between raw trace and AgentDebugX report, rate usefulness,
+  and mark incorrect/misleading diagnoses.
+
+### Q4. Operational Viability
+
+Can teams run this in real workflows?
+
+Metrics:
+
+- analyzer latency by profile: rule, judge, DeepDebug
+- token cost by trace length
+- local storage overhead
+- UI load time for 100, 1k, 10k events
+- scrubber redaction hit rate
+- scrubber false positives on benign strings
+- Error Hub bundle size and push/pull time
+
+## 5. Data Scale
+
+Minimum credible submission target:
+
+- 500 failed trajectories.
+- 100 successful trajectories for false-positive calibration.
+- At least 30 examples per high-level family where source benchmarks permit.
+- Two annotators per newly labeled trace plus adjudication.
+- DeepDebug on a stratified hard subset of 100 traces.
+
+Stronger target:
+
+- 1,000 failed trajectories.
+- 200 successful trajectories.
+- DeepDebug on every trace where rule and single-pass judge disagree.
+- 50-100 private pilot traces, scrubbed and reported only in aggregate.
+
+## 6. API and Infra Needed
+
+Model APIs:
+
+- OpenAI-compatible endpoint as the default abstraction.
+- OpenAI, Gemini, Anthropic-through-proxy/LiteLLM, and local vLLM/Ollama where
+  feasible.
+
+Benchmark APIs:
+
+- WebShop/ALFWorld/GAIA loaders for AgentErrorBench.
+- MAST/Who&When/AgentRx importers preserving existing labels.
+- tau-bench user/tool simulator wrapper.
+- WebArena browser harness with DOM/text/action capture.
+- SWE-bench Docker harness with shell, patch, and test-output capture.
+- OSWorld capture path for screenshot, accessibility tree, click/action, and
+  verifier result.
+
+Storage/export:
+
+- SQLite for local experiments.
+- Error Hub bundles for sharing.
+- Parquet manifest roll-up for large result analysis.
+
+## 7. What Should Go in the Appendix
+
+- Full benchmark matrix.
+- Exact label schema and examples.
+- Prompts for LLM judge, attributor, and DeepDebug.
+- Model settings and token budgets.
+- Human study instructions and consent/safety notes.
+- Redaction examples.
+- Two or three full trace/report examples.
+- Failure cases where AgentDebugX is wrong.
+
+## 8. Immediate TODO Before Submission
+
+1. Convert at least two public benchmark sources into AgentTrajectory.
+2. Produce a first 100-trace labeled set.
+3. Add an evaluation runner that outputs a single CSV/JSONL.
+4. Run rule, judge, All-at-Once, and DeepDebug on the same split.
+5. Add a small human/expert review with raw trace vs paired trace view.
+6. Trim main body to 6 pages while keeping appendix rich.

agentdebugx-0.2.2/docs/23_status_v0_2.md

@@ -0,0 +1,108 @@
+# 23 — Capability + Test Coverage Status (v0.2.2)
+
+A live audit of what's implemented, what's tested, and what's specced but
+not yet built. Pair this with [docs/15_roadmap.md](./15_roadmap.md), which is
+the forward-looking plan; this doc is the rear-view mirror.
+
+## 1. What ships in v0.2.2 (live on PyPI)
+
+| Layer | Module | Status | Tests |
+|---|---|---|---|
+| Trace IR | `agentdebug.models` | ✅ stable | round-trip + enum tests |
+| Storage | `agentdebug.storage` (JSONL + SQLite) | ✅ stable | round-trip + ctx-mgr |
+| Recorder | `agentdebug.recorder` (`AgentDebug`, `TraceSession`) | ✅ stable | record + analyze flow |
+| Rule analyzer | `agentdebug.analyzers.HeuristicAnalyzer` | ✅ stable | match + suggest |
+| Taxonomy | `agentdebug.taxonomy` (19 seed modes) | ✅ stable | get_mode + list |
+| Function instrumentation | `agentdebug.instrumentation.traced_tool` | ✅ stable | happy + raise |
+| Event bus | `agentdebug.events.EventBus` | ✅ stable | fan-out + auto-detach |
+| LLM client | `agentdebug.llm.OpenAICompatClient` | ✅ stable | mocked httpx + env |
+| LLM judge | `agentdebug.judges.LLMJudgeAnalyzer` | ✅ stable | scripted-LLM happy + silent |
+| Attribution | `agentdebug.attribution.HeuristicAttributor` | ✅ stable | first-finding + tiebreak |
+| Attribution | `agentdebug.attribution.AllAtOnceAttributor` | ✅ stable | mocked LLM + fallback |
+| Attribution | `agentdebug.attribution.StepByStepAttributor` | ✅ **new 0.2.2** | scripted-LLM + fallback |
+| Recovery | `agentdebug.recovery.ReflexionSuggestion` | ✅ stable | per-finding + empty |
+| DeepDebug | `agentdebug.deep.DeepDebugAnalyzer` | ✅ stable | full loop + silent LLM |
+| Cascade view | `agentdebug.traceback.format_traceback` | ✅ stable | cascade + step-order + ANSI + empty |
+| Detectors | `agentdebug.detectors.RepeatedToolCall / RepeatedState / StepCountLimit` | ✅ **new 0.2.2** | threshold + window + budget |
+| Hub bundle | `agentdebug.hub.Bundle / pack_bundle / unpack_bundle` | ✅ stable | round-trip |
+| Hub scrubber | `agentdebug.hub.Scrubber` | ✅ stable | 12 redactions + idempotent |
+| Hub backends | `LocalHubBackend`, `GitHubBackend`, `HuggingFaceBackend` | ✅ stable | local-bare-git + local |
+| Adapters | `agentdebug.adapters.raw` (`trace_loop`, `mark_step`) | ✅ stable | end-to-end + ctxvar |
+| Adapters | `agentdebug.adapters.langgraph.LangChainCallbackAdapter` | ✅ stable | gracefully degrades w/o dep |
+| Adapters | `agentdebug.adapters.otel.OTelExportAdapter` | ✅ stable | branch test |
+| Integrations | `agentdebug.integrations.claude_skill` | ✅ stable | skill-bundle write |
+| Integrations | `agentdebug.integrations.openhands` (microagent + bridge) | ⚠️ microagent stable; bridge needs live OpenHands | microagent YAML test |
+| CLI | `agentdebug.cli` (`analyze | judge | deep | list | show | hub | integrations | serve | doctor`) | ✅ stable | 12 subcommand smoke tests |
+| Local UI | `agentdebug.ui` (FastAPI + vanilla JS console) | ✅ stable | endpoint round-trip |
+
+**Test counts:** 60+ unit tests + 1 live-LLM smoke test, `mypy --strict` clean
+across 32 source files.
+
+## 2. Designed in docs, not yet implemented
+
+| Doc | Component | Why deferred | Realistic ship |
+|---|---|---|---|
+| [06_detectors.md](./06_detectors.md) | `trajectory_perplexity` (TrajAD) | needs token-level LM perplexity API or embedding model + baseline calibration | v0.3 |
+| [06_detectors.md](./06_detectors.md) | `topic_drift` (embedding cosine) | needs embedding client; consider reusing `OpenAICompatClient` `/embeddings` | v0.3 |
+| [06_detectors.md](./06_detectors.md) | LTL spec monitors | requires user-supplied spec or LLM-synthesized monitors; gated on RV research | v1.2 |
+| [07_attribution.md](./07_attribution.md) | `BinarySearchAttributor` (ddmin) | requires replayable environment; few frameworks expose it | v0.3 |
+| [07_attribution.md](./07_attribution.md) | `CounterfactualAttributor` | requires re-rolling agent actions; same replay constraint | v0.3 |
+| [07_attribution.md](./07_attribution.md) | `SBFLAttributor` (Tarantula/Ochiai) | needs corpus of passing + failing traces of same task; gated on Hub adoption | v0.4 |
+| [07_attribution.md](./07_attribution.md) | `DeltaDebugAttributor` (Zeller) | same replay constraint | v0.3 |
+| [07_attribution.md](./07_attribution.md) | `EnsembleAttributor` | trivial once 2+ heavy backends ship; awaits BinarySearch/Counterfactual | v0.3 |
+| [08_recovery.md](./08_recovery.md) | `SelfRefineLoop` | small but needs a generator-critic-refiner orchestration | v0.3 |
+| [08_recovery.md](./08_recovery.md) | `CriticRecoverer` | needs a verifier registry (search, code-exec, type-check) | v0.3 |
+| [08_recovery.md](./08_recovery.md) | `AutoManualRules` | needs persistent project manual + injection into next-run prompts | v0.3 |
+| [08_recovery.md](./08_recovery.md) | `LangGraphRewind` | depends on LangGraph checkpointer; ships when we have a real LangGraph user | v0.3 |
+| [08_recovery.md](./08_recovery.md) | `SagaRollback` | needs compensation registry on tool definitions; new schema | v0.3 |
+| [08_recovery.md](./08_recovery.md) | `MCTSBranchExploration` (LATS) | heavy; v2 feature | v2.0 |
+| [09_error_database.md](./09_error_database.md) | DuckDB analytical + Parquet archive | optional; Hub bundles already give per-project corpus | v0.3 |
+| [09_error_database.md](./09_error_database.md) | Vector similarity search | needs embedding model + index choice | v0.3 |
+| [10_taxonomy_induction.md](./10_taxonomy_induction.md) | TnT-LLM + BERTopic pipeline | needs ≥ 1k labeled traces to be useful | v0.4 |
+| [11_multimodal.md](./11_multimodal.md) | Screenshot/DOM capture, VLM judge | gated on multimodal user (Claude Computer Use / OpenAI CUA / OpenHands browser) | v1.1 |
+| [12_ui_dashboard.md](./12_ui_dashboard.md) | TUI (Textual) | low priority; CLI + web UI cover the use cases | v0.4 |
+| [12_ui_dashboard.md](./12_ui_dashboard.md) | VSCode extension | needs TS extension scaffolding | v1.0 |
+| [05_adapters.md](./05_adapters.md) | CrewAI, OpenAI Agents SDK, AutoGen, smolagents, LlamaIndex, DSPy, Pydantic-AI | each is ~150 LOC + conformance test; ship as users land | rolling |
+
+## 3. Implementation gaps surfaced by the audit
+
+The audit found one real bug and a handful of test gaps:
+
+1. **`agentdebug.hub.build_manifest` was used by the CLI but not re-exported** —
+   would have surfaced as `ImportError` for any user calling
+   `agentdebug hub push`. Fixed in 0.2.2 (`hub/__init__.py`) and locked in by a
+   CLI smoke test.
+2. **`cli.py` had 0% coverage** — every subcommand now has a smoke test that
+   exercises the argparse path; the LLM-required commands assert the
+   "missing credentials" exit code without hitting the network.
+3. **`instrumentation.py` (`traced_tool`) had 0% coverage** — happy and
+   exception paths now tested.
+4. **`llm.OpenAICompatClient.complete` had no test** — covered by a custom
+   `httpx.BaseTransport` that returns canned JSON without a network call.
+5. **`recovery.ReflexionSuggestion`** had only an indirect test from DeepDebug
+   examples; now has direct happy + empty tests.
+
+## 4. Coverage matrix (post-0.2.2)
+
+Run `PYTHONPATH=src pytest --cov=agentdebug --cov-report=term`. The two largest
+remaining gaps are deliberate:
+
+- `agentdebug.adapters.langgraph` — exercised only when `langchain_core` is
+  installed. The status-test verifies graceful degradation when it isn't.
+- `agentdebug.hub.backends.HuggingFaceBackend` — gated on `huggingface_hub`.
+  Round-tripping through real HF requires `HF_TOKEN`; covered by the local
+  bare-git test for the analogous push/pull flow.
+
+## 5. Acceptance gates for v0.3 (next minor)
+
+Before v0.3 ships, this doc should record green checkmarks for:
+
+- [ ] One replayable counterfactual attributor (`BinarySearchAttributor` is
+      the cheapest entry).
+- [ ] One tool-grounded recovery strategy (`CriticRecoverer`) wired against
+      a `Verifier` Protocol.
+- [ ] One additional framework adapter that goes through the full conformance
+      suite (CrewAI is the most-requested).
+- [ ] HuggingFace Hub round-trip live test (gated on `HF_TOKEN`).
+- [ ] Bench harness extended with one published-benchmark loader (Who&When
+      is the obvious first target — we already cite its method).

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/docs/README.md

@@ -31,6 +31,8 @@ This `docs/` directory contains the full design specification.
 | 19 | [19_error_hub.md](./19_error_hub.md) | **Error Hub** — bundle format, Local / Git / HF backends, scrubbing |
 | 20 | [20_deep_debug.md](./20_deep_debug.md) | **DeepDebug** — iterative multi-turn analysis (plan → hypothesize → verify → refine) |
 | 21 | [21_integrations.md](./21_integrations.md) | **Claude Code Skill** + **OpenHands** microagent + EventStream bridge |
+| 22 | [22_industry_track_paper_eval_plan.md](./22_industry_track_paper_eval_plan.md) | EMNLP Industry Track writing strategy + benchmark / human-study evaluation plan |
+| 23 | [23_status_v0_2.md](./23_status_v0_2.md) | **Capability + test coverage status (v0.2.2)** — what's implemented, what's tested, what's specced but not built |
 
 Plus three **narrative** docs that pre-dated this engineering spec and are kept for paper-style framing:
 
@@ -45,7 +47,7 @@ Plus three **narrative** docs that pre-dated this engineering spec and are kept
 ## How to read this
 
 - **First-time reader:** start with [00_overview.md](./00_overview.md), then [02_architecture.md](./02_architecture.md), then [14_api_reference.md](./14_api_reference.md).
-- **Researcher / paper author:** read [01_literature_survey.md](./01_literature_survey.md), [03_taxonomy.md](./03_taxonomy.md), [07_attribution.md](./07_attribution.md), [10_taxonomy_induction.md](./10_taxonomy_induction.md).
+- **Researcher / paper author:** read [01_literature_survey.md](./01_literature_survey.md), [03_taxonomy.md](./03_taxonomy.md), [07_attribution.md](./07_attribution.md), [10_taxonomy_induction.md](./10_taxonomy_induction.md), and [22_industry_track_paper_eval_plan.md](./22_industry_track_paper_eval_plan.md).
 - **Framework integrator:** read [04_trace_schema.md](./04_trace_schema.md), [05_adapters.md](./05_adapters.md), [13_class_design.md](./13_class_design.md).
 - **UI / product:** read [12_ui_dashboard.md](./12_ui_dashboard.md), [09_error_database.md](./09_error_database.md).
 - **Runtime / agent UX designer:** read [17_claude_code_design_patterns.md](./17_claude_code_design_patterns.md), then [02_architecture.md](./02_architecture.md), [08_recovery.md](./08_recovery.md), and [14_api_reference.md](./14_api_reference.md).

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "agentdebugx"
-version = "0.2.1"
+version = "0.2.2"
 description = "Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`."
 authors = ["ULab @ UIUC <ulab@illinois.edu>"]
 license = "MIT"

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/src/agentdebug/__init__.py

@@ -15,6 +15,16 @@ from agentdebug.attribution import (
     Attributor,
     Blame,
     HeuristicAttributor,
+    StepByStepAttributor,
+)
+from agentdebug.detectors import (
+    Detector,
+    DetectorConfig,
+    RepeatedStateDetector,
+    RepeatedToolCallDetector,
+    StepCountLimitDetector,
+    default_detectors,
+    run_detectors,
 )
 from agentdebug.events import DEFAULT_BUS, BusEvent, EventBus, EventSubscription
 from agentdebug.models import (
@@ -44,8 +54,16 @@ __all__ = [
     'Blame',
     'BusEvent',
     'CascadeFrame',
+    'Detector',
+    'DetectorConfig',
+    'RepeatedStateDetector',
+    'RepeatedToolCallDetector',
+    'StepByStepAttributor',
+    'StepCountLimitDetector',
     'build_cascade',
+    'default_detectors',
     'format_traceback',
+    'run_detectors',
     'DEFAULT_BUS',
     'DiagnosticReport',
     'EventBus',
@@ -66,4 +84,4 @@ __all__ = [
     'get_failure_mode',
 ]
 
-__version__ = '0.2.1'
+__version__ = '0.2.2'

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/src/agentdebug/attribution.py

@@ -19,10 +19,10 @@ from __future__ import annotations
 
 import logging
 from dataclasses import dataclass, field
-from typing import Any, Dict, List, Optional, Protocol
+from typing import Any, Dict, List, Optional, Protocol, cast
 
 from agentdebug.llm import LLMClient, extract_json_block
-from agentdebug.models import AgentTrajectory, FailureFinding, new_id
+from agentdebug.models import AgentEvent, AgentTrajectory, FailureFinding, new_id
 
 LOG = logging.getLogger('agentdebug.attribution')
 
@@ -227,4 +227,173 @@ class AllAtOnceAttributor:
         return [str(value)]
 
 
-__all__ = ['Attributor', 'Blame', 'AttributionResult', 'HeuristicAttributor', 'AllAtOnceAttributor']
+_STEP_SYSTEM_PROMPT = """You are AgentDebugX-Attributor, scanning a failed
+agent trajectory one step at a time (the Who&When "Step-by-Step" method,
+arXiv:2505.00212).
+
+You will be given the goal, framework, the prior judge findings, AND a
+SINGLE step from the trajectory plus a short window of preceding steps for
+context. Decide whether THIS step is the decisive failure step.
+
+Respond ONLY with a JSON object matching this schema (no prose, no markdown):
+
+{
+  "is_failure_step": true | false,
+  "confidence": <float in [0,1]>,
+  "rationale": "<one sentence>",
+  "evidence": ["<short quoted evidence>", ...]
+}
+
+Be CONSERVATIVE: only return true when the trajectory evidence on this step
+specifically caused the cascading failure.
+"""
+
+
+class StepByStepAttributor:
+    """LLM-based attributor mirroring Who&When's Step-by-Step method.
+
+    Walks the trajectory in order, asking the LLM about each step. Returns
+    every step that answered ``is_failure_step=true`` as a Blame hypothesis,
+    sorted by step index. Costs O(N) LLM calls; pair with ``max_steps`` to
+    bound the budget for long trajectories.
+    """
+
+    id = 'step_by_step'
+
+    def __init__(
+        self,
+        llm: LLMClient,
+        *,
+        fallback: Optional[Attributor] = None,
+        max_steps: int = 30,
+        context_window: int = 3,
+        max_tokens: int = 1024,
+    ) -> None:
+        self.llm = llm
+        self.fallback: Attributor = fallback or HeuristicAttributor()
+        self.max_steps = max_steps
+        self.context_window = context_window
+        self.max_tokens = max_tokens
+
+    def attribute(
+        self,
+        trajectory: AgentTrajectory,
+        findings: List[FailureFinding],
+    ) -> AttributionResult:
+        events = list(trajectory.events)
+        if not events:
+            return self.fallback.attribute(trajectory, findings)
+        # Scan only the suffix of size max_steps so long traces stay affordable.
+        budget = min(self.max_steps, len(events))
+        scanned = events[-budget:]
+        hypotheses: List[Blame] = []
+        for idx, evt in enumerate(scanned):
+            absolute_index = len(events) - budget + idx
+            ctx_start = max(0, absolute_index - self.context_window)
+            context = events[ctx_start:absolute_index]
+            verdict = self._classify_step(
+                trajectory, findings, evt, context=context
+            )
+            if verdict is None or not verdict.get('is_failure_step'):
+                continue
+            hypotheses.append(Blame(
+                span_id=evt.event_id,
+                step_index=evt.step_index,
+                agent_name=evt.agent_name,
+                confidence=self._coerce_float(verdict.get('confidence'), 0.5),
+                rationale=str(verdict.get('rationale') or ''),
+                evidence=self._coerce_str_list(verdict.get('evidence')),
+                sources=[self.id],
+            ))
+        if not hypotheses:
+            return AttributionResult(
+                method=self.id,
+                hypotheses=[],
+                raw={'scanned_steps': len(scanned)},
+            )
+        hypotheses.sort(
+            key=lambda h: (
+                h.step_index is None,
+                h.step_index if h.step_index is not None else 10**9,
+            )
+        )
+        return AttributionResult(
+            method=self.id,
+            hypotheses=hypotheses,
+            raw={'scanned_steps': len(scanned)},
+        )
+
+    def _classify_step(
+        self,
+        trajectory: AgentTrajectory,
+        findings: List[FailureFinding],
+        event: 'AgentEvent',
+        *,
+        context: List['AgentEvent'],
+    ) -> Optional[Dict[str, Any]]:
+        findings_doc = '\n'.join(self._render_finding(f) for f in findings[:10]) \
+            or '(no judge findings yet)'
+        context_doc = '\n'.join(
+            f'context event_id={e.event_id} step={e.step_index} agent={e.agent_name}'
+            f' type={getattr(e.event_type, "value", e.event_type)}'
+            f' output={str(e.output)[:120]} error={str(e.error)[:120]}'
+            for e in context
+        ) or '(no preceding context)'
+        prompt = (
+            f'GOAL: {trajectory.goal!r}\n'
+            f'FRAMEWORK: {trajectory.framework!r}\n\n'
+            f'JUDGE FINDINGS:\n{findings_doc}\n\n'
+            f'PRECEDING CONTEXT:\n{context_doc}\n\n'
+            f'CANDIDATE STEP:\n'
+            f'  event_id={event.event_id}\n'
+            f'  step={event.step_index} agent={event.agent_name} '
+            f'module={event.module}\n'
+            f'  type={getattr(event.event_type, "value", event.event_type)}\n'
+            f'  input={str(event.input)[:300]}\n'
+            f'  output={str(event.output)[:300]}\n'
+            f'  error={str(event.error)[:300]}\n'
+        )
+        try:
+            result = self.llm.complete(
+                messages=[
+                    {'role': 'system', 'content': _STEP_SYSTEM_PROMPT},
+                    {'role': 'user', 'content': prompt},
+                ],
+                max_tokens=self.max_tokens,
+            )
+        except Exception as exc:  # pragma: no cover
+            LOG.warning('step_by_step LLM call failed at step %s: %s',
+                        event.step_index, exc)
+            return None
+        parsed = extract_json_block(result.text)
+        if parsed is None:
+            return None
+        return cast(Dict[str, Any], parsed)
+
+    def _render_finding(self, finding: FailureFinding) -> str:
+        return (
+            f'- mode={finding.failure_mode.mode_id} '
+            f'agent={finding.agent_name} step={finding.step_index} '
+            f'confidence={finding.confidence:.2f}'
+        )
+
+    @staticmethod
+    def _coerce_float(value: Any, default: float) -> float:
+        try:
+            return float(value)
+        except (TypeError, ValueError):
+            return default
+
+    @staticmethod
+    def _coerce_str_list(value: Any) -> List[str]:
+        if value is None:
+            return []
+        if isinstance(value, list):
+            return [str(v) for v in value]
+        return [str(value)]
+
+
+__all__ = [
+    'Attributor', 'Blame', 'AttributionResult',
+    'HeuristicAttributor', 'AllAtOnceAttributor', 'StepByStepAttributor',
+]

agentdebugx-0.2.2/src/agentdebug/detectors.py

@@ -0,0 +1,284 @@
+"""Lightweight rule + anomaly detectors that don't need an LLM.
+
+These were specified in `docs/06_detectors.md` but not implemented in the
+initial v0.1 ship. The :class:`HeuristicAnalyzer` covers per-event string
+matching; the detectors here cover *cross-event* signals (loops, repeated
+tool calls, no-op streaks) which the heuristic analyzer cannot see in a
+single pass.
+
+Each detector implements :class:`Detector` and returns a list of
+:class:`FailureFinding`. They compose with the existing analyzer pipeline
+via :func:`run_detectors`.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+from typing import List, Optional, Protocol
+
+from agentdebug.models import (
+    AgentEvent,
+    AgentTrajectory,
+    EventType,
+    FailureFinding,
+    FailureMode,
+    new_id,
+)
+from agentdebug.taxonomy import SEED_FAILURE_MODES
+
+LOG = logging.getLogger('agentdebug.detectors')
+
+
+@dataclass
+class DetectorConfig:
+    """Tunables for the rule + anomaly detectors."""
+
+    repeated_tool_call_threshold: int = 3
+    repeated_state_window: int = 4
+    repeated_state_threshold: int = 3
+    step_count_limit: int = 50
+
+
+class Detector(Protocol):
+    id: str
+
+    def detect(self, trajectory: AgentTrajectory) -> List[FailureFinding]:
+        ...
+
+
+# ---------------------------------------------------------------------------
+# RepeatedToolCallDetector
+# ---------------------------------------------------------------------------
+
+class RepeatedToolCallDetector:
+    """Flag a tool that's called with identical args >= threshold times.
+
+    Matches ``FM-1.3 step_repetition`` / ``planning.inefficient_plan``.
+    """
+
+    id = 'repeated_tool_call'
+
+    def __init__(self, *, threshold: int = 3) -> None:
+        self.threshold = threshold
+
+    def detect(self, trajectory: AgentTrajectory) -> List[FailureFinding]:
+        counts: dict[tuple[str, str], List[AgentEvent]] = {}
+        for evt in trajectory.events:
+            if evt.event_type != EventType.TOOL_CALL:
+                continue
+            key = (evt.agent_name, _normalize(evt.input))
+            counts.setdefault(key, []).append(evt)
+        findings: List[FailureFinding] = []
+        mode = SEED_FAILURE_MODES['planning.inefficient_plan']
+        for (agent, signature), events in counts.items():
+            if len(events) < self.threshold:
+                continue
+            target = events[-1]
+            findings.append(
+                FailureFinding(
+                    finding_id=new_id('finding'),
+                    failure_mode=mode,
+                    event_id=target.event_id,
+                    agent_name=target.agent_name,
+                    step_index=target.step_index,
+                    confidence=min(0.5 + 0.1 * len(events), 0.95),
+                    evidence=[
+                        f'{len(events)} identical TOOL_CALL events with '
+                        f'signature={_truncate(signature, 80)} on agent={agent}',
+                    ],
+                    suggestion=_suggestion(mode),
+                    metadata={'source': self.id, 'detected_count': len(events)},
+                )
+            )
+        return findings
+
+
+# ---------------------------------------------------------------------------
+# RepeatedStateDetector
+# ---------------------------------------------------------------------------
+
+class RepeatedStateDetector:
+    """Flag a sliding window where the agent's outputs/observations don't change.
+
+    Catches no-progress loops the per-event matcher cannot see (e.g., agent
+    keeps responding "checking..." or producing the same plan over several
+    steps). Matches ``planning.inefficient_plan``.
+    """
+
+    id = 'repeated_state'
+
+    def __init__(self, *, window: int = 4, threshold: int = 3) -> None:
+        if threshold > window:
+            raise ValueError('threshold must be <= window')
+        self.window = window
+        self.threshold = threshold
+
+    def detect(self, trajectory: AgentTrajectory) -> List[FailureFinding]:
+        events = [
+            e for e in trajectory.events
+            if e.event_type in {
+                EventType.OBSERVATION,
+                EventType.AGENT_STEP,
+                EventType.PLAN,
+                EventType.TOOL_RESULT,
+            }
+        ]
+        findings: List[FailureFinding] = []
+        if len(events) < self.window:
+            return findings
+        mode = SEED_FAILURE_MODES['planning.inefficient_plan']
+        flagged_event_ids: set[str] = set()
+        for i in range(len(events) - self.window + 1):
+            window = events[i : i + self.window]
+            sigs = [_normalize(_state_signature(e)) for e in window]
+            most_common, count = _mode_count(sigs)
+            if count < self.threshold:
+                continue
+            target = window[-1]
+            if target.event_id in flagged_event_ids:
+                continue
+            flagged_event_ids.add(target.event_id)
+            findings.append(
+                FailureFinding(
+                    finding_id=new_id('finding'),
+                    failure_mode=mode,
+                    event_id=target.event_id,
+                    agent_name=target.agent_name,
+                    step_index=target.step_index,
+                    confidence=min(0.5 + 0.1 * count, 0.9),
+                    evidence=[
+                        f'state repeated {count}x within window of '
+                        f'{self.window} events; signature={_truncate(most_common, 80)}',
+                    ],
+                    suggestion=_suggestion(mode),
+                    metadata={
+                        'source': self.id,
+                        'window': self.window,
+                        'repeated_count': count,
+                    },
+                )
+            )
+        return findings
+
+
+# ---------------------------------------------------------------------------
+# StepCountLimitDetector
+# ---------------------------------------------------------------------------
+
+class StepCountLimitDetector:
+    """Flag a trajectory that exceeded a configured step budget.
+
+    Matches ``AgentBench TLE`` and ``verification.premature_stop`` proxies.
+    """
+
+    id = 'step_count_limit'
+
+    def __init__(self, *, max_steps: int = 50) -> None:
+        self.max_steps = max_steps
+
+    def detect(self, trajectory: AgentTrajectory) -> List[FailureFinding]:
+        if len(trajectory.events) <= self.max_steps:
+            return []
+        target = trajectory.events[-1]
+        mode = SEED_FAILURE_MODES['planning.inefficient_plan']
+        return [FailureFinding(
+            finding_id=new_id('finding'),
+            failure_mode=mode,
+            event_id=target.event_id,
+            agent_name=target.agent_name,
+            step_index=target.step_index,
+            confidence=0.7,
+            evidence=[
+                f'{len(trajectory.events)} events exceeds configured '
+                f'max_steps={self.max_steps}',
+            ],
+            suggestion=_suggestion(mode),
+            metadata={'source': self.id, 'event_count': len(trajectory.events)},
+        )]
+
+
+# ---------------------------------------------------------------------------
+# Pipeline runner
+# ---------------------------------------------------------------------------
+
+def default_detectors(config: Optional[DetectorConfig] = None) -> List[Detector]:
+    cfg = config or DetectorConfig()
+    return [
+        RepeatedToolCallDetector(threshold=cfg.repeated_tool_call_threshold),
+        RepeatedStateDetector(
+            window=cfg.repeated_state_window,
+            threshold=cfg.repeated_state_threshold,
+        ),
+        StepCountLimitDetector(max_steps=cfg.step_count_limit),
+    ]
+
+
+def run_detectors(
+    trajectory: AgentTrajectory,
+    detectors: Optional[List[Detector]] = None,
+) -> List[FailureFinding]:
+    """Run a list of detectors over a trajectory and return merged findings."""
+    detectors = detectors or default_detectors()
+    out: List[FailureFinding] = []
+    for d in detectors:
+        try:
+            out.extend(d.detect(trajectory))
+        except Exception as exc:  # pragma: no cover - defensive
+            LOG.warning('detector %s raised: %s', d.id, exc)
+    return out
+
+
+# ---------------------------------------------------------------------------
+# helpers
+# ---------------------------------------------------------------------------
+
+def _state_signature(event: AgentEvent) -> str:
+    """A compact, comparison-friendly view of an event's observable state."""
+    return '|'.join([
+        getattr(event.event_type, 'value', str(event.event_type)),
+        str(event.agent_name or ''),
+        str(event.module or ''),
+        str(event.output or '')[:200],
+        str(event.error or '')[:200],
+    ])
+
+
+def _normalize(value: object) -> str:
+    text = '' if value is None else str(value)
+    return ' '.join(text.split())
+
+
+def _truncate(text: str, max_chars: int) -> str:
+    if len(text) > max_chars:
+        return text[:max_chars] + '…'
+    return text
+
+
+def _mode_count(items: List[str]) -> tuple[str, int]:
+    counts: dict[str, int] = {}
+    best: str = ''
+    best_count = 0
+    for it in items:
+        counts[it] = counts.get(it, 0) + 1
+        if counts[it] > best_count:
+            best_count = counts[it]
+            best = it
+    return best, best_count
+
+
+def _suggestion(mode: FailureMode) -> Optional[str]:
+    if mode.suggestion_templates:
+        return str(mode.suggestion_templates[0])
+    return None
+
+
+__all__ = [
+    'Detector',
+    'DetectorConfig',
+    'RepeatedStateDetector',
+    'RepeatedToolCallDetector',
+    'StepCountLimitDetector',
+    'default_detectors',
+    'run_detectors',
+]

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/src/agentdebug/hub/__init__.py

@@ -34,7 +34,13 @@ from agentdebug.hub.backends import (
     LocalHubBackend,
     backend_from_spec,
 )
-from agentdebug.hub.bundle import Bundle, BundleManifest, pack_bundle, unpack_bundle
+from agentdebug.hub.bundle import (
+    Bundle,
+    BundleManifest,
+    build_manifest,
+    pack_bundle,
+    unpack_bundle,
+)
 from agentdebug.hub.scrub import (
     DEFAULT_REDACTIONS,
     ScrubReport,
@@ -53,6 +59,7 @@ __all__ = [
     'ScrubReport',
     'Scrubber',
     'backend_from_spec',
+    'build_manifest',
     'pack_bundle',
     'parse_spec',
     'scrub_trajectory',

{agentdebugx-0.2.1 → agentdebugx-0.2.2}/src/agentdebug/ui/server.py

@@ -2,12 +2,12 @@
 
 Endpoints:
 
-* ``GET  /``                       — single-page HTML console.
-* ``GET  /api/v1/traces``          — list trace IDs in the store.
-* ``GET  /api/v1/traces/{tid}``    — fetch a trajectory + freshly analyzed report.
-* ``GET  /api/v1/traces/{tid}/raw``— raw trajectory JSON.
-* ``GET  /api/v1/taxonomy``        — list seed failure modes.
-* ``GET  /healthz``                — liveness.
+* ``GET  /``                       - single-page HTML console.
+* ``GET  /api/v1/traces``          - list trace IDs in the store.
+* ``GET  /api/v1/traces/{tid}``    - fetch a trajectory + freshly analyzed report.
+* ``GET  /api/v1/traces/{tid}/raw``- raw trajectory JSON.
+* ``GET  /api/v1/taxonomy``        - list seed failure modes.
+* ``GET  /healthz``                - liveness.
 
 The server is intentionally tiny and built on a no-build (vanilla JS) frontend
 so it ships with the wheel and runs without `npm`.
@@ -126,7 +126,7 @@ def store_from_path(path: str) -> TraceStore:
     return JsonlTraceStore(path)
 
 
-# Single-file HTML console. Plain DOM + fetch — no build step required.
+# Single-file HTML console. Plain DOM + fetch; no build step required.
 _INDEX_HTML = """<!doctype html>
 <html lang="en">
 <head>
@@ -394,7 +394,7 @@ function eventProblem(ev) {
   const payload = (fmt(ev.error) + ' ' + fmt(ev.output) + ' ' + fmt(ev.metadata)).toLowerCase();
   return Boolean(ev.error || payload.includes('missing context') || payload.includes('premature') || payload.includes('loop') || payload.includes('handoff'));
 }
-async function loadTraceList() {
+async function loadTraceList(selectFirst) {
   const data = await api('/api/v1/traces');
   const ul = document.getElementById('trace-list');
   ul.innerHTML = '';
@@ -407,7 +407,7 @@ async function loadTraceList() {
     li.dataset.tid = tid;
     li.onclick = () => { selectTrace(tid, li); };
     ul.appendChild(li);
-    if (idx === 0) selectTrace(tid, li);
+    if (idx === 0 && selectFirst) selectTrace(tid, li);
   });
   if (data.traces.length === 0) {
     document.getElementById('detail').innerHTML = '<div class="empty">No traces in store.</div>';
@@ -442,6 +442,7 @@ function renderTrace(traj, report) {
   const events = traj.events || [];
   const findings = report.findings || [];
   const rootId = report.root_cause_event_id;
+  const alignmentEvents = events.filter(ev => ev.step_index !== null && ev.step_index !== undefined);
   const families = [...new Set(findings.map(f => f.failure_mode?.family).filter(Boolean))];
   const errorEvents = events.filter(eventProblem).length;
   const rootEvent = events.find(ev => ev.event_id === rootId) || {};
@@ -467,7 +468,7 @@ function renderTrace(traj, report) {
   html += '<div class="trace-legend"><div class="legend-cell"><div class="legend-label">Agent native trace</div><div class="legend-title">What the agent logged, thought, called, or observed.</div></div>';
   html += '<div class="legend-cell"><div class="legend-label">AgentDebugX error trace</div><div class="legend-title">Normalized failure signal, attribution, and repair hint for human review.</div></div></div>';
   html += '<div class="timeline">';
-  for (const ev of events) html += renderEvent(ev, ev.event_id === rootId, findingForEvent(findings, ev.event_id));
+  for (const ev of alignmentEvents) html += renderEvent(ev, ev.event_id === rootId, findingForEvent(findings, ev.event_id));
   html += '</div></div></div>';
 
   html += '<div class="rail">';
@@ -589,7 +590,7 @@ if (BOOTSTRAP && BOOTSTRAP.traces) {
     document.getElementById('detail').innerHTML = '<div class="empty">No traces in store.</div>';
   }
 }
-loadTraceList();
+loadTraceList(!(BOOTSTRAP && BOOTSTRAP.selected));
 </script>
 </body>
 </html>