agentdebugx 0.1.0__tar.gz

agentdebugx-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Haofei Yu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agentdebugx-0.1.0/PKG-INFO

@@ -0,0 +1,217 @@
+Metadata-Version: 2.4
+Name: agentdebugx
+Version: 0.1.0
+Summary: Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`.
+License: MIT
+License-File: LICENSE
+Keywords: llm,agents,debugging,observability,failure-analysis,agent-debugging,agentic-ai,tracing,evaluation
+Author: ULab @ UIUC
+Author-email: ulab@illinois.edu
+Requires-Python: >=3.9,<4.0
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Debuggers
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: System :: Monitoring
+Classifier: Typing :: Typed
+Provides-Extra: all
+Provides-Extra: langgraph
+Provides-Extra: otel
+Provides-Extra: ui
+Requires-Dist: httpx (>=0.24,<1.0)
+Requires-Dist: pydantic (>=1.10,<3.0)
+Project-URL: Documentation, https://github.com/ulab-uiuc/AgentDebugX/tree/main/docs
+Project-URL: Homepage, https://github.com/ulab-uiuc/AgentDebugX
+Project-URL: Repository, https://github.com/ulab-uiuc/AgentDebugX
+Description-Content-Type: text/markdown
+
+# AgentDebugX
+
+AgentDebugX is an open-source framework for tracing, diagnosing, and recovering
+from failures in agentic AI systems. It is packaged as `agentdebug` so existing
+agent stacks can add structured debugging with a small import:
+
+```python
+from agentdebug import AgentDebug, EventType, SQLiteTraceStore
+
+debugger = AgentDebug(store=SQLiteTraceStore(".agentdebug/errors.sqlite"))
+
+with debugger.trace(goal="Book a refundable flight", framework="my-agent") as trace:
+    trace.record(EventType.PLAN, agent_name="planner", output="Search flights first")
+    trace.record(
+        EventType.TOOL_RESULT,
+        agent_name="browser",
+        error="Timeout while loading checkout page",
+        step_index=3,
+    )
+    report = trace.analyze()
+
+print(report.summary)
+```
+
+## Vision
+
+Most observability tools show what an agent did. AgentDebugX aims to explain why
+it failed, who or which module caused the issue, when the first critical error
+occurred, and what should be changed next.
+
+The long-term scope is:
+
+- A portable trajectory IR for single-agent, multi-agent, tool-use, UI, and
+  multimodal runs.
+- An agentic error database for storing traces, diagnostic reports, taxonomy
+  labels, artifacts, and recovery outcomes.
+- Automated failure localization across step, module, agent, and run levels.
+- Automated taxonomy generation that clusters new failures and proposes new
+  labels when existing taxonomies are insufficient.
+- Recovery suggestions that can be used by humans, CI workflows, or self-healing
+  agents.
+- UI and notebook experiences for timeline debugging, attribution, replay, and
+  dataset curation.
+
+## Current Status
+
+This repository now contains the first framework skeleton:
+
+- `agentdebug.models`: normalized event, artifact, trajectory, taxonomy, and
+  diagnostic report schemas.
+- `agentdebug.recorder`: high-level `AgentDebug` and `TraceSession` APIs.
+- `agentdebug.analyzers`: deterministic baseline analyzer for immediate local
+  feedback.
+- `agentdebug.taxonomy`: seed taxonomy inspired by AgentDebug, MAST, Who&When,
+  AgentRx, AgentSight, and multimodal extensions.
+- `agentdebug.storage`: JSONL and SQLite trace stores.
+- `agentdebug.instrumentation`: lightweight function/tool tracing helper.
+- `agentdebug.cli`: initial `agentdebug analyze` command.
+
+Many advanced modules are intentionally not implemented yet. The design docs
+spell out the proposed path before we lock into heavy abstractions.
+
+## Install
+
+```bash
+# From PyPI (distribution name: agentdebugx; import as `agentdebug`)
+pip install agentdebugx
+
+# With the optional local dashboard
+pip install 'agentdebugx[ui]'
+
+# With LangChain/LangGraph callback adapter
+pip install 'agentdebugx[langgraph]'
+
+# With OpenTelemetry GenAI export shim
+pip install 'agentdebugx[otel]'
+
+# Everything
+pip install 'agentdebugx[all]'
+```
+
+From source:
+
+```bash
+pip install -e .         # or: poetry install
+```
+
+## Quick Start
+
+```python
+from agentdebug import AgentDebug, EventType
+from agentdebug.models import model_to_json
+
+debugger = AgentDebug()
+trajectory = debugger.start_trace(
+    goal="Find a paper and summarize the method",
+    framework="custom-react-agent",
+)
+
+debugger.record_event(
+    trajectory,
+    EventType.LLM_CALL,
+    agent_name="researcher",
+    module="planning",
+    step_index=1,
+    input="Find the latest AgentDebug paper",
+)
+debugger.record_event(
+    trajectory,
+    EventType.TOOL_RESULT,
+    agent_name="search",
+    module="action",
+    step_index=2,
+    error="JSON schema validation failed: missing parameter query",
+)
+debugger.finish_trace(trajectory, success=False)
+
+report = debugger.analyze(trajectory)
+print(model_to_json(report, indent=2))
+```
+
+CLI:
+
+```bash
+# Run the rule-based analyzer (works offline, no LLM)
+agentdebug analyze examples/sample_trace.json --out .agentdebug/report.json
+
+# List traces in a store
+agentdebug list --store-sqlite .agentdebug/errors.sqlite
+
+# Run the LLM judge (requires AGENTDEBUG_LLM_BASE_URL + AGENTDEBUG_LLM_API_KEY)
+agentdebug judge examples/sample_trace.json --attribute
+
+# Launch the local dashboard at http://127.0.0.1:7777
+agentdebug serve --store-sqlite .agentdebug/errors.sqlite
+
+# Diagnose which adapters / integrations are available
+agentdebug doctor
+```
+
+End-to-end demo (judge + attribution + Reflexion recovery, all live against an
+LLM): `examples/llm_judge_demo.py`.
+
+## Benchmark
+
+`scripts/eval_v0_1.py` runs the rule analyzer + LLM judge + All-at-Once
+attributor across six realistic failing traces (action/format,
+planning/loop, verification, system/tool error, multi-agent handoff loss,
+memory retrieval). Latest results:
+[docs/benchmarks/v0_1_smoke.md](docs/benchmarks/v0_1_smoke.md).
+
+## Documentation
+
+Narrative (paper-style):
+
+- [Research Survey](docs/RESEARCH_SURVEY.md)
+- [Open-Source Development Plan](docs/OPEN_SOURCE_DEVELOPMENT_PLAN.md)
+- [Seed Error Taxonomy](docs/ERROR_TAXONOMY.md)
+
+Engineering spec (numbered docs 00–18):
+
+- [docs/README.md](docs/README.md) — full doc index
+- [docs/00_overview.md](docs/00_overview.md) — vision + north-star UX
+- [docs/02_architecture.md](docs/02_architecture.md) — 7-layer architecture
+- [docs/18_comparison_codex_vs_design.md](docs/18_comparison_codex_vs_design.md) — Codex scaffold ↔ design-spec reconciliation that drove v0.1
+
+## Development
+
+```bash
+pytest
+mypy --strict ./
+ruff check .
+```
+
+## License
+
+[MIT License](LICENSE) — see the `LICENSE` file. Original template copyright
+Haofei Yu, 2024; additions copyright ULab @ UIUC and contributors, 2026.
+

agentdebugx-0.1.0/README.md

@@ -0,0 +1,178 @@
+# AgentDebugX
+
+AgentDebugX is an open-source framework for tracing, diagnosing, and recovering
+from failures in agentic AI systems. It is packaged as `agentdebug` so existing
+agent stacks can add structured debugging with a small import:
+
+```python
+from agentdebug import AgentDebug, EventType, SQLiteTraceStore
+
+debugger = AgentDebug(store=SQLiteTraceStore(".agentdebug/errors.sqlite"))
+
+with debugger.trace(goal="Book a refundable flight", framework="my-agent") as trace:
+    trace.record(EventType.PLAN, agent_name="planner", output="Search flights first")
+    trace.record(
+        EventType.TOOL_RESULT,
+        agent_name="browser",
+        error="Timeout while loading checkout page",
+        step_index=3,
+    )
+    report = trace.analyze()
+
+print(report.summary)
+```
+
+## Vision
+
+Most observability tools show what an agent did. AgentDebugX aims to explain why
+it failed, who or which module caused the issue, when the first critical error
+occurred, and what should be changed next.
+
+The long-term scope is:
+
+- A portable trajectory IR for single-agent, multi-agent, tool-use, UI, and
+  multimodal runs.
+- An agentic error database for storing traces, diagnostic reports, taxonomy
+  labels, artifacts, and recovery outcomes.
+- Automated failure localization across step, module, agent, and run levels.
+- Automated taxonomy generation that clusters new failures and proposes new
+  labels when existing taxonomies are insufficient.
+- Recovery suggestions that can be used by humans, CI workflows, or self-healing
+  agents.
+- UI and notebook experiences for timeline debugging, attribution, replay, and
+  dataset curation.
+
+## Current Status
+
+This repository now contains the first framework skeleton:
+
+- `agentdebug.models`: normalized event, artifact, trajectory, taxonomy, and
+  diagnostic report schemas.
+- `agentdebug.recorder`: high-level `AgentDebug` and `TraceSession` APIs.
+- `agentdebug.analyzers`: deterministic baseline analyzer for immediate local
+  feedback.
+- `agentdebug.taxonomy`: seed taxonomy inspired by AgentDebug, MAST, Who&When,
+  AgentRx, AgentSight, and multimodal extensions.
+- `agentdebug.storage`: JSONL and SQLite trace stores.
+- `agentdebug.instrumentation`: lightweight function/tool tracing helper.
+- `agentdebug.cli`: initial `agentdebug analyze` command.
+
+Many advanced modules are intentionally not implemented yet. The design docs
+spell out the proposed path before we lock into heavy abstractions.
+
+## Install
+
+```bash
+# From PyPI (distribution name: agentdebugx; import as `agentdebug`)
+pip install agentdebugx
+
+# With the optional local dashboard
+pip install 'agentdebugx[ui]'
+
+# With LangChain/LangGraph callback adapter
+pip install 'agentdebugx[langgraph]'
+
+# With OpenTelemetry GenAI export shim
+pip install 'agentdebugx[otel]'
+
+# Everything
+pip install 'agentdebugx[all]'
+```
+
+From source:
+
+```bash
+pip install -e .         # or: poetry install
+```
+
+## Quick Start
+
+```python
+from agentdebug import AgentDebug, EventType
+from agentdebug.models import model_to_json
+
+debugger = AgentDebug()
+trajectory = debugger.start_trace(
+    goal="Find a paper and summarize the method",
+    framework="custom-react-agent",
+)
+
+debugger.record_event(
+    trajectory,
+    EventType.LLM_CALL,
+    agent_name="researcher",
+    module="planning",
+    step_index=1,
+    input="Find the latest AgentDebug paper",
+)
+debugger.record_event(
+    trajectory,
+    EventType.TOOL_RESULT,
+    agent_name="search",
+    module="action",
+    step_index=2,
+    error="JSON schema validation failed: missing parameter query",
+)
+debugger.finish_trace(trajectory, success=False)
+
+report = debugger.analyze(trajectory)
+print(model_to_json(report, indent=2))
+```
+
+CLI:
+
+```bash
+# Run the rule-based analyzer (works offline, no LLM)
+agentdebug analyze examples/sample_trace.json --out .agentdebug/report.json
+
+# List traces in a store
+agentdebug list --store-sqlite .agentdebug/errors.sqlite
+
+# Run the LLM judge (requires AGENTDEBUG_LLM_BASE_URL + AGENTDEBUG_LLM_API_KEY)
+agentdebug judge examples/sample_trace.json --attribute
+
+# Launch the local dashboard at http://127.0.0.1:7777
+agentdebug serve --store-sqlite .agentdebug/errors.sqlite
+
+# Diagnose which adapters / integrations are available
+agentdebug doctor
+```
+
+End-to-end demo (judge + attribution + Reflexion recovery, all live against an
+LLM): `examples/llm_judge_demo.py`.
+
+## Benchmark
+
+`scripts/eval_v0_1.py` runs the rule analyzer + LLM judge + All-at-Once
+attributor across six realistic failing traces (action/format,
+planning/loop, verification, system/tool error, multi-agent handoff loss,
+memory retrieval). Latest results:
+[docs/benchmarks/v0_1_smoke.md](docs/benchmarks/v0_1_smoke.md).
+
+## Documentation
+
+Narrative (paper-style):
+
+- [Research Survey](docs/RESEARCH_SURVEY.md)
+- [Open-Source Development Plan](docs/OPEN_SOURCE_DEVELOPMENT_PLAN.md)
+- [Seed Error Taxonomy](docs/ERROR_TAXONOMY.md)
+
+Engineering spec (numbered docs 00–18):
+
+- [docs/README.md](docs/README.md) — full doc index
+- [docs/00_overview.md](docs/00_overview.md) — vision + north-star UX
+- [docs/02_architecture.md](docs/02_architecture.md) — 7-layer architecture
+- [docs/18_comparison_codex_vs_design.md](docs/18_comparison_codex_vs_design.md) — Codex scaffold ↔ design-spec reconciliation that drove v0.1
+
+## Development
+
+```bash
+pytest
+mypy --strict ./
+ruff check .
+```
+
+## License
+
+[MIT License](LICENSE) — see the `LICENSE` file. Original template copyright
+Haofei Yu, 2024; additions copyright ULab @ UIUC and contributors, 2026.

agentdebugx-0.1.0/docs/00_overview.md

@@ -0,0 +1,138 @@
+# 00 — Project Overview
+
+## 1. The problem
+
+LLM agent frameworks have proliferated — AutoGen, LangGraph, CrewAI, OpenAI Agents SDK, OpenHands, smolagents, LlamaIndex, DSPy, Pydantic-AI, MetaGPT, Camel-AI — but **failure analysis has not kept up**. The field is fragmented along four axes:
+
+1. **No common trace schema.** Each framework emits its own event format; observability vendors (LangSmith, Langfuse, AgentOps, Phoenix, Opik, Weave) each reimplement integration per framework, and traces cannot be compared across frameworks.
+2. **No standard error taxonomy.** Every paper invents one — AgentDebug's 5-module taxonomy (memory/reflection/planning/action/system; Zhu et al. arXiv:2509.25370), MAST's 14 multi-agent failure modes (Cemri et al. arXiv:2503.13657), Galileo's 7 operational families, τ-bench's 4 labels, AgentBench's TLE/CLE/IF/IA codes. These rarely cross-walk.
+3. **Attribution remains an open research problem.** Best published methods (Zhang et al. "Who & When", arXiv:2505.00212; AgenTracer, arXiv:2509.03312) hit ~53% agent-blame and ~14% step-blame accuracy. Counterfactual replay, spectrum-based fault localization, and classical delta debugging (Zeller 1999) have **never been benchmarked head-to-head** on agent traces.
+4. **Recovery is scattered.** Reflexion, Self-Refine, CRITIC, RCI, AutoManual, LATS, LangGraph checkpointing, SagaLLM compensating actions — each lives in its own repo with its own interface.
+
+The cost of this fragmentation is concrete: developers cannot answer "*which* agent broke, *when*, *why*, and *how do I fix it*" reliably for any non-trivial agent. Production agents ship with bespoke logging and ad-hoc post-mortems. Research papers cannot reuse each other's failure corpora.
+
+## 2. The vision
+
+**AgentDebugX is to LLM agents what OpenTelemetry + Sentry + a fault-tree analyzer is to distributed services.**
+
+A single import gives any agent:
+
+- **Uniform trajectory capture** in an OpenTelemetry-GenAI-aligned schema, regardless of framework.
+- **Layered error detection** — fast rule checks, anomaly detection, LLM-as-judge, spec monitors.
+- **Pluggable attribution** — which agent, which step, with confidence and counterfactual evidence.
+- **Composable recovery** — verbal reflection, refine loops, tool-grounded critique, saga rollback.
+- **A community error database** — every trace contributes (opt-in) to a versioned, searchable corpus of agent failures keyed by taxonomy.
+- **Automated taxonomy induction** — as the corpus grows, AgentDebugX surfaces new failure modes and proposes taxonomy revisions.
+- **Concrete fix suggestions** — for each detected error, an LLM-grounded recommendation backed by similar resolved cases.
+- **Multimodal support** — screenshots, DOM snapshots, click traces for computer-use / GUI agents.
+- **A web UI and IDE integration** for trace browsing, time-travel debugging, and review workflows.
+
+## 3. Scope
+
+### In scope (v0 → v1)
+
+- Python library with adapters for the 12+ frameworks listed above.
+- Canonical trace schema based on OTel GenAI semantic conventions.
+- AgentDebugX-Tax v0.1 — MAST-14 + AgentDebug-5 + security/tool/grounding extensions.
+- v1 detectors: rule-based, repeated-state, perplexity anomaly, LLM-judge.
+- v1 attribution: All-at-Once, Step-by-Step, Binary-Search (Who&When reproductions) + Counterfactual Replay.
+- v1 recovery: Reflexion memory, Self-Refine loop, CRITIC-with-tools, LangGraph-checkpoint rewind.
+- Local error database (SQLite + DuckDB) + optional sync to a hosted community instance.
+- Web UI for trace browsing, attribution review, fix suggestion.
+
+### In scope (v2)
+
+- Hierarchical delta debugging on traces.
+- Goal-conditioned TnT-LLM taxonomy induction.
+- Saga-pattern compensating-action layer.
+- LTL/temporal-logic monitor synthesis from natural-language specs.
+- HITL taxonomy refinement workflow for community contributions.
+- Multimodal trace recorder (screenshot + DOM diff per step).
+- AgentDebugX-as-a-Judge fine-tuned small model (AgenTracer-style).
+
+### Explicitly out of scope
+
+- Replacing an agent framework. AgentDebugX is *additive*, not a runtime.
+- Hosting LLM inference. We never call models the user didn't configure.
+- Acting as a commercial observability platform — we *export to* LangSmith/Langfuse/Phoenix, we are not a competitor.
+- Building yet another eval harness. We integrate with Inspect AI for evals.
+
+## 4. North-star UX
+
+```python
+# 1. Install
+# pip install agentdebugx  (PyPI distribution — short import is `agentdebug`)
+
+# 2. Add a handful of lines around your agent (v0.1 explicit API)
+from agentdebug import AgentDebug, EventType, SQLiteTraceStore
+
+debugger = AgentDebug(store=SQLiteTraceStore(".agentdebug/errors.sqlite"))
+with debugger.trace(goal="Book a refundable flight", framework="my-agent") as trace:
+    trace.record(EventType.PLAN, agent_name="planner", output="Search flights first")
+    trace.record(EventType.TOOL_RESULT, agent_name="browser",
+                 error="Timeout while loading checkout page", step_index=3)
+    report = trace.analyze()
+print(report.summary)
+
+# v0.2+ adds zero-line auto-attach: `agentdebug.init(project="my-agent")`
+# auto-detects the framework and attaches the right adapter.
+
+# 4. After it runs (or fails), inspect — v0.1
+report = debugger.analyze(trajectory)
+report.summary
+# -> Trace ID: tr_01J...
+#    Status: failed
+#    Detected errors: 2
+#      [HIGH]   FM-2.3 task_derailment at step 7 (agent: researcher)
+#      [MEDIUM] tool_arg_format_error at step 4 (tool: search_web)
+#    Attribution: step 4 caused the cascade (confidence 0.71, method: counterfactual)
+#    Suggested fix: tighten search_web JSON schema; add a retry-with-clarification loop
+#    Similar past cases: 14 (see https://errors.agentdebugx.dev/q/tr_01J...)
+
+# 5. Open the dashboard
+agentdebugx.serve()  # → http://localhost:7777
+```
+
+For v0.1, the LLM judge + attributor are wired directly:
+
+```python
+from agentdebug import AgentDebug, SQLiteTraceStore
+from agentdebug.llm import OpenAICompatClient
+from agentdebug.judges import LLMJudgeAnalyzer
+from agentdebug.attribution import AllAtOnceAttributor
+
+llm = OpenAICompatClient(base_url="...", api_key="...", model="gemini-3-flash")
+debugger = AgentDebug(store=SQLiteTraceStore())
+trajectory = debugger.start_trace(goal="...", framework="langgraph")
+# ... record events ...
+debugger.finish_trace(trajectory, success=False)
+
+report = LLMJudgeAnalyzer(llm=llm).analyze(trajectory)
+blame  = AllAtOnceAttributor(llm=llm).attribute(trajectory, report.findings)
+```
+
+## 5. Design principles
+
+1. **OTel-first.** Wire format is OpenTelemetry GenAI semantic conventions. Free interop with Phoenix, Langfuse, Datadog, SigNoz, MLflow, Logfire.
+2. **Framework-agnostic.** No assumption about which agent framework is in use; auto-detect.
+3. **Thin adapters.** Each framework adapter is ≤ 200 LOC and uses the framework's native extension point (callback / hook / event bus / OTel) — never deep monkey-patching unless that's the only option.
+4. **Composable, not monolithic.** Detectors, attributors, recoverers all share small interfaces and can be mixed.
+5. **Honest about uncertainty.** Attribution and recovery surface confidence scores. We never claim "the agent broke at step 7" — we say "candidate hypothesis: step 7 (confidence 0.71, evidence: …)".
+6. **Local-first, community-augmented.** Local SQLite database by default; opt-in sync to the community corpus. User owns the data.
+7. **Open governance.** Taxonomy and schema evolve via public RFCs.
+8. **Skeptical of marketing.** Galileo-style "7 modes" are derivative repackagings. We cite primary methodologies.
+
+## 6. Non-goals stated as risks
+
+- **We will not present attribution as ground truth.** SOTA is too unreliable.
+- **We will not auto-apply recovery actions that have side effects** (sending emails, DB writes) without explicit user opt-in and a saga-pattern compensation registered.
+- **We will not lock users into our taxonomy.** Custom taxonomy registries are first-class.
+
+## 7. Naming
+
+- PyPI distribution: `agentdebugx`
+- Python import: `agentdebug` (short, ergonomic — chosen for adoption UX)
+- Repository: `AgentDebugX` (brand)
+- Default taxonomy: `AgentDebugX-Tax v0.1` — 19 seed modes (see [03_taxonomy.md](./03_taxonomy.md))
+- Community corpus: `ADBXCorpus` (deferred to v0.3)
+- Reference web UI: `AgentDebugX Console` (deferred to v0.2)