proveai-agentpulse 0.2.0__tar.gz

proveai_agentpulse-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Prove AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

proveai_agentpulse-0.2.0/PKG-INFO

@@ -0,0 +1,361 @@
+Metadata-Version: 2.4
+Name: proveai-agentpulse
+Version: 0.2.0
+Summary: Observability and drift detection for multi-agent systems (AutoGen, LangChain/LangGraph)
+License: MIT License
+        
+        Copyright (c) 2026 Prove AI
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: flask>=3.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Provides-Extra: ai
+Requires-Dist: anthropic>=0.40; extra == "ai"
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0; extra == "mcp"
+Provides-Extra: otel
+Requires-Dist: opentelemetry-sdk>=1.24; extra == "otel"
+Requires-Dist: opentelemetry-api>=1.24; extra == "otel"
+Provides-Extra: all
+Requires-Dist: proveai-agentpulse[ai,mcp,otel]; extra == "all"
+Dynamic: license-file
+
+# AgentPulse
+
+[![tests](https://github.com/prove-ai/agentpulse/actions/workflows/tests.yml/badge.svg)](https://github.com/prove-ai/agentpulse/actions/workflows/tests.yml)
+[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+> Traces tell you what happened. They don't tell you where to start investigating.
+
+AgentPulse is an **open-source reference implementation** of drift investigation for multi-agent systems. It turns raw agent traces (tokens, cost, latency, tool calls, handoffs, DAG structure) into run, agent, handoff, route, and drift views. When an outcome degrades, it traces the drift upstream through the handoff graph to the component where it originated and presents the whole causal path.
+
+**Who it's for.** Teams building in-house observability for multi-agent systems, and anyone exploring how agent failures should be investigated. Use the ideas, the schema, or the whole thing.
+
+**What it's not.** A production observability platform. If you need managed tracing at scale today, use Langfuse, LangSmith, or similar tools. AgentPulse explores the layer above traces: the investigation.
+
+If you try it, I would like to hear what worked and what did not. See [Feedback](#feedback).
+
+![Drift Investigation](docs/screenshots/light/drift-investigation.png)
+
+*The core view: writer drifted after a prompt and model change at run 18, and critic's success dropped as a consequence. The fix belongs in writer, not critic.*
+
+---
+
+## Architecture
+
+One drift engine, three surfaces. The dashboard, the CLI, and the MCP server call the same engine functions, so they always agree on what "drift" means. Data flows left to right:
+
+```mermaid
+flowchart LR
+    A["your multi-agent app<br/>+ instrument()"] --> B["sdk/<br/>intercepts LLM calls,<br/>tool calls, handoffs"]
+    B --> C[("storage/<br/>SQLite, one file<br/>per project")]
+    C --> D["analysis/<br/>metrics, anomalies,<br/>drift, causal chains"]
+    D --> E["Dashboard (Flask)"]
+    D --> F["CLI (today-drift)"]
+    D --> G["MCP server"]
+    G --> H["Claude Code /<br/>Claude Desktop"]
+```
+
+In words:
+
+1. You launch your app through `agentpulse run` — no code changes. It patches the OpenAI and Anthropic SDKs inside your process and hooks AutoGen and LangChain when they are present. With those frameworks, every LLM call, tool call, and handoff is captured automatically; custom orchestrators are not officially supported (see [Framework support](#framework-support)).
+2. Captured events are written through `storage/` into a plain SQLite file per project, `~/.agentpulse/db/<project>.db`. No server, no agent daemon.
+3. The `analysis/` engine reads those runs and computes metrics, anomaly reports, drift findings, and causal chains.
+4. Three surfaces present the same findings: the Flask dashboard, the `today-drift` CLI, and the MCP server that Claude Code or Claude Desktop connects to.
+
+| Surface | What it's for |
+|---|---|
+| **Dashboard** (Flask) | Run explorer, timelines, DAGs, trends, and the Drift Investigation view |
+| **CLI** (`today-drift`) | Drift findings as readable cards in your terminal, useful as a daily standup check |
+| **MCP server** | The same findings inside Claude Code / Claude Desktop, so a coding agent can run the investigation |
+
+Works with the OpenAI and Anthropic SDKs. How much is captured automatically depends on your framework; see [Framework support](#framework-support).
+
+---
+
+## Ideas you can reuse
+
+Even if you never run AgentPulse, these design decisions carry over to any in-house build:
+
+1. **Detect agent, handoff, and route drift separately.** An agent getting slower, a handoff payload shrinking, and the execution path changing are different failure classes with different signals. Collapsing them into one "anomaly score" hides where to look. (`analysis/drift_detect.py`, `analysis/version_drift.py`)
+
+2. **Severity needs corroboration, not just magnitude.** A Tier-0 (outcome) breach only escalates to critical when co-timed strong supporting signals and a plausible change event line up. A lone moving metric stays a low-confidence candidate. (`analysis/drift_detect.py`, thresholds in `config/drift_rules.yaml`)
+
+3. **Trace the symptom to its upstream origination.** Walk the handoff graph upstream from the breached outcome and stop at the component whose input is stable but whose output drifted. That is where the fix belongs. The downstream agent that "failed" often did not change at all. (`analysis/drift_chains.py`)
+
+4. **A change can only explain a drift if it could have caused it.** Config, prompt, and model changes are attributed only when they happened at or before the drift start, on the same or an upstream component. (`_nearby_change` in `analysis/drift_detect.py`)
+
+5. **Give the investigation to a coding agent, not just a dashboard.** The MCP server exposes findings as structured, root-cause-led cards, so Claude can triage drift, compare releases, and propose next checks conversationally. (`src/agentpulse/mcp_server.py`, `.claude/skills/`)
+
+6. **Pin your metric engine with a snapshot test.** Every chart value for the sample data is pinned by a fixture. A refactor that silently changes a metric fails CI loudly. (`tests/test_series_snapshot.py`)
+
+---
+
+## Getting started
+
+### Step 1: install
+
+```bash
+pip install proveai-agentpulse
+```
+
+To try the dashboard with the bundled sample project first (a 4-agent content pipeline with 120 runs across 4 prompt versions and one real drift), clone the repo and point AgentPulse at its data:
+
+```bash
+git clone https://github.com/prove-ai/agentpulse.git
+AGENTPULSE_HOME=./agentpulse agentpulse dashboard
+```
+
+Open <http://localhost:5001> — the Drift Investigation view above is the first thing you can reproduce.
+
+### Step 2: capture your own system
+
+No code changes — launch your app through `agentpulse run` instead of `python`:
+
+```bash
+agentpulse run python main.py
+```
+
+If your system runs on a supported framework (see the table below), that is the whole integration. Every LLM call, agent turn, tool call, and handoff is captured into `~/.agentpulse/db/runs.db`. Then run `agentpulse dashboard` and your project appears in the sidebar picker. Your data stays on your machine — it's plain SQLite in your home directory.
+
+To monitor several systems side by side, give each its own database:
+
+```bash
+AGENTPULSE_DB=my-system agentpulse run python main.py
+```
+
+If you can't change how your app is launched (a notebook, a process manager), add two lines at the top of your entry point instead — this is exactly what `agentpulse run` does for you:
+
+```python
+import agentpulse
+agentpulse.instrument()
+```
+
+### Framework support
+
+`agentpulse run` is the whole story only when your framework tells AgentPulse where agent boundaries are:
+
+| Your stack | Integration effort | What you get |
+|---|---|---|
+| AutoGen (`SelectorGroupChat`) | `agentpulse run`, nothing else | Full capture: turns, tokens, tools, handoffs, termination |
+| LangChain, LangGraph, and frameworks built on LangChain callbacks (e.g. CrewAI) | `agentpulse run`, nothing else | Full capture: each chain/node reports its name through the callback system; tokens come from the SDK patches |
+| Plain OpenAI/Anthropic SDK with your own orchestration (e.g. `asyncio.gather`) | Not officially supported | — |
+
+The reason for the difference: AutoGen and LangChain expose agent boundaries through their event and callback systems, so AgentPulse can attribute every call to the right agent automatically. A hand-rolled orchestrator has no such signal, so automatic capture is not possible and custom pipelines are not officially supported. (If you really need one, the session primitives in [`src/agentpulse/sdk/session.py`](src/agentpulse/sdk/session.py) are what the adapters in [`src/agentpulse/sdk/patches/`](src/agentpulse/sdk/patches) drive — but you're off the supported path.)
+
+### What gets captured
+
+| Layer | What |
+|---|---|
+| Per-call | start/end timestamps, input/output tokens, model, latency |
+| Per-agent turn | aggregated tokens, duration, tool calls, status, parent agent |
+| Per-run | total cost, wall-clock, termination reason, prompt version |
+| DAG | parent → child edges (when your orchestrator exposes them), parallel branches, join waits |
+
+No API keys are needed to capture data or browse the dashboard; it only reads SQLite. An `ANTHROPIC_API_KEY` is needed only for the optional AI "suggest next checks" feature (see [Configuration](#configuration)).
+
+---
+
+## The dashboard
+
+### Drift Investigation (`/drift2`)
+The core view, shown at the top of this README: findings ranked by severity, the causal path, what changed, why it matters, potentially related changes, and suggested next checks.
+
+### Run explorer (`/`)
+Every captured run with status, route, and cost. Click through to the per-run detail page with an execution timeline (gantt), the interactive agent-chain DAG, anomalies vs the baseline, and parallel-group efficiency.
+
+![Run explorer](docs/screenshots/light/run-history.png)
+
+### Metrics Explorer (`/explore`)
+Chart any metric for any agent, handoff, or the whole system across runs, with the same baseline bands and version markers as Drift Investigation, plus custom metrics and thresholds.
+
+![Metrics Explorer](docs/screenshots/light/metrics-explorer.png)
+
+There is also a trend view (`/trends`) with agent health cards and a handoff health leaderboard, and an event timeline (`/timeline`) of prompt, model, and tool changes.
+
+---
+
+## CLI: `agentpulse drift`
+
+The drift findings as terminal cards, with no server and no Claude involved:
+
+```bash
+agentpulse drift                          # all projects, active drifts
+agentpulse drift --project demo           # one project
+agentpulse drift --range 7d               # narrower look-back window (default 30d)
+agentpulse drift --min-severity drift     # hide low-signal watches
+agentpulse drift --next demo:chain0       # next investigation checks for one finding
+agentpulse drift --compare --project demo # version comparison (baseline vs newest)
+agentpulse drift --compare --project demo --all   # step through every version pair
+```
+
+Sample output:
+
+```
+AgentPulse — 1 drift finding  · as of 2026-07-16 17:09 · range 30d · demo
+
+● writer  ·  Agent behaviour  ·  demo
+    Severity: Drift    Confidence: High
+    Path:    writer → … → critic
+    Why:     A change in writer propagated downstream to critic, whose outcome
+             (success) worsened. The fix belongs in writer, not critic.
+    Trigger: writer prompt changed near run 110
+    Metrics changed (3):
+        • writer latency_s +51.2%
+        • writer cost_usd +1446.9%
+        • critic success -9.3 pp
+    active 23d, since Jun 23  ·  id demo:chain0
+```
+
+Findings carry an `id` (like `demo:chain0`). Pass it to `--next` to get the recommended follow-up checks for that finding.
+
+There is also `agentpulse report`, a per-run metrics report for a single project (`agentpulse report --all --db demo`).
+
+---
+
+## MCP server: let Claude run the investigation
+
+`agentpulse mcp` (install with `pip install "proveai-agentpulse[mcp]"`) exposes the drift engine to Claude Code and Claude Desktop as three tools:
+
+| Tool | What it returns |
+|---|---|
+| `get_todays_finding` | The drift findings active right now, as root-cause-led investigation cards (which component drifted, why, how long it's been active) |
+| `get_version_comparison` | Baseline-vs-newest (or any pair, or every consecutive pair) version comparison, leading with the change that broke an outcome |
+| `get_next_check_steps` | Recommended next investigation steps for a finding. AI-generated when an `ANTHROPIC_API_KEY` is configured, deterministic checks otherwise |
+
+### Claude Code
+
+The repo ships with a project-scoped [`.mcp.json`](.mcp.json), so if you followed Getting started (venv at `.venv/`), just open Claude Code inside the repo and approve the server when prompted:
+
+```bash
+cd agentpulse
+claude
+```
+
+Then ask things like *"what drifted today?"*, *"compare versions of the demo project"*, or *"what should I check next for demo:chain0?"*.
+
+To register it from another directory instead:
+
+```bash
+claude mcp add agentpulse -- agentpulse mcp
+```
+
+The repo also bundles three Claude Code **skills** under [`.claude/skills/`](.claude/skills) that build on these tools: `drift-triage` (daily standup-style triage), `drift-root-cause-report` (a written root-cause report), and `release-regression-check` (did the last release break anything?).
+
+### Claude Desktop
+
+Add to `claude_desktop_config.json`. Use absolute paths, because Desktop spawns servers without a working directory:
+
+```json
+{
+  "mcpServers": {
+    "agentpulse": {
+      "command": "/path/to/your/venv/bin/agentpulse",
+      "args": ["mcp"]
+    }
+  }
+}
+```
+
+---
+
+## Configuration
+
+All AgentPulse data lives under one home directory, `~/.agentpulse` by default (override with `AGENTPULSE_HOME`):
+
+```
+~/.agentpulse/
+├── db/        one SQLite file per project
+├── config/    optional overrides, e.g. drift_rules.yaml
+└── .env       optional KEY=VALUE env file
+```
+
+Set `ANTHROPIC_API_KEY` in `~/.agentpulse/.env` (or your shell environment) to enable the AI "suggest next checks" feature (dashboard button, CLI `--next`, MCP `get_next_check_steps`). Everything else works without it; the MCP tool falls back to deterministic checks.
+
+Drift detection thresholds and handoff rules ship inside the package ([`src/agentpulse/config/drift_rules.yaml`](src/agentpulse/config/drift_rules.yaml)); to tune them, copy that file to `~/.agentpulse/config/drift_rules.yaml` and edit — it takes precedence, and the dashboard reloads it on every request, no restart needed.
+
+Per-run labels are set with environment variables when launching: `AGENTPULSE_DB` (project database name), `AGENTPULSE_TASK_TYPE` (groups like-with-like in drift reports), and `AGENTPULSE_PROMPT_VERSION` (bump when you change prompts). All are optional.
+
+---
+
+## What I want to learn
+
+This project is an experiment in how agent failures should be investigated. If you build or operate multi-agent systems, I want to know:
+
+- What telemetry do you actually collect for multi-agent systems, and what does AgentPulse's schema miss?
+- Does the agent / handoff / route drift split match how you triage failures?
+- Where is the drift detector wrong? Thresholds live in [`src/agentpulse/config/drift_rules.yaml`](src/agentpulse/config/drift_rules.yaml). If it over-fires or under-fires on your data, that is useful feedback.
+- Should agent observability stay a dashboard, or become structured context for a coding agent that performs the investigation? The MCP server is a bet on the second answer.
+
+## Feedback
+
+**Feedback is welcome.** Please open a [GitHub issue](https://github.com/prove-ai/agentpulse/issues) for questions, ideas, or bug reports. Telemetry schemas, stories about agent failures that were hard to localize, and design disagreements are all useful.
+
+For private feedback, contact me at [leyla@proveai.com](mailto:leyla@proveai.com).
+
+---
+
+## Project layout
+
+```
+agentpulse/
+├── pyproject.toml            Package definition (`pip install .`, `agentpulse` CLI)
+├── src/agentpulse/
+│   ├── sdk/                  Patches for OpenAI/Anthropic + the instrument() entry point
+│   ├── storage/              SQLite store (multi-DB aware via ContextVar)
+│   ├── analysis/             Metric engine: raw → derived → anomalies → trends → drift → DAG
+│   ├── reporter/             Flask dashboard + Jinja templates
+│   ├── config/               Default drift rules + prompt manifests (shipped in the package)
+│   ├── runner.py             `agentpulse run` — zero-code-change launcher
+│   ├── cli.py                The `agentpulse` command (run/report/dashboard/drift/mcp)
+│   ├── drift_cli.py          Drift findings as terminal cards
+│   ├── report.py             Per-run metrics report
+│   └── mcp_server.py         MCP server (3 tools over the same engine)
+├── scripts/                  Demo generators & import helpers
+├── tests/                    pytest suite (incl. a metric snapshot guard)
+├── .claude/skills/           Claude Code skills built on the MCP tools
+└── db/                       demo.db sample data
+```
+
+## Development
+
+```bash
+git clone https://github.com/prove-ai/agentpulse.git
+cd agentpulse
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e ".[all]" pytest
+python -m pytest tests/          # run the test suite
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the snapshot-test workflow and guidelines.
+
+## Requirements
+
+- Python 3.10+
+- Optional extras: `proveai-agentpulse[mcp]` (MCP server), `proveai-agentpulse[ai]` (AI next-check suggestions), `proveai-agentpulse[otel]` (OTel export), `proveai-agentpulse[all]`
+- The multi-agent system you observe needs `openai` and/or `anthropic` installed in **its** environment. AgentPulse patches whichever it finds; neither is a hard dependency of AgentPulse itself.
+
+## License
+
+[MIT](LICENSE)