agent-ablation 0.1.0 → 0.3.0

package/README.md

@@ -1,15 +1,19 @@
 # agent-ablation
 
-Leave-one-out ablation testing for multi-agent decision systems. You have a set of
-per-agent findings (scores, confidences, whatever your pipeline produces) and a
-function that turns those findings into a verdict. `agent-ablation` answers one
-question: **which of my agents' findings actually changed that verdict, and which
-were along for the ride?**
+[![CI](https://github.com/AyushCipher/agent-ablation/actions/workflows/ci.yml/badge.svg)](https://github.com/AyushCipher/agent-ablation/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/agent-ablation.svg)](https://www.npmjs.com/package/agent-ablation)
 
-It removes each finding one at a time, re-runs your decision function on what's
-left, and reports which removals flipped the outcome. Zero runtime dependencies,
-zero opinions about how your agents work — you supply the findings and the
-decision function, it does the leave-one-out loop and the bookkeeping.
+Leave-one-out ablation testing, backward elimination, and ROI evaluation for multi-agent systems. You have a set of per-agent findings (scores, confidences, telemetry) and a function or model that turns those findings into a verdict. `agent-ablation` answers the key production questions:
+
+1. **Load-Bearing Influence:** Which agents' findings actually changed the verdict, and which were along for the ride?
+2. **Cost & Token ROI:** How many dollars and tokens did each specialist burn per verdict flip? Does that small accuracy bump justify the API bill?
+3. **Correlated Agents & Pruning:** Which redundant agents can be safely pruned via greedy backward elimination without breaking the final verdict?
+4. **Protective vs. Harmful Signals:** When ground truth is provided, did an agent's presence prevent an error (protective), or did it cause a hallucination/false positive (harmful)?
+5. **Stochastic & Async LLM Judges:** Handles async decisions and repeated sampling with majority voting to filter out LLM temperature variance.
+
+Zero runtime dependencies. Native TypeScript.
+
+---
 
 ## Install
 
@@ -17,7 +21,11 @@ decision function, it does the leave-one-out loop and the bookkeeping.
 npm install agent-ablation
 ```
 
-## Quick example
+---
+
+## Core Features & Usage
+
+### 1. Basic Leave-One-Out Ablation
 
 ```typescript
 import { runAblation, type Finding } from "agent-ablation";
@@ -40,144 +48,189 @@ const findings: Finding[] = [
 const result = runAblation(findings, decide);
 
 console.log(result.baseline);        // "decline"
-console.log(result.loadBearingRatio); // fraction of agents whose removal changed the verdict
+console.log(result.loadBearingRatio); // 0.33 (1 out of 3 agents flipped the outcome)
 for (const p of result.perAgent) {
   console.log(p.removedAgentId, "->", p.verdictWithout, p.changed ? "(load-bearing)" : "");
 }
 ```
 
-For a batch of cases, `batchAblation` runs the same ablation over each one and
-aggregates the results — including, per agent, the fraction of cases in which
-removing that agent changed the outcome:
+---
+
+### 2. Cost & Token ROI Analysis ("Cost per Verdict Flip")
+
+Pass telemetry (`cost`, `tokens`, `latencyMs`) inside your findings. `batchAblation` computes the exact ROI metrics and identifies expensive agents with low decision impact:
 
 ```typescript
-import { batchAblation } from "agent-ablation";
+import { batchAblation, formatMarkdownReport, type Finding } from "agent-ablation";
+
+const cases: Finding[][] = [
+  [
+    { agentId: "expensive_reasoner", score: 90, cost: 0.15, tokens: 3000 },
+    { agentId: "cheap_heuristic", score: 10, cost: 0.002, tokens: 50 },
+  ],
+  [
+    { agentId: "expensive_reasoner", score: 20, cost: 0.15, tokens: 3000 },
+    { agentId: "cheap_heuristic", score: 85, cost: 0.002, tokens: 50 },
+  ],
+];
 
-const { results, summary } = batchAblation(allCases, decide);
+const { results, summary } = batchAblation(cases, decide);
 
-console.log(summary.averageLoadBearingRatio);
-console.log(summary.perAgentInfluence); // { transaction_pattern: 0.17, identity_signal: 0.83, ... }
+console.log(summary.roi?.agents.expensive_reasoner.costPerVerdictFlip); // Cost per decision flip
+console.log(summary.roi?.recommendations); // Automated pruning/downgrade advice
+
+// Format into a Markdown report for PRs or documentation
+console.log(formatMarkdownReport(summary));
 ```
 
-## Worked example: reproducing SentryMesh's 33% multi-signal-share finding
-
-[SentryMesh](https://github.com/AyushCipher/Sentry-Mesh) is a four-specialist
-multi-agent fraud investigation system. Its own eval harness runs an ablation over
-its 23-case bank and reports the result plainly in its README: of 9 cases the
-system auto-resolved without escalating to a human, **only 3 survive removal of
-their single loudest specialist — 6 collapse to `escalate`.** SentryMesh calls
-this "the most important number in the report," because it means two-thirds of
-those auto-decisions rested on one specialist's finding, with the other three
-specialists' LLM calls spent for nothing.
-
-Those six cases, straight from SentryMesh's ablation table:
-
-| Case | Decision | Remove | Becomes |
-|---|---|---|---|
-| SM-001 | auto_decline | `identity_signal` (100) | escalate |
-| SM-002 | auto_decline | `identity_signal` (75) | escalate |
-| SM-005 | auto_decline | `identity_signal` (80) | escalate |
-| SM-006 | auto_decline | `identity_signal` (70) | escalate |
-| SM-012 | auto_decline | `network_analysis` (90) | escalate |
-| SM-016 | auto_approve | `transaction_pattern` (26) | escalate |
-
-`tests/ablation.test.ts` in this repo reproduces all six as a worked example: it
-builds each case's four-specialist `Finding[]` (with the named specialist's score
-matching SentryMesh's table exactly), runs it through a decision function modeled
-on SentryMesh's own description of its aggregation — findings combine via
-noisy-OR, gated by panel confidence — and asserts that `runAblation` correctly
-identifies the named specialist's removal as the one that flips each case to
-`escalate`. It's the credibility anchor for this package: if `agent-ablation`
-couldn't reproduce a real, previously-published ablation result on real case
-data, it wouldn't be trustworthy on your data either.
-
-## Limitations
-
-**This tool detects verdict *change*, not verdict quality.** A flipped verdict
-after removing an agent tells you that agent was load-bearing for that decision —
-it says nothing about whether the original verdict or the post-removal one was
-*correct*. Conversely, a low `loadBearingRatio` is not automatically a flaw:
-redundancy across agents can be exactly what you want (independent corroboration
-is the point of running more than one specialist), and this tool has no way to
-distinguish "healthy redundancy" from "wasted compute" for you.
-
-**Leave-one-out misses agents that only matter in pairs.** This is a real gap,
-not a hedge. If removing agent A alone doesn't flip the verdict, and removing
-agent B alone doesn't either, but removing *both together* would, leave-one-out
-ablation will report both as not load-bearing. Catching that requires ablating
-combinations, which this package deliberately does not do — the combinatorics
-blow up fast, and a leave-one-out pass over a modest agent panel is already the
-useful 80% case. If you suspect joint effects, ablate the suspected pair
-manually by filtering `findings` yourself before calling `decide`.
-
-**`decide()` must be pure and deterministic.** `runAblation` calls `decide` once
-per finding removed, expecting each call to depend only on the findings it's
-given. If your decision logic calls an LLM internally, this tool does not apply
-to that call — it only makes sense as a probe over a deterministic aggregation
-step that runs *after* the LLM reasoning is done. This mirrors SentryMesh's own
-architecture: its supervisor's model call interprets *why* specialists conflict
-and produces a combined risk and confidence, but turning those numbers into an
-auto-decline/auto-approve/escalate action is `decide()`, a pure function with no
-model call in it — "an LLM is a good place for the interpretation and a bad place
-for a threshold that compliance will one day have to explain in writing," in
-SentryMesh's own words. `agent-ablation` ablates that downstream pure function,
-not the LLM call that fed it.
-
-## API reference
+---
+
+### 3. Async & Stochastic Decision Functions (LLM-as-a-Judge)
+
+When your decision step is an async LLM call with temperature, use `runAblationAsync` or `batchAblationAsync`. Set `samples: k` to take a majority-vote consensus across runs to eliminate sampling noise:
 
 ```typescript
-interface Finding {
-  agentId: string;
-  score: number;
-  confidence?: number;
-  metadata?: Record<string, unknown>;
+import { runAblationAsync } from "agent-ablation";
+
+async function llmSupervisorDecide(findings: Finding[]): Promise<string> {
+  const res = await callLlmJudge(findings);
+  return res.verdict;
 }
 
-type DecisionFn<TVerdict> = (findings: Finding[]) => TVerdict;
+const result = await runAblationAsync(findings, llmSupervisorDecide, {
+  samples: 5, // Runs 5 samples per ablation to filter out temperature noise
+});
+```
+
+---
 
-interface PerAgentAblation<TVerdict> {
-  removedAgentId: string;
-  verdictWithout: TVerdict;
-  changed: boolean;
-}
+### 4. Greedy Backward Elimination & Minimal Viable Panel
 
-interface AblationResult<TVerdict> {
-  baseline: TVerdict;
-  perAgent: PerAgentAblation<TVerdict>[];
-  loadBearingCount: number;
-  totalAgents: number;
-  loadBearingRatio: number;
-}
+If you have correlated or redundant agents (e.g., three critics looking at the same context), simple leave-one-out might mark all of them as not load-bearing because the others compensate. 
 
-function runAblation<TVerdict>(
-  findings: Finding[],
-  decide: DecisionFn<TVerdict>,
-  equals?: (a: TVerdict, b: TVerdict) => boolean
-): AblationResult<TVerdict>;
+`runBackwardElimination` iteratively eliminates agents one-by-one until removing any further agent flips the verdict, revealing the **minimal viable panel**:
 
-interface BatchAblationSummary {
-  cases: number;
-  averageLoadBearingRatio: number;
-  perAgentInfluence: Record<string, number>;
-}
+```typescript
+import { runBackwardElimination } from "agent-ablation";
+
+const result = runBackwardElimination(allSpecialists, decide);
 
-function batchAblation<TVerdict>(
-  cases: Finding[][],
-  decide: DecisionFn<TVerdict>,
-  equals?: (a: TVerdict, b: TVerdict) => boolean
-): { results: AblationResult<TVerdict>[]; summary: BatchAblationSummary };
+console.log(result.minimalAgentIds);   // ["critic_1", "security_auditor"]
+console.log(result.eliminatedAgentIds); // ["critic_2", "critic_3", "scout_noisy"]
+console.log(result.steps);             // Step-by-step elimination trace
 ```
 
-`equals` defaults to `===`. If `TVerdict` is an object (or anything else compared
-by reference rather than value), you must supply your own `equals` — otherwise
-every ablation will read as "changed" purely because two structurally identical
-verdict objects are never `===` to each other, regardless of whether the
-decision actually differed.
+For detecting 2nd-order joint dependencies, `runPairwiseAblation(findings, decide)` evaluates all pairs $(A, B)$ to catch cases where neither agent alone is load-bearing, but removing both together flips the outcome.
 
-## License
+---
+
+### 5. Ground-Truth & Net Accuracy Impact ("Protective vs. Harmful")
+
+Supply ground truth labels in `batchAblation` to measure whether an agent's load-bearing presence actually **improved** accuracy or **injected errors / hallucinations**:
+
+```typescript
+const { summary } = batchAblation(cases, decide, {
+  groundTruth: ["approve", "decline", "approve", "escalate"],
+});
+
+// Per-agent stats:
+// - Protective: removing the agent caused a correct verdict to become incorrect
+// - Harmful: removing the agent fixed an incorrect verdict
+console.log(summary.perAgentStats?.["hallucinating_agent"]?.role); // "Harmful"
+console.log(summary.perAgentStats?.["hallucinating_agent"]?.netAccuracyImpact); // -0.25
+```
+
+---
+
+## Framework Adapters
 
-MIT
+Zero-dependency adapters to map telemetry and messages from popular agent frameworks directly into `Finding[]`:
+
+### LangGraph
+```typescript
+import { fromLangGraphMessages } from "agent-ablation";
+
+const findings = fromLangGraphMessages(state.messages, {
+  scoreOf: (msg) => (msg.content as any).score,
+  confidenceOf: (msg) => (msg.content as any).confidence,
+});
+```
+
+### CrewAI
+```typescript
+import { fromCrewAITasks } from "agent-ablation";
+
+const findings = fromCrewAITasks(crewOutput.tasks_output, {
+  scoreOf: (task) => task.json_dict?.score ?? 0,
+});
+```
+
+### AutoGen
+```typescript
+import { fromAutoGenMessages } from "agent-ablation";
+
+const findings = fromAutoGenMessages(chatHistory, {
+  scoreOf: (msg) => (msg.content as any).riskScore,
+});
+```
+
+### Vercel AI SDK / Step Traces
+```typescript
+import { fromAISDKSteps } from "agent-ablation";
+
+const findings = fromAISDKSteps(steps, {
+  scoreOf: (step) => (step.result as any).score,
+});
+```
+
+### Generic Custom Records
+```typescript
+import { fromRecords } from "agent-ablation";
+
+const findings = fromRecords(customAuditRecords, {
+  agentId: (r) => r.specialistId,
+  scoreOf: (r) => r.riskScore,
+  confidenceOf: (r) => r.confidenceLevel,
+});
+```
 
 ---
 
-Ayush Verma — ayushv3533e@gmail.com
+## Worked Example: SentryMesh 33% Multi-Signal Finding
+
+[SentryMesh](https://github.com/AyushCipher/Sentry-Mesh) is a four-specialist multi-agent fraud investigation system. Its eval harness runs an ablation over its 23-case bank and reports: of 9 cases auto-resolved without human escalation, **only 3 survive removal of their single loudest specialist — 6 collapse to `escalate`.**
+
+`tests/ablation.test.ts` reproduces all 6 cases verbatim with `agent-ablation`.
+
+---
+
+## Architectural Note: Leaf Ablation vs. DAG Subgraph Replay
+
+* **Leaf Finding Ablation (This Package):** Best for **parallel / fan-out / fan-in** panels where specialists independently produce findings that feed a decision gate. Because findings are generated independently, dropping an item at `decide()` measures causal weight with **zero LLM re-invocation cost**.
+* **Sequential DAG Replay:** If your pipeline is sequential (Agent A feeds intermediate prompt context to Agent B), removing Agent A at the final decision gate misses that Agent B's output already reflects Agent A. Measuring sequential pipelines requires replaying downstream subgraphs or injecting mock messages into the trace.
+
+---
+
+## API Summary
+
+| Function | Description |
+| :--- | :--- |
+| `runAblation(findings, decide, equals?)` | Synchronous leave-one-out ablation for a single case. |
+| `runAblationAsync(findings, decide, options?)` | Async leave-one-out ablation with optional $K$-sampling / majority voting. |
+| `batchAblation(cases, decide, options?)` | Batch ablation with telemetry ROI and ground-truth metrics. |
+| `batchAblationAsync(cases, decide, options?)` | Async batch ablation. |
+| `runBackwardElimination(findings, decide, equals?)` | Greedy backward elimination to find the minimal agent panel. |
+| `runPairwiseAblation(findings, decide, equals?)` | Evaluates all 2-agent pairs to detect interaction/redundancy effects. |
+| `formatMarkdownReport(summary, options?)` | Formats summary into a GitHub/Dev.to markdown report with ROI tables. |
+| `formatAsciiTable(summary)` | Formats summary into a clean terminal ASCII table. |
+| `fromLangGraphMessages(...)` | Adapter for LangGraph message arrays. |
+| `fromCrewAITasks(...)` | Adapter for CrewAI task outputs. |
+| `fromAutoGenMessages(...)` | Adapter for AutoGen chat histories. |
+| `fromAISDKSteps(...)` | Adapter for Vercel AI SDK step traces. |
+| `fromRecords(...)` | Generic record mapper. |
+
+---
+
+## License
+
+MIT © Ayush Verma — ayushv3533e@gmail.com