agent-ablation 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ayush Verma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,183 @@
+# agent-ablation
+
+Leave-one-out ablation testing for multi-agent decision systems. You have a set of
+per-agent findings (scores, confidences, whatever your pipeline produces) and a
+function that turns those findings into a verdict. `agent-ablation` answers one
+question: **which of my agents' findings actually changed that verdict, and which
+were along for the ride?**
+
+It removes each finding one at a time, re-runs your decision function on what's
+left, and reports which removals flipped the outcome. Zero runtime dependencies,
+zero opinions about how your agents work — you supply the findings and the
+decision function, it does the leave-one-out loop and the bookkeeping.
+
+## Install
+
+```bash
+npm install agent-ablation
+```
+
+## Quick example
+
+```typescript
+import { runAblation, type Finding } from "agent-ablation";
+
+type Verdict = "approve" | "decline" | "escalate";
+
+function decide(findings: Finding[]): Verdict {
+  const risk = 1 - findings.reduce((p, f) => p * (1 - f.score / 100), 1);
+  if (risk >= 0.7) return "decline";
+  if (risk <= 0.3) return "approve";
+  return "escalate";
+}
+
+const findings: Finding[] = [
+  { agentId: "transaction_pattern", score: 25 },
+  { agentId: "identity_signal", score: 90 },
+  { agentId: "network_analysis", score: 20 },
+];
+
+const result = runAblation(findings, decide);
+
+console.log(result.baseline);        // "decline"
+console.log(result.loadBearingRatio); // fraction of agents whose removal changed the verdict
+for (const p of result.perAgent) {
+  console.log(p.removedAgentId, "->", p.verdictWithout, p.changed ? "(load-bearing)" : "");
+}
+```
+
+For a batch of cases, `batchAblation` runs the same ablation over each one and
+aggregates the results — including, per agent, the fraction of cases in which
+removing that agent changed the outcome:
+
+```typescript
+import { batchAblation } from "agent-ablation";
+
+const { results, summary } = batchAblation(allCases, decide);
+
+console.log(summary.averageLoadBearingRatio);
+console.log(summary.perAgentInfluence); // { transaction_pattern: 0.17, identity_signal: 0.83, ... }
+```
+
+## Worked example: reproducing SentryMesh's 33% multi-signal-share finding
+
+[SentryMesh](https://github.com/AyushCipher/Sentry-Mesh) is a four-specialist
+multi-agent fraud investigation system. Its own eval harness runs an ablation over
+its 23-case bank and reports the result plainly in its README: of 9 cases the
+system auto-resolved without escalating to a human, **only 3 survive removal of
+their single loudest specialist — 6 collapse to `escalate`.** SentryMesh calls
+this "the most important number in the report," because it means two-thirds of
+those auto-decisions rested on one specialist's finding, with the other three
+specialists' LLM calls spent for nothing.
+
+Those six cases, straight from SentryMesh's ablation table:
+
+| Case | Decision | Remove | Becomes |
+|---|---|---|---|
+| SM-001 | auto_decline | `identity_signal` (100) | escalate |
+| SM-002 | auto_decline | `identity_signal` (75) | escalate |
+| SM-005 | auto_decline | `identity_signal` (80) | escalate |
+| SM-006 | auto_decline | `identity_signal` (70) | escalate |
+| SM-012 | auto_decline | `network_analysis` (90) | escalate |
+| SM-016 | auto_approve | `transaction_pattern` (26) | escalate |
+
+`tests/ablation.test.ts` in this repo reproduces all six as a worked example: it
+builds each case's four-specialist `Finding[]` (with the named specialist's score
+matching SentryMesh's table exactly), runs it through a decision function modeled
+on SentryMesh's own description of its aggregation — findings combine via
+noisy-OR, gated by panel confidence — and asserts that `runAblation` correctly
+identifies the named specialist's removal as the one that flips each case to
+`escalate`. It's the credibility anchor for this package: if `agent-ablation`
+couldn't reproduce a real, previously-published ablation result on real case
+data, it wouldn't be trustworthy on your data either.
+
+## Limitations
+
+**This tool detects verdict *change*, not verdict quality.** A flipped verdict
+after removing an agent tells you that agent was load-bearing for that decision —
+it says nothing about whether the original verdict or the post-removal one was
+*correct*. Conversely, a low `loadBearingRatio` is not automatically a flaw:
+redundancy across agents can be exactly what you want (independent corroboration
+is the point of running more than one specialist), and this tool has no way to
+distinguish "healthy redundancy" from "wasted compute" for you.
+
+**Leave-one-out misses agents that only matter in pairs.** This is a real gap,
+not a hedge. If removing agent A alone doesn't flip the verdict, and removing
+agent B alone doesn't either, but removing *both together* would, leave-one-out
+ablation will report both as not load-bearing. Catching that requires ablating
+combinations, which this package deliberately does not do — the combinatorics
+blow up fast, and a leave-one-out pass over a modest agent panel is already the
+useful 80% case. If you suspect joint effects, ablate the suspected pair
+manually by filtering `findings` yourself before calling `decide`.
+
+**`decide()` must be pure and deterministic.** `runAblation` calls `decide` once
+per finding removed, expecting each call to depend only on the findings it's
+given. If your decision logic calls an LLM internally, this tool does not apply
+to that call — it only makes sense as a probe over a deterministic aggregation
+step that runs *after* the LLM reasoning is done. This mirrors SentryMesh's own
+architecture: its supervisor's model call interprets *why* specialists conflict
+and produces a combined risk and confidence, but turning those numbers into an
+auto-decline/auto-approve/escalate action is `decide()`, a pure function with no
+model call in it — "an LLM is a good place for the interpretation and a bad place
+for a threshold that compliance will one day have to explain in writing," in
+SentryMesh's own words. `agent-ablation` ablates that downstream pure function,
+not the LLM call that fed it.
+
+## API reference
+
+```typescript
+interface Finding {
+  agentId: string;
+  score: number;
+  confidence?: number;
+  metadata?: Record<string, unknown>;
+}
+
+type DecisionFn<TVerdict> = (findings: Finding[]) => TVerdict;
+
+interface PerAgentAblation<TVerdict> {
+  removedAgentId: string;
+  verdictWithout: TVerdict;
+  changed: boolean;
+}
+
+interface AblationResult<TVerdict> {
+  baseline: TVerdict;
+  perAgent: PerAgentAblation<TVerdict>[];
+  loadBearingCount: number;
+  totalAgents: number;
+  loadBearingRatio: number;
+}
+
+function runAblation<TVerdict>(
+  findings: Finding[],
+  decide: DecisionFn<TVerdict>,
+  equals?: (a: TVerdict, b: TVerdict) => boolean
+): AblationResult<TVerdict>;
+
+interface BatchAblationSummary {
+  cases: number;
+  averageLoadBearingRatio: number;
+  perAgentInfluence: Record<string, number>;
+}
+
+function batchAblation<TVerdict>(
+  cases: Finding[][],
+  decide: DecisionFn<TVerdict>,
+  equals?: (a: TVerdict, b: TVerdict) => boolean
+): { results: AblationResult<TVerdict>[]; summary: BatchAblationSummary };
+```
+
+`equals` defaults to `===`. If `TVerdict` is an object (or anything else compared
+by reference rather than value), you must supply your own `equals` — otherwise
+every ablation will read as "changed" purely because two structurally identical
+verdict objects are never `===` to each other, regardless of whether the
+decision actually differed.
+
+## License
+
+MIT
+
+---
+
+Ayush Verma — ayushv3533e@gmail.com

package/dist/index.cjs

@@ -0,0 +1,81 @@
+"use strict";
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+
+// src/index.ts
+var index_exports = {};
+__export(index_exports, {
+  batchAblation: () => batchAblation,
+  runAblation: () => runAblation
+});
+module.exports = __toCommonJS(index_exports);
+function defaultEquals(a, b) {
+  return a === b;
+}
+function runAblation(findings, decide, equals = defaultEquals) {
+  const baseline = decide(findings.slice());
+  const perAgent = findings.map((finding, index) => {
+    const without = findings.slice(0, index).concat(findings.slice(index + 1));
+    const verdictWithout = decide(without);
+    return {
+      removedAgentId: finding.agentId,
+      verdictWithout,
+      changed: !equals(baseline, verdictWithout)
+    };
+  });
+  const totalAgents = findings.length;
+  const loadBearingCount = perAgent.filter((p) => p.changed).length;
+  const loadBearingRatio = totalAgents === 0 ? 0 : loadBearingCount / totalAgents;
+  return {
+    baseline,
+    perAgent,
+    loadBearingCount,
+    totalAgents,
+    loadBearingRatio
+  };
+}
+function batchAblation(cases, decide, equals = defaultEquals) {
+  const results = cases.map((findings) => runAblation(findings, decide, equals));
+  const appearances = /* @__PURE__ */ new Map();
+  const changedCounts = /* @__PURE__ */ new Map();
+  for (const result of results) {
+    for (const perAgent of result.perAgent) {
+      const id = perAgent.removedAgentId;
+      appearances.set(id, (appearances.get(id) ?? 0) + 1);
+      if (perAgent.changed) {
+        changedCounts.set(id, (changedCounts.get(id) ?? 0) + 1);
+      }
+    }
+  }
+  const perAgentInfluence = {};
+  for (const [id, count] of appearances) {
+    perAgentInfluence[id] = (changedCounts.get(id) ?? 0) / count;
+  }
+  const averageLoadBearingRatio = results.length === 0 ? 0 : results.reduce((sum, r) => sum + r.loadBearingRatio, 0) / results.length;
+  const summary = {
+    cases: cases.length,
+    averageLoadBearingRatio,
+    perAgentInfluence
+  };
+  return { results, summary };
+}
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  batchAblation,
+  runAblation
+});

package/dist/index.d.cts

@@ -0,0 +1,47 @@
+interface Finding {
+    agentId: string;
+    score: number;
+    confidence?: number;
+    metadata?: Record<string, unknown>;
+}
+type DecisionFn<TVerdict> = (findings: Finding[]) => TVerdict;
+interface PerAgentAblation<TVerdict> {
+    removedAgentId: string;
+    verdictWithout: TVerdict;
+    changed: boolean;
+}
+interface AblationResult<TVerdict> {
+    baseline: TVerdict;
+    perAgent: PerAgentAblation<TVerdict>[];
+    loadBearingCount: number;
+    totalAgents: number;
+    loadBearingRatio: number;
+}
+interface BatchAblationSummary {
+    cases: number;
+    averageLoadBearingRatio: number;
+    perAgentInfluence: Record<string, number>;
+}
+/**
+ * Runs a leave-one-out ablation over `findings`: computes the baseline verdict,
+ * then re-runs `decide` once per finding with that finding removed, comparing each
+ * result back to the baseline via `equals`.
+ *
+ * `equals` defaults to `===`. If TVerdict is an object or otherwise compared by
+ * reference (not a primitive like a string or number), you MUST supply your own
+ * `equals` — otherwise every ablation will spuriously read as "changed" because two
+ * structurally identical objects are never `===` to one another, even when nothing
+ * about the decision actually differed.
+ */
+declare function runAblation<TVerdict>(findings: Finding[], decide: DecisionFn<TVerdict>, equals?: (a: TVerdict, b: TVerdict) => boolean): AblationResult<TVerdict>;
+/**
+ * Runs `runAblation` over a batch of independent cases and aggregates the results:
+ * the mean load-bearing ratio across cases, and, per agent ID, the fraction of the
+ * cases containing that agent in which removing it flipped the verdict.
+ */
+declare function batchAblation<TVerdict>(cases: Finding[][], decide: DecisionFn<TVerdict>, equals?: (a: TVerdict, b: TVerdict) => boolean): {
+    results: AblationResult<TVerdict>[];
+    summary: BatchAblationSummary;
+};
+
+export { type AblationResult, type BatchAblationSummary, type DecisionFn, type Finding, type PerAgentAblation, batchAblation, runAblation };

package/dist/index.d.ts

@@ -0,0 +1,47 @@
+interface Finding {
+    agentId: string;
+    score: number;
+    confidence?: number;
+    metadata?: Record<string, unknown>;
+}
+type DecisionFn<TVerdict> = (findings: Finding[]) => TVerdict;
+interface PerAgentAblation<TVerdict> {
+    removedAgentId: string;
+    verdictWithout: TVerdict;
+    changed: boolean;
+}
+interface AblationResult<TVerdict> {
+    baseline: TVerdict;
+    perAgent: PerAgentAblation<TVerdict>[];
+    loadBearingCount: number;
+    totalAgents: number;
+    loadBearingRatio: number;
+}
+interface BatchAblationSummary {
+    cases: number;
+    averageLoadBearingRatio: number;
+    perAgentInfluence: Record<string, number>;
+}
+/**
+ * Runs a leave-one-out ablation over `findings`: computes the baseline verdict,
+ * then re-runs `decide` once per finding with that finding removed, comparing each
+ * result back to the baseline via `equals`.
+ *
+ * `equals` defaults to `===`. If TVerdict is an object or otherwise compared by
+ * reference (not a primitive like a string or number), you MUST supply your own
+ * `equals` — otherwise every ablation will spuriously read as "changed" because two
+ * structurally identical objects are never `===` to one another, even when nothing
+ * about the decision actually differed.
+ */
+declare function runAblation<TVerdict>(findings: Finding[], decide: DecisionFn<TVerdict>, equals?: (a: TVerdict, b: TVerdict) => boolean): AblationResult<TVerdict>;
+/**
+ * Runs `runAblation` over a batch of independent cases and aggregates the results:
+ * the mean load-bearing ratio across cases, and, per agent ID, the fraction of the
+ * cases containing that agent in which removing it flipped the verdict.
+ */
+declare function batchAblation<TVerdict>(cases: Finding[][], decide: DecisionFn<TVerdict>, equals?: (a: TVerdict, b: TVerdict) => boolean): {
+    results: AblationResult<TVerdict>[];
+    summary: BatchAblationSummary;
+};
+
+export { type AblationResult, type BatchAblationSummary, type DecisionFn, type Finding, type PerAgentAblation, batchAblation, runAblation };

package/dist/index.js

@@ -0,0 +1,55 @@
+// src/index.ts
+function defaultEquals(a, b) {
+  return a === b;
+}
+function runAblation(findings, decide, equals = defaultEquals) {
+  const baseline = decide(findings.slice());
+  const perAgent = findings.map((finding, index) => {
+    const without = findings.slice(0, index).concat(findings.slice(index + 1));
+    const verdictWithout = decide(without);
+    return {
+      removedAgentId: finding.agentId,
+      verdictWithout,
+      changed: !equals(baseline, verdictWithout)
+    };
+  });
+  const totalAgents = findings.length;
+  const loadBearingCount = perAgent.filter((p) => p.changed).length;
+  const loadBearingRatio = totalAgents === 0 ? 0 : loadBearingCount / totalAgents;
+  return {
+    baseline,
+    perAgent,
+    loadBearingCount,
+    totalAgents,
+    loadBearingRatio
+  };
+}
+function batchAblation(cases, decide, equals = defaultEquals) {
+  const results = cases.map((findings) => runAblation(findings, decide, equals));
+  const appearances = /* @__PURE__ */ new Map();
+  const changedCounts = /* @__PURE__ */ new Map();
+  for (const result of results) {
+    for (const perAgent of result.perAgent) {
+      const id = perAgent.removedAgentId;
+      appearances.set(id, (appearances.get(id) ?? 0) + 1);
+      if (perAgent.changed) {
+        changedCounts.set(id, (changedCounts.get(id) ?? 0) + 1);
+      }
+    }
+  }
+  const perAgentInfluence = {};
+  for (const [id, count] of appearances) {
+    perAgentInfluence[id] = (changedCounts.get(id) ?? 0) / count;
+  }
+  const averageLoadBearingRatio = results.length === 0 ? 0 : results.reduce((sum, r) => sum + r.loadBearingRatio, 0) / results.length;
+  const summary = {
+    cases: cases.length,
+    averageLoadBearingRatio,
+    perAgentInfluence
+  };
+  return { results, summary };
+}
+export {
+  batchAblation,
+  runAblation
+};

package/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "agent-ablation",
+  "version": "0.1.0",
+  "description": "Leave-one-out ablation testing for multi-agent decision systems — find out which agents' findings actually change the outcome.",
+  "type": "module",
+  "main": "./dist/index.cjs",
+  "module": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "require": "./dist/index.cjs"
+    }
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "build": "tsup src/index.ts --format esm,cjs --dts --clean",
+    "test": "vitest run",
+    "typecheck": "tsc --noEmit"
+  },
+  "keywords": [
+    "multi-agent",
+    "ai-agents",
+    "ablation",
+    "explainability",
+    "llm-agents",
+    "agent-orchestration",
+    "interpretability"
+  ],
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/AyushCipher/agent-ablation.git"
+  },
+  "author": "Ayush Verma <ayushv3533e@gmail.com>",
+  "devDependencies": {
+    "@types/node": "^26.2.0",
+    "tsup": "^8.5.1",
+    "typescript": "^5.7.3",
+    "vitest": "^4.1.10"
+  }
+}