agent-alibi 0.1.0__py3-none-any.whl

agent_alibi-0.1.0.dist-info/METADATA

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: agent-alibi
+Version: 0.1.0
+Summary: We check whether your agent has an alibi for what it did.
+License-Expression: MIT
+Requires-Dist: langchain-typesafe>=0.0.1a2
+Requires-Dist: langsmith>=0.12.6
+Requires-Dist: mcp>=2.2,<3
+Requires-Dist: numpy>=2.5.3
+Requires-Dist: openai>=3.16.0
+Requires-Dist: pyarrow>=25.0.1
+Requires-Dist: pydantic>=2
+Requires-Dist: python-dotenv>=1.2.3
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/ahmedezz26/alibi
+Description-Content-Type: text/markdown
+
+# Alibi
+
+[![CI](https://github.com/ahmedezz26/alibi/actions/workflows/ci.yml/badge.svg)](https://github.com/ahmedezz26/alibi/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](pyproject.toml)
+
+**We check whether your agent has an alibi for what it did.**
+
+Alibi finds where a long AI-agent run went wrong. It borrows the playbook of automotive
+driver-assistance systems: treat the run as a time series, filter it forward, detect the
+fault, then smooth backward to the moment it began. The only sensor is Jev, TypeSafe AI's
+System One model, which answers with calibrated probabilities instead of text.
+
+It is built for long traces, where reading everything at once breaks down: on real coding
+failures of 59K to 120K tokens, a single whole-trace read found the root-cause step in 0%
+of cases; Alibi found it in 16% and 7.8%, and points to the right neighbourhood (within
+3 steps) in 21% and 17%.
+
+## How it works
+
+```mermaid
+flowchart LR
+  A[Trace] --> B[10K-token chapters]
+  B --> C[Forward filter:<br/>Jev rates each chapter]
+  C -->|memory card| C
+  C --> D[CUSUM alarm<br/>on health]
+  D --> E[Look back from the alarm:<br/>chapter + per-step evidence]
+  E --> F[Top 3 steps to read,<br/>earliest strong suspect first]
+```
+
+| Stage | What happens | Driver-assistance analogue |
+|---|---|---|
+| Chapters | 10K-token windows with overlap, read one at a time | Measurement frames |
+| Forward filter | Jev rates health and 4 warning signs per chapter; a memory card of numbers carries state forward | Recursive filter (predict + update) |
+| CUSUM | Accumulates health drift; the first alarm marks the failure chapter | Fault detection |
+| Look-back | Re-reads the alarm chapter and every earlier one in parallel, with hindsight | Fixed-interval (RTS-style) smoothing |
+| Pick | P(chapter) x evidence per step; the earliest step within 80% of the top score | Fault-onset estimation |
+
+## What you get
+
+A short trace is refused, without spending a call:
+
+    $ alibi diagnose examples/sample_trace.json
+    Trace is 219 tokens, under the 50,000-token threshold: a single direct read is
+    enough; Alibi adds value on long traces.
+
+A long one comes back as three steps to read, in order. This is a real run from the
+locked TrajErrBench set (a Claude Opus coding agent failing on a qutebrowser issue),
+replayed from its recorded result:
+
+    $ alibi diagnose trace.json
+    Read these 3 steps first, in order.
+    80,778 tokens, 10 chapters, alarm at chapter 6; 17 Jev calls, 35 s, $0.0096
+    1. step 74 (assistant), chapter 6, score 0.277
+       'Now I see the full picture. The test on line 458 expects
+        `str(proc.outcome) == 'Testprocess crashed.'` for SIGSEGV...'
+    2. step 76 (assistant), chapter 6, score 0.202
+       '## Phase 5: FIX ANALYSIS\n\nNow I have a clear understanding. Let me
+        implement the changes to `guiprocess.py`...'
+    3. step 78 (assistant), chapter 6, score 0.178
+       'Now let me implement all the changes:\nTool calls:\nstr_replace_editor(...'
+
+Step 74 is the labelled root cause: the agent reads the test wrong and every later
+edit builds on that reading. Being right at rank 1 happens on 16% of these traces;
+the honest claim is that three steps out of 118 is a much smaller haystack.
+
+## Results
+
+Locked test sets, each run once against a pass bar written down beforehand:
+
+| Test set | Traces | Median length | Alibi exact step | Alibi within 3 steps | Whole-trace read, exact |
+|---|---|---|---|---|---|
+| TrajErrBench SWE-Bench Pro (real coding failures) | 56 | 59K tokens | 16% | 21% | 0% (p = 0.004) |
+| LongRCA SWE-bench Pro (real coding failures) | 90 | 120K tokens | 7.8% | 16.7% | 0% (p = 0.016) |
+| LongRCA WebArena (real web-task failures) | 48 | 38K tokens | 12.5% | 20.8% | 16.9% published (no clear difference) |
+
+Against published methods on the LongRCA leaderboard (all on DeepSeek-V4-Flash; exact
+root step, same 128 SWE-bench Pro failures):
+
+| Method | Exact root step |
+|---|---|
+| RCTA | 38.3% |
+| **Alibi (Jev)** | **10.2%** |
+| ECHO | 7.8% |
+| FALAT | 2.3% |
+| All-at-once (whole trace) | 1.6% |
+| Step-by-step | 0.8% |
+| Binary search | 0.8% |
+
+Alibi ties ECHO (Fisher p = 0.66) and trails RCTA (p < 0.000001), which traces each
+suspect back to the handoff instruction between agents.
+
+## When to use it, and when not to
+
+- Use it for traces of tens of thousands of tokens or more (default gate: 50K tokens).
+- Do not use it for short traces: a single direct read by any capable model is enough,
+  and Alibi says so without spending a call.
+- Treat the output as "read these 3 steps first", not as a verdict.
+
+## Speed and cost
+
+- Each chapter is one Jev call, and chapters are judged in parallel on the way back. A
+  38K-token trace is 4.6 chapters and 15 seconds of judge time; a 120K-token trace is
+  17 chapters and about 48 seconds. No reasoning tokens are generated.
+- Cost scales with trace length: about $0.10 per million trace tokens at Jev's price of
+  $0.042 per million input tokens (roughly $0.01 for a 100K-token trace).
+
+## Install
+
+Claude Code:
+
+    /plugin marketplace add ahmedezz26/alibi
+    /plugin install alibi@alibi
+
+Set `TYPESAFE_API_KEY` in your environment. Then ask Claude Code: "Why did my last
+session go wrong?"
+
+Command line:
+
+    uv tool install git+https://github.com/ahmedezz26/alibi
+    ALIBI_JUDGE_BACKEND=typesafe ALIBI_ALLOW_PAID_MODELS=1 TYPESAFE_API_KEY=... \
+      alibi diagnose path/to/trace.json
+
+## Privacy
+
+Traces above the length gate are sent to TypeSafe's API; below it, nothing leaves your
+machine. There is no telemetry. Claude Code session parsing is best effort: the transcript
+format is internal to Claude Code and may change. Session transcripts often contain source
+code and secrets, so read [SECURITY.md](SECURITY.md) before diagnosing one.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). Plumbing, adapters, bug fixes and docs are welcome as
+ordinary pull requests; changes to the estimation method need a pre-registered measurement,
+for the reason the research log makes obvious.
+
+## Background
+
+Alibi started as an experiment by an ADAS engineer: can the tracking filters used in cars
+work on AI-agent traces? The research log with every pre-registered test, including the
+ones that failed, is in [docs/research-log.md](docs/research-log.md).
+
+## License
+
+MIT

agent_alibi-0.1.0.dist-info/RECORD

@@ -0,0 +1,30 @@
+alibi/__init__.py,sha256=106VQ9QhFYONJcypz1Q-u9gGGsQL4FP06mi0_1E4K2w,80
+alibi/backward.py,sha256=-9Zxav59z5WkS5D__qLnDmhkj7QDKKkv3TLgP0tFSLY,20217
+alibi/chunking.py,sha256=TZOtcmST3MbA-YDHFo_x82pUJ9MF_-GjMa_dNbpiVHQ,4426
+alibi/cli.py,sha256=TPxw44R69XUkQaVUD4GCtJ2mQyqLMqXeMuQ-1q0OOOY,15828
+alibi/config.py,sha256=unF7Q6zqZo6AP-hEJE8HPZH6X59wQYHVIKeGYdECNro,3500
+alibi/drift.py,sha256=9VKW_tAfDUN8Z_44Rc8zrHxoJlczHrorgOcCmxrBz0M,1940
+alibi/evaluate.py,sha256=ZCsbH1wuszSiaD3JXjTTN0J-lJHj21lGW5ExT-sQCaA,12893
+alibi/forward.py,sha256=MSeB44M0hvN6Hn5rQGsLK3V_re2JISXyltnrmSLkFc0,6654
+alibi/judges/__init__.py,sha256=esY3Zy9UKV4kOxvwsslPR96v52gSyh1PlXjtBZ2-TRc,2365
+alibi/judges/base.py,sha256=4OOcvFBHsWYyWe-aO0k8xYP6a9FBwxUFTrewh6mCirc,588
+alibi/judges/openrouter.py,sha256=mM0Gro7ZCM5m0wxnw5Xxa2iT0ym3WDjpnNp9-q34KdA,2916
+alibi/judges/typesafe.py,sha256=fLleDwjj4vMfRSQeUcBPNzshE3tsFFqIPqzTIWSrKkE,3510
+alibi/localize.py,sha256=fQ5PkALVGhTL0OTWDAe1xtIBaq-lbWzC8_yKdoscfdY,4851
+alibi/mcp_server.py,sha256=nJGbiTUZeRDmAPF_VLuC9nnb9dOjqBxs49UQpcLc6gE,1563
+alibi/routing.py,sha256=UgevgRxCT6ASnWIlWOkETpN47rPb8bpK9oPsuJAjscM,1963
+alibi/sources/__init__.py,sha256=7r-4QCNMrd-1Cd-rcOimGybjZ7iTa2L_LXzLFnZTNDA,70
+alibi/sources/agentracer.py,sha256=PR8Xl1IymsO8ToowFhBMwcpRnQB_tq1Lvw1kq3UWiB8,2404
+alibi/sources/agentrx.py,sha256=dp5BmeFG74uGMQ0f8XEwvEv6P5Jmcng0tzM2htMisFQ,2716
+alibi/sources/auto.py,sha256=4wuVmyGkaNurxEKa2WNPfERh4x_GkezgwkGy5TFfdMQ,1080
+alibi/sources/base.py,sha256=qOkCtOOauIMR7rR0VyLZ295S04ToQYgigedrWd8WU1Q,265
+alibi/sources/claudecode.py,sha256=ch_pRwpDI9W19TetJUrn96iB2ok90wdlprXzWO57OQ4,1817
+alibi/sources/jsonfile.py,sha256=yxEz1-d3ohElppOEfh3Q2SnyMi_se0I39n5tr-Bt088,1131
+alibi/sources/langsmith.py,sha256=bZyKrN5a7qRjwXbuBp8zG_tmJF2p-3en2f9NeQUbu6w,1333
+alibi/sources/longrca.py,sha256=g1kFv-OlJYgUYN7BjU4SNCGhrcuA2B6Nmmu5op3tbkk,2635
+alibi/sources/trajerrbench.py,sha256=RTHRsM86kU0pjrKikU1qNzG_A0_kzivnDF0pVRqtqps,2358
+alibi/types.py,sha256=vhSt0GEeAmGjXmAxs1h7J3qMOhH2KmD4_noNfHbBCcw,1599
+agent_alibi-0.1.0.dist-info/WHEEL,sha256=R1d3uUTbmXM1FHXH_itQashbrqrOSVj-hvBCpmkIIGE,81
+agent_alibi-0.1.0.dist-info/entry_points.txt,sha256=YKWFOthViAh9z8XRpHrDbfAkJLVkagDWAs3iHQdqD0o,76
+agent_alibi-0.1.0.dist-info/METADATA,sha256=zEHVBTRGBytqbuRVw7j4YH80Rb6Cwk8w675f8vD62Ls,6952
+agent_alibi-0.1.0.dist-info/RECORD,,

agent_alibi-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: uv 0.12.17
+Root-Is-Purelib: true
+Tag: py3-none-any

agent_alibi-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,4 @@
+[console_scripts]
+alibi = alibi.cli:main
+alibi-mcp = alibi.mcp_server:main
+

alibi/__init__.py

@@ -0,0 +1 @@
+"""Alibi: agent-trace triage via state estimation over windowed judge calls."""