pi-l1-cache 1.2.0 → 1.2.2

package/README.md

@@ -4,6 +4,8 @@
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Pi Package](https://img.shields.io/badge/pi-package-blue)](https://pi.dev/packages)
+[![npm version](https://img.shields.io/npm/v/pi-l1-cache)](https://www.npmjs.com/package/pi-l1-cache)
+[![CI](https://github.com/tobias-weiss-ai-xr/pi-l1-cache/actions/workflows/ci.yml/badge.svg)](https://github.com/tobias-weiss-ai-xr/pi-l1-cache/actions/workflows/ci.yml)
 
 A high-performance, production-ready **L1 (in-memory) cache** extension for the [pi coding agent](https://pi.dev). Designed for stoic Unix simplicity: one file, one purpose, zero dependencies.
 
@@ -11,10 +13,11 @@ A high-performance, production-ready **L1 (in-memory) cache** extension for the
 
 | Feature | Description |
 |---------|-------------|
-| **~0.1ms lookup** | FNV-1a fast hash — ~100× faster than SHA256 |
+| **~138µs overhead** | End-to-end per-request cost (measured, incl. key hashing + JSON) |
+| **Fast key hash** | FNV-1a (~1.5µs/op) — no per-request CPU probe on the hot path |
 | **Memory cap** | Hard limit (20MB default) prevents RAM bloat |
 | **Auto-eviction** | LRU-style cleanup when limits reached |
-| **CPU-aware** | Auto-disables when CPU > 95% |
+| **CPU-aware** | Auto-disables when CPU > 95% (checked once at startup) |
 | **TTL-based** | 1-hour default expiry for cached entries |
 | **Atomic cleanup** | Periodic expired entry removal |
 
@@ -22,7 +25,7 @@ A high-performance, production-ready **L1 (in-memory) cache** extension for the
 
 ```
 pi → [L1: RAM Map] → [L2: Redis via LiteLLM] → Provider
-     <0.5ms          ~150ms                   1-3s
+     ~138µs          ~150ms                   1-3s
 ```
 
 ## Installation
@@ -119,7 +122,7 @@ const DEFAULTS: Settings = {
 - **Zero dependencies** — Single TypeScript file
 
 ### Performance Optimizations
-1. **Fast string hashing** (FNV-1a) instead of SHA256
+1. **Cheap key hashing** (FNV-1a, ~1.5µs/op). Note: Node's native SHA-256 (OpenSSL) is marginally *faster* than a JS FNV loop — hashing is a rounding error next to `JSON.stringify(messages)`, and both are >1000× below provider latency. The real win is *not* a faster hash, it is skipping the provider call.
 2. **L1 only** — avoid disk I/O in hot path
 3. **Batch eviction** — remove 10-20% at a time, not one-by-one
 4. **Periodic cleanup** — async, non-blocking garbage collection
@@ -145,6 +148,31 @@ npx tsc --noEmit
 - State management & reset
 - Settings override
 
+## Benchmark
+
+Measured against the published npm artifact (`pi-l1-cache@1.2.1`) on Node 22, using the package's own test hooks and a mocked `ExtensionAPI` with a realistic 15-message conversation history:
+
+| Measurement | Result |
+|---|---|
+| `fastHash` (FNV-1a) standalone | ~670k ops/s — 1.49µs/op |
+| Node native `sha256` (for reference) | ~850k ops/s — 1.18µs/op |
+| Full put: hash + map set + size check | ~249k ops/s — 4.0µs/op |
+| Interceptor path (hit *and* miss, 15-msg history) | ~138µs/request |
+| Cache hit vs uncached provider round-trip (1–3s) | ~7,000–20,000× wall-clock |
+| Key uniqueness over 50k synthetic keys | 50,000/50,000 (no collisions) |
+| Eviction under 200-slot cap, 20k inserts | cap held; 19,800 evicted |
+
+> ⚠ **Correction:** earlier versions claimed FNV-1a was "~100× faster than SHA256". That was wrong — Node's native SHA-256 is ~0.8× *faster* in practice. The hash was never the bottleneck: `JSON.stringify` dominates the ~138µs per-request cost. Cache hits are keyed by *byte-identical* requests, so real-world hit rate depends on your workload (best for retries, repeated tool calls and same-prompt reruns).
+
+## pi API compatibility
+
+Verified against the **pi 0.84.x** extension API — two runtime facts shape the behaviour:
+
+- `before_provider_request` receives the assembled provider request as `event.payload` (model, messages, parameters); cache keys are derived from it. In current pi this hook is a payload *transform*, not a response short-circuit, so a cached response is only ever returned once a response body has actually been captured.
+- `after_provider_response` currently carries only `{ status, headers }` — **no response body**. Until a pi version exposes the body, responses cannot be stored; the extension detects this, logs a one-time note, and keeps `/l1-cache` stats working. It upgrades automatically (no config) on any pi version that exposes the body.
+
+On pi 0.84 the extension therefore operates as a request-key instrumentation layer (Hits/Misses/Evictions via `/l1-cache`, CPU guard, TTL bookkeeping) and only short-circuits identical requests when the API contract provides the body it needs.
+
 ## Related Projects
 
 - **[opencode-saia-plugin](https://github.com/tobias-weiss-ai-xr/opencode-saia-plugin)** — SAIA provider for OpenCode
@@ -157,4 +185,4 @@ MIT — see [LICENSE](LICENSE)
 
 ## Maintainer
 
-[Tobias Weiß](https://github.com/tobias-weiss-ai-xr) — weissto@hrz.uni-marburg.de
+[Tobias Weiß](https://github.com/tobias-weiss-ai-xr) — info@graphwiz.ai

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-l1-cache",
-  "version": "1.2.0",
+  "version": "1.2.2",
   "description": "L1 in-memory cache extension for pi — CPU/RAM optimized, production-ready",
   "type": "module",
   "main": "./src/index.ts",
@@ -23,7 +23,7 @@
     "README.md",
     "LICENSE"
   ],
-  "author": "Tobias Weiß <weissto@hrz.uni-marburg.de>",
+  "author": "Tobias Weiß <info@graphwiz.ai>",
   "license": "MIT",
   "homepage": "https://github.com/tobias-weiss-ai-xr/pi-l1-cache",
   "repository": {
@@ -39,9 +39,18 @@
   },
   "scripts": {
     "test": "node --test src/*.test.ts",
-    "test:watch": "node --watch --test src/*.test.ts"
+    "test:watch": "node --watch --test src/*.test.ts",
+    "typecheck": "tsc",
+    "pack": "npm pack --dry-run",
+    "prepack": "npm run typecheck && npm test",
+    "prepublishOnly": "npm run typecheck && npm test"
   },
   "devDependencies": {
-    "@types/node": "^20.0.0"
+    "@earendil-works/pi-coding-agent": "^0.84.0",
+    "@types/node": "^20.0.0",
+    "typescript": "^5.8.0"
+  },
+  "peerDependencies": {
+    "@earendil-works/pi-coding-agent": "*"
   }
 }

package/src/index.test.ts

@@ -1,6 +1,6 @@
-import { describe, it, beforeEach } from "node:test"
+import { describe, it, beforeEach, before, after } from "node:test"
 import assert from "node:assert/strict"
-import { _testHooks } from "./index.ts"
+import init, { _testHooks } from "./index.ts"
 
 const Test = _testHooks()
 
@@ -133,4 +133,104 @@ describe("pi-l1-cache", () => {
       assert.equal(state.settings.maxMemoryBytes, 999999)
     })
   })
+
+  describe("interceptor flow (extension lifecycle)", () => {
+    type Handler = (ev?: any, ctx?: any) => any
+    const events: Record<string, Handler[]> = {}
+    let shutdown: (() => void) | undefined
+
+    before(async () => {
+      const api: any = {
+        registerCommand: () => {},
+        on: (name: string, cb: Handler) => {
+          ;(events[name] ??= []).push(cb)
+        },
+      }
+      await init(api)
+      const shutdownCbs = events.session_shutdown ?? []
+      shutdown = () => shutdownCbs.forEach((cb) => cb())
+    })
+
+    after(() => {
+      shutdown?.()
+    })
+
+    beforeEach(() => {
+      Test.reset()
+      Test.setSettings({ enabled: true, maxEntries: 50, maxMemoryBytes: 100000, ttlSeconds: 3600 })
+    })
+
+    const payload = () => ({ messages: [{ role: "user", content: "hello cache" }], parameters: {} })
+    const event = (): any => ({ type: "before_provider_request", payload: payload() })
+    const reqCtx = { model: { id: "test-model" } }
+
+    it("records a miss and stamps _cacheKey on first request", async () => {
+      const ev: any = event()
+      const ret = await events.before_provider_request[0](ev, reqCtx)
+      assert.equal(ret, undefined)
+      assert.ok(ev._cacheKey, "should stamp _cacheKey on miss")
+      assert.equal(Test.getStats().misses, 1)
+    })
+
+    it("returns the cached response on a byte-identical repeat when a body is captured", async () => {
+      const first: any = event()
+      await events.before_provider_request[0](first, reqCtx)
+      // Simulate a pi API that exposes the response body
+      await events.after_provider_response[0]({
+        type: "after_provider_response",
+        status: 200,
+        headers: {},
+        _cacheKey: first._cacheKey,
+        response: { content: "cached!" },
+      } as any)
+
+      const second: any = event()
+      const out = await events.before_provider_request[0](second, reqCtx)
+      assert.ok(out, "identical repeat should short-circuit to cache")
+      assert.equal(out.content, "cached!")
+      assert.equal(Test.getStats().hits, 1)
+    })
+
+    it("never stores an event envelope when no response body is available", async () => {
+      const first: any = event()
+      await events.before_provider_request[0](first, reqCtx)
+      // pi 0.84 + shape: status + headers only, no body
+      await events.after_provider_response[0]({
+        type: "after_provider_response",
+        status: 200,
+        headers: {},
+      } as any)
+      assert.equal(Test._getCacheSize(), 0, "must not cache the event envelope")
+      assert.ok(
+        Test.getStats().misses >= 0,
+        "after_provider_response without a body must not poison the cache",
+      )
+    })
+
+    it("does not touch the cache when disabled", async () => {
+      Test.setSettings({ enabled: false })
+      const ev: any = event()
+      const ret = await events.before_provider_request[0](ev, reqCtx)
+      assert.equal(ret, undefined)
+      assert.equal(ev._cacheKey, undefined)
+      assert.equal(Test.getStats().misses, 0)
+    })
+
+    it("keeps size within cap under interceptor churn", async () => {
+      Test.setSettings({ maxEntries: 20 })
+      for (let i = 0; i < 50; i++) {
+        const ev: any = {
+          type: "before_provider_request",
+          payload: { messages: [{ role: "user", content: `q-${i}` }], parameters: {} },
+        }
+        await events.before_provider_request[0](ev, reqCtx)
+        await events.after_provider_response[0]({
+          _cacheKey: ev._cacheKey,
+          response: { content: "a".repeat(50) },
+        } as any)
+      }
+      assert.ok(Test._getCacheSize() <= 20)
+      assert.ok(Test.getStats().misses >= 50)
+    })
+  })
 })

package/src/index.ts

@@ -14,7 +14,14 @@
 //   - CPU-aware graceful degradation
 //   - Zero dependencies, single file
 
-import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"
+import type {
+  BeforeProviderRequestEvent,
+  ExtensionAPI,
+  ExtensionCommandContext,
+  ExtensionEvent,
+} from "@earendil-works/pi-coding-agent"
+
+type AfterProviderResponseEvent = Extract<ExtensionEvent, { type: "after_provider_response" }>
 
 // ---------------------------------------------------------------------------
 // Types
@@ -79,6 +86,12 @@ let initialCpuStatus: "ok" | "disabled" | "error" = "ok"
 let lastCleanup = Date.now()
 const stats: Stats = { hits: 0, misses: 0, evictions: 0, cpuSkips: 0 }
 
+// Capability flags (learned at runtime, never assumed):
+// `canServe` turns true only after we capture a real response body, and gates
+// short-circuiting so we never replace the outgoing request payload with garbage.
+let canServe = false
+let bodyWarningShown = false
+
 // ---------------------------------------------------------------------------
 // Core helpers
 // ---------------------------------------------------------------------------
@@ -231,12 +244,17 @@ export default async function (pi: ExtensionAPI) {
   pi.on("session_shutdown", () => clearInterval(cleanupTimer))
 
   // Interceptor: cache lookup before provider request
-  pi.on("before_provider_request", async (event, ctx) => {
+  pi.on("before_provider_request", async (event: BeforeProviderRequestEvent, ctx) => {
     if (!settings.enabled) return
 
     const model = ctx.model?.id || "unknown"
-    const messages = event.messages || []
-    const params = event.parameters || {}
+    // pi passes the assembled provider request as `event.payload`
+    const payload = event.payload as
+      | { messages?: unknown[]; parameters?: unknown }
+      | null
+      | undefined
+    const messages = payload?.messages ?? []
+    const params = payload?.parameters ?? {}
 
     // Fast hash; no per-request CPU check to stay on the hot path
     const key = fastHash(model + JSON.stringify(messages) + JSON.stringify(params))
@@ -244,22 +262,42 @@ export default async function (pi: ExtensionAPI) {
 
     if (entry && Date.now() - entry.timestamp <= settings.ttlSeconds * 1000) {
       stats.hits++
-      return entry.response as never
+      // Only replace the payload when we have actually captured a response body.
+      // Without one (pi 0.84 +) a return value would overwrite the outgoing
+      // request payload instead of short-circuiting — so we count the hit and
+      // leave the request untouched.
+      if (canServe) return entry.response
+      return
     }
 
     // Miss — remember the key for after_provider_response
-    event._cacheKey = key
+    ;(event as unknown as { _cacheKey?: string })._cacheKey = key
     stats.misses++
   })
 
   // Interceptor: store response after provider finishes
-  pi.on("after_provider_response", async (event) => {
+  pi.on("after_provider_response", async (event: AfterProviderResponseEvent) => {
     if (!settings.enabled) return
-    if (!event._cacheKey) return
 
-    const key = event._cacheKey as string
-    const response = event.response ?? event.choices ?? event
+    // pi 0.84 + exposes only { status, headers } here — no response body.
+    // Guard so we never cache the event envelope as if it were a response.
+    const e = event as unknown as { _cacheKey?: string; response?: unknown; choices?: unknown }
+    const response = e.response ?? e.choices
+    if (response === undefined) {
+      if (!bodyWarningShown) {
+        bodyWarningShown = true
+        console.log(
+          "[l1-cache] note: this pi version does not expose a response body in 'after_provider_response'; " +
+            "responses cannot be cached. Stats remain available via /l1-cache.",
+        )
+      }
+      return
+    }
+    if (!e._cacheKey) return
+
+    const key = e._cacheKey
     const size = estimateSize(response)
+    canServe = true
 
     cache.set(key, { response, timestamp: Date.now(), sizeBytes: size })
     totalMemory += size
@@ -269,7 +307,7 @@ export default async function (pi: ExtensionAPI) {
   // Commands
   pi.registerCommand("l1-cache", {
     description: "Show L1 cache stats, or use 'clear' to reset",
-    handler: async (args: string, ctx: ExtensionContext) => {
+    handler: async (args: string, ctx: ExtensionCommandContext) => {
       const arg = (args ?? "").trim()
 
       if (arg === "clear") {
@@ -278,7 +316,7 @@ export default async function (pi: ExtensionAPI) {
         stats.hits = 0
         stats.misses = 0
         stats.evictions = 0
-        ctx.ui.notify("L1 cache cleared", "success")
+        ctx.ui.notify("L1 cache cleared", "info")
         return
       }
 
@@ -287,6 +325,7 @@ export default async function (pi: ExtensionAPI) {
           stats.hits + stats.misses > 0 ? ((stats.hits / (stats.hits + stats.misses)) * 100).toFixed(1) : "0.0"
         const lines = [
           `L1 cache status: ${settings.enabled ? "ENABLED" : "disabled"}`,
+          `Response caching: ${canServe ? "active" : "unavailable (no response body in this pi API)"}`,
           `Entries: ${cache.size} / ${settings.maxEntries}`,
           `Memory: ${(totalMemory / 1024 / 1024).toFixed(1)}MB / ${(settings.maxMemoryBytes / 1024 / 1024).toFixed(1)}MB`,
           `TTL: ${settings.ttlSeconds}s | CPU threshold: ${settings.cpuThreshold}%`,
@@ -296,23 +335,22 @@ export default async function (pi: ExtensionAPI) {
           `Last cleanup: ${new Date(lastCleanup).toISOString()}`,
         ]
         ctx.ui.notify(lines.join("\n"), "info")
-        return ""
+        return
       }
 
       if (arg === "enable") {
         settings.enabled = true
-        ctx.ui.notify("L1 cache enabled", "success")
+        ctx.ui.notify("L1 cache enabled", "info")
         return
       }
 
       if (arg === "disable") {
         settings.enabled = false
-        ctx.ui.notify("L1 cache disabled", "success")
+        ctx.ui.notify("L1 cache disabled", "info")
         return
       }
 
       ctx.ui.notify(`Unknown command: /l1-cache ${arg}\nUsage: /l1-cache [stats|clear|enable|disable]`, "error")
-      return ""
     },
   })