llmbrake 0.1.0 → 0.2.0

package/README.md

@@ -2,6 +2,7 @@
 
 **Per-customer cost caps for AI apps: put a monthly dollar budget on every customer, switch them to a cheaper model near the cap, and block them at the cap.**
 
+[![npm version](https://img.shields.io/npm/v/llmbrake.svg)](https://www.npmjs.com/package/llmbrake)
 [![CI](https://github.com/llmbrake/llmbrake/actions/workflows/ci.yml/badge.svg)](https://github.com/llmbrake/llmbrake/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
@@ -19,6 +20,8 @@ Status: `0.1.0`, early. Website: [llmbrake.com](https://llmbrake.com)
 npm install llmbrake
 ```
 
+Published on npm as [`llmbrake`](https://www.npmjs.com/package/llmbrake). To try the latest `main` instead, run `npm install github:llmbrake/llmbrake` (it builds on install).
+
 `llmbrake` is ESM-only (`import`, not `require`).
 
 ## 30-second quickstart
@@ -85,6 +88,16 @@ With `stream: true`, `llmbrake` sets `stream_options.include_usage = true` and r
 
 Any `create(params)` function that accepts `{ model, ... }` and returns `{ model, usage: { prompt_tokens, completion_tokens } }` works, for example OpenRouter, Anthropic's OpenAI-compatible endpoint and local gateways. Add their model IDs to the price table.
 
+## Examples: Vercel AI SDK, OpenRouter, LangChain
+
+Tested integrations live in [`examples/`](examples/):
+
+- **[Vercel AI SDK](examples/vercel-ai-sdk/)**: `guard.check()` before `generateText` / `streamText`, then charge from `result.usage` (cached tokens included). Also has a Next.js route handler sketch.
+- **[OpenRouter](examples/openrouter/)**: `guard.wrap()` on the OpenAI SDK pointed at OpenRouter, with OpenRouter model IDs in the price table.
+- **[LangChain.js](examples/langchain/)**: `guard.check()` before `invoke()`, then charge from `AIMessage.usage_metadata`.
+
+CI runs them against mock providers (no API keys).
+
 ## Storage (Redis example)
 
 The default `MemoryStore` keeps totals in process memory: fine for one server, tests and demos, but lost on restart and not shared between instances. For multiple instances or serverless, implement the two-method `UsageStore` interface. With [ioredis](https://github.com/redis/ioredis):
@@ -118,6 +131,24 @@ const guard = createGuard({ budgetUsd: 5, store: redisStore });
 
 `INCRBYFLOAT` is atomic, so totals stay correct across instances. If the store throws (or returns a non-numeric value), calls **fail open** by default so your product keeps working; set `failOpen: false` to fail closed instead.
 
+## Shared budgets across servers (`HostedStore`)
+
+`HostedStore` keeps the counters in LLMBrake Cloud, so every server and serverless instance using your project's API key shares one counter per customer, and budgets you edit in the dashboard apply on the next call, with no Redis to run:
+
+```ts
+import { createGuard, HostedStore } from "llmbrake";
+
+const guard = createGuard({
+  budgetUsd: 5, // default; a budget set in the dashboard for a customer wins
+  downgradeMap: { "gpt-5.5": "gpt-5.4-mini" },
+  store: new HostedStore({ apiKey: process.env.LLMBRAKE_API_KEY! }),
+});
+```
+
+It also closes the concurrent-overshoot gap: before each call the guard **reserves** an estimate of the call's cost (prompt size plus `max_tokens`, or `estimateUsd`), and the reservation only succeeds if spend plus other in-flight reservations is under the budget, in one atomic step on the server. After the call it **settles** the real cost and releases the hold (failed provider calls release it at $0). Parallel calls can then overshoot by at most about one call, instead of by everything in flight. If the API is unreachable the guard follows `failOpen` (default: let the call through).
+
+Any store can opt in to this by implementing `reserve` and `settle` (the `ReservingStore` interface).
+
 ## Options
 
 `createGuard(options)` returns `{ wrap, check, status }`.
@@ -133,6 +164,7 @@ const guard = createGuard({ budgetUsd: 5, store: redisStore });
 | `unknownModel` | `"throw"` | Model missing from `prices`: `"throw"` an `UnknownModelPriceError` before the call, or `"allow"` it and record `costUsd: null`. |
 | `failOpen` | `true` | If the store errors: `true` lets calls through, `false` rethrows. |
 | `now` | `() => new Date()` | Clock override, mainly for tests. |
+| `estimateUsd` | estimated | Reserving stores only (`HostedStore`): USD to hold while a call is in flight, a number or `(ctx, params) => number`. Default: prompt length / 4 tokens plus `max_completion_tokens`/`max_tokens` (or 1,000) at the requested model's price. |
 
 The call context passed as the second argument to the wrapped function is `{ customerId: string; plan?: string; feature?: string }`.
 
@@ -158,15 +190,27 @@ Dated snapshot IDs such as `gpt-5.4-mini-2026-03-17` fall back to the undated en
 
 ## Known limitations
 
-- **Concurrent overshoot.** The budget is checked before a call and the cost is recorded after it. Several in-flight calls for the same customer can each pass the check, so the cap can be overshot by roughly the cost of those concurrent calls. Keep a margin in your budgets if this matters.
+- **Concurrent overshoot.** With `MemoryStore` or a plain `UsageStore`, the budget is checked before a call and the cost is recorded after it, so several in-flight calls for the same customer can each pass the check and overshoot the cap by roughly their combined cost. `HostedStore` (or any `ReservingStore`) reserves before each call, which bounds the overshoot to about one call.
 - **Prices need verification.** See above. Long-context tiers, batch discounts, regional or data-residency uplifts and priority tiers are not modeled, and the Anthropic IDs in the default table may differ from what your endpoint accepts.
 - **Chat completions only.** Only chat-completions-style token usage is metered: no images, audio, embeddings or tool fees yet.
 - **Streaming needs usage.** If a provider ignores `include_usage`, or a stream is abandoned before the usage chunk arrives, that call is recorded with zero cost.
 - **UTC calendar months.** Budget periods reset at 00:00 UTC on the 1st.
 
-## Hosted dashboard (coming)
+## Python
+
+A Python port with the same behavior lives in [`python/`](python/README.md): sync and async `openai` clients, streaming, pluggable stores (in-memory, Redis), zero required dependencies, Python 3.9+.
+
+`pip install llmbrake` is coming soon to PyPI. For now, install from GitHub:
+
+```bash
+pip install "git+https://github.com/llmbrake/llmbrake#subdirectory=python"
+```
+
+See [python/README.md](python/README.md) for the quickstart.
+
+## Hosted dashboard (LLMBrake Cloud)
 
-The library is free and stays MIT licensed. We're building an optional hosted dashboard on top of it: cost per customer, feature and model, shared counters, alerts near and at caps, and AI cost vs. Stripe revenue per customer. It is **not available yet**; join the waitlist at [llmbrake.com](https://llmbrake.com).
+The library is free and stays MIT licensed. LLMBrake Cloud is the optional paid layer on top: shared counters (`HostedStore`), a dashboard with spend, budget and status per customer (edit budgets live), Slack/webhook alerts at 80% and 100%, and AI cost vs. Stripe revenue per customer. See [llmbrake.com](https://llmbrake.com).
 
 ## Contributing
 

package/dist/guard.d.ts

@@ -41,7 +41,15 @@ export interface GuardOptions {
     failOpen?: boolean;
     /** Clock override for tests. */
     now?: () => Date;
+    /**
+     * Only used with a reserving store (e.g. HostedStore): the cost to hold while a call is in flight.
+     * Default: estimated from the prompt size (~4 chars/token) plus `max_completion_tokens`/`max_tokens`
+     * (or 1,000 output tokens) at the requested model's price.
+     */
+    estimateUsd?: number | ((ctx: CallContext, params: Record<string, unknown>) => number);
 }
+/** Rough upper-ish estimate of a call's cost, used only to size reservations. */
+export declare function estimateCallUsd(model: string, params: Record<string, unknown>, prices?: PriceTable): number;
 type ChatParams = {
     model: string;
     stream?: boolean | null;
@@ -58,6 +66,7 @@ export declare function createGuard(opts: GuardOptions): {
         spent: number;
         model: string;
         downgraded: boolean;
+        holdId: string | null;
     }>;
     status: (ctx: CallContext) => Promise<{
         period: string;

package/dist/guard.js

@@ -1,6 +1,17 @@
 import { BudgetExceededError, UnknownModelPriceError } from "./errors.js";
 import { DEFAULT_PRICES, computeCostUsd } from "./prices.js";
-import { MemoryStore } from "./store.js";
+import { MemoryStore, isReservingStore } from "./store.js";
+/** Rough upper-ish estimate of a call's cost, used only to size reservations. */
+export function estimateCallUsd(model, params, prices = DEFAULT_PRICES) {
+    let chars = 0;
+    try {
+        chars = JSON.stringify(params.messages ?? params.input ?? params.prompt ?? "").length;
+    }
+    catch { /* circular: ignore */ }
+    const maxOut = Number(params.max_completion_tokens ?? params.max_tokens ?? 1000);
+    const cost = computeCostUsd(model, { prompt_tokens: Math.ceil(chars / 4), completion_tokens: Number.isFinite(maxOut) && maxOut > 0 ? maxOut : 1000 }, prices);
+    return cost ?? 0;
+}
 export function periodOf(d) {
     return `${d.getUTCFullYear()}-${String(d.getUTCMonth() + 1).padStart(2, "0")}`;
 }
@@ -49,7 +60,63 @@ export function createGuard(opts) {
             effective = downgradeMap[model];
         if (unknownModel === "throw" && computeCostUsd(effective, {}, prices) === null)
             throw new UnknownModelPriceError(effective);
-        return { period, budget, spent, model: effective, downgraded: effective !== model };
+        return { period, budget, spent, model: effective, downgraded: effective !== model, holdId: null };
+    }
+    function decide(ctx, model, period, budget, spent) {
+        if (spent >= budget)
+            throw new BudgetExceededError(ctx.customerId, spent, budget, period);
+        let effective = model;
+        if (budget !== Infinity && spent >= budget * downgradeAt && downgradeMap[model])
+            effective = downgradeMap[model];
+        if (unknownModel === "throw" && computeCostUsd(effective, {}, prices) === null)
+            throw new UnknownModelPriceError(effective);
+        return { period, budget, spent, model: effective, downgraded: effective !== model, holdId: null };
+    }
+    /** Reserving stores: atomic check-and-reserve before the call (see ReservingStore). */
+    async function reserveCheck(ctx, params) {
+        const period = periodOf(now());
+        const budget = await budgetFor(ctx);
+        if (!isReservingStore(store))
+            throw new Error("unreachable");
+        let holdUsd = 0;
+        try {
+            holdUsd = typeof opts.estimateUsd === "function" ? opts.estimateUsd(ctx, params) : opts.estimateUsd ?? estimateCallUsd(params.model, params, prices);
+            if (!Number.isFinite(holdUsd) || holdUsd < 0)
+                holdUsd = 0;
+        }
+        catch {
+            holdUsd = 0;
+        }
+        let r;
+        try {
+            r = await store.reserve({ customerId: ctx.customerId, period, holdUsd, budgetUsd: budget, downgradeAt, stream: !!params.stream });
+            if (typeof r?.spentUsd !== "number" || !Number.isFinite(r.spentUsd))
+                throw new TypeError("llmbrake: store.reserve returned a non-numeric spentUsd");
+        }
+        catch (e) {
+            if (!failOpen)
+                throw e;
+            return decide(ctx, params.model, period, budget, 0);
+        }
+        const effBudget = typeof r.budgetUsd === "number" && !Number.isNaN(r.budgetUsd) ? r.budgetUsd : budget;
+        if (!r.allowed)
+            throw new BudgetExceededError(ctx.customerId, r.spentUsd, effBudget, period);
+        // The server already admitted the call; only the downgrade and unknown-model rules apply here.
+        const model = params.model;
+        const effective = effBudget !== Infinity && r.spentUsd >= effBudget * downgradeAt && downgradeMap[model] ? downgradeMap[model] : model;
+        if (unknownModel === "throw" && computeCostUsd(effective, {}, prices) === null) {
+            await release(ctx, period, r.holdId);
+            throw new UnknownModelPriceError(effective);
+        }
+        return { period, budget: effBudget, spent: r.spentUsd, model: effective, downgraded: effective !== model, holdId: r.holdId };
+    }
+    async function release(ctx, period, holdId) {
+        if (!holdId || !isReservingStore(store))
+            return;
+        try {
+            await store.settle({ holdId, customerId: ctx.customerId, period, costUsd: 0, downgraded: false });
+        }
+        catch { /* hold expires on its own */ }
     }
     async function record(ctx, d, requestedModel, respModel, usage) {
         const u = usage ?? {};
@@ -57,7 +124,16 @@ export function createGuard(opts) {
         const priceModel = respModel && computeCostUsd(respModel, {}, prices) !== null ? respModel : d.model;
         const cost = computeCostUsd(priceModel, u, prices);
         let spent = d.spent;
-        if (cost !== null && cost > 0) {
+        if (d.holdId && isReservingStore(store)) {
+            try {
+                spent = await store.settle({ holdId: d.holdId, customerId: ctx.customerId, period: d.period, costUsd: cost ?? 0, downgraded: d.downgraded });
+            }
+            catch (e) {
+                if (!failOpen)
+                    throw e;
+            }
+        }
+        else if (cost !== null && cost > 0) {
             try {
                 spent = await store.addSpend(ctx.customerId, d.period, cost);
             }
@@ -91,14 +167,28 @@ export function createGuard(opts) {
             if (!ctx?.customerId)
                 throw new TypeError("llmbrake: ctx.customerId is required");
             const requestedModel = params.model;
-            const d = await check(ctx, requestedModel);
+            const d = isReservingStore(store) ? await reserveCheck(ctx, params) : await check(ctx, requestedModel);
             const finalParams = { ...params, model: d.model };
             if (params.stream) {
                 finalParams.stream_options = { ...(params.stream_options ?? {}), include_usage: true };
-                const stream = (await createFn(finalParams, ...rest));
+                let stream;
+                try {
+                    stream = (await createFn(finalParams, ...rest));
+                }
+                catch (e) {
+                    await release(ctx, d.period, d.holdId);
+                    throw e;
+                }
                 return wrapStream(stream, (model, usage) => record(ctx, d, requestedModel, model, usage));
             }
-            const res = await createFn(finalParams, ...rest);
+            let res;
+            try {
+                res = await createFn(finalParams, ...rest);
+            }
+            catch (e) {
+                await release(ctx, d.period, d.holdId);
+                throw e;
+            }
             await record(ctx, d, requestedModel, res?.model, res?.usage);
             return res;
         };

package/dist/hosted.d.ts

@@ -0,0 +1,42 @@
+import type { Reservation, ReserveRequest, ReservingStore, SettleRequest } from "./store.js";
+export declare const DEFAULT_HOSTED_URL = "https://app.llmbrake.com";
+export interface HostedStoreOptions {
+    /** Your project's SDK API key (lbsk_...). */
+    apiKey: string;
+    /** API base URL. Default https://app.llmbrake.com */
+    baseUrl?: string;
+    /** Per-request timeout in ms. Default 3000. On timeout the guard follows `failOpen`. */
+    timeoutMs?: number;
+    /** fetch implementation (defaults to global fetch; Node 18+). */
+    fetch?: typeof fetch;
+}
+export declare class HostedStoreError extends Error {
+    readonly status?: number | undefined;
+    constructor(message: string, status?: number | undefined);
+}
+/**
+ * Shared, hosted spend counters (LLMBrake Cloud). One counter per customer per month across every
+ * server using the same API key; budgets edited in the dashboard override the guard's `budgetUsd`.
+ *
+ *   const guard = createGuard({ budgetUsd: 5, store: new HostedStore({ apiKey: process.env.LLMBRAKE_API_KEY! }) });
+ */
+export declare class HostedStore implements ReservingStore {
+    private readonly base;
+    private readonly key;
+    private readonly timeoutMs;
+    private readonly f;
+    constructor(opts: HostedStoreOptions);
+    private call;
+    getSpend(customerId: string, period: string): Promise<number>;
+    addSpend(customerId: string, period: string, usd: number): Promise<number>;
+    reserve(req: ReserveRequest): Promise<Reservation>;
+    settle(req: SettleRequest): Promise<number>;
+    /** Server view of a customer this period (spend, in-flight holds, effective budget, status). */
+    customer(customerId: string, period: string): Promise<{
+        spentUsd: number;
+        pendingUsd: number;
+        budgetUsd: number | null;
+        budgetSource: string;
+        status: string;
+    }>;
+}

package/dist/hosted.js

@@ -0,0 +1,79 @@
+export const DEFAULT_HOSTED_URL = "https://app.llmbrake.com";
+export class HostedStoreError extends Error {
+    status;
+    constructor(message, status) {
+        super(message);
+        this.status = status;
+        this.name = "HostedStoreError";
+    }
+}
+/**
+ * Shared, hosted spend counters (LLMBrake Cloud). One counter per customer per month across every
+ * server using the same API key; budgets edited in the dashboard override the guard's `budgetUsd`.
+ *
+ *   const guard = createGuard({ budgetUsd: 5, store: new HostedStore({ apiKey: process.env.LLMBRAKE_API_KEY! }) });
+ */
+export class HostedStore {
+    base;
+    key;
+    timeoutMs;
+    f;
+    constructor(opts) {
+        if (!opts?.apiKey)
+            throw new TypeError("llmbrake: HostedStore needs an apiKey");
+        this.key = opts.apiKey;
+        this.base = (opts.baseUrl ?? DEFAULT_HOSTED_URL).replace(/\/+$/, "");
+        this.timeoutMs = opts.timeoutMs ?? 3000;
+        const f = opts.fetch ?? globalThis.fetch;
+        if (!f)
+            throw new TypeError("llmbrake: HostedStore needs fetch (Node 18+), or pass opts.fetch");
+        this.f = f;
+    }
+    async call(method, path, body) {
+        const ctl = new AbortController();
+        const t = setTimeout(() => ctl.abort(), this.timeoutMs);
+        try {
+            const r = await this.f(this.base + path, {
+                method,
+                headers: { authorization: `Bearer ${this.key}`, "content-type": "application/json", "user-agent": "llmbrake-js" },
+                body: body === undefined ? undefined : JSON.stringify(body),
+                signal: ctl.signal,
+            });
+            const data = await r.json().catch(() => ({}));
+            if (!r.ok)
+                throw new HostedStoreError(`llmbrake: hosted API ${r.status} ${data?.error ?? ""}`.trim(), r.status);
+            return data;
+        }
+        catch (e) {
+            if (e instanceof HostedStoreError)
+                throw e;
+            throw new HostedStoreError(`llmbrake: hosted API unreachable (${e?.name === "AbortError" ? `timeout after ${this.timeoutMs}ms` : String(e?.message ?? e)})`);
+        }
+        finally {
+            clearTimeout(t);
+        }
+    }
+    async getSpend(customerId, period) {
+        const q = new URLSearchParams({ customerId, period });
+        return (await this.call("GET", `/v1/spend?${q}`)).spentUsd;
+    }
+    async addSpend(customerId, period, usd) {
+        return (await this.call("POST", "/v1/spend/add", { customerId, period, usd })).spentUsd;
+    }
+    async reserve(req) {
+        const r = await this.call("POST", "/v1/reserve", {
+            customerId: req.customerId, period: req.period, holdUsd: req.holdUsd,
+            budgetUsd: Number.isFinite(req.budgetUsd) ? req.budgetUsd : null,
+            downgradeAt: req.downgradeAt, ttlSec: req.stream ? 900 : 120,
+        });
+        return { allowed: !!r.allowed, spentUsd: r.spentUsd, budgetUsd: r.budgetUsd === null ? Infinity : r.budgetUsd, holdId: r.holdId ?? null };
+    }
+    async settle(req) {
+        return (await this.call("POST", "/v1/settle", req)).spentUsd;
+    }
+    /** Server view of a customer this period (spend, in-flight holds, effective budget, status). */
+    async customer(customerId, period) {
+        const q = new URLSearchParams({ customerId, period });
+        return this.call("GET", `/v1/spend?${q}`);
+    }
+}

package/dist/index.d.ts

@@ -1,7 +1,9 @@
-export { createGuard, periodOf } from "./guard.js";
+export { createGuard, periodOf, estimateCallUsd } from "./guard.js";
 export type { GuardOptions, CallContext, UsageEvent } from "./guard.js";
-export { MemoryStore } from "./store.js";
-export type { UsageStore } from "./store.js";
+export { MemoryStore, isReservingStore } from "./store.js";
+export type { UsageStore, ReservingStore, Reservation, ReserveRequest, SettleRequest } from "./store.js";
+export { HostedStore, HostedStoreError, DEFAULT_HOSTED_URL } from "./hosted.js";
+export type { HostedStoreOptions } from "./hosted.js";
 export { DEFAULT_PRICES, computeCostUsd, lookupPrice } from "./prices.js";
 export type { ModelPrice, PriceTable, ChatUsage } from "./prices.js";
 export { BudgetExceededError, UnknownModelPriceError, isBudgetExceededError } from "./errors.js";

package/dist/index.js

@@ -1,4 +1,5 @@
-export { createGuard, periodOf } from "./guard.js";
-export { MemoryStore } from "./store.js";
+export { createGuard, periodOf, estimateCallUsd } from "./guard.js";
+export { MemoryStore, isReservingStore } from "./store.js";
+export { HostedStore, HostedStoreError, DEFAULT_HOSTED_URL } from "./hosted.js";
 export { DEFAULT_PRICES, computeCostUsd, lookupPrice } from "./prices.js";
 export { BudgetExceededError, UnknownModelPriceError, isBudgetExceededError } from "./errors.js";

package/dist/store.d.ts

@@ -5,6 +5,46 @@ export interface UsageStore {
     /** Atomically add USD and return the new total. */
     addSpend(customerId: string, period: string, usd: number): Promise<number>;
 }
+/** What a {@link ReservingStore} answers before a call. */
+export interface Reservation {
+    /** false = the customer is at/over budget (counting other in-flight calls); the call must not be made. */
+    allowed: boolean;
+    /** Settled spend this period (USD). */
+    spentUsd: number;
+    /** Effective budget (USD). Infinity = unlimited. May differ from the one you sent if a dashboard override is set. */
+    budgetUsd: number;
+    /** Pass back to `settle`. null when not allowed (or when the store failed open). */
+    holdId: string | null;
+}
+export interface ReserveRequest {
+    customerId: string;
+    period: string;
+    /** Estimated cost of the call, held until it settles. */
+    holdUsd: number;
+    /** Budget from your guard config. Infinity = unlimited. */
+    budgetUsd: number;
+    downgradeAt: number;
+    /** Streams get a longer hold before it expires on its own. */
+    stream: boolean;
+}
+export interface SettleRequest {
+    holdId: string | null;
+    customerId: string;
+    period: string;
+    /** Actual cost; 0 releases the hold without charging (e.g. the provider call failed). */
+    costUsd: number;
+    downgraded: boolean;
+}
+/**
+ * Optional extension of {@link UsageStore} for shared stores (e.g. {@link HostedStore}).
+ * When the guard's store implements it, wrapped calls *check-and-reserve* atomically before the
+ * provider call and *settle* after it, so parallel calls across servers can't overshoot the cap together.
+ */
+export interface ReservingStore extends UsageStore {
+    reserve(req: ReserveRequest): Promise<Reservation>;
+    settle(req: SettleRequest): Promise<number>;
+}
+export declare function isReservingStore(s: UsageStore): s is ReservingStore;
 /** In-memory store. Fine for a single process, tests and demos; state is lost on restart. */
 export declare class MemoryStore implements UsageStore {
     private data;

package/dist/store.js

@@ -1,3 +1,6 @@
+export function isReservingStore(s) {
+    return typeof s.reserve === "function" && typeof s.settle === "function";
+}
 /** In-memory store. Fine for a single process, tests and demos; state is lost on restart. */
 export class MemoryStore {
     data = new Map();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "llmbrake",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Per-customer monthly USD budgets for OpenAI-compatible chat calls: track cost, auto-downgrade near the cap, block at the cap.",
   "license": "MIT",
   "author": "LLMBrake contributors",