ai-sdk-rate-limiter 0.12.0 → 0.13.0

package/CHANGELOG.md

@@ -0,0 +1,195 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [Unreleased]
+
+---
+
+## [0.13.0] - 2026-03-13
+
+### Added
+- **`limiter.reset()`** — clear all rate-limit, queue, cost, and circuit-breaker state without recreating the instance. Queued requests are rejected with `ShutdownError`. Primarily useful in tests to reset between cases with a shared limiter instance.
+- **`queue.onFull: 'drop-low'`** fully implemented — when the queue is at capacity and a `high` or `normal` priority request arrives, the tail `low`-priority waiter is evicted (rejected with `QueueFullError`) to make room. Useful for mixed workloads where background batch jobs should never block user-facing requests.
+
+### Changed
+- Unknown models with zero pricing now emit a one-time `console.warn` on first use, pointing at the exact `config.limits` fix needed to enable cost tracking. Known registry models and models with user-supplied pricing are silent.
+
+---
+
+## [0.12.0] - 2026-03-12
+
+### Added
+- **`getCostForecast()`** — project end-of-period spend based on the current hourly rate. Returns `{ hour, day, month }` each with `spentUsd`, `projectedUsd`, and `ratePerHourUsd`. Useful for alerting before a budget cap is hit.
+- **`createModelPool(models, options?)`** — round-robin (or random) load balancer across multiple wrapped model instances. Distributes calls evenly across API keys or model variants. Import from `ai-sdk-rate-limiter`.
+- **Request deduplication** — pass `dedupKey` in `providerOptions.rateLimiter` to make concurrent identical requests share a single API call. All callers receive the same result; the dedup entry is cleared on completion so the next request always makes a fresh call.
+
+---
+
+## [0.11.0] - 2026-03-12
+
+### Added
+- **Debug mode** — set `debug: true` on `createRateLimiter()` to enable structured console logging for every rate-limit decision, queue entry/exit, slot acquisition, circuit breaker state change, and cost recording. Zero overhead when disabled.
+- **Config validation** — `createRateLimiter()` now validates your config at construction time and emits `console.warn` for common misconfigurations:
+  - `cost.store` set without calling `warmUp()` reminder
+  - `circuit.failureThreshold < 3` (too sensitive, risks false trips)
+  - `retry.retryOn` explicitly excludes 429 (defeats rate-limit retry)
+  - `queue.timeout < 3000ms` (too short, requests will time out before serving)
+  - `cost.budget` set without `onExceeded` (uses silent default `'throw'`)
+  - `cost.onExceeded: 'fallback'` reminder to configure fallback model
+- GitHub Actions CI workflow (Node 18 / 20 / 22 matrix)
+- `CHANGELOG.md` with retroactive entries from v0.1.0
+
+### Fixed
+- `DebugLogger` details serialization: empty objects no longer emit trailing `()`
+
+---
+
+## [0.10.0] - 2026-03-12
+
+### Added
+- `ai-sdk-rate-limiter/middleware` entry point
+  - `createRateLimiterMiddleware(limiter, opts)` — returns `{ middleware, errorHandler }` for Express
+  - `createRateLimiterErrorHandler(opts)` — standalone 4-arg Express error handler
+  - `createHonoMiddleware(limiter, opts)` — Hono middleware with `c.var.rateLimiter`
+  - `mapErrorToResponse(err)` — utility for Fastify and custom frameworks
+  - `RateLimiterRequestContext` type + `http.IncomingMessage` augmentation for `req.rateLimiter`
+  - Automatic error → HTTP mapping: `QueueTimeoutError` → 503, `BudgetExceededError` → 402, etc.
+  - `injectHeaders` option: adds `X-RateLimit-*` informational headers to responses
+
+### Added (examples)
+- `examples/multi-tenant-express/` — Express API with per-user scoped rate limits (free/pro tiers)
+- `examples/batch-processing/` — Concurrent batch jobs with priority queuing + graceful shutdown
+- `examples/budget-alerts/` — Slack/webhook budget alerts with per-scope spend breakdown
+
+---
+
+## [0.9.0] - 2026-03-12
+
+### Added
+- **Circuit breaker** — auto-opens on repeated 5xx failures, half-open probe, configurable thresholds
+  - `CircuitBreakerConfig` in `RateLimiterConfig.circuit`
+  - `CircuitOpenError` thrown when circuit is open
+  - `circuitOpen` / `circuitClosed` events
+- **Graceful shutdown** — `limiter.shutdown({ drainMs })`, `ShutdownError`
+- **Persistent cost tracking** — `CostStore` interface + `RedisCostStore` in `ai-sdk-rate-limiter/redis`
+  - `limiter.warmUp()` pre-loads historical spend on startup
+- **Per-scope cost attribution** — `getCostReport().byScope` breakdown per user/org/tenant
+- **Fallback chains** — `fallback` accepts `WrappableModel[]`, walked in order on `BudgetExceededError`
+- **Call timeout** — `callTimeout` in retry config and per-request options (uses `Promise.race`)
+- **Auto-detected limits** — parses `x-ratelimit-limit-*` response headers, user config wins
+  - `limitsDetected` event
+- **Prometheus metrics** — `createPrometheusPlugin()` in `ai-sdk-rate-limiter/prometheus`
+- **StatsD / DogStatsD** — `createStatsDPlugin(client)` in `ai-sdk-rate-limiter/statsd`
+- **Drop hooks** — `DroppedEvent` now includes `reason`, `waitedMs`, `queueDepth`, `scope`, `metadata`
+
+---
+
+## [0.8.0] - 2026-03-11
+
+### Added
+- Redis store for multi-instance rate limiting (`ai-sdk-rate-limiter/redis`)
+  - `RedisStore` — Lua-script-based atomic sliding window shared across instances
+  - Fail-open on Redis errors (enforcement suspended, never blocks requests)
+  - Compatible with ioredis, node-redis, Upstash Redis
+- `rpd` (requests per day) limit support — rolling 24-hour window
+- `otpm` (output tokens per minute) limit support — based on completed request actuals
+
+---
+
+## [0.7.1] - 2026-03-10
+
+### Fixed
+- `Retry-After` header parsing: correctly handles duration strings like `"6m30s"` (previously parsed as 6s)
+
+---
+
+## [0.7.0] - 2026-03-10
+
+### Added
+- Raw SDK proxy — `limiter.rawProxy(client)` wraps native OpenAI/Anthropic/Groq/Mistral/Cohere clients
+  - Transparent `Proxy`-based drop-in with no API changes
+  - Streaming support via `AsyncIterable` wrapping for usage chunk capture
+  - `rateLimited(client, opts)` standalone factory
+- Budget fallback routing — `onExceeded: 'fallback'` transparently reroutes to a cheaper model
+  - `limiter.wrap(model, { fallback: cheaperModel })`
+  - `usingFallback` field on `budgetHit` event
+
+---
+
+## [0.6.0] - 2026-03-09
+
+### Added
+- OpenTelemetry plugin (`ai-sdk-rate-limiter/otel`)
+  - `createOtelPlugin(tracer)` — emits GenAI-spec spans for every request
+  - No hard dependency on `@opentelemetry/api` (structural typing)
+  - Span duration reconstructed from `latencyMs` for accurate wall-clock timing
+- Testing utilities (`ai-sdk-rate-limiter/testing`)
+  - `createTestLimiter()` — records all completed calls for assertions
+  - `limiter.getCalls()` / `limiter.reset()`
+
+---
+
+## [0.5.0] - 2026-03-08
+
+### Added
+- Concurrency limits — `maxConcurrent` per model, enforced as a semaphore
+- Multi-tenant scoped limits — `config.scopes` with `*` wildcard patterns
+  - Each scope gets its own independent sliding window
+  - Per-request scope via `providerOptions.rateLimiter.scope`
+- `queue.onFull: 'drop-low'` — evict lowest-priority requests before throwing `QueueFullError`
+- `AbortSignal` propagation through both rate-limit and concurrency queues
+
+---
+
+## [0.4.0] - 2026-03-07
+
+### Added
+- Priority queue — `high` / `normal` / `low` priorities; FIFO within same priority
+- Per-request options via `providerOptions.rateLimiter` (priority, timeout, scope)
+- `limiter.estimatedWait(modelId)` — returns ms until next available slot
+- `QueueFullError` when queue is at `maxSize` capacity
+
+---
+
+## [0.3.0] - 2026-03-06
+
+### Added
+- Cost tracking — records actual token usage per request
+  - `getCostReport()` with hourly / daily / monthly rolling windows
+  - `byModel` breakdown in cost report
+- Budget caps — `cost.budget` with `hourly`, `daily`, `monthly` limits
+  - `onExceeded: 'throw' | 'queue'` behavior
+  - `BudgetExceededError` with period, current spend, and limit
+- `nextBudgetClearMs()` used internally for queue-mode budget holds
+
+---
+
+## [0.2.0] - 2026-03-05
+
+### Added
+- Model registry expanded to include Groq, Mistral, and Cohere
+  - `GROQ_MODELS`, `MISTRAL_MODELS`, `COHERE_MODELS`
+  - `isKnownModel(modelId, provider)` utility
+
+---
+
+## [0.1.0] - 2026-03-04
+
+### Added
+- Initial release
+- Sliding window rate limiting (RPM + ITPM) for OpenAI, Anthropic, Google Gemini models
+- Priority queue with drain timer (`scheduleDrain` per model)
+- Exponential backoff retry with jitter, `Retry-After` header support
+- Vercel AI SDK `.wrap()` adapter via `LanguageModelV4Middleware`
+- In-memory store (default, zero config)
+- `createRateLimiter()` factory
+- `limiter.getStatus()` — queue depths and window state per model
+- Event system: `queued`, `dequeued`, `retrying`, `rateLimited`, `budgetHit`, `dropped`, `completed`
+- `RateLimiterError` hierarchy: `RateLimitExceededError`, `QueueTimeoutError`, `RetryExhaustedError`
+- Built-in model registry for OpenAI and Anthropic with pricing data

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 ai-sdk-rate-limiter contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -4,12 +4,16 @@ Smart rate limiting, queuing, and cost tracking for AI API calls. Works across p
 
 [![npm](https://img.shields.io/npm/v/ai-sdk-rate-limiter)](https://www.npmjs.com/package/ai-sdk-rate-limiter)
 [![CI](https://github.com/piyushgupta344/ai-sdk-rate-limiter/actions/workflows/ci.yml/badge.svg)](https://github.com/piyushgupta344/ai-sdk-rate-limiter/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/piyushgupta344/ai-sdk-rate-limiter/graph/badge.svg)](https://codecov.io/gh/piyushgupta344/ai-sdk-rate-limiter)
 [![npm downloads](https://img.shields.io/npm/dm/ai-sdk-rate-limiter)](https://www.npmjs.com/package/ai-sdk-rate-limiter)
+[![docs](https://img.shields.io/badge/docs-piyushgupta344.github.io-blue)](https://piyushgupta344.github.io/ai-sdk-rate-limiter/)
 
 ```
 npm install ai-sdk-rate-limiter
 ```
 
+**[Documentation →](https://piyushgupta344.github.io/ai-sdk-rate-limiter/)**
+
 ---
 
 ## The problem

package/dist/cli.js

@@ -1021,7 +1021,7 @@ async function main() {
 Run with --help for usage.`);
     process.exit(1);
   }
-  await runAudit({ provider, json });
+  await runAudit({ ...provider !== void 0 && { provider }, json });
 }
 main().catch((err) => {
   console.error(err instanceof Error ? err.message : String(err));

package/dist/index.cjs

@@ -271,6 +271,13 @@ var InMemoryStore = class {
     }
     return now + WINDOW_MS + 1;
   }
+  /** Clear all state — useful in tests to reset between runs. */
+  reset() {
+    this.windows.clear();
+    this.dailyWindows.clear();
+    this.backoffs.clear();
+    this.pruneCounter = 0;
+  }
   /**
    * Periodically sweep stale entries from the scope/model maps.
    * Runs every 200 checkAndRecord calls to avoid per-request overhead.
@@ -360,7 +367,18 @@ var RateLimitEngine = class {
     }
     if (nextSlotAtMs > Date.now()) {
       if (local.waiters.length >= this.maxQueueSize) {
-        throw new QueueFullError(key, this.maxQueueSize);
+        if (opts.onFull === "drop-low" && opts.priority !== "low") {
+          const lastIdx = local.waiters.length - 1;
+          const victim = local.waiters[lastIdx];
+          if (victim !== void 0 && victim.priority === "low") {
+            local.waiters.splice(lastIdx, 1);
+            victim.reject(new QueueFullError(key, this.maxQueueSize));
+          } else {
+            throw new QueueFullError(key, this.maxQueueSize);
+          }
+        } else {
+          throw new QueueFullError(key, this.maxQueueSize);
+        }
       }
       const estimatedWaitMs = Math.max(0, nextSlotAtMs - Date.now());
       opts.onQueued?.(local.waiters.length, estimatedWaitMs);
@@ -481,6 +499,18 @@ var RateLimitEngine = class {
     }
     return total;
   }
+  /**
+   * Clear all rate-limit state — sliding windows, queues, backoffs.
+   * Queued requests are rejected with ShutdownError before clearing.
+   * Useful in tests to reset between runs without recreating the instance.
+   */
+  reset() {
+    this.shutdown();
+    this.localStates.clear();
+    if (this.store instanceof InMemoryStore) {
+      this.store.reset();
+    }
+  }
   /**
    * Immediately reject all queued and concurrency-waiting requests with
    * ShutdownError. Called by Pipeline.shutdown() before draining.
@@ -724,6 +754,10 @@ var CostTracker = class {
     }
     return { hour, day, month, byModel, byScope };
   }
+  /** Clear all in-memory cost entries. Useful in tests to reset between runs. */
+  reset() {
+    this.entries = [];
+  }
   // -------------------------------------------------------------------------
   // Private helpers
   // -------------------------------------------------------------------------
@@ -1720,6 +1754,8 @@ var DebugLogger = class {
 };
 
 // src/core/pipeline.ts
+var WARN_PREFIX = "\x1B[33m\u26A0 ai-sdk-rate-limiter\x1B[0m";
+var WARN_RESET = "\x1B[0m";
 function resolveRetryConfig(config) {
   const r = config.retry ?? {};
   return {
@@ -1760,8 +1796,12 @@ var Pipeline = class {
     this.detectedLimits = /* @__PURE__ */ new Map();
     /** In-flight promises indexed by dedup key, shared across identical concurrent requests */
     this.dedupMap = /* @__PURE__ */ new Map();
+    /** Counter used to trigger periodic keyMeta GC without a setInterval */
+    this.executeCount = 0;
     /** Set to true after shutdown() is called */
     this.shutdownRequested = false;
+    /** Models that have already received an "unknown model" warning (dedup) */
+    this.warnedModels = /* @__PURE__ */ new Set();
     this.config = config;
     this.log = new DebugLogger(config.debug === true);
     this.engine = new RateLimitEngine({
@@ -1907,6 +1947,7 @@ var Pipeline = class {
         priority: opts.priority,
         timeoutMs: opts.timeoutMs,
         ...opts.signal !== void 0 && { signal: opts.signal },
+        ...this.config.queue?.onFull !== void 0 && { onFull: this.config.queue.onFull },
         onQueued: (queueDepth, estimatedWaitMs) => {
           this.log.log(modelId, "queuing", { queueDepth, estimatedWaitMs, priority: opts.priority });
           this.emitter.emit("queued", { model: modelId, provider, priority: opts.priority, queueDepth, estimatedWaitMs });
@@ -1917,6 +1958,7 @@ var Pipeline = class {
           this.emitter.emit("dequeued", { model: modelId, provider, waitedMs, priority: opts.priority });
         }
       });
+      if (++this.executeCount % 1e3 === 0) this.pruneKeyMeta();
     } catch (acquireErr) {
       if (acquireErr instanceof QueueFullError) {
         const maxSize = this.config.queue?.maxSize;
@@ -2046,6 +2088,7 @@ var Pipeline = class {
       const snapshot = this.engine.windowSnapshot(key);
       const queueDepth = this.engine.queueDepth(key);
       const backoffUntil = this.engine.backoffUntil(key);
+      if (snapshot.requests === 0 && queueDepth === 0 && backoffUntil === null) continue;
       totalQueueDepth += queueDepth;
       models.push({
         modelId,
@@ -2099,6 +2142,22 @@ var Pipeline = class {
       await new Promise((resolve) => setTimeout(resolve, 50));
     }
   }
+  /**
+   * Clear all rate-limit, queue, cost, and circuit-breaker state.
+   * Any currently queued requests are rejected with ShutdownError.
+   * Useful in tests to reset between runs without recreating the limiter.
+   */
+  reset() {
+    this.engine.reset();
+    this.costTracker.reset();
+    this.keyMeta.clear();
+    this.circuits.clear();
+    this.detectedLimits.clear();
+    this.dedupMap.clear();
+    this.warnedModels.clear();
+    this.executeCount = 0;
+    this.shutdownRequested = false;
+  }
   /** Pre-load historical cost data from the persistent cost store. */
   async warmUp() {
     if (this.config.cost?.store) {
@@ -2134,6 +2193,21 @@ var Pipeline = class {
   // -------------------------------------------------------------------------
   // Private helpers
   // -------------------------------------------------------------------------
+  /**
+   * Remove keyMeta entries for keys that have no recent activity.
+   * A key is safe to evict when its sliding window is empty (all entries
+   * older than 60s) and it has no queued or in-flight requests.
+   * Called every 1000 executions — no setInterval, no GC interference.
+   */
+  pruneKeyMeta() {
+    for (const key of this.keyMeta.keys()) {
+      const snapshot = this.engine.windowSnapshot(key);
+      const queueDepth = this.engine.queueDepth(key);
+      if (snapshot.requests === 0 && queueDepth === 0) {
+        this.keyMeta.delete(key);
+      }
+    }
+  }
   getOrCreateCircuit(key) {
     let cb = this.circuits.get(key);
     if (!cb) {
@@ -2145,13 +2219,24 @@ var Pipeline = class {
   resolveModelLimits(modelId, provider) {
     const base = resolveModelLimits(modelId, provider, this.config.limits ?? {});
     const detected = this.detectedLimits.get(`${provider}:${modelId}`);
-    if (!detected) return base;
-    const userOverride = this.config.limits?.[modelId] ?? {};
-    return {
-      ...base,
-      ...!("rpm" in userOverride) && detected.rpm !== void 0 && { rpm: detected.rpm },
-      ...!("itpm" in userOverride) && detected.itpm !== void 0 && { itpm: detected.itpm }
-    };
+    const limits = detected ? (() => {
+      const userOverride = this.config.limits?.[modelId] ?? {};
+      return {
+        ...base,
+        ...!("rpm" in userOverride) && detected.rpm !== void 0 && { rpm: detected.rpm },
+        ...!("itpm" in userOverride) && detected.itpm !== void 0 && { itpm: detected.itpm }
+      };
+    })() : base;
+    const warnKey = `${provider}:${modelId}`;
+    if (!this.warnedModels.has(warnKey) && !isKnownModel(modelId, provider) && limits.inputPricePerMillion === 0 && limits.outputPricePerMillion === 0 && !(this.config.limits?.[modelId]?.inputPricePerMillion !== void 0 || this.config.limits?.[modelId]?.outputPricePerMillion !== void 0)) {
+      this.warnedModels.add(warnKey);
+      console.warn(
+        `${WARN_PREFIX}: Unknown model '${modelId}' (provider: '${provider}'). Using fallback limits (${limits.rpm} RPM). Cost tracking is disabled.
+  Add pricing to config.limits to enable it:
+  limits: { '${modelId}': { inputPricePerMillion: <n>, outputPricePerMillion: <n> } }` + WARN_RESET
+      );
+    }
+    return limits;
   }
 };
 
@@ -2576,6 +2661,9 @@ function createRateLimiter(config = {}) {
     },
     warmUp() {
       return pipeline.warmUp();
+    },
+    reset() {
+      pipeline.reset();
     }
   };
 }