ai-sdk-rate-limiter 0.3.0 → 0.5.0

package/README.md

@@ -52,7 +52,9 @@ The wrapped model is a drop-in replacement. Every Vercel AI SDK feature — stre
 
 **Cost tracking** — Records actual token usage from every response. Reports hourly, daily, and monthly spend per model. Optionally enforces budget caps.
 
-**Built-in model registry** — Knows the RPM, ITPM, and per-token pricing for every major OpenAI, Anthropic, and Google model out of the box. Nothing to configure to get started.
+**Built-in model registry** — Knows the RPM, ITPM, and per-token pricing for every major OpenAI, Anthropic, Google, Groq, Mistral, and Cohere model out of the box. Nothing to configure to get started.
+
+**Raw SDK support** — Works with the native OpenAI, Anthropic, Groq, Mistral, and Cohere SDKs directly via a transparent JavaScript Proxy. No Vercel AI SDK required.
 
 ---
 
@@ -212,6 +214,123 @@ const result = await generateText({ model, prompt })
 
 ---
 
+## Multi-instance Redis store
+
+By default, rate limit state is in-memory (per-process). In multi-instance deployments — serverless functions, multiple pods, workers — each instance has its own counters. Install the Redis store to share state across all instances:
+
+```
+npm install ioredis
+```
+
+```typescript
+import { createRateLimiter } from 'ai-sdk-rate-limiter'
+import { RedisStore } from 'ai-sdk-rate-limiter/redis'
+import Redis from 'ioredis'
+
+const limiter = createRateLimiter({
+  store: new RedisStore(new Redis(process.env.REDIS_URL)),
+  // ... rest of your config
+})
+```
+
+That's the entire change. All APIs — `wrap()`, `rawProxy()`, events, cost reports — work identically. The Redis store enforces rate limits collectively so no two instances can jointly exceed the API limits.
+
+**How it works:**
+
+Each request atomically runs a Lua script that:
+1. Removes entries older than 60 seconds from a sorted set (`ZREMRANGEBYSCORE`)
+2. Counts remaining requests and sums input tokens
+3. Checks against RPM and ITPM limits
+4. If allowed: reserves the slot (`ZADD`) and returns immediately
+5. If blocked: returns the timestamp when the next slot opens
+
+The local queue (priority ordering, drain timer, timeout handling) stays in-memory per instance — only the window counters are shared.
+
+**Options:**
+
+```typescript
+new RedisStore(redis, {
+  keyPrefix: 'rl:myapp:',  // namespace if multiple apps share Redis
+  windowMs:  60_000,        // window size; match your provider's limit window
+})
+```
+
+**Compatible clients** — any Redis client with `eval()`, `get()`, and `set()` works: `ioredis`, `node-redis`, Upstash Redis.
+
+**Single-instance deployments:** the default `InMemoryStore` is more accurate (true sliding window, no network round-trips) and zero-config. Only switch to `RedisStore` when you actually need cross-instance coordination.
+
+---
+
+## Raw SDK proxy
+
+If you're using the OpenAI, Anthropic, Groq, Mistral, or Cohere SDK directly — without the Vercel AI SDK — use `limiter.rawProxy()` to add rate limiting as a transparent drop-in:
+
+```typescript
+import { createRateLimiter } from 'ai-sdk-rate-limiter'
+import OpenAI from 'openai'
+import Anthropic from '@anthropic-ai/sdk'
+
+const limiter = createRateLimiter({
+  cost: { budget: { daily: 50 }, onExceeded: 'throw' },
+  on: { rateLimited: ({ model }) => console.warn(`${model} rate limited`) },
+})
+
+// Every API call goes through the same rate limiter and cost tracker
+const openai   = limiter.rawProxy(new OpenAI())
+const anthropic = limiter.rawProxy(new Anthropic())
+
+// Use exactly as before — no other changes needed
+const completion = await openai.chat.completions.create({
+  model: 'gpt-4o',
+  messages: [{ role: 'user', content: 'Hello!' }],
+})
+
+const message = await anthropic.messages.create({
+  model: 'claude-opus-4-6',
+  max_tokens: 1024,
+  messages: [{ role: 'user', content: 'Hello!' }],
+})
+
+// Cost from both clients tracked together
+const report = limiter.getCostReport()
+```
+
+**Streaming works too** — the proxy wraps the returned `AsyncIterable` to capture the final usage chunk automatically:
+
+```typescript
+const stream = await openai.chat.completions.create({
+  model: 'gpt-4o',
+  messages: [{ role: 'user', content: 'Stream this' }],
+  stream: true,
+  stream_options: { include_usage: true },  // tells OpenAI to include usage in final chunk
+})
+
+for await (const chunk of stream) {
+  process.stdout.write(chunk.choices[0]?.delta?.content ?? '')
+}
+// After the loop, usage is recorded in limiter.getCostReport()
+```
+
+**Zero-config standalone version** — if you don't need to share the limiter with other models:
+
+```typescript
+import { rateLimited } from 'ai-sdk-rate-limiter'
+
+const openai = rateLimited(new OpenAI(), {
+  config: { cost: { budget: { daily: 20 } } },
+})
+```
+
+**Provider is auto-detected** from the client's constructor name (`OpenAI`, `Anthropic`, `Groq`, etc.). Override it explicitly if needed:
+
+```typescript
+const client = limiter.rawProxy(new OpenAI({ baseURL: 'https://api.groq.com/openai/v1' }), {
+  provider: 'groq',  // use Groq's limits and pricing instead of OpenAI's
+})
+```
+
+---
+
 ## Backpressure — know before you send
 
 Check estimated wait time before committing to a request. Useful for showing loading states or shedding load gracefully.
@@ -449,6 +568,7 @@ console.log(isKnownModel('my-fine-tune', 'openai'))
 | | ai-sdk-rate-limiter | bottleneck | p-limit | SDK built-in retry | LangChain |
 |---|:---:|:---:|:---:|:---:|:---:|
 | Vercel AI SDK `.wrap()` | yes | no | no | — | no |
+| Raw SDK proxy | yes | no | no | — | no |
 | Model-aware limits | yes | no | no | no | partial |
 | ITPM / token tracking | yes | no | no | no | no |
 | Priority queue | yes | yes | no | no | no |