next-semantic-cache 1.0.0 → 1.0.1

package/README.md

@@ -1,303 +1,188 @@
-# next-semantic-cache
-
-### Drop-in local semantic caching for the Vercel AI SDK.
-
-**Cut your OpenAI / Anthropic bill. Slash latency to ~50 ms. Compute embeddings for free — locally, on every request.**
-
-[![npm version](https://img.shields.io/npm/v/next-semantic-cache?color=6366f1&style=flat-square)](https://www.npmjs.com/package/next-semantic-cache)
-[![license](https://img.shields.io/npm/l/next-semantic-cache?color=22c55e&style=flat-square)](./LICENSE)
-[![bundle size](https://img.shields.io/bundlephobia/minzip/next-semantic-cache?color=f59e0b&style=flat-square)](https://bundlephobia.com/package/next-semantic-cache)
-
----
-
-## Why next-semantic-cache?
-
-Every time a user asks *"What is the return policy?"* your app pays for a full GPT-4o call — even when you answered the same question 200 times yesterday. That is wasted money and wasted latency.
-
-`next-semantic-cache` sits **between your app and the LLM** as a Vercel AI SDK middleware. It:
-
-1. **Embeds** the incoming prompt using `all-MiniLM-L6-v2` — a 22 MB model that runs locally inside your serverless function. **Zero embedding API cost.**
-2. **Searches** your vector store for a semantically equivalent previous prompt (cosine similarity ≥ threshold).
-3. **Returns** the cached response in **~50 ms** if a match is found — no LLM call, no token spend.
-4. **Saves** every new LLM response asynchronously for future cache hits — without adding latency to the current request.
-
-The result: **identical answers are served instantly, and you only pay the LLM for genuinely novel questions.**
-
----
-
-## How it works
-
-```
-Incoming prompt
-      │
-      ▼
-┌─────────────────────────────┐
-│  transformParams()          │  ← embed prompt locally (all-MiniLM-L6-v2)
-│  vector search in Redis     │  ← cosine similarity ≥ threshold?
-└────────────┬────────────────┘
-             │
-      ┌──────┴──────┐
-      │ HIT (~50ms) │  NO LLM call. Return cached string immediately.
-      └─────────────┘
-             │
-      ┌──────┴──────┐
-      │ MISS        │  Call real LLM → stream response to user
-      └──────┬──────┘
-             │
-    ┌─────────────────┐
-    │ fire-and-forget │  save(vector, response) → Redis, with optional TTL
-    └─────────────────┘
-```
-
-> **Key insight:** Embeddings are computed with `@huggingface/transformers` using quantised ONNX weights loaded **once per cold start** as a singleton. Warm requests pay ~2–5 ms for the embedding step — everything else is a Redis RTT.
-
----
-
-## Installation
-
-```bash
-npm install next-semantic-cache @upstash/redis uuid ai
-npm install -D @types/uuid
-```
-
-Set your Upstash credentials in `.env.local`:
-
-```bash
-UPSTASH_REDIS_REST_URL=https://your-db.upstash.io
-UPSTASH_REDIS_REST_TOKEN=your-token-here
-```
-
-> **Upstash requirement:** Your Redis database must have **RediSearch / Redis Stack** enabled. All new Upstash databases have this by default. For self-hosted Redis, use `redis-stack` or load the `redisearch` module.
-
----
-
-## Quick start
-
-```ts
-import { withSemanticCache, RedisVectorAdapter } from "next-semantic-cache";
-import { openai } from "@ai-sdk/openai";
-
-const adapter = new RedisVectorAdapter({
-  redisUrl:   process.env.UPSTASH_REDIS_REST_URL!,
-  redisToken: process.env.UPSTASH_REDIS_REST_TOKEN!,
-  ttlSeconds: 86_400, // cache entries expire after 24 hours
-});
-
-// Wrap your model once. Use it everywhere.
-export const cachedModel = withSemanticCache(openai("gpt-4o"), {
-  adapter,
-  threshold: 0.92, // cosine similarity — tune to taste
-});
-```
-
-That's it. Pass `cachedModel` to `streamText`, `generateText`, or `generateObject` exactly as you would with the unwrapped model.
-
----
-
-## Next.js Edge function example
-
-```ts
-// app/api/chat/route.ts
-import { withSemanticCache, RedisVectorAdapter } from "next-semantic-cache";
-import { openai }     from "@ai-sdk/openai";
-import { streamText } from "ai";
-
-export const runtime = "edge"; // ← works in Edge runtime
-
-// ── Build adapter & cached model outside the handler ─────────────────────
-// On warm lambdas this is reused across requests. The embedding model
-// singleton is loaded once and stays in memory between invocations.
-const adapter = new RedisVectorAdapter({
-  redisUrl:            process.env.UPSTASH_REDIS_REST_URL!,
-  redisToken:          process.env.UPSTASH_REDIS_REST_TOKEN!,
-  ttlSeconds:          60 * 60 * 24 * 7, // 7-day TTL — FAQ answers stay fresh
-  connectionTimeoutMs: 1_500,            // fail fast; fall back to LLM
-});
-
-const cachedModel = withSemanticCache(openai("gpt-4o"), {
-  adapter,
-  threshold: 0.92,
-  debug:     process.env.NODE_ENV === "development",
-});
-
-// ── Edge handler ──────────────────────────────────────────────────────────
-export async function POST(req: Request) {
-  const { messages } = await req.json();
-
-  // Drop-in replacement — the API is identical to the unwrapped model.
-  const result = streamText({
-    model:    cachedModel,
-    messages,
-    system:   "You are a helpful customer support assistant.",
-  });
-
-  return result.toDataStreamResponse();
-}
-```
-
-A cache hit response includes `providerMetadata.semanticCache.hit = true` so you can log it or expose it in your analytics:
-
-```ts
-const result = await generateText({ model: cachedModel, prompt });
-
-if (result.experimental_providerMetadata?.semanticCache?.hit) {
-  console.log("Served from cache — 0 tokens billed.");
-}
-```
-
----
-
-## API reference
-
-### `withSemanticCache(model, options)`
-
-The primary entry point. Returns the wrapped model — fully compatible with all Vercel AI SDK functions.
-
-```ts
-function withSemanticCache<M extends LanguageModelV1>(
-  model:   M,
-  options: SemanticCacheOptions
-): LanguageModelV1
-```
-
-| Option | Type | Default | Description |
-|---|---|---|---|
-| `adapter` | `VectorStoreAdapter` | **required** | Storage backend. Use the built-in `RedisVectorAdapter` or bring your own. |
-| `threshold` | `number` | `0.92` | Cosine similarity cutoff in `[0, 1]`. Higher = stricter matching. |
-| `debug` | `boolean` | `false` | Log cache hits/misses to `console.debug`. |
-
----
-
-### `RedisVectorAdapter`
-
-Built-in adapter for Upstash Redis (and any Redis Stack instance).
-
-```ts
-const adapter = new RedisVectorAdapter(options);
-```
-
-| Option | Type | Default | Description |
-|---|---|---|---|
-| `redisUrl` | `string` | — | Upstash REST endpoint URL. |
-| `redisToken` | `string` | — | Upstash REST API token. |
-| `client` | `Redis` | — | Pass an existing `@upstash/redis` client instead of `url` + `token`. |
-| `ttlSeconds` | `number` | `0` (no expiry) | Seconds until a cache entry is automatically deleted by Redis. |
-| `connectionTimeoutMs` | `number` | `2000` | Max ms to wait for Redis before treating the call as failed and falling back to the LLM. |
-
-```ts
-// Flush all cache entries (testing / manual invalidation)
-const deleted = await adapter.flush();
-console.log(`Removed ${deleted} entries from the semantic cache.`);
-```
-
----
-
-### `VectorStoreAdapter` interface
-
-Bring your own storage backend by implementing two methods:
-
-```ts
-export interface VectorStoreAdapter {
-  /**
-   * Return the cached response string if a similar prompt exists,
-   * or null if no match exceeds the threshold.
-   */
-  search(vector: number[], threshold: number): Promise<string | null>;
-
-  /**
-   * Persist the prompt embedding alongside its LLM response.
-   */
-  save(promptVector: number[], response: string): Promise<void>;
-}
-```
-
-#### Example — in-memory adapter (for unit testing)
-
-```ts
-import type { VectorStoreAdapter } from "next-semantic-cache";
-
-class InMemoryAdapter implements VectorStoreAdapter {
-  private store: Array<{ vector: number[]; response: string }> = [];
-
-  private cosineSimilarity(a: number[], b: number[]): number {
-    const dot = a.reduce((sum, v, i) => sum + v * b[i], 0);
-    const magA = Math.sqrt(a.reduce((s, v) => s + v * v, 0));
-    const magB = Math.sqrt(b.reduce((s, v) => s + v * v, 0));
-    return dot / (magA * magB);
-  }
-
-  async search(vector: number[], threshold: number) {
-    for (const entry of this.store) {
-      if (this.cosineSimilarity(vector, entry.vector) >= threshold) {
-        return entry.response;
-      }
-    }
-    return null;
-  }
-
-  async save(promptVector: number[], response: string) {
-    this.store.push({ vector: promptVector, response });
-  }
-}
-```
-
----
-
-## Architecture notes
-
-### Singleton embedding model
-
-`@huggingface/transformers` loads `Xenova/all-MiniLM-L6-v2` (quantised q8 ONNX, ~23 MB) **once per process**. The pipeline instance is held in a module-level variable and reused on every subsequent request. Cold-start cost: ~400–800 ms. Warm embedding cost: ~2–5 ms.
-
-### Middleware interception flow
-
-The middleware implements two hooks from the Vercel AI SDK's `LanguageModelV1Middleware` spec:
-
-- **`transformParams`** — runs synchronously *before* the LLM call. Embeds the prompt and attaches a `__semanticCacheHit` sentinel to `providerMetadata` on a hit.
-- **`wrapGenerate`** — reads the sentinel. If present, returns the cached result and **never calls `doGenerate()`**. On a miss it calls `doGenerate()` and fire-and-forgets the `save()`.
-
-### Failure isolation
-
-If Redis is unreachable or times out:
-- `search()` returns `null` → the middleware calls the real LLM transparently.
-- `save()` logs the error and returns — the LLM response has already been streamed to the user.
-
-**Redis failures are completely invisible to your users.**
-
----
-
-## Threshold tuning guide
-
-| Threshold | Behaviour | Use-case |
-|---|---|---|
-| **0.98** | Near-identical phrasing only | Deduplicate exact retries |
-| **0.92** *(default)* | Catches paraphrases & minor rewording | General-purpose FAQ / support bots |
-| **0.88** | Broad semantic matching | High-traffic apps where some imprecision is acceptable |
-| **0.80** | Very aggressive | Only for tightly constrained, single-domain prompts |
-
----
-
-## FAQ
-
-**Does it work in the Vercel Edge Runtime?**
-Yes. `@huggingface/transformers` with ONNX backend runs in Edge (V8 isolate) environments. The `@upstash/redis` client uses the REST API so there are no raw TCP sockets.
-
-**Can I use a different LLM — Anthropic Claude, Google Gemini, etc.?**
-Yes. `withSemanticCache` accepts any `LanguageModelV1`-compatible model. Swap `openai(...)` for `anthropic(...)`, `google(...)`, or any community provider.
-
-**Does it support streaming?**
-Cache misses stream normally via the underlying model. Cache hits return a non-streaming synthetic result — the text is complete and the SDK delivers it as a resolved promise. To surface hits as streams, extend `buildCachedGenerateResult` to use `wrapStream` instead.
-
-**Does it work with `generateObject` / structured output?**
-The middleware hooks into `wrapGenerate`, which also underlies `generateObject`. Cached responses are raw text; if your object-generation prompt is highly templated and structurally identical across calls, hits will work correctly. For variable schema inputs, raise `threshold` to ≥ 0.97.
-
-**Will it slow down my first request on a cold start?**
-The embedding model loads in ~400–800 ms on the first invocation. Vercel pre-warms functions in production, reducing the impact. You can also explicitly warm the pipeline by calling `getEmbeddingPipeline()` from a `warmup` route.
-
----
-
-## License
-
-MIT © 2026 — contributions welcome.
-
-> **Install deps:** `npm i next-semantic-cache @upstash/redis uuid ai`
+# next-semantic-cache
+
+> Semantic caching middleware for the [Vercel AI SDK](https://sdk.vercel.ai) — short-circuit LLM calls with vector similarity lookups.
+
+[![npm version](https://img.shields.io/npm/v/next-semantic-cache.svg)](https://www.npmjs.com/package/next-semantic-cache)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+
+`next-semantic-cache` is a drop-in `LanguageModelV4Middleware` that intercepts prompts, embeds them locally, and serves cached responses when a semantically similar prompt has been seen before. It ships with a pluggable `VectorStoreAdapter` interface and a production-ready Redis (RediSearch / Upstash) implementation.
+
+---
+
+## Why?
+
+LLM calls are **slow** and **expensive**. Traditional caching only matches *exact* strings — but users phrase the same question in countless ways. `next-semantic-cache` matches on **meaning**, not exact text:
+
+| Prompt A | Prompt B | Exact cache | Semantic cache |
+| --- | --- | --- | --- |
+| "How do I reset my password?" | "What's the steps to change my password?" | ❌ Miss | ✅ Hit |
+
+---
+
+## Features
+
+- 🧠 **Semantic matching** — embeds prompts with `all-MiniLM-L6-v2` and matches on cosine similarity.
+- 🔌 **Pluggable storage** — implement `VectorStoreAdapter` for any vector DB (Pinecone, pgvector, Weaviate, in-memory, …).
+- 🚀 **Redis adapter included** — works with **Upstash Redis** (serverless) or self-hosted **Redis Stack**.
+- 🛡️ **Defensive by design** — every cache operation is wrapped in try/catch. If the store is down, you transparently fall back to the live LLM. **A cache failure never breaks your app.**
+- ⏱️ **Configurable TTL** — auto-expire entries via Redis `EXPIRE`.
+- 📦 **Zero-config embeddings** — the embedding model runs locally; no extra API keys required.
+
+---
+
+## Installation
+
+```bash
+npm install next-semantic-cache ai @huggingface/transformers
+```
+
+For the Redis adapter, also install:
+
+```bash
+npm install @upstash/redis uuid
+npm install -D @types/uuid
+```
+
+---
+
+## Quick Start
+
+```ts
+import { openai } from "@ai-sdk/openai";
+import { wrapLanguageModel, generateText } from "ai";
+import { SemanticCacheMiddleware } from "next-semantic-cache";
+import { RedisVectorAdapter } from "next-semantic-cache/redis-vector-adapter";
+
+// 1. Create a vector store adapter.
+const vectorStore = new RedisVectorAdapter({
+  redisUrl:   process.env.UPSTASH_REDIS_REST_URL!,
+  redisToken: process.env.UPSTASH_REDIS_REST_TOKEN!,
+  ttlSeconds: 60 * 60 * 24, // 24-hour cache expiry
+});
+
+// 2. Wrap your model with the middleware.
+const model = wrapLanguageModel({
+  model: openai("gpt-4o"),
+  middleware: SemanticCacheMiddleware({
+    vectorStore,
+    similarityThreshold: 0.92, // 0–1; higher = stricter match
+  }),
+});
+
+// 3. Use it exactly like any AI SDK model.
+const { text } = await generateText({
+  model,
+  prompt: "Explain quantum entanglement in one sentence.",
+});
+
+console.log(text);
+// First call  → hits the LLM, caches the result.
+// Later calls with a similar prompt → served instantly from cache.
+```
+
+---
+
+## Configuration
+
+### `SemanticCacheMiddleware(options)`
+
+| Option | Type | Default | Description |
+| --- | --- | --- | --- |
+| `vectorStore` | `VectorStoreAdapter` | **required** | The persistence backend for embeddings and responses. |
+| `similarityThreshold` | `number` | `0.92` | Minimum cosine similarity (0–1) for a cache hit. Higher = fewer, more accurate hits. |
+
+### `RedisVectorAdapter(options)`
+
+| Option | Type | Default | Description |
+| --- | --- | --- | --- |
+| `client` | `Redis` | — | A pre-configured `@upstash/redis` client. Mutually exclusive with `redisUrl` + `redisToken`. |
+| `redisUrl` | `string` | — | Upstash REST endpoint. Required if `client` is omitted. |
+| `redisToken` | `string` | — | Upstash REST token. Required if `client` is omitted. |
+| `ttlSeconds` | `number` | `0` | Auto-expire entries after N seconds. `0` = keep indefinitely. |
+| `connectionTimeoutMs` | `number` | `2000` | Timeout per Redis command before falling back to the LLM. |
+
+---
+
+## Writing a Custom Adapter
+
+Implement the `VectorStoreAdapter` interface to plug in any backend:
+
+```ts
+import type { VectorStoreAdapter } from "next-semantic-cache";
+
+export class MyVectorAdapter implements VectorStoreAdapter {
+  /**
+   * Return the cached response if a stored vector is within `threshold`
+   * cosine similarity of `vector`, otherwise `null`.
+   */
+  async search(vector: number[], threshold: number): Promise<string | null> {
+    // ... query your vector DB ...
+    return null;
+  }
+
+  /**
+   * Persist a prompt embedding and its LLM response.
+   * Failures should be swallowed — caching is best-effort.
+   */
+  async save(promptVector: number[], response: string): Promise<void> {
+    // ... upsert into your vector DB ...
+  }
+}
+```
+
+---
+
+## How It Works
+
+```
+                      ┌──────────────────────────────┐
+   prompt ──────────► │  transformParams             │
+                      │  • embed prompt (MiniLM-L6)   │
+                      │  • vectorStore.search(...)    │
+                      └───────────┬──────────────────┘
+                                  │
+                 ┌────────────────┴─────────────────┐
+                 ▼                                    ▼
+          Cache HIT                             Cache MISS
+   ┌──────────────────────┐            ┌──────────────────────────┐
+   │ return synthetic      │            │ wrapGenerate:            │
+   │ result — LLM is       │            │ • call real model        │
+   │ NEVER invoked         │            │ • vectorStore.save(...)  │
+   └──────────────────────┘            │   (fire-and-forget)      │
+                                        └──────────────────────────┘
+```
+
+1. **`transformParams`** embeds the incoming prompt and queries the vector store.
+2. **Cache hit** → a synthetic `LanguageModelV4GenerateResult` is returned immediately; the real model is short-circuited.
+3. **Cache miss** → the request falls through to the real model, and the response is saved to the store asynchronously (never blocking the response path).
+
+---
+
+## Redis Storage Layout
+
+The `RedisVectorAdapter` stores each entry as a Redis `HASH` and creates an HNSW vector index over them:
+
+| Component | Value |
+| --- | --- |
+| Key prefix | `sc:<uuid>` |
+| Fields | `response`, `vector` (Float32 binary), `createdAt` (ISO-8601) |
+| Index | `FT.CREATE` — HNSW, `FLOAT32`, `DIM 384`, `COSINE` distance |
+
+The index is created **lazily** on first `save()`/`search()` and is idempotent.
+
+> ⚠️ **Note on Upstash + binary vectors:** `@upstash/redis` transports commands as JSON over HTTP. Binary vector payloads may not round-trip reliably through the REST layer. If you encounter KNN query errors, use a **Redis Stack** instance with a native `ioredis` client for full binary support.
+
+---
+
+## Requirements
+
+- **Node.js** ≥ 18
+- **Vercel AI SDK** (`ai`) v4+
+- A vector store (Redis Stack / Upstash, or your own adapter)
+- Embedding model: `all-MiniLM-L6-v2` (384-dim) via `@huggingface/transformers`
+
+---
+
+## License
+
+MIT © [Prateek Chaturvedi](https://github.com/chaturvedi-prateek)

package/dist/redis-vector-adapter.d.ts

@@ -0,0 +1,135 @@
+/**
+ * redis-vector-adapter.ts
+ *
+ * Concrete implementation of VectorStoreAdapter using Upstash Redis
+ * (compatible with any Redis instance that has the RediSearch / Redis Stack
+ * module enabled — the same FT.* command surface is used in both cases).
+ *
+ * Storage layout
+ * ─────────────
+ *  Key prefix : sc:<uuid>          (Redis HASH)
+ *  Fields     : response  — raw LLM response text
+ *               vector    — Float32 embedding encoded as a binary string
+ *               createdAt — ISO-8601 timestamp for observability
+ *
+ * Index       : FT.CREATE on the HASH prefix using an HNSW VECTOR field
+ *               with COSINE distance.  Created lazily on first save/search.
+ *
+ * Failure policy (defensive-first)
+ * ──────────────────────────────────
+ *  Every Redis operation is wrapped in try/catch.  If Redis is unreachable,
+ *  times out, or returns an unexpected payload:
+ *    • search() returns null  → middleware falls back to the real LLM.
+ *    • save()   is a no-op    → the miss is simply not cached; no crash.
+ *
+ * Dependencies:
+ *   npm i @upstash/redis uuid
+ *   npm i -D @types/node @types/uuid
+ */
+import { Redis } from "@upstash/redis";
+import type { VectorStoreAdapter } from "./semantic-cache-middleware";
+export interface RedisVectorAdapterOptions {
+    /**
+     * Pre-configured @upstash/redis client instance.
+     * Pass this if you want to share a client across your application.
+     * Mutually exclusive with `redisUrl` + `redisToken`.
+     */
+    client?: Redis;
+    /**
+     * Upstash Redis REST endpoint URL.
+     * Required when `client` is not provided.
+     * Example: "https://your-db.upstash.io"
+     */
+    redisUrl?: string;
+    /**
+     * Upstash Redis REST API token.
+     * Required when `client` is not provided.
+     */
+    redisToken?: string;
+    /**
+     * Optional TTL in seconds.  When set, each cache entry will automatically
+     * expire after this many seconds via Redis EXPIRE.
+     * Omit (or set to 0) to keep entries indefinitely.
+     */
+    ttlSeconds?: number;
+    /**
+     * Maximum number of milliseconds to wait for a Redis command before
+     * treating the operation as a failure and falling back to the LLM.
+     * @default 2000
+     */
+    connectionTimeoutMs?: number;
+}
+/**
+ * A {@link VectorStoreAdapter} backed by Redis with the RediSearch (Redis
+ * Stack) vector similarity extension.
+ *
+ * Compatible with:
+ *  • Upstash Redis (recommended — serverless, REST-based)
+ *  • Redis Stack (self-hosted or Redis Cloud with Search enabled)
+ *
+ * @example
+ * ```ts
+ * const adapter = new RedisVectorAdapter({
+ *   redisUrl:   process.env.UPSTASH_REDIS_REST_URL!,
+ *   redisToken: process.env.UPSTASH_REDIS_REST_TOKEN!,
+ *   ttlSeconds: 60 * 60 * 24, // 24-hour cache expiry
+ * });
+ *
+ * const model = wrapLanguageModel({
+ *   model: openai("gpt-4o"),
+ *   middleware: SemanticCacheMiddleware({
+ *     vectorStore: adapter,
+ *     similarityThreshold: 0.92,
+ *   }),
+ * });
+ * ```
+ */
+export declare class RedisVectorAdapter implements VectorStoreAdapter {
+    private readonly redis;
+    private readonly ttlSeconds;
+    private readonly timeoutMs;
+    /**
+     * Whether the RediSearch HNSW index has been confirmed to exist this
+     * process lifetime.  Avoids issuing FT.CREATE on every request.
+     */
+    private indexReady;
+    constructor(options: RedisVectorAdapterOptions);
+    /**
+     * Lazily creates the RediSearch HNSW index on first use.
+     * The operation is idempotent — if the index already exists Redis returns
+     * an error that we swallow silently.
+     */
+    private ensureIndex;
+    /**
+     * Performs a KNN-1 cosine similarity search against the HNSW index.
+     *
+     * Cosine *distance* (returned by Redis) = 1 − cosine *similarity*.
+     * Therefore the `threshold` parameter (expressed as similarity, e.g. 0.92)
+     * is converted: `maxDistance = 1 − threshold` (e.g. 0.08).
+     *
+     * If Redis is unavailable or returns an error, `null` is returned and the
+     * middleware gracefully falls through to a live LLM call.
+     *
+     * @param vector    - The float32 embedding of the incoming prompt.
+     * @param threshold - Minimum cosine similarity score in [0, 1].
+     * @returns The cached LLM response string, or `null` on a miss / error.
+     */
+    search(vector: number[], threshold: number): Promise<string | null>;
+    /**
+     * Stores a prompt embedding and its LLM response as a Redis HASH.
+     *
+     * Failures are logged but never re-thrown — caching is best-effort.
+     *
+     * @param promptVector - The float32 embedding of the prompt.
+     * @param response     - The LLM-generated text to cache.
+     */
+    save(promptVector: number[], response: string): Promise<void>;
+    /**
+     * Deletes ALL semantic cache entries from Redis.
+     * Useful for testing or manual cache invalidation.
+     *
+     *     This scans the keyspace — avoid on large production databases.
+     *     Prefer TTL-based expiry for automatic eviction in production.
+     */
+    flush(): Promise<number>;
+}

package/dist/redis-vector-adapter.js

@@ -0,0 +1,323 @@
+"use strict";
+/**
+ * redis-vector-adapter.ts
+ *
+ * Concrete implementation of VectorStoreAdapter using Upstash Redis
+ * (compatible with any Redis instance that has the RediSearch / Redis Stack
+ * module enabled — the same FT.* command surface is used in both cases).
+ *
+ * Storage layout
+ * ─────────────
+ *  Key prefix : sc:<uuid>          (Redis HASH)
+ *  Fields     : response  — raw LLM response text
+ *               vector    — Float32 embedding encoded as a binary string
+ *               createdAt — ISO-8601 timestamp for observability
+ *
+ * Index       : FT.CREATE on the HASH prefix using an HNSW VECTOR field
+ *               with COSINE distance.  Created lazily on first save/search.
+ *
+ * Failure policy (defensive-first)
+ * ──────────────────────────────────
+ *  Every Redis operation is wrapped in try/catch.  If Redis is unreachable,
+ *  times out, or returns an unexpected payload:
+ *    • search() returns null  → middleware falls back to the real LLM.
+ *    • save()   is a no-op    → the miss is simply not cached; no crash.
+ *
+ * Dependencies:
+ *   npm i @upstash/redis uuid
+ *   npm i -D @types/node @types/uuid
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.RedisVectorAdapter = void 0;
+const redis_1 = require("@upstash/redis");
+const uuid_1 = require("uuid");
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+/** Prefix for every cache entry stored as a Redis HASH. */
+const KEY_PREFIX = "sc:";
+/** Name of the RediSearch index created over the cache entries. */
+const INDEX_NAME = "semantic_cache_idx";
+/**
+ * Dimensionality of the all-MiniLM-L6-v2 embedding space.
+ * Must match the model used in the middleware (384 for MiniLM-L6).
+ */
+const VECTOR_DIM = 384;
+/**
+ * Number of HNSW graph neighbours to explore at query time.
+ * Higher = more accurate recall, higher latency.  32 is a safe default.
+ */
+const HNSW_EF_RUNTIME = 32;
+// ---------------------------------------------------------------------------
+// Helpers — raw command dispatch
+// ---------------------------------------------------------------------------
+/**
+ * Dispatches a raw Redis command through the @upstash/redis client.
+ *
+ * The @upstash/redis SDK does not expose a public `sendCommand` /
+ * `executeCommand` method for RediSearch (FT.*) or other low-level commands.
+ * Internally every client forwards a command array via its private request
+ * pipeline. We funnel ALL raw commands through this single helper so the
+ * cast to the internal shape is isolated to one place rather than scattered
+ * across the class.
+ *
+ * @param redis - The @upstash/redis client instance.
+ * @param args  - The command name followed by its arguments, as a flat array.
+ * @returns The decoded command result, typed by the caller via `T`.
+ */
+async function rawCommand(redis, args) {
+    // The internal `request` method accepts `{ command: [...] }` and returns
+    // `{ result }`. It is not part of the public type surface, so we cast here
+    // deliberately and keep the cast confined to this helper.
+    const client = redis;
+    const { result } = await client.request({ command: args });
+    return result;
+}
+// ---------------------------------------------------------------------------
+// Helpers — vector encoding / decoding
+// ---------------------------------------------------------------------------
+/**
+ * Encodes a JS number[] into a Buffer of raw Float32 bytes.
+ * Redis vector fields require the embedding as a binary blob.
+ */
+function encodeVector(vector) {
+    const float32 = new Float32Array(vector);
+    return Buffer.from(float32.buffer);
+}
+/**
+ * Decodes a Base64 string back into a Buffer containing raw Float32 bytes.
+ * Used when converting the stored value back for a KNN query blob.
+ */
+function base64ToBuffer(b64) {
+    return Buffer.from(b64, "base64");
+}
+/**
+ * Wraps a promise with a timeout.
+ * Rejects with a timeout error if the promise does not settle within
+ * `ms` milliseconds.
+ */
+function withTimeout(promise, ms) {
+    return new Promise((resolve, reject) => {
+        const timer = setTimeout(() => reject(new Error(`Redis operation timed out after ${ms}ms`)), ms);
+        promise.then((value) => { clearTimeout(timer); resolve(value); }, (err) => { clearTimeout(timer); reject(err); });
+    });
+}
+/**
+ * A {@link VectorStoreAdapter} backed by Redis with the RediSearch (Redis
+ * Stack) vector similarity extension.
+ *
+ * Compatible with:
+ *  • Upstash Redis (recommended — serverless, REST-based)
+ *  • Redis Stack (self-hosted or Redis Cloud with Search enabled)
+ *
+ * @example
+ * ```ts
+ * const adapter = new RedisVectorAdapter({
+ *   redisUrl:   process.env.UPSTASH_REDIS_REST_URL!,
+ *   redisToken: process.env.UPSTASH_REDIS_REST_TOKEN!,
+ *   ttlSeconds: 60 * 60 * 24, // 24-hour cache expiry
+ * });
+ *
+ * const model = wrapLanguageModel({
+ *   model: openai("gpt-4o"),
+ *   middleware: SemanticCacheMiddleware({
+ *     vectorStore: adapter,
+ *     similarityThreshold: 0.92,
+ *   }),
+ * });
+ * ```
+ */
+class RedisVectorAdapter {
+    redis;
+    ttlSeconds;
+    timeoutMs;
+    /**
+     * Whether the RediSearch HNSW index has been confirmed to exist this
+     * process lifetime.  Avoids issuing FT.CREATE on every request.
+     */
+    indexReady = false;
+    constructor(options) {
+        const { client, redisUrl, redisToken, ttlSeconds = 0, connectionTimeoutMs = 2000 } = options;
+        if (client) {
+            this.redis = client;
+        }
+        else if (redisUrl && redisToken) {
+            this.redis = new redis_1.Redis({
+                url: redisUrl,
+                token: redisToken,
+            });
+        }
+        else {
+            throw new Error("RedisVectorAdapter: supply either `client` or both `redisUrl` and `redisToken`.");
+        }
+        this.ttlSeconds = ttlSeconds;
+        this.timeoutMs = connectionTimeoutMs;
+    }
+    // =========================================================================
+    // Index bootstrap
+    // =========================================================================
+    /**
+     * Lazily creates the RediSearch HNSW index on first use.
+     * The operation is idempotent — if the index already exists Redis returns
+     * an error that we swallow silently.
+     */
+    async ensureIndex() {
+        if (this.indexReady)
+            return;
+        try {
+            await withTimeout(rawCommand(this.redis, [
+                "FT.CREATE", INDEX_NAME,
+                "ON", "HASH",
+                "PREFIX", "1", KEY_PREFIX,
+                "SCHEMA",
+                // ── Full-text field (for human inspection / future hybrid search) ──
+                "response", "TEXT",
+                "createdAt", "TEXT",
+                // ── Vector field — HNSW with Cosine distance ──────────────────────
+                "vector", "VECTOR", "HNSW",
+                "6", // number of attribute/value pairs below
+                "TYPE", "FLOAT32",
+                "DIM", String(VECTOR_DIM),
+                "DISTANCE_METRIC", "COSINE",
+            ]), this.timeoutMs);
+        }
+        catch (err) {
+            // Redis returns "Index already exists" as an error string; swallow it.
+            // Any other error is also swallowed here — if the index genuinely fails
+            // to create, FT.SEARCH will error too and be caught in search().
+            const message = err instanceof Error ? err.message : String(err);
+            if (!message.includes("already exists") && !message.includes("Index already exists")) {
+                console.warn("[RedisVectorAdapter] FT.CREATE warning (non-fatal):", message);
+            }
+        }
+        // Mark as ready regardless — subsequent failures will be caught per-call.
+        this.indexReady = true;
+    }
+    // =========================================================================
+    // VectorStoreAdapter — search
+    // =========================================================================
+    /**
+     * Performs a KNN-1 cosine similarity search against the HNSW index.
+     *
+     * Cosine *distance* (returned by Redis) = 1 − cosine *similarity*.
+     * Therefore the `threshold` parameter (expressed as similarity, e.g. 0.92)
+     * is converted: `maxDistance = 1 − threshold` (e.g. 0.08).
+     *
+     * If Redis is unavailable or returns an error, `null` is returned and the
+     * middleware gracefully falls through to a live LLM call.
+     *
+     * @param vector    - The float32 embedding of the incoming prompt.
+     * @param threshold - Minimum cosine similarity score in [0, 1].
+     * @returns The cached LLM response string, or `null` on a miss / error.
+     */
+    async search(vector, threshold) {
+        try {
+            await this.ensureIndex();
+            // Convert similarity threshold → cosine distance upper bound.
+            const maxDistance = 1 - threshold;
+            // Encode the query vector as a binary buffer for the KNN parameter.
+            const queryBlob = encodeVector(vector);
+            // ── FT.SEARCH KNN query ──────────────────────────────────────────────
+            const rawResult = await withTimeout(rawCommand(this.redis, [
+                "FT.SEARCH", INDEX_NAME,
+                `*=>[KNN 1 @vector $vec AS __score]`,
+                "PARAMS", "2",
+                "vec", queryBlob.toString("binary"), // binary string for the REST layer
+                "RETURN", "2", "__score", "response",
+                "SORTBY", "__score", "ASC",
+                "DIALECT", "2",
+            ]), this.timeoutMs);
+            // rawResult layout (RESP2): [ totalCount, key1, [field, value, ...], ... ]
+            const totalCount = rawResult[0];
+            if (totalCount === 0) {
+                return null; // no entries in the index yet
+            }
+            // The first result's field-value array is at index 2.
+            const fields = rawResult[2];
+            // Parse the flat field-value array into a map.
+            const fieldMap = {};
+            for (let i = 0; i < fields.length; i += 2) {
+                fieldMap[fields[i]] = fields[i + 1];
+            }
+            const score = parseFloat(fieldMap["__score"] ?? "1");
+            const response = fieldMap["response"] ?? null;
+            // `score` is cosine distance; reject if it exceeds the max allowed distance.
+            if (score > maxDistance || response === null) {
+                return null;
+            }
+            return response;
+        }
+        catch (err) {
+            // ── Defensive fallback ────────────────────────────────────────────────
+            console.error("[RedisVectorAdapter] search() failed (falling back to LLM):", err instanceof Error ? err.message : err);
+            return null;
+        }
+    }
+    // =========================================================================
+    // VectorStoreAdapter — save
+    // =========================================================================
+    /**
+     * Stores a prompt embedding and its LLM response as a Redis HASH.
+     *
+     * Failures are logged but never re-thrown — caching is best-effort.
+     *
+     * @param promptVector - The float32 embedding of the prompt.
+     * @param response     - The LLM-generated text to cache.
+     */
+    async save(promptVector, response) {
+        try {
+            await this.ensureIndex();
+            const key = `${KEY_PREFIX}${(0, uuid_1.v4)()}`;
+            const vectorBuf = encodeVector(promptVector);
+            // Store the vector as a binary string field so RediSearch can index it.
+            await withTimeout(rawCommand(this.redis, [
+                "HSET", key,
+                "vector", vectorBuf.toString("binary"),
+                "response", response,
+                "createdAt", new Date().toISOString(),
+            ]), this.timeoutMs);
+            // Apply optional TTL — Redis will remove the key after `ttlSeconds`.
+            if (this.ttlSeconds > 0) {
+                await withTimeout(rawCommand(this.redis, ["EXPIRE", key, String(this.ttlSeconds)]), this.timeoutMs);
+            }
+        }
+        catch (err) {
+            // ── Defensive fallback ────────────────────────────────────────────────
+            // A failed cache write must never degrade the primary LLM response
+            // that was already returned to the user.
+            console.error("[RedisVectorAdapter] save() failed (cache write skipped):", err instanceof Error ? err.message : err);
+        }
+    }
+    // =========================================================================
+    // Utility — explicit teardown (optional, for graceful shutdown)
+    // =========================================================================
+    /**
+     * Deletes ALL semantic cache entries from Redis.
+     * Useful for testing or manual cache invalidation.
+     *
+     *     This scans the keyspace — avoid on large production databases.
+     *     Prefer TTL-based expiry for automatic eviction in production.
+     */
+    async flush() {
+        try {
+            let cursor = "0";
+            let deletedCount = 0;
+            do {
+                const [nextCursor, keys] = await withTimeout(rawCommand(this.redis, [
+                    "SCAN", cursor, "MATCH", `${KEY_PREFIX}*`, "COUNT", "100"
+                ]), this.timeoutMs);
+                cursor = nextCursor;
+                if (keys.length > 0) {
+                    await withTimeout(rawCommand(this.redis, ["DEL", ...keys]), this.timeoutMs);
+                    deletedCount += keys.length;
+                }
+            } while (cursor !== "0");
+            return deletedCount;
+        }
+        catch (err) {
+            console.error("[RedisVectorAdapter] flush() failed:", err instanceof Error ? err.message : err);
+            return 0;
+        }
+    }
+}
+exports.RedisVectorAdapter = RedisVectorAdapter;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "next-semantic-cache",
-  "version": "1.0.0",
+  "version": "1.0.1",
   "description": "**Cut your OpenAI / Anthropic bill. Slash latency to ~50 ms. Compute embeddings for free — locally, on every request.**",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
@@ -11,7 +11,10 @@
     "dist/index.js",
     "dist/index.d.ts",
     "dist/semantic-cache-middleware.js",
-    "dist/semantic-cache-middleware.d.ts"
+    "dist/semantic-cache-middleware.d.ts",
+    "dist/redis-vector-adapter.js",
+    "dist/redis-vector-adapter.d.ts",
+    "README.md"
   ],
   "keywords": [],
   "author": "Prateek Chaturvedi",