krusch-cascade-router 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 kruschdev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,158 @@
+<p align="center">
+  <img src="docs/assets/banner.png" alt="Krusch Cascade Router" width="800" />
+</p>
+
+<p align="center">
+  <strong>Latency-aware LLM router that dynamically cascades between edge and cloud models via logprob inspection.</strong>
+</p>
+
+<p align="center">
+  <a href="https://www.npmjs.com/package/krusch-cascade-router"><img src="https://img.shields.io/github/package-json/v/kruschdev/krusch-cascade-router.svg?style=flat-square" alt="NPM Version"></a>
+  <a href="https://github.com/kruschdev/krusch-cascade-router/blob/main/LICENSE"><img src="https://img.shields.io/github/license/kruschdev/krusch-cascade-router.svg?style=flat-square" alt="License"></a>
+  <img src="https://img.shields.io/badge/node-%3E%3D18-blue.svg?style=flat-square" alt="Node Version">
+</p>
+
+---
+
+## ⚡ Why Krusch Cascade Router?
+
+**"LLM routing an LLM is a trap."**
+
+Using a massive third LLM to decide which LLM to route a query to adds severe TTFT (Time To First Token) latency and API costs. `krusch-cascade-router` solves this by combining a fast predictive heuristic classifier (<50ms latency) with a reactive logprob-based speculative cascade. Designed specifically for agentic developers building with local AI, it allows you to optimize for cost, performance, and reliability without sacrificing capability.
+
+### Key Features
+- **🚀 Sub-50ms Heuristic Classifier:** Evaluates prompt complexity instantly.
+- **🧠 Logprob Speculative Execution:** Reactively cascades to heavy cloud models if the edge model's confidence drops.
+- **🔌 Framework Agnostic:** Can be plugged into any Node.js AI architecture.
+- **🛡️ Custom Heuristics:** Support for `customRules` to inject your own prompt complexity detection logic.
+- **🛑 Native AbortSignal Support:** Manage request timeouts natively via `ChatOptions`.
+- **📦 Dual CJS/ESM Support:** Works in modern ECMAScript and legacy environments.
+
+---
+
+## 🧠 Architecture: How It Works
+
+1. **Predictive Classifier**: Instantly evaluates the prompt's complexity via string heuristics (length, code blocks, complex cognitive verbs, or your `customRules`). If classified as complex, it routes directly to the heavy cloud model.
+2. **Speculative Cascade**: If classified as simple, it streams the fast local edge model. It buffers and inspects the logprobs of the first N tokens. If the confidence (probability) dips below your configured threshold, it silently aborts the stream and falls back to the heavy cloud model.
+
+```mermaid
+graph TD;
+    A[Incoming Prompt] --> B{Heuristic Classifier};
+    B -- Complex --> C[Heavy Cloud Model];
+    B -- Simple --> D[Local Edge Model];
+    D --> E{Evaluate Logprobs first N tokens};
+    E -- Confidence >= Threshold --> F[Stream Edge Response];
+    E -- Confidence < Threshold --> G[Abort Edge];
+    G --> C;
+```
+
+---
+
+## 📦 Installation
+
+```bash
+npm install krusch-cascade-router
+```
+
+> **Note**: Requires Node.js 18+ for native fetch and `AbortSignal` support.
+
+---
+
+## 🚀 Quick Start Guide
+
+```javascript
+import { CascadeRouter } from 'krusch-cascade-router';
+
+// 1. Initialize the router with your edge and cloud models
+const router = new CascadeRouter({
+  fastModel: { 
+    url: 'http://localhost:11434/v1/chat/completions', 
+    model: 'qwen2.5:3b' // Edge node tag resolution
+  },
+  heavyModel: { 
+    apiKey: process.env.GEMINI_API_KEY, 
+    model: 'gemini-2.5-pro', 
+    provider: 'gemini' 
+  },
+  cascadeThreshold: 0.85, // Abort if average probability of first 5 tokens is < 85%
+  tokensToEvaluate: 5
+});
+
+// 2. Send a chat request
+const response = await router.chat("Write a complex architectural plan...");
+
+// 3. Check where it was routed
+console.log(`Routed to: ${response.routedTo}`);
+console.log(response.text);
+```
+
+---
+
+## 🛠️ Advanced Usage
+
+### Custom Heuristic Rules (`customRules`)
+You can inject your own detection logic to fine-tune what goes directly to the cloud model:
+
+```javascript
+const router = new CascadeRouter({
+  // ...models config
+  customRules: [
+    (prompt) => prompt.includes('PostgreSQL'), // Always route DB questions to cloud
+    (prompt) => prompt.length > 2000 // Override default length heuristics
+  ]
+});
+```
+
+### Timeouts and AbortSignals
+Native integration with `AbortSignal` for graceful timeout handling:
+
+```javascript
+const controller = new AbortController();
+setTimeout(() => controller.abort(), 10000); // 10s timeout
+
+try {
+  const response = await router.chat("Analyze this dataset", {
+    signal: controller.signal
+  });
+} catch (err) {
+  if (err.name === 'AbortError') {
+    console.log('Request was timed out or aborted manually.');
+  }
+}
+```
+
+---
+
+## 📚 API Reference
+
+### `new CascadeRouter(config)`
+
+| Property | Type | Description |
+|---|---|---|
+| `fastModel` | `ModelConfig` | Configuration for your fast, local edge model (e.g. Ollama). |
+| `heavyModel` | `ModelConfig` | Configuration for your heavy cloud fallback (e.g. Gemini, OpenAI). |
+| `cascadeThreshold` | `number` | Confidence probability (0.0 to 1.0). If logprobs dip below this, it cascades. |
+| `tokensToEvaluate` | `number` | How many tokens to buffer before making the speculative decision. |
+| `customRules` | `Array<(prompt: string) => boolean>` | *(Optional)* Array of heuristic functions to override complex prompt detection. |
+
+### `router.chat(prompt, options?)`
+
+| Parameter | Type | Description |
+|---|---|---|
+| `prompt` | `string` | The user's input prompt. |
+| `options` | `ChatOptions` | *(Optional)* Options like `{ signal: AbortSignal }`. |
+
+**Returns:** `Promise<{ text: string, routedTo: 'fast' | 'heavy' }>`
+
+---
+
+## 🤝 Contributing
+
+We welcome contributions! Please follow the established homelab conventions:
+- Library code must NEVER use `console.warn` or `console.log` directly. Route diagnostics through callback options (`onEvent` pattern).
+- Ensure your `AbortSignal` listeners use `{ once: true }` to prevent leaks.
+- Run tests via `npm test` before submitting PRs.
+
+## 📄 License
+
+MIT License © 2026 kruschdev

package/dist/index.cjs

@@ -0,0 +1,254 @@
+"use strict";
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+
+// src/index.ts
+var index_exports = {};
+__export(index_exports, {
+  CascadeRouter: () => CascadeRouter,
+  isComplexPrompt: () => isComplexPrompt
+});
+module.exports = __toCommonJS(index_exports);
+
+// src/classifier.ts
+function isComplexPrompt(messages, options) {
+  const lengthThreshold = options?.lengthThreshold || 2e3;
+  const fullText = Array.isArray(messages) ? messages.map((m) => m.content).join("\n") : messages;
+  if (fullText.length > lengthThreshold) {
+    return true;
+  }
+  const complexMarkers = [
+    /```[a-z]*/i,
+    // Contains code blocks
+    /<\/?([a-z][a-z0-9]*)\b[^>]*>/i,
+    // Contains XML/HTML tags
+    /\{[\s\S]*"[\s\S]*\}/,
+    // Contains JSON-like structures
+    /\b(analyze|evaluate|architect|synthesize|speculate|refactor)\b/i
+    // Complex cognitive verbs
+  ];
+  if (options?.customRules) {
+    complexMarkers.push(...options.customRules);
+  }
+  for (const marker of complexMarkers) {
+    if (marker.test(fullText)) {
+      return true;
+    }
+  }
+  return false;
+}
+
+// src/cascade.ts
+var CascadeRouter = class {
+  config;
+  fetchFn;
+  constructor(config) {
+    this.config = {
+      ...config,
+      cascadeThreshold: config.cascadeThreshold ?? 0.85,
+      tokensToEvaluate: config.tokensToEvaluate ?? 5
+    };
+    this.fetchFn = config.fetch ?? (typeof globalThis !== "undefined" ? globalThis.fetch : fetch);
+    if (!this.fetchFn) {
+      throw new Error("A global fetch API is required, or a custom fetch implementation must be provided in RouterConfig.");
+    }
+  }
+  /**
+   * Complete a chat request, routing automatically.
+   */
+  async chat(messages, systemPrompt, options) {
+    const formattedMessages = this.formatMessages(messages, systemPrompt);
+    const isComplex = isComplexPrompt(formattedMessages, this.config.classifier);
+    if (isComplex) {
+      this.config.onEvent?.("route_heavy", { reason: "classifier_heuristic" });
+      const text = await this.fetchHeavyModel(formattedMessages, options);
+      return { text, routedTo: "heavy", aborted: false };
+    }
+    try {
+      const fastResult = await this.streamAndEvaluateFastModel(formattedMessages, options);
+      if (fastResult.aborted) {
+        this.config.onEvent?.("route_heavy", { reason: "cascade_fallback" });
+        const heavyText = await this.fetchHeavyModel(formattedMessages, options);
+        return { text: heavyText, routedTo: "heavy", aborted: true };
+      }
+      this.config.onEvent?.("route_fast", { reason: "high_confidence" });
+      return { text: fastResult.text, routedTo: "fast", aborted: false };
+    } catch (err) {
+      this.config.onEvent?.("route_heavy", { reason: "fast_model_error", error: err.message });
+      const heavyText = await this.fetchHeavyModel(formattedMessages, options);
+      return { text: heavyText, routedTo: "heavy", aborted: true };
+    }
+  }
+  formatMessages(messages, systemPrompt) {
+    const msgs = [];
+    if (systemPrompt) {
+      msgs.push({ role: "system", content: systemPrompt });
+    }
+    if (typeof messages === "string") {
+      msgs.push({ role: "user", content: messages });
+    } else {
+      msgs.push(...messages);
+    }
+    return msgs;
+  }
+  /**
+   * Streams the fast model, buffering the first N tokens to check logprobs.
+   * If confidence is lower than threshold, aborts and returns { aborted: true }.
+   */
+  async streamAndEvaluateFastModel(messages, options) {
+    const { fastModel } = this.config;
+    const url = fastModel.url || "http://localhost:11434/v1/chat/completions";
+    const headers = { "Content-Type": "application/json" };
+    if (fastModel.apiKey) headers["Authorization"] = `Bearer ${fastModel.apiKey}`;
+    const controller = new AbortController();
+    if (options?.signal) {
+      options.signal.addEventListener("abort", () => controller.abort(), { once: true });
+      if (options.signal.aborted) controller.abort();
+    }
+    const response = await this.fetchFn(url, {
+      method: "POST",
+      headers,
+      signal: controller.signal,
+      body: JSON.stringify({
+        model: fastModel.model,
+        messages,
+        stream: true,
+        logprobs: true
+        // Request logprobs (OpenAI format)
+      })
+    });
+    if (!response.ok) {
+      throw new Error(`Fast model HTTP ${response.status}`);
+    }
+    if (!response.body) throw new Error("No response body");
+    const reader = response.body.getReader();
+    const decoder = new TextDecoder("utf-8");
+    let fullText = "";
+    let tokenCount = 0;
+    let accumulatedProb = 0;
+    let buffer = "";
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      buffer += decoder.decode(value, { stream: true });
+      const lines = buffer.split("\n");
+      buffer = lines.pop() || "";
+      for (const line of lines) {
+        const trimmed = line.trim();
+        if (!trimmed || trimmed === "data: [DONE]") continue;
+        if (!trimmed.startsWith("data: ")) continue;
+        try {
+          const data = JSON.parse(trimmed.slice(6));
+          const choice = data.choices?.[0];
+          const delta = choice?.delta?.content || "";
+          if (delta) fullText += delta;
+          const logprobsObj = choice?.logprobs?.content;
+          if (logprobsObj && Array.isArray(logprobsObj) && logprobsObj.length > 0) {
+            for (const lp of logprobsObj) {
+              const logprob = lp.logprob;
+              if (logprob !== void 0) {
+                const linearProb = Math.exp(logprob);
+                accumulatedProb += linearProb;
+                tokenCount++;
+                if (tokenCount === this.config.tokensToEvaluate) {
+                  const avgProb = accumulatedProb / tokenCount;
+                  if (avgProb < (this.config.cascadeThreshold || 0.85)) {
+                    this.config.onEvent?.("cascade_triggered", { tokenCount, avgProb, threshold: this.config.cascadeThreshold || 0.85 });
+                    controller.abort();
+                    return { text: "", aborted: true };
+                  }
+                }
+              }
+            }
+          }
+        } catch (e) {
+          if (!(e instanceof SyntaxError)) throw e;
+        }
+      }
+    }
+    if (buffer.trim().startsWith("data: ")) {
+      try {
+        const data = JSON.parse(buffer.trim().slice(6));
+        const delta = data.choices?.[0]?.delta?.content || "";
+        if (delta) fullText += delta;
+      } catch (e) {
+        if (!(e instanceof SyntaxError)) throw e;
+      }
+    }
+    return { text: fullText, aborted: false };
+  }
+  /**
+   * Fallback to heavy model. Only returns the full string for now.
+   */
+  async fetchHeavyModel(messages, options) {
+    const { heavyModel } = this.config;
+    const provider = heavyModel.provider || "openai";
+    if (provider === "gemini") {
+      return this.fetchGemini(messages, options);
+    }
+    const url = heavyModel.url || "https://api.openai.com/v1/chat/completions";
+    const headers = { "Content-Type": "application/json" };
+    if (heavyModel.apiKey) headers["Authorization"] = `Bearer ${heavyModel.apiKey}`;
+    const response = await this.fetchFn(url, {
+      method: "POST",
+      headers,
+      signal: options?.signal,
+      body: JSON.stringify({
+        model: heavyModel.model,
+        messages,
+        stream: false
+      })
+    });
+    if (!response.ok) {
+      throw new Error(`Heavy model HTTP ${response.status}`);
+    }
+    const data = await response.json();
+    return data.choices?.[0]?.message?.content || "";
+  }
+  async fetchGemini(messages, options) {
+    const { heavyModel } = this.config;
+    const apiKey = heavyModel.apiKey;
+    if (!apiKey) throw new Error("Gemini requires an API key");
+    const url = `https://generativelanguage.googleapis.com/v1beta/models/${heavyModel.model}:generateContent?key=${apiKey}`;
+    const systemPrompt = messages.find((m) => m.role === "system")?.content;
+    const contents = messages.filter((m) => m.role !== "system").map((m) => ({
+      role: m.role === "assistant" ? "model" : "user",
+      parts: [{ text: m.content }]
+    }));
+    const body = { contents };
+    if (systemPrompt) {
+      body.system_instruction = { parts: [{ text: systemPrompt }] };
+    }
+    const response = await this.fetchFn(url, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      signal: options?.signal,
+      body: JSON.stringify(body)
+    });
+    if (!response.ok) {
+      throw new Error(`Gemini HTTP ${response.status}: ${await response.text()}`);
+    }
+    const data = await response.json();
+    return data.candidates?.[0]?.content?.parts?.[0]?.text || "";
+  }
+};
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  CascadeRouter,
+  isComplexPrompt
+});

package/dist/index.d.cts

@@ -0,0 +1,60 @@
+interface Message {
+    role: 'system' | 'user' | 'assistant';
+    content: string;
+}
+interface ClassifierOptions {
+    lengthThreshold?: number;
+    customRules?: RegExp[];
+}
+/**
+ * A fast, <50ms heuristic classifier to predict if a prompt is "simple" or "complex".
+ * Evaluates message length and structural markers (code blocks, XML, JSON).
+ */
+declare function isComplexPrompt(messages: Message[] | string, options?: ClassifierOptions): boolean;
+
+interface ModelConfig {
+    url?: string;
+    apiKey?: string;
+    model: string;
+    provider?: 'openai' | 'gemini';
+}
+type TelemetryEvent = 'route_fast' | 'route_heavy' | 'cascade_triggered';
+interface RouterConfig {
+    fastModel: ModelConfig;
+    heavyModel: ModelConfig;
+    cascadeThreshold?: number;
+    tokensToEvaluate?: number;
+    classifier?: ClassifierOptions;
+    fetch?: typeof fetch;
+    onEvent?: (event: TelemetryEvent, metadata?: Record<string, any>) => void;
+}
+interface CascadeResponse {
+    text: string;
+    routedTo: 'fast' | 'heavy';
+    aborted: boolean;
+}
+interface ChatOptions {
+    signal?: AbortSignal;
+}
+declare class CascadeRouter {
+    private config;
+    private fetchFn;
+    constructor(config: RouterConfig);
+    /**
+     * Complete a chat request, routing automatically.
+     */
+    chat(messages: Message[] | string, systemPrompt?: string, options?: ChatOptions): Promise<CascadeResponse>;
+    private formatMessages;
+    /**
+     * Streams the fast model, buffering the first N tokens to check logprobs.
+     * If confidence is lower than threshold, aborts and returns { aborted: true }.
+     */
+    private streamAndEvaluateFastModel;
+    /**
+     * Fallback to heavy model. Only returns the full string for now.
+     */
+    private fetchHeavyModel;
+    private fetchGemini;
+}
+
+export { type CascadeResponse, CascadeRouter, type ChatOptions, type ClassifierOptions, type Message, type ModelConfig, type RouterConfig, type TelemetryEvent, isComplexPrompt };

package/dist/index.d.ts

@@ -0,0 +1,60 @@
+interface Message {
+    role: 'system' | 'user' | 'assistant';
+    content: string;
+}
+interface ClassifierOptions {
+    lengthThreshold?: number;
+    customRules?: RegExp[];
+}
+/**
+ * A fast, <50ms heuristic classifier to predict if a prompt is "simple" or "complex".
+ * Evaluates message length and structural markers (code blocks, XML, JSON).
+ */
+declare function isComplexPrompt(messages: Message[] | string, options?: ClassifierOptions): boolean;
+
+interface ModelConfig {
+    url?: string;
+    apiKey?: string;
+    model: string;
+    provider?: 'openai' | 'gemini';
+}
+type TelemetryEvent = 'route_fast' | 'route_heavy' | 'cascade_triggered';
+interface RouterConfig {
+    fastModel: ModelConfig;
+    heavyModel: ModelConfig;
+    cascadeThreshold?: number;
+    tokensToEvaluate?: number;
+    classifier?: ClassifierOptions;
+    fetch?: typeof fetch;
+    onEvent?: (event: TelemetryEvent, metadata?: Record<string, any>) => void;
+}
+interface CascadeResponse {
+    text: string;
+    routedTo: 'fast' | 'heavy';
+    aborted: boolean;
+}
+interface ChatOptions {
+    signal?: AbortSignal;
+}
+declare class CascadeRouter {
+    private config;
+    private fetchFn;
+    constructor(config: RouterConfig);
+    /**
+     * Complete a chat request, routing automatically.
+     */
+    chat(messages: Message[] | string, systemPrompt?: string, options?: ChatOptions): Promise<CascadeResponse>;
+    private formatMessages;
+    /**
+     * Streams the fast model, buffering the first N tokens to check logprobs.
+     * If confidence is lower than threshold, aborts and returns { aborted: true }.
+     */
+    private streamAndEvaluateFastModel;
+    /**
+     * Fallback to heavy model. Only returns the full string for now.
+     */
+    private fetchHeavyModel;
+    private fetchGemini;
+}
+
+export { type CascadeResponse, CascadeRouter, type ChatOptions, type ClassifierOptions, type Message, type ModelConfig, type RouterConfig, type TelemetryEvent, isComplexPrompt };

package/dist/index.js

@@ -0,0 +1,226 @@
+// src/classifier.ts
+function isComplexPrompt(messages, options) {
+  const lengthThreshold = options?.lengthThreshold || 2e3;
+  const fullText = Array.isArray(messages) ? messages.map((m) => m.content).join("\n") : messages;
+  if (fullText.length > lengthThreshold) {
+    return true;
+  }
+  const complexMarkers = [
+    /```[a-z]*/i,
+    // Contains code blocks
+    /<\/?([a-z][a-z0-9]*)\b[^>]*>/i,
+    // Contains XML/HTML tags
+    /\{[\s\S]*"[\s\S]*\}/,
+    // Contains JSON-like structures
+    /\b(analyze|evaluate|architect|synthesize|speculate|refactor)\b/i
+    // Complex cognitive verbs
+  ];
+  if (options?.customRules) {
+    complexMarkers.push(...options.customRules);
+  }
+  for (const marker of complexMarkers) {
+    if (marker.test(fullText)) {
+      return true;
+    }
+  }
+  return false;
+}
+
+// src/cascade.ts
+var CascadeRouter = class {
+  config;
+  fetchFn;
+  constructor(config) {
+    this.config = {
+      ...config,
+      cascadeThreshold: config.cascadeThreshold ?? 0.85,
+      tokensToEvaluate: config.tokensToEvaluate ?? 5
+    };
+    this.fetchFn = config.fetch ?? (typeof globalThis !== "undefined" ? globalThis.fetch : fetch);
+    if (!this.fetchFn) {
+      throw new Error("A global fetch API is required, or a custom fetch implementation must be provided in RouterConfig.");
+    }
+  }
+  /**
+   * Complete a chat request, routing automatically.
+   */
+  async chat(messages, systemPrompt, options) {
+    const formattedMessages = this.formatMessages(messages, systemPrompt);
+    const isComplex = isComplexPrompt(formattedMessages, this.config.classifier);
+    if (isComplex) {
+      this.config.onEvent?.("route_heavy", { reason: "classifier_heuristic" });
+      const text = await this.fetchHeavyModel(formattedMessages, options);
+      return { text, routedTo: "heavy", aborted: false };
+    }
+    try {
+      const fastResult = await this.streamAndEvaluateFastModel(formattedMessages, options);
+      if (fastResult.aborted) {
+        this.config.onEvent?.("route_heavy", { reason: "cascade_fallback" });
+        const heavyText = await this.fetchHeavyModel(formattedMessages, options);
+        return { text: heavyText, routedTo: "heavy", aborted: true };
+      }
+      this.config.onEvent?.("route_fast", { reason: "high_confidence" });
+      return { text: fastResult.text, routedTo: "fast", aborted: false };
+    } catch (err) {
+      this.config.onEvent?.("route_heavy", { reason: "fast_model_error", error: err.message });
+      const heavyText = await this.fetchHeavyModel(formattedMessages, options);
+      return { text: heavyText, routedTo: "heavy", aborted: true };
+    }
+  }
+  formatMessages(messages, systemPrompt) {
+    const msgs = [];
+    if (systemPrompt) {
+      msgs.push({ role: "system", content: systemPrompt });
+    }
+    if (typeof messages === "string") {
+      msgs.push({ role: "user", content: messages });
+    } else {
+      msgs.push(...messages);
+    }
+    return msgs;
+  }
+  /**
+   * Streams the fast model, buffering the first N tokens to check logprobs.
+   * If confidence is lower than threshold, aborts and returns { aborted: true }.
+   */
+  async streamAndEvaluateFastModel(messages, options) {
+    const { fastModel } = this.config;
+    const url = fastModel.url || "http://localhost:11434/v1/chat/completions";
+    const headers = { "Content-Type": "application/json" };
+    if (fastModel.apiKey) headers["Authorization"] = `Bearer ${fastModel.apiKey}`;
+    const controller = new AbortController();
+    if (options?.signal) {
+      options.signal.addEventListener("abort", () => controller.abort(), { once: true });
+      if (options.signal.aborted) controller.abort();
+    }
+    const response = await this.fetchFn(url, {
+      method: "POST",
+      headers,
+      signal: controller.signal,
+      body: JSON.stringify({
+        model: fastModel.model,
+        messages,
+        stream: true,
+        logprobs: true
+        // Request logprobs (OpenAI format)
+      })
+    });
+    if (!response.ok) {
+      throw new Error(`Fast model HTTP ${response.status}`);
+    }
+    if (!response.body) throw new Error("No response body");
+    const reader = response.body.getReader();
+    const decoder = new TextDecoder("utf-8");
+    let fullText = "";
+    let tokenCount = 0;
+    let accumulatedProb = 0;
+    let buffer = "";
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      buffer += decoder.decode(value, { stream: true });
+      const lines = buffer.split("\n");
+      buffer = lines.pop() || "";
+      for (const line of lines) {
+        const trimmed = line.trim();
+        if (!trimmed || trimmed === "data: [DONE]") continue;
+        if (!trimmed.startsWith("data: ")) continue;
+        try {
+          const data = JSON.parse(trimmed.slice(6));
+          const choice = data.choices?.[0];
+          const delta = choice?.delta?.content || "";
+          if (delta) fullText += delta;
+          const logprobsObj = choice?.logprobs?.content;
+          if (logprobsObj && Array.isArray(logprobsObj) && logprobsObj.length > 0) {
+            for (const lp of logprobsObj) {
+              const logprob = lp.logprob;
+              if (logprob !== void 0) {
+                const linearProb = Math.exp(logprob);
+                accumulatedProb += linearProb;
+                tokenCount++;
+                if (tokenCount === this.config.tokensToEvaluate) {
+                  const avgProb = accumulatedProb / tokenCount;
+                  if (avgProb < (this.config.cascadeThreshold || 0.85)) {
+                    this.config.onEvent?.("cascade_triggered", { tokenCount, avgProb, threshold: this.config.cascadeThreshold || 0.85 });
+                    controller.abort();
+                    return { text: "", aborted: true };
+                  }
+                }
+              }
+            }
+          }
+        } catch (e) {
+          if (!(e instanceof SyntaxError)) throw e;
+        }
+      }
+    }
+    if (buffer.trim().startsWith("data: ")) {
+      try {
+        const data = JSON.parse(buffer.trim().slice(6));
+        const delta = data.choices?.[0]?.delta?.content || "";
+        if (delta) fullText += delta;
+      } catch (e) {
+        if (!(e instanceof SyntaxError)) throw e;
+      }
+    }
+    return { text: fullText, aborted: false };
+  }
+  /**
+   * Fallback to heavy model. Only returns the full string for now.
+   */
+  async fetchHeavyModel(messages, options) {
+    const { heavyModel } = this.config;
+    const provider = heavyModel.provider || "openai";
+    if (provider === "gemini") {
+      return this.fetchGemini(messages, options);
+    }
+    const url = heavyModel.url || "https://api.openai.com/v1/chat/completions";
+    const headers = { "Content-Type": "application/json" };
+    if (heavyModel.apiKey) headers["Authorization"] = `Bearer ${heavyModel.apiKey}`;
+    const response = await this.fetchFn(url, {
+      method: "POST",
+      headers,
+      signal: options?.signal,
+      body: JSON.stringify({
+        model: heavyModel.model,
+        messages,
+        stream: false
+      })
+    });
+    if (!response.ok) {
+      throw new Error(`Heavy model HTTP ${response.status}`);
+    }
+    const data = await response.json();
+    return data.choices?.[0]?.message?.content || "";
+  }
+  async fetchGemini(messages, options) {
+    const { heavyModel } = this.config;
+    const apiKey = heavyModel.apiKey;
+    if (!apiKey) throw new Error("Gemini requires an API key");
+    const url = `https://generativelanguage.googleapis.com/v1beta/models/${heavyModel.model}:generateContent?key=${apiKey}`;
+    const systemPrompt = messages.find((m) => m.role === "system")?.content;
+    const contents = messages.filter((m) => m.role !== "system").map((m) => ({
+      role: m.role === "assistant" ? "model" : "user",
+      parts: [{ text: m.content }]
+    }));
+    const body = { contents };
+    if (systemPrompt) {
+      body.system_instruction = { parts: [{ text: systemPrompt }] };
+    }
+    const response = await this.fetchFn(url, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      signal: options?.signal,
+      body: JSON.stringify(body)
+    });
+    if (!response.ok) {
+      throw new Error(`Gemini HTTP ${response.status}: ${await response.text()}`);
+    }
+    const data = await response.json();
+    return data.candidates?.[0]?.content?.parts?.[0]?.text || "";
+  }
+};
+export {
+  CascadeRouter,
+  isComplexPrompt
+};

package/package.json

@@ -0,0 +1,54 @@
+{
+  "name": "krusch-cascade-router",
+  "version": "1.0.0",
+  "description": "Latency-aware LLM router that dynamically cascades between edge and cloud models via logprob inspection.",
+  "main": "dist/index.cjs",
+  "module": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "type": "module",
+  "exports": {
+    ".": {
+      "import": {
+        "types": "./dist/index.d.ts",
+        "default": "./dist/index.js"
+      },
+      "require": {
+        "types": "./dist/index.d.cts",
+        "default": "./dist/index.cjs"
+      }
+    }
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "build": "tsup src/index.ts --format cjs,esm --dts",
+    "dev": "tsup src/index.ts --format cjs,esm --watch",
+    "test": "node --test"
+  },
+  "keywords": [
+    "llm",
+    "router",
+    "cascade",
+    "logprobs",
+    "agentic",
+    "local-ai"
+  ],
+  "author": "kruschdev",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/kruschdev/krusch-cascade-router.git"
+  },
+  "homepage": "https://github.com/kruschdev/krusch-cascade-router#readme",
+  "bugs": {
+    "url": "https://github.com/kruschdev/krusch-cascade-router/issues"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "tsup": "^8.0.0",
+    "typescript": "^5.0.0"
+  }
+}