lynkr 9.0.1 → 9.0.2

package/README.md

@@ -3,7 +3,7 @@
 ### Run Claude Code, Cursor, and Codex on any model. One proxy, every provider.
 
 [![npm version](https://img.shields.io/npm/v/lynkr.svg)](https://www.npmjs.com/package/lynkr)
-[![Tests](https://img.shields.io/badge/tests-652%20passing-brightgreen)](https://github.com/vishalveerareddy123/Lynkr)
+[![Tests](https://img.shields.io/badge/tests-699%20passing-brightgreen)](https://github.com/Fast-Editor/Lynkr)
 [![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
 [![Node.js](https://img.shields.io/badge/node-20%2B-green)](https://nodejs.org)
 [![Homebrew Tap](https://img.shields.io/badge/homebrew-lynkr-brightgreen.svg)](https://github.com/vishalveerareddy123/homebrew-lynkr)
@@ -11,9 +11,9 @@
 
 <table>
 <tr>
-<td align="center"><strong>10+</strong><br/>LLM Providers</td>
+<td align="center"><strong>12+</strong><br/>LLM Providers</td>
 <td align="center"><strong>60-80%</strong><br/>Cost Reduction</td>
-<td align="center"><strong>652</strong><br/>Tests Passing</td>
+<td align="center"><strong>699</strong><br/>Tests Passing</td>
 <td align="center"><strong>0</strong><br/>Code Changes Required</td>
 </tr>
 </table>
@@ -55,6 +55,12 @@ lynkr start
 
 ### Install
 
+**One-line install (recommended):**
+```bash
+curl -fsSL https://raw.githubusercontent.com/Fast-Editor/Lynkr/main/install.sh | bash
+```
+
+**Or via npm:**
 ```bash
 npm install -g pino-pretty && npm install -g lynkr
 ```
@@ -125,7 +131,24 @@ const { text } = await generateText({
 });
 ```
 
-> Works with any OpenAI-compatible client: Cline, Continue.dev, ClawdBot, KiloCode, and more.
+**OpenClaw**
+```json
+// openclaw.json
+{
+  "models": {
+    "providers": [{
+      "name": "lynkr",
+      "type": "openai-compatible",
+      "base_url": "http://localhost:8081/v1",
+      "api_key": "any-value",
+      "models": ["auto"]
+    }]
+  }
+}
+```
+Set `OPENCLAW_MODE=true` in Lynkr's `.env` to show actual provider/model in responses.
+
+> Works with any OpenAI-compatible client: Cline, Continue.dev, OpenClaw, KiloCode, and more.
 
 ---
 
@@ -139,12 +162,16 @@ const { text } = await generateText({
 | **MLX Server** | Local | Apple Silicon optimized | **Free** |
 | **AWS Bedrock** | Cloud | 100+ (Claude, Llama, Mistral, Titan) | $$ |
 | **OpenRouter** | Cloud | 100+ (GPT, Claude, Llama, Gemini) | $-$$ |
-| **Databricks** | Cloud | Claude Sonnet 4.5, Opus 4.5 | $$$ |
-| **Azure OpenAI** | Cloud | GPT-4o, GPT-5, o1, o3 | $$$ |
+| **Databricks** | Cloud | Claude Sonnet 4.5, Opus 4.6 | $$$ |
+| **Azure OpenAI** | Cloud | GPT-4o, o1, o3 | $$$ |
 | **Azure Anthropic** | Cloud | Claude models | $$$ |
-| **OpenAI** | Cloud | GPT-4o, o1, o3 | $$$ |
+| **OpenAI** | Cloud | GPT-4o, o3, o4-mini | $$$ |
+| **Google Vertex** | Cloud | Gemini 2.5 Pro/Flash | $$$ |
+| **Moonshot AI** | Cloud | Kimi K2 Thinking/Turbo | $$ |
+| **Z.AI** | Cloud | GLM-4.7 | $$ |
+| **DeepSeek** | Cloud | DeepSeek Reasoner, R1 | $ |
 
-4 local providers for **100% offline, free** usage. 6+ cloud providers for scale.
+4 local providers for **100% offline, free** usage. 10+ cloud providers for scale.
 
 ---
 
@@ -166,6 +193,9 @@ const { text } = await generateText({
 | **Transaction fees** | None | None (OSS) / Paid enterprise | 5.5% on credits | Free tier / Paid |
 | **Dependencies** | Node.js only | Python, Prisma, PostgreSQL | N/A | Docker, Python |
 | **Format conversion** | Anthropic <-> OpenAI (automatic) | Automatic | N/A | Automatic |
+| **Code intelligence** | Graphify (19-lang AST graph) | No | No | No |
+| **Routing telemetry** | Built-in (SQLite + REST API) | No | Dashboard | Dashboard |
+| **Admin hot-reload** | Yes (no restart) | Requires restart | N/A | Requires restart |
 | **License** | Apache 2.0 | MIT | Proprietary | MIT (gateway) |
 
 **Lynkr's edge:** Purpose-built for AI coding tools. Not a general LLM gateway — a proxy that understands Claude Code, Cursor, and Codex natively, with built-in token optimization, complexity-based routing, and a memory system designed for coding workflows. Installs in one command, runs on Node.js, zero infrastructure required.
@@ -188,20 +218,29 @@ const { text } = await generateText({
 
 Lynkr isn't just a passthrough proxy. It's an optimization layer.
 
-### Smart Routing
-Routes requests to the right model based on task complexity. Simple questions go to fast/cheap models. Complex architectural tasks go to powerful models. You configure the tiers.
+### Smart Routing (5-Phase)
+Routes requests to the right model based on 5-phase complexity analysis. Simple questions go to fast/cheap models. Complex architectural tasks go to powerful models. Includes Graphify structural analysis for code-aware routing.
+
+- **Complexity scoring** — 15-dimension weighted scoring with agentic workflow detection
+- **Graphify integration** — AST-based knowledge graph detects god nodes, community cohesion, blast radius across 19 languages
+- **Routing telemetry** — every decision recorded with quality scoring (0-100) and latency tracking (P50/P95/P99)
 
-### Token Optimization
+### Token Optimization (7 Phases)
 - **Smart tool selection** — only sends tools relevant to the current task
-- **Prompt compression** — removes redundant context before sending
+- **Code Mode** — replaces 100+ MCP tools with 4 meta-tools (~96% token reduction)
+- **Distill compression** — structural similarity, delta rendering, smart dedup of repetitive tool outputs
+- **Prompt caching** — SHA-256 keyed LRU cache
 - **Memory deduplication** — eliminates repeated information across turns
-- **TOON format** — compact serialization that cuts token count
+- **History compression** — sliding window with Distill-powered structural dedup
+- **Headroom sidecar** — optional 47-92% ML-based compression (Smart Crusher, CCR, LLMLingua)
 
 ### Enterprise Resilience
-- **Circuit breakers** — automatic failover when a provider goes down
+- **Circuit breakers** — automatic failover with half-open probe recovery
+- **Admin hot-reload** — `POST /v1/admin/reload` reloads config + resets circuit breakers without restart
 - **Load shedding** — graceful degradation under high load
 - **Prometheus metrics** — full observability at `/metrics`
 - **Health checks** — K8s-ready endpoints at `/health`
+- **Performance timer** — per-request timing breakdown with `PERF_TIMER=true`
 
 ### Memory System
 Titans-inspired long-term memory with surprise-based filtering. The system remembers important context across sessions and forgets noise — reducing token waste from repeated context.
@@ -214,14 +253,23 @@ SEMANTIC_CACHE_ENABLED=true
 SEMANTIC_CACHE_THRESHOLD=0.95
 ```
 
-### MCP Integration
-Automatic Model Context Protocol server discovery and orchestration. Your MCP tools work through Lynkr without configuration.
+### MCP Integration + Code Mode
+Automatic Model Context Protocol server discovery and orchestration. Your MCP tools work through Lynkr without configuration. Enable Code Mode to replace 100+ MCP tool definitions with 4 lightweight meta-tools:
+
+```bash
+CODE_MODE_ENABLED=true  # ~96% reduction in tool-catalog tokens
+```
 
 ---
 
 ## Deployment Options
 
-**NPM (recommended)**
+**One-line install (recommended)**
+```bash
+curl -fsSL https://raw.githubusercontent.com/Fast-Editor/Lynkr/main/install.sh | bash
+```
+
+**NPM**
 ```bash
 npm install -g lynkr && lynkr start
 ```
@@ -233,7 +281,7 @@ docker-compose up -d
 
 **Git Clone**
 ```bash
-git clone https://github.com/vishalveerareddy123/Lynkr.git
+git clone https://github.com/Fast-Editor/Lynkr.git
 cd Lynkr && npm install && cp .env.example .env
 npm start
 ```
@@ -251,9 +299,10 @@ brew install lynkr
 | Guide | Description |
 |-------|-------------|
 | [Installation](documentation/installation.md) | All installation methods |
-| [Provider Config](documentation/providers.md) | Setup for all 10+ providers |
+| [Provider Config](documentation/providers.md) | Setup for all 12+ providers |
 | [Claude Code CLI](documentation/claude-code-cli.md) | Detailed Claude Code integration |
 | [Codex CLI](documentation/codex-cli.md) | Codex config.toml setup |
+| [OpenClaw](documentation/openclaw-integration.md) | OpenClaw integration with tier routing |
 | [Cursor IDE](documentation/cursor-integration.md) | Cursor integration + troubleshooting |
 | [Embeddings](documentation/embeddings.md) | @Codebase semantic search (4 options) |
 | [Token Optimization](documentation/token-optimization.md) | 60-80% cost reduction strategies |
@@ -293,8 +342,8 @@ Apache 2.0 — See [LICENSE](LICENSE).
 
 ## Community
 
-- [GitHub Discussions](https://github.com/vishalveerareddy123/Lynkr/discussions) — Questions and tips
-- [Report Issues](https://github.com/vishalveerareddy123/Lynkr/issues) — Bug reports and feature requests
+- [GitHub Discussions](https://github.com/Fast-Editor/Lynkr/discussions) — Questions and tips
+- [Report Issues](https://github.com/Fast-Editor/Lynkr/issues) — Bug reports and feature requests
 - [NPM Package](https://www.npmjs.com/package/lynkr) — Official package
 - [DeepWiki](https://deepwiki.com/vishalveerareddy123/Lynkr) — AI-powered docs search
 

package/bin/cli.js

@@ -17,11 +17,15 @@ Usage:
   lynkr [options]
 
 Options:
-  -h, --help     Show this help message
-  -v, --version  Show version number
+  -h, --help      Show this help message
+  -v, --version   Show version number
+  --cluster       Enable cluster mode (multi-core)
+  --workers N     Number of worker processes (default: auto)
 
 Environment Variables:
-  See .env.example for configuration options
+  CLUSTER_ENABLED=true    Enable multi-core cluster mode
+  CLUSTER_WORKERS=auto    Worker count (auto = CPU cores - 1)
+  See .env.example for all configuration options
 
 Documentation:
   ${pkg.homepage}
@@ -29,4 +33,13 @@ Documentation:
   process.exit(0);
 }
 
+// CLI flags for cluster mode
+if (process.argv.includes('--cluster')) {
+  process.env.CLUSTER_ENABLED = 'true';
+}
+const workersIdx = process.argv.indexOf('--workers');
+if (workersIdx !== -1 && process.argv[workersIdx + 1]) {
+  process.env.CLUSTER_WORKERS = process.argv[workersIdx + 1];
+}
+
 require("../index.js");

package/index.js

@@ -1,3 +1,7 @@
-const { start } = require("./src/server");
-
-start();
+if (process.env.CLUSTER_ENABLED === 'true') {
+  const { startCluster } = require('./src/cluster');
+  startCluster();
+} else {
+  const { start } = require('./src/server');
+  start();
+}

package/install.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 #
 # Lynkr Installation Script
-# Usage: curl -fsSL https://raw.githubusercontent.com/vishalveerareddy123/Lynkr/main/install.sh | bash
+# Usage: curl -fsSL https://raw.githubusercontent.com/Fast-Editor/Lynkr/main/install.sh | bash
 #
 # This script installs Lynkr, a self-hosted Claude Code proxy with multi-provider support.
 #
@@ -125,7 +125,7 @@ create_env_file() {
             # Fallback: create minimal .env if .env.example doesn't exist
             cat > "$INSTALL_DIR/.env" << 'EOF'
 # Lynkr Configuration
-# For full options, see: https://github.com/vishalveerareddy123/Lynkr/blob/main/.env.example
+# For full options, see: https://github.com/Fast-Editor/Lynkr/blob/main/.env.example
 
 # Model Provider (databricks, openai, azure-openai, azure-anthropic, openrouter, ollama, llamacpp)
 MODEL_PROVIDER=ollama
@@ -247,7 +247,7 @@ print_next_steps() {
     echo "💡 ${YELLOW}Tip:${NC} Memory system is enabled by default"
     echo "   Lynkr remembers preferences and project context across sessions"
     echo ""
-    echo "📚 Documentation: ${BLUE}https://github.com/vishalveerareddy123/Lynkr${NC}"
+    echo "📚 Documentation: ${BLUE}https://github.com/Fast-Editor/Lynkr${NC}"
     echo "💬 Discord: ${BLUE}https://discord.gg/qF7DDxrX${NC}"
     echo ""
 }

package/native/Cargo.toml

@@ -0,0 +1,26 @@
+[package]
+name = "lynkr-native"
+version = "0.1.0"
+edition = "2024"
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+napi = { version = "3", features = ["napi9", "serde-json"] }
+napi-derive = "3"
+serde = { version = "1", features = ["derive"] }
+serde_json = { version = "1", features = ["preserve_order"] }
+regex = "1"
+sha2 = "0.10"
+hex = "0.4"
+mimalloc = { version = "0.1", default-features = false }
+
+[build-dependencies]
+napi-build = "2"
+
+[profile.release]
+opt-level = 3
+lto = "fat"
+codegen-units = 1
+strip = true

package/native/index.js

@@ -0,0 +1,29 @@
+/**
+ * Lynkr Native — Rust-powered hot-path functions
+ *
+ * Loads the native .node addon for 10-50x speedup on:
+ * - Complexity analysis (regex patterns)
+ * - Cache key computation (recursive sort + SHA-256)
+ * - Structural similarity (Jaccard on line sets)
+ * - Text normalization (ANSI strip + whitespace collapse)
+ * - Payload size estimation
+ *
+ * Falls back to JS implementations if the native addon is unavailable.
+ */
+
+let native = null;
+
+try {
+  native = require('./lynkr-native.node');
+} catch {
+  // Native addon not available — fall back to JS
+}
+
+module.exports = {
+  available: native !== null,
+  analyzeComplexityNative: native?.analyzeComplexityNative ?? null,
+  computeCacheKey: native?.computeCacheKey ?? null,
+  structuralSimilarity: native?.structuralSimilarity ?? null,
+  normalizeText: native?.normalizeText ?? null,
+  estimatePayloadSize: native?.estimatePayloadSize ?? null,
+};

package/native/src/lib.rs

@@ -0,0 +1,321 @@
+use mimalloc::MiMalloc;
+
+#[global_allocator]
+static GLOBAL: MiMalloc = MiMalloc;
+
+use napi::bindgen_prelude::*;
+use napi_derive::napi;
+use regex::Regex;
+use sha2::{Digest, Sha256};
+use std::collections::BTreeMap;
+use std::sync::LazyLock;
+
+// ── 1. Complexity Analysis (15+ regex patterns at native speed) ─────
+
+/// Pre-compiled regex patterns — compiled once, reused forever
+static GREETING_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)^(hi|hello|hey|thanks?|bye|goodbye|good morning|good evening|good afternoon|good night|howdy|greetings|welcome)\b").unwrap());
+
+static YES_NO_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)^(yes|no|ok|okay|sure|y|n|yep|nope|yea|nah|affirmative|negative|roger|copy)\s*[.!?]*$").unwrap());
+
+static SIMPLE_QUESTION_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)^(what|where|when|who|how|why|which|is|are|do|does|can|could|will|would|should)\b.{0,80}[?]?\s*$").unwrap());
+
+static TECHNICAL_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(function|class|module|import|export|async|await|promise|api|database|server|client|component|interface|struct|enum|trait|impl|const|let|var|def|return|throw|catch|try|if|else|for|while|loop|match|switch|case)\b").unwrap());
+
+static SECURITY_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(security|audit|vulnerab|exploit|injection|xss|csrf|auth|encrypt|decrypt|certificate|tls|ssl|oauth|jwt|token|permission|privilege|sanitize|escape|hash|salt)\b").unwrap());
+
+static ARCHITECTURE_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(architect|design|pattern|microservice|monolith|scale|distributed|event.?driven|cqrs|saga|domain.?driven|hexagonal|clean.?arch|solid|dry|kiss)\b").unwrap());
+
+static REFACTOR_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(refactor|restructure|reorganize|rewrite|rearchitect|decompos|extract|consolidat|simplif|clean.?up|tech.?debt)\b").unwrap());
+
+static MULTI_FILE_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(all files|every file|entire|codebase|project.?wide|across.?the|multiple files|several files|many files)\b").unwrap());
+
+static CONCURRENCY_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(async|await|concurrent|parallel|thread|mutex|lock|deadlock|race.?condition|semaphore|channel|atomic|worker|pool)\b").unwrap());
+
+static PERFORMANCE_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(performance|optimize|bottleneck|profil|benchmark|latency|throughput|cache|memory.?leak|cpu|heap|gc|garbage)\b").unwrap());
+
+static DATABASE_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(database|sql|query|migration|schema|index|transaction|join|aggregate|stored.?proc|trigger|view|orm|sequelize|prisma|knex|typeorm)\b").unwrap());
+
+static REASONING_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"(?i)\b(step.?by.?step|think.*through|analyz|compar|trade.?off|pros?.?and?.?cons|evaluat|assess|consider|weigh|reason|logic|deduc)\b").unwrap());
+
+static FORCE_CLOUD_RE: LazyLock<Vec<Regex>> = LazyLock::new(|| {
+    vec![
+        Regex::new(r"(?i)\bsecurity\s+(audit|review)\b").unwrap(),
+        Regex::new(r"(?i)\barchitect(ure)?\s+(design|review)\b").unwrap(),
+        Regex::new(r"(?i)\b(complete|full|entire)\s+codebase\s+refactor").unwrap(),
+        Regex::new(r"(?i)\bcode\s+review\b").unwrap(),
+        Regex::new(r"(?i)\bpr\s+review\b").unwrap(),
+        Regex::new(r"(?i)\bcomplex\s+debug").unwrap(),
+        Regex::new(r"(?i)\bproduction\s+(incident|outage|issue)\b").unwrap(),
+    ]
+});
+
+static FORCE_LOCAL_RE: LazyLock<Vec<Regex>> = LazyLock::new(|| {
+    vec![
+        Regex::new(r"(?i)^(hi|hello|hey|thanks?|bye|goodbye)\s*[.!?]*$").unwrap(),
+        Regex::new(r"(?i)^what\s+time\s+is\s+it").unwrap(),
+        Regex::new(r"(?i)^(yes|no|ok|okay|sure|y|n)\s*[.!?]*$").unwrap(),
+        Regex::new(r"(?i)^(help|commands?|menu)\s*[.!?]*$").unwrap(),
+    ]
+});
+
+#[napi(object)]
+pub struct ComplexityResult {
+    pub score: u32,
+    pub force_local: bool,
+    pub force_cloud: bool,
+    pub token_score: u32,
+    pub task_type_score: u32,
+    pub code_complexity_score: u32,
+    pub reasoning_score: u32,
+}
+
+/// Analyze request complexity — Rust regex engine is 10-50x faster than JS RegExp
+#[napi]
+pub fn analyze_complexity_native(content: String, token_estimate: u32, tool_count: u32) -> ComplexityResult {
+    // Force patterns (short-circuit)
+    let force_local = FORCE_LOCAL_RE.iter().any(|re| re.is_match(&content));
+    if force_local {
+        return ComplexityResult {
+            score: 0,
+            force_local: true,
+            force_cloud: false,
+            token_score: 0,
+            task_type_score: 0,
+            code_complexity_score: 0,
+            reasoning_score: 0,
+        };
+    }
+
+    let force_cloud = FORCE_CLOUD_RE.iter().any(|re| re.is_match(&content));
+
+    // Token score (0-20)
+    let token_score = match token_estimate {
+        0..500 => 0,
+        500..1000 => 4,
+        1000..2000 => 8,
+        2000..4000 => 12,
+        4000..8000 => 16,
+        _ => 20,
+    };
+
+    // Tool score (0-20)
+    let tool_score = match tool_count {
+        0 => 0,
+        1..=3 => 4,
+        4..=6 => 8,
+        7..=10 => 12,
+        11..=15 => 16,
+        _ => 20,
+    };
+
+    // Task type (0-25)
+    let task_type_score = if GREETING_RE.is_match(&content) || YES_NO_RE.is_match(&content) {
+        0
+    } else if SIMPLE_QUESTION_RE.is_match(&content) {
+        3
+    } else if REFACTOR_RE.is_match(&content) {
+        16
+    } else if MULTI_FILE_RE.is_match(&content) {
+        22
+    } else if force_cloud {
+        25
+    } else if TECHNICAL_RE.is_match(&content) {
+        10
+    } else {
+        5
+    };
+
+    // Code complexity (0-20)
+    let mut code_score: u32 = 0;
+    if MULTI_FILE_RE.is_match(&content) { code_score += 5; }
+    if ARCHITECTURE_RE.is_match(&content) { code_score += 5; }
+    if SECURITY_RE.is_match(&content) { code_score += 4; }
+    if CONCURRENCY_RE.is_match(&content) { code_score += 3; }
+    if PERFORMANCE_RE.is_match(&content) { code_score += 3; }
+    if DATABASE_RE.is_match(&content) { code_score += 3; }
+    let code_complexity_score = code_score.min(20);
+
+    // Reasoning (0-15)
+    let reasoning_score = if REASONING_RE.is_match(&content) { 4 } else { 0 };
+
+    let total = (token_score + tool_score + task_type_score + code_complexity_score + reasoning_score).min(100);
+
+    ComplexityResult {
+        score: if force_cloud { total.max(76) } else { total },
+        force_local: false,
+        force_cloud,
+        token_score,
+        task_type_score,
+        code_complexity_score,
+        reasoning_score,
+    }
+}
+
+// ── 2. Cache Key Computation (recursive sort + SHA-256) ─────────────
+
+/// Recursively sort all object keys and produce a stable SHA-256 hash.
+/// This is the hot path for prompt cache key generation.
+#[napi]
+pub fn compute_cache_key(json_str: String) -> String {
+    let normalized = match serde_json::from_str::<serde_json::Value>(&json_str) {
+        Ok(val) => normalize_value(&val),
+        Err(_) => {
+            // Fallback: hash the raw string
+            let mut hasher = Sha256::new();
+            hasher.update(json_str.as_bytes());
+            return hex::encode(hasher.finalize());
+        }
+    };
+
+    let stable = serde_json::to_string(&normalized).unwrap_or(json_str);
+    let mut hasher = Sha256::new();
+    hasher.update(stable.as_bytes());
+    hex::encode(hasher.finalize())
+}
+
+/// Recursively normalize a JSON value: sort object keys, preserve arrays
+fn normalize_value(val: &serde_json::Value) -> serde_json::Value {
+    match val {
+        serde_json::Value::Object(map) => {
+            let mut sorted = BTreeMap::new();
+            for (k, v) in map {
+                sorted.insert(k.clone(), normalize_value(v));
+            }
+            serde_json::Value::Object(sorted.into_iter().collect())
+        }
+        serde_json::Value::Array(arr) => {
+            serde_json::Value::Array(arr.iter().map(normalize_value).collect())
+        }
+        other => other.clone(),
+    }
+}
+
+// ── 3. Structural Similarity (Jaccard on line sets) ─────────────────
+
+/// Compute Jaccard similarity between two text blocks using normalized line sets.
+/// Used by Distill compression for dedup detection.
+#[napi]
+pub fn structural_similarity(a: String, b: String) -> f64 {
+    if a.is_empty() && b.is_empty() {
+        return 1.0;
+    }
+    if a.is_empty() || b.is_empty() {
+        return 0.0;
+    }
+
+    let set_a: std::collections::HashSet<&str> = a.lines()
+        .map(|l| l.trim())
+        .filter(|l| !l.is_empty())
+        .collect();
+
+    let set_b: std::collections::HashSet<&str> = b.lines()
+        .map(|l| l.trim())
+        .filter(|l| !l.is_empty())
+        .collect();
+
+    if set_a.is_empty() && set_b.is_empty() {
+        return 1.0;
+    }
+
+    let intersection = set_a.intersection(&set_b).count();
+    let union = set_a.union(&set_b).count();
+
+    if union == 0 { 0.0 } else { intersection as f64 / union as f64 }
+}
+
+// ── 4. Text Normalization (ANSI strip + whitespace collapse) ────────
+
+static ANSI_RE: LazyLock<Regex> =
+    LazyLock::new(|| Regex::new(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])").unwrap());
+
+/// Strip ANSI escape codes and normalize whitespace.
+/// Used by Distill compression on every tool result.
+#[napi]
+pub fn normalize_text(text: String) -> String {
+    let stripped = ANSI_RE.replace_all(&text, "");
+    let normalized = stripped
+        .replace("\r\n", "\n")
+        .replace('\r', "\n");
+
+    // Collapse whitespace runs
+    let mut result = String::with_capacity(normalized.len());
+    let mut prev_space = false;
+    let mut newline_count = 0;
+
+    for ch in normalized.chars() {
+        if ch == '\n' {
+            newline_count += 1;
+            if newline_count <= 2 {
+                result.push('\n');
+            }
+            prev_space = false;
+        } else if ch == ' ' || ch == '\t' {
+            if !prev_space {
+                result.push(' ');
+                prev_space = true;
+            }
+            newline_count = 0;
+        } else {
+            result.push(ch);
+            prev_space = false;
+            newline_count = 0;
+        }
+    }
+
+    result.trim().to_string()
+}
+
+// ── 5. Payload Size Estimation ──────────────────────────────────────
+
+/// Estimate payload content size without full JSON serialization.
+/// Scans for base64 image data and text content lengths.
+#[napi]
+pub fn estimate_payload_size(json_str: String) -> u64 {
+    let val: serde_json::Value = match serde_json::from_str(&json_str) {
+        Ok(v) => v,
+        Err(_) => return json_str.len() as u64,
+    };
+
+    let messages = match val.get("messages").and_then(|m| m.as_array()) {
+        Some(m) => m,
+        None => return 0,
+    };
+
+    let mut size: u64 = 0;
+
+    for msg in messages {
+        if let Some(content) = msg.get("content") {
+            if let Some(s) = content.as_str() {
+                size += s.len() as u64;
+            } else if let Some(arr) = content.as_array() {
+                for block in arr {
+                    if let Some(text) = block.get("text").and_then(|t| t.as_str()) {
+                        size += text.len() as u64;
+                    }
+                    if let Some(data) = block.pointer("/source/data").and_then(|d| d.as_str()) {
+                        size += data.len() as u64;
+                    }
+                    if let Some(url) = block.pointer("/image_url/url").and_then(|u| u.as_str()) {
+                        if url.starts_with("data:") {
+                            size += url.len() as u64;
+                        }
+                    }
+                }
+            }
+        }
+    }
+
+    size
+}