brevit 0.1.4 → 1.0.0

package/README.md

@@ -1,24 +1,24 @@
-# Brevit.js
+# brevit
 
 A high-performance JavaScript library for semantically compressing and optimizing data before sending it to a Large Language Model (LLM). Dramatically reduce token costs while maintaining data integrity and readability.
 
 ## Table of Contents
 
-- [Why Brevit.js?](#why-brevitjs)
+- [Why brevit?](#why-brevit)
 - [Key Features](#key-features)
-- [When Not to Use Brevit.js](#when-not-to-use-brevitjs)
+- [When Not to Use brevit](#when-not-to-use-brevit)
 - [Benchmarks](#benchmarks)
 - [Installation & Quick Start](#installation--quick-start)
 - [Playgrounds](#playgrounds)
 - [CLI](#cli)
 - [Format Overview](#format-overview)
 - [API](#api)
-- [Using Brevit.js in LLM Prompts](#using-brevitjs-in-llm-prompts)
+- [Using brevit in LLM Prompts](#using-brevit-in-llm-prompts)
 - [Syntax Cheatsheet](#syntax-cheatsheet)
 - [Other Implementations](#other-implementations)
 - [Full Specification](#full-specification)
 
-## Why Brevit.js?
+## Why brevit?
 
 ### JavaScript-Specific Advantages
 
@@ -53,7 +53,7 @@ const explicit = await brevit.optimize(complexOrder);
 
 ### Automatic Strategy Selection
 
-Brevit.js now includes the `.brevity()` method that automatically analyzes your data and selects the optimal optimization strategy:
+brevit now includes the `.brevity()` method that automatically analyzes your data and selects the optimal optimization strategy:
 
 ```javascript
 const data = {
@@ -72,7 +72,7 @@ const optimized = await brevit.brevity(data);
 ## Key Features
 
 - **JSON Optimization**: Flatten nested JSON structures into token-efficient key-value pairs
-- **Text Optimization**: Clean and summarize long text documents
+- **Text Optimization**: Deterministic TextRank-based compression for plain text (no LLM required)
 - **Image Optimization**: Extract text from images via OCR
 - **Lightweight**: Zero dependencies (optional YAML support)
 - **Universal**: Works in Node.js, browsers, and modern JavaScript environments
@@ -110,7 +110,7 @@ pnpm add brevit
 
 ### TypeScript Support
 
-Brevit.js includes full TypeScript definitions. Simply import and use with full type safety:
+brevit includes full TypeScript definitions. Simply import and use with full type safety:
 
 ```typescript
 import {
@@ -130,7 +130,7 @@ const client = new BrevitClient(new BrevitConfig(config));
 
 ## Complete Usage Examples
 
-Brevit.js supports three main data types: **JSON objects/strings**, **text files/strings**, and **images**. Here's how to use each:
+brevit supports three main data types: **JSON objects/strings**, **text files/strings**, and **images**. Here's how to use each:
 
 ### 1. JSON Optimization Examples
 
@@ -219,20 +219,6 @@ const optimized = await brevit.brevity(jsonString);
 // @o.status:SHIPPED
 ```
 
-#### Example 1.2a: Abbreviations Disabled
-
-```javascript
-const brevitNoAbbr = new BrevitClient(new BrevitConfig({ 
-  jsonMode: JsonOptimizationMode.Flatten,
-  enableAbbreviations: false  // Disable abbreviations
-}));
-
-const jsonString = '{"order": {"id": "o-456", "status": "SHIPPED"}}';
-const optimized = await brevitNoAbbr.brevity(jsonString);
-// Output (without abbreviations):
-// order.id:o-456
-// order.status:SHIPPED
-```
 
 #### Example 1.3: Complex Nested JSON with Arrays
 
@@ -276,51 +262,6 @@ const optimized = await brevit.brevity(complexData);
 // luis,9.2,540,2,Ridge Overlook,false
 ```
 
-#### Example 1.3a: Complex Data with Abbreviations Disabled
-
-```javascript
-const brevitNoAbbr = new BrevitClient(new BrevitConfig({ 
-  jsonMode: JsonOptimizationMode.Flatten,
-  enableAbbreviations: false  // Disable abbreviations
-}));
-
-const complexData = {
-  context: {
-    task: "Our favorite hikes together",
-    location: "Boulder",
-    season: "spring_2025"
-  },
-  friends: ["ana", "luis", "sam"],
-  hikes: [
-    {
-      id: 1,
-      name: "Blue Lake Trail",
-      distanceKm: 7.5,
-      elevationGain: 320,
-      companion: "ana",
-      wasSunny: true
-    },
-    {
-      id: 2,
-      name: "Ridge Overlook",
-      distanceKm: 9.2,
-      elevationGain: 540,
-      companion: "luis",
-      wasSunny: false
-    }
-  ]
-};
-
-const optimized = await brevitNoAbbr.brevity(complexData);
-// Output (without abbreviations):
-// context.task:Our favorite hikes together
-// context.location:Boulder
-// context.season:spring_2025
-// friends[3]:ana,luis,sam
-// hikes[2]{companion,distanceKm,elevationGain,id,name,wasSunny}:
-// ana,7.5,320,1,Blue Lake Trail,true
-// luis,9.2,540,2,Ridge Overlook,false
-```
 
 #### Example 1.4: Different JSON Optimization Modes
 
@@ -357,16 +298,23 @@ The text goes on for many lines...
 [Repeated content many times]
 `.repeat(50);
 
-// Automatic detection: If text exceeds threshold, applies text optimization
+// Automatic detection: plain text is compressed by default
 const optimized = await brevit.brevity(longText);
 
-// Explicit text optimization
+// Explicit text compression via the main pipeline (ratio optional; defaults to 0.0 = auto)
 const config = new BrevitConfig({ 
   textMode: TextOptimizationMode.Clean,
-  longTextThreshold: 500  // Characters threshold
+  longTextThreshold: 500  // (JSON heuristics only; plain text is compressed regardless)
 });
 const brevitWithText = new BrevitClient(config);
-const cleaned = await brevitWithText.optimize(longText);
+const cleanedAuto = await brevitWithText.optimize(longText);       // auto
+const cleaned60 = await brevitWithText.optimize(longText, 0.6);    // ratio
+const cleanedIntent = await brevitWithText.optimize(longText, 0.6, "keep key details"); // ratio + intent (3rd arg)
+
+// Explicit TextRank compression APIs (recommended when you want direct control)
+const compressedAuto = await brevit.compressText(longText);        // AUTO mode
+const compressed60 = await brevit.optimizeText(longText, 0.6);     // Keep ~60% of sentences
+const compressedDefault = await brevit.optimizeText(longText, 0.0); // Same as compressText()
 ```
 
 #### Example 2.2: Reading Text from File (Node.js)
@@ -388,19 +336,19 @@ const optimized = await brevit.brevity(textContent);
 const cleanConfig = new BrevitConfig({ 
   textMode: TextOptimizationMode.Clean 
 });
-// Removes signatures, headers, repetitive content
+// Built-in deterministic TextRank extractive compression (no LLM required)
 
 // Summarize Fast
 const fastConfig = new BrevitConfig({ 
   textMode: TextOptimizationMode.SummarizeFast 
 });
-// Fast summarization (requires custom text optimizer implementation)
+// Reserved for custom LLM summarization (or use built-in TextRank via compressText/optimizeText)
 
 // Summarize High Quality
 const qualityConfig = new BrevitConfig({ 
   textMode: TextOptimizationMode.SummarizeHighQuality 
 });
-// High-quality summarization (requires custom text optimizer with LLM integration)
+// Reserved for custom LLM summarization (or use built-in TextRank via compressText/optimizeText)
 ```
 
 ### 3. Image Optimization Examples
@@ -643,7 +591,7 @@ processOrder(order).then(console.log);
 <!DOCTYPE html>
 <html>
 <head>
-  <title>Brevit.js Example</title>
+  <title>brevit Example</title>
 </head>
 <body>
   <script type="module">
@@ -880,14 +828,16 @@ const user = {
 };
 
 const optimized = await brevit.optimize(user);
-// Output:
-// id: u-123
-// name: Javian
-// isActive: true
-// contact.email: support@javianpicardo.com
-// contact.phone: null
-// orders[0].orderId: o-456
-// orders[0].status: SHIPPED
+// Output (with abbreviations enabled by default):
+// @c=contact
+// @o=orders
+// id:u-123
+// name:Javian
+// isActive:true
+// @c.email:support@javianpicardo.com
+// @c.phone:null
+// @o[0].orderId:o-456
+// @o[0].status:SHIPPED
 ```
 
 ### Example 2: Optimize JSON String
@@ -911,7 +861,7 @@ const optimized = await brevit.optimize(json);
 ```javascript
 const longDocument = '...very long text...';
 const optimized = await brevit.optimize(longDocument);
-// Will trigger text optimization if length > longTextThreshold
+// Plain text is compressed by default; use optimize(longDocument, 0.6) for ratio compression
 ```
 
 ### Example 4: Process Image (ArrayBuffer)
@@ -925,7 +875,7 @@ const optimized = await brevit.optimize(imageData);
 // Will trigger image optimization
 ```
 
-## When Not to Use Brevit.js
+## When Not to Use brevit
 
 Consider alternatives when:
 
@@ -1199,7 +1149,7 @@ class BrevitConfig {
 - `Ocr` - Extract text via OCR
 - `Metadata` - Extract metadata only
 
-## Using Brevit.js in LLM Prompts
+## Using brevit in LLM Prompts
 
 ### Best Practices
 

package/TYPESCRIPT.md

@@ -7,7 +7,7 @@ Brevit.js includes comprehensive TypeScript definitions for full type safety and
 No additional installation required! TypeScript definitions are included in the package.
 
 ```bash
-npm install brevit-js
+npm install brevit
 ```
 
 ## Basic Usage
@@ -17,7 +17,7 @@ import {
   BrevitClient,
   BrevitConfig,
   JsonOptimizationMode,
-} from 'brevit-js';
+} from 'brevit';
 
 const config = new BrevitConfig({
   jsonMode: JsonOptimizationMode.Flatten,
@@ -38,7 +38,7 @@ import {
   JsonOptimizationMode,
   TextOptimizationMode,
   ImageOptimizationMode,
-} from 'brevit-js';
+} from 'brevit';
 
 // Usage
 const mode: typeof JsonOptimizationMode.Flatten = JsonOptimizationMode.Flatten;
@@ -51,7 +51,7 @@ import type {
   JsonOptimizationModeType,
   TextOptimizationModeType,
   ImageOptimizationModeType,
-} from 'brevit-js';
+} from 'brevit';
 
 function setMode(mode: JsonOptimizationModeType) {
   // Type-safe mode setting
@@ -66,7 +66,7 @@ import type {
   BrevitClientOptions,
   TextOptimizerFunction,
   ImageOptimizerFunction,
-} from 'brevit-js';
+} from 'brevit';
 
 // Configuration options
 const config: BrevitConfigOptions = {
@@ -91,14 +91,14 @@ import {
   BrevitConfig,
   JsonOptimizationMode,
   type BrevitConfigOptions,
-} from 'brevit-js';
+} from 'brevit';
 
 const configOptions: BrevitConfigOptions = {
   jsonMode: JsonOptimizationMode.Flatten,
   textMode: 'Clean',
   imageMode: 'Ocr',
   jsonPathsToKeep: ['user.name', 'order.orderId'],
-  longTextThreshold: 1000,
+  longTextThreshold: 1000, // (plain text is compressed regardless; this is mostly for JSON heuristics)
 };
 
 const config = new BrevitConfig(configOptions);
@@ -113,7 +113,7 @@ import {
   BrevitConfig,
   type TextOptimizerFunction,
   type ImageOptimizerFunction,
-} from 'brevit-js';
+} from 'brevit';
 
 const customTextOptimizer: TextOptimizerFunction = async (longText, intent) => {
   const response = await fetch('/api/summarize', {
@@ -140,6 +140,13 @@ const client = new BrevitClient(new BrevitConfig(), {
   textOptimizer: customTextOptimizer,
   imageOptimizer: customImageOptimizer,
 });
+
+// Text compression (TextRank) is built-in:
+const text = 'Alpha sentence. Beta sentence. Gamma sentence.';
+const compressedAuto = await client.brevity(text);        // auto compression
+const compressedAuto2 = await client.optimize(text);      // auto compression (ratio defaults to 0.0)
+const compressed60 = await client.optimize(text, 0.6);    // ratio compression
+const compressed60WithIntent = await client.optimize(text, 0.6, 'keep key details'); // ratio + intent
 ```
 
 ### Example 3: Type-Safe Data Structures
@@ -184,7 +191,7 @@ const optimizedOrder = await client.optimize(order);
 ### Example 4: Generic Helper Function
 
 ```typescript
-import { BrevitClient, BrevitConfig } from 'brevit-js';
+import { BrevitClient, BrevitConfig } from 'brevit';
 
 async function optimizeData<T>(data: T): Promise<string> {
   const client = new BrevitClient();

package/example.ts

@@ -37,7 +37,7 @@ async function example2() {
     jsonMode: JsonOptimizationMode.Flatten,
     textMode: TextOptimizationMode.Clean,
     imageMode: ImageOptimizationMode.Ocr,
-    longTextThreshold: 1000,
+    longTextThreshold: 1000, // (plain text is compressed regardless; this is mostly for JSON heuristics)
   };
 
   const client = new BrevitClient(new BrevitConfig(config));
@@ -70,7 +70,11 @@ async function example3() {
   });
 
   const longText = '...very long text...';
+  // For text, optimize() defaults to deterministic TextRank compression unless you provide a custom text optimizer.
+  // Ratio compression is supported via optimize(longText, ratio, intent?).
   const optimized = await client.optimize(longText);
+  const optimized60 = await client.optimize(longText, 0.6);
+  const optimized60WithIntent = await client.optimize(longText, 0.6, 'keep key details');
   console.log(optimized);
 }
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "brevit",
-  "version": "0.1.4",
-  "description": "A high-performance JavaScript library for semantically compressing and optimizing data before sending it to a Large Language Model (LLM).",
+  "version": "1.0.0",
+  "description": "A high-performance JavaScript library for optimizing LLM prompt inputs: token-efficient JSON flattening + deterministic TextRank-based text compression.",
   "main": "src/brevit.js",
   "types": "src/brevit.d.ts",
   "type": "module",
@@ -25,6 +25,9 @@
     "type": "git",
     "url": "https://github.com/JavianDev/Brevit.js.git"
   },
+  "dependencies": {
+    "compromise": "^14.14.4"
+  },
   "optionalDependencies": {
     "js-yaml": "^4.1.0"
   }

package/src/brevit.d.ts

@@ -221,7 +221,34 @@ export class BrevitClient {
    * // Returns OCR text or metadata
    * ```
    */
-  optimize(rawData: unknown, intent?: string | null): Promise<string>;
+  /**
+   * Optimizes any supported input type.
+   * For plain text inputs, this performs deterministic TextRank compression by default.
+   *
+   * - optimize(text) => auto compression (ratio defaults to 0.0)
+   * - optimize(text, 0.6) => ratio compression
+   * - optimize(text, 0.6, intent) => ratio compression with intent hint
+   * - optimize(obj, intent) => JSON/object pipeline with intent hint
+   */
+  optimize(rawData: unknown, ratioOrIntent?: number | string | null, intent?: string | null): Promise<string>;
+
+  /**
+   * Intelligently optimizes data by automatically selecting the best strategy.
+   * For plain text inputs, this performs deterministic TextRank compression by default.
+   */
+  brevity(rawData: unknown, intent?: string | null): Promise<string>;
+
+  /**
+   * Explicit text compression (AUTO mode).
+   * Always attempts to compress, even for short / single-sentence inputs.
+   */
+  compressText(text: string): Promise<string>;
+
+  /**
+   * Explicit text compression (RATIO mode).
+   * If ratio <= 0, behaves like compressText().
+   */
+  optimizeText(text: string, ratio?: number): Promise<string>;
 }
 
 // Re-export types for convenience

package/src/brevit.js

@@ -7,7 +7,7 @@
  *
  * Project: Brevit
  * Author: Javian
- * Version: 0.1.0
+ * Version: 1.0.0
  * =================================================================================
  */
 
@@ -80,6 +80,36 @@ export class BrevitClient {
     this._config = config;
     this._textOptimizer = options.textOptimizer || this._defaultTextOptimizer.bind(this);
     this._imageOptimizer = options.imageOptimizer || this._defaultImageOptimizer.bind(this);
+    this._semanticCompressor = null;
+  }
+
+  /**
+   * Explicit text compression (AUTO mode).
+   * Always attempts to compress, even for short / single-sentence inputs.
+   * @param {string} text
+   * @returns {Promise<string>}
+   */
+  async compressText(text) {
+    if (!this._semanticCompressor) {
+      const { SemanticCompressor } = await import('./semanticCompressor.js');
+      this._semanticCompressor = new SemanticCompressor();
+    }
+    return this._semanticCompressor.compress(String(text ?? ''));
+  }
+
+  /**
+   * Explicit text compression (RATIO mode).
+   * If ratio <= 0, behaves like compressText().
+   * @param {string} text
+   * @param {number} ratio
+   * @returns {Promise<string>}
+   */
+  async optimizeText(text, ratio = 0.0) {
+    if (!this._semanticCompressor) {
+      const { SemanticCompressor } = await import('./semanticCompressor.js');
+      this._semanticCompressor = new SemanticCompressor();
+    }
+    return this._semanticCompressor.optimize(String(text ?? ''), Number(ratio ?? 0));
   }
 
   /**
@@ -592,24 +622,12 @@ export class BrevitClient {
             (trimmed.startsWith('[') && trimmed.endsWith(']'))) {
           inputObject = JSON.parse(rawData);
         } else {
-          // It's plain text - analyze and optimize
-          const analysis = this._analyzeDataStructure(rawData);
-          const strategy = this._selectOptimalStrategy(analysis);
-          
-          if (strategy.name === 'TextOptimization') {
-            return await this._textOptimizer(rawData, intent);
-          }
-          return rawData;
+          // Plain text: always compress via TextRank (auto).
+          return await this.compressText(rawData);
         }
       } catch (e) {
-        // Not JSON - treat as text
-        const analysis = this._analyzeDataStructure(rawData);
-        const strategy = this._selectOptimalStrategy(analysis);
-        
-        if (strategy.name === 'TextOptimization') {
-          return await this._textOptimizer(rawData, intent);
-        }
-        return rawData;
+        // Not valid JSON - treat as plain text and compress.
+        return await this.compressText(rawData);
       }
     } else if (inputType === 'object' && rawData !== null) {
       // Check if it's image data
@@ -671,12 +689,25 @@ export class BrevitClient {
    * or text into a token-efficient string.
    *
    * @param {any} rawData - The data to optimize (object, JSON string, text, ArrayBuffer).
-   * @param {string} [intent] - (Optional) A hint about the user's goal.
+   * @param {number|string|null} [ratioOrIntent] - If number: sentence ratio for TextRank compression (0..1). If string: intent.
+   * @param {string|null} [intent] - (Optional) A hint about the user's goal (use this as 3rd arg when passing ratio).
    * @returns {Promise<string>} A promise that resolves to the optimized string.
    */
-  async optimize(rawData, intent = null) {
+  async optimize(rawData, ratioOrIntent = null, intent = null) {
     let inputObject = null;
     let inputType = typeof rawData;
+    let ratio = 0.0;
+    let resolvedIntent = intent;
+
+    // Backwards-compatible argument parsing:
+    // - optimize(text, 0.6, intent?) => ratio-based text compression
+    // - optimize(text, intent?) => auto text compression (ratio defaults to 0.0)
+    // - optimize(obj, intent?) => JSON/object pipeline
+    if (typeof ratioOrIntent === 'number' && Number.isFinite(ratioOrIntent)) {
+      ratio = ratioOrIntent;
+    } else if (resolvedIntent == null && typeof ratioOrIntent === 'string') {
+      resolvedIntent = ratioOrIntent;
+    }
 
     if (inputType === 'string') {
       // Could be JSON string or just text
@@ -691,20 +722,15 @@ export class BrevitClient {
       }
 
       if (!inputObject) {
-        // It's text
-        if (rawData.length > this._config.longTextThreshold) {
-          // It's long text, apply text optimization
-          return await this._textOptimizer(rawData, intent);
-        }
-        // It's short text, return as-is
-        return rawData;
+        // Plain text: always compress via TextRank.
+        return await this.optimizeText(rawData, ratio);
       }
     } else if (inputType === 'object' && rawData !== null) {
       // Check if it's an ArrayBuffer or TypedArray (image data)
       if (rawData instanceof ArrayBuffer || 
           rawData instanceof Uint8Array ||
           (rawData.constructor && rawData.constructor.name === 'Buffer')) {
-        return await this._imageOptimizer(rawData, intent);
+        return await this._imageOptimizer(rawData, resolvedIntent);
       }
       // It's a plain JS object
       inputObject = rawData;
@@ -745,12 +771,13 @@ export class BrevitClient {
    * @private
    */
   async _defaultTextOptimizer(longText, intent) {
-    // STUB: A real frontend app would call its backend for this.
-    // NEVER put LLM API keys in a frontend app.
-    console.warn('[Brevit] Text summarization should be done on a secure backend.');
-    const mode = this._config.textMode;
-    const stubSummary = longText.substring(0, 150);
-    return `[${mode} Stub: Summary of text follows...]\n${stubSummary}...\n[End of summary]`;
+    if (this._config.textMode === TextOptimizationMode.None) {
+      return String(longText ?? '');
+    }
+
+    // Built-in deterministic extractive compression (TextRank).
+    // If callers want LLM summarization, they can pass a custom textOptimizer.
+    return await this.compressText(String(longText ?? ''));
   }
 
   /**

package/src/semanticCompressor.js

@@ -0,0 +1,157 @@
+import nlp from 'compromise';
+
+/**
+ * Deterministic extractive semantic compressor using a TextRank-style graph over sentences.
+ */
+export class SemanticCompressor {
+  constructor(options = {}) {
+    const {
+      stopWords,
+      damping = 0.85,
+      iterations = 20,
+      autoThresholdMultiplier = 0.9,
+    } = options;
+
+    this._stopWords = new Set(
+      stopWords || [
+        'the', 'is', 'in', 'at', 'of', 'on', 'and', 'a', 'to', 'it', 'for',
+        'with', 'as', 'by', 'this', 'that', 'are', 'was', 'be', 'or', 'an',
+        'if', 'not', 'but', 'from', 'they', 'we', 'he', 'she', 'which',
+      ],
+    );
+
+    this._damping = damping;
+    this._iterations = iterations;
+    this._autoThresholdMultiplier = autoThresholdMultiplier;
+  }
+
+  /**
+   * AUTO MODE: Keep sentences with above-average importance (threshold = mean * multiplier).
+   */
+  compress(text) {
+    return this._runTextRank(text, 'auto');
+  }
+
+  /**
+   * MANUAL MODE: Keep top-ranked sentences by ratio.
+   * If ratio <= 0, behaves like `compress`.
+   */
+  optimize(text, ratio = 0.0) {
+    return this._runTextRank(text, 'ratio', ratio);
+  }
+
+  _runTextRank(text, mode, ratioValue = 0.0) {
+    if (text == null) return '';
+    const str = String(text);
+
+    const rawSentences = this._splitSentences(str);
+    if (rawSentences.length === 0) return str;
+
+    // 1) Extract features
+    const nodes = rawSentences.map((sent, index) => {
+      const terms = nlp(sent)
+        .nouns()
+        .out('array')
+        .map((t) => String(t).toLowerCase().trim())
+        .filter((t) => t.length > 2 && !this._stopWords.has(t));
+
+      return {
+        id: index,
+        text: sent,
+        terms: new Set(terms),
+        score: 1.0,
+      };
+    });
+
+    // 2) Build graph (adjacency list) using similarity > 0 as an edge.
+    const edges = Array.from({ length: nodes.length }, () => []);
+    for (let i = 0; i < nodes.length; i++) {
+      for (let j = i + 1; j < nodes.length; j++) {
+        const sim = this._calculateSimilarity(nodes[i].terms, nodes[j].terms);
+        if (sim > 0) {
+          edges[i].push(j);
+          edges[j].push(i);
+        }
+      }
+    }
+
+    // 3) Iterate (PageRank-style)
+    const base = 1 - this._damping;
+    for (let iter = 0; iter < this._iterations; iter++) {
+      const newScores = nodes.map((n) => n.score);
+      for (let i = 0; i < nodes.length; i++) {
+        let sum = 0;
+        for (const neighborIdx of edges[i]) {
+          sum += nodes[neighborIdx].score / (edges[neighborIdx].length || 1);
+        }
+        newScores[i] = base + this._damping * sum;
+      }
+      nodes.forEach((n, i) => {
+        n.score = newScores[i];
+      });
+    }
+
+    // 4) Selection strategy
+    const keptIndices = new Set();
+
+    if (mode === 'auto' || ratioValue <= 0) {
+      const totalScore = nodes.reduce((sum, n) => sum + n.score, 0);
+      const avgScore = totalScore / (nodes.length || 1);
+      const threshold = avgScore * this._autoThresholdMultiplier;
+
+      nodes.forEach((n) => {
+        if (n.score >= threshold) keptIndices.add(n.id);
+      });
+
+      if (keptIndices.size === 0 && nodes.length > 0) {
+        const topNode = nodes.reduce((prev, current) =>
+          prev.score > current.score ? prev : current,
+        );
+        keptIndices.add(topNode.id);
+      }
+    } else {
+      if (ratioValue >= 1) {
+        nodes.forEach((n) => keptIndices.add(n.id));
+      } else {
+        const sorted = [...nodes].sort((a, b) => b.score - a.score);
+        const count = Math.max(1, Math.floor(nodes.length * ratioValue));
+        sorted.slice(0, count).forEach((n) => keptIndices.add(n.id));
+      }
+    }
+
+    // 5) Reconstruct in original order
+    return nodes
+      .filter((n) => keptIndices.has(n.id))
+      .map((n) => n.text)
+      .join(' ');
+  }
+
+  _splitSentences(text) {
+    try {
+      const doc = nlp(text);
+      const arr = doc.sentences().out('array');
+      if (Array.isArray(arr) && arr.length > 0) {
+        return arr.map((s) => String(s).trim()).filter(Boolean);
+      }
+    } catch {
+      // fall back
+    }
+
+    // Conservative fallback split
+    return String(text)
+      .split(/(?<=[.!?])\s+/)
+      .map((s) => s.trim())
+      .filter(Boolean);
+  }
+
+  _calculateSimilarity(setA, setB) {
+    if (!setA || !setB || setA.size === 0 || setB.size === 0) return 0;
+    let intersection = 0;
+    for (const elem of setA) if (setB.has(elem)) intersection++;
+    if (intersection === 0) return 0;
+    const denom = Math.log(setA.size) + Math.log(setB.size);
+    return intersection / (denom || 1);
+  }
+}
+
+

package/test/test.js

@@ -6,9 +6,9 @@ async function runTests() {
   let passed = 0;
   let failed = 0;
 
-  function test(name, fn) {
+  async function test(name, fn) {
     try {
-      fn();
+      await fn();
       console.log(`✓ ${name}`);
       passed++;
     } catch (error) {
@@ -18,7 +18,7 @@ async function runTests() {
   }
 
   // Test 1: Flatten JSON object
-  test('Flatten JSON object', async () => {
+  await test('Flatten JSON object', async () => {
     const config = new BrevitConfig({ jsonMode: JsonOptimizationMode.Flatten });
     const brevit = new BrevitClient(config);
     
@@ -30,26 +30,26 @@ async function runTests() {
     };
 
     const result = await brevit.optimize(testObject);
-    if (!result.includes('user.name: Javian') || !result.includes('user.email: support@javianpicardo.com')) {
+    if (!result.includes('user.name:Javian') || !result.includes('user.email:support@javianpicardo.com')) {
       throw new Error('Flattened output does not contain expected values');
     }
   });
 
   // Test 2: Flatten JSON string
-  test('Flatten JSON string', async () => {
+  await test('Flatten JSON string', async () => {
     const config = new BrevitConfig({ jsonMode: JsonOptimizationMode.Flatten });
     const brevit = new BrevitClient(config);
     
     const jsonString = '{"order": {"orderId": "o-456", "status": "SHIPPED"}}';
     const result = await brevit.optimize(jsonString);
     
-    if (!result.includes('order.orderId: o-456') || !result.includes('order.status: SHIPPED')) {
+    if (!result.includes('order.orderId:o-456') || !result.includes('order.status:SHIPPED')) {
       throw new Error('Flattened output does not contain expected values');
     }
   });
 
   // Test 3: Short text returns as-is
-  test('Short text returns as-is', async () => {
+  await test('Short text returns as-is', async () => {
     const config = new BrevitConfig({ longTextThreshold: 500 });
     const brevit = new BrevitClient(config);
     
@@ -61,8 +61,59 @@ async function runTests() {
     }
   });
 
-  // Test 4: Array handling
-  test('Array handling', async () => {
+  // Test 4: compressText always attempts compression
+  await test('compressText returns a string and is deterministic', async () => {
+    const brevit = new BrevitClient();
+    const text = 'Alpha sentence about cats. Beta sentence about cats. Gamma unrelated sentence.';
+
+    const r1 = await brevit.compressText(text);
+    const r2 = await brevit.compressText(text);
+    if (typeof r1 !== 'string' || r1.length === 0) {
+      throw new Error('compressText did not return a non-empty string');
+    }
+    if (r1 !== r2) {
+      throw new Error('compressText should be deterministic');
+    }
+  });
+
+  // Test 5: optimizeText ratio=0 behaves like compressText
+  await test('optimizeText ratio=0 behaves like compressText', async () => {
+    const brevit = new BrevitClient();
+    const text = 'One sentence about hiking. Another sentence about hiking. A third sentence about coffee.';
+
+    const auto = await brevit.compressText(text);
+    const zero = await brevit.optimizeText(text, 0.0);
+    if (auto !== zero) {
+      throw new Error('optimizeText(text, 0.0) should equal compressText(text)');
+    }
+  });
+
+  // Test 6: optimize(text) defaults to auto compression
+  await test('optimize(text) defaults to auto compression', async () => {
+    const brevit = new BrevitClient();
+    const text = 'One sentence about hiking. Another sentence about hiking. A third sentence about coffee.';
+
+    const auto = await brevit.compressText(text);
+    const result = await brevit.optimize(text);
+    if (result !== auto) {
+      throw new Error('optimize(text) should behave like compressText(text)');
+    }
+  });
+
+  // Test 7: optimize(text, ratio) routes to optimizeText
+  await test('optimize(text, ratio) routes to optimizeText', async () => {
+    const brevit = new BrevitClient();
+    const text = 'Alpha cats. Beta cats. Gamma coffee. Delta cats.';
+
+    const direct = await brevit.optimizeText(text, 0.6);
+    const routed = await brevit.optimize(text, 0.6);
+    if (direct !== routed) {
+      throw new Error('optimize(text, ratio) should equal optimizeText(text, ratio)');
+    }
+  });
+
+  // Test 8: Array handling
+  await test('Array handling', async () => {
     const config = new BrevitConfig({ jsonMode: JsonOptimizationMode.Flatten });
     const brevit = new BrevitClient(config);
     
@@ -74,8 +125,12 @@ async function runTests() {
     };
 
     const result = await brevit.optimize(testObject);
-    if (!result.includes('items[0].sku: A-88') || !result.includes('items[1].sku: T-22')) {
-      throw new Error('Array flattening failed');
+    // Expect tabular optimization for uniform object arrays
+    if (!result.includes('items[2]{sku,name}:')) {
+      throw new Error('Tabular array header missing');
+    }
+    if (!result.includes('A-88,Brevit Pro') || !result.includes('T-22,Toon Handbook')) {
+      throw new Error('Tabular array rows missing');
     }
   });