toon-formatter 2.2.0 → 2.3.0

package/CHANGELOG.md

@@ -4,6 +4,25 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [2.3.0] - 2026-01-27
+
+### Added
+- **🧠 Smart Code Optimization**
+  - New preprocessing pipeline to maximize token efficiency for LLM contexts
+  - **Code Detection & Reduction**: Automatic identification, comment stripping, and whitespace compression for scripts and commands
+  - **Safe Expensive Words Replacement**: Case-insensitive abbreviation of 100+ verbose phrases in natural language text
+  - **Security & Integrity**: Replacement logic strictly avoids altering code blocks or data (JSON/XML/CSV) to prevent syntactic issues
+  - **Seamless Integration**: Automatically active for all mixed-text conversion methods (`jsonToToon`, `xmlToToon`, `csvToToon`, etc.)
+- **🧪 Expanded Test Infrastructure**
+  - Dedicated `code-optimization.test.js` for unit verification
+  - New `optimization-integration.test.js` for high-level mixed-content validation
+  - Total test suite expanded to 189 tests with 100% pass rate
+
+### Changed
+- Refined `isCode` heuristics to support common Node.js and shell execution patterns
+- Optimized asynchronous pipeline in `dataManagerAsync` for better performance
+- Improved mixed-text extraction robustness in `utils.js`
+
 
 ## [2.1.1] - 2025-12-18
 

package/README.md

@@ -61,7 +61,7 @@ users[3]{id,name,active}:
 
 ## 💻 CLI Utility
 
-**NEW in v2.1.0**: TOON Formatter now includes a powerful CLI utility for fast data conversion directly from your terminal!
+**NEW in v2.2.0**: TOON Formatter now includes a powerful CLI utility for fast data conversion directly from your terminal!
 
 ### Global Installation
 To use the `toon-formatter` command anywhere:
@@ -305,6 +305,64 @@ All conversion functions are available in both **synchronous** and **asynchronou
 
 ---
 
+## 🧠 Smart Code Optimization
+
+**NEW in v2.3.0**: The TOON Converter now features **Smart Code Optimization**, a preprocessing pipeline designed to maximize token efficiency when dealing with mixed text and code blocks.
+
+### Overview
+
+LLMs often process documentation or chat history that contains a mix of natural language, code snippets, and data (JSON/XML/CSV). Smart Code Optimization automatically cleans and compresses this content before conversion, resulting in even lower token counts.
+
+### Core Components
+
+#### 1. Code Detection & Reduction
+The library uses heuristics to detect code blocks (npm/git commands, shebangs, common programming patterns). Detected blocks are:
+- **Comment Stripped**: Single-line comments (`#` or `//`) are removed.
+- **Whitespace Compressed**: Multiple newlines are collapsed into single breaks.
+- **Preserved**: The semantic structure of the code is maintained while removing "token noise".
+
+#### 2. Expensive Words Replacement (Safe)
+The library contains a dictionary of **100+ verbose phrases** (Contracted/Common/Technical) and automatically replaces them with token-efficient abbreviations. 
+
+> [!IMPORTANT]
+> To maintain syntactic integrity, these replacements are **ONLY applied to natural language text**. Both **code blocks and data blocks (JSON/XML/CSV/TOON) are strictly preserved** and never modified by this process.
+
+#### 3. Targeted Data Extraction
+When using `jsonToToon`, `xmlToToon`, or `csvToToon`, the library:
+1. Extracts valid data blocks and code snippets from the text.
+2. Applies phrase replacements **only** to the remaining natural language text.
+3. Re-inserts the converted data and reduced code blocks into the resulting string.
+
+### How it works (Example)
+
+**Input Mixed Text:**
+```
+Please review this large language model configuration as soon as possible:
+{"model": "gpt-4", "temp": 0.7}
+
+npm install openai // install the helper library
+```
+
+**Optimized TOON Output:**
+```
+pls review this llm configuration asap:
+model: "gpt-4"
+temp: 0.7
+
+npm install openai
+```
+
+### Supported Formats
+
+Smart Code Optimization is active by default for all methods starting with `jsonTo`, `xmlTo`, and `csvTo`:
+
+- ✅ `jsonToToon()`, `jsonToYaml()`, `jsonToXml()`...
+- ✅ `xmlToToon()`, `xmlToJson()`, `xmlToCsv()`...
+- ✅ `csvToToon()`, `csvToJson()`, `csvToXml()`...
+- ❌ `yamlToToon()`, `toonToJson()` (These remain **Pure Data** only)
+
+---
+
 ## 🎯 Mixed Text Support
 
 ### What is Mixed Text?

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "toon-formatter",
-    "version": "2.2.0",
+    "version": "2.3.0",
     "funding": {
         "type": "github",
         "url": "https://github.com/sponsors/ankitpal181"
@@ -72,4 +72,4 @@
         "./xml-converter": "./src/xml_formatter/index.js",
         "./csv-converter": "./src/csv_formatter/index.js"
     }
-}
+}

package/src/constants.js

@@ -0,0 +1,142 @@
+/**
+ * TOON Converter Constants
+ * 
+ * Contains configuration values and optimization dictionaries.
+ */
+
+/**
+ * EXPENSIVE_WORDS Dictionary
+ * 
+ * Maps verbose phrases to their token-efficient abbreviations.
+ * Used by Smart Code Optimization to reduce token count in text.
+ * 
+ * These replacements are ONLY applied to non-code text to preserve
+ * code syntax integrity.
+ */
+export const EXPENSIVE_WORDS = {
+    // Contractions
+    "i am": "i'm",
+    "do not": "don't",
+    "would not": "wouldn't",
+    "i will": "i'll",
+    "i have": "i've",
+    "you are": "you're",
+    "we will": "we'll",
+    "they are": "they're",
+    "it is": "it's",
+    "could have": "could've",
+    "should not": "shouldn't",
+    "has not": "hasn't",
+    "there is": "there's",
+
+    // Abbreviations - Common Phrases
+    "as soon as possible": "asap",
+    "frequently asked questions": "faq",
+    "user interface": "ui",
+    "to be determined": "tbd",
+    "by the way": "btw",
+    "artificial intelligence": "ai",
+    "large language model": "llm",
+    "point of view": "pov",
+    "for your information": "fyi",
+    "estimated time of arrival": "eta",
+    "end of day": "eod",
+
+    // Informal Contractions
+    "going to": "gonna",
+    "want to": "wanna",
+    "got to": "gotta",
+    "give me": "gimme",
+    "kind of": "kinda",
+    "don't know": "dunno",
+    "thank you": "thanks",
+
+    // Abbreviations - Titles/Places
+    "doctor": "dr.",
+    "street": "st.",
+    "avenue": "ave.",
+    "incorporated": "inc.",
+
+    // Abbreviations - General
+    "utility": "util",
+    "laboratory": "lab",
+    "photograph": "photo",
+    "refrigerator": "fridge",
+    "vegetables": "veggies",
+    "mathematics": "math",
+    "gymnasium": "gym",
+    "advertisement": "ad",
+    "vegetarian": "veg",
+
+    // Synonym Simplifications
+    "approximate": "approx",
+    "regarding": "re:",
+    "information": "info",
+    "amount": "amt",
+    "reference": "ref",
+    "management": "mgmt",
+    "number": "no.",
+    "because": "as",
+    "without": "w/o",
+    "before": "b4",
+    "between": "btw",
+    "example": "e.g.",
+    "demonstrate": "show",
+    "utilize": "use",
+    "purchase": "buy",
+    "sufficient": "enough",
+    "assist": "help",
+    "request": "ask",
+    "inexpensive": "cheap",
+    "tomorrow": "tmrw",
+    "message": "msg",
+    "people": "ppl",
+    "please": "pls",
+    "business": "biz",
+    "favorite": "fav",
+    "every": "ea.",
+    "minute": "min",
+    "second": "sec",
+    "annually": "yearly",
+    "and": "&",
+    "resulting in": "leads to",
+    "important": "imp.",
+    "versus": "vs",
+    "determine": "find",
+    "terminate": "end",
+    "initialize": "start",
+    "commence": "begin",
+    "equivalent": "same",
+    "acquire": "get",
+    "maximum": "max",
+    "minimum": "min",
+    "miscellaneous": "misc",
+    "introduction": "intro",
+    "definition": "def",
+    "character": "char",
+    "equation": "eq",
+    "weight": "wt",
+    "height": "ht",
+    "quantity": "qty",
+    "average": "avg",
+    "standard": "std",
+    "package": "pkg",
+    "document": "doc",
+    "government": "govt",
+
+    // Days/Months
+    "january": "jan",
+    "february": "feb",
+    "august": "aug",
+    "september": "sept",
+    "october": "oct",
+    "november": "nov",
+    "december": "dec",
+    "monday": "mon",
+    "tuesday": "tue",
+    "wednesday": "wed",
+    "thursday": "thur",
+    "friday": "fri",
+    "saturday": "sat",
+    "sunday": "sun"
+};

package/src/csv_formatter/index.js

@@ -7,6 +7,7 @@ import { csvToJson, csvToJsonSync, jsonToCsv, jsonToCsvSync } from '../json_form
 import { csvToYaml, csvToYamlSync, yamlToCsv, yamlToCsvSync } from '../yaml_formatter/csv.js';
 import { csvToXml, csvToXmlSync, xmlToCsv, xmlToCsvSync } from '../xml_formatter/csv.js';
 import { validateCsvString, validateCsvStringSync } from './validator.js';
+import { dataManager, dataManagerAsync, extractJsonFromString, extractXmlFromString, extractCsvFromString } from '../utils.js';
 
 export class CsvConverter {
     /**
@@ -120,8 +121,9 @@ export class CsvConverter {
      */
     toToon(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManager(csvToToonSync, extractCsvFromString);
         return this._convertWithEncryption(
-            (data) => csvToToonSync(data),
+            (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -136,8 +138,9 @@ export class CsvConverter {
      */
     async toToonAsync(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManagerAsync(csvToToon, extractCsvFromString);
         return this._convertWithEncryptionAsync(
-            async (data) => csvToToon(data),
+            async (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -154,8 +157,9 @@ export class CsvConverter {
      */
     fromJson(jsonData, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManager(jsonToCsvSync, extractJsonFromString);
         return this._convertWithEncryption(
-            (data) => jsonToCsvSync(data),
+            (data) => optimizedConverterFn(data),
             jsonData,
             conversionMode
         );
@@ -170,8 +174,9 @@ export class CsvConverter {
      */
     async fromJsonAsync(jsonData, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManagerAsync(jsonToCsv, extractJsonFromString);
         return this._convertWithEncryptionAsync(
-            async (data) => jsonToCsv(data),
+            async (data) => optimizedConverterFn(data),
             jsonData,
             conversionMode
         );
@@ -186,8 +191,9 @@ export class CsvConverter {
      */
     toJson(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManager(csvToJsonSync, extractCsvFromString);
         return this._convertWithEncryption(
-            (data) => csvToJsonSync(data),
+            (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -202,8 +208,9 @@ export class CsvConverter {
      */
     async toJsonAsync(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManagerAsync(csvToJson, extractCsvFromString);
         return this._convertWithEncryptionAsync(
-            async (data) => csvToJson(data),
+            async (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -245,8 +252,9 @@ export class CsvConverter {
      */
     toYaml(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManager(csvToYamlSync, extractCsvFromString);
         return this._convertWithEncryption(
-            (data) => csvToYamlSync(data),
+            (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -254,8 +262,9 @@ export class CsvConverter {
 
     async toYamlAsync(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManagerAsync(csvToYaml, extractCsvFromString);
         return this._convertWithEncryptionAsync(
-            async (data) => csvToYaml(data),
+            async (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -272,8 +281,9 @@ export class CsvConverter {
      */
     fromXml(xmlString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManager(xmlToCsvSync, extractXmlFromString);
         return this._convertWithEncryption(
-            (data) => xmlToCsvSync(data),
+            (data) => optimizedConverterFn(data),
             xmlString,
             conversionMode
         );
@@ -281,8 +291,9 @@ export class CsvConverter {
 
     async fromXmlAsync(xmlString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManagerAsync(xmlToCsv, extractXmlFromString);
         return this._convertWithEncryptionAsync(
-            async (data) => xmlToCsv(data),
+            async (data) => optimizedConverterFn(data),
             xmlString,
             conversionMode
         );
@@ -297,8 +308,9 @@ export class CsvConverter {
      */
     toXml(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManager(csvToXmlSync, extractCsvFromString);
         return this._convertWithEncryption(
-            (data) => csvToXmlSync(data),
+            (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -306,8 +318,9 @@ export class CsvConverter {
 
     async toXmlAsync(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
+        const optimizedConverterFn = dataManagerAsync(csvToXml, extractCsvFromString);
         return this._convertWithEncryptionAsync(
-            async (data) => csvToXml(data),
+            async (data) => optimizedConverterFn(data),
             csvString,
             conversionMode
         );
@@ -356,7 +369,8 @@ export class CsvConverter {
      * @returns {string} TOON formatted string
      */
     static toToon(csvString) {
-        return csvToToonSync(csvString);
+        const optimizedConverterFn = dataManager(csvToToonSync, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -365,7 +379,8 @@ export class CsvConverter {
      * @returns {Promise<string>} TOON formatted string
      */
     static async toToonAsync(csvString) {
-        return csvToToon(csvString);
+        const optimizedConverterFn = dataManagerAsync(csvToToon, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -374,7 +389,8 @@ export class CsvConverter {
      * @returns {string} CSV formatted string
      */
     static fromJson(jsonData) {
-        return jsonToCsvSync(jsonData);
+        const optimizedConverterFn = dataManager(jsonToCsvSync, extractJsonFromString);
+        return optimizedConverterFn(jsonData);
     }
 
     /**
@@ -383,7 +399,8 @@ export class CsvConverter {
      * @returns {Promise<string>} CSV formatted string
      */
     static async fromJsonAsync(jsonData) {
-        return jsonToCsv(jsonData);
+        const optimizedConverterFn = dataManagerAsync(jsonToCsv, extractJsonFromString);
+        return optimizedConverterFn(jsonData);
     }
 
     /**
@@ -392,7 +409,8 @@ export class CsvConverter {
      * @returns {Array|string} JSON result
      */
     static toJson(csvString) {
-        return csvToJsonSync(csvString);
+        const optimizedConverterFn = dataManager(csvToJsonSync, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -401,7 +419,8 @@ export class CsvConverter {
      * @returns {Promise<Array|string>} JSON result
      */
     static async toJsonAsync(csvString) {
-        return csvToJson(csvString);
+        const optimizedConverterFn = dataManagerAsync(csvToJson, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -428,7 +447,8 @@ export class CsvConverter {
      * @returns {string} YAML formatted string
      */
     static toYaml(csvString) {
-        return csvToYamlSync(csvString);
+        const optimizedConverterFn = dataManager(csvToYamlSync, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -437,7 +457,8 @@ export class CsvConverter {
      * @returns {Promise<string>} YAML formatted string
      */
     static async toYamlAsync(csvString) {
-        return csvToYaml(csvString);
+        const optimizedConverterFn = dataManagerAsync(csvToYaml, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -446,7 +467,8 @@ export class CsvConverter {
      * @returns {string} CSV formatted string
      */
     static fromXml(xmlString) {
-        return xmlToCsvSync(xmlString);
+        const optimizedConverterFn = dataManager(xmlToCsvSync, extractXmlFromString);
+        return optimizedConverterFn(xmlString);
     }
 
     /**
@@ -455,7 +477,8 @@ export class CsvConverter {
      * @returns {Promise<string>} CSV formatted string
      */
     static async fromXmlAsync(xmlString) {
-        return xmlToCsv(xmlString);
+        const optimizedConverterFn = dataManagerAsync(xmlToCsv, extractXmlFromString);
+        return optimizedConverterFn(xmlString);
     }
 
     /**
@@ -464,7 +487,8 @@ export class CsvConverter {
      * @returns {string} XML formatted string
      */
     static toXml(csvString) {
-        return csvToXmlSync(csvString);
+        const optimizedConverterFn = dataManager(csvToXmlSync, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -473,7 +497,8 @@ export class CsvConverter {
      * @returns {Promise<string>} XML formatted string
      */
     static async toXmlAsync(csvString) {
-        return csvToXml(csvString);
+        const optimizedConverterFn = dataManagerAsync(csvToXml, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**

package/src/index.js

@@ -21,7 +21,9 @@ import {
     formatValue,
     extractJsonFromString,
     extractXmlFromString,
-    extractCsvFromString
+    extractCsvFromString,
+    dataManager,
+    dataManagerAsync
 } from './utils.js';
 import { Encryptor } from './encryptor.js';
 import { JsonConverter } from './json_formatter/index.js';
@@ -168,7 +170,8 @@ export class ToonConverter {
      */
     fromJson(jsonData, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
-        return this._convertWithEncryption(jsonToToonSync, jsonData, conversionMode);
+        const optimizedConverterFn = dataManager(jsonToToonSync, extractJsonFromString);
+        return this._convertWithEncryption(optimizedConverterFn, jsonData, conversionMode);
     }
 
     /**
@@ -180,7 +183,8 @@ export class ToonConverter {
      */
     async fromJsonAsync(jsonData, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
-        return this._convertWithEncryptionAsync(jsonToToon, jsonData, conversionMode);
+        const optimizedConverterFn = dataManagerAsync(jsonToToon, extractJsonFromString);
+        return this._convertWithEncryptionAsync(optimizedConverterFn, jsonData, conversionMode);
     }
 
     /**
@@ -274,7 +278,8 @@ export class ToonConverter {
      */
     fromXml(xmlString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
-        return this._convertWithEncryption(xmlToToonSync, xmlString, conversionMode);
+        const optimizedConverterFn = dataManager(xmlToToonSync, extractXmlFromString);
+        return this._convertWithEncryption(optimizedConverterFn, xmlString, conversionMode);
     }
 
     /**
@@ -286,7 +291,8 @@ export class ToonConverter {
      */
     async fromXmlAsync(xmlString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
-        return this._convertWithEncryptionAsync(xmlToToon, xmlString, conversionMode);
+        const optimizedConverterFn = dataManagerAsync(xmlToToon, extractXmlFromString);
+        return this._convertWithEncryptionAsync(optimizedConverterFn, xmlString, conversionMode);
     }
 
     /**
@@ -322,7 +328,8 @@ export class ToonConverter {
      */
     fromCsv(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
-        return this._convertWithEncryption(csvToToonSync, csvString, conversionMode);
+        const optimizedConverterFn = dataManager(csvToToonSync, extractCsvFromString);
+        return this._convertWithEncryption(optimizedConverterFn, csvString, conversionMode);
     }
 
     /**
@@ -334,7 +341,8 @@ export class ToonConverter {
      */
     async fromCsvAsync(csvString, options = {}) {
         const { conversionMode = 'no_encryption' } = options;
-        return this._convertWithEncryptionAsync(csvToToon, csvString, conversionMode);
+        const optimizedConverterFn = dataManagerAsync(csvToToon, extractCsvFromString);
+        return this._convertWithEncryptionAsync(optimizedConverterFn, csvString, conversionMode);
     }
 
     /**
@@ -391,7 +399,8 @@ export class ToonConverter {
      * @returns {string} TOON formatted string
      */
     static fromJson(jsonData) {
-        return jsonToToonSync(jsonData);
+        const optimizedConverterFn = dataManager(jsonToToonSync, extractJsonFromString);
+        return optimizedConverterFn(jsonData);
     }
 
     /**
@@ -400,7 +409,8 @@ export class ToonConverter {
      * @returns {Promise<string>} TOON formatted string
      */
     static async fromJsonAsync(jsonData) {
-        return jsonToToon(jsonData);
+        const optimizedConverterFn = dataManagerAsync(jsonToToon, extractJsonFromString);
+        return optimizedConverterFn(jsonData);
     }
 
     /**
@@ -465,7 +475,8 @@ export class ToonConverter {
      * @returns {string} TOON formatted string
      */
     static fromXml(xmlString) {
-        return xmlToToonSync(xmlString);
+        const optimizedConverterFn = dataManager(xmlToToonSync, extractXmlFromString);
+        return optimizedConverterFn(xmlString);
     }
 
     /**
@@ -474,7 +485,8 @@ export class ToonConverter {
      * @returns {Promise<string>} TOON formatted string
      */
     static async fromXmlAsync(xmlString) {
-        return xmlToToon(xmlString);
+        const optimizedConverterFn = dataManagerAsync(xmlToToon, extractXmlFromString);
+        return optimizedConverterFn(xmlString);
     }
 
     /**
@@ -501,7 +513,8 @@ export class ToonConverter {
      * @returns {Promise<string>} TOON formatted string
      */
     static async fromCsvAsync(csvString) {
-        return csvToToon(csvString);
+        const optimizedConverterFn = dataManagerAsync(csvToToon, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**
@@ -510,7 +523,8 @@ export class ToonConverter {
      * @returns {string} TOON formatted string
      */
     static fromCsv(csvString) {
-        return csvToToonSync(csvString);
+        const optimizedConverterFn = dataManager(csvToToonSync, extractCsvFromString);
+        return optimizedConverterFn(csvString);
     }
 
     /**