convert-buddy-js 0.12.4 → 0.12.5

package/README.md

@@ -8,6 +8,33 @@ A **high-performance, streaming-first** parser and converter for **CSV, XML, NDJ
 
 ---
 
+## Table of Contents
+
+- [Install](#install)
+- [Quick Start](#quick-start)
+- [API Overview](#api-overview)
+  - [Simple API](#1-simple-api-for-small-files)
+  - [Streaming API](#2-streaming-api-for-large-files--real-time-processing)
+  - [Instance-based API](#3-instance-based-api)
+  - [Low-level API](#4-low-level-api-for-advanced-use-cases)
+- [Stats & Monitoring](#stats--monitoring)
+- [Format Detection & Structure Analysis](#format-detection--structure-analysis)
+- [Real-World Examples](#real-world-examples)
+- [Formats & Configuration](#formats--configuration)
+- [Transformations & Field Mapping](#transformations--field-mapping)
+  - [Transform Basics](#transform-basics)
+  - [Transform Modes](#transform-modes)
+  - [Field Mapping](#field-mapping)
+  - [Compute Functions](#compute-functions)
+  - [Type Coercion](#type-coercion)
+  - [Error Handling](#transform-error-handling)
+  - [Transform Examples](#transform-examples)
+- [How it Works](#how-it-works)
+- [Benchmarks](#benchmarks-repository)
+- [License](#license)
+
+---
+
 ## Status & Quality
 
 [![Known Vulnerabilities](https://snyk.io/test/github/brunohanss/convert-buddy/badge.svg)](https://snyk.io/test/github/brunohanss/convert-buddy)
@@ -678,16 +705,26 @@ Supported
 
 ## Transformations & Field Mapping
 
-Convert Buddy supports field-level transformations during conversion via the `transform` option. Example:
+Convert Buddy includes a **powerful Rust-based transform engine** that enables field-level transformations, computed fields, type coercion, and data reshaping during conversion — all executed in WASM for maximum performance with zero JavaScript overhead.
+
+### Transform Basics
+
+Apply transforms using the `transform` option in any conversion:
 
 ```ts
-const out = await buddy.convert(csvString, {
+import { ConvertBuddy } from "convert-buddy-js";
+
+const buddy = new ConvertBuddy();
+
+const result = await buddy.convert(csvData, {
   outputFormat: "json",
   transform: {
-    mode: "augment",
+    mode: "replace",
     fields: [
+      { targetFieldName: "id", required: true },
       { targetFieldName: "full_name", compute: "concat(first, ' ', last)" },
       { targetFieldName: "age", coerce: { type: "i64" }, defaultValue: 0 },
+      { targetFieldName: "email", compute: "lower(trim(email))" },
     ],
     onMissingField: "null",
     onCoerceError: "null",
@@ -695,7 +732,511 @@ const out = await buddy.convert(csvString, {
 });
 ```
 
-Runtime compute helpers depend on the Rust/WASM core build. For complex transforms consider pre/post-processing in JS.
+**Key Features:**
+- 🚀 **WASM performance** — all transforms execute in Rust/WASM
+- 🔄 **Streaming** — transforms work on line-by-line records without buffering
+- 🧮 **30+ compute functions** — string manipulation, math, logic, type conversions
+- 🎯 **Type coercion** — convert strings to numbers, booleans, timestamps
+- ⚡ **Zero memory overhead** — no intermediate objects or copies in JS
+- 🛡️ **Error policies** — control behavior for missing fields and coercion failures
+
+### Transform Modes
+
+**`mode: "replace"`** (default)  
+Outputs only the fields you explicitly map. Original fields are discarded.
+
+```ts
+// Input:  { "first": "Alice", "last": "Smith", "age": 30, "city": "NYC" }
+// Output: { "full_name": "Alice Smith", "age": 30 }
+
+transform: {
+  mode: "replace",
+  fields: [
+    { targetFieldName: "full_name", compute: "concat(first, ' ', last)" },
+    { targetFieldName: "age" },
+  ],
+}
+```
+
+**`mode: "augment"`**  
+Preserves all original fields and adds/overwrites mapped fields.
+
+```ts
+// Input:  { "name": "Alice", "age": 30 }
+// Output: { "name": "Alice", "age": 30, "name_upper": "ALICE", "is_adult": true }
+
+transform: {
+  mode: "augment",
+  fields: [
+    { targetFieldName: "name_upper", compute: "upper(name)" },
+    { targetFieldName: "is_adult", compute: "gte(age, 18)" },
+  ],
+}
+```
+
+### Field Mapping
+
+Each field in the `fields` array defines a mapping:
+
+```ts
+type FieldMap = {
+  targetFieldName: string;        // Output field name
+  originFieldName?: string;       // Input field name (defaults to targetFieldName)
+  required?: boolean;             // Fail if missing
+  defaultValue?: any;             // Fallback value
+  coerce?: Coerce;                // Type conversion
+  compute?: string;               // Expression to compute value
+};
+```
+
+**Basic field pass-through:**
+```ts
+{ targetFieldName: "name" }  // Maps input "name" to output "name"
+```
+
+**Rename fields:**
+```ts
+{ targetFieldName: "fullName", originFieldName: "full_name" }  // Rename full_name → fullName
+```
+
+**Required fields:**
+```ts
+{ targetFieldName: "id", required: true }  // Fail conversion if "id" is missing
+```
+
+**Default values:**
+```ts
+{ targetFieldName: "status", defaultValue: "active" }  // Use "active" if field is missing or null
+```
+
+**Computed fields:**
+```ts
+{ targetFieldName: "display", compute: "concat(first, ' ', last)" }
+```
+
+**Type coercion:**
+```ts
+{ targetFieldName: "age", coerce: { type: "i64" } }  // Convert string "30" → number 30
+```
+
+**Combined:**
+```ts
+{
+  targetFieldName: "age_years",
+  originFieldName: "age",
+  coerce: { type: "i64" },
+  defaultValue: 0,
+  required: false,
+}
+```
+
+### Compute Functions
+
+Transform expressions support **30+ built-in functions** for data manipulation:
+
+#### String Functions
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `concat(...)` | Join strings | `concat(first, " ", last)` → `"Alice Smith"` |
+| `lower(s)` | Lowercase | `lower("HELLO")` → `"hello"` |
+| `upper(s)` | Uppercase | `upper("hello")` → `"HELLO"` |
+| `trim(s)` | Remove whitespace | `trim("  text  ")` → `"text"` |
+| `trim_start(s)` | Remove leading whitespace | `trim_start("  text")` → `"text"` |
+| `trim_end(s)` | Remove trailing whitespace | `trim_end("text  ")` → `"text"` |
+| `substring(s, start, end)` | Extract substring | `substring("hello", 0, 3)` → `"hel"` |
+| `replace(s, old, new)` | Replace text | `replace("foo bar", "foo", "baz")` → `"baz bar"` |
+| `len(s)` | String length | `len("hello")` → `5` |
+| `starts_with(s, prefix)` | Check prefix | `starts_with("https://...", "https")` → `true` |
+| `ends_with(s, suffix)` | Check suffix | `ends_with("file.pdf", ".pdf")` → `true` |
+| `contains(s, substr)` | Check substring | `contains("hello world", "world")` → `true` |
+| `pad_start(s, len, char)` | Left-pad | `pad_start("42", 5, "0")` → `"00042"` |
+| `pad_end(s, len, char)` | Right-pad | `pad_end("x", 3, "_")` → `"x__"` |
+| `repeat(s, n)` | Repeat string | `repeat("ab", 3)` → `"ababab"` |
+| `reverse(s)` | Reverse string | `reverse("hello")` → `"olleh"` |
+| `split(s, delim)` | Split to array | `split("a,b,c", ",")` → `["a","b","c"]` |
+| `join(arr, delim)` | Join array | `join(["a","b"], ",")` → `"a,b"` |
+
+#### Math Functions
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `+`, `-`, `*`, `/` | Arithmetic operators | `price * 1.1` |
+| `round(n)` | Round to nearest integer | `round(3.7)` → `4` |
+| `floor(n)` | Round down | `floor(3.9)` → `3` |
+| `ceil(n)` | Round up | `ceil(3.1)` → `4` |
+| `abs(n)` | Absolute value | `abs(-5)` → `5` |
+| `min(...)` | Minimum value | `min(a, b, c)` → smallest |
+| `max(...)` | Maximum value | `max(a, b, c)` → largest |
+
+#### Logic & Comparison
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `if(cond, true_val, false_val)` | Conditional | `if(age >= 18, "adult", "minor")` |
+| `not(bool)` | Boolean NOT | `not(active)` → opposite |
+| `eq(a, b)` | Equals | `eq(status, "active")` → `true/false` |
+| `ne(a, b)` | Not equals | `ne(a, b)` |
+| `gt(a, b)` | Greater than | `gt(age, 18)` |
+| `gte(a, b)` | Greater than or equal | `gte(age, 18)` |
+| `lt(a, b)` | Less than | `lt(price, 100)` |
+| `lte(a, b)` | Less than or equal | `lte(score, 50)` |
+
+#### Type Checking
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `is_null(val)` | Check if null | `is_null(optional_field)` → `true/false` |
+| `is_number(val)` | Check if number | `is_number(val)` → `true/false` |
+| `is_string(val)` | Check if string | `is_string(val)` → `true/false` |
+| `is_bool(val)` | Check if boolean | `is_bool(val)` → `true/false` |
+
+#### Type Conversion
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `to_string(val)` | Convert to string | `to_string(42)` → `"42"` |
+| `parse_int(s)` | Parse string to integer | `parse_int("123")` → `123` |
+| `parse_float(s)` | Parse string to float | `parse_float("3.14")` → `3.14` |
+
+#### Utility Functions
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| `coalesce(...)` | First non-null value | `coalesce(alt1, alt2, "default")` |
+| `default(val, fallback)` | Fallback for null | `default(optional, "N/A")` |
+
+**Nested expressions:**
+```ts
+compute: "upper(trim(concat(first, ' ', last)))"
+// Input: { first: "  alice  ", last: "  smith  " }
+// Output: "ALICE SMITH"
+```
+
+**Complex logic:**
+```ts
+compute: "if(gte(age, 18), 'adult', if(gte(age, 13), 'teen', 'child'))"
+```
+
+### Type Coercion
+
+Convert field types automatically:
+
+```ts
+type Coerce =
+  | { type: "string" }
+  | { type: "i64" }      // 64-bit integer
+  | { type: "f64" }      // 64-bit float
+  | { type: "bool" }
+  | { type: "timestamp_ms"; format?: "iso8601" | "unix_ms" | "unix_s" };
+```
+
+**String coercion:**
+```ts
+// Number → String: 42 → "42"
+// Boolean → String: true → "true"
+// Null → String: null → ""
+{ targetFieldName: "age_str", originFieldName: "age", coerce: { type: "string" } }
+```
+
+**Integer coercion (`i64`):**
+```ts
+// String → Integer: "42" → 42
+// Float → Integer: 3.7 → 3 (truncates)
+// Boolean → Integer: true → 1, false → 0
+{ targetFieldName: "count", coerce: { type: "i64" } }
+```
+
+**Float coercion (`f64`):**
+```ts
+// String → Float: "3.14" → 3.14
+// Integer → Float: 42 → 42.0
+{ targetFieldName: "price", coerce: { type: "f64" } }
+```
+
+**Boolean coercion:**
+```ts
+// String → Boolean: "true"/"false" → true/false, "1"/"0" → true/false
+// Number → Boolean: 0 → false, non-zero → true
+{ targetFieldName: "active", coerce: { type: "bool" } }
+```
+
+**Timestamp coercion:**
+```ts
+// ISO8601 string → Unix timestamp (ms)
+{
+  targetFieldName: "created_ts",
+  originFieldName: "created_at",
+  coerce: { type: "timestamp_ms", format: "iso8601" }
+}
+// "2024-01-15T10:30:00Z" → 1705315800000
+
+// Unix seconds → milliseconds
+{
+  targetFieldName: "updated_ts",
+  coerce: { type: "timestamp_ms", format: "unix_s" }
+}
+// 1705315800 → 1705315800000
+```
+
+### Transform Error Handling
+
+Control behavior when fields are missing or coercion fails:
+
+**`onMissingField`** — What to do when a field is missing:
+- `"error"` — Fail the entire conversion
+- `"null"` — Insert `null` for missing fields
+- `"drop"` — Omit the field from output
+
+```ts
+transform: {
+  fields: [{ targetFieldName: "optional_field" }],
+  onMissingField: "null",  // Missing fields become null
+}
+```
+
+**`onMissingRequired`** — What to do when a required field is missing:
+- `"error"` — Fail the conversion (default)
+- `"abort"` — Stop processing this record
+
+```ts
+transform: {
+  fields: [{ targetFieldName: "id", required: true }],
+  onMissingRequired: "error",  // Fail if "id" is missing
+}
+```
+
+**`onCoerceError`** — What to do when type coercion fails:
+- `"error"` — Fail the entire conversion
+- `"null"` — Set field to `null` on coercion failure
+- `"dropRecord"` — Silently skip records that fail coercion
+
+```ts
+transform: {
+  fields: [{ targetFieldName: "age", coerce: { type: "i64" } }],
+  onCoerceError: "null",  // "invalid" → null instead of throwing
+}
+```
+
+**Skip invalid records:**
+```ts
+// Drop records where age cannot be parsed as integer
+transform: {
+  fields: [
+    { targetFieldName: "name" },
+    { targetFieldName: "age", coerce: { type: "i64" } },
+  ],
+  onCoerceError: "dropRecord",  // Skip bad records silently
+}
+```
+
+### Transform Examples
+
+**Example 1: Clean and normalize user data**
+
+```ts
+const controller = buddy.stream("users.csv", {
+  outputFormat: "json",
+  transform: {
+    mode: "replace",
+    fields: [
+      // Required ID
+      { targetFieldName: "id", required: true },
+      
+      // Normalize name
+      { 
+        targetFieldName: "full_name", 
+        compute: "trim(concat(first_name, ' ', last_name))" 
+      },
+      
+      // Lowercase email
+      { targetFieldName: "email", compute: "lower(trim(email))" },
+      
+      // Parse age as integer with default
+      { 
+        targetFieldName: "age", 
+        coerce: { type: "i64" }, 
+        defaultValue: 0 
+      },
+      
+      // Active status as boolean
+      { targetFieldName: "active", coerce: { type: "bool" } },
+    ],
+    onMissingField: "null",
+    onCoerceError: "null",
+  },
+  onRecords: async (ctrl, records) => {
+    await db.users.insertMany(records);
+  },
+});
+```
+
+**Example 2: Calculate derived fields**
+
+```ts
+// Add computed fields to existing data
+transform: {
+  mode: "augment",  // Keep all original fields
+  fields: [
+    // Calculate total with tax
+    { 
+      targetFieldName: "total_with_tax", 
+      compute: "round(price * (1 + tax_rate))" 
+    },
+    
+    // Discount percentage
+    { 
+      targetFieldName: "discount_pct", 
+      compute: "round((original_price - price) / original_price * 100)" 
+    },
+    
+    // Is on sale?
+    { 
+      targetFieldName: "on_sale", 
+      compute: "lt(price, original_price)" 
+    },
+  ],
+}
+```
+
+**Example 3: Data validation and filtering**
+
+```ts
+// Only keep records with valid email and age >= 18
+const validRecords = [];
+
+const controller = buddy.stream(csvData, {
+  outputFormat: "ndjson",
+  transform: {
+    mode: "replace",
+    fields: [
+      { targetFieldName: "email", required: true },
+      { targetFieldName: "age", coerce: { type: "i64" }, required: true },
+      { 
+        targetFieldName: "is_adult", 
+        compute: "gte(age, 18)" 
+      },
+    ],
+    onMissingRequired: "error",
+    onCoerceError: "dropRecord",  // Skip records with invalid age
+  },
+  onRecords: (ctrl, records) => {
+    // Filter for adults only
+    const adults = records.filter(r => r.is_adult);
+    validRecords.push(...adults);
+  },
+});
+```
+
+**Example 4: Format conversion with field mapping**
+
+```ts
+// Convert XML to CSV with field restructuring
+const controller = buddy.stream("data.xml", {
+  inputFormat: "xml",
+  outputFormat: "csv",
+  xmlConfig: { recordElement: "user" },
+  transform: {
+    mode: "replace",
+    fields: [
+      { targetFieldName: "user_id", originFieldName: "id" },
+      { targetFieldName: "name", compute: "concat(firstName, ' ', lastName)" },
+      { targetFieldName: "email_domain", compute: "split(email, '@')[1]" },
+      { 
+        targetFieldName: "created_date", 
+        originFieldName: "createdAt",
+        coerce: { type: "timestamp_ms", format: "iso8601" }
+      },
+    ],
+  },
+  onData: (chunk) => {
+    fs.appendFileSync("output.csv", chunk);
+  },
+});
+```
+
+**Example 5: Streaming with real-time transforms**
+
+```ts
+// Process large file with transforms in streaming mode
+let processedCount = 0;
+
+const controller = buddy.stream("massive-dataset.csv", {
+  outputFormat: "ndjson",
+  recordBatchSize: 1000,
+  transform: {
+    mode: "replace",
+    fields: [
+      { targetFieldName: "id" },
+      { targetFieldName: "name_upper", compute: "upper(name)" },
+      { targetFieldName: "price", coerce: { type: "f64" } },
+      { 
+        targetFieldName: "price_category",
+        compute: "if(gt(price, 100), 'premium', if(gt(price, 50), 'mid', 'budget'))"
+      },
+    ],
+    onMissingField: "drop",
+  },
+  onRecords: async (ctrl, records, stats) => {
+    processedCount += records.length;
+    console.log(`Transformed ${processedCount} records at ${stats.throughputMbPerSec.toFixed(2)} MB/s`);
+    
+    // Stream directly to database
+    await db.products.insertMany(records);
+  },
+});
+
+await controller.done;
+```
+
+**Example 6: Complex field transformations**
+
+```ts
+transform: {
+  mode: "replace",
+  fields: [
+    // Combine multiple fields with formatting
+    {
+      targetFieldName: "address",
+      compute: "concat(street, ', ', city, ', ', state, ' ', zip)"
+    },
+    
+    // Conditional logic
+    {
+      targetFieldName: "shipping_fee",
+      compute: "if(eq(country, 'US'), 5.00, if(eq(country, 'CA'), 7.50, 12.00))"
+    },
+    
+    // Nested string operations
+    {
+      targetFieldName: "username",
+      compute: "lower(trim(replace(email, '@', '_at_')))"
+    },
+    
+    // Math with multiple fields
+    {
+      targetFieldName: "bmi",
+      compute: "round(weight / ((height / 100) * (height / 100)))"
+    },
+    
+    // String manipulation chain
+    {
+      targetFieldName: "display_name",
+      compute: "concat(upper(substring(first, 0, 1)), '. ', last)"
+    },
+  ],
+}
+```
+
+### Performance Notes
+
+- ✅ **Zero JS overhead** — transforms execute entirely in Rust/WASM
+- ✅ **Streaming** — processes line-by-line, no buffering of entire dataset
+- ✅ **Type safety** — expression parser validates syntax at compile time
+- ✅ **Memory efficient** — uses WASM linear memory, no JS objects created
+- ⚡ **Throughput** — typically adds <10% overhead vs. raw conversion
+
+For extremely complex transformations (e.g., API lookups, database joins), consider using `onRecords` callback for post-processing in JavaScript.
 
 ---
 

package/dist/browser.d.ts

@@ -1,5 +1,5 @@
 import { ConvertOptions, Format } from './index.js';
-export { BuddyController, BuddyInput, BuddyStats, BuddyStreamOptions, Coerce, ConvertBuddy, ConvertBuddyOptions, ConvertBuddyTransformStream, ConvertBuddy as Converter, CsvConfig, CsvDetection, DetectInput, DetectOptions, FieldMap, JsonDetection, NdjsonDetection, ProgressCallback, Stats, StreamController, StreamInput, StreamOptions, StreamResult, StructureDetection, TransformConfig, TransformMode, XmlConfig, XmlDetection, autoDetectConfig, convertAny, convertAnyToString, createStreamController, detectCsvFieldsAndDelimiter, detectFormat, detectStructure, detectXmlElements, getExtension, getMimeType, getOptimalThreadCount, getSuggestedFilename, getThreadingInfo, isWasmThreadingSupported, processStream, stream, streamRecords } from './index.js';
+export { BuddyController, BuddyStats, BuddyStreamOptions, Coerce, ConvertBuddy, ConvertBuddyOptions, ConvertBuddyTransformStream, ConvertBuddy as Converter, CsvConfig, CsvDetection, DetectInput, DetectOptions, FieldMap, JsonDetection, NdjsonDetection, ProgressCallback, Stats, StructureDetection, TransformConfig, TransformMode, XmlConfig, XmlDetection, autoDetectConfig, convertAny, convertAnyToString, detectCsvFieldsAndDelimiter, detectFormat, detectStructure, detectXmlElements, getExtension, getMimeType, getOptimalThreadCount, getSuggestedFilename, getThreadingInfo, isWasmThreadingSupported, stream } from './index.js';
 
 /**
  * Browser entry point for convert-buddy-js