rowpipe 1.0.0 → 1.2.0

package/README.md

@@ -1,10 +1,10 @@
 # Rowpipe
 
-> **A stream-first tabular data toolkit for CSV, JSON, JSONL, and XLSX.**
+> **A stream-first tabular data toolkit for CSV, TSV, JSON, JSONL, XLSX, Apache Parquet, and Markdown.**
 >
 > *Don't load the dataset. Stream through it.*
 
-Rowpipe is a high-performance, bounded-memory command-line toolkit and Node.js library for reading, inspecting, analyzing, transforming, validating, and converting tabular datasets across CSV, JSON, JSONL, and XLSX formats.
+Rowpipe is a high-performance, bounded-memory command-line toolkit and Node.js library for reading, inspecting, analyzing, transforming, validating, and converting tabular datasets across CSV, TSV, PSV, JSON, JSONL, XLSX, Apache Parquet (`.parquet`), Markdown tables (`.md`), and transparent Gzip (`.gz`) streams.
 
 Designed around stream backpressure and batch processing, Rowpipe processes multi-gigabyte and multi-million-row datasets with constant $O(1)$ memory usage.
 
@@ -13,8 +13,14 @@ Designed around stream backpressure and batch processing, Rowpipe processes mult
 ## Highlights
 
 - **Stream-First Architecture**: Datasets are never buffered entirely in memory; rows flow in configurable batches through async generator pipelines.
-- **Cross-Format Conversion & Multi-Sheet Export**: Convert seamlessly between CSV, JSON (streaming array & nested path), JSONL / NDJSON, and XLSX. Export individual sheets or bulk export all worksheets with `--all-sheets`.
-- **Zero-`eval` Expression Engine**: Safe filter expressions compiled into AST evaluators (`country == "TR" && revenue > 1000`).
+- **Rich Format Ecosystem**: Native streaming support for **CSV, TSV / PSV, JSON, JSONL, XLSX, Apache Parquet (`.parquet`), Markdown tables (`.md`), and Gzip (`.gz`)**.
+- **Apache Parquet Support**: High-performance columnar binary format reader (Snappy, Zstd, Gzip decompression via pure JS) and streaming writer.
+- **Cross-Format Conversion & Multi-Sheet Export**: Convert seamlessly across all formats. Export individual Excel sheets or bulk export all worksheets with `--all-sheets`.
+- **Filesystem as Data (`rowpipe files`)**: First-class streaming filesystem source adapter. Directories and files become streams of tabular records (`path`, `size`, `type`, `extension`, `modified_at`, streaming `hash`, `mime`) composable with `filter`, `select`, `stats`, `convert`, and `diff`.
+- **Cross-Format Dataset Diffing (`rowpipe diff`)**: High-performance, stream-first comparison across CSV, JSONL, XLSX, and Parquet with bounded memory and automatic spill-to-disk. Supports composite keys, duplicate policies, value tolerance, column filtering, schema diffing, JSONL patch generation, and CI `--fail-on-diff` mode.
+
+- **Zero-`eval` Safe Expression Engine**: Ultra-fast JIT closure-compiled evaluators supporting arithmetic (`revenue - cost`), string/date pipe functions (`email | lower | trim`), and direct JSON dot-notation (`payload.user.city`).
+- **Map & Reduce Streaming Engine**: Compute row derivations (`rowpipe map`) and global or group-by aggregations (`rowpipe reduce --by country`) in a single pass.
 - **Numerically Stable Streaming Stats**: Online Welford statistics (mean, variance, stddev, min, max, sum) and $O(1)$ HyperLogLog distinct cardinality estimation.
 - **Incremental Schema Inference & Semantic Types**: Detects data types with confidence scoring and semantic patterns (`email`, `url`, `uuid`, `ipv4`, `country-code`, `currency`, `phone`).
 - **Streaming Schema Validation**: Validates stream against JSON schemas with detailed violation reporting.
@@ -45,14 +51,19 @@ npm install rowpipe
 
 ## CLI Usage & Examples
 
-### 1. Inspect Dataset & Multi-Sheet Excel Analysis
+### 1. Inspect Dataset & Multi-Sheet Excel / Parquet Analysis
 Inspect format, row count, column list, data types, null percentage, and approximate distinct counts:
 
 ```bash
+# CSV / TSV / JSONL
 rowpipe inspect users.csv
+rowpipe inspect data.tsv
+
+# Apache Parquet dataset
+rowpipe inspect dataset.parquet
 
 # Machine-readable JSON output
-rowpipe inspect users.csv --json
+rowpipe inspect dataset.parquet --json
 
 # Analyze all sheets in an Excel workbook
 rowpipe inspect workbook.xlsx
@@ -61,39 +72,24 @@ rowpipe inspect workbook.xlsx
 rowpipe inspect workbook.xlsx --sheet Users
 ```
 
-Example Excel workbook overview:
-```text
-File: workbook.xlsx
-Format: XLSX
-Size: 1.25 MB
-Sheets: 3
-
-SHEET         ROWS       COLUMNS   PREVIEW COLUMNS
-Users         1,200      5         id, name, email, age, created_at
-Orders        8,400      4         order_id, user_id, total, status
-Products        300      3         sku, title, price
-
-Tip: Run with --sheet <name> to inspect a specific sheet in full detail.
-```
-
-### 2. Sheet-Based Export & Format Conversion
+### 2. Format Conversion & Export
 Convert across formats without intermediate buffering:
 
 ```bash
-# Export specific Excel worksheet to CSV
-rowpipe convert workbook.xlsx users.csv --sheet Users
+# CSV / TSV to Apache Parquet
+rowpipe convert sales.csv sales.parquet
 
-# Export specific Excel worksheet to JSONL
-rowpipe convert workbook.xlsx orders.jsonl --sheet Orders
+# Parquet to Gzipped JSONL
+rowpipe convert sales.parquet sales.jsonl.gz
 
-# Bulk export ALL worksheets to individual files in a directory
-rowpipe convert workbook.xlsx --all-sheets --out-dir ./exported/ --to csv
+# Filter and output directly as a GitHub Markdown Table
+rowpipe filter dataset.parquet "status == 'ACTIVE' && score >= 90" --to markdown
 
-# CSV to JSONL
-rowpipe convert users.csv users.jsonl
+# Export specific Excel worksheet to CSV / TSV
+rowpipe convert workbook.xlsx users.tsv --sheet Users
 
-# CSV to JSON array
-rowpipe convert users.csv users.json
+# Bulk export ALL worksheets to individual files in a directory
+rowpipe convert workbook.xlsx --all-sheets --out-dir ./exported/ --to csv
 
 # Streaming through gzip
 rowpipe convert data.csv.gz data.jsonl.gz
@@ -312,6 +308,253 @@ cat transactions.csv |
 
 ---
 
+## Compare Datasets (`rowpipe diff`)
+
+`rowpipe diff` is a production-grade, stream-first comparison engine that compares two tabular datasets by primary or composite key and reports:
+
+- **Added rows** (present in right dataset, absent in left)
+- **Removed rows** (present in left dataset, absent in right)
+- **Changed rows** (matching key, different values)
+- **Unchanged rows** (matching key, identical values)
+- **Changed columns breakdown** (ranked by frequency of changes)
+- **Schema differences** (added/removed columns and type changes)
+
+```bash
+rowpipe diff yesterday.csv today.csv --key id
+```
+
+Example Summary Output:
+
+```text
+Comparing:
+  yesterday.csv
+  today.csv
+
+Key: id
+
+Rows
+  added          1,283
+  removed          412
+  changed        8,921
+  unchanged     91,204
+
+Changed columns
+  price          5,821
+  status         2,101
+  email            999
+
+Schema
+  + loyalty_level: string
+  ~ price: integer -> number
+  - legacy_code: string
+```
+
+### Key Capabilities & Examples
+
+#### 1. Composite Keys
+Use comma-separated column names for multi-column composite keys with collision-proof typed encoding:
+```bash
+rowpipe diff old.csv new.csv --key country,user_id
+```
+
+#### 2. Cross-Format Comparison
+Compare datasets seamlessly across any combination of supported formats (CSV, TSV, JSONL, XLSX, Apache Parquet):
+```bash
+# Compare CSV with Parquet
+rowpipe diff old.csv new.parquet --key id
+
+# Compare JSONL with Excel
+rowpipe diff old.jsonl new.xlsx --key user_id --sheet Users
+
+# Compare specific sheets between two Excel workbooks
+rowpipe diff old.xlsx new.xlsx --left-sheet Nov --right-sheet Dec --key sku
+```
+
+#### 3. Ignore or Select Specific Columns
+```bash
+# Ignore volatile timestamp or audit columns
+rowpipe diff old.csv new.csv --key id --ignore updated_at,last_seen
+
+# Compare only specific columns for changes
+rowpipe diff old.csv new.csv --key id --columns price,status
+```
+
+#### 4. Value Equality, Tolerance & Coercion
+- **Strict by default**: `null != ""` and `0 != "0"`.
+- **`--coerce`**: Coerces types (e.g. `"42" == 42`, `"true" == true`) across formats.
+- **`--epsilon <val>`**: Floating point comparison tolerance (`abs(a - b) <= epsilon`).
+- **`--trim` & `--ignore-case`**: String normalization options.
+
+```bash
+rowpipe diff old.csv new.csv --key id --epsilon 0.001 --trim --ignore-case
+```
+
+#### 5. Duplicate Key Policies
+Duplicate keys in either dataset are never silently ignored:
+- `--duplicate-key error` *(default)*: Exits immediately with row context.
+- `--duplicate-key first`: Keeps the first occurrence.
+- `--duplicate-key last`: Keeps the latest occurrence.
+
+#### 6. Detailed Row & JSONL Patch Output Modes
+```bash
+# Detailed row change output with limit
+rowpipe diff old.csv new.csv --key id --format rows --only changed --limit 50
+
+# Stream JSONL patches (suitable for ETL change feeds or audit logs)
+rowpipe diff old.csv new.csv --key id --format patch > changes.jsonl
+
+# Machine-readable JSON summary metadata
+rowpipe diff old.csv new.csv --key id --json
+```
+
+Example JSONL patch output:
+```json
+{"op":"update","key":{"id":24},"changes":{"price":{"old":120,"new":125}}}
+{"op":"insert","key":{"id":31},"row":{"id":31,"name":"Alice"}}
+{"op":"delete","key":{"id":51},"row":{"id":51,"name":"Bob"}}
+```
+
+#### 7. CI/CD Validation (`--fail-on-diff`)
+Exit with non-zero code (`1`) when differences exist, or `0` when datasets are equivalent:
+
+```bash
+rowpipe diff expected.csv actual.csv \
+  --key id \
+  --fail-on-diff
+```
+
+This makes Rowpipe ideal for automated verification of:
+- **Database migrations**
+- **ETL jobs & data pipelines**
+- **API export validation**
+- **Regression test fixtures**
+- **Scheduled report auditing**
+
+#### 8. Bounded Memory Architecture & Automatic Spill-to-Disk
+Keyed diffing is classified as **Global State / Spillable**. Rowpipe buffers indexed keys in memory and automatically spills to disk using an append-only temp file index when `--memory-limit` is reached (default: 256MB).
+
+```bash
+# Handle multi-million-row diffs with bounded RAM
+rowpipe diff huge-old.csv huge-new.csv \
+  --key id \
+  --memory-limit 128mb
+```
+
+---
+
+## Filesystem as Data (`rowpipe files`)
+
+> **Everything becomes a stream of records.**
+
+Rowpipe treats files and directories as another first-class streaming record source. A directory tree becomes a stream of metadata records that can be filtered, selected, mapped, reduced, converted, and diffed using the exact same stream pipeline used for CSV, JSONL, XLSX, and Parquet.
+
+```bash
+rowpipe files ./uploads
+```
+
+Example Output (as Table / CSV / JSONL):
+
+```text
+path                      name             ext    size       modified_at
+uploads/avatar.png        avatar.png       png    182312     2026-09-16T08:13:11.923Z
+uploads/users.csv         users.csv        csv    8123102    2026-09-15T19:42:01.114Z
+uploads/archive/data.zip  data.zip         zip    12931231   2026-09-12T11:05:30.852Z
+```
+
+### Standard Record Fields
+Each file record contains rich, standardized metadata:
+- `path`: Full normalized path
+- `relative_path`: Normalized path relative to root directory (uses `/` across all operating systems)
+- `name`: File name with extension (e.g. `logo.png`)
+- `basename`: File name without extension (e.g. `logo`)
+- `extension`: Lowercase extension without dot (e.g. `png`)
+- `directory`: Parent directory path
+- `type`: Record type (`"file"`, `"directory"`, `"symlink"`)
+- `size`: Numeric size in bytes (`0` for directories)
+- `created_at`, `modified_at`, `accessed_at`: ISO 8601 timestamp strings
+- `mode`: File permission mode
+- `is_file`, `is_directory`, `is_symlink`: Boolean flags
+- `hash` *(optional)*: Streaming content hash (`sha256`, `sha1`, `md5`, `fast`)
+- `mime` & `category` *(optional)*: Inferred MIME type (`image/png`, `text/csv`) and category (`image`, `data`, `code`, `archive`, `document`, `video`, `audio`)
+
+---
+
+### Filesystem CLI Workflows & Examples
+
+#### 1. Recursive Traversal & Glob Filtering
+```bash
+# Scan directory recursively (default)
+rowpipe files ./src
+
+# Limit traversal depth
+rowpipe files . --max-depth 2
+
+# Include specific globs
+rowpipe files ./src --include "**/*.ts"
+
+# Exclude patterns (e.g. node_modules, build artifacts)
+rowpipe files . --exclude "dist/**" --exclude "*.tmp"
+
+# Include hidden files & dot directories
+rowpipe files . --hidden
+```
+
+#### 2. Streaming Hashes & MIME Type Detection
+```bash
+# Compute streaming SHA-256 hashes (never loads full file into memory)
+rowpipe files ./assets --hash sha256
+
+# Fast non-cryptographic fingerprinting (size + mtime + content sampling)
+rowpipe files ./dataset --hash fast
+
+# Infer MIME types and high-level categories
+rowpipe files ./uploads --mime
+```
+
+#### 3. Unix Pipeline Composition
+Compose filesystem metadata directly into Rowpipe streams:
+
+```bash
+# Filter large files (> 1MB)
+rowpipe files ./uploads | rowpipe filter - 'size > 1000000'
+
+# Project specific columns
+rowpipe files ./src | rowpipe select - path,size,extension
+
+# Calculate disk usage stats across file types
+rowpipe files ./src --to jsonl | \
+  rowpipe reduce - "total_size=sum(size)" "file_count=count()" --by extension
+```
+
+#### 4. Deterministic Snapshots & Build Verification
+Create stable, reproducible file inventory snapshots for build artifacts, deployments, or backups:
+
+```bash
+# Generate deterministic snapshot
+rowpipe files snapshot ./dist \
+  --hash sha256 \
+  --to jsonl > dist.snapshot.v1.jsonl
+
+# Generate snapshot of next build
+rowpipe files snapshot ./dist \
+  --hash sha256 \
+  --to jsonl > dist.snapshot.v2.jsonl
+
+# Compare snapshots using Rowpipe's diff engine
+rowpipe diff dist.snapshot.v1.jsonl dist.snapshot.v2.jsonl \
+  --key relative_path \
+  --ignore modified_at,accessed_at
+```
+
+#### 5. Native Directory Diffing (`rowpipe files diff`)
+Compare two directory trees directly:
+
+```bash
+rowpipe files diff ./dist-v1 ./dist-v2 --hash sha256
+```
+
+---
+
 ## Node.js Library API
 
 Rowpipe can be imported directly in your Node.js projects:
@@ -322,12 +565,15 @@ import {
   CSVReader,
   JSONLWriter,
   XLSXReader,
+  ParquetReader,
+  FileSystemReader,
   filterRows,
   selectColumns,
   castColumns,
   mapRows,
   reduceRows,
-  DatasetStatsAggregator,
+  diffRows,
+  computeDiff,
 } from "rowpipe";
 import { createReadStream, createWriteStream } from "node:fs";
 
@@ -351,10 +597,31 @@ await createPipeline(reader)
   }))
   .to(writer);
 
-// 2. Multi-Sheet Excel Inspection
-const xlsxReader = new XLSXReader("workbook.xlsx");
-const metadata = await xlsxReader.inspect();
-console.log("Workbook Sheets:", metadata.sheets);
+// 2. Stream Filesystem Directory as Tabular Rows
+const fsReader = new FileSystemReader({
+  root: "./src",
+  include: ["**/*.ts"],
+  hash: "sha256",
+  mime: true,
+});
+
+for await (const batch of fsReader.read()) {
+  for (const file of batch.rows) {
+    console.log(`${file.relative_path} (${file.size} bytes) - Hash: ${file.hash}`);
+  }
+}
+
+// 3. Streaming Dataset Diffing (Async Iterable Events)
+for await (const event of diffRows({
+  leftPath: "old.csv",
+  rightPath: "new.parquet",
+  keys: ["id"],
+  ignore: ["updated_at"],
+})) {
+  if (event.type === "changed") {
+    console.log(`Key ${JSON.stringify(event.key)} changed:`, event.changes);
+  }
+}
 ```
 
 ---
@@ -366,32 +633,49 @@ Rowpipe uses standardized POSIX exit codes:
 | Code | Meaning | Description |
 |------|---------|-------------|
 | `0`  | Success | Command executed successfully |
-| `1`  | Generic Failure | Unexpected runtime or internal error |
+| `1`  | Generic / Diff Mismatch | Runtime error or diff mismatch when `--fail-on-diff` is specified |
 | `2`  | Invalid Arguments | Missing or invalid CLI flags or arguments |
 | `3`  | Parse Error | Corrupted or malformed CSV/JSON/XLSX input with row/column context |
 | `4`  | Validation Failure | Schema validation violations detected |
+| `5`  | Duplicate Key Error | Duplicate key encountered during keyed diff under `--duplicate-key error` |
 
 ---
 
 ## Memory & Performance Benchmarks
 
-Tested on 1,000,000 synthetic rows through a complete pipeline (`Filter` -> `Select` -> `Cast` -> `Online Stats`):
+### 1. Pipeline Benchmark (`Filter` -> `Select` -> `Cast` -> `Online Stats`):
 
 | Dataset Size | Throughput | Execution Time | Peak RSS Memory | Memory Scaling |
 |--------------|------------|----------------|-----------------|----------------|
 | **100,000 rows** | **~318,000 rows/s** | 0.31s | ~106 MB | $O(1)$ Bounded |
 | **1,000,000 rows** | **~500,000 rows/s** | 2.00s | ~153 MB | $O(1)$ Bounded |
 
-To run the benchmark locally:
+### 2. Diff Engine Benchmark (`rowpipe diff`):
+
+| Comparison Size | Memory Limit | Throughput | Peak RSS Memory | Disk Spilled | Result |
+|-----------------|--------------|------------|-----------------|--------------|--------|
+| **2,000,000 rows** (1M vs 1M) | 64 MB | **~73,000 rows/s** | ~145 MB | Active | 100% Exact Match |
+
+### 3. Filesystem Streaming Benchmark (`rowpipe files`):
+
+| File Count | Operation | Throughput | Peak RSS Memory | Memory Scaling |
+|------------|-----------|------------|-----------------|----------------|
+| **50,000 files** | Stream Traversal + Metadata | **~24,100 files/s** | ~147 MB | $O(1)$ Bounded |
+| **50,000 files** | Filter + Select Pipeline | **~13,800 files/s** | ~147 MB | $O(1)$ Bounded |
+| **50,000 files** | Fast Fingerprint Hashing | **~3,470 files/s** | ~148 MB | $O(1)$ Bounded |
+
+To run the benchmarks locally:
 
 ```bash
 npm run bench
+npm run benchmark:diff
+npm run benchmark:files
 ```
 
 To enable real-time memory debugging on any command:
 
 ```bash
-ROWPIPE_DEBUG_MEMORY=1 rowpipe stats huge.csv
+ROWPIPE_DEBUG_MEMORY=1 rowpipe files ./huge-directory --hash fast
 ```
 
 ---
@@ -399,3 +683,5 @@ ROWPIPE_DEBUG_MEMORY=1 rowpipe stats huge.csv
 ## License
 
 MIT
+
+

package/dist/cli/commands/diff.d.ts

@@ -0,0 +1,30 @@
+import type { DiffOnlyFilter, DiffOutputFormat, DuplicateKeyPolicy } from "../../diff/types.js";
+export interface DiffCommandOptions {
+    key?: string;
+    format?: DiffOutputFormat;
+    only?: DiffOnlyFilter;
+    columns?: string;
+    ignore?: string;
+    limit?: string | number;
+    json?: boolean;
+    output?: string;
+    failOnDiff?: boolean;
+    duplicateKey?: DuplicateKeyPolicy;
+    coerce?: boolean;
+    epsilon?: string | number;
+    ignoreCase?: boolean;
+    trim?: boolean;
+    memoryLimit?: string;
+    sheet?: string;
+    leftSheet?: string;
+    rightSheet?: string;
+    from?: string;
+    fromLeft?: string;
+    fromRight?: string;
+    schema?: boolean;
+    quiet?: boolean;
+    noProgress?: boolean;
+    batchSize?: string | number;
+}
+export declare function diffCommand(leftPath: string, rightPath: string, options?: DiffCommandOptions): Promise<void>;
+//# sourceMappingURL=diff.d.ts.map

package/dist/cli/commands/diff.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"diff.d.ts","sourceRoot":"","sources":["../../../src/cli/commands/diff.ts"],"names":[],"mappings":"AAUA,OAAO,KAAK,EAEV,cAAc,EACd,gBAAgB,EAChB,kBAAkB,EACnB,MAAM,qBAAqB,CAAC;AAI7B,MAAM,WAAW,kBAAkB;IACjC,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,MAAM,CAAC,EAAE,gBAAgB,CAAC;IAC1B,IAAI,CAAC,EAAE,cAAc,CAAC;IACtB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;IACxB,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,YAAY,CAAC,EAAE,kBAAkB,CAAC;IAClC,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;IAC1B,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,SAAS,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;CAC7B;AAED,wBAAsB,WAAW,CAC/B,QAAQ,EAAE,MAAM,EAChB,SAAS,EAAE,MAAM,EACjB,OAAO,GAAE,kBAAuB,GAC/B,OAAO,CAAC,IAAI,CAAC,CA+Jf"}

package/dist/cli/commands/diff.js

@@ -0,0 +1,143 @@
+import { createWriteStream } from "node:fs";
+import { DiffMismatchError, InvalidArgumentError } from "../../core/errors.js";
+import { computeDiff, diffRows } from "../../diff/engine.js";
+import { formatJson, formatPatchEvent, formatRowEvent, formatSummary, } from "../../diff/reporter.js";
+import { formatNumber } from "../../utils/formatting.js";
+import { parseMemoryLimit } from "../../diff/storage/spillable-index.js";
+export async function diffCommand(leftPath, rightPath, options = {}) {
+    if (leftPath === "-" && rightPath === "-") {
+        throw new InvalidArgumentError("Cannot diff stdin against stdin ('- -'). One side must be a file or named source.");
+    }
+    if (!options.key) {
+        throw new InvalidArgumentError("Missing required option: --key <column1,column2,...>");
+    }
+    const keys = options.key.split(",").map((k) => k.trim()).filter(Boolean);
+    const compareColumns = options.columns
+        ? options.columns.split(",").map((c) => c.trim()).filter(Boolean)
+        : undefined;
+    const ignoreColumns = options.ignore
+        ? options.ignore.split(",").map((c) => c.trim()).filter(Boolean)
+        : undefined;
+    const leftSheet = options.leftSheet || options.sheet;
+    const rightSheet = options.rightSheet || options.sheet;
+    const leftFormat = options.fromLeft || options.from;
+    const rightFormat = options.fromRight || options.from;
+    const batchSize = Number(options.batchSize) || 1000;
+    const leftOptions = {
+        sheet: leftSheet,
+        format: leftFormat,
+        batchSize,
+        filePath: leftPath,
+    };
+    const rightOptions = {
+        sheet: rightSheet,
+        format: rightFormat,
+        batchSize,
+        filePath: rightPath,
+    };
+    const isTty = process.stderr.isTTY && !options.quiet && options.noProgress !== true;
+    const onProgress = (info) => {
+        if (!isTty)
+            return;
+        const isSpilledText = info.isSpilled ? " (disk spilled)" : "";
+        if (info.phase === "indexing_left") {
+            process.stderr.write(`\r\x1b[2K  Indexing left: ${formatNumber(info.leftRowsProcessed)} rows${isSpilledText}...`);
+        }
+        else if (info.phase === "comparing_right") {
+            process.stderr.write(`\r\x1b[2K  Comparing right: ${formatNumber(info.rightRowsProcessed)} rows [added: ${formatNumber(info.addedCount)}, changed: ${formatNumber(info.changedCount)}, unchanged: ${formatNumber(info.unchangedCount)}]${isSpilledText}...`);
+        }
+        else if (info.phase === "finishing") {
+            process.stderr.write(`\r\x1b[2K  Done comparing.\n`);
+        }
+    };
+    const engineOptions = {
+        left: leftPath,
+        right: rightPath,
+        keys,
+        columns: compareColumns,
+        ignore: ignoreColumns,
+        duplicateKey: options.duplicateKey || "error",
+        coerce: options.coerce,
+        epsilon: options.epsilon !== undefined ? Number(options.epsilon) : undefined,
+        ignoreCase: options.ignoreCase,
+        trim: options.trim,
+        memoryLimitBytes: parseMemoryLimit(options.memoryLimit),
+        includeSchema: options.schema !== false,
+        leftOptions,
+        rightOptions,
+        onProgress,
+    };
+    const formatMode = options.format || "summary";
+    const limit = options.limit !== undefined ? Number(options.limit) : undefined;
+    const onlyFilter = options.only;
+    let outStream = process.stdout;
+    let fileStream = null;
+    if (options.output) {
+        fileStream = createWriteStream(options.output, "utf-8");
+        outStream = fileStream;
+    }
+    const writeOut = async (chunk) => {
+        if (!outStream.write(chunk)) {
+            await new Promise((resolve) => outStream.once("drain", resolve));
+        }
+    };
+    try {
+        if (options.json || formatMode === "summary") {
+            const summary = await computeDiff(engineOptions);
+            if (isTty) {
+                process.stderr.write("\r\x1b[2K");
+            }
+            if (options.json) {
+                await writeOut(formatJson(summary));
+            }
+            else {
+                await writeOut(formatSummary(summary));
+            }
+            const totalDiffs = summary.rows.added + summary.rows.removed + summary.rows.changed;
+            if (options.failOnDiff && totalDiffs > 0) {
+                throw new DiffMismatchError(totalDiffs);
+            }
+            return;
+        }
+        // -----------------------------------------------------------
+        // Streaming row-level output (rows or patch)
+        // -----------------------------------------------------------
+        let emittedCount = 0;
+        let totalDiffCount = 0;
+        for await (const event of diffRows(engineOptions)) {
+            if (event.type === "added" || event.type === "removed" || event.type === "changed") {
+                totalDiffCount++;
+            }
+            // Filter by --only
+            if (onlyFilter && event.type !== onlyFilter) {
+                continue;
+            }
+            if (limit !== undefined && emittedCount >= limit) {
+                continue;
+            }
+            let formattedText = null;
+            if (formatMode === "rows") {
+                formattedText = formatRowEvent(event);
+            }
+            else if (formatMode === "patch") {
+                formattedText = formatPatchEvent(event);
+            }
+            if (formattedText) {
+                await writeOut(formattedText);
+                emittedCount++;
+            }
+        }
+        if (isTty) {
+            process.stderr.write("\r\x1b[2K");
+        }
+        if (options.failOnDiff && totalDiffCount > 0) {
+            throw new DiffMismatchError(totalDiffCount);
+        }
+    }
+    finally {
+        if (fileStream) {
+            await new Promise((resolve) => fileStream.end(() => resolve()));
+        }
+    }
+}
+//# sourceMappingURL=diff.js.map

package/dist/cli/commands/diff.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"diff.js","sourceRoot":"","sources":["../../../src/cli/commands/diff.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,SAAS,CAAC;AAC5C,OAAO,EAAE,iBAAiB,EAAE,oBAAoB,EAAE,MAAM,sBAAsB,CAAC;AAE/E,OAAO,EAAE,WAAW,EAAE,QAAQ,EAAE,MAAM,sBAAsB,CAAC;AAC7D,OAAO,EACL,UAAU,EACV,gBAAgB,EAChB,cAAc,EACd,aAAa,GACd,MAAM,wBAAwB,CAAC;AAOhC,OAAO,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AACzD,OAAO,EAAE,gBAAgB,EAAE,MAAM,uCAAuC,CAAC;AA8BzE,MAAM,CAAC,KAAK,UAAU,WAAW,CAC/B,QAAgB,EAChB,SAAiB,EACjB,UAA8B,EAAE;IAEhC,IAAI,QAAQ,KAAK,GAAG,IAAI,SAAS,KAAK,GAAG,EAAE,CAAC;QAC1C,MAAM,IAAI,oBAAoB,CAC5B,mFAAmF,CACpF,CAAC;IACJ,CAAC;IAED,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,CAAC;QACjB,MAAM,IAAI,oBAAoB,CAAC,sDAAsD,CAAC,CAAC;IACzF,CAAC;IAED,MAAM,IAAI,GAAG,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IACzE,MAAM,cAAc,GAAG,OAAO,CAAC,OAAO;QACpC,CAAC,CAAC,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC;QACjE,CAAC,CAAC,SAAS,CAAC;IACd,MAAM,aAAa,GAAG,OAAO,CAAC,MAAM;QAClC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC;QAChE,CAAC,CAAC,SAAS,CAAC;IAEd,MAAM,SAAS,GAAG,OAAO,CAAC,SAAS,IAAI,OAAO,CAAC,KAAK,CAAC;IACrD,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,OAAO,CAAC,KAAK,CAAC;IACvD,MAAM,UAAU,GAAG,OAAO,CAAC,QAAQ,IAAI,OAAO,CAAC,IAAI,CAAC;IACpD,MAAM,WAAW,GAAG,OAAO,CAAC,SAAS,IAAI,OAAO,CAAC,IAAI,CAAC;IACtD,MAAM,SAAS,GAAG,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,IAAI,CAAC;IAEpD,MAAM,WAAW,GAAkB;QACjC,KAAK,EAAE,SAAS;QAChB,MAAM,EAAE,UAAU;QAClB,SAAS;QACT,QAAQ,EAAE,QAAQ;KACnB,CAAC;IAEF,MAAM,YAAY,GAAkB;QAClC,KAAK,EAAE,UAAU;QACjB,MAAM,EAAE,WAAW;QACnB,SAAS;QACT,QAAQ,EAAE,SAAS;KACpB,CAAC;IAEF,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,KAAK,IAAI,CAAC,OAAO,CAAC,KAAK,IAAI,OAAO,CAAC,UAAU,KAAK,IAAI,CAAC;IAEpF,MAAM,UAAU,GAAG,CAAC,IAAS,EAAE,EAAE;QAC/B,IAAI,CAAC,KAAK;YAAE,OAAO;QACnB,MAAM,aAAa,GAAG,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,iBAAiB,CAAC,CAAC,CAAC,EAAE,CAAC;QAC9D,IAAI,IAAI,CAAC,KAAK,KAAK,eAAe,EAAE,CAAC;YACnC,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,6BAA6B,YAAY,CAAC,IAAI,CAAC,iBAAiB,CAAC,QAAQ,aAAa,KAAK,CAC5F,CAAC;QACJ,CAAC;aAAM,IAAI,IAAI,CAAC,KAAK,KAAK,iBAAiB,EAAE,CAAC;YAC5C,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,+BAA+B,YAAY,CAAC,IAAI,CAAC,kBAAkB,CAAC,iBAAiB,YAAY,CAAC,IAAI,CAAC,UAAU,CAAC,cAAc,YAAY,CAAC,IAAI,CAAC,YAAY,CAAC,gBAAgB,YAAY,CAAC,IAAI,CAAC,cAAc,CAAC,IAAI,aAAa,KAAK,CACvO,CAAC;QACJ,CAAC;aAAM,IAAI,IAAI,CAAC,KAAK,KAAK,WAAW,EAAE,CAAC;YACtC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,8BAA8B,CAAC,CAAC;QACvD,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,aAAa,GAAsB;QACvC,IAAI,EAAE,QAAQ;QACd,KAAK,EAAE,SAAS;QAChB,IAAI;QACJ,OAAO,EAAE,cAAc;QACvB,MAAM,EAAE,aAAa;QACrB,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,OAAO;QAC7C,MAAM,EAAE,OAAO,CAAC,MAAM;QACtB,OAAO,EAAE,OAAO,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,SAAS;QAC5E,UAAU,EAAE,OAAO,CAAC,UAAU;QAC9B,IAAI,EAAE,OAAO,CAAC,IAAI;QAClB,gBAAgB,EAAE,gBAAgB,CAAC,OAAO,CAAC,WAAW,CAAC;QACvD,aAAa,EAAE,OAAO,CAAC,MAAM,KAAK,KAAK;QACvC,WAAW;QACX,YAAY;QACZ,UAAU;KACX,CAAC;IAEF,MAAM,UAAU,GAAG,OAAO,CAAC,MAAM,IAAI,SAAS,CAAC;IAC/C,MAAM,KAAK,GAAG,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IAC9E,MAAM,UAAU,GAAG,OAAO,CAAC,IAAI,CAAC;IAEhC,IAAI,SAAS,GAAG,OAAO,CAAC,MAAM,CAAC;IAC/B,IAAI,UAAU,GAAQ,IAAI,CAAC;IAE3B,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;QACnB,UAAU,GAAG,iBAAiB,CAAC,OAAO,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QACxD,SAAS,GAAG,UAAU,CAAC;IACzB,CAAC;IAED,MAAM,QAAQ,GAAG,KAAK,EAAE,KAAa,EAAiB,EAAE;QACtD,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,KAAK,CAAC,EAAE,CAAC;YAC5B,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC,CAAC;QACzE,CAAC;IACH,CAAC,CAAC;IAEF,IAAI,CAAC;QACH,IAAI,OAAO,CAAC,IAAI,IAAI,UAAU,KAAK,SAAS,EAAE,CAAC;YAC7C,MAAM,OAAO,GAAG,MAAM,WAAW,CAAC,aAAa,CAAC,CAAC;YACjD,IAAI,KAAK,EAAE,CAAC;gBACV,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,WAAW,CAAC,CAAC;YACpC,CAAC;YAED,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;gBACjB,MAAM,QAAQ,CAAC,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC;YACtC,CAAC;iBAAM,CAAC;gBACN,MAAM,QAAQ,CAAC,aAAa,CAAC,OAAO,CAAC,CAAC,CAAC;YACzC,CAAC;YAED,MAAM,UAAU,GACd,OAAO,CAAC,IAAI,CAAC,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC;YACnE,IAAI,OAAO,CAAC,UAAU,IAAI,UAAU,GAAG,CAAC,EAAE,CAAC;gBACzC,MAAM,IAAI,iBAAiB,CAAC,UAAU,CAAC,CAAC;YAC1C,CAAC;YACD,OAAO;QACT,CAAC;QAED,8DAA8D;QAC9D,6CAA6C;QAC7C,8DAA8D;QAC9D,IAAI,YAAY,GAAG,CAAC,CAAC;QACrB,IAAI,cAAc,GAAG,CAAC,CAAC;QAEvB,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,QAAQ,CAAC,aAAa,CAAC,EAAE,CAAC;YAClD,IAAI,KAAK,CAAC,IAAI,KAAK,OAAO,IAAI,KAAK,CAAC,IAAI,KAAK,SAAS,IAAI,KAAK,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;gBACnF,cAAc,EAAE,CAAC;YACnB,CAAC;YAED,mBAAmB;YACnB,IAAI,UAAU,IAAI,KAAK,CAAC,IAAI,KAAK,UAAU,EAAE,CAAC;gBAC5C,SAAS;YACX,CAAC;YAED,IAAI,KAAK,KAAK,SAAS,IAAI,YAAY,IAAI,KAAK,EAAE,CAAC;gBACjD,SAAS;YACX,CAAC;YAED,IAAI,aAAa,GAAkB,IAAI,CAAC;YACxC,IAAI,UAAU,KAAK,MAAM,EAAE,CAAC;gBAC1B,aAAa,GAAG,cAAc,CAAC,KAAK,CAAC,CAAC;YACxC,CAAC;iBAAM,IAAI,UAAU,KAAK,OAAO,EAAE,CAAC;gBAClC,aAAa,GAAG,gBAAgB,CAAC,KAAK,CAAC,CAAC;YAC1C,CAAC;YAED,IAAI,aAAa,EAAE,CAAC;gBAClB,MAAM,QAAQ,CAAC,aAAa,CAAC,CAAC;gBAC9B,YAAY,EAAE,CAAC;YACjB,CAAC;QACH,CAAC;QAED,IAAI,KAAK,EAAE,CAAC;YACV,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,WAAW,CAAC,CAAC;QACpC,CAAC;QAED,IAAI,OAAO,CAAC,UAAU,IAAI,cAAc,GAAG,CAAC,EAAE,CAAC;YAC7C,MAAM,IAAI,iBAAiB,CAAC,cAAc,CAAC,CAAC;QAC9C,CAAC;IACH,CAAC;YAAS,CAAC;QACT,IAAI,UAAU,EAAE,CAAC;YACf,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC;QACxE,CAAC;IACH,CAAC;AACH,CAAC"}