rowpipe 1.0.0 → 1.2.0

package/skills/rowpipe/SKILL.md

@@ -0,0 +1,263 @@
+---
+name: rowpipe
+description: Stream-first tabular data toolkit for inspecting, analyzing, transforming, validating, converting, mapping, and reducing large datasets across CSV, TSV, JSON, JSONL, XLSX, Apache Parquet, and Markdown formats with bounded O(1) memory and extreme throughput (~500,000 rows/s).
+---
+
+# Rowpipe: Stream-First Tabular Data Toolkit
+
+Rowpipe is a high-performance, stream-first command-line toolkit and Node.js library for tabular data processing. It processes gigabyte-scale datasets with bounded $O(1)$ memory usage, backpressure, batching, and zero-`eval` safe expression execution.
+
+---
+
+## 🚀 Quick Command Reference
+
+| Task | Command Syntax | Description |
+|---|---|---|
+| **Inspect** | `rowpipe inspect <file> [--sheet <name>] [--json]` | Inspect row count, column types, null %, and HLL uniqueness |
+| **Convert** | `rowpipe convert <input> <output> [--all-sheets]` | Stream convert between CSV, TSV, JSON, JSONL, XLSX, Parquet, Markdown |
+| **Schema** | `rowpipe schema <file> [--full] [--sample <n>]` | Infer column types and semantic patterns (email, url, uuid) |
+| **Stats** | `rowpipe stats <file> [--column <name>]` | Streaming Welford numeric stats & HyperLogLog distinct count |
+| **Map** | `rowpipe map <file> "<col1=expr1>" "<col2=expr2>"` | Derive and compute columns per row with arithmetic/JSON dot-notation |
+| **Filter** | `rowpipe filter <file> "<expression>"` | Filter rows using safe expression engine and pipe syntax (`\|`) |
+| **Reduce** | `rowpipe reduce <file> "<agg1=func(col)>" [--by <cols>]` | Global & group-by streaming aggregations (`sum`, `avg`, `min`, `max`, `count`, `countDistinct`, `stddev`) |
+| **Select** | `rowpipe select <file> <col1,col2,...>` | Project subset of columns in order |
+| **Rename** | `rowpipe rename <file> <old1=new1> <old2=new2>` | Rename columns in stream |
+| **Cast** | `rowpipe cast <file> <col1:type> [--on-error null]` | Stream cast types (`integer`, `number`, `boolean`, `date`) |
+| **Sample** | `rowpipe sample <file> --rows <n> [--seed <s>]` | Reservoir sampling with bounded memory |
+| **Validate** | `rowpipe validate <file> --schema <schema.json>` | Validate stream against JSON schema definition |
+| **Diff** | `rowpipe diff <left> <right> --key <cols> [opts]` | Stream compare datasets by key across formats with bounded RAM and disk spilling |
+| **Files** | `rowpipe files <path> [--include <glob>] [opts]` | Stream filesystem directory as tabular records with bounded memory, globs, hashes, and MIME |
+
+---
+
+## 🛠️ Common Workflows & Examples
+
+### 1. Inspect & Schema Inference (CSV, TSV, Parquet, Excel)
+```bash
+# Inspect CSV / TSV / JSONL file
+rowpipe inspect dataset.csv
+rowpipe inspect data.tsv
+
+# Inspect Apache Parquet file directly
+rowpipe inspect dataset.parquet --json
+
+# Inspect multi-sheet Excel workbook
+rowpipe inspect workbook.xlsx
+rowpipe inspect workbook.xlsx --sheet Orders
+
+# Full schema inference with semantic types (email, URL, UUID, date)
+rowpipe schema huge.csv --full
+```
+
+### 2. Format Conversion, Parquet & Gzip Streams
+```bash
+# Convert CSV to Apache Parquet
+rowpipe convert sales.csv sales.parquet
+
+# Convert Parquet to compressed JSONL (.jsonl.gz)
+rowpipe convert sales.parquet sales.jsonl.gz
+
+# Filter Parquet and output directly as GitHub Markdown table
+rowpipe filter dataset.parquet "amount > 500" --to markdown
+
+# Convert specific Excel worksheet to TSV
+rowpipe convert workbook.xlsx orders.tsv --sheet Orders
+
+# Bulk export all worksheets in workbook to a directory
+rowpipe convert workbook.xlsx --all-sheets --out-dir ./exported_sheets --to csv
+```
+
+### 3. Row Derivations & Calculations (`map`)
+Derive calculated columns using arithmetic, strings, dates, and direct JSON dot-notation:
+```bash
+# Arithmetic & percentage formulas
+rowpipe map sales.csv \
+  "profit = revenue - cost" \
+  "tax = revenue * 0.20" \
+  "margin = ((revenue - cost) / revenue) * 100"
+
+# Direct JSON dot-notation & pipe transformations
+rowpipe map events.jsonl \
+  "city = payload.user.address.city" \
+  "email = payload.user.email | lower | trim" \
+  "item_price = payload.items.0.price" \
+  "year = created_at | year"
+
+# Conditional mapping
+rowpipe map orders.csv "net_total = total | if(is_vip, total * 0.90, total) | round(2)"
+```
+
+### 4. Advanced Row Filtering (`filter`)
+```bash
+# Arithmetic & comparison on Parquet / CSV
+rowpipe filter sales.parquet '(revenue - cost) / revenue >= 0.25'
+
+# Pipe syntax for string & date functions
+rowpipe filter users.csv 'email | lower | endsWith("@corp.com")'
+rowpipe filter orders.csv 'created_at | year == 2026 && total > 500'
+
+# Direct JSON dot-notation in filter
+rowpipe filter events.jsonl.gz 'payload.user.city == "Istanbul" && payload.items.0.price > 50'
+
+# Set membership and range checks
+rowpipe filter products.tsv 'category | in("Electronics", "Computers") && price | between(100, 1000)'
+```
+
+### 5. Group-By & Global Aggregations (`reduce`)
+```bash
+# 1. Global summary (Single row output)
+rowpipe reduce sales.parquet \
+  "total_revenue = sum(revenue)" \
+  "avg_margin = avg(margin)" \
+  "min_price = min(price)" \
+  "max_price = max(price)" \
+  "total_orders = count()" \
+  "unique_customers = countDistinct(customer_id)"
+
+# 2. Group-By Streaming Aggregations
+rowpipe reduce sales.csv \
+  "total_revenue = sum(revenue)" \
+  "avg_profit = avg(profit)" \
+  "order_count = count()" \
+  --by country,category
+```
+
+Supported reduce functions:
+- `sum(col)` — Numeric sum
+- `avg(col)` / `mean(col)` — Average
+- `min(col)` / `max(col)` — Minimum / Maximum
+- `count()` / `count(col)` — Row / non-null count
+- `countDistinct(col)` — HyperLogLog distinct cardinality ($O(1)$ memory)
+- `stddev(col)` / `variance(col)` — Welford single-pass online standard deviation / variance
+- `first(col)` / `last(col)` — First non-null / last value
+
+### 6. End-to-End Unix Pipe Processing Chain
+```bash
+cat transactions.csv.gz | \
+  rowpipe map - "profit=revenue - cost" "margin=((revenue - cost) / revenue) * 100" | \
+  rowpipe filter - "margin >= 15" | \
+  rowpipe reduce - "total_profit=sum(profit)" "avg_margin=avg(margin)" "count=count()" --by country | \
+  rowpipe convert - --to markdown
+```
+
+### 7. Dataset Diffing & Change Feeds (`diff`)
+Compare two datasets by key across formats with bounded memory ($O(1)$) and automatic disk spilling:
+
+```bash
+# 1. Basic Keyed Diff
+rowpipe diff yesterday.csv today.csv --key id
+
+# 2. Composite Key & Cross-Format (CSV vs Parquet)
+rowpipe diff old.csv new.parquet --key country,user_id
+
+# 3. Multi-Sheet Excel Workbook Diff
+rowpipe diff old.xlsx new.xlsx --sheet Users --key id
+
+# 4. Filter Columns & String Normalization
+rowpipe diff old.csv new.csv \
+  --key id \
+  --ignore updated_at,last_seen \
+  --trim \
+  --ignore-case \
+  --epsilon 0.001
+
+# 5. Output Modes: Rows, JSONL Patch, and JSON
+rowpipe diff old.csv new.csv --key id --format rows --only changed --limit 50
+rowpipe diff old.csv new.csv --key id --format patch > changes.jsonl
+rowpipe diff old.csv new.csv --key id --json
+
+# 6. CI/CD Validation (Fails with exit code 1 if differences found)
+rowpipe diff expected.csv actual.csv --key id --fail-on-diff
+```
+
+### 8. Filesystem as Streaming Tabular Records (`files`)
+Stream files and directory trees with bounded memory, globs, streaming hashes, and MIME detection:
+
+```bash
+# 1. Stream directory records
+rowpipe files ./src --include "**/*.ts"
+
+# 2. Filter large files and select columns via Unix pipe
+rowpipe files ./uploads | rowpipe filter - 'size > 1000000' | rowpipe select - relative_path,size
+
+# 3. Stream SHA-256 or fast hashes
+rowpipe files ./assets --hash sha256 --to jsonl
+rowpipe files ./dataset --hash fast
+
+# 4. Generate deterministic build snapshot
+rowpipe files snapshot ./dist --hash sha256 --to jsonl > dist.snapshot.jsonl
+
+# 5. Direct directory diffing
+rowpipe files diff ./dist-v1 ./dist-v2 --hash sha256
+```
+
+---
+
+## 💻 Node.js Library API
+
+```typescript
+import {
+  createPipeline,
+  createReader,
+  createWriter,
+  CSVReader,
+  JSONLWriter,
+  ParquetReader,
+  ParquetWriter,
+  MarkdownWriter,
+  XLSXReader,
+  FileSystemReader,
+  mapRows,
+  filterRows,
+  reduceRows,
+  diffRows,
+  computeDiff,
+} from "rowpipe";
+import { createReadStream, createWriteStream } from "node:fs";
+
+// 1. Streaming Parquet -> Filter -> Markdown Pipeline
+const reader = new ParquetReader("sales.parquet");
+const writer = new MarkdownWriter(createWriteStream("summary.md"));
+
+await createPipeline(reader)
+  .pipe(mapRows({
+    profit: "revenue - cost",
+    margin: "((revenue - cost) / revenue) * 100",
+  }))
+  .pipe(filterRows("margin >= 15"))
+  .pipe(reduceRows({
+    by: ["country"],
+    aggregations: [
+      "total_profit = sum(profit)",
+      "avg_margin = avg(margin)",
+      "count = count()",
+    ],
+  }))
+  .to(writer);
+
+// 2. Stream Filesystem Directory as Tabular Rows
+const fsReader = new FileSystemReader({
+  root: "./src",
+  include: ["**/*.ts"],
+  hash: "sha256",
+  mime: true,
+});
+
+for await (const batch of fsReader.read()) {
+  for (const file of batch.rows) {
+    console.log(`${file.relative_path} (${file.size} bytes) - Hash: ${file.hash}`);
+  }
+}
+
+// 3. Streaming Diff Events
+for await (const event of diffRows({
+  leftPath: "old.csv",
+  rightPath: "new.parquet",
+  keys: ["id"],
+  ignore: ["updated_at"],
+})) {
+  if (event.type === "changed") {
+    console.log(`Row ${JSON.stringify(event.key)} changed:`, event.changes);
+  }
+}
+```

package/benchmarks/memory-bench.ts

@@ -1,109 +0,0 @@
-import { createPipeline } from "../src/core/pipeline.js";
-import { filterRows } from "../src/transforms/filter.js";
-import { selectColumns } from "../src/transforms/select.js";
-import { castColumns } from "../src/transforms/cast.js";
-import { DatasetStatsAggregator } from "../src/analytics/stats.js";
-import type { DataBatch, DataStream, Row } from "../src/core/types.js";
-import { formatBytes, formatNumber } from "../src/utils/formatting.js";
-
-/**
- * Generator producing synthetic data batches without buffering.
- */
-async function* generateSyntheticStream(
-  totalRows: number,
-  batchSize = 2000
-): DataStream {
-  let offset = 0;
-  while (offset < totalRows) {
-    const currentBatchSize = Math.min(batchSize, totalRows - offset);
-    const rows: Row[] = new Array(currentBatchSize);
-
-    for (let i = 0; i < currentBatchSize; i++) {
-      const id = offset + i;
-      rows[i] = {
-        id: String(id),
-        name: `User_${id}`,
-        age: String(20 + (id % 50)),
-        email: `user${id}@example.com`,
-        revenue: String((id % 1000) * 1.5),
-        country: id % 3 === 0 ? "TR" : id % 3 === 1 ? "US" : "DE",
-      };
-    }
-
-    yield {
-      rows,
-      offset,
-    };
-    offset += currentBatchSize;
-  }
-}
-
-async function runBenchmark(totalRows: number): Promise<void> {
-  process.stdout.write(`\n======================================================\n`);
-  process.stdout.write(`Running Benchmark with ${formatNumber(totalRows)} synthetic rows...\n`);
-  process.stdout.write(`Pipeline: Input -> Filter(age > 25) -> Select(id, name, age, revenue) -> Cast(revenue:number) -> Stats\n`);
-  process.stdout.write(`------------------------------------------------------\n`);
-
-  const initialMem = process.memoryUsage();
-  let peakRss = initialMem.rss;
-  let peakHeap = initialMem.heapUsed;
-
-  const startTime = Date.now();
-  let processedRows = 0;
-
-  const rawStream = generateSyntheticStream(totalRows, 2000);
-
-  const pipeline = createPipeline(rawStream)
-    .pipe(filterRows("age > 25"))
-    .pipe(selectColumns(["id", "name", "age", "revenue"]))
-    .pipe(castColumns({ revenue: "number" }));
-
-  const statsAgg = new DatasetStatsAggregator();
-
-  for await (const batch of pipeline.batches()) {
-    processedRows += batch.rows.length;
-    for (const row of batch.rows) {
-      statsAgg.add(row);
-    }
-
-    const currentMem = process.memoryUsage();
-    if (currentMem.rss > peakRss) peakRss = currentMem.rss;
-    if (currentMem.heapUsed > peakHeap) peakHeap = currentMem.heapUsed;
-
-    if (processedRows % 250000 === 0 || processedRows === totalRows) {
-      const elapsed = (Date.now() - startTime) / 1000 || 0.001;
-      const speed = Math.round(processedRows / elapsed);
-      process.stdout.write(
-        `  Processed: ${formatNumber(processedRows)} rows | Elapsed: ${elapsed.toFixed(1)}s | RSS: ${formatBytes(currentMem.rss)} | Heap: ${formatBytes(currentMem.heapUsed)} | Speed: ${formatNumber(speed)} rows/s\n`
-      );
-    }
-  }
-
-  const totalTimeMs = Date.now() - startTime;
-  const elapsedSec = totalTimeMs / 1000 || 0.001;
-  const rowsPerSec = Math.round(totalRows / elapsedSec);
-  // Estimate ~70 bytes per row
-  const estimatedMB = (totalRows * 70) / (1024 * 1024);
-  const mbPerSec = (estimatedMB / elapsedSec).toFixed(1);
-
-  const results = statsAgg.result();
-
-  process.stdout.write(`------------------------------------------------------\n`);
-  process.stdout.write(`Benchmark Results:\n`);
-  process.stdout.write(`  Total Input Rows:  ${formatNumber(totalRows)}\n`);
-  process.stdout.write(`  Filtered Rows:     ${formatNumber(processedRows)}\n`);
-  process.stdout.write(`  Execution Time:    ${elapsedSec.toFixed(2)}s\n`);
-  process.stdout.write(`  Throughput:        ${formatNumber(rowsPerSec)} rows/s (~${mbPerSec} MB/s)\n`);
-  process.stdout.write(`  Initial RSS:       ${formatBytes(initialMem.rss)}\n`);
-  process.stdout.write(`  Peak RSS:          ${formatBytes(peakRss)}\n`);
-  process.stdout.write(`  Peak Heap Used:    ${formatBytes(peakHeap)}\n`);
-  process.stdout.write(`  Memory Growth:     Bounded ($O(1)$ constant memory)\n`);
-  process.stdout.write(`======================================================\n`);
-}
-
-async function main() {
-  await runBenchmark(100_000);
-  await runBenchmark(1_000_000);
-}
-
-main().catch(console.error);