rowpipe 1.1.0 → 1.2.0

package/skills/rowpipe/SKILL.md

@@ -25,6 +25,8 @@ Rowpipe is a high-performance, stream-first command-line toolkit and Node.js lib
 | **Cast** | `rowpipe cast <file> <col1:type> [--on-error null]` | Stream cast types (`integer`, `number`, `boolean`, `date`) |
 | **Sample** | `rowpipe sample <file> --rows <n> [--seed <s>]` | Reservoir sampling with bounded memory |
 | **Validate** | `rowpipe validate <file> --schema <schema.json>` | Validate stream against JSON schema definition |
+| **Diff** | `rowpipe diff <left> <right> --key <cols> [opts]` | Stream compare datasets by key across formats with bounded RAM and disk spilling |
+| **Files** | `rowpipe files <path> [--include <glob>] [opts]` | Stream filesystem directory as tabular records with bounded memory, globs, hashes, and MIME |
 
 ---
 
@@ -138,6 +140,57 @@ cat transactions.csv.gz | \
   rowpipe convert - --to markdown
 ```
 
+### 7. Dataset Diffing & Change Feeds (`diff`)
+Compare two datasets by key across formats with bounded memory ($O(1)$) and automatic disk spilling:
+
+```bash
+# 1. Basic Keyed Diff
+rowpipe diff yesterday.csv today.csv --key id
+
+# 2. Composite Key & Cross-Format (CSV vs Parquet)
+rowpipe diff old.csv new.parquet --key country,user_id
+
+# 3. Multi-Sheet Excel Workbook Diff
+rowpipe diff old.xlsx new.xlsx --sheet Users --key id
+
+# 4. Filter Columns & String Normalization
+rowpipe diff old.csv new.csv \
+  --key id \
+  --ignore updated_at,last_seen \
+  --trim \
+  --ignore-case \
+  --epsilon 0.001
+
+# 5. Output Modes: Rows, JSONL Patch, and JSON
+rowpipe diff old.csv new.csv --key id --format rows --only changed --limit 50
+rowpipe diff old.csv new.csv --key id --format patch > changes.jsonl
+rowpipe diff old.csv new.csv --key id --json
+
+# 6. CI/CD Validation (Fails with exit code 1 if differences found)
+rowpipe diff expected.csv actual.csv --key id --fail-on-diff
+```
+
+### 8. Filesystem as Streaming Tabular Records (`files`)
+Stream files and directory trees with bounded memory, globs, streaming hashes, and MIME detection:
+
+```bash
+# 1. Stream directory records
+rowpipe files ./src --include "**/*.ts"
+
+# 2. Filter large files and select columns via Unix pipe
+rowpipe files ./uploads | rowpipe filter - 'size > 1000000' | rowpipe select - relative_path,size
+
+# 3. Stream SHA-256 or fast hashes
+rowpipe files ./assets --hash sha256 --to jsonl
+rowpipe files ./dataset --hash fast
+
+# 4. Generate deterministic build snapshot
+rowpipe files snapshot ./dist --hash sha256 --to jsonl > dist.snapshot.jsonl
+
+# 5. Direct directory diffing
+rowpipe files diff ./dist-v1 ./dist-v2 --hash sha256
+```
+
 ---
 
 ## 💻 Node.js Library API
@@ -153,9 +206,12 @@ import {
   ParquetWriter,
   MarkdownWriter,
   XLSXReader,
+  FileSystemReader,
   mapRows,
   filterRows,
   reduceRows,
+  diffRows,
+  computeDiff,
 } from "rowpipe";
 import { createReadStream, createWriteStream } from "node:fs";
 
@@ -178,4 +234,30 @@ await createPipeline(reader)
     ],
   }))
   .to(writer);
+
+// 2. Stream Filesystem Directory as Tabular Rows
+const fsReader = new FileSystemReader({
+  root: "./src",
+  include: ["**/*.ts"],
+  hash: "sha256",
+  mime: true,
+});
+
+for await (const batch of fsReader.read()) {
+  for (const file of batch.rows) {
+    console.log(`${file.relative_path} (${file.size} bytes) - Hash: ${file.hash}`);
+  }
+}
+
+// 3. Streaming Diff Events
+for await (const event of diffRows({
+  leftPath: "old.csv",
+  rightPath: "new.parquet",
+  keys: ["id"],
+  ignore: ["updated_at"],
+})) {
+  if (event.type === "changed") {
+    console.log(`Row ${JSON.stringify(event.key)} changed:`, event.changes);
+  }
+}
 ```