rowpipe 2.1.0 → 2.2.1

package/README.md

@@ -13,11 +13,15 @@ Designed around stream backpressure and batch processing, Rowpipe processes mult
 ## Highlights
 
 - **Stream-First Architecture**: Datasets are never buffered entirely in memory; rows flow in configurable batches through async generator pipelines.
-- **Rich Format Ecosystem**: Native streaming support for **CSV, TSV / PSV, JSON, JSONL, XLSX, Apache Parquet (`.parquet`), Markdown tables (`.md`), and Gzip (`.gz`)**.
+- **Rich Format Ecosystem**: Native streaming support for **PostgreSQL, MySQL, SQLite, CSV, TSV / PSV, JSON, JSONL, XLSX, Apache Parquet (`.parquet`), Markdown tables (`.md`), and Gzip (`.gz`)**.
+- **First-Class Database Streaming (`rowpipe db`)**: True cursor-based streaming sources and sinks for PostgreSQL, MySQL, and SQLite. "Databases are just another Rowpipe source and sink" — rows normalize directly into standard `DataBatch` objects with zero parallel data models.
+- **Intelligent Pushdown Optimizer**: Automatically pushes compatible operations (`SELECT`, `WHERE`, `ORDER BY`, `LIMIT`, `OFFSET`) down into database queries in `--table` mode while seamlessly executing non-pushdownable transforms in the local streaming pipeline.
+- **Database Writing, Migration & DDL**: Stream directly into database tables (`--to-db`) with automatic table creation (`--create-table`), batched inserts, transactions (`--transaction`), dialect-specific upsert (`--upsert --conflict <cols>`), and truncate (`--truncate`).
+- **Lossless Type Precision**: Preserves full `BIGINT` and arbitrary precision `DECIMAL` / `NUMERIC` values without floating point corruption. Safely maps `JSONB`, `BYTEA`/`BLOB`, dates, and timestamps.
+- **Connection Security & Credential Masking**: Automatically sanitizes database passwords across all logs, explain plans, errors, and metadata (`postgres://user:***@host/db`).
 - **Apache Parquet Support**: High-performance columnar binary format reader (Snappy, Zstd, Gzip decompression via pure JS) and streaming writer.
-- **Cross-Format Conversion & Multi-Sheet Export**: Convert seamlessly across all formats. Export individual Excel sheets or bulk export all worksheets with `--all-sheets`.
 - **Filesystem as Data (`rowpipe files`)**: First-class streaming filesystem source adapter. Directories and files become streams of tabular records (`path`, `size`, `type`, `extension`, `modified_at`, streaming `hash`, `mime`) composable with `filter`, `select`, `stats`, `convert`, and `diff`.
-- **Cross-Format Dataset Diffing (`rowpipe diff`)**: High-performance, stream-first comparison across CSV, JSONL, XLSX, and Parquet with bounded memory and automatic spill-to-disk. Supports composite keys, duplicate policies, value tolerance, column filtering, schema diffing, JSONL patch generation, and CI `--fail-on-diff` mode.
+- **Cross-Format Dataset Diffing (`rowpipe diff`)**: High-performance, stream-first comparison across CSV, JSONL, XLSX, Parquet, and Database tables with bounded memory and automatic spill-to-disk. Supports composite keys, duplicate policies, value tolerance, column filtering, schema diffing, JSONL patch generation, and CI `--fail-on-diff` mode.
 
 - **Zero-`eval` Safe Expression Engine**: Ultra-fast JIT closure-compiled evaluators supporting arithmetic (`revenue - cost`), string/date pipe functions (`email | lower | trim`), and direct JSON dot-notation (`payload.user.city`).
 - **Map & Reduce Streaming Engine**: Compute row derivations (`rowpipe map`) and global or group-by aggregations (`rowpipe reduce --by country`) in a single pass.
@@ -549,11 +553,185 @@ rowpipe diff dist.snapshot.v1.jsonl dist.snapshot.v2.jsonl \
 #### 5. Native Directory Diffing (`rowpipe files diff`)
 Compare two directory trees directly:
 
+---
+
+## Databases
+
+Rowpipe provides first-class streaming database support for **PostgreSQL, MySQL, and SQLite**.
+
+The key architectural rule is:
+
+> **Databases are just another Rowpipe source and sink.**
+
+Database rows normalize directly into standard Rowpipe `DataBatch` and `Row` objects. There is no parallel execution engine or secondary row type — existing operations (`filter`, `select`, `sort`, `group`, `reduce`, `stats`, `diff`) compose natively without intermediate files or memory buffering.
+
+```text
+PostgreSQL / MySQL / SQLite / CSV / JSONL / Parquet / Files
+                           ↓
+                     DataBatch / Row
+                           ↓
+             filter / select / map / sort
+             group / reduce / stats / diff
+                           ↓
+                     TabularWriter
+                           ↓
+PostgreSQL / MySQL / SQLite / CSV / JSONL / Parquet / Markdown
+```
+
+### 1. Database Streaming Sources
+
+Read directly from database tables or custom queries in bounded, cursor-based streams:
+
 ```bash
-rowpipe files diff ./dist-v1 ./dist-v2 --hash sha256
+# Read from PostgreSQL table in streaming batches
+rowpipe db postgres://localhost/mydb --table users
+
+# Read from MySQL database
+rowpipe db mysql://root@localhost/app --table orders
+
+# Read from SQLite database file
+rowpipe db sqlite://./data.db --table events
+# or direct file path
+rowpipe db ./data.db --table events
+
+# Execute custom SQL query
+rowpipe db postgres://localhost/mydb \
+  --query 'SELECT id, email, created_at FROM users WHERE active = true'
+
+# Parameterized queries (driver-level binding)
+rowpipe db postgres://localhost/mydb \
+  --query 'SELECT * FROM users WHERE age > $1 AND status = $2' \
+  --params '[30, "ACTIVE"]'
 ```
 
----
+### 2. Output to Any Rowpipe Format
+
+Database streams flow directly through Rowpipe writers to CSV, JSONL, Parquet, or Markdown tables:
+
+```bash
+# Stream PostgreSQL table to JSONL
+rowpipe db postgres://localhost/app --table users --to jsonl > users.jsonl
+
+# Stream MySQL query to CSV
+rowpipe db mysql://localhost/app --query 'SELECT * FROM orders' --to csv > orders.csv
+
+# Stream SQLite table to GitHub Markdown table
+rowpipe db sqlite://./sales.db --table sales --limit 20 --to markdown
+```
+
+### 3. Pipeline Operations & Pushdown Optimizer
+
+When querying tables with `--table`, Rowpipe's **Intelligent Pushdown Optimizer** analyzes the pipeline and pushes compatible operations directly into database SQL:
+
+* `select` $\rightarrow$ `SELECT col1, col2`
+* `filter` $\rightarrow$ `WHERE condition`
+* `sort` $\rightarrow$ `ORDER BY col DESC`
+* `limit` $\rightarrow$ `LIMIT N`
+* `offset` $\rightarrow$ `OFFSET N`
+
+Unsupported or complex operations safely remain in the local streaming pipeline.
+
+```bash
+# Pushes down WHERE, SELECT, ORDER BY, and LIMIT to database SQL
+rowpipe db postgres://localhost/app \
+  --table users \
+  --filter 'active == true and age > 25' \
+  --select id,email,created_at \
+  --sort created_at:desc \
+  --limit 100 \
+  --to jsonl
+
+# Mixed execution: pushdown WHERE + local aggregations
+rowpipe db sqlite://./sales.db \
+  --table sales \
+  --where 'year = 2026' \
+  --group-by country \
+  --sum revenue \
+  --sort revenue_sum:desc
+```
+
+### 4. Database Sinks & File-to-Database Loading
+
+Load CSV, JSONL, XLSX, Parquet, or other databases directly into database tables in batched transactions:
+
+```bash
+# Load CSV into PostgreSQL table with automatic DDL table creation
+rowpipe users.csv \
+  --to-db postgres://localhost/app \
+  --to-table users \
+  --create-table
+
+# Load JSONL into SQLite database
+rowpipe events.jsonl \
+  --to-db sqlite://./app.db \
+  --to-table events \
+  --create-table
+
+# Dialect-specific Upsert on conflict keys
+rowpipe users.csv \
+  --to-db postgres://localhost/app \
+  --to-table users \
+  --upsert \
+  --conflict id
+
+# Destructive Truncate before bulk inserting
+rowpipe new_catalog.parquet \
+  --to-db mysql://localhost/store \
+  --to-table catalog \
+  --truncate \
+  --transaction
+```
+
+### 5. Database-to-Database Migrations (Zero Intermediate Files)
+
+Stream data directly between different database dialects with on-the-fly transformations:
+
+```bash
+# Migrate PostgreSQL table directly to SQLite with filtering & projection
+rowpipe db postgres://production/app \
+  --table users \
+  --filter 'deleted_at == null' \
+  --select id,email,created_at \
+  --to-db sqlite://./backup.db \
+  --to-table active_users \
+  --create-table
+```
+
+### 6. Cross-Format Diffing with Databases
+
+Diff a database table directly against a CSV, JSONL, or Parquet file using Rowpipe's diff engine:
+
+```bash
+# Compare PostgreSQL table against CSV export
+rowpipe diff \
+  'postgres://localhost/app?table=users' \
+  users_backup.csv \
+  --key id \
+  --coerce
+```
+
+### 7. Database Introspection
+
+Inspect tables and schema types without third-party database clients:
+
+```bash
+# List all tables in database
+rowpipe db postgres://localhost/app --tables
+
+# Inspect column schema, data types, and nullability
+rowpipe db postgres://localhost/app --schema users
+
+# Output schema as machine-readable JSON
+rowpipe db postgres://localhost/app --schema users --json
+```
+
+### 8. Lossless Type Mapping & Security
+
+- **BIGINT**: Values outside JavaScript safe integer range (e.g. `9223372036854775807`) are preserved losslessly as `bigint` without silent precision loss.
+- **DECIMAL / NUMERIC**: High-precision numbers survive database $\leftrightarrow$ file workflows without floating point truncation.
+- **JSON / JSONB**: Native objects and arrays are preserved and parsed appropriately.
+- **BYTEA / BLOB**: Binary buffers are mapped cleanly without dumping corrupted text to stdout.
+- **Credential Masking**: Connection URLs containing passwords (e.g. `postgres://admin:secret@host/db`) are automatically masked (`postgres://admin:***@host/db`) across all logs, explain plans, progress outputs, and errors.
 
 ---
 

package/dist/analytics/profiler.d.ts

@@ -0,0 +1,56 @@
+import type { Aggregator, ColumnType, Row, SemanticType } from "../core/types.js";
+export interface ColumnProfileResult {
+    name: string;
+    inferredType: ColumnType;
+    typeConfidence: number;
+    typeBreakdown: Record<string, number>;
+    anomalies: unknown[];
+    totalRows: number;
+    nullCount: number;
+    nullPercentage: number;
+    emptyStringCount: number;
+    approxDistinctCount: number;
+    distinctPercentage: number;
+    semanticType?: SemanticType;
+    semanticConfidence?: number;
+    isNumeric: boolean;
+    min?: number;
+    max?: number;
+    sum?: number;
+    mean?: number;
+    stddev?: number;
+    zerosCount?: number;
+    negativeCount?: number;
+    isString: boolean;
+    minLength?: number;
+    maxLength?: number;
+    avgLength?: number;
+    topValues?: Array<{
+        value: string;
+        count: number;
+        percentage: number;
+    }>;
+}
+export interface DatasetProfileResult {
+    totalRows: number;
+    totalColumns: number;
+    columns: ColumnProfileResult[];
+}
+/**
+ * Single-pass Streaming Dataset Profiler.
+ */
+export declare class DatasetProfiler implements Aggregator<DatasetProfileResult> {
+    private columnProfilers;
+    private totalRows;
+    add(row: Row): void;
+    result(): DatasetProfileResult;
+}
+/**
+ * Formats profile results as an aligned terminal table.
+ */
+export declare function formatProfileTerminal(profile: DatasetProfileResult): string;
+/**
+ * Formats profile results as a GitHub Flavored Markdown document.
+ */
+export declare function formatProfileMarkdown(profile: DatasetProfileResult): string;
+//# sourceMappingURL=profiler.d.ts.map

package/dist/analytics/profiler.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"profiler.d.ts","sourceRoot":"","sources":["../../src/analytics/profiler.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAMlF,MAAM,WAAW,mBAAmB;IAClC,IAAI,EAAE,MAAM,CAAC;IACb,YAAY,EAAE,UAAU,CAAC;IACzB,cAAc,EAAE,MAAM,CAAC;IACvB,aAAa,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACtC,SAAS,EAAE,OAAO,EAAE,CAAC;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,MAAM,CAAC;IACvB,gBAAgB,EAAE,MAAM,CAAC;IACzB,mBAAmB,EAAE,MAAM,CAAC;IAC5B,kBAAkB,EAAE,MAAM,CAAC;IAC3B,YAAY,CAAC,EAAE,YAAY,CAAC;IAC5B,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAG5B,SAAS,EAAE,OAAO,CAAC;IACnB,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,aAAa,CAAC,EAAE,MAAM,CAAC;IAGvB,QAAQ,EAAE,OAAO,CAAC;IAClB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,CAAC,EAAE,KAAK,CAAC;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;CACzE;AAED,MAAM,WAAW,oBAAoB;IACnC,SAAS,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,MAAM,CAAC;IACrB,OAAO,EAAE,mBAAmB,EAAE,CAAC;CAChC;AA+LD;;GAEG;AACH,qBAAa,eAAgB,YAAW,UAAU,CAAC,oBAAoB,CAAC;IACtE,OAAO,CAAC,eAAe,CAAqC;IAC5D,OAAO,CAAC,SAAS,CAAK;IAEtB,GAAG,CAAC,GAAG,EAAE,GAAG,GAAG,IAAI;IAYnB,MAAM,IAAI,oBAAoB;CAY/B;AAED;;GAEG;AACH,wBAAgB,qBAAqB,CAAC,OAAO,EAAE,oBAAoB,GAAG,MAAM,CAsC3E;AAED;;GAEG;AACH,wBAAgB,qBAAqB,CAAC,OAAO,EAAE,oBAAoB,GAAG,MAAM,CA+B3E"}

package/dist/analytics/profiler.js

@@ -0,0 +1,277 @@
+import { classifyPrimitiveType } from "./schema-inference.js";
+import { detectSemanticType } from "./semantic-types.js";
+import { HyperLogLog } from "./stats.js";
+import { formatNumber, formatTable } from "../utils/formatting.js";
+class ColumnProfiler {
+    name;
+    totalCount = 0;
+    nullCount = 0;
+    emptyStringCount = 0;
+    typeCounts = {
+        string: 0,
+        integer: 0,
+        bigint: 0,
+        number: 0,
+        decimal: 0,
+        boolean: 0,
+        date: 0,
+        datetime: 0,
+        binary: 0,
+        json: 0,
+        null: 0,
+        mixed: 0,
+    };
+    semanticCounts = {};
+    hll = new HyperLogLog(10);
+    frequencies = new Map();
+    // Numeric tracking (Welford's algorithm)
+    numericCount = 0;
+    numMin = Number.POSITIVE_INFINITY;
+    numMax = Number.NEGATIVE_INFINITY;
+    numSum = 0;
+    welfordMean = 0;
+    welfordM2 = 0;
+    zerosCount = 0;
+    negativeCount = 0;
+    // String tracking
+    stringCount = 0;
+    strMinLen = Number.POSITIVE_INFINITY;
+    strMaxLen = 0;
+    strSumLen = 0;
+    // Anomaly tracking
+    anomalies = [];
+    constructor(name) {
+        this.name = name;
+    }
+    add(val) {
+        this.totalCount++;
+        if (val === null || val === undefined) {
+            this.nullCount++;
+            this.typeCounts.null++;
+            return;
+        }
+        if (val === "") {
+            this.emptyStringCount++;
+            this.typeCounts.null++;
+            return;
+        }
+        const valStr = String(val);
+        this.hll.add(valStr);
+        // Track top frequency for up to 1000 distinct items
+        if (this.frequencies.size < 1000 || this.frequencies.has(valStr)) {
+            this.frequencies.set(valStr, (this.frequencies.get(valStr) || 0) + 1);
+        }
+        const primType = classifyPrimitiveType(val);
+        this.typeCounts[primType] = (this.typeCounts[primType] || 0) + 1;
+        // Semantic type detection
+        if (typeof val === "string" && val.length > 2) {
+            const sem = detectSemanticType(val);
+            if (sem) {
+                this.semanticCounts[sem] = (this.semanticCounts[sem] || 0) + 1;
+            }
+        }
+        // Numeric aggregation
+        const numVal = typeof val === "number" ? val : (typeof val === "string" && /^-?\d+(?:\.\d+)?$/.test(val.trim()) ? Number(val) : NaN);
+        if (!Number.isNaN(numVal)) {
+            this.numericCount++;
+            this.numSum += numVal;
+            if (numVal < this.numMin)
+                this.numMin = numVal;
+            if (numVal > this.numMax)
+                this.numMax = numVal;
+            if (numVal === 0)
+                this.zerosCount++;
+            if (numVal < 0)
+                this.negativeCount++;
+            // Welford's algorithm for online variance
+            const delta = numVal - this.welfordMean;
+            this.welfordMean += delta / this.numericCount;
+            const delta2 = numVal - this.welfordMean;
+            this.welfordM2 += delta * delta2;
+        }
+        // String aggregation
+        if (typeof val === "string") {
+            this.stringCount++;
+            const len = val.length;
+            if (len < this.strMinLen)
+                this.strMinLen = len;
+            if (len > this.strMaxLen)
+                this.strMaxLen = len;
+            this.strSumLen += len;
+        }
+    }
+    computeResult() {
+        const nonNullRows = this.totalCount - this.nullCount - this.emptyStringCount;
+        let dominantType = "string";
+        let maxTypeCount = 0;
+        for (const [t, count] of Object.entries(this.typeCounts)) {
+            if (t === "null")
+                continue;
+            if (count > maxTypeCount) {
+                maxTypeCount = count;
+                dominantType = t;
+            }
+        }
+        const typeConfidence = nonNullRows > 0 ? maxTypeCount / nonNullRows : 1;
+        const isNumeric = dominantType === "integer" || dominantType === "number" || dominantType === "decimal" || dominantType === "bigint";
+        const isString = dominantType === "string" || dominantType === "date" || dominantType === "datetime";
+        // Top values calculation
+        const sortedFreq = Array.from(this.frequencies.entries())
+            .sort((a, b) => b[1] - a[1])
+            .slice(0, 5)
+            .map(([value, count]) => ({
+            value,
+            count,
+            percentage: this.totalCount > 0 ? Math.round((count / this.totalCount) * 1000) / 10 : 0,
+        }));
+        // Semantic type
+        let semanticType;
+        let semanticConfidence;
+        let maxSemCount = 0;
+        for (const [sem, count] of Object.entries(this.semanticCounts)) {
+            if (count > maxSemCount) {
+                maxSemCount = count;
+                semanticType = sem;
+            }
+        }
+        if (semanticType && nonNullRows > 0) {
+            semanticConfidence = Math.round((maxSemCount / nonNullRows) * 100) / 100;
+        }
+        const approxDistinct = this.hll.count();
+        const res = {
+            name: this.name,
+            inferredType: dominantType,
+            typeConfidence: Math.round(typeConfidence * 100) / 100,
+            typeBreakdown: { ...this.typeCounts },
+            anomalies: this.anomalies,
+            totalRows: this.totalCount,
+            nullCount: this.nullCount + this.emptyStringCount,
+            nullPercentage: this.totalCount > 0 ? Math.round(((this.nullCount + this.emptyStringCount) / this.totalCount) * 1000) / 10 : 0,
+            emptyStringCount: this.emptyStringCount,
+            approxDistinctCount: approxDistinct,
+            distinctPercentage: this.totalCount > 0 ? Math.round((approxDistinct / this.totalCount) * 1000) / 10 : 0,
+            semanticType,
+            semanticConfidence,
+            isNumeric,
+            isString,
+            topValues: sortedFreq,
+        };
+        if (isNumeric && this.numericCount > 0) {
+            res.min = this.numMin === Number.POSITIVE_INFINITY ? undefined : this.numMin;
+            res.max = this.numMax === Number.NEGATIVE_INFINITY ? undefined : this.numMax;
+            res.sum = this.numSum;
+            res.mean = this.welfordMean;
+            res.stddev = this.numericCount > 1 ? Math.sqrt(this.welfordM2 / (this.numericCount - 1)) : 0;
+            res.zerosCount = this.zerosCount;
+            res.negativeCount = this.negativeCount;
+        }
+        if (isString && this.stringCount > 0) {
+            res.minLength = this.strMinLen === Number.POSITIVE_INFINITY ? 0 : this.strMinLen;
+            res.maxLength = this.strMaxLen;
+            res.avgLength = Math.round((this.strSumLen / this.stringCount) * 10) / 10;
+        }
+        return res;
+    }
+}
+/**
+ * Single-pass Streaming Dataset Profiler.
+ */
+export class DatasetProfiler {
+    columnProfilers = new Map();
+    totalRows = 0;
+    add(row) {
+        this.totalRows++;
+        for (const [col, val] of Object.entries(row)) {
+            let profiler = this.columnProfilers.get(col);
+            if (!profiler) {
+                profiler = new ColumnProfiler(col);
+                this.columnProfilers.set(col, profiler);
+            }
+            profiler.add(val);
+        }
+    }
+    result() {
+        const columns = [];
+        for (const profiler of this.columnProfilers.values()) {
+            columns.push(profiler.computeResult());
+        }
+        return {
+            totalRows: this.totalRows,
+            totalColumns: columns.length,
+            columns,
+        };
+    }
+}
+/**
+ * Formats profile results as an aligned terminal table.
+ */
+export function formatProfileTerminal(profile) {
+    const headers = ["COLUMN", "TYPE", "DISTINCT", "NULLS (%)", "MIN / MIN-LEN", "MAX / MAX-LEN", "MEAN / TOP-VALUE"];
+    const rows = profile.columns.map((c) => {
+        const nullStr = `${formatNumber(c.nullCount)} (${c.nullPercentage}%)`;
+        const distinctStr = `${formatNumber(c.approxDistinctCount)} (${c.distinctPercentage}%)`;
+        let minStr = "-";
+        let maxStr = "-";
+        let meanStr = "-";
+        if (c.isNumeric && c.min !== undefined && c.max !== undefined) {
+            minStr = String(c.min);
+            maxStr = String(c.max);
+            meanStr = c.mean !== undefined ? (Math.round(c.mean * 100) / 100).toString() : "-";
+        }
+        else if (c.isString) {
+            minStr = `${c.minLength ?? 0} chars`;
+            maxStr = `${c.maxLength ?? 0} chars`;
+            if (c.topValues && c.topValues.length > 0) {
+                meanStr = `${c.topValues[0].value} (${c.topValues[0].count})`;
+            }
+        }
+        return [
+            c.name,
+            c.semanticType ? `${c.inferredType} (${c.semanticType})` : c.inferredType,
+            distinctStr,
+            nullStr,
+            minStr,
+            maxStr,
+            meanStr,
+        ];
+    });
+    let out = `\n======================================================\n`;
+    out += `  Rowpipe Dataset Profile (${formatNumber(profile.totalRows)} rows, ${profile.totalColumns} columns)\n`;
+    out += `======================================================\n\n`;
+    out += formatTable(headers, rows);
+    out += "\n";
+    return out;
+}
+/**
+ * Formats profile results as a GitHub Flavored Markdown document.
+ */
+export function formatProfileMarkdown(profile) {
+    let md = `# Dataset Profile Report\n\n`;
+    md += `* **Total Rows**: ${formatNumber(profile.totalRows)}\n`;
+    md += `* **Total Columns**: ${profile.totalColumns}\n\n`;
+    md += `| Column | Type | Distinct | Nulls (%) | Min | Max | Mean / Top Value |\n`;
+    md += `| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n`;
+    for (const c of profile.columns) {
+        const typeStr = c.semanticType ? `${c.inferredType} (${c.semanticType})` : c.inferredType;
+        const nullStr = `${formatNumber(c.nullCount)} (${c.nullPercentage}%)`;
+        const distinctStr = `${formatNumber(c.approxDistinctCount)} (${c.distinctPercentage}%)`;
+        let minStr = "-";
+        let maxStr = "-";
+        let meanStr = "-";
+        if (c.isNumeric && c.min !== undefined && c.max !== undefined) {
+            minStr = String(c.min);
+            maxStr = String(c.max);
+            meanStr = c.mean !== undefined ? (Math.round(c.mean * 100) / 100).toString() : "-";
+        }
+        else if (c.isString) {
+            minStr = `${c.minLength ?? 0} chars`;
+            maxStr = `${c.maxLength ?? 0} chars`;
+            if (c.topValues && c.topValues.length > 0) {
+                meanStr = `${c.topValues[0].value} (${c.topValues[0].count})`;
+            }
+        }
+        md += `| **${c.name}** | \`${typeStr}\` | ${distinctStr} | ${nullStr} | ${minStr} | ${maxStr} | ${meanStr} |\n`;
+    }
+    return md;
+}
+//# sourceMappingURL=profiler.js.map

package/dist/analytics/profiler.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"profiler.js","sourceRoot":"","sources":["../../src/analytics/profiler.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AAC9D,OAAO,EAAE,kBAAkB,EAAE,MAAM,qBAAqB,CAAC;AACzD,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,WAAW,EAAE,MAAM,wBAAwB,CAAC;AAyCnE,MAAM,cAAc;IACT,IAAI,CAAS;IACtB,UAAU,GAAG,CAAC,CAAC;IACf,SAAS,GAAG,CAAC,CAAC;IACd,gBAAgB,GAAG,CAAC,CAAC;IACrB,UAAU,GAA+B;QACvC,MAAM,EAAE,CAAC;QACT,OAAO,EAAE,CAAC;QACV,MAAM,EAAE,CAAC;QACT,MAAM,EAAE,CAAC;QACT,OAAO,EAAE,CAAC;QACV,OAAO,EAAE,CAAC;QACV,IAAI,EAAE,CAAC;QACP,QAAQ,EAAE,CAAC;QACX,MAAM,EAAE,CAAC;QACT,IAAI,EAAE,CAAC;QACP,IAAI,EAAE,CAAC;QACP,KAAK,EAAE,CAAC;KACT,CAAC;IACF,cAAc,GAA2B,EAAE,CAAC;IAC5C,GAAG,GAAG,IAAI,WAAW,CAAC,EAAE,CAAC,CAAC;IAC1B,WAAW,GAAG,IAAI,GAAG,EAAkB,CAAC;IAExC,yCAAyC;IACzC,YAAY,GAAG,CAAC,CAAC;IACjB,MAAM,GAAG,MAAM,CAAC,iBAAiB,CAAC;IAClC,MAAM,GAAG,MAAM,CAAC,iBAAiB,CAAC;IAClC,MAAM,GAAG,CAAC,CAAC;IACX,WAAW,GAAG,CAAC,CAAC;IAChB,SAAS,GAAG,CAAC,CAAC;IACd,UAAU,GAAG,CAAC,CAAC;IACf,aAAa,GAAG,CAAC,CAAC;IAElB,kBAAkB;IAClB,WAAW,GAAG,CAAC,CAAC;IAChB,SAAS,GAAG,MAAM,CAAC,iBAAiB,CAAC;IACrC,SAAS,GAAG,CAAC,CAAC;IACd,SAAS,GAAG,CAAC,CAAC;IAEd,mBAAmB;IACnB,SAAS,GAAc,EAAE,CAAC;IAE1B,YAAY,IAAY;QACtB,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;IACnB,CAAC;IAED,GAAG,CAAC,GAAY;QACd,IAAI,CAAC,UAAU,EAAE,CAAC;QAElB,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;YACtC,IAAI,CAAC,SAAS,EAAE,CAAC;YACjB,IAAI,CAAC,UAAU,CAAC,IAAI,EAAE,CAAC;YACvB,OAAO;QACT,CAAC;QAED,IAAI,GAAG,KAAK,EAAE,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,EAAE,CAAC;YACxB,IAAI,CAAC,UAAU,CAAC,IAAI,EAAE,CAAC;YACvB,OAAO;QACT,CAAC;QAED,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;QAC3B,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QAErB,oDAAoD;QACpD,IAAI,IAAI,CAAC,WAAW,CAAC,IAAI,GAAG,IAAI,IAAI,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;YACjE,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QACxE,CAAC;QAED,MAAM,QAAQ,GAAG,qBAAqB,CAAC,GAAG,CAAC,CAAC;QAC5C,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAEjE,0BAA0B;QAC1B,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC9C,MAAM,GAAG,GAAG,kBAAkB,CAAC,GAAG,CAAC,CAAC;YACpC,IAAI,GAAG,EAAE,CAAC;gBACR,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;YACjE,CAAC;QACH,CAAC;QAED,sBAAsB;QACtB,MAAM,MAAM,GAAG,OAAO,GAAG,KAAK,QAAQ,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,GAAG,KAAK,QAAQ,IAAI,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;QACrI,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC;YAC1B,IAAI,CAAC,YAAY,EAAE,CAAC;YACpB,IAAI,CAAC,MAAM,IAAI,MAAM,CAAC;YACtB,IAAI,MAAM,GAAG,IAAI,CAAC,MAAM;gBAAE,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;YAC/C,IAAI,MAAM,GAAG,IAAI,CAAC,MAAM;gBAAE,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;YAC/C,IAAI,MAAM,KAAK,CAAC;gBAAE,IAAI,CAAC,UAAU,EAAE,CAAC;YACpC,IAAI,MAAM,GAAG,CAAC;gBAAE,IAAI,CAAC,aAAa,EAAE,CAAC;YAErC,0CAA0C;YAC1C,MAAM,KAAK,GAAG,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;YACxC,IAAI,CAAC,WAAW,IAAI,KAAK,GAAG,IAAI,CAAC,YAAY,CAAC;YAC9C,MAAM,MAAM,GAAG,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;YACzC,IAAI,CAAC,SAAS,IAAI,KAAK,GAAG,MAAM,CAAC;QACnC,CAAC;QAED,qBAAqB;QACrB,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;YAC5B,IAAI,CAAC,WAAW,EAAE,CAAC;YACnB,MAAM,GAAG,GAAG,GAAG,CAAC,MAAM,CAAC;YACvB,IAAI,GAAG,GAAG,IAAI,CAAC,SAAS;gBAAE,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC;YAC/C,IAAI,GAAG,GAAG,IAAI,CAAC,SAAS;gBAAE,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC;YAC/C,IAAI,CAAC,SAAS,IAAI,GAAG,CAAC;QACxB,CAAC;IACH,CAAC;IAED,aAAa;QACX,MAAM,WAAW,GAAG,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,gBAAgB,CAAC;QAC7E,IAAI,YAAY,GAAe,QAAQ,CAAC;QACxC,IAAI,YAAY,GAAG,CAAC,CAAC;QAErB,KAAK,MAAM,CAAC,CAAC,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,UAAU,CAAC,EAAE,CAAC;YACzD,IAAI,CAAC,KAAK,MAAM;gBAAE,SAAS;YAC3B,IAAI,KAAK,GAAG,YAAY,EAAE,CAAC;gBACzB,YAAY,GAAG,KAAK,CAAC;gBACrB,YAAY,GAAG,CAAe,CAAC;YACjC,CAAC;QACH,CAAC;QAED,MAAM,cAAc,GAAG,WAAW,GAAG,CAAC,CAAC,CAAC,CAAC,YAAY,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC;QACxE,MAAM,SAAS,GAAG,YAAY,KAAK,SAAS,IAAI,YAAY,KAAK,QAAQ,IAAI,YAAY,KAAK,SAAS,IAAI,YAAY,KAAK,QAAQ,CAAC;QACrI,MAAM,QAAQ,GAAG,YAAY,KAAK,QAAQ,IAAI,YAAY,KAAK,MAAM,IAAI,YAAY,KAAK,UAAU,CAAC;QAErG,yBAAyB;QACzB,MAAM,UAAU,GAAG,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,WAAW,CAAC,OAAO,EAAE,CAAC;aACtD,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;aAC3B,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC;aACX,GAAG,CAAC,CAAC,CAAC,KAAK,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,CAAC;YACxB,KAAK;YACL,KAAK;YACL,UAAU,EAAE,IAAI,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,KAAK,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,CAAC;SACxF,CAAC,CAAC,CAAC;QAEN,gBAAgB;QAChB,IAAI,YAAsC,CAAC;QAC3C,IAAI,kBAAsC,CAAC;QAC3C,IAAI,WAAW,GAAG,CAAC,CAAC;QACpB,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,cAAc,CAAC,EAAE,CAAC;YAC/D,IAAI,KAAK,GAAG,WAAW,EAAE,CAAC;gBACxB,WAAW,GAAG,KAAK,CAAC;gBACpB,YAAY,GAAG,GAAmB,CAAC;YACrC,CAAC;QACH,CAAC;QACD,IAAI,YAAY,IAAI,WAAW,GAAG,CAAC,EAAE,CAAC;YACpC,kBAAkB,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,WAAW,GAAG,WAAW,CAAC,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;QAC3E,CAAC;QAED,MAAM,cAAc,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,CAAC;QAExC,MAAM,GAAG,GAAwB;YAC/B,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,YAAY,EAAE,YAAY;YAC1B,cAAc,EAAE,IAAI,CAAC,KAAK,CAAC,cAAc,GAAG,GAAG,CAAC,GAAG,GAAG;YACtD,aAAa,EAAE,EAAE,GAAG,IAAI,CAAC,UAAU,EAAE;YACrC,SAAS,EAAE,IAAI,CAAC,SAAS;YACzB,SAAS,EAAE,IAAI,CAAC,UAAU;YAC1B,SAAS,EAAE,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,gBAAgB;YACjD,cAAc,EAAE,IAAI,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,gBAAgB,CAAC,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,CAAC;YAC9H,gBAAgB,EAAE,IAAI,CAAC,gBAAgB;YACvC,mBAAmB,EAAE,cAAc;YACnC,kBAAkB,EAAE,IAAI,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,cAAc,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,CAAC;YACxG,YAAY;YACZ,kBAAkB;YAClB,SAAS;YACT,QAAQ;YACR,SAAS,EAAE,UAAU;SACtB,CAAC;QAEF,IAAI,SAAS,IAAI,IAAI,CAAC,YAAY,GAAG,CAAC,EAAE,CAAC;YACvC,GAAG,CAAC,GAAG,GAAG,IAAI,CAAC,MAAM,KAAK,MAAM,CAAC,iBAAiB,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YAC7E,GAAG,CAAC,GAAG,GAAG,IAAI,CAAC,MAAM,KAAK,MAAM,CAAC,iBAAiB,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YAC7E,GAAG,CAAC,GAAG,GAAG,IAAI,CAAC,MAAM,CAAC;YACtB,GAAG,CAAC,IAAI,GAAG,IAAI,CAAC,WAAW,CAAC;YAC5B,GAAG,CAAC,MAAM,GAAG,IAAI,CAAC,YAAY,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,GAAG,CAAC,IAAI,CAAC,YAAY,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YAC7F,GAAG,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;YACjC,GAAG,CAAC,aAAa,GAAG,IAAI,CAAC,aAAa,CAAC;QACzC,CAAC;QAED,IAAI,QAAQ,IAAI,IAAI,CAAC,WAAW,GAAG,CAAC,EAAE,CAAC;YACrC,GAAG,CAAC,SAAS,GAAG,IAAI,CAAC,SAAS,KAAK,MAAM,CAAC,iBAAiB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC;YACjF,GAAG,CAAC,SAAS,GAAG,IAAI,CAAC,SAAS,CAAC;YAC/B,GAAG,CAAC,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,WAAW,CAAC,GAAG,EAAE,CAAC,GAAG,EAAE,CAAC;QAC5E,CAAC;QAED,OAAO,GAAG,CAAC;IACb,CAAC;CACF;AAED;;GAEG;AACH,MAAM,OAAO,eAAe;IAClB,eAAe,GAAG,IAAI,GAAG,EAA0B,CAAC;IACpD,SAAS,GAAG,CAAC,CAAC;IAEtB,GAAG,CAAC,GAAQ;QACV,IAAI,CAAC,SAAS,EAAE,CAAC;QACjB,KAAK,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC7C,IAAI,QAAQ,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;YAC7C,IAAI,CAAC,QAAQ,EAAE,CAAC;gBACd,QAAQ,GAAG,IAAI,cAAc,CAAC,GAAG,CAAC,CAAC;gBACnC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;YAC1C,CAAC;YACD,QAAQ,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACpB,CAAC;IACH,CAAC;IAED,MAAM;QACJ,MAAM,OAAO,GAA0B,EAAE,CAAC;QAC1C,KAAK,MAAM,QAAQ,IAAI,IAAI,CAAC,eAAe,CAAC,MAAM,EAAE,EAAE,CAAC;YACrD,OAAO,CAAC,IAAI,CAAC,QAAQ,CAAC,aAAa,EAAE,CAAC,CAAC;QACzC,CAAC;QAED,OAAO;YACL,SAAS,EAAE,IAAI,CAAC,SAAS;YACzB,YAAY,EAAE,OAAO,CAAC,MAAM;YAC5B,OAAO;SACR,CAAC;IACJ,CAAC;CACF;AAED;;GAEG;AACH,MAAM,UAAU,qBAAqB,CAAC,OAA6B;IACjE,MAAM,OAAO,GAAG,CAAC,QAAQ,EAAE,MAAM,EAAE,UAAU,EAAE,WAAW,EAAE,eAAe,EAAE,eAAe,EAAE,kBAAkB,CAAC,CAAC;IAClH,MAAM,IAAI,GAAG,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE;QACrC,MAAM,OAAO,GAAG,GAAG,YAAY,CAAC,CAAC,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,cAAc,IAAI,CAAC;QACtE,MAAM,WAAW,GAAG,GAAG,YAAY,CAAC,CAAC,CAAC,mBAAmB,CAAC,KAAK,CAAC,CAAC,kBAAkB,IAAI,CAAC;QACxF,IAAI,MAAM,GAAG,GAAG,CAAC;QACjB,IAAI,MAAM,GAAG,GAAG,CAAC;QACjB,IAAI,OAAO,GAAG,GAAG,CAAC;QAElB,IAAI,CAAC,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,SAAS,EAAE,CAAC;YAC9D,MAAM,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;YACvB,MAAM,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;YACvB,OAAO,GAAG,CAAC,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC;QACrF,CAAC;aAAM,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC;YACtB,MAAM,GAAG,GAAG,CAAC,CAAC,SAAS,IAAI,CAAC,QAAQ,CAAC;YACrC,MAAM,GAAG,GAAG,CAAC,CAAC,SAAS,IAAI,CAAC,QAAQ,CAAC;YACrC,IAAI,CAAC,CAAC,SAAS,IAAI,CAAC,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAC1C,OAAO,GAAG,GAAG,CAAC,CAAC,SAAS,CAAC,CAAC,CAAE,CAAC,KAAK,KAAK,CAAC,CAAC,SAAS,CAAC,CAAC,CAAE,CAAC,KAAK,GAAG,CAAC;YAClE,CAAC;QACH,CAAC;QAED,OAAO;YACL,CAAC,CAAC,IAAI;YACN,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,YAAY,KAAK,CAAC,CAAC,YAAY,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,YAAY;YACzE,WAAW;YACX,OAAO;YACP,MAAM;YACN,MAAM;YACN,OAAO;SACR,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,IAAI,GAAG,GAAG,4DAA4D,CAAC;IACvE,GAAG,IAAI,8BAA8B,YAAY,CAAC,OAAO,CAAC,SAAS,CAAC,UAAU,OAAO,CAAC,YAAY,aAAa,CAAC;IAChH,GAAG,IAAI,4DAA4D,CAAC;IACpE,GAAG,IAAI,WAAW,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC;IAClC,GAAG,IAAI,IAAI,CAAC;IACZ,OAAO,GAAG,CAAC;AACb,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,qBAAqB,CAAC,OAA6B;IACjE,IAAI,EAAE,GAAG,8BAA8B,CAAC;IACxC,EAAE,IAAI,qBAAqB,YAAY,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,CAAC;IAC/D,EAAE,IAAI,wBAAwB,OAAO,CAAC,YAAY,MAAM,CAAC;IACzD,EAAE,IAAI,2EAA2E,CAAC;IAClF,EAAE,IAAI,sDAAsD,CAAC;IAE7D,KAAK,MAAM,CAAC,IAAI,OAAO,CAAC,OAAO,EAAE,CAAC;QAChC,MAAM,OAAO,GAAG,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,YAAY,KAAK,CAAC,CAAC,YAAY,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,YAAY,CAAC;QAC1F,MAAM,OAAO,GAAG,GAAG,YAAY,CAAC,CAAC,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,cAAc,IAAI,CAAC;QACtE,MAAM,WAAW,GAAG,GAAG,YAAY,CAAC,CAAC,CAAC,mBAAmB,CAAC,KAAK,CAAC,CAAC,kBAAkB,IAAI,CAAC;QACxF,IAAI,MAAM,GAAG,GAAG,CAAC;QACjB,IAAI,MAAM,GAAG,GAAG,CAAC;QACjB,IAAI,OAAO,GAAG,GAAG,CAAC;QAElB,IAAI,CAAC,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,SAAS,EAAE,CAAC;YAC9D,MAAM,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;YACvB,MAAM,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;YACvB,OAAO,GAAG,CAAC,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC;QACrF,CAAC;aAAM,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC;YACtB,MAAM,GAAG,GAAG,CAAC,CAAC,SAAS,IAAI,CAAC,QAAQ,CAAC;YACrC,MAAM,GAAG,GAAG,CAAC,CAAC,SAAS,IAAI,CAAC,QAAQ,CAAC;YACrC,IAAI,CAAC,CAAC,SAAS,IAAI,CAAC,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAC1C,OAAO,GAAG,GAAG,CAAC,CAAC,SAAS,CAAC,CAAC,CAAE,CAAC,KAAK,KAAK,CAAC,CAAC,SAAS,CAAC,CAAC,CAAE,CAAC,KAAK,GAAG,CAAC;YAClE,CAAC;QACH,CAAC;QAED,EAAE,IAAI,OAAO,CAAC,CAAC,IAAI,UAAU,OAAO,QAAQ,WAAW,MAAM,OAAO,MAAM,MAAM,MAAM,MAAM,MAAM,OAAO,MAAM,CAAC;IAClH,CAAC;IAED,OAAO,EAAE,CAAC;AACZ,CAAC"}

package/dist/analytics/schema-inference.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"schema-inference.d.ts","sourceRoot":"","sources":["../../src/analytics/schema-inference.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAGlF,MAAM,WAAW,qBAAqB;IACpC,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,UAAU,CAAC;IACjB,QAAQ,EAAE,OAAO,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B,aAAa,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACtC,WAAW,EAAE,MAAM,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,oBAAoB;IACnC,gBAAgB,EAAE,MAAM,CAAC;IACzB,OAAO,EAAE,qBAAqB,EAAE,CAAC;CAClC;AAKD;;GAEG;AACH,wBAAgB,qBAAqB,CAAC,KAAK,EAAE,OAAO,GAAG,UAAU,CAuChE;AAED,qBAAa,yBAA0B,YAAW,UAAU,CAAC,oBAAoB,CAAC;IAChF,OAAO,CAAC,SAAS,CAAK;IACtB,OAAO,CAAC,SAAS,CAAS;IAC1B,OAAO,CAAC,WAAW,CAQf;gBAEQ,OAAO,CAAC,EAAE;QAAE,MAAM,CAAC,EAAE,MAAM,CAAA;KAAE;IAIzC,GAAG,CAAC,GAAG,EAAE,GAAG,GAAG,IAAI;IAuCnB,MAAM,IAAI,oBAAoB;CA6F/B"}
+{"version":3,"file":"schema-inference.d.ts","sourceRoot":"","sources":["../../src/analytics/schema-inference.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAGlF,MAAM,WAAW,qBAAqB;IACpC,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,UAAU,CAAC;IACjB,QAAQ,EAAE,OAAO,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B,aAAa,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACtC,WAAW,EAAE,MAAM,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,oBAAoB;IACnC,gBAAgB,EAAE,MAAM,CAAC;IACzB,OAAO,EAAE,qBAAqB,EAAE,CAAC;CAClC;AAKD;;GAEG;AACH,wBAAgB,qBAAqB,CAAC,KAAK,EAAE,OAAO,GAAG,UAAU,CA+ChE;AAED,qBAAa,yBAA0B,YAAW,UAAU,CAAC,oBAAoB,CAAC;IAChF,OAAO,CAAC,SAAS,CAAK;IACtB,OAAO,CAAC,SAAS,CAAS;IAC1B,OAAO,CAAC,WAAW,CAQf;gBAEQ,OAAO,CAAC,EAAE;QAAE,MAAM,CAAC,EAAE,MAAM,CAAA;KAAE;IAIzC,GAAG,CAAC,GAAG,EAAE,GAAG,GAAG,IAAI;IA2CnB,MAAM,IAAI,oBAAoB;CA6F/B"}

package/dist/analytics/schema-inference.js

@@ -10,9 +10,19 @@ export function classifyPrimitiveType(value) {
     }
     if (typeof value === "boolean")
         return "boolean";
+    if (typeof value === "bigint")
+        return "bigint";
     if (typeof value === "number") {
         return Number.isInteger(value) ? "integer" : "number";
     }
+    if (Buffer.isBuffer(value) || value instanceof Uint8Array) {
+        return "binary";
+    }
+    if (typeof value === "object") {
+        if (value instanceof Date)
+            return "datetime";
+        return "json";
+    }
     const str = String(value).trim();
     // Boolean strings
     if (["true", "false"].includes(str.toLowerCase())) {
@@ -56,10 +66,14 @@ export class SchemaInferenceAggregator {
                     counts: {
                         string: 0,
                         integer: 0,
+                        bigint: 0,
                         number: 0,
+                        decimal: 0,
                         boolean: 0,
                         date: 0,
                         datetime: 0,
+                        binary: 0,
+                        json: 0,
                         null: 0,
                         mixed: 0,
                     },

package/dist/analytics/schema-inference.js.map

@@ -1 +1 @@
-{"version":3,"file":"schema-inference.js","sourceRoot":"","sources":["../../src/analytics/schema-inference.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,kBAAkB,EAAE,MAAM,qBAAqB,CAAC;AAmBzD,MAAM,UAAU,GAAG,qBAAqB,CAAC;AACzC,MAAM,cAAc,GAAG,4EAA4E,CAAC;AAEpG;;GAEG;AACH,MAAM,UAAU,qBAAqB,CAAC,KAAc;IAClD,IAAI,KAAK,KAAK,IAAI,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,KAAK,EAAE,EAAE,CAAC;QAC1D,OAAO,MAAM,CAAC;IAChB,CAAC;IAED,IAAI,OAAO,KAAK,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IACjD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,OAAO,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,CAAC;IACxD,CAAC;IAED,MAAM,GAAG,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,CAAC;IAEjC,kBAAkB;IAClB,IAAI,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;QAClD,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,gBAAgB;IAChB,IAAI,SAAS,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACxB,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,uBAAuB;IACvB,IAAI,cAAc,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,6BAA6B,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACxE,OAAO,QAAQ,CAAC;IAClB,CAAC;IAED,kBAAkB;IAClB,IAAI,UAAU,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACzB,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC,CAAC;QACxB,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,EAAE,CAAC;YAAE,OAAO,MAAM,CAAC;IAChD,CAAC;IAED,IAAI,cAAc,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QAC7B,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC,CAAC;QACxB,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,EAAE,CAAC;YAAE,OAAO,UAAU,CAAC;IACpD,CAAC;IAED,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,MAAM,OAAO,yBAAyB;IAC5B,SAAS,GAAG,CAAC,CAAC;IACd,SAAS,CAAS;IAClB,WAAW,GAAG,IAAI,GAAG,EAQ1B,CAAC;IAEJ,YAAY,OAA6B;QACvC,IAAI,CAAC,SAAS,GAAG,OAAO,EAAE,MAAM,IAAI,MAAM,CAAC,iBAAiB,CAAC;IAC/D,CAAC;IAED,GAAG,CAAC,GAAQ;QACV,IAAI,IAAI,CAAC,SAAS,IAAI,IAAI,CAAC,SAAS;YAAE,OAAO;QAC7C,IAAI,CAAC,SAAS,EAAE,CAAC;QAEjB,KAAK,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC7C,IAAI,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;gBAC/B,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,EAAE;oBACxB,MAAM,EAAE;wBACN,MAAM,EAAE,CAAC;wBACT,OAAO,EAAE,CAAC;wBACV,MAAM,EAAE,CAAC;wBACT,OAAO,EAAE,CAAC;wBACV,IAAI,EAAE,CAAC;wBACP,QAAQ,EAAE,CAAC;wBACX,IAAI,EAAE,CAAC;wBACP,KAAK,EAAE,CAAC;qBACT;oBACD,cAAc,EAAE,EAAE;oBAClB,SAAS,EAAE,CAAC;oBACZ,UAAU,EAAE,CAAC;iBACd,CAAC,CAAC;YACL,CAAC;YAED,MAAM,IAAI,GAAG,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,CAAE,CAAC;YACxC,IAAI,CAAC,UAAU,EAAE,CAAC;YAElB,MAAM,KAAK,GAAG,qBAAqB,CAAC,GAAG,CAAC,CAAC;YACzC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;YACrB,IAAI,KAAK,KAAK,MAAM,EAAE,CAAC;gBACrB,IAAI,CAAC,SAAS,EAAE,CAAC;YACnB,CAAC;iBAAM,CAAC;gBACN,MAAM,GAAG,GAAG,kBAAkB,CAAC,GAAG,CAAC,CAAC;gBACpC,IAAI,GAAG,EAAE,CAAC;oBACR,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;gBACjE,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;IAED,MAAM;QACJ,MAAM,OAAO,GAA4B,EAAE,CAAC;QAE5C,KAAK,MAAM,CAAC,GAAG,EAAE,IAAI,CAAC,IAAI,IAAI,CAAC,WAAW,CAAC,OAAO,EAAE,EAAE,CAAC;YACrD,MAAM,YAAY,GAAG,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC;YACtD,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,GAAG,CAAC,CAAC;YAEpC,IAAI,YAAY,KAAK,CAAC,EAAE,CAAC;gBACvB,OAAO,CAAC,IAAI,CAAC;oBACX,IAAI,EAAE,GAAG;oBACT,IAAI,EAAE,MAAM;oBACZ,QAAQ,EAAE,IAAI;oBACd,UAAU,EAAE,GAAG;oBACf,aAAa,EAAE,EAAE,IAAI,EAAE,GAAG,EAAE;oBAC5B,WAAW,EAAE,IAAI,CAAC,UAAU;oBAC5B,SAAS,EAAE,IAAI,CAAC,SAAS;iBAC1B,CAAC,CAAC;gBACH,SAAS;YACX,CAAC;YAED,mCAAmC;YACnC,MAAM,SAAS,GAA2B,EAAE,CAAC;YAC7C,IAAI,YAAY,GAAe,QAAQ,CAAC;YACxC,IAAI,aAAa,GAAG,CAAC,CAAC;YAEtB,MAAM,UAAU,GAAiB;gBAC/B,SAAS;gBACT,QAAQ;gBACR,SAAS;gBACT,UAAU;gBACV,MAAM;gBACN,QAAQ;aACT,CAAC;YAEF,KAAK,MAAM,CAAC,IAAI,UAAU,EAAE,CAAC;gBAC3B,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,YAAY,CAAC,GAAG,GAAG,CAAC;gBAClD,IAAI,GAAG,GAAG,CAAC,EAAE,CAAC;oBACZ,SAAS,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,GAAG,EAAE,CAAC,GAAG,EAAE,CAAC;gBAC3C,CAAC;gBACD,IAAI,GAAG,GAAG,aAAa,EAAE,CAAC;oBACxB,aAAa,GAAG,GAAG,CAAC;oBACpB,YAAY,GAAG,CAAC,CAAC;gBACnB,CAAC;YACH,CAAC;YAED,0DAA0D;YAC1D,IACE,YAAY,KAAK,SAAS;gBAC1B,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC;gBACzB,CAAC,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,GAAG,YAAY,IAAI,IAAI,EACvE,CAAC;gBACD,YAAY,GAAG,QAAQ,CAAC;gBACxB,aAAa,GAAG,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,GAAG,YAAY,CAAC,GAAG,GAAG,CAAC;YAC1F,CAAC;YAED,IAAI,YAAY,GAAe,YAAY,CAAC;YAC5C,IAAI,UAAU,GAAG,IAAI,CAAC,KAAK,CAAC,aAAa,GAAG,EAAE,CAAC,GAAG,EAAE,CAAC;YAErD,IAAI,aAAa,GAAG,EAAE,IAAI,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAC5D,YAAY,GAAG,OAAO,CAAC;YACzB,CAAC;YAED,0BAA0B;YAC1B,IAAI,QAAkC,CAAC;YACvC,IAAI,kBAAsC,CAAC;YAE3C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,cAAc,CAAC,EAAE,CAAC;gBAC/D,MAAM,MAAM,GAAG,CAAC,KAAK,GAAG,YAAY,CAAC,GAAG,GAAG,CAAC;gBAC5C,IAAI,MAAM,IAAI,EAAE,EAAE,CAAC;oBACjB,QAAQ,GAAG,GAAmB,CAAC;oBAC/B,kBAAkB,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,GAAG,EAAE,CAAC,GAAG,EAAE,CAAC;oBAClD,MAAM;gBACR,CAAC;YACH,CAAC;YAED,OAAO,CAAC,IAAI,CAAC;gBACX,IAAI,EAAE,GAAG;gBACT,IAAI,EAAE,YAAY;gBAClB,QAAQ;gBACR,UAAU;gBACV,QAAQ;gBACR,kBAAkB;gBAClB,aAAa,EAAE,SAAS;gBACxB,WAAW,EAAE,IAAI,CAAC,UAAU;gBAC5B,SAAS,EAAE,IAAI,CAAC,SAAS;aAC1B,CAAC,CAAC;QACL,CAAC;QAED,OAAO;YACL,gBAAgB,EAAE,IAAI,CAAC,SAAS;YAChC,OAAO;SACR,CAAC;IACJ,CAAC;CACF"}
+{"version":3,"file":"schema-inference.js","sourceRoot":"","sources":["../../src/analytics/schema-inference.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,kBAAkB,EAAE,MAAM,qBAAqB,CAAC;AAmBzD,MAAM,UAAU,GAAG,qBAAqB,CAAC;AACzC,MAAM,cAAc,GAAG,4EAA4E,CAAC;AAEpG;;GAEG;AACH,MAAM,UAAU,qBAAqB,CAAC,KAAc;IAClD,IAAI,KAAK,KAAK,IAAI,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,KAAK,EAAE,EAAE,CAAC;QAC1D,OAAO,MAAM,CAAC;IAChB,CAAC;IAED,IAAI,OAAO,KAAK,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IACjD,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,QAAQ,CAAC;IAC/C,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,OAAO,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,CAAC;IACxD,CAAC;IACD,IAAI,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,KAAK,YAAY,UAAU,EAAE,CAAC;QAC1D,OAAO,QAAQ,CAAC;IAClB,CAAC;IACD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,KAAK,YAAY,IAAI;YAAE,OAAO,UAAU,CAAC;QAC7C,OAAO,MAAM,CAAC;IAChB,CAAC;IAED,MAAM,GAAG,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,CAAC;IAEjC,kBAAkB;IAClB,IAAI,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;QAClD,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,gBAAgB;IAChB,IAAI,SAAS,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACxB,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,uBAAuB;IACvB,IAAI,cAAc,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,6BAA6B,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACxE,OAAO,QAAQ,CAAC;IAClB,CAAC;IAED,kBAAkB;IAClB,IAAI,UAAU,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACzB,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC,CAAC;QACxB,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,EAAE,CAAC;YAAE,OAAO,MAAM,CAAC;IAChD,CAAC;IAED,IAAI,cAAc,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QAC7B,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC,CAAC;QACxB,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,EAAE,CAAC;YAAE,OAAO,UAAU,CAAC;IACpD,CAAC;IAED,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,MAAM,OAAO,yBAAyB;IAC5B,SAAS,GAAG,CAAC,CAAC;IACd,SAAS,CAAS;IAClB,WAAW,GAAG,IAAI,GAAG,EAQ1B,CAAC;IAEJ,YAAY,OAA6B;QACvC,IAAI,CAAC,SAAS,GAAG,OAAO,EAAE,MAAM,IAAI,MAAM,CAAC,iBAAiB,CAAC;IAC/D,CAAC;IAED,GAAG,CAAC,GAAQ;QACV,IAAI,IAAI,CAAC,SAAS,IAAI,IAAI,CAAC,SAAS;YAAE,OAAO;QAC7C,IAAI,CAAC,SAAS,EAAE,CAAC;QAEjB,KAAK,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC7C,IAAI,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;gBAC/B,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,EAAE;oBACxB,MAAM,EAAE;wBACN,MAAM,EAAE,CAAC;wBACT,OAAO,EAAE,CAAC;wBACV,MAAM,EAAE,CAAC;wBACT,MAAM,EAAE,CAAC;wBACT,OAAO,EAAE,CAAC;wBACV,OAAO,EAAE,CAAC;wBACV,IAAI,EAAE,CAAC;wBACP,QAAQ,EAAE,CAAC;wBACX,MAAM,EAAE,CAAC;wBACT,IAAI,EAAE,CAAC;wBACP,IAAI,EAAE,CAAC;wBACP,KAAK,EAAE,CAAC;qBACT;oBACD,cAAc,EAAE,EAAE;oBAClB,SAAS,EAAE,CAAC;oBACZ,UAAU,EAAE,CAAC;iBACd,CAAC,CAAC;YACL,CAAC;YAED,MAAM,IAAI,GAAG,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,CAAE,CAAC;YACxC,IAAI,CAAC,UAAU,EAAE,CAAC;YAElB,MAAM,KAAK,GAAG,qBAAqB,CAAC,GAAG,CAAC,CAAC;YACzC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;YACrB,IAAI,KAAK,KAAK,MAAM,EAAE,CAAC;gBACrB,IAAI,CAAC,SAAS,EAAE,CAAC;YACnB,CAAC;iBAAM,CAAC;gBACN,MAAM,GAAG,GAAG,kBAAkB,CAAC,GAAG,CAAC,CAAC;gBACpC,IAAI,GAAG,EAAE,CAAC;oBACR,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;gBACjE,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;IAED,MAAM;QACJ,MAAM,OAAO,GAA4B,EAAE,CAAC;QAE5C,KAAK,MAAM,CAAC,GAAG,EAAE,IAAI,CAAC,IAAI,IAAI,CAAC,WAAW,CAAC,OAAO,EAAE,EAAE,CAAC;YACrD,MAAM,YAAY,GAAG,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC;YACtD,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,GAAG,CAAC,CAAC;YAEpC,IAAI,YAAY,KAAK,CAAC,EAAE,CAAC;gBACvB,OAAO,CAAC,IAAI,CAAC;oBACX,IAAI,EAAE,GAAG;oBACT,IAAI,EAAE,MAAM;oBACZ,QAAQ,EAAE,IAAI;oBACd,UAAU,EAAE,GAAG;oBACf,aAAa,EAAE,EAAE,IAAI,EAAE,GAAG,EAAE;oBAC5B,WAAW,EAAE,IAAI,CAAC,UAAU;oBAC5B,SAAS,EAAE,IAAI,CAAC,SAAS;iBAC1B,CAAC,CAAC;gBACH,SAAS;YACX,CAAC;YAED,mCAAmC;YACnC,MAAM,SAAS,GAA2B,EAAE,CAAC;YAC7C,IAAI,YAAY,GAAe,QAAQ,CAAC;YACxC,IAAI,aAAa,GAAG,CAAC,CAAC;YAEtB,MAAM,UAAU,GAAiB;gBAC/B,SAAS;gBACT,QAAQ;gBACR,SAAS;gBACT,UAAU;gBACV,MAAM;gBACN,QAAQ;aACT,CAAC;YAEF,KAAK,MAAM,CAAC,IAAI,UAAU,EAAE,CAAC;gBAC3B,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,YAAY,CAAC,GAAG,GAAG,CAAC;gBAClD,IAAI,GAAG,GAAG,CAAC,EAAE,CAAC;oBACZ,SAAS,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,GAAG,EAAE,CAAC,GAAG,EAAE,CAAC;gBAC3C,CAAC;gBACD,IAAI,GAAG,GAAG,aAAa,EAAE,CAAC;oBACxB,aAAa,GAAG,GAAG,CAAC;oBACpB,YAAY,GAAG,CAAC,CAAC;gBACnB,CAAC;YACH,CAAC;YAED,0DAA0D;YAC1D,IACE,YAAY,KAAK,SAAS;gBAC1B,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC;gBACzB,CAAC,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,GAAG,YAAY,IAAI,IAAI,EACvE,CAAC;gBACD,YAAY,GAAG,QAAQ,CAAC;gBACxB,aAAa,GAAG,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,GAAG,YAAY,CAAC,GAAG,GAAG,CAAC;YAC1F,CAAC;YAED,IAAI,YAAY,GAAe,YAAY,CAAC;YAC5C,IAAI,UAAU,GAAG,IAAI,CAAC,KAAK,CAAC,aAAa,GAAG,EAAE,CAAC,GAAG,EAAE,CAAC;YAErD,IAAI,aAAa,GAAG,EAAE,IAAI,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAC5D,YAAY,GAAG,OAAO,CAAC;YACzB,CAAC;YAED,0BAA0B;YAC1B,IAAI,QAAkC,CAAC;YACvC,IAAI,kBAAsC,CAAC;YAE3C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,cAAc,CAAC,EAAE,CAAC;gBAC/D,MAAM,MAAM,GAAG,CAAC,KAAK,GAAG,YAAY,CAAC,GAAG,GAAG,CAAC;gBAC5C,IAAI,MAAM,IAAI,EAAE,EAAE,CAAC;oBACjB,QAAQ,GAAG,GAAmB,CAAC;oBAC/B,kBAAkB,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,GAAG,EAAE,CAAC,GAAG,EAAE,CAAC;oBAClD,MAAM;gBACR,CAAC;YACH,CAAC;YAED,OAAO,CAAC,IAAI,CAAC;gBACX,IAAI,EAAE,GAAG;gBACT,IAAI,EAAE,YAAY;gBAClB,QAAQ;gBACR,UAAU;gBACV,QAAQ;gBACR,kBAAkB;gBAClB,aAAa,EAAE,SAAS;gBACxB,WAAW,EAAE,IAAI,CAAC,UAAU;gBAC5B,SAAS,EAAE,IAAI,CAAC,SAAS;aAC1B,CAAC,CAAC;QACL,CAAC;QAED,OAAO;YACL,gBAAgB,EAAE,IAAI,CAAC,SAAS;YAChC,OAAO;SACR,CAAC;IACJ,CAAC;CACF"}

package/dist/cli/commands/clean.d.ts

@@ -0,0 +1,15 @@
+import { UnifiedPipelineCliOptions } from "./pipeline.js";
+export interface CleanCommandOptions extends UnifiedPipelineCliOptions {
+    trim?: boolean;
+    nullValues?: string;
+    fillNulls?: string | string[];
+    coerce?: boolean;
+    stripChars?: string;
+    case?: "lower" | "upper" | "title";
+    dropEmptyRows?: boolean;
+}
+/**
+ * Handles `rowpipe clean [input] [options]`
+ */
+export declare function cleanCommand(inputPath?: string, options?: CleanCommandOptions): Promise<void>;
+//# sourceMappingURL=clean.d.ts.map

package/dist/cli/commands/clean.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"clean.d.ts","sourceRoot":"","sources":["../../../src/cli/commands/clean.ts"],"names":[],"mappings":"AASA,OAAO,EAA8B,yBAAyB,EAAE,MAAM,eAAe,CAAC;AAEtF,MAAM,WAAW,mBAAoB,SAAQ,yBAAyB;IACpE,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,SAAS,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC9B,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,OAAO,GAAG,OAAO,GAAG,OAAO,CAAC;IACnC,aAAa,CAAC,EAAE,OAAO,CAAC;CACzB;AAED;;GAEG;AACH,wBAAsB,YAAY,CAChC,SAAS,SAAM,EACf,OAAO,GAAE,mBAAwB,GAChC,OAAO,CAAC,IAAI,CAAC,CAqGf"}