df-script 1.4.0 → 1.6.0

package/README.md

@@ -1,9 +1,39 @@
 # 🚀 DFScript
 
+[![GitHub Repository](https://img.shields.io/badge/GitHub-Repository-blue?style=for-the-badge&logo=github)](https://github.com/trentamorris/df-script)
+[![Donate](https://img.shields.io/badge/Donate-Support-green?style=for-the-badge)](DONATIONS.md)
+
 DFScript is a lightweight, high-performance, and **zero-dependency** data analysis library for TypeScript and JavaScript. Heavily inspired by modern dataframe libraries like **Polars** and **Pandas**, DFScript brings a robust, expression-based columnar data processing engine directly to the JavaScript ecosystem.
 
 With optimized columnar storage under the hood, DFScript enables you to build clean, maintainable, and type-safe data pipelines using a declarative expression API.
 
+## 💡 Why DFScript?
+
+Like many developers, I spent years working on frontend codebases filled with deeply nested, unoptimized data processing loops. We've all seen them: chained `.map()`, `.filter()`, `.sort()`, and `.forEach()` loops processing arrays of objects. Each step in the chain allocates new intermediate arrays, triggers garbage collection thrashing, and operates on row-based layouts that slow down browser main threads.
+
+When you need to perform complex analytical transformations on the client-side—handling missing values, formatting timezone-aware datetimes, parsing decimals, or computing rolling averages—relying on standard JavaScript array methods quickly becomes a performance and maintenance nightmare. There had to be a better, cleaner, and faster way to express these pipelines. I realized that if nobody else had built a lightweight, zero-dependency columnar expression engine for the JavaScript ecosystem yet, why not roll up my sleeves and build it myself?
+
+DFScript was born from that frustration. It brings Polars-like columnar execution, strict domain namespaces (`.str`, `.dt`, `.arr`, `.struct`), and lazy/declarative expression compilation to JavaScript. Under the hood, it avoids intermediate array allocations and uses highly optimized loops with cached lengths, so your frontend code stays clean, type-safe, and incredibly fast.
+
+---
+
+## 🗺️ Table of Contents
+
+- [✨ Key Features](#-key-features)
+- [⚙️ Compatibility & Design Principles](#-compatibility--design-principles)
+- [🤝 Contributing & Collective Wisdom](#-contributing--collective-wisdom)
+- [📦 Installation](#-installation)
+- [🚀 Quick Start](#-quick-start)
+- [📖 Core Concepts](#-core-concepts)
+- [🛠️ DataFrame API Reference](#-dataframe-api-reference)
+- [📂 File / Data I/O](#-file--data-io)
+- [🧮 Expressions API Reference](#-expressions-api-reference)
+- [📂 Namespaces](#-namespaces)
+- [🪟 Window & Rolling Expressions](#-window--rolling-expressions)
+- [🛡️ Typing and Schema Registry](#-typing-and-schema-registry)
+- [🧑‍💻 Contributing & Development](#-contributing--development)
+- [📄 License](#-license)
+
 ---
 
 ## ✨ Key Features
@@ -14,13 +44,34 @@ With optimized columnar storage under the hood, DFScript enables you to build cl
 - 📂 **Strict Namespaces** — Clear API organization for specific domains:
   - `.str` for advanced string manipulations.
   - `.dt` for microsecond-precision datetimes, timezones, and duration calculations.
-  - `.list` for robust array/list column operations.
+  - `.arr` for robust array/list column operations.
+  - `.struct` for nested object and struct manipulation.
 - 🪟 **Analytical Window Functions** — Windowing (`over()`), cumulative aggregations (`cum_sum()`, `cum_max()`), and rolling metrics (`rolling_mean()`, `rolling_std()`).
 - 🛠️ **Relational Operations** — Rich, high-speed joins, pivots, unpivots, vertical/horizontal concatenations, and group-by aggregations.
 - 🛡️ **Defensive & Type-Safe** — Native type-coercion, robust null-safety, and strict schema validation.
 
 ---
 
+## ⚙️ Compatibility & Design Principles
+
+DFScript is designed with a **low-abstraction, zero-dependency** philosophy to guarantee maximum compatibility, predictability, and runtime performance:
+
+- 📦 **Zero External Dependencies** — Lightweight footprint with zero runtime overhead or supply chain vulnerabilities.
+- 🌐 **Universal Compatibility** — Works out-of-the-box in any JavaScript/TypeScript environment, including Node.js, Deno, Bun, web browsers, and cloud/edge workers.
+- 🧱 **Built-in Standards** — Prioritizes native, built-in APIs (like standard `Date`, `Intl` formatting, and `TextEncoder`) and standard arrays rather than custom wrappers or heavy runtime abstractions.
+- ⚡ **Optimized Execution Paths** — Under the hood, performance-critical code avoids higher-level array iterators and short-lived intermediate allocations in favor of simple, fast `for` and `while` loops with cached lengths, keeping garbage collection overhead to an absolute minimum.
+- 🔄 **Easy Transpilation** — Relies strictly on low-level native operations, making it fully compatible with older environments (like ES6 or even ES5) without requiring complex polyfills or modern engine-specific features.
+
+---
+
+## 🤝 Contributing & Collective Wisdom
+
+We don’t pretend to have encountered every localized date format, database quirk, or environment-specific edge case. DFScript is built on the belief that software correctness is a collective endeavor. 
+
+If you run into an unsupported edge case or unexpected behavior in any of our functions, we want to hear about it. Help us harden this engine by opening a GitHub Issue or submitting a PR—every report makes the library more robust for everyone.
+
+---
+
 ## 📦 Installation
 
 Install DFScript using your favorite package manager:
@@ -43,10 +94,10 @@ pnpm add df-script
 Here is a quick example showing how to load data, run expressions, perform aggregations, and compute rolling statistics.
 
 ```typescript
-import { $tbl } from "df-script";
+import { $df } from "df-script";
 
 // 1. Create a DataFrame with structured data and automatic schema inference
-const df = $tbl.data([
+const df = $df.data([
   { id: 1, name: "Alice", join_date: "2026-01-15", sales: 1200.50, tags: ["sales", "east"] },
   { id: 2, name: "Bob", join_date: "2026-02-20", sales: 850.00, tags: ["support", "west"] },
   { id: 3, name: "Charlie", join_date: "2026-03-05", sales: 2300.00, tags: ["sales", "north"] },
@@ -55,11 +106,11 @@ const df = $tbl.data([
 
 // 2. Select columns, transform strings, format dates, and fill missing values
 const processedDf = df.select(
-  $tbl.col("id"),
-  $tbl.col("name").str.upper().alias("NAME_UPPER"),
-  $tbl.col("join_date").str.to_datetime().dt.year().alias("join_year"),
-  $tbl.col("sales").add(500).alias("sales_adjusted"),
-  $tbl.col("tags").list.lengths().alias("tag_count")
+  $df.col("id"),
+  $df.col("name").str.upper().alias("NAME_UPPER"),
+  $df.col("join_date").str.to_datetime().dt.year().alias("join_year"),
+  $df.col("sales").add(500).alias("sales_adjusted"),
+  $df.col("tags").arr.lengths().alias("tag_count")
 );
 
 console.log(processedDf.to_dicts());
@@ -77,14 +128,23 @@ console.log(processedDf.to_dicts());
 
 ## 📖 Core Concepts
 
-### The `$tbl` Entry Point
+### The `$df` Entry Point
 
-DFScript uses the `$tbl` namespace to bootstrap DataFrames, refer to columns, and access general types.
+DFScript uses the `$df` namespace to bootstrap DataFrames, refer to columns, build expressions, and access data types.
 
-- `$tbl.data(dataRowsOrCols, schema?)`: Instantiates a new `DataFrame`.
-- `$tbl.col(name)`: Creates a column reference expression.
-- `$tbl.all()`: Selects all columns in the DataFrame.
-- `$tbl.DataType`: Direct access to the `DataTypeRegistry` for schema specification.
+- `$df.data(dataRowsOrCols, schema?)`: Instantiates a new `DataFrame`.
+- `$df.read_json(content, options?)`: Reads JSON/NDJSON content into a new `DataFrame`.
+- `$df.read_csv(content, options?)`: Reads CSV content into a new `DataFrame`.
+- `$df.col(name)`: Creates a column reference expression.
+- `$df.all()`: Selects all columns in the DataFrame.
+- `$df.exclude(columns)`: Creates an expression matching all columns except the specified ones.
+- `$df.coalesce(...exprs)`: Returns the first non-null value among columns or literal expressions.
+- `$df.lit(val)`: Explicitly wraps a raw value into a literal expression.
+- `$df.when(predicate).then(value)...otherwise(value)`: Constructs a conditional expression (when-then-otherwise chain).
+- `$df.implode(column)`: Aggregates a column's rows (or grouped values) into a list.
+- `$df.seq_range(value, options?)`: Generates a sequence range of values.
+- `$df.element()`: References the current array element within an `.arr.eval(...)` expression.
+- `$df.DataType`: Direct access to the `DataTypeRegistry` for schema specification.
 
 ### DataFrames vs. Columns
 
@@ -96,7 +156,7 @@ DFScript uses the `$tbl` namespace to bootstrap DataFrames, refer to columns, an
 ## 🛠️ DataFrame API Reference
 
 ### 1. Transformations & Projection
-- **`select(...exprs)`**: Projects columns. Supports strings, raw column names, `$tbl.col(...)` expressions, and `$tbl.all()`.
+- **`select(...exprs)`**: Projects columns. Supports strings, raw column names, `$df.col(...)` expressions, and `$df.all()`.
 - **`with_columns(...exprs)`**: Adds or overrides columns. Accepts expressions, strings, or options objects mapping keys to values/expressions.
 - **`drop(...names)`**: Drops one or more columns from the DataFrame.
 - **`rename(mapping)`**: Renames columns using a `{ oldName: newName }` object.
@@ -107,13 +167,14 @@ DFScript uses the `$tbl` namespace to bootstrap DataFrames, refer to columns, an
 - **`limit(n, options?)`**: Returns the first `n` rows. Options include `offset` and direction `from: "start" | "end"`.
 - **`head(n)`** / **`tail(n)`**: Shortcuts for `limit` from the start or end of the DataFrame.
 - **`slice(start, end?)`**: Extract a subset of rows using standard index slicing.
+- **`gather(indices, options?)`**: Gathers rows at specified indices. Supports single index, arrays of indices, and negative indexing. Options include `{ null_on_oob?: boolean }` (default: `false` which throws an error on out-of-bounds indices; if `true`, out-of-bounds indices result in `null` values).
 
 ### 3. Sorting
 - **`sort({ by, descending?, nullsLast?, custom? })`**: Sorts rows. Supports single or multiple columns/expressions, custom descending configurations per column, custom null sorting rules, and custom comparator functions.
 
 ### 4. Grouping & Aggregations
 - **`groupby(keys)`**: Groups the data by one or more columns, returning a `GroupedData` object.
-- **`GroupedData.agg(...exprs)`**: Run aggregations on grouped data (e.g. `$tbl.col("sales").sum()`).
+- **`GroupedData.agg(...exprs)`**: Run aggregations on grouped data (e.g. `$df.col("sales").sum()`).
 
 ### 5. Reshaping & Joining
 - **`join(other, on, how, suffixes?)`**: Merges two DataFrames on join keys. Supported join types: `"inner" | "left" | "right" | "outer"`.
@@ -123,6 +184,50 @@ DFScript uses the `$tbl` namespace to bootstrap DataFrames, refer to columns, an
 
 ---
 
+## 📂 File / Data I/O
+
+DFScript provides helpers to serialize and parse data formats like JSON and CSV.
+
+### Reading Data
+- **`$df.read_json(content, options?)`**: Reads a JSON array or Newline Delimited JSON (NDJSON) string and loads it into a new DataFrame.
+  ```typescript
+  import { $df } from "df-script";
+
+  // Read standard JSON array
+  const df = $df.read_json('[{"id": 1, "name": "Alice"}]');
+
+  // Read Newline Delimited JSON (NDJSON)
+  const dfNdjson = $df.read_json('{"id": 1}\n{"id": 2}', { format: "ndjson" });
+  ```
+- **`$df.read_csv(content, options?)`**: Reads a CSV string and loads it into a new DataFrame, with automatic data type inference.
+  ```typescript
+  import { $df } from "df-script";
+
+  const csvContent = "id,name,active\n1,Alice,true\n2,Bob,false";
+  const df = $df.read_csv(csvContent, {
+    separator: ",",
+    hasHeader: true,
+    inferSchema: true
+  });
+  ```
+
+### Writing Data
+- **`df.write_json(file?, options?)`**: Serializes a DataFrame into a JSON or NDJSON string. If a file path or writable stream/object (with a `.write` method) is provided, writes/streams the content as a side-effect. Always returns the serialized string.
+  ```typescript
+  // Write to a file and get the string
+  const jsonStr = df.write_json("output.json");
+  ```
+- **`df.write_csv(file?, options?)`**: Serializes a DataFrame into a CSV string. Supports options for headers, custom separators, quote styles, float precision, and BOM. If a file path or writable stream/object (with a `.write` method) is provided, writes/streams the content as a side-effect. Always returns the serialized string.
+  ```typescript
+  // Serialize to a CSV string
+  const csvStr = df.write_csv();
+
+  // Write to a file with custom separator
+  df.write_csv("output.csv", { separator: ";" });
+  ```
+
+---
+
 ## 🧮 Expressions API Reference
 
 All column expressions inherit from `ExprBase` and support standard operators.
@@ -148,6 +253,23 @@ Chained mathematical functions execute cleanly with built-in null-safety (Kleene
 - `.first()`, `.last()`
 - `.any()`, `.all()`, `.any_null()`, `.all_null()`, `.n_unique()`
 
+### 🔀 Control Flow & Conditionals
+Construct dynamic `CASE WHEN` branches using the `$df.when` API:
+```typescript
+import { $df } from "df-script";
+
+df.select(
+  $df.col("sales"),
+  $df.when($df.col("sales").gt(2000)).then("High Performance")
+     .when($df.col("sales").gt(1000)).then("Standard Performance")
+     .otherwise("Low Performance")
+     .alias("sales_category")
+);
+```
+- `$df.when(predicate).then(value)`: Starts a conditional evaluation.
+- `.when(predicate).then(value)`: Chains additional conditions.
+- `.otherwise(value)`: Specifies the fallback value when no conditions match (returns a complete `ColumnExpr`).
+
 ---
 
 ## 📂 Namespaces
@@ -157,29 +279,52 @@ To maintain a clean and uncluttered API namespace, specific data transforms are
 ### 🔤 String Operations (`.str`)
 Available on any expression via `.str`:
 ```typescript
-$tbl.col("name").str.lower()
-$tbl.col("code").str.starts_with("A")
-$tbl.col("description").str.replace(/foo/i, "bar")
+$df.col("name").str.lower()
+$df.col("code").str.starts_with("A")
+$df.col("description").str.replace(/foo/i, "bar")
 ```
 - **Methods**: `lower()`, `upper()`, `len()`, `len_bytes()`, `len_chars()`, `trim()`, `trim_start()`, `trim_end()`, `starts_with(prefix)`, `ends_with(suffix)`, `contains(pattern)`, `replace(pattern, repl)`, `replace_all(pattern, repl)`, `slice(offset, length?)`, `split(delimiter)`, `explode()`, `reverse()`, `lpad(w, f)`, `rpad(w, f)`, `zfill(w)`, `strip_chars(chars?)`, `strip_chars_start(chars?)`, `strip_chars_end(chars?)`, `strip_prefix(pfx)`, `strip_suffix(sfx)`, `to_titlecase()`, `strptime(format, strict?)`, `to_integer()`, `to_decimal(p, s)`, `to_date()`, `to_datetime()`, `to_time()`.
 
 ### 📅 Temporal Operations (`.dt`)
 Available on datetime or duration values via `.dt`:
 ```typescript
-$tbl.col("timestamp").dt.year()
-$tbl.col("timestamp").dt.strftime("%Y-%m-%d %H:%M:%S")
-$tbl.col("duration").dt.total_seconds()
+$df.col("timestamp").dt.year()
+$df.col("timestamp").dt.strftime("%Y-%m-%d %H:%M:%S")
+$df.col("duration").dt.total_seconds()
 ```
 - **Datetime Methods**: `year()`, `month()`, `day()`, `hour()`, `minute()`, `second()`, `millisecond()`, `microsecond()`, `nanosecond()`, `weekday()`, `week()`, `quarter()`, `century()`, `millennium()`, `ordinal_day()`, `is_leap_year()`, `month_start()`, `month_end()`, `date()`, `time()`, `datetime()`, `epoch(unit)`, `timestamp(unit)`, `strftime(format, locale?)`.
 - **Duration Methods**: `total_days()`, `total_hours()`, `total_minutes()`, `total_seconds()`, `total_milliseconds()`, `total_microseconds()`, `total_nanoseconds()`.
 
-### 📊 List Operations (`.list`)
-Available on arrays or lists via `.list`:
+### 📊 Array/List Operations (`.arr`)
+Available on any array or list column expression via `.arr`:
 ```typescript
-$tbl.col("tags").list.contains("vip")
-$tbl.col("matrix").list.get(2)
+$df.col("tags").arr.contains("vip")
+$df.col("matrix").arr.get(2)
+
+// Element-wise manipulation inside arrays:
+$df.col("numbers").arr.eval(element().mul(2)).alias("numbers_doubled")
+$df.col("tags").arr.eval(element().str.to_uppercase()).alias("upper_tags")
+```
+- **Methods**: `lengths()`, `len()`, `get(idx, null_on_oob?)`, `first(null_on_oob?)`, `last(null_on_oob?)`, `gather(indices, null_on_oob?)`, `gather_every(n, offset?)`, `slice(offset, length?)`, `contains(item)`, `count_matches(item)`, `join(separator)`, `sort(descending?)`, `reverse()`, `unique()`, `sum()`, `mean()`, `median()`, `mode()`, `min()`, `max()`, `eval(expr)`.
+
+### 🗃️ Struct/Object Operations (`.struct`)
+Available on any struct or nested object column expression via `.struct`. You can access fields dynamically via properties or explicit methods:
+```typescript
+// Sibling fields access via Proxy
+$df.col("address").struct.city.alias("city")
+
+// Or using the explicit field method
+$df.col("address").struct.field("city")
 ```
-- **Methods**: `lengths()`, `len()`, `get(idx, null_on_oob?)`, `first(null_on_oob?)`, `last(null_on_oob?)`, `gather(indices, null_on_oob?)`, `gather_every(n, offset?)`, `slice(offset, length?)`, `contains(item)`, `count_matches(item)`, `join(separator)`, `sort(descending?)`, `reverse()`, `unique()`, `sum()`, `mean()`, `median()`, `mode()`, `min()`, `max()`.
+- **Methods**:
+  - `field(name)`: Accesses a field within the struct.
+  - `rename_fields(mapping)`: Renames fields in the struct based on a `{ oldKey: newKey }` mapping.
+  - `with_fields(fields)`: Adds or overrides fields in the struct. Accepts an array of aliased expressions or an object.
+  - `unnest()`: Expands the fields of the struct into individual top-level columns in a select projection.
+    ```typescript
+    // Flattens the address struct into "city", "state", etc. at the top-level
+    df.select($df.col("address").struct.unnest())
+    ```
 
 ---
 
@@ -190,11 +335,11 @@ DFScript provides full support for analytic partition window operations using `.
 ```typescript
 // Calculate partition cumulative sums and row numbers
 df.select(
-  $tbl.col("department"),
-  $tbl.col("sales"),
-  $tbl.col("sales").sum().over("department").alias("dept_total_sales"),
-  $tbl.col("sales").cum_sum().over("department").alias("dept_running_sales"),
-  $tbl.all().row_number().over("department").alias("dept_rank")
+  $df.col("department"),
+  $df.col("sales"),
+  $df.col("sales").sum().over("department").alias("dept_total_sales"),
+  $df.col("sales").cum_sum().over("department").alias("dept_running_sales"),
+  $df.all().row_number().over("department").alias("dept_rank")
 );
 ```
 
@@ -230,16 +375,36 @@ Apply moving calculations over a fixed window size:
 You can optionally declare schemas to enforce precise data types and automatic type coercion during construction.
 
 ```typescript
-import { $tbl } from "df-script";
+import { $df } from "df-script";
 
 const schema = {
-  id: $tbl.DataType.Int32,
-  price: $tbl.DataType.Decimal(10, 2),
-  active: $tbl.DataType.Boolean,
-  created_at: $tbl.DataType.Datetime
+  id: $df.DataType.Int32,
+  price: $df.DataType.Decimal(10, 2),
+  active: $df.DataType.Boolean,
+  created_at: $df.DataType.Datetime
 };
 
-const df = $tbl.data(rawData, schema);
+const df = $df.data(rawData, schema);
+```
+
+### 🧠 TypeScript Type Inference & IDE Safety
+
+When you pass a schema to `$df.data()`, DFScript's types automatically infer the target types of the fields, turning dynamic dataframes into compile-safe records. Your IDE will auto-complete column names and validate that operations match the underlying types.
+
+```typescript
+import { $df } from "df-script";
+
+const schema = {
+  name: $df.DataType.Utf8,
+  age: $df.DataType.Int32,
+  is_active: $df.DataType.Boolean
+};
+
+// Inferred DataFrame type is DataFrame<{ name: string; age: number; is_active: boolean }>
+const df = $df.data(rawData, schema);
+
+// Full IDE autocomplete, type validation, and compiler safety!
+const activeUsers = df.filter($df.col("is_active").eq(true));
 ```
 
 ### Supported Data Types

package/dist/api.d.ts

@@ -1,17 +1,24 @@
 import { DataFrame } from "./dataframe";
-import { ColumnExpr, lit, all, exclude, coalesce, when } from "./columnExpressions";
+import { ColumnExpr, lit, all, exclude, coalesce, when, implode, seq_range, element } from "./columnExpressions";
 import { DataType } from "./datatypes";
-import { concat } from "./functions";
-import type { RowRecord } from "./types";
-export declare const $tbl: {
-    data: <T extends RowRecord>(data: T[], schema?: Record<string, DataType>) => DataFrame<T>;
-    col: <T = any>(name: keyof T | string) => ColumnExpr<T>;
+import { concat, read_json, read_csv } from "./functions";
+import type { RowRecord, DataFrameSchema, ColumnDict, InferSchema } from "./types";
+declare function data<S extends DataFrameSchema>(data: any[] | ColumnDict, schema: S): DataFrame<InferSchema<S>>;
+declare function data<T extends RowRecord = any>(data: T[] | ColumnDict, schema?: DataFrameSchema): DataFrame<T>;
+export declare const $df: {
+    data: typeof data;
+    col: <T = any>(name: keyof T | string | (keyof T | string)[] | DataType | Function | (DataType | Function)[]) => ColumnExpr<T>;
     all: typeof all;
     exclude: typeof exclude;
     coalesce: typeof coalesce;
     concat: typeof concat;
+    read_json: typeof read_json;
+    read_csv: typeof read_csv;
     lit: typeof lit;
     when: typeof when;
+    implode: typeof implode;
+    seq_range: typeof seq_range;
+    element: typeof element;
     DataType: {
         Int8: import("./datatypes").Int8Type;
         Int16: import("./datatypes").Int16Type;
@@ -33,7 +40,8 @@ export declare const $tbl: {
         Duration: import("./datatypes").DurationType;
         Object: import("./datatypes").ObjectType;
         Null: import("./datatypes").NullType;
-        List: <TInner>(inner: DataType<TInner>) => import("./datatypes").ListType<TInner>;
-        Struct: <TFields extends RowRecord>(fields: { [K in keyof TFields]: DataType<TFields[K]>; }) => import("./datatypes").StructType<TFields>;
+        Array: <TInner>(inner: import("./datatypes").RegisteredDataType & DataType<TInner>) => import("./datatypes").ArrayType<TInner>;
+        Struct: <TFields extends RowRecord>(fields: { [K in keyof TFields]: import("./datatypes").RegisteredDataType & DataType<TFields[K]>; }) => import("./datatypes").StructType<TFields>;
     };
 };
+export {};