df-script 1.8.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +153 -203
- package/dist/api.d.ts +41 -36
- package/dist/columnExpressions/ColumnExpr.d.ts +5 -8
- package/dist/columnExpressions/ExprBase.d.ts +7 -0
- package/dist/columnExpressions/constants.d.ts +1 -0
- package/dist/columnExpressions/functions/all.d.ts +13 -13
- package/dist/columnExpressions/functions/coalesce.d.ts +2 -2
- package/dist/columnExpressions/functions/duration.d.ts +16 -21
- package/dist/columnExpressions/functions/element.d.ts +10 -10
- package/dist/columnExpressions/functions/exclude.d.ts +14 -14
- package/dist/columnExpressions/functions/implode.d.ts +7 -7
- package/dist/columnExpressions/functions/lit.d.ts +9 -9
- package/dist/columnExpressions/functions/seqRange.d.ts +69 -0
- package/dist/columnExpressions/functions/struct.d.ts +6 -6
- package/dist/columnExpressions/functions/when.d.ts +31 -32
- package/dist/columnExpressions/index.d.ts +4 -7
- package/dist/columnExpressions/mixins/AggregationExpr.d.ts +672 -141
- package/dist/columnExpressions/mixins/ArithmeticExpr.d.ts +701 -327
- package/dist/columnExpressions/mixins/ArrayExpr.d.ts +543 -231
- package/dist/columnExpressions/mixins/ComparisonExpr.d.ts +398 -201
- package/dist/columnExpressions/mixins/LogicalExpr.d.ts +59 -29
- package/dist/columnExpressions/mixins/ManipulationExpr.d.ts +23 -9
- package/dist/columnExpressions/mixins/StandardExpr.d.ts +3234 -0
- package/dist/columnExpressions/mixins/StringExpr.d.ts +1299 -396
- package/dist/columnExpressions/mixins/StructExpr.d.ts +72 -30
- package/dist/columnExpressions/mixins/TemporalExpr.d.ts +518 -212
- package/dist/columnExpressions/mixins/WindowExpr.d.ts +270 -102
- package/dist/columnExpressions/typeInference.d.ts +13 -0
- package/dist/columnExpressions/types.d.ts +6 -1
- package/dist/columnExpressions/utils.d.ts +16 -0
- package/dist/constants.d.ts +38 -0
- package/dist/dataframe/dataframe.d.ts +755 -608
- package/dist/dataframe/grouped/grouped.d.ts +24 -6
- package/dist/dataframe/grouped.d.ts +70 -0
- package/dist/dataframe/index.d.ts +1 -1
- package/dist/dataframe/lazy.d.ts +37 -0
- package/dist/dataframe/types.d.ts +46 -22
- package/dist/dataframe/utils.d.ts +10 -4
- package/dist/datatypes/index.d.ts +11 -4
- package/dist/expressions.js +1 -0
- package/dist/expressions.mjs +1 -0
- package/dist/functions/concat.d.ts +68 -16
- package/dist/functions/index.d.ts +2 -2
- package/dist/functions/readCsv.d.ts +35 -0
- package/dist/functions/readJson.d.ts +33 -0
- package/dist/index.js +5 -6
- package/dist/index.mjs +5 -6
- package/dist/types.d.ts +148 -7
- package/dist/utils/array.d.ts +54 -18
- package/dist/utils/binary.d.ts +6 -2
- package/dist/utils/csv.d.ts +4 -1
- package/dist/utils/date.d.ts +3 -19
- package/dist/utils/duration.d.ts +7 -5
- package/dist/utils/json.d.ts +56 -2
- package/dist/utils/number.d.ts +5 -2
- package/dist/utils/object.d.ts +7 -12
- package/dist/utils/string.d.ts +83 -2
- package/dist/utils/table.d.ts +76 -0
- package/dist/utils.js +4 -0
- package/dist/utils.mjs +4 -0
- package/package.json +29 -8
- package/dist/assets/index-DBhGK6Tp.css +0 -1
- package/dist/assets/index-DEJEV_tU.js +0 -195
- package/dist/index.html +0 -17
package/README.md
CHANGED
|
@@ -1,38 +1,57 @@
|
|
|
1
|
-
# 🚀
|
|
2
|
-
|
|
3
|
-
[](https://github.com/trentamorris/df-script)
|
|
4
|
+
[](https://www.npmjs.com/package/df-script)
|
|
5
|
+
[](https://bundlephobia.com/package/df-script)
|
|
6
|
+
[](#)
|
|
7
|
+
[](https://www.typescriptlang.org/)
|
|
8
|
+
[](#)
|
|
9
|
+
[](LICENSE)
|
|
4
10
|
[](DONATIONS.md)
|
|
5
|
-
|
|
6
|
-
|
|
11
|
+
**df-script** is a blazing-fast, **zero-dependency**, expression-based DataFrame and data manipulation library for **TypeScript** and **JavaScript**. Heavily inspired by modern columnar engines like **Polars** and **Pandas**, `df-script` brings declarative, high-performance columnar analytical queries and ETL workflows directly to JavaScript environments (Node.js, Browser, Bun, Deno, and Edge Workers).
|
|
12
|
+
|
|
13
|
+
With cache-optimized columnar storage and flat memory layout under the hood, `df-script` eliminates garbage collection thrashing caused by intermediate array allocations in chained `.map()`, `.filter()`, and `.reduce()` calls.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## 🌐 Universal Language & Environment Support
|
|
7
18
|
|
|
8
|
-
|
|
19
|
+
`df-script` is built with **zero native dependencies** and ships with dual **ESM** (`dist/index.mjs`) and **CommonJS** (`dist/index.js`) modules alongside comprehensive `.d.ts` type declarations. It runs natively everywhere JavaScript or TypeScript runs:
|
|
9
20
|
|
|
10
|
-
|
|
21
|
+
- 📄 **Languages & File Formats**: Full first-class support in **TypeScript (`.ts`, `.tsx`)**, **JavaScript (`.js`, `.jsx`)**, and module formats (`.mjs`, `.cjs`).
|
|
22
|
+
- ⚛️ **UI Frameworks & Bundlers**: React (JSX/TSX), Next.js, Vue, Nuxt, Svelte, SolidJS, Astro, Vite, Webpack, and esbuild.
|
|
23
|
+
- ⚙️ **Runtimes & Target Standard**: Standard **ES2020+** compatible. Fully tested on **Node.js** (14+), **Bun**, **Deno**, modern **Web Browsers** (Chrome 80+, Safari 13.1+, Firefox 74+, Edge 80+), Cloudflare Workers, Fastly Compute, and AWS Lambda.
|
|
24
|
+
- 🌐 **Browser / Client-Side Compatibility**: 100% in-memory analytical transformations, joins, grouping, and expressions run natively in all browser runtimes. File writing methods (`df.writeCsv()`, `df.writeJson()`) automatically fallback to returning strings or writing to custom stream objects when running in browser environments.
|
|
25
|
+
- 📦 **Package Managers**: Works seamlessly with `npm`, `pnpm`, `bun`, and `yarn`.
|
|
11
26
|
|
|
12
|
-
|
|
27
|
+
---
|
|
13
28
|
|
|
14
|
-
|
|
29
|
+
## 💡 Why df-script? (Polars & Pandas for TypeScript)
|
|
15
30
|
|
|
16
|
-
|
|
31
|
+
In modern web apps and Node.js backend services, data transformation code often degrades into nested chains of `.map()`, `.filter()`, and `.sort()` on arrays of objects. Each step in the chain allocates new intermediate arrays, slows down garbage collection, and creates maintenance overhead.
|
|
17
32
|
|
|
18
|
-
|
|
33
|
+
`df-script` provides:
|
|
34
|
+
- ⚡ **Columnar Execution**: Column-oriented arrays with cached loop lengths for fast computation and minimal memory allocation.
|
|
35
|
+
- 🔗 **Fluent Expressions**: Declarative, composable queries using `$df.col(...)` expressions with automatic post-operation schema deduction.
|
|
36
|
+
- 📂 **Strict Domain Namespaces**: Clean, dedicated namespaces (`.str`, `.dt`, `.arr`, `.struct`) to prevent method clutter and ensure discoverable APIs.
|
|
37
|
+
- 🛡️ **Zero External Dependencies**: Lightweight runtime footprint with zero supply-chain risk.
|
|
38
|
+
- 🧠 **TypeScript First**: Full IDE autocomplete and compile-time type safety for column selections and schemas.
|
|
19
39
|
|
|
20
40
|
---
|
|
21
41
|
|
|
22
42
|
## 🗺️ Table of Contents
|
|
23
43
|
|
|
24
44
|
- [✨ Key Features](#-key-features)
|
|
25
|
-
- [
|
|
26
|
-
- [🤝 Contributing & Collective Wisdom](#-contributing--collective-wisdom)
|
|
45
|
+
- [🌐 Universal Language & Environment Support](#-universal-language--environment-support)
|
|
27
46
|
- [📦 Installation](#-installation)
|
|
28
47
|
- [🚀 Quick Start](#-quick-start)
|
|
29
48
|
- [📖 Core Concepts](#-core-concepts)
|
|
30
|
-
- [🛠️ DataFrame API Reference](
|
|
49
|
+
- [🛠️ DataFrame API Reference](#️-dataframe-api-reference)
|
|
31
50
|
- [📂 File / Data I/O](#-file--data-io)
|
|
32
51
|
- [🧮 Expressions API Reference](#-expressions-api-reference)
|
|
33
52
|
- [📂 Namespaces](#-namespaces)
|
|
34
53
|
- [🪟 Window & Rolling Expressions](#-window--rolling-expressions)
|
|
35
|
-
- [🛡️ Typing and Schema Registry](
|
|
54
|
+
- [🛡️ Typing and Schema Registry](#️-typing-and-schema-registry)
|
|
36
55
|
- [🧑💻 Contributing & Development](#-contributing--development)
|
|
37
56
|
- [📄 License](#-license)
|
|
38
57
|
|
|
@@ -40,88 +59,67 @@ DFScript was born from that frustration. It brings Polars-like columnar executio
|
|
|
40
59
|
|
|
41
60
|
## ✨ Key Features
|
|
42
61
|
|
|
43
|
-
- 📦 **Zero Dependencies** —
|
|
44
|
-
- ⚡ **Columnar Execution** —
|
|
45
|
-
- 🔗 **Expression-Based API** — Compose complex
|
|
46
|
-
- 📂 **Strict Namespaces
|
|
47
|
-
- `.str`
|
|
48
|
-
- `.dt`
|
|
49
|
-
- `.arr`
|
|
50
|
-
- `.struct`
|
|
51
|
-
- 🪟 **Analytical Window Functions** —
|
|
52
|
-
- 🛠️ **Relational Operations** —
|
|
53
|
-
- 🛡️ **Defensive & Type-Safe** —
|
|
54
|
-
|
|
55
|
-
---
|
|
56
|
-
|
|
57
|
-
## ⚙️ Compatibility & Design Principles
|
|
58
|
-
|
|
59
|
-
DFScript is designed with a **low-abstraction, zero-dependency** philosophy to guarantee maximum compatibility, predictability, and runtime performance:
|
|
60
|
-
|
|
61
|
-
- 📦 **Zero External Dependencies** — Lightweight footprint with zero runtime overhead or supply chain vulnerabilities.
|
|
62
|
-
- 🌐 **Universal Compatibility** — Works out-of-the-box in any JavaScript/TypeScript environment, including Node.js, Deno, Bun, web browsers, and cloud/edge workers.
|
|
63
|
-
- 🧱 **Built-in Standards** — Prioritizes native, built-in APIs (like standard `Date`, `Intl` formatting, and `TextEncoder`) and standard arrays rather than custom wrappers or heavy runtime abstractions.
|
|
64
|
-
- ⚡ **Optimized Execution Paths** — Under the hood, performance-critical code avoids higher-level array iterators and short-lived intermediate allocations in favor of simple, fast `for` and `while` loops with cached lengths, keeping garbage collection overhead to an absolute minimum.
|
|
65
|
-
- 🔄 **Easy Transpilation** — Relies strictly on low-level native operations, making it fully compatible with older environments (like ES6 or even ES5) without requiring complex polyfills or modern engine-specific features.
|
|
66
|
-
|
|
67
|
-
---
|
|
68
|
-
|
|
69
|
-
## 🤝 Contributing & Collective Wisdom
|
|
70
|
-
|
|
71
|
-
We don’t pretend to have encountered every localized date format, database quirk, or environment-specific edge case. DFScript is built on the belief that software correctness is a collective endeavor.
|
|
72
|
-
|
|
73
|
-
If you run into an unsupported edge case or unexpected behavior in any of our functions, we want to hear about it. Help us harden this engine by opening a GitHub Issue or submitting a PR—every report makes the library more robust for everyone.
|
|
62
|
+
- 📦 **Zero Dependencies** — 0 external runtime dependencies; completely standalone.
|
|
63
|
+
- ⚡ **Columnar Execution** — Fast columnar processing that eliminates intermediate array allocations.
|
|
64
|
+
- 🔗 **Expression-Based API** — Compose complex transformations, aggregations, and conditions using fluent `$df` expressions.
|
|
65
|
+
- 📂 **Strict Namespaces**:
|
|
66
|
+
- `.str` — Unicode string manipulations, regex extractions, and JSON path lookups.
|
|
67
|
+
- `.dt` — Timezone conversions, business days, microsecond-precision datetimes, and durations.
|
|
68
|
+
- `.arr` — Array/list column operations and element-wise `.arr.eval()` mapping.
|
|
69
|
+
- `.struct` — Nested object handling and `.struct.unnest()` column flattening.
|
|
70
|
+
- 🪟 **Analytical Window Functions** — Partitioned windowing (`.over()`), cumulative aggregations (`cumSum()`, `cumMax()`), and rolling moving statistics (`rollingMean()`, `rollingStd()`).
|
|
71
|
+
- 🛠️ **Relational Operations** — Inner/left/right/outer/cross joins, `joinAsof` time-series matching, pivots, unpivots, and multi-axis concatenations.
|
|
72
|
+
- 🛡️ **Defensive & Type-Safe** — Automatic type coercion, Kleene three-valued logic for null safety, and strict schema validation.
|
|
74
73
|
|
|
75
74
|
---
|
|
76
75
|
|
|
77
76
|
## 📦 Installation
|
|
78
77
|
|
|
79
|
-
Install
|
|
78
|
+
Install `df-script` using your package manager:
|
|
80
79
|
|
|
81
80
|
```bash
|
|
82
81
|
npm install df-script
|
|
83
82
|
```
|
|
84
83
|
|
|
85
|
-
Or with Yarn
|
|
84
|
+
Or with Yarn, PNPM, or Bun:
|
|
86
85
|
|
|
87
86
|
```bash
|
|
88
87
|
yarn add df-script
|
|
89
88
|
pnpm add df-script
|
|
89
|
+
bun add df-script
|
|
90
90
|
```
|
|
91
91
|
|
|
92
92
|
---
|
|
93
93
|
|
|
94
94
|
## 🚀 Quick Start
|
|
95
95
|
|
|
96
|
-
Here is a quick example showing how to load data, run expressions, perform aggregations, and compute rolling statistics.
|
|
97
|
-
|
|
98
96
|
```typescript
|
|
99
97
|
import { $df } from "df-script";
|
|
100
98
|
|
|
101
99
|
// 1. Create a DataFrame with structured data and automatic schema inference
|
|
102
100
|
const df = $df.data([
|
|
103
|
-
{ id: 1, name: "Alice",
|
|
104
|
-
{ id: 2, name: "Bob",
|
|
105
|
-
{ id: 3, name: "Charlie",
|
|
106
|
-
{ id: 4, name: "David",
|
|
101
|
+
{ id: 1, name: "Alice", joinDate: "2026-01-15", sales: 1200.50, tags: ["sales", "east"] },
|
|
102
|
+
{ id: 2, name: "Bob", joinDate: "2026-02-20", sales: 850.00, tags: ["support", "west"] },
|
|
103
|
+
{ id: 3, name: "Charlie", joinDate: "2026-03-05", sales: 2300.00, tags: ["sales", "north"] },
|
|
104
|
+
{ id: 4, name: "David", joinDate: "2026-03-12", sales: null, tags: ["marketing"] },
|
|
107
105
|
]);
|
|
108
106
|
|
|
109
|
-
// 2. Select columns, transform strings, format dates, and
|
|
107
|
+
// 2. Select columns, transform strings, format dates, and compute expressions
|
|
110
108
|
const processedDf = df.select(
|
|
111
109
|
$df.col("id"),
|
|
112
110
|
$df.col("name").str.upper().alias("NAME_UPPER"),
|
|
113
|
-
$df.col("
|
|
114
|
-
$df.col("sales").add(500).alias("
|
|
115
|
-
$df.col("tags").arr.lengths().alias("
|
|
111
|
+
$df.col("joinDate").str.toDatetime().dt.year().alias("joinYear"),
|
|
112
|
+
$df.col("sales").add(500).alias("salesAdjusted"),
|
|
113
|
+
$df.col("tags").arr.lengths().alias("tagCount")
|
|
116
114
|
);
|
|
117
115
|
|
|
118
|
-
console.log(processedDf.
|
|
116
|
+
console.log(processedDf.toDicts());
|
|
119
117
|
/* Output:
|
|
120
118
|
[
|
|
121
|
-
{ id: 1, NAME_UPPER: 'ALICE',
|
|
122
|
-
{ id: 2, NAME_UPPER: 'BOB',
|
|
123
|
-
{ id: 3, NAME_UPPER: 'CHARLIE',
|
|
124
|
-
{ id: 4, NAME_UPPER: 'DAVID',
|
|
119
|
+
{ id: 1, NAME_UPPER: 'ALICE', joinYear: 2026, salesAdjusted: 1700.5, tagCount: 2 },
|
|
120
|
+
{ id: 2, NAME_UPPER: 'BOB', joinYear: 2026, salesAdjusted: 1350, tagCount: 2 },
|
|
121
|
+
{ id: 3, NAME_UPPER: 'CHARLIE', joinYear: 2026, salesAdjusted: 2800, tagCount: 2 },
|
|
122
|
+
{ id: 4, NAME_UPPER: 'DAVID', joinYear: 2026, salesAdjusted: null, tagCount: 1 }
|
|
125
123
|
]
|
|
126
124
|
*/
|
|
127
125
|
```
|
|
@@ -132,81 +130,88 @@ console.log(processedDf.to_dicts());
|
|
|
132
130
|
|
|
133
131
|
### The `$df` Entry Point
|
|
134
132
|
|
|
135
|
-
|
|
133
|
+
`df-script` uses the `$df` namespace to instantiate DataFrames, reference columns, construct expressions, and specify data types.
|
|
136
134
|
|
|
137
135
|
- `$df.data(dataRowsOrCols, schema?)`: Instantiates a new `DataFrame`.
|
|
138
|
-
- `$df.
|
|
139
|
-
- `$df.
|
|
140
|
-
- `$df.col(
|
|
136
|
+
- `$df.readJson(content, options?)`: Reads JSON/NDJSON content into a new `DataFrame`.
|
|
137
|
+
- `$df.readCsv(content, options?)`: Reads CSV content into a new `DataFrame` with automatic schema inference.
|
|
138
|
+
- `$df.col(selector)`: Creates a column reference expression by column name (`"a"`), multiple names (`["a", "b"]`), RegExp pattern (`/^user_/`), or DataType selector (`$df.Float64`, `$df.Numeric`).
|
|
141
139
|
- `$df.all()`: Selects all columns in the DataFrame.
|
|
142
|
-
- `$df.exclude(columns)`:
|
|
143
|
-
- `$df.coalesce(...exprs)`: Returns the first non-null value among columns or
|
|
140
|
+
- `$df.exclude(columns)`: Matches all columns except the specified ones.
|
|
141
|
+
- `$df.coalesce(...exprs)`: Returns the first non-null value among columns or expressions.
|
|
144
142
|
- `$df.lit(val)`: Explicitly wraps a raw value into a literal expression.
|
|
145
|
-
- `$df.
|
|
146
|
-
- `$df.
|
|
147
|
-
- `$df.
|
|
143
|
+
- `$df.duration(optionsOrString)`: Constructs a `Duration` expression from component options (`{ days: 1, hours: 12 }`) or compound duration strings (`"1d 12h 30m"`).
|
|
144
|
+
- `$df.struct(fields)`: Constructs a nested struct object expression from named expressions or sibling columns.
|
|
145
|
+
- `$df.when(predicate).then(value)...otherwise(value)`: Constructs a conditional `CASE WHEN` expression chain.
|
|
146
|
+
- `$df.implode(column)`: Aggregates a column's rows or grouped values into a list.
|
|
147
|
+
- `$df.seqRange(value, options?)`: Generates a sequence range of values.
|
|
148
148
|
- `$df.element()`: References the current array element within an `.arr.eval(...)` expression.
|
|
149
|
-
- `$df.
|
|
150
|
-
|
|
151
|
-
### DataFrames vs. Columns
|
|
152
|
-
|
|
153
|
-
- **`DataFrame`** holds data in a columnar-oriented object: `columns: Record<string, any[]>`.
|
|
154
|
-
- **`ColumnExpr`** represents an evaluation sequence over rows. Operations (arithmetic, strings, lists, date-time, comparisons) are chained to build a tree of computations evaluated lazily.
|
|
149
|
+
- `$df.Float64`, `$df.Int32`, `$df.Utf8`, etc.: Direct access to data types and constructors for schema definitions and type-based column selection.
|
|
155
150
|
|
|
156
151
|
---
|
|
157
152
|
|
|
158
153
|
## 🛠️ DataFrame API Reference
|
|
159
154
|
|
|
160
155
|
### 1. Transformations & Projection
|
|
161
|
-
- **`select(...exprs)`**: Projects columns. Supports strings,
|
|
162
|
-
- **`
|
|
156
|
+
- **`select(...exprs)`**: Projects columns. Supports strings, `$df.col(...)` expressions, `$df.all()`, RegExp patterns (`/^prefix_/`), DataType selectors (`$df.Numeric`), and `$df.col("struct").struct.unnest()`.
|
|
157
|
+
- **`withColumns(...exprs)`**: Adds or overrides columns. Accepts expressions, options mapping keys, RegExp patterns, or DataType selectors.
|
|
163
158
|
- **`drop(...names)`**: Drops one or more columns from the DataFrame.
|
|
164
159
|
- **`rename(mapping)`**: Renames columns using a `{ oldName: newName }` object.
|
|
160
|
+
- **`explode(columns)`**: Unnests list-like columns into multiple rows, replicating other columns per list element.
|
|
161
|
+
- **`implode(columns)`**: Groups values in specified columns back into a list element per column.
|
|
165
162
|
|
|
166
163
|
### 2. Filtering & Row Selection
|
|
167
|
-
- **`filter(...predicates)`**: Filters rows where all predicate expressions evaluate to `true
|
|
168
|
-
- **`
|
|
164
|
+
- **`filter(...predicates)`**: Filters rows where all predicate expressions evaluate to `true`.
|
|
165
|
+
- **`find(predicate)`**: Returns the first matching row record object (or `undefined`).
|
|
166
|
+
- **`unique(columns?)`**: Returns unique rows, optionally deduplicating based on a subset of columns.
|
|
169
167
|
- **`limit(n, options?)`**: Returns the first `n` rows. Options include `offset` and direction `from: "start" | "end"`.
|
|
170
168
|
- **`head(n)`** / **`tail(n)`**: Shortcuts for `limit` from the start or end of the DataFrame.
|
|
171
|
-
- **`slice(start, end?)`**:
|
|
172
|
-
- **`gather(indices, options?)`**: Gathers rows at specified indices
|
|
169
|
+
- **`slice(start, end?)`**: Extracts a subset of rows using standard index slicing.
|
|
170
|
+
- **`gather(indices, options?)`**: Gathers rows at specified indices (supports negative indexing and `{ nullOnOob?: boolean }`).
|
|
173
171
|
|
|
174
|
-
### 3. Sorting
|
|
175
|
-
- **`sort({ by, descending?, nullsLast?, custom? })`**: Sorts rows
|
|
172
|
+
### 3. Sorting & Structural Operations
|
|
173
|
+
- **`sort({ by, descending?, nullsLast?, custom? })`**: Sorts rows by single/multiple columns, custom null ordering, or comparator functions.
|
|
174
|
+
- **`clone()`**: Performs a deep copy of the `DataFrame`, replicating all underlying column arrays and schema metadata.
|
|
175
|
+
- **`transpose(options?)`**: Transposes the DataFrame (swapping rows and columns).
|
|
176
176
|
|
|
177
177
|
### 4. Grouping & Aggregations
|
|
178
|
-
- **`
|
|
179
|
-
- **`
|
|
178
|
+
- **`groupBy(keys)`**: Groups data by one or more columns, returning a `GroupedData` object.
|
|
179
|
+
- **`groupByDynamic(indexColumn, options)`**: Dynamic time-series / numeric window grouping over tumbling, sliding, or rolling temporal aggregation intervals (`every`, `period`, `offset`, `closed`, `label`, `startBy`, `includeBoundaries`, `by`).
|
|
180
|
+
- **`GroupedData.agg(...exprs)`**: Computes aggregations on grouped data (e.g. `$df.col("sales").sum()`).
|
|
180
181
|
|
|
181
182
|
### 5. Reshaping & Joining
|
|
182
|
-
- **`join(other,
|
|
183
|
+
- **`join(other, onOrOptions, how?, suffixes?)`**: Merges two DataFrames. Supports:
|
|
184
|
+
- Join modes (`how`): `"inner" | "left" | "right" | "outer" | "semi" | "anti" | "cross"`.
|
|
185
|
+
- Keys: `leftOn`, `rightOn`, `coalesce`, and order preservation options.
|
|
186
|
+
- **`joinAsof(other, options)`**: Inexact time-series / nearest-neighbor joins on sorted key columns.
|
|
187
|
+
- Parameters: `on`, `leftOn`, `rightOn`, `by`, `strategy` (`"backward" | "forward" | "nearest"`), and `tolerance`.
|
|
183
188
|
- **`pivot(index, columns, values)`**: Pivots the table, converting unique values in `columns` into column headers.
|
|
184
|
-
- **`unpivot(idVars, valueVars, varName?, valueName?)`**: Melts/unpivots
|
|
185
|
-
- **`concat(items, options?)`**: Concatenates multiple DataFrames
|
|
189
|
+
- **`unpivot(idVars, valueVars, varName?, valueName?)`**: Melts/unpivots wide columns into long format name-value pairs.
|
|
190
|
+
- **`concat(items, options?)`**: Concatenates multiple DataFrames (`"vertical" | "horizontal" | "diagonal"`).
|
|
186
191
|
|
|
187
192
|
---
|
|
188
193
|
|
|
189
194
|
## 📂 File / Data I/O
|
|
190
195
|
|
|
191
|
-
|
|
196
|
+
`df-script` provides built-in parsers and serializers for JSON and CSV formats.
|
|
192
197
|
|
|
193
198
|
### Reading Data
|
|
194
|
-
- **`$df.
|
|
199
|
+
- **`$df.readJson(content, options?)`**: Reads standard JSON or Newline Delimited JSON (NDJSON) string.
|
|
195
200
|
```typescript
|
|
196
201
|
import { $df } from "df-script";
|
|
197
202
|
|
|
198
|
-
// Read standard JSON
|
|
199
|
-
const df = $df.
|
|
203
|
+
// Read standard JSON
|
|
204
|
+
const df = $df.readJson('[{"id": 1, "name": "Alice"}]');
|
|
200
205
|
|
|
201
|
-
// Read
|
|
202
|
-
const dfNdjson = $df.
|
|
206
|
+
// Read NDJSON
|
|
207
|
+
const dfNdjson = $df.readJson('{"id": 1}\n{"id": 2}', { format: "ndjson" });
|
|
203
208
|
```
|
|
204
|
-
- **`$df.
|
|
209
|
+
- **`$df.readCsv(content, options?)`**: Reads a CSV string with automatic type inference.
|
|
205
210
|
```typescript
|
|
206
211
|
import { $df } from "df-script";
|
|
207
212
|
|
|
208
213
|
const csvContent = "id,name,active\n1,Alice,true\n2,Bob,false";
|
|
209
|
-
const df = $df.
|
|
214
|
+
const df = $df.readCsv(csvContent, {
|
|
210
215
|
separator: ",",
|
|
211
216
|
hasHeader: true,
|
|
212
217
|
inferSchema: true
|
|
@@ -214,46 +219,45 @@ DFScript provides helpers to serialize and parse data formats like JSON and CSV.
|
|
|
214
219
|
```
|
|
215
220
|
|
|
216
221
|
### Writing Data
|
|
217
|
-
- **`df.
|
|
222
|
+
- **`df.writeJson(file?, options?)`**: Serializes a DataFrame into a JSON or NDJSON string.
|
|
218
223
|
```typescript
|
|
219
|
-
|
|
220
|
-
|
|
224
|
+
const jsonStr = df.writeJson();
|
|
225
|
+
df.writeJson("output.json");
|
|
221
226
|
```
|
|
222
|
-
- **`df.
|
|
227
|
+
- **`df.writeCsv(file?, options?)`**: Serializes a DataFrame into a CSV string.
|
|
223
228
|
```typescript
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
// Write to a file with custom separator
|
|
228
|
-
df.write_csv("output.csv", { separator: ";" });
|
|
229
|
+
const csvStr = df.writeCsv();
|
|
230
|
+
df.writeCsv("output.csv", { separator: ";" });
|
|
229
231
|
```
|
|
230
232
|
|
|
231
233
|
---
|
|
232
234
|
|
|
233
235
|
## 🧮 Expressions API Reference
|
|
234
236
|
|
|
235
|
-
All column expressions inherit from `ExprBase` and support
|
|
237
|
+
All column expressions inherit from `ExprBase` and support chaining.
|
|
236
238
|
|
|
237
239
|
### ➕ Arithmetic Expressions
|
|
238
|
-
Chained mathematical functions execute cleanly with built-in null-safety (Kleene logic).
|
|
239
240
|
- `.add(val)`, `.sub(val)`, `.mul(val)`, `.div(val)`, `.floordiv(val)`, `.mod(val)`, `.pow(val)`
|
|
240
241
|
- `.abs()`, `.sqrt()`, `.cbrt()`, `.exp()`, `.expm1()`, `.log(base?)`, `.log1p()`
|
|
241
242
|
- `.ceil()`, `.floor()`, `.trunc()`, `.round(decimals)`, `.clip(lower, upper)`, `.sign()`, `.negate()`
|
|
242
|
-
- `.sin()`, `.cos()`, `.tan()`, `.sinh()`, `.cosh()`, `.tanh()`, `.
|
|
243
|
+
- `.sin()`, `.cos()`, `.tan()`, `.cot()`, `.sinh()`, `.cosh()`, `.tanh()`, `.arcsin()`, `.arccos()`, `.arctan()`, `.arctan2(val)`, `.arcsinh()`, `.arccosh()`, `.arctanh()`, `.degrees()`, `.radians()`, `.hypot(val)`
|
|
243
244
|
|
|
244
245
|
### 🔍 Comparison Expressions
|
|
245
|
-
- `.eq(val)`, `.ne(val)` — Strict value equivalence (
|
|
246
|
-
- `.
|
|
246
|
+
- `.eq(val)`, `.ne(val)` — Strict value equivalence (Kleene null-propagation).
|
|
247
|
+
- `.eqMissing(val)`, `.neMissing(val)` — Null-safe equality treating null/undefined as equal.
|
|
247
248
|
- `.gt(val)`, `.ge(val)`, `.lt(val)`, `.le(val)`
|
|
248
|
-
- `.
|
|
249
|
-
- `.
|
|
250
|
-
- `.
|
|
249
|
+
- `.isNull()`, `.isNotNull()`
|
|
250
|
+
- `.isFinite()`, `.isInfinite()`, `.isNan()`, `.isNotNan()`
|
|
251
|
+
- `.isNDistinct(index, nullOnOob?)` — Matches the N-th distinct value by positive/negative index position.
|
|
252
|
+
- `.isIn(arrayOrExpr)`, `.notIn(arrayOrExpr)`
|
|
251
253
|
|
|
252
254
|
### ⚡ Aggregations
|
|
253
|
-
- `.sum()`, `.avg()` / `.mean()`, `.median()`, `.mode()`, `.std()`, `.
|
|
254
|
-
- `.
|
|
255
|
+
- `.sum()`, `.product()`, `.avg()` / `.mean()`, `.median()`, `.mode()`, `.variance()`, `.std()`, `.skew()`, `.kurtosis()`, `.entropy(base?, normalize?)`
|
|
256
|
+
- `.min()`, `.max()`, `.nanMin()`, `.nanMax()`, `.minBy(by)`, `.maxBy(by)`, `.argMin()`, `.argMax()`
|
|
257
|
+
- `.count(options?)` — Options: `{ includeNulls: boolean }`.
|
|
255
258
|
- `.first()`, `.last()`
|
|
256
|
-
- `.any()`, `.all()`, `.
|
|
259
|
+
- `.any()`, `.all()`, `.anyNull()`, `.allNull()`, `.nUnique()`, `.nullCount()`
|
|
260
|
+
- `.bitwiseAnd()`, `.bitwiseOr()`, `.bitwiseXor()`
|
|
257
261
|
|
|
258
262
|
### 🔀 Control Flow & Conditionals
|
|
259
263
|
Construct dynamic `CASE WHEN` branches using the `$df.when` API:
|
|
@@ -265,117 +269,84 @@ df.select(
|
|
|
265
269
|
$df.when($df.col("sales").gt(2000)).then("High Performance")
|
|
266
270
|
.when($df.col("sales").gt(1000)).then("Standard Performance")
|
|
267
271
|
.otherwise("Low Performance")
|
|
268
|
-
.alias("
|
|
272
|
+
.alias("salesCategory")
|
|
269
273
|
);
|
|
270
274
|
```
|
|
271
|
-
- `$df.when(predicate).then(value)`: Starts a conditional evaluation.
|
|
272
|
-
- `.when(predicate).then(value)`: Chains additional conditions.
|
|
273
|
-
- `.otherwise(value)`: Specifies the fallback value when no conditions match (returns a complete `ColumnExpr`).
|
|
274
275
|
|
|
275
276
|
---
|
|
276
277
|
|
|
277
278
|
## 📂 Namespaces
|
|
278
279
|
|
|
279
|
-
|
|
280
|
+
Specific domain transforms are grouped under dedicated namespaces:
|
|
280
281
|
|
|
281
282
|
### 🔤 String Operations (`.str`)
|
|
282
283
|
Available on any expression via `.str`:
|
|
283
284
|
```typescript
|
|
284
285
|
$df.col("name").str.lower()
|
|
285
|
-
$df.col("code").str.
|
|
286
|
+
$df.col("code").str.startsWith("A")
|
|
286
287
|
$df.col("description").str.replace(/foo/i, "bar")
|
|
287
288
|
```
|
|
288
|
-
- **Methods**: `lower()`, `upper()`, `len()`, `
|
|
289
|
+
- **Methods**: `lower()`, `upper()`, `toTitlecase()`, `len()`, `lenBytes()`, `lenChars()`, `trim()`, `trimStart()`, `trimEnd()`, `startsWith(pfx)`, `endsWith(sfx)`, `contains(pat)`, `containsAny(pats)`, `countMatches(pat)`, `find(pat)`, `findMany(pats)`, `replace(pat, repl)`, `replaceAll(pat, repl)`, `replaceMany(pats, repls)`, `slice(offset, len?)`, `split(delim, options?)`, `explode()`, `reverse()`, `lpad(w, f)`, `rpad(w, f)`, `zfill(w)`, `stripChars(chars?)`, `stripCharsStart(chars?)`, `stripCharsEnd(chars?)`, `stripPrefix(pfx)`, `stripSuffix(sfx)`, `escapeRegex()`, `extract(pat, group?)`, `extractAll(pat)`, `extractGroups(pat)`, `extractMany(pats)`, `encode(enc)`, `decode(enc, strict?)`, `jsonDecode(options?)`, `jsonPathMatch(path)`, `normalize(form?)`, `join(sep)`, `strptime(fmt, strict?)`, `toInteger()`, `toDecimal(p, s)`, `toDate()`, `toDatetime()`, `toTime()`.
|
|
289
290
|
|
|
290
291
|
### 📅 Temporal Operations (`.dt`)
|
|
291
292
|
Available on datetime or duration values via `.dt`:
|
|
292
293
|
```typescript
|
|
293
294
|
$df.col("timestamp").dt.year()
|
|
294
|
-
$df.col("timestamp").dt.
|
|
295
|
-
$df.col("duration").dt.
|
|
295
|
+
$df.col("timestamp").dt.convertTimeZone("America/New_York")
|
|
296
|
+
$df.col("duration").dt.totalSeconds()
|
|
296
297
|
```
|
|
297
|
-
- **Datetime Methods**: `year()`, `month()`, `day()`, `hour()`, `minute()`, `second()`, `millisecond()`, `microsecond()`, `nanosecond()`, `weekday()`, `week()`, `quarter()`, `century()`, `millennium()`, `
|
|
298
|
-
- **Duration Methods**: `
|
|
298
|
+
- **Datetime Methods**: `year()`, `month()`, `day()`, `hour()`, `minute()`, `second()`, `millisecond()`, `microsecond()`, `nanosecond()`, `weekday()`, `week()`, `quarter()`, `century()`, `millennium()`, `ordinalDay()`, `isLeapYear()`, `monthStart()`, `monthEnd()`, `date()`, `time()`, `offsetDay(n, options?)`, `offsetBusinessDay(n, options?)`, `convertTimeZone(tz)`, `castTimeUnit(unit)`, `withTimeUnit(unit)`, `replace(options)`, `truncate(every)`, `utcOffset(tz?, options?)`, `epoch(unit)`, `timestamp(unit)`, `strftime(fmt, locale?)`.
|
|
299
|
+
- **Duration Methods**: `totalDays()`, `totalHours()`, `totalMinutes()`, `totalSeconds()`, `totalMilliseconds()`, `totalMicroseconds()`, `totalNanoseconds()`.
|
|
299
300
|
|
|
300
301
|
### 📊 Array/List Operations (`.arr`)
|
|
301
|
-
Available on
|
|
302
|
+
Available on array/list column expressions via `.arr`:
|
|
302
303
|
```typescript
|
|
303
304
|
$df.col("tags").arr.contains("vip")
|
|
304
|
-
$df.col("
|
|
305
|
-
|
|
306
|
-
// Element-wise manipulation inside arrays:
|
|
307
|
-
$df.col("numbers").arr.eval(element().mul(2)).alias("numbers_doubled")
|
|
308
|
-
$df.col("tags").arr.eval(element().str.to_uppercase()).alias("upper_tags")
|
|
305
|
+
$df.col("numbers").arr.eval($df.element().mul(2)).alias("numbersDoubled")
|
|
309
306
|
```
|
|
310
|
-
- **Methods**: `lengths()`, `len()`, `get(idx,
|
|
307
|
+
- **Methods**: `lengths()`, `len()`, `get(idx, nullOnOob?)`, `first(nullOnOob?)`, `last(nullOnOob?)`, `gather(indices, nullOnOob?)`, `gatherEvery(n, offset?)`, `slice(offset, len?)`, `contains(item)`, `countMatches(item)`, `join(sep)`, `sort(descending?)`, `reverse()`, `unique()`, `sum()`, `mean()`, `median()`, `mode()`, `min()`, `max()`, `argMin()`, `argMax()`, `agg(expr)`, `eval(expr)`.
|
|
311
308
|
|
|
312
309
|
### 🗃️ Struct/Object Operations (`.struct`)
|
|
313
|
-
Available on
|
|
310
|
+
Available on nested struct/object column expressions via `.struct`:
|
|
314
311
|
```typescript
|
|
315
312
|
// Sibling fields access via Proxy
|
|
316
313
|
$df.col("address").struct.city.alias("city")
|
|
317
314
|
|
|
318
|
-
//
|
|
319
|
-
$df.col("address").struct.
|
|
315
|
+
// Struct unnesting (flattens fields to top-level columns in select)
|
|
316
|
+
df.select($df.col("address").struct.unnest())
|
|
320
317
|
```
|
|
321
|
-
- **Methods**:
|
|
322
|
-
- `field(name)`: Accesses a field within the struct.
|
|
323
|
-
- `rename_fields(mapping)`: Renames fields in the struct based on a `{ oldKey: newKey }` mapping.
|
|
324
|
-
- `with_fields(fields)`: Adds or overrides fields in the struct. Accepts an array of aliased expressions or an object.
|
|
325
|
-
- `unnest()`: Expands the fields of the struct into individual top-level columns in a select projection.
|
|
326
|
-
```typescript
|
|
327
|
-
// Flattens the address struct into "city", "state", etc. at the top-level
|
|
328
|
-
df.select($df.col("address").struct.unnest())
|
|
329
|
-
```
|
|
318
|
+
- **Methods**: `field(name)`, `renameFields(mapping)`, `withFields(fields)`, `unnest()`.
|
|
330
319
|
|
|
331
320
|
---
|
|
332
321
|
|
|
333
322
|
## 🪟 Window & Rolling Expressions
|
|
334
323
|
|
|
335
|
-
|
|
324
|
+
Analytical partition window operations using `.over()` and moving calculations:
|
|
336
325
|
|
|
337
326
|
```typescript
|
|
338
|
-
// Calculate partition cumulative sums and row numbers
|
|
339
327
|
df.select(
|
|
340
328
|
$df.col("department"),
|
|
341
329
|
$df.col("sales"),
|
|
342
|
-
$df.col("sales").sum().over("department").alias("
|
|
343
|
-
$df.col("sales").
|
|
344
|
-
$df.all().
|
|
330
|
+
$df.col("sales").sum().over("department").alias("deptTotalSales"),
|
|
331
|
+
$df.col("sales").cumSum().over("department").alias("deptRunningSales"),
|
|
332
|
+
$df.all().rowNumber().over("department").alias("deptRank")
|
|
345
333
|
);
|
|
346
334
|
```
|
|
347
335
|
|
|
348
336
|
### 1. Cumulative Windows
|
|
349
|
-
- `.
|
|
350
|
-
- `.cum_prod(reverse?)`
|
|
351
|
-
- `.cum_min(reverse?)`
|
|
352
|
-
- `.cum_max(reverse?)`
|
|
353
|
-
- `.cum_count(reverse?)`
|
|
337
|
+
- `.cumSum(reverse?)`, `.cumProd(reverse?)`, `.cumMin(reverse?)`, `.cumMax(reverse?)`, `.cumCount(reverse?)`
|
|
354
338
|
|
|
355
339
|
### 2. Rolling Metrics (Moving Window)
|
|
356
|
-
|
|
357
|
-
- `.
|
|
358
|
-
- `.rolling_mean(size)`
|
|
359
|
-
- `.rolling_median(size)`
|
|
360
|
-
- `.rolling_min(size)`
|
|
361
|
-
- `.rolling_max(size)`
|
|
362
|
-
- `.rolling_std(size)`
|
|
363
|
-
- `.rolling_rank(size)`
|
|
364
|
-
- `.rolling_quantile(quantile, size)`
|
|
340
|
+
- **Generic Rolling Reducer**: `.rolling(sizeOrOptions, exprOrFn)` (evaluates custom functions or `$df` column expressions over each sliding window).
|
|
341
|
+
- **Specialized Rolling Reducers**: `.rollingSum(size)`, `.rollingMean(size)`, `.rollingMedian(size)`, `.rollingMin(size)`, `.rollingMax(size)`, `.rollingStd(size)`, `.rollingRank(size)`, `.rollingQuantile(quantile, size)`
|
|
365
342
|
|
|
366
343
|
### 3. Positional & Rank Windows
|
|
367
|
-
- `.lead(offset, defaultVal?)`
|
|
368
|
-
- `.lag(offset, defaultVal?)`
|
|
369
|
-
- `.rank()`
|
|
370
|
-
- `.dense_rank()`
|
|
371
|
-
- `.row_number()`
|
|
344
|
+
- `.lead(offset, defaultVal?)`, `.lag(offset, defaultVal?)`, `.rank()`, `.denseRank()`, `.rowNumber()`
|
|
372
345
|
|
|
373
346
|
---
|
|
374
347
|
|
|
375
348
|
## 🛡️ Typing and Schema Registry
|
|
376
349
|
|
|
377
|
-
You can optionally declare schemas to enforce precise data types and automatic type coercion during construction.
|
|
378
|
-
|
|
379
350
|
```typescript
|
|
380
351
|
import { $df } from "df-script";
|
|
381
352
|
|
|
@@ -383,30 +354,10 @@ const schema = {
|
|
|
383
354
|
id: $df.DataType.Int32,
|
|
384
355
|
price: $df.DataType.Decimal(10, 2),
|
|
385
356
|
active: $df.DataType.Boolean,
|
|
386
|
-
|
|
387
|
-
};
|
|
388
|
-
|
|
389
|
-
const df = $df.data(rawData, schema);
|
|
390
|
-
```
|
|
391
|
-
|
|
392
|
-
### 🧠 TypeScript Type Inference & IDE Safety
|
|
393
|
-
|
|
394
|
-
When you pass a schema to `$df.data()`, DFScript's types automatically infer the target types of the fields, turning dynamic dataframes into compile-safe records. Your IDE will auto-complete column names and validate that operations match the underlying types.
|
|
395
|
-
|
|
396
|
-
```typescript
|
|
397
|
-
import { $df } from "df-script";
|
|
398
|
-
|
|
399
|
-
const schema = {
|
|
400
|
-
name: $df.DataType.Utf8,
|
|
401
|
-
age: $df.DataType.Int32,
|
|
402
|
-
is_active: $df.DataType.Boolean
|
|
357
|
+
createdAt: $df.DataType.Datetime
|
|
403
358
|
};
|
|
404
359
|
|
|
405
|
-
// Inferred DataFrame type is DataFrame<{ name: string; age: number; is_active: boolean }>
|
|
406
360
|
const df = $df.data(rawData, schema);
|
|
407
|
-
|
|
408
|
-
// Full IDE autocomplete, type validation, and compiler safety!
|
|
409
|
-
const activeUsers = df.filter($df.col("is_active").eq(true));
|
|
410
361
|
```
|
|
411
362
|
|
|
412
363
|
### Supported Data Types
|
|
@@ -420,17 +371,16 @@ const activeUsers = df.filter($df.col("is_active").eq(true));
|
|
|
420
371
|
|
|
421
372
|
## 🧑💻 Contributing & Development
|
|
422
373
|
|
|
423
|
-
We welcome contributions! Please make sure to review our [Developer Guidelines](DEVELOPER_GUIDELINES.md) when writing code.
|
|
424
|
-
|
|
425
|
-
### Running Project Tests
|
|
426
|
-
DFScript has a comprehensive suite of unit tests. Run them using:
|
|
427
|
-
|
|
428
374
|
```bash
|
|
429
|
-
|
|
375
|
+
# Run test suite
|
|
376
|
+
npm test
|
|
377
|
+
|
|
378
|
+
# Build production bundles
|
|
379
|
+
npm run build
|
|
430
380
|
```
|
|
431
381
|
|
|
432
382
|
---
|
|
433
383
|
|
|
434
384
|
## 📄 License
|
|
435
385
|
|
|
436
|
-
|
|
386
|
+
`df-script` is open-source software licensed under the [MIT License](LICENSE).
|