@tidyjs/tidy 2.5.2 → 2.6.1

package/genai-docs/api-vector.md

@@ -0,0 +1,239 @@
+# Vector Functions API Reference
+
+Vector functions produce an array of values equal in length to the input collection. They operate across items (not one at a time), enabling running totals, lookbacks, sliding windows, and row numbering.
+
+**CRITICAL:** Vector functions go inside `mutateWithSummary()`, NOT inside `mutate()`. `mutate` passes items one at a time; `mutateWithSummary` passes the full array. Using vector functions inside `mutate()` will silently produce wrong results.
+
+```js
+import { tidy, mutateWithSummary, cumsum, lag, lead, roll, rowNumber } from '@tidyjs/tidy';
+```
+
+---
+
+<!-- keywords: mutateWithSummary, mutate summary, vector mutate, cross-item, broadcast, array function -->
+## mutateWithSummary
+
+Add or replace columns using functions that receive the full array of items.
+
+**Signature:** `mutateWithSummary(spec: Record<string, (items[]) => value[] | value | NonFunctionValue>)`
+**Goes inside:** `tidy()`
+
+### Parameters
+- `spec` -- an object where each key is a new (or existing) column name and each value is one of:
+  - A function `(items: T[]) => T[keyof T][] | T[keyof T]` -- receives the full array. If it returns an **array**, each element maps to the corresponding item. If it returns a **single value**, that value is broadcast to all items.
+  - A literal value (number, string, etc.) -- broadcast to all items.
+
+### Example
+```js
+const data = [
+  { name: 'a', val: 1 },
+  { name: 'b', val: 2 },
+  { name: 'c', val: 3 },
+];
+
+tidy(data, mutateWithSummary({
+  // array return: each element maps to the corresponding row
+  doubled: (items) => items.map(d => d.val * 2),
+  // single value return: broadcast to every row
+  total: (items) => items.reduce((s, d) => s + d.val, 0),
+  // literal value: broadcast to every row
+  flag: true,
+}));
+// output:
+// [
+//   { name: 'a', val: 1, doubled: 2, total: 6, flag: true },
+//   { name: 'b', val: 2, doubled: 4, total: 6, flag: true },
+//   { name: 'c', val: 3, doubled: 6, total: 6, flag: true },
+// ]
+```
+
+### Why not mutate?
+`mutate` processes items one at a time: `(item, index) => value`. It cannot look at other rows. Vector and summary functions need the full array, so they must go inside `mutateWithSummary`.
+
+---
+
+<!-- keywords: cumsum, cumulative sum, running total, running sum -->
+## cumsum
+
+Cumulative sum. Returns an array of running totals (uses full-precision floating-point summation).
+
+**Signature:** `cumsum(key: keyof T | (d: T, index: number, array: Iterable<T>) => number | null | undefined)`
+**Goes inside:** `mutateWithSummary()`
+
+### Parameters
+- `key` -- a property name (string) or accessor function returning a number. `null`/`undefined` values are skipped (running total stays the same).
+
+### Example
+```js
+const data = [
+  { item: 'a', val: 3 },
+  { item: 'b', val: 1 },
+  { item: 'c', val: null },
+  { item: 'd', val: 5 },
+];
+
+tidy(data, mutateWithSummary({
+  running: cumsum('val'),
+}));
+// output:
+// [
+//   { item: 'a', val: 3,    running: 3 },
+//   { item: 'b', val: 1,    running: 4 },
+//   { item: 'c', val: null, running: 4 },
+//   { item: 'd', val: 5,    running: 9 },
+// ]
+```
+
+---
+
+<!-- keywords: lag, previous value, lookback, offset, delta, difference -->
+## lag
+
+Value from N rows before the current row. Useful for computing deltas (current minus previous).
+
+**Signature:** `lag(key: keyof T | (d: T, index: number, array: Iterable<T>) => any, options?)`
+**Goes inside:** `mutateWithSummary()`
+
+### Parameters
+- `key` -- a property name or accessor function.
+- `options` (optional):
+  - `n` (number, default `1`) -- how many rows back to look.
+  - `default` (any, default `undefined`) -- fill value for the first N items where no previous row exists.
+
+### Example
+```js
+const data = [
+  { day: 1, val: 10 },
+  { day: 2, val: 15 },
+  { day: 3, val: 12 },
+];
+
+tidy(data, mutateWithSummary({
+  prev: lag('val'),
+  prev0: lag('val', { default: 0 }),
+  prev2: lag('val', { n: 2 }),
+}));
+// output:
+// [
+//   { day: 1, val: 10, prev: undefined, prev0: 0,  prev2: undefined },
+//   { day: 2, val: 15, prev: 10,        prev0: 10, prev2: undefined },
+//   { day: 3, val: 12, prev: 15,        prev0: 15, prev2: 10 },
+// ]
+```
+
+---
+
+<!-- keywords: lead, next value, lookahead, forward, offset -->
+## lead
+
+Value from N rows after the current row. Useful for computing forward differences.
+
+**Signature:** `lead(key: keyof T | (d: T, index: number, array: Iterable<T>) => any, options?)`
+**Goes inside:** `mutateWithSummary()`
+
+### Parameters
+- `key` -- a property name or accessor function.
+- `options` (optional):
+  - `n` (number, default `1`) -- how many rows ahead to look.
+  - `default` (any, default `undefined`) -- fill value for the last N items where no next row exists.
+
+### Example
+```js
+const data = [
+  { day: 1, val: 10 },
+  { day: 2, val: 15 },
+  { day: 3, val: 12 },
+];
+
+tidy(data, mutateWithSummary({
+  next: lead('val'),
+  next0: lead('val', { default: 0 }),
+  next2: lead('val', { n: 2 }),
+}));
+// output:
+// [
+//   { day: 1, val: 10, next: 15,        next0: 15, next2: 12 },
+//   { day: 2, val: 15, next: 12,        next0: 12, next2: undefined },
+//   { day: 3, val: 12, next: undefined, next0: 0,  next2: undefined },
+// ]
+```
+
+---
+
+<!-- keywords: roll, rolling, sliding window, moving average, running mean, window function -->
+## roll
+
+Rolling/sliding window operation. Applies a summary function to a window of items as it slides across the array.
+
+**Signature:** `roll(width: number, rollFn: (itemsInWindow: T[], endIndex: number) => any, options?)`
+**Goes inside:** `mutateWithSummary()`
+
+### Parameters
+- `width` (number) -- the window size (number of items).
+- `rollFn` -- a function `(itemsInWindow, endIndex) => value` applied to each window. Typically a summary function like `mean('col')`.
+- `options` (optional):
+  - `partial` (boolean, default `false`) -- if `true`, compute for windows smaller than `width` at the edges. If `false`, those positions are `undefined`.
+  - `align` (`'right'` | `'center'` | `'left'`, default `'right'`) -- window alignment relative to the current row:
+    - `'right'`: current row is the last item in the window (looks back).
+    - `'left'`: current row is the first item in the window (looks forward).
+    - `'center'`: current row is the center of the window.
+
+### Example
+```js
+import { mean } from '@tidyjs/tidy';
+
+const data = [
+  { day: 1, val: 3 },
+  { day: 2, val: 1 },
+  { day: 3, val: 3 },
+  { day: 4, val: 1 },
+  { day: 5, val: 7 },
+];
+
+tidy(data, mutateWithSummary({
+  avg3: roll(3, mean('val')),
+  avg3p: roll(3, mean('val'), { partial: true }),
+}));
+// output:
+// [
+//   { day: 1, val: 3, avg3: undefined,    avg3p: 3/1 },  // partial
+//   { day: 2, val: 1, avg3: undefined,    avg3p: 4/2 },  // partial
+//   { day: 3, val: 3, avg3: 7/3,          avg3p: 7/3 },
+//   { day: 4, val: 1, avg3: 5/3,          avg3p: 5/3 },
+//   { day: 5, val: 7, avg3: 11/3,         avg3p: 11/3 },
+// ]
+```
+
+---
+
+<!-- keywords: rowNumber, row number, index, sequential, numbering -->
+## rowNumber
+
+Assigns sequential row numbers starting from 0 (configurable).
+
+**Signature:** `rowNumber(options?)`
+**Goes inside:** `mutateWithSummary()`
+
+### Parameters
+- `options` (optional):
+  - `startAt` (number, default `0`) -- the number to assign to the first row.
+
+### Example
+```js
+const data = [
+  { name: 'a', val: 10 },
+  { name: 'b', val: 20 },
+  { name: 'c', val: 30 },
+];
+
+tidy(data, mutateWithSummary({
+  row: rowNumber(),
+  row1: rowNumber({ startAt: 1 }),
+}));
+// output:
+// [
+//   { name: 'a', val: 10, row: 0, row1: 1 },
+//   { name: 'b', val: 20, row: 1, row1: 2 },
+//   { name: 'c', val: 30, row: 2, row1: 3 },
+// ]
+```

package/genai-docs/gotchas.md

@@ -0,0 +1,193 @@
+# Gotchas and Anti-Patterns
+
+Common mistakes AI assistants make when generating tidyjs code, ordered by severity.
+
+---
+
+## 1. Using summary functions inside `mutate()` instead of `mutateWithSummary()`
+
+This is the **most dangerous mistake** — it produces code that runs without errors but returns wrong data.
+
+```js
+// WRONG — sum() receives one item at a time, not the full array
+tidy(data, mutate({ total: sum('value') }))
+
+// CORRECT — mutateWithSummary passes the full array to sum()
+tidy(data, mutateWithSummary({ total: sum('value') }))
+```
+
+**Rule:** If the function needs to see all items (summary functions like `sum`, `mean`, `median`, or vector functions like `cumsum`, `lag`, `lead`), use `mutateWithSummary`. If it only needs the current item, use `mutate`.
+
+---
+
+## 2. Using string column names instead of accessor functions
+
+tidyjs is **not** pandas or SQL. Most verbs require accessor functions.
+
+```js
+// WRONG
+tidy(data, filter('value > 10'))
+tidy(data, mutate({ doubled: 'value * 2' }))
+
+// CORRECT
+tidy(data, filter((d) => d.value > 10))
+tidy(data, mutate({ doubled: (d) => d.value * 2 }))
+```
+
+**Exception:** Summary functions and sort helpers accept string keys as shorthand:
+```js
+sum('value')       // OK — shorthand for sum((d) => d.value)
+asc('name')        // OK — shorthand for ascending sort by name
+desc('value')      // OK — shorthand for descending sort by value
+```
+
+---
+
+## 3. Forgetting the outer `tidy()` wrapper
+
+Every transformation should go through `tidy()`. Verbs are curried — they return functions, not results.
+
+```js
+// WRONG — filter() returns a function, not filtered data
+const result = filter((d) => d.active);
+// result is a function, not an array!
+
+// CORRECT
+const result = tidy(data, filter((d) => d.active));
+```
+
+---
+
+## 4. Confusing `TMath.rate()` with item-level `rate()`
+
+These are two different functions with the same name but different purposes:
+
+```js
+import { rate, TMath } from '@tidyjs/tidy';
+
+// TMath.rate — simple math: (numerator, denominator) => number
+TMath.rate(10, 100); // => 0.1
+
+// rate — item-level mutator for use inside mutate()
+tidy(data, mutate({
+  convRate: rate('conversions', 'impressions')
+}))
+// rate('conversions', 'impressions') creates (d) => d.conversions / d.impressions
+```
+
+---
+
+## 5. Not understanding groupBy export modes
+
+Without an export option, `groupBy` returns a flat array. With one, the output shape changes completely.
+
+```js
+// Returns flat array (default)
+tidy(data, groupBy('type', [summarize({ n: n() })]))
+// => [{ type: 'A', n: 5 }, { type: 'B', n: 3 }]
+
+// Returns a plain object — DIFFERENT output shape
+tidy(data, groupBy('type', [summarize({ n: n() })], groupBy.object()))
+// => { A: [{ type: 'A', n: 5 }], B: [{ type: 'B', n: 3 }] }
+```
+
+**Important:** When using an export mode, `groupBy` must be the **last step** in the pipeline. It returns a non-array type that cannot be piped further.
+
+---
+
+## 6. `select` negation syntax requires negation first
+
+The `-` prefix for excluding columns only works when negation keys come first (or all keys are negations).
+
+```js
+// CORRECT — all negations
+tidy(data, select(['-password', '-secret']))
+
+// CORRECT — negations after selector that produces all keys
+tidy(data, select([everything(), '-password']))
+
+// WRONG — mixing positive and negative keys unpredictably
+tidy(data, select(['name', '-password', 'email']))
+```
+
+---
+
+## 7. `fill()` only fills forward (down), not backward
+
+`fill` replaces `null`/`undefined` with the most recent non-null value going **downward** through the array.
+
+```js
+const data = [
+  { group: 'A', value: 10 },
+  { group: null, value: 20 },
+  { group: null, value: 30 },
+  { group: 'B', value: 40 },
+];
+
+tidy(data, fill('group'))
+// => [{ group: 'A', value: 10 }, { group: 'A', value: 20 },
+//     { group: 'A', value: 30 }, { group: 'B', value: 40 }]
+```
+
+If you need backward fill, sort your data in reverse first, fill, then sort back.
+
+---
+
+## 8. Wrapping accessors in `asc()` unnecessarily
+
+`arrange` auto-promotes single-argument accessor functions to ascending comparators. You don't need `asc()` for the basic case.
+
+```js
+// These are equivalent:
+tidy(data, arrange((d) => d.name))
+tidy(data, arrange(asc('name')))
+
+// Use desc() explicitly for descending:
+tidy(data, arrange(desc('value')))
+
+// Multiple sort keys:
+tidy(data, arrange(asc('category'), desc('value')))
+```
+
+---
+
+## 9. Relying on join auto-detection for `by` keys
+
+`innerJoin`, `leftJoin`, and `fullJoin` auto-detect matching keys from the **first element** of each array. This can break with inconsistent data shapes.
+
+```js
+// RISKY — auto-detects join keys from first element
+tidy(data, leftJoin(lookupTable))
+
+// SAFER — explicitly specify join keys
+tidy(data, leftJoin(lookupTable, { by: 'id' }))
+tidy(data, leftJoin(lookupTable, { by: { id: 'lookupId' } }))
+```
+
+---
+
+## 10. Exceeding the 10-step pipeline type limit
+
+`tidy()` has TypeScript overloads for up to 10 pipeline steps. Beyond that, types fall back to `any`.
+
+```js
+// If you need more than 10 steps, split into multiple tidy() calls:
+const intermediate = tidy(data, step1(), step2(), /* ...up to 10 */);
+const result = tidy(intermediate, step11(), step12());
+```
+
+---
+
+## 11. Using `summarize` when you want `total`
+
+`summarize` reduces to a single row. `total` appends a summary row while keeping all original rows.
+
+```js
+// summarize: replaces all rows with one summary row
+tidy(data, summarize({ sum: sum('value') }))
+// => [{ sum: 150 }]
+
+// total: keeps all rows, adds a summary row at the end
+tidy(data, total({ value: sum('value') }))
+// => [{ name: 'a', value: 50 }, { name: 'b', value: 100 }, { name: 'Total', value: 150 }]
+```

package/genai-docs/index.md

@@ -0,0 +1,44 @@
+# @tidyjs/tidy — AI Documentation
+
+> Tidy up your data with JavaScript! A data wrangling library inspired by R's dplyr and the tidyverse.
+
+## How to Use These Docs
+
+1. **Start here** — Read `mental-model.md` first. It teaches the pipeline pattern, accessor conventions, and function taxonomy that are essential for writing correct tidyjs code.
+2. **Look up functions** — Use the `api-*.md` files to find specific function signatures, parameters, and examples.
+3. **Find recipes** — Check `patterns.md` for multi-step transformation recipes.
+4. **Avoid mistakes** — Read `gotchas.md` for common AI mistakes and anti-patterns.
+5. **Quick lookup** — Use `quick-reference.md` to map a task ("I want to filter rows") to the right function.
+
+## File Index
+
+| File | Purpose |
+|------|---------|
+| [`mental-model.md`](mental-model.md) | Core concepts: pipeline pattern, accessors, function taxonomy, mutate vs mutateWithSummary |
+| [`quick-reference.md`](quick-reference.md) | Task-to-function cheat sheet |
+| [`gotchas.md`](gotchas.md) | Common mistakes and anti-patterns |
+| [`patterns.md`](patterns.md) | Multi-verb recipes for real-world tasks |
+| [`api-core.md`](api-core.md) | tidy, filter, mutate, transmute, arrange, select, distinct, rename, when, debug, map |
+| [`api-grouping.md`](api-grouping.md) | groupBy + all 8 export modes |
+| [`api-summarize.md`](api-summarize.md) | summarize, summarizeAll/At/If + summary functions (sum, mean, median, etc.) |
+| [`api-vector.md`](api-vector.md) | mutateWithSummary + vector functions (cumsum, lag, lead, roll, rowNumber) |
+| [`api-joins.md`](api-joins.md) | innerJoin, leftJoin, fullJoin |
+| [`api-pivot.md`](api-pivot.md) | pivotWider, pivotLonger |
+| [`api-slice.md`](api-slice.md) | slice, sliceHead, sliceTail, sliceMin, sliceMax, sliceSample |
+| [`api-selectors.md`](api-selectors.md) | everything, startsWith, endsWith, contains, matches, numRange, negate |
+| [`api-sequences.md`](api-sequences.md) | fullSeq, fullSeqDate, fullSeqDateISOString, vectorSeq, vectorSeqDate |
+| [`api-other.md`](api-other.md) | complete, expand, fill, replaceNully, count, tally, total, addRows, TMath |
+
+## Quick Install
+
+```bash
+npm install @tidyjs/tidy
+```
+
+```js
+import { tidy, filter, mutate, arrange, desc, groupBy, summarize, sum } from '@tidyjs/tidy';
+```
+
+## Extension Packages
+
+- `@tidyjs/tidy-moment` — Moment.js date/time extensions (not covered in these docs)