@rokkit/data 1.0.0-next.98 → 1.0.1

package/README.md

@@ -1,50 +1,162 @@
 # @rokkit/data
 
-This package provides a set of tools for managing and manipulating data in user interfaces. It is framework agnostic and can be used with any ui framework.
+Data manipulation utilities for the Rokkit design system — filtering, joining, schema inference, and dataset views.
 
-## Features
+## Install
 
-- **Datasets:** Manage and manipulate datasets with ease
-  - filtering, sorting, and aggregation.
-  - **Joins:** Combine datasets effortlessly with support for inner, outer, left, and right joins.
-  - **Rollup:** Summarize data across multiple dimensions, calculating aggregate statistics like sum, average, min, and max.
-- **Modeling:** Create and manage data models, including merging, renaming, and resolving conflicts.
-- **Renaming:** Simplify key renaming in JavaScript objects, with options for prefixes, suffixes, and custom separators.
-- **Dataviews:** Create interactive views for exploring and analyzing datasets, including sortable and filterable columns.
+```sh
+npm install @rokkit/data
+# or
+bun add @rokkit/data
+```
+
+## Overview
+
+`@rokkit/data` provides general-purpose data utilities for working with arrays of objects. It is used internally by `@rokkit/ui` table and grid components and can also be used standalone.
+
+- **Type inference** — detect column types from data samples
+- **Filtering** — apply single or multiple filters with standard operators
+- **Joins** — relational join operations across two datasets
+- **Dataset / Dataview** — composable dataset abstraction with filter, sort, and pagination
+- **Metadata** — derive column definitions and types from raw data
+- **Renaming** — remap object keys with a rename spec
+
+## Usage
+
+### Type inference
+
+```js
+import { typeOf } from '@rokkit/data'
+
+typeOf('hello') // 'string'
+typeOf(42) // 'number'
+typeOf(new Date()) // 'date'
+typeOf([1, 2]) // 'array'
+typeOf({ a: 1 }) // 'object'
+typeOf(true) // 'boolean'
+```
+
+### Filtering
 
-## Dataset
+```js
+import { filterObjectArray, filterData } from '@rokkit/data'
+
+// Single filter
+const adults = filterObjectArray(users, {
+  column: 'age',
+  value: 18,
+  operator: 'gte'
+})
+
+// Multiple filters combined
+const results = filterData(users, [
+  { column: 'status', value: 'active', operator: 'eq' },
+  { column: 'age', value: 18, operator: 'gte' }
+])
+```
 
-- The `dataset` function initializes a dataset from an array of objects.
-- This dataset comprises data rows that are sortable, filterable, and groupable.
-- Leverage dataset for SQL-like operations such as selecting, updating, and deleting rows.
-- Employ aggregate functions like sum, average, and count for summarizing multiple attributes of the data.
-- Utilize the dataset for various data operations like joining and merging data from disparate sources.
-- Establish connected data views with joins to visualize data relationships, particularly for tracking parent-child connections.
+Supported operators: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `contains`, `startsWith`, `endsWith`, `in`, `notIn`.
 
-This makes it easier to build a tweenable data structure for animations. Here is an example of how to use the `dataset` to generate a tweenable data:
+### Joins
 
 ```js
-function tweenable(data, by, group, template) {
-  return dataset(data)
-    .groupBy(by)
-    .using(template)
-    .alignBy(...group)
-    .rollup()
-    .sortBy(by)
-    .select()
-}
+import { leftJoin, innerJoin } from '@rokkit/data'
+
+const result = leftJoin(orders, customers, 'customerId', 'id')
+// Each order row merged with its matching customer row (nulls if unmatched)
 ```
 
-Refer to [dataset](docs/dataset.md) documentation for more details.
+Available joins: `innerJoin`, `leftJoin`, `rightJoin`, `fullJoin`, `crossJoin`, `antiJoin`, `semiJoin`.
+
+### Schema inference
+
+```js
+import { deriveMetadata, deriveColumns } from '@rokkit/data'
+
+const metadata = deriveMetadata(data)
+// [{ column: 'name', type: 'string' }, { column: 'age', type: 'number' }, ...]
+
+const columns = deriveColumns(data)
+// Column definitions ready for use in a table component
+```
 
-## DataView
+### Dataset and dataview
 
-- Utilize the `dataview` function to create a data view, enabling dynamic data manipulation and observation.
-- This view facilitates sorting, filtering, and grouping data while providing insight into selected or hidden rows.
-- The original dataset remains untouched while the view offers flexible data presentation options.
-- Associate actions such as select, edit, and delete with data rows using action columns within the view.
-- Manage data effectively, including parent-child links, even within a flat array structure.
+```js
+import { dataset, dataview } from '@rokkit/data'
+
+const ds = dataset(rawData)
+const view = dataview(ds, {
+  filters: [{ column: 'active', value: true, operator: 'eq' }],
+  sort: [{ column: 'name', direction: 'asc' }],
+  page: { size: 20, index: 0 }
+})
+```
+
+### Key renaming
+
+```js
+import { renamer } from '@rokkit/data'
+
+const rename = renamer({ firstName: 'first_name', lastName: 'last_name' })
+const remapped = data.map(rename)
+```
+
+### Filter string parsing
+
+```js
+import { parseFilters } from '@rokkit/data'
+
+const filters = parseFilters('age >= 18 AND status = "active"')
+```
+
+## API
+
+| Export                            | Description                                           |
+| --------------------------------- | ----------------------------------------------------- |
+| `typeOf(value)`                   | Infer type string from a value                        |
+| `filterObjectArray(data, filter)` | Apply a single filter object to an array              |
+| `filterData(data, filters)`       | Apply an array of filters to a dataset                |
+| `parseFilters(string)`            | Parse a filter expression string into filter objects  |
+| `innerJoin(a, b, keyA, keyB)`     | Inner join two arrays on matching keys                |
+| `leftJoin(a, b, keyA, keyB)`      | Left join — all rows from `a`, matched rows from `b`  |
+| `rightJoin(a, b, keyA, keyB)`     | Right join — all rows from `b`, matched rows from `a` |
+| `fullJoin(a, b, keyA, keyB)`      | Full outer join                                       |
+| `crossJoin(a, b)`                 | Cartesian product of two arrays                       |
+| `antiJoin(a, b, keyA, keyB)`      | Rows in `a` with no match in `b`                      |
+| `semiJoin(a, b, keyA, keyB)`      | Rows in `a` that have a match in `b`                  |
+| `deriveMetadata(data)`            | Infer column names and types from a data sample       |
+| `deriveColumns(data)`             | Derive column definitions from data                   |
+| `deriveSortableColumn(col)`       | Mark a column as sortable                             |
+| `dataset(data, options?)`         | Create a dataset abstraction                          |
+| `dataview(dataset, options?)`     | Create a filtered/sorted/paged view over a dataset    |
+| `renamer(mapping)`                | Create a function that renames object keys            |
+| `model(data)`                     | Create a reactive data model                          |
+
+## Benchmarks
+
+Results from `benchmark/results/p4-lazy-fusion.json` — Bun 1.3.11, Apple Silicon (arm64).
+All times in **ms (mean)** over 10 iterations. Dataset sizes: small = 100 rows, medium = 1 000 rows, large = 10 000 rows.
+
+| Operation                   | Small | Medium |  Large |
+| --------------------------- | ----: | -----: | -----: |
+| `groupBy` + `rollup`        | 0.113 |  0.236 |  1.617 |
+| `groupBy` + `summarize` + `rollup` | 0.042 | 0.097 | 0.534 |
+| `groupBy` + `alignBy` + `rollup`   | 0.133 | 0.090 | 0.085 |
+| `where` + `apply` + `sortBy`| 0.024 |  0.120 |  1.328 |
+| `filterData`                | 0.020 |  0.022 |  0.081 |
+| `leftJoin`                  | 0.044 |  0.448 | 23.620 |
+| `antiJoin`                  | 0.045 |  0.340 | 13.798 |
+| `semiJoin`                  | 0.032 |  0.172 | 11.318 |
+| `crossJoin`                 | 0.034 |  0.023 |      — |
+
+To run benchmarks locally:
+
+```sh
+bun run benchmark          # print results to console
+bun run benchmark:save     # save results to benchmark/results/baseline.json
+```
 
-Refer to [dataview](docs/dataview.md) documentation for more details.
+---
 
-The `dataview` function initializes a manageable data state with methods that allow for manipulation and observation of data changes, integrating smoothly with UI updates.
+Part of [Rokkit](https://github.com/jerrythomas/rokkit) — a Svelte 5 component library and design system.

package/package.json

@@ -1,30 +1,31 @@
 {
   "name": "@rokkit/data",
-  "version": "1.0.0-next.98",
+  "version": "1.0.1",
   "description": "Contains generic data manipulation functions that can be used in various components.",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/jerrythomas/rokkit.git"
+  },
   "author": "Jerry Thomas <me@jerrythomas.name>",
   "license": "MIT",
-  "main": "index.js",
-  "module": "src/index.js",
-  "types": "dist/index.d.ts",
   "type": "module",
   "publishConfig": {
     "access": "public"
   },
-  "devDependencies": {
-    "@sveltejs/vite-plugin-svelte": "^3.1.2",
-    "@testing-library/svelte": "^5.2.1",
-    "@types/ramda": "^0.30.2",
-    "@vitest/coverage-v8": "^1.6.0",
-    "@vitest/ui": "~1.6.0",
-    "typescript": "^5.6.2",
-    "vite": "^5.4.6",
-    "vitest": "~1.6.0",
-    "shared-config": "1.0.0-next.98"
+  "scripts": {
+    "prepublishOnly": "cp ../../LICENSE . && tsc --project tsconfig.build.json",
+    "postpublish": "rm -f LICENSE",
+    "clean": "rm -rf dist",
+    "build": "bun clean && bun prepublishOnly",
+    "benchmark": "bun benchmark/index.js",
+    "benchmark:save": "bun benchmark/index.js --save"
   },
   "files": [
     "src/**/*.js",
-    "src/**/*.svelte"
+    "dist/**/*.d.ts",
+    "README.md",
+    "package.json",
+    "LICENSE"
   ],
   "exports": {
     "./src": "./src",
@@ -37,20 +38,10 @@
   },
   "dependencies": {
     "@lukeed/uuid": "^2.0.1",
+    "@rokkit/core": "1.0.1",
     "d3-array": "^3.2.4",
     "d3-collection": "^1.0.7",
-    "ramda": "^0.30.1",
-    "svelte": "^4.2.19",
-    "@rokkit/core": "1.0.0-next.98"
-  },
-  "scripts": {
-    "format": "prettier --write .",
-    "lint": "eslint --fix .",
-    "test:ci": "vitest run",
-    "test:ui": "vitest --ui",
-    "test": "vitest",
-    "coverage": "vitest run --coverage",
-    "latest": "pnpm upgrade --latest && pnpm test:ci",
-    "release": "pnpm publish --access public"
+    "ramda": "^0.32.0",
+    "svelte": "^5.53.5"
   }
-}
+}

package/src/constants.js

@@ -1,6 +1,11 @@
-import { identity, pick } from 'ramda'
-
-export const pickAllowedConfig = pick(['children', 'actual_flag'])
+const ALLOWED_CONFIG_KEYS = new Set(['children', 'actual_flag'])
+export const pickAllowedConfig = (obj) => {
+	const result = {}
+	for (const k of ALLOWED_CONFIG_KEYS) {
+		if (k in obj) result[k] = obj[k]
+	}
+	return result
+}
 
 /** @type {import('./types').DataFrameConfig */
 export const defaultConfig = {
@@ -63,7 +68,7 @@ export const typeConverters = {
 	number: (value) => Number(value),
 	boolean: (value) => value === 'true' || value === true,
 	date: (value) => new Date(value),
-	mixed: identity
+	mixed: (x) => x
 }
 
 /**