@nteract/sift 0.0.1 → 0.5.5-nightly.38500811922

package/README.md

@@ -1,13 +1,106 @@
-# @nteract/sift 0.0.1
+# Sift
 
-This version only reserves the package name so nteract's release workflow can
-publish through npm trusted publishing. It contains no code.
+Crossfilter data explorer. [Pretext](https://github.com/chenglou/pretext) × [Arrow](https://arrow.apache.org/).
 
-Install a real build instead:
+![Sift — crossfilter in action](https://img.runt.run/sift/crossfilter-demo.gif)
+
+Published as [`@nteract/sift`](https://www.npmjs.com/package/@nteract/sift). Try it at [rgbkrk.github.io/sift](https://rgbkrk.github.io/sift/).
+
+## Features
+
+- **Local CSV/TSV, Parquet, and Arrow IPC streams** — bounded file adapters, no notebook or server needed
+- **Stream hosted Arrow IPC or load Parquet URLs** — WASM-native
+- **Crossfilter** — brush histograms to filter, all other summaries update instantly
+- **Column profiling** — null%, distinct count shown inline in headers
+- **Smart summaries** — histograms, category bars, boolean ratios, binary value bars, searchable popovers. Adapts visualization to column type × cardinality.
+- **Column type casting** — right-click to treat text as numbers, dates, booleans — with full undo
+- **Smart column detection** — pandas index columns, HuggingFace ClassLabel features
+- **Virtual scroll** — 100k+ rows at 120fps. Pretext layout at 0.0002ms/cell.
+- **Odometer status bar** — rolling digit animation for row count, visible range, FPS
+- **Multi-instance** — embed multiple tables on one page (notebook demo included)
+- **RxJS streaming** — cancellable dataset loading via `switchMap`
+- **Mobile** — tap any row for a detail sheet with all column values
+- **Dark mode**, keyboard nav, pinned columns, ARIA grid
+- **8 HuggingFace datasets** built into the demo
+
+## Quick start
+
+From the nteract repository root:
+
+```sh
+pnpm install --frozen-lockfile
+cargo xtask wasm sift --skip-renderer-plugins
+pnpm --dir packages/sift dev
+```
+
+## Use as a library
+
+The package ships JavaScript, compiled CSS, TypeScript declarations, and WASM,
+with notices for the Rust crates compiled into the WASM in
+`lib/THIRD_PARTY_NOTICES.md`. React and React DOM are peer dependencies; no
+notebook, kernel, or daemon is needed. Tested with React 19. Types resolve
+under TypeScript's `bundler` module resolution (Vite, webpack, esbuild).
+
+```tsx
+import { loadTableFile, SiftTable } from '@nteract/sift'
+import '@nteract/sift/style.css'
+
+const load = useCallback(() => loadTableFile(bytes), [bytes]) // Parquet or Arrow IPC
+<SiftTable source={{ kind: 'loader', load }} style={{ height: 600 }} />
+```
+
+Files load whole or fail; defaults are 16 MiB, 100,000 rows, and 256 columns.
+CSV and TSV go through [`@nteract/sift-csv`](../sift-csv), a separate WASM
+module that returns Arrow IPC bytes for `loadTableFile`, so apps that only open
+Parquet or Arrow do not carry the CSV parser.
+
+See the [embedding guide](docs/embedding-guide.md) for WASM asset setup, memory
+limits, lifecycle, styling, and host viewer/editor integration.
+
+## Engine API
+
+```ts
+engine.setSort('age', 'desc')
+engine.setFilter(colIndex, { kind: 'range', min: 18, max: 65 })
+engine.clearAllFilters()
+
+// Inspect the current sort, filters, and counts.
+const state = engine.getState()
+```
+
+See [`docs/embedding-guide.md`](docs/embedding-guide.md) for the full API.
+
+## WASM compute (`sift-wasm`)
+
+Rust crate wrapping arrow-rs. Parquet decoding, sort, histograms, value counts, string search, column casting — all in one WASM binary. Data lives in WASM memory; JS holds handles.
 
 ```sh
-npm install --save-exact @nteract/sift@nightly
+cargo xtask wasm sift --skip-renderer-plugins
 ```
 
-Stable versions publish to `latest` from the nteract release workflow. See
-https://github.com/nteract/nteract/tree/main/packages/sift.
+## Stack
+
+| Layer | What |
+|-------|------|
+| Layout | [@chenglou/pretext](https://github.com/chenglou/pretext) — DOM-free text measurement |
+| Data | [Apache Arrow](https://arrow.apache.org/) — columnar, streamed via RecordBatchReader |
+| Compute | [sift-wasm](../../crates/sift-wasm/) — arrow-rs WASM kernels |
+| Reactivity | [RxJS](https://rxjs.dev/) — streaming pipeline |
+| Charts | SVG histograms + HTML bars |
+
+## Development
+
+Run from the repository root after the setup above:
+
+```sh
+pnpm --dir packages/sift test          # Vitest unit tests
+pnpm --dir packages/sift test:e2e      # Playwright E2E
+pnpm --dir packages/sift build         # production build (demo)
+pnpm --dir packages/sift build:lib     # lib/ (JS, CSS, types, WASM)
+pnpm --dir packages/sift test:package  # packed consumer outside the workspace
+```
+
+## Demo pages
+
+- `/` — main demo with dataset picker
+- `/notebook.html` — multi-table notebook simulation

package/docs/embedding-guide.md

@@ -0,0 +1,185 @@
+# Embedding Sift
+
+`@nteract/sift` is a browser table viewer with compiled JavaScript, CSS, TypeScript
+interfaces, and a matching WebAssembly decoder. It does not require a notebook,
+kernel, daemon, Tauri, Electron, or an nteract runtime connection. React and React
+DOM are required even for the imperative API because column controls use React.
+
+## Install and assets
+
+```sh
+npm install @nteract/sift react react-dom
+```
+
+```tsx
+import { SiftTable, setWasmUrl } from '@nteract/sift'
+import '@nteract/sift/style.css'
+// Vite example: emit the packaged binary through the host's asset pipeline.
+import wasmUrl from '@nteract/sift/sift.wasm?url'
+
+setWasmUrl(wasmUrl) // Configure once, before the first binary table loads.
+```
+
+Other bundlers can copy `@nteract/sift/sift.wasm` into their local assets and pass
+its served URL to `setWasmUrl`. Without an override, the shipped wasm-bindgen glue
+resolves `sift_wasm_bg.wasm` relative to itself; preserve that layout when serving
+unbundled modules. Serve the binary with `application/wasm`. Keep the JavaScript
+and WASM from the same package version. No CDN or data upload is required.
+
+The host owns file access and its existing sandbox/CSP policy. In an Electron
+app, read selected files through the approved main/preload bridge and pass text
+or bytes to the renderer; Sift does not need Node integration or filesystem
+access. The package's URL API explicitly fetches URLs supplied by the host.
+
+## Local CSV and TSV
+
+CSV parsing is a separate package, `@nteract/sift-csv`, so apps that only open
+Parquet or Arrow do not carry it. It converts text or bytes to an Arrow IPC
+stream that `loadTableFile` accepts:
+
+```tsx
+import { csvToArrowIpc } from '@nteract/sift-csv'
+import { loadTableFile, SiftTable } from '@nteract/sift'
+
+const load = useCallback(async () => {
+  const ipc = await csvToArrowIpc(text, { delimiter: '\t' })
+  return loadTableFile(ipc, { format: 'arrow-ipc' })
+}, [text])
+
+<SiftTable source={{ kind: 'loader', load }} style={{ height: 600 }} />
+```
+
+Type inference reads the whole file. Booleans, Int64, Float64 (including
+exponents), ISO dates, and ISO timestamps are detected; a column with mixed
+shapes stays text. Empty fields are null in every type. Malformed quoting and
+ragged rows reject with the reason; nothing is truncated or sampled. The host
+owns text decoding and delimiter choice; there is no guessing. Defaults are
+16 MiB of input, 100,000 rows, and 256 columns, each an option. See the
+package README for the WASM asset setup.
+
+## Local Parquet and Arrow
+
+```ts
+import { loadTableFile, createTable } from '@nteract/sift'
+
+const data = await loadTableFile(bytes) // Uint8Array; detects Parquet magic.
+const engine = createTable(container, data)
+// When the document closes or changes:
+engine.destroy() // Also disposes the data's WASM store.
+```
+
+For React, use a memoized loader returning fresh data for every invocation.
+This also supports React StrictMode's setup/cleanup replay:
+
+```tsx
+import { useCallback } from 'react'
+import { loadTableFile, SiftTable } from '@nteract/sift'
+
+function ParquetView({ bytes }: { bytes: Uint8Array }) {
+  const load = useCallback(() => loadTableFile(bytes), [bytes])
+  return <SiftTable source={{ kind: 'loader', load }} style={{ height: 600 }} />
+}
+```
+
+Keep the bytes snapshot stable until the file changes. Do not capture and return
+one previously loaded disposable table from the callback.
+
+`loadTableFile` supports Parquet and uncompressed Arrow IPC streams. The optional
+`format` is `'parquet'` or `'arrow-ipc'`. CSV and TSV go through `@nteract/sift-csv`. Arrow file
+containers and compressed IPC streams are not supported by this path.
+
+Binary input defaults to 16 MiB, 100,000 rows, 256 columns, and 128 MiB of
+retained Arrow memory (`maxBytes`, `maxRows`, `maxColumns`, `maxDecodedBytes`).
+Rows and columns are checked from file metadata before decoding. Retained memory
+is checked after each decoded batch, and the decoder can allocate one batch past
+the limit before that check runs. These limits are meant for files you chose to
+open, not a sandbox for hostile input: a crafted Parquet file can make one batch
+large. If you load untrusted files, do it in a Worker you can terminate.
+
+This API loads a complete file. Virtual scrolling bounds rendered DOM, not the
+dataset: sorting, filtering, and summaries cover every accepted row. There is no
+paging or sampled preview.
+
+Lists, structs, and maps display as text. Known image structures have dedicated
+rendering.
+
+## Lifecycle and host controls
+
+A `TableData` instance returned by `loadTableFile` owns one WASM store. The React
+loader source creates a fresh store for each setup and discards stale completions;
+the mounted engine disposes the store on teardown. Avoid passing preloaded binary
+data directly to the React `data` prop, because effect replay would reuse a freed
+store. For the imperative API, pass each result to one `createTable` instance. If
+an imperative load is stale or never mounted, call `data.dispose?.()`, using an
+activation token to recognize results completed after switching or closing files.
+Reopen/reload creates fresh data. Do not share disposable data between tables.
+
+Sift is a read-only exploration surface. Sorting, filtering, and type casts do
+not rewrite the file. Keep editor text and dirty state in the host; a viewer/editor
+toggle should derive its preview from the current text buffer and preserve edits
+on parse errors or mode changes. Save, Save As/export, reveal-in-folder, file
+watching, dirty-file conflict handling, and loading/error messages remain host
+responsibilities. Parquet can use a view-only mode.
+
+For already hosted data, `<SiftTable url="/data.arrow" />` and
+`<SiftTable url="/data.parquet" />` remain available. The URL/manifest paths have
+different loading behavior and do not inherit `loadTableFile` admission limits.
+A sampled Arrow manifest must include its `summary.sampled` and included/total
+row metadata; its sorting, filtering, and summaries cover only the supplied rows.
+
+## Interaction and appearance
+
+```ts
+engine.setSort('age', 'desc')
+engine.setFilter(0, { kind: 'range', min: 18, max: 65 })
+engine.clearAllFilters()
+const state = engine.getState()
+```
+
+The container needs a defined height. The table supplies keyboard navigation and
+ARIA grid semantics.
+
+## Styling
+
+`style.css` is compiled and self-contained. It writes nothing to `:root`,
+`html`, or `body` and ships no document-wide CSS reset, so it can load into a host document
+next to the host's own stylesheet. Every rule is keyed on `.sift-root`, which
+Sift puts on the table container and on each element it appends to `<body>`
+(popovers, the column menu, the row detail sheet, the image viewer).
+
+Dark mode follows the host: put `.dark` or `data-theme="dark"` on `<html>` or
+`<body>`, so body-mounted overlays receive the same theme.
+
+Sift reads eight colors, a font, and a radius. Set them on `<html>` or `<body>`. A table-local wrapper does not theme overlays
+that are portaled to `<body>`:
+
+```css
+:root {
+  --sift-color-bg: #fff;        /* table background */
+  --sift-color-panel: #fff;     /* header, popovers, menus, sheets */
+  --sift-color-ink: #111;       /* text */
+  --sift-color-muted: #666;     /* secondary text, null markers */
+  --sift-color-rule: #e5e5e5;   /* borders and grid lines */
+  --sift-color-accent: #2563eb; /* histogram bars, selection, focus ring */
+  --sift-color-true: #22c55e;   /* boolean true */
+  --sift-color-false: #ef4444;  /* boolean false */
+  --sift-font-family: Inter, system-ui, sans-serif;
+  --sift-radius-base: 0.5rem;
+}
+```
+
+Every other color (row stripes, scrollbars, badges, skeletons) is derived from
+these with `color-mix()`, so they follow light and dark automatically.
+
+The neutrals fall back to the shadcn token names when the host defines them:
+`--background`, `--card`, `--foreground`, `--muted-foreground`, `--border`, and
+`--radius`. A shadcn app gets Sift in its own palette without setting anything.
+Accent and boolean colors keep Sift's defaults unless set, since a shadcn
+`--primary` is a button color and often near-black.
+
+## Notices
+
+`lib/THIRD_PARTY_NOTICES.md` lists the Rust crates compiled into the WASM with
+their license texts and NOTICE files. Ship it with any app that redistributes
+Sift. Building and verifying the package from source is covered in the
+package's `AGENTS.md`.

package/lib/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, runtimed
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.