@nteract/sift 0.5.6 → 0.6.0-nightly.39864690728

package/README.md

@@ -52,6 +52,9 @@ const load = useCallback(() => loadTableFile(bytes), [bytes]) // Parquet or Arro
 ```
 
 Files load whole or fail; defaults are 16 MiB, 100,000 rows, and 256 columns.
+Parquet can also be read by byte range through a `ByteSource` (footer first,
+then one row group at a time), which drops the 16 MiB input cap; see the
+embedding guide.
 CSV and TSV go through [`@nteract/sift-csv`](https://www.npmjs.com/package/@nteract/sift-csv), a separate WASM
 module that returns Arrow IPC bytes for `loadTableFile`, so apps that only open
 Parquet or Arrow do not carry the CSV parser.

package/docs/embedding-guide.md

@@ -111,6 +111,67 @@ paging or sampled preview.
 Lists, structs, and maps display as text. Known image structures have dedicated
 rendering.
 
+## Parquet by byte range
+
+For Parquet files you would rather not read into memory, give Sift a
+`ByteSource` instead of bytes. Sift reads the footer, then fetches each row
+group's column chunks as it decodes them; the whole file is never buffered in
+JS or WASM memory.
+
+```ts
+import type { ByteSource } from '@nteract/sift'
+
+const source: ByteSource = {
+  byteLength: stat.size,
+  // Resolve with exactly `length` bytes. The signal aborts when Sift no
+  // longer needs them (unmount, source change).
+  read: (offset, length, signal) => files.readRange(path, offset, length, { signal }),
+}
+```
+
+Back `read` with whatever the host already trusts: a file handle behind an
+IPC bridge, or a URL that honors HTTP `Range`. Return a transferable
+`Uint8Array`; avoid base64.
+
+Two ways to use it:
+
+```tsx
+// Complete or error, like loadTableFile(bytes). Resolves after every row group.
+// This path passes no AbortSignal to `read`.
+const data = await loadTableFile(source)
+
+// Progressive: the table mounts after the first row group and the rest stream
+// in behind the streaming indicator. Memoize `source`; a new identity reloads
+// and aborts the previous source's reads.
+<SiftTable source={{ kind: 'byte-source', source }} onLoadError={report} />
+```
+
+The footer must carry the `PAR1` magic; anything else is rejected after a
+single read of the file tail. `maxRows`, `maxColumns`, and `maxDecodedBytes`
+apply as for bytes, from the footer and cumulatively as row groups append.
+`maxBytes` does not: the file length is not the cap, retained memory is. A file
+whose decoded rows exceed `maxDecodedBytes` still fails; raise the limit for
+hosts with memory to spare. Rows become resident as their row group loads and
+stay resident, so sort, filter, and summaries cover every loaded row.
+
+What `maxDecodedBytes` bounds is retained Arrow memory, checked after each
+decoded batch. Two guards keep transient memory near it: a footer longer than
+the limit is rejected before it is read, and a row group whose compressed or
+footer-declared size exceeds the remaining budget is rejected before it is
+fetched. One row group's compressed bytes still pass through JS and WASM
+memory while it decodes. Files with column chunks past 4 GiB are rejected at
+open; the WASM reader cannot address them.
+
+If a row group fails after the table has mounted (a read error, the decoded
+limit), the rows already loaded stay on screen, the streaming indicator
+settles, Sift shows the error, and `onLoadError` fires.
+
+Only Parquet can be read this way. A `ByteSource` with `format: 'arrow-ipc'` is
+read whole and goes through the bytes path with its `maxBytes` check.
+
+`onLoadError` fires for `loader`, `url`, `byte-source`, and manifest sources
+when a load fails, with the same `Error` Sift renders in its error state.
+
 ## Lifecycle and host controls
 
 A `TableData` instance returned by `loadTableFile` owns one WASM store. The React
@@ -131,7 +192,9 @@ responsibilities. Parquet can use a view-only mode.
 
 For already hosted data, `<SiftTable url="/data.arrow" />` and
 `<SiftTable url="/data.parquet" />` remain available. The URL/manifest paths have
-different loading behavior and do not inherit `loadTableFile` admission limits.
+different loading behavior and do not inherit `loadTableFile` admission limits:
+a Parquet URL streams row groups through the same loader as `byte-source`, with
+the limits set to their maxima. A schema-only Parquet URL mounts an empty table.
 A sampled Arrow manifest must include its `summary.sampled` and included/total
 row metadata; its sorting, filtering, and summaries cover only the supplied rows.
 

package/lib/THIRD_PARTY_NOTICES.md

@@ -40,11 +40,14 @@ Generated by `cargo about` from `about.toml` for sift-wasm.
 | const-random-macro | 0.1.16 | MIT OR Apache-2.0 | https://github.com/tkaitchuck/constrandom |
 | crc32fast | 1.5.0 | MIT OR Apache-2.0 | https://github.com/srijs/rust-crc32fast |
 | crunchy | 0.2.4 | MIT | https://github.com/eira-fransham/crunchy |
+| equivalent | 1.0.2 | Apache-2.0 OR MIT | https://github.com/indexmap-rs/equivalent |
 | flatbuffers | 24.12.23 | Apache-2.0 | https://github.com/google/flatbuffers |
 | flate2 | 1.1.9 | MIT OR Apache-2.0 | https://github.com/rust-lang/flate2-rs |
 | getrandom | 0.2.17 | MIT OR Apache-2.0 | https://github.com/rust-random/getrandom |
 | half | 2.7.1 | MIT OR Apache-2.0 | https://github.com/VoidStarKat/half-rs |
 | hashbrown | 0.15.5 | MIT OR Apache-2.0 | https://github.com/rust-lang/hashbrown |
+| hashbrown | 0.17.0 | MIT OR Apache-2.0 | https://github.com/rust-lang/hashbrown |
+| indexmap | 2.14.0 | Apache-2.0 OR MIT | https://github.com/indexmap-rs/indexmap |
 | integer-encoding | 3.0.4 | MIT | https://github.com/dermesser/integer-encoding-rs |
 | itoa | 1.0.18 | MIT OR Apache-2.0 | https://github.com/dtolnay/itoa |
 | js-sys | 0.3.95 | MIT OR Apache-2.0 | https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys |
@@ -821,7 +824,7 @@ Apache License
 
 ### Apache License 2.0 (Apache-2.0)
 
-Used by: ahash@0.8.12, base64@0.22.1, bitflags@1.3.2, bitflags@2.11.1, bumpalo@3.20.2, cfg-if@1.0.4, const-random-macro@0.1.16, const-random@0.1.18, flate2@1.1.9, hashbrown@0.15.5, js-sys@0.3.95, num-bigint@0.4.6, num-complex@0.4.6, num-integer@0.1.46, num-iter@0.1.45, num-rational@0.4.2, num-traits@0.2.19, num@0.4.3, once_cell@1.21.4, regex-automata@0.4.14, regex-syntax@0.8.10, regex@1.12.3, wasm-bindgen-macro-support@0.2.118, wasm-bindgen-macro@0.2.118, wasm-bindgen-shared@0.2.118, wasm-bindgen@0.2.118
+Used by: ahash@0.8.12, base64@0.22.1, bitflags@1.3.2, bitflags@2.11.1, bumpalo@3.20.2, cfg-if@1.0.4, const-random-macro@0.1.16, const-random@0.1.18, equivalent@1.0.2, flate2@1.1.9, hashbrown@0.15.5, hashbrown@0.17.0, indexmap@2.14.0, js-sys@0.3.95, num-bigint@0.4.6, num-complex@0.4.6, num-integer@0.1.46, num-iter@0.1.45, num-rational@0.4.2, num-traits@0.2.19, num@0.4.3, once_cell@1.21.4, regex-automata@0.4.14, regex-syntax@0.8.10, regex@1.12.3, wasm-bindgen-macro-support@0.2.118, wasm-bindgen-macro@0.2.118, wasm-bindgen-shared@0.2.118, wasm-bindgen@0.2.118
 
 ```
 Apache License