@nteract/sift 0.6.0 → 0.7.0-nightly.42433965924

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. package/README.md +11 -2
  2. package/docs/embedding-guide.md +417 -10
  3. package/lib/THIRD_PARTY_NOTICES.md +29 -4
  4. package/lib/handoff.css +12 -1
  5. package/lib/index.js +2847 -2052
  6. package/lib/protocol-B_bRY_76.js +885 -0
  7. package/lib/style.css +1 -1
  8. package/lib/types/packages/sift/src/auto-width.d.ts +9 -3
  9. package/lib/types/packages/sift/src/components/icons.d.ts +16 -0
  10. package/lib/types/packages/sift/src/file-data.d.ts +34 -5
  11. package/lib/types/packages/sift/src/filter-schema.d.ts +11 -0
  12. package/lib/types/packages/sift/src/index.d.ts +11 -4
  13. package/lib/types/packages/sift/src/parquet-file.d.ts +52 -5
  14. package/lib/types/packages/sift/src/permutation.d.ts +24 -0
  15. package/lib/types/packages/sift/src/predicate.d.ts +56 -3
  16. package/lib/types/packages/sift/src/query.d.ts +100 -0
  17. package/lib/types/packages/sift/src/react.d.ts +108 -6
  18. package/lib/types/packages/sift/src/table.d.ts +168 -2
  19. package/lib/types/packages/sift/src/wasm-summaries.d.ts +19 -1
  20. package/lib/types/packages/sift/src/windowed-resident.d.ts +20 -0
  21. package/lib/types/packages/sift/src/windowed.d.ts +116 -0
  22. package/lib/types/packages/sift/src/worker/client.d.ts +87 -0
  23. package/lib/types/packages/sift/src/worker/entry.d.ts +11 -0
  24. package/lib/types/packages/sift/src/worker/factory.d.ts +11 -0
  25. package/lib/types/packages/sift/src/worker/host.d.ts +31 -0
  26. package/lib/types/packages/sift/src/worker/protocol.d.ts +191 -0
  27. package/lib/wasm/sift_wasm.d.ts +67 -3
  28. package/lib/wasm/sift_wasm.js +109 -3
  29. package/lib/wasm/sift_wasm_bg.wasm +0 -0
  30. package/lib/worker.js +277 -0
  31. package/package.json +9 -6
package/README.md CHANGED
@@ -53,8 +53,17 @@ const load = useCallback(() => loadTableFile(bytes), [bytes]) // Parquet or Arro
53
53
 
54
54
  Files load whole or fail; defaults are 16 MiB, 100,000 rows, and 256 columns.
55
55
  Parquet can also be read by byte range through a `ByteSource` (footer first,
56
- then one row group at a time), which drops the 16 MiB input cap; see the
57
- embedding guide.
56
+ then one row group at a time), which drops the 16 MiB input cap. With
57
+ `overflow: 'truncate'` it shows the leading rows that fit the row and memory
58
+ budget instead of failing, and `autoLoad` pauses at a soft budget with a
59
+ "Load more" button; see the embedding guide. Every summary labels what
60
+ its numbers cover (`scope`): histograms are over loaded rows, while range and
61
+ null count come from Parquet footer statistics for the whole file when present.
62
+ Data behind a database or scan service plugs in through a `query-session`
63
+ source: the host implements `TableQuerySession`, Sift shows a bounded head and
64
+ applies brushes to the whole dataset on request; see the embedding guide.
65
+ `worker: true` on a byte-range Parquet source decodes in the `@nteract/sift/worker`
66
+ data worker under an allocation cap, once the host registers `setWorkerFactory`.
58
67
  CSV and TSV go through [`@nteract/sift-csv`](https://www.npmjs.com/package/@nteract/sift-csv), a separate WASM
59
68
  module that returns Arrow IPC bytes for `loadTableFile`, so apps that only open
60
69
  Parquet or Arrow do not carry the CSV parser.
@@ -106,7 +106,8 @@ large. If you load untrusted files, do it in a Worker you can terminate.
106
106
 
107
107
  This API loads a complete file. Virtual scrolling bounds rendered DOM, not the
108
108
  dataset: sorting, filtering, and summaries cover every accepted row. There is no
109
- paging or sampled preview.
109
+ paging or sampled preview. Byte-range Parquet can load a bounded leading
110
+ prefix instead; see [Bounded previews](#bounded-previews).
110
111
 
111
112
  Lists, structs, and maps display as text. Known image structures have dedicated
112
113
  rendering.
@@ -163,14 +164,368 @@ memory while it decodes. Files with column chunks past 4 GiB are rejected at
163
164
  open; the WASM reader cannot address them.
164
165
 
165
166
  If a row group fails after the table has mounted (a read error, the decoded
166
- limit), the rows already loaded stay on screen, the streaming indicator
167
- settles, Sift shows the error, and `onLoadError` fires.
167
+ limit), the rows already loaded stay on screen and usable. Sift does not show
168
+ its blocking error box. Instead its footer reads "Stopped loading: <message> ·
169
+ Showing loaded rows" and the status indicator turns to a warning mark whose
170
+ tooltip names the failure. `onLoadError` still fires, so the host can log it.
171
+ A failure before mount shows the error box as usual. To tell the two apart,
172
+ check whether the `engine-mounted` milestone arrived for this load. The next
173
+ source to load clears the notice. A Parquet `url` streams through the same
174
+ loader and behaves the same way.
175
+
176
+ ### Bounded previews
177
+
178
+ To open files of any size and show the leading rows, set `overflow: 'truncate'`.
179
+ Rows and decoded bytes become a budget instead of an admission check:
180
+
181
+ ```tsx
182
+ <SiftTable
183
+ source={{
184
+ kind: 'byte-source',
185
+ source,
186
+ limits: { maxRows: 1_000_000, maxDecodedBytes: 512 * 1024 * 1024 },
187
+ overflow: 'truncate',
188
+ }}
189
+ onLoadMilestone={(m) => {
190
+ if (m.phase === 'streaming-complete' && m.truncated) {
191
+ console.log(`Showing first ${m.rowCount} of ${m.totalRowCount} rows`)
192
+ }
193
+ }}
194
+ />
195
+
196
+ // loadTableFile accepts the same option for a ByteSource.
197
+ const data = await loadTableFile(source, { maxRows: 100_000, overflow: 'truncate' })
198
+ ```
199
+
200
+ - The footer is not rejected for its total row count. `maxColumns` and the
201
+ footer-size check still reject, and so does a file where one row's
202
+ fixed-width columns alone exceed `maxDecodedBytes`.
203
+ - Row groups decode in 256-row batches. Loading stops at exactly `maxRows`,
204
+ at the rows whose fixed-width values fit what is left of the budget, or
205
+ before the first batch that would take retained memory past
206
+ `maxDecodedBytes`. Once the budget is spent, later row groups are never
207
+ read.
208
+ - A row group whose compressed column chunks exceed `maxDecodedBytes` fails
209
+ before it is fetched, and gaps between chunks are fetched only while the
210
+ request stays within that size. If the budget cannot hold even the first
211
+ batch, the load fails: there is nothing to preview.
212
+ - String and binary view columns are copied out of the decoded pages, so
213
+ retained memory is what the budget counts.
214
+ - The `streaming-complete` milestone carries `rowCount` (loaded rows),
215
+ `totalRowCount` (footer rows), and `truncated`. When truncated, Sift's footer
216
+ shows "Showing first N of M rows" and the status tooltip reads "Preview
217
+ loaded". Sorting, filtering, and summaries cover only the loaded rows.
218
+
219
+ The default for `byte-source` and `loadTableFile`, `overflow: 'error'`, keeps
220
+ the behavior described above; a Parquet `url` source defaults to `'truncate'`
221
+ (see below). Only streaming Parquet honors `overflow`; in-memory buffers and
222
+ Arrow IPC still fail on overflow.
223
+
224
+ The budget is a loading guardrail, like the other limits, not isolation from
225
+ hostile files. Page decompression and one decoder batch allocate before a
226
+ batch is measured. Load untrusted files in a Worker you can terminate.
227
+
228
+ ### Load more on demand
229
+
230
+ A truncating load can also stop early on purpose. `autoLoad` sets a soft budget
231
+ below `limits`: Sift loads row groups until the committed rows or decoded bytes
232
+ reach it, then pauses with the rows loaded so far and lets the user ask for
233
+ more.
234
+
235
+ ```tsx
236
+ <SiftTable
237
+ source={{
238
+ kind: 'byte-source',
239
+ source,
240
+ limits: { maxRows: 5_000_000, maxDecodedBytes: 1024 * 1024 * 1024 },
241
+ overflow: 'truncate',
242
+ autoLoad: { rows: 100_000, decodedBytes: 64 * 1024 * 1024, trigger: 'scroll' },
243
+ }}
244
+ />
245
+ ```
246
+
247
+ - `autoLoad` needs `overflow: 'truncate'`; with `'error'` the source is
248
+ rejected before anything is read. It applies to `byte-source` (including
249
+ `worker: true`, where the worker pauses and waits) and to Parquet behind a
250
+ `url` source. `rows` and `decodedBytes` are each optional; the pause
251
+ comes at whichever is reached first.
252
+ - At a pause the table is at rest: the status indicator settles, casts are
253
+ allowed again, and the footer reads "Showing first N of M rows" with a
254
+ `Load more` button (`.sift-load-more`). Each press extends the soft budget by
255
+ the same `rows` and `decodedBytes` and loads until it is reached again, the
256
+ hard budget is spent, or the file ends. While a resume is in flight the button
257
+ reads "Loading…" and is disabled. With `trigger: 'scroll'`, scrolling within
258
+ the overscan of the last loaded row also resumes; the button stays.
259
+ - Milestones: a pause emits `streaming-complete` with `truncated: true` and
260
+ `resumable: true`, so hosts that clear their loading state on
261
+ `streaming-complete` keep doing so. A resume emits `streaming-resumed` with
262
+ `rowCount`, then the usual `chunk-appended` milestones, then
263
+ `streaming-complete` again: `resumable: true` for another pause, or
264
+ `resumable: false` once the file finished or the hard budget cut it, with
265
+ `truncated` meaning what it means for any truncating load.
266
+ - `TableData.coverage` reports `truncated: true` and `resumable: true` while
267
+ paused, and `resumable: false` once the load finished. Summaries, sort, and
268
+ filters cover the loaded rows throughout.
269
+ - Unmounting or changing the source while paused closes the footer and aborts a
270
+ resume in flight, as for any streaming load.
168
271
 
169
272
  Only Parquet can be read this way. A `ByteSource` with `format: 'arrow-ipc'` is
170
273
  read whole and goes through the bytes path with its `maxBytes` check.
171
274
 
172
275
  `onLoadError` fires for `loader`, `url`, `byte-source`, and manifest sources
173
- when a load fails, with the same `Error` Sift renders in its error state.
276
+ when a load fails, with the same `Error` Sift shows. For streaming Parquet it
277
+ may arrive after rows mounted; see above.
278
+
279
+ ### What the numbers cover
280
+
281
+ A table loaded from a file carries three pieces of provenance:
282
+
283
+ - `TableData.coverage` says how much of the dataset is resident: `loadedRows`,
284
+ `totalRows` (with its own scope, precision, and source), the file's
285
+ `rowGroups`, and `truncated`. `loadTableFile` sets it on every path, so a
286
+ caller of `loadTableFile(source, { overflow: 'truncate' })` reads
287
+ `data.coverage.truncated` to learn the result was cut. Absent means every row
288
+ is resident.
289
+ - `TableData.datasetStats` holds one `ColumnFileStats` per column, read from
290
+ the Parquet footer at open and never changed: a whole-file null count when
291
+ every row group recorded one, min/max when every row group had usable bounds,
292
+ and flags for inexact (truncated string) bounds and infinite values. Columns
293
+ without usable footer statistics are `null`.
294
+ - Each non-null `ColumnSummary` has `scope`, which labels every metric
295
+ separately. `scope.distribution` covers the histogram bins, category bars,
296
+ and true/false counts; it is always computed over loaded rows and says
297
+ `dataset` only when every row is resident. `scope.range` and
298
+ `scope.nullCount` come from footer metadata when the footer has them, and
299
+ then describe the whole file; otherwise they describe loaded rows.
300
+ `scope.distinctCount` is always over loaded rows. A cast column ignores its
301
+ footer entry, since that entry describes the stored type.
302
+
303
+ The header renders these as text. The histogram and its brush domain stay over
304
+ loaded rows. When the file has more rows than are loaded, a numeric or timestamp
305
+ header adds a line with the whole-file bounds ("0 – 100 · whole file", with "≈"
306
+ when the bounds are inexact), the null percentage names its scope ("20% null ·
307
+ whole file" or "20% null · loaded rows"), the category popover keeps its "Search
308
+ N values…" and "All" controls and adds a note "Values from loaded rows · N of M"
309
+ that both controls reference through `aria-describedby`, and a boolean column keeps its loaded-rows bar
310
+ and shows whole-file nulls on a separate line. Captions carry `data-scope` and
311
+ `data-precision` attributes. `TableEngineState.loadedCount` reports resident
312
+ rows alongside `totalCount`, which keeps its current meaning.
313
+
314
+ ## Query sessions
315
+
316
+ When the data lives behind a database, a scan service, or anything that can
317
+ answer a filter without shipping the whole table, the host implements
318
+ `TableQuerySession` and hands it to a `query-session` source. Sift sends a
319
+ `TableQuery` and receives one bounded Arrow IPC stream; it loads that stream
320
+ through the same resident path as a file and records that the rows already
321
+ satisfy the query. It does not filter them again, and the footer and pills
322
+ describe the committed query rather than a local one.
323
+
324
+ ```tsx
325
+ import type { TableQuerySession } from '@nteract/sift'
326
+
327
+ const session: TableQuerySession = {
328
+ capabilities: {
329
+ filters: { price: ['between'], category: ['in', 'not_in', 'is_null'] },
330
+ sort: [],
331
+ casts: [],
332
+ counts: ['exact'],
333
+ compound: ['or'], // for Sift's category exclusion, or(is_null, not_in)
334
+ },
335
+ async query(q, ctx) {
336
+ // Compile q.filters to your dialect with bound parameters, run it with
337
+ // ctx.signal and ctx.budget, and return one uncompressed IPC stream.
338
+ return { ipc, returnedRows, matchedCount: { kind: 'exact', value }, continuation: { kind: 'more' } }
339
+ },
340
+ async count(q, mode, ctx) { /* COUNT(*) or an estimate */ },
341
+ async close() { /* release the connection */ },
342
+ }
343
+
344
+ <SiftTable source={{ kind: 'query-session', session, headRows: 10_000 }} />
345
+ ```
346
+
347
+ The contract (`src/query.ts`):
348
+
349
+ - `TableQuery` is `casts`, `filters` (top-level AND of `FilterPredicate`), and
350
+ `sort`. The vocabulary is the same name-addressed predicate tree that
351
+ `engineStateToExplorerState` produces; SQL and pandas text are exports, not the
352
+ wire format. The host owns the SQL dialect, the trusted base relation, the
353
+ connection, credentials, and placement. Sift never receives SQL text.
354
+ - `capabilities` says what the provider can run: leaf ops per column, sortable
355
+ columns, casts, which count kinds it offers, and which compound predicates it
356
+ accepts. `supportsQuery(caps, q)` checks a query against it and returns the
357
+ reason when something is missing. A provider still validates the whole query
358
+ and refuses what it cannot run; it never narrows a result silently.
359
+ - `query` takes a `window` of `{ mode: 'head', limit }` and returns a
360
+ `QueryPage`: the IPC stream (schema-only when empty), `returnedRows`,
361
+ `matchedCount` (a `Count`: exact, estimate, lower bound, or unknown, always for
362
+ the query, never the base relation), and a `continuation` that says whether
363
+ more rows matched. `RequestContext` carries a `requestId`, an `AbortSignal`,
364
+ and a byte budget. Sift aborts a request when a newer query supersedes it or
365
+ the source unmounts; a late answer is dropped and its store freed.
366
+ - `count` is separate so first paint never waits on `COUNT(*)`. Sift asks for a
367
+ `'cheap'` count when the page's `matchedCount` is `unknown` and the provider
368
+ lists a count kind.
369
+
370
+ What the user sees:
371
+
372
+ - The footer's primary line is the whole-dataset count with the shown rows:
373
+ "N whole-dataset matches · K rows shown", "≈N whole-dataset matches ·
374
+ Estimated · K rows shown", "≥N …", or "Counting whole-dataset matches… · K
375
+ rows shown" while the count is pending. A query with no filters says "rows in
376
+ whole dataset" instead of "matches".
377
+ - Committed filters render as pills with `.sift-filter-pill--committed` and the
378
+ title "Applied to the whole dataset". Clearing one does not touch the resident
379
+ rows; it runs the reduced query against the source. Removing a filter must
380
+ query the source, not the already-filtered result, or the rows it hid could
381
+ never come back.
382
+ - Brushes and category clicks stay local, over the shown rows, exactly as for a
383
+ truncated file: the footer adds "J of K shown rows · Filters use shown rows"
384
+ and the summaries recompute over the resident rows. When local filters exist,
385
+ the footer offers "Apply filters to whole dataset" (`.sift-apply-query`). It
386
+ commits the committed filters plus the local ones as one `TableQuery` against
387
+ the source and replaces the resident rows with the new head; it does not layer
388
+ on the previous result or union rows. While the query runs the button reads
389
+ "Querying…" and the old rows stay visible. If the merged query is outside
390
+ `capabilities`, the button is disabled with the reason as its title.
391
+ - `onQueryStateChange` reports each generation (`pending`, `committedQuery`,
392
+ `resultCount`); `onChange` keeps reporting the local state, and
393
+ `TableEngineState` gains `committedQuery` and `resultCount`.
394
+ `engineStateToExplorerState` carries them as `committedQuery` and `counts`;
395
+ its existing fields keep meaning resident rows.
396
+
397
+ What stays local in this version: sort (the head is sorted in the browser, so
398
+ it orders the shown rows, not the whole result), casts (they apply to the
399
+ resident rows and are not part of `committedQuery`), and summaries (computed
400
+ over the shown rows; the interface has no summary request yet). `limits` bounds
401
+ decoded memory and columns per result; `headRows` is the row limit, 10,000 by
402
+ default. Only the head window exists: there is no paging or windowing past it.
403
+
404
+ For the imperative API the same ownership is available directly:
405
+ `engine.replaceData(data, { committedQuery })` swaps in rows that satisfy a
406
+ query without re-applying local filters or sort, `engine.setResultCount(count)`
407
+ sets the footer count, and `onCommittedQueryChange` in `createTable` options
408
+ receives the reduced query when a committed pill is cleared.
409
+
410
+ ## Windowed data (experimental)
411
+
412
+ `createTable` also accepts a `WindowedTableData`: a source whose cells,
413
+ permutation, filtered summaries, and image bytes arrive asynchronously. The
414
+ data side owns the filter-and-sort permutation and answers `setQuery` with the
415
+ filtered count and summaries, `requestWindow(viewStart, viewEnd, generation)`
416
+ with the data rows and cells for a span of view positions, and `requestImage`
417
+ with one cell's bytes; it pushes `rows-appended` and `streaming-done` through
418
+ `subscribe`. The engine shows skeleton placeholders (`.sift-cell-skeleton`,
419
+ rows carry `.sift-row-placeholder`, the grid `aria-busy="true"`) for view rows
420
+ whose cells have not landed, drops any answer tagged with a stale generation,
421
+ and reports `pending: true` in `TableEngineState` while a query is in flight.
422
+ Geometry stays dense: `rowCount` is a resident row count. `head` holds cells
423
+ for data rows `[0, headRows)` in identity order, whatever the current query; the
424
+ engine paints from it only when no sort or filter is active, and ignores a head
425
+ whose `dataRows` are not `0, 1, 2, …`. A data side that rewrites cell strings
426
+ (cast, undo) refreshes `head` the same way.
427
+
428
+ What is painted stays painted. A `rows-appended` push under the identity
429
+ permutation extends the map in place and calls `onChange` with the new
430
+ `totalCount` and `filteredCount`; under a sort or filter it re-queries, and
431
+ the answer is held until the window covering the visible span arrives, then
432
+ both commit in one frame under the scroll anchor (`onChange` fires when the
433
+ answer arrives). The scroll anchor is the row under the header's bottom edge:
434
+ landing windows, appends, and commits keep that row at the same screen
435
+ position; `scrollTop` changes only by row height measured above it. A user's
436
+ own sort or filter shows placeholders at once (the viewport returns to the
437
+ top), and a refused query puts the delivered cells back. A
438
+ `{ type: "error", fatal: true }` event
439
+ sets `TableEngineState.dataFailed`: the rows on screen stay, and sort, filter,
440
+ and cast controls are disabled with the title "Data source stopped; table tools
441
+ are unavailable".
442
+
443
+ `createResidentWindowedData(tableData)` adapts a loaded `TableData` to this
444
+ shape on the main thread. It exists to exercise the mode and is the contract a
445
+ worker-backed data side will implement over a `MessagePort`; it is not faster
446
+ than resident mode. `byte-source` and Parquet `url` sources take
447
+ `windowed: true` to mount through it, with the same milestones as today. Off
448
+ by default. Lists render as text in this mode, and casts and summaries go
449
+ through the data side.
450
+
451
+ ## Running the data side in a worker (experimental)
452
+
453
+ `worker: true` on a `byte-source` moves decoding and the table store into a
454
+ dedicated worker. The worker is the resident windowed adapter behind a
455
+ `MessagePort`: it owns the WASM module, the Parquet file, the permutation, and
456
+ the summaries, and answers the engine's windows over messages. The main thread
457
+ keeps the engine, the host's `ByteSource`, and the rows it has already been
458
+ shown.
459
+
460
+ The library ships the worker bundle as `@nteract/sift/worker` but not the
461
+ spawn. Register one before mounting:
462
+
463
+ ```ts
464
+ import { setWorkerFactory } from '@nteract/sift'
465
+
466
+ // Same-origin module worker (Desktop, a browser page you control).
467
+ setWorkerFactory(
468
+ () => new Worker(new URL('@nteract/sift/worker', import.meta.url), { type: 'module' }),
469
+ )
470
+
471
+ <SiftTable source={{ kind: 'byte-source', source, overflow: 'truncate', worker: true }} />
472
+ ```
473
+
474
+ With no factory the source fails at validation with a visible error. Sift does
475
+ not fall back to the main thread: a file that needs the worker's containment
476
+ must not run without it. A factory that throws (a CSP or sandbox
477
+ `SecurityError`, for example) takes the same path as any other load error:
478
+ Sift's error state shows "Worker factory threw: …" and `onLoadError` fires; it
479
+ does not reach your error boundary. The notebook output frame has an opaque origin and
480
+ cannot load a same-origin module worker; a blob-URL worker build for it is a
481
+ follow-up and that host is not covered yet.
482
+
483
+ Reads are relayed: the worker asks for a byte range, the main thread calls
484
+ `source.read` and transfers the copy. `setWasmUrl` state is forwarded in the
485
+ `open` message, so configure it before mounting as today.
486
+
487
+ What the worker contains: WASM traps, decoder hangs, and retained memory. A trap
488
+ (including the allocation cap below) ends that worker: Sift terminates it,
489
+ keeps the rows already loaded, shows "Stopped loading: row group N exceeded the
490
+ worker memory limit … · Showing loaded rows" in the footer, and fires
491
+ `onLoadError`. Rows not yet shown stay as placeholders; a respawn with
492
+ row-group quarantine is a follow-up. What it does not contain: the renderer
493
+ process's memory (a dedicated worker lives in it) and the host's own I/O. The
494
+ allocation cap is what bounds memory: the worker calls `set_allocation_cap`
495
+ with `allocationCap` (default `2.5 × maxDecodedBytes`) before loading, so a
496
+ decompression bomb traps in Rust before the process pays for it. Between row
497
+ groups the host also compares live heap bytes against the cap and stops a load
498
+ that has already exceeded it.
499
+
500
+ After a trap the table keeps its rows, but sort, filter, and cast are disabled
501
+ (`getState().dataFailed` is `true`). A host calling `createWorkerWindowedData`
502
+ directly gets a `WorkerDataError` with `code: "terminated"` from every pending
503
+ and later request; other codes are `factory`, `open`, `aborted`, and `closed`
504
+ (after `dispose()`). A failed or aborted open terminates its worker.
505
+
506
+ A failure that is not a trap, such as a `source.read` rejection for a later row
507
+ group, stops loading without ending the worker. The footer shows "Stopped
508
+ loading: <message> · Showing loaded rows", `onLoadError` fires, and the rows
509
+ already loaded keep their sort, filter, and cast tools.
510
+
511
+ With `overflow: 'truncate'`, a load the budget stops early shows the same
512
+ "Showing first N of M rows · Table tools use shown rows" notice as the
513
+ main-thread loader, and the `streaming-complete` milestone carries `rowCount`,
514
+ `totalRowCount`, and `truncated`.
515
+
516
+ `autoLoad` works the same as on the main thread: the worker pauses at the soft
517
+ budget and waits, the footer offers `Load more`, and `trigger: 'scroll'`
518
+ resumes near the end. The milestones match (`streaming-complete` with
519
+ `resumable: true`, then `streaming-resumed`).
520
+
521
+ A worker that never starts (its script is refused, as with a `blob:` URL under
522
+ a CSP without `worker-src blob:`) fails the open with "Sift worker failed to
523
+ start; the host's Content-Security-Policy may block this worker URL (…)"
524
+ (`WorkerDataError` code `open`). A worker that dies after answering reports
525
+ "Sift worker terminated: …".
526
+
527
+ Not in this version: `url` and manifest sources, a request-latency watchdog,
528
+ and respawn.
174
529
 
175
530
  ## Lifecycle and host controls
176
531
 
@@ -192,11 +547,31 @@ responsibilities. Parquet can use a view-only mode.
192
547
 
193
548
  For already hosted data, `<SiftTable url="/data.arrow" />` and
194
549
  `<SiftTable url="/data.parquet" />` remain available. The URL/manifest paths have
195
- different loading behavior and do not inherit `loadTableFile` admission limits:
196
- a Parquet URL streams row groups through the same loader as `byte-source`, with
197
- the limits set to their maxima. A schema-only Parquet URL mounts an empty table.
198
- A sampled Arrow manifest must include its `summary.sampled` and included/total
199
- row metadata; its sorting, filtering, and summaries cover only the supplied rows.
550
+ their own loading behavior. A Parquet URL is downloaded whole, then streams row
551
+ groups through the same loader as `byte-source`, with the same budget and
552
+ preview footer. Its defaults are `URL_PARQUET_LIMITS` (1,000,000 rows, 512 MiB
553
+ of decoded memory, 256 columns) and `overflow: 'truncate'`, so a file past the
554
+ budget shows its leading rows and the footer reads "Showing first N of M rows".
555
+ The `streaming-complete` milestone carries `rowCount`, `totalRowCount`, and
556
+ `truncated`, and the mounted data carries `coverage`, as for `byte-source`.
557
+ Override either on the source:
558
+
559
+ ```tsx
560
+ <SiftTable
561
+ source={{
562
+ kind: 'url',
563
+ url: '/data.parquet',
564
+ limits: { maxRows: 5_000_000, maxDecodedBytes: 2 * 1024 * 1024 * 1024 },
565
+ overflow: 'error', // fail instead of truncating
566
+ }}
567
+ />
568
+ ```
569
+
570
+ The download itself is not bounded; the budget bounds decoded memory. Arrow IPC
571
+ over a URL is read whole and ignores `limits` and `overflow`. A schema-only
572
+ Parquet URL mounts an empty table. A sampled Arrow manifest must include its
573
+ `summary.sampled` and included/total row metadata; its sorting, filtering, and
574
+ summaries cover only the supplied rows.
200
575
 
201
576
  ## Interaction and appearance
202
577
 
@@ -234,11 +609,17 @@ that are portaled to `<body>`:
234
609
  --sift-color-accent: #2563eb; /* histogram bars, selection, focus ring */
235
610
  --sift-color-true: #22c55e; /* boolean true */
236
611
  --sift-color-false: #ef4444; /* boolean false */
237
- --sift-font-family: Inter, system-ui, sans-serif;
612
+ --sift-font-family: Inter, "Helvetica Neue", "Segoe UI", Arial, sans-serif;
238
613
  --sift-radius-base: 0.5rem;
239
614
  }
240
615
  ```
241
616
 
617
+ Row heights are computed from text measured in the cell's font, read from the
618
+ DOM when the table mounts. Set `--sift-font-family` before creating the table.
619
+ Lead with a named family such as `Inter` or `"Helvetica Neue"`: on macOS,
620
+ `system-ui` measures a little differently in canvas than it paints, so an
621
+ occasional row can come out a line short or tall.
622
+
242
623
  Every other color (row stripes, scrollbars, badges, skeletons) is derived from
243
624
  these with `color-mix()`, so they follow light and dark automatically.
244
625
 
@@ -256,3 +637,29 @@ Sift. The JavaScript in `lib/` is first-party; every npm dependency is
256
637
  installed by the host under its own license. Building and
257
638
  verifying the package from source is covered in `packages/sift/AGENTS.md` in
258
639
  the nteract repository.
640
+
641
+ ## Verified in an Electron harness
642
+
643
+ The worker path (see "Running the data side in a worker") has been run in
644
+ Electron 45 under Kilo Desktop's renderer policy: a custom standard scheme
645
+ registered `secure` with fetch and CORS privileges, `script-src 'self'
646
+ 'wasm-unsafe-eval'` with no `worker-src`, and Parquet read over a ranged IPC
647
+ `ByteSource`. What an embedder in that setup needs to know:
648
+
649
+ - `new Worker(new URL('@nteract/sift/worker', import.meta.url), { type: 'module' })`
650
+ resolves under Vite and runs with no `worker-src` and no CSP warnings. Set
651
+ `worker: { format: 'es' }`; the worker bundle code-splits.
652
+ - The worker needs `'wasm-unsafe-eval'`. Its script response carries your
653
+ CSP, and it instantiates the WASM itself.
654
+ - Serve `.wasm` as `application/wasm` and `.js` as `text/javascript`.
655
+ - A `blob:` worker blocked by CSP does not throw from `new Worker`; Chromium
656
+ fires `error` instead, and Sift reports "Sift worker failed to start; the
657
+ host's Content-Security-Policy may block this worker URL (…)" rather than
658
+ "Worker factory threw: …".
659
+ - `autoLoad` pauses and resumes the worker. On both loaders, resumed appends
660
+ keep the row under the header in place; `scrollTop` may move to do it.
661
+ - `allocationCap` defaults to 2.5 × `maxDecodedBytes`. Decoding a row group
662
+ briefly needs about twice the retained heap, so a 2× cap can trap on the last
663
+ row group that fits a truncating budget instead of ending in a clean preview.
664
+
665
+ The harness is `apps/sift-electron-harness` in the nteract repository.
@@ -1,8 +1,9 @@
1
1
  # Third-party notices
2
2
 
3
3
  This package ships WebAssembly compiled from the Rust crates below, under
4
- the licenses listed. The JavaScript in `lib/` is first-party; its npm
5
- dependencies are installed separately and carry their own licenses.
4
+ the licenses listed. The JavaScript in `lib/` is first-party apart from the
5
+ vendored assets listed; its npm dependencies are installed separately and
6
+ carry their own licenses.
6
7
 
7
8
  Generated by `cargo about` from `about.toml` for sift-wasm.
8
9
 
@@ -31,7 +32,7 @@ Generated by `cargo about` from `about.toml` for sift-wasm.
31
32
  | bitflags | 2.11.1 | MIT OR Apache-2.0 | https://github.com/bitflags/bitflags |
32
33
  | bumpalo | 3.20.2 | MIT OR Apache-2.0 | https://github.com/fitzgen/bumpalo |
33
34
  | byteorder | 1.5.0 | Unlicense OR MIT | https://github.com/BurntSushi/byteorder |
34
- | bytes | 1.11.1 | MIT | https://github.com/tokio-rs/bytes |
35
+ | bytes | 1.12.1 | MIT | https://github.com/tokio-rs/bytes |
35
36
  | cfg-if | 1.0.4 | MIT OR Apache-2.0 | https://github.com/rust-lang/cfg-if |
36
37
  | chrono | 0.4.44 | MIT OR Apache-2.0 | https://github.com/chronotope/chrono |
37
38
  | chrono-tz | 0.10.4 | MIT OR Apache-2.0 | https://github.com/chronotope/chrono-tz |
@@ -196,6 +197,30 @@ its NOTICE file:
196
197
  (c) Copyright [2014-2015] Hewlett-Packard Development Company, L.P
197
198
  ```
198
199
 
200
+ ## Vendored assets
201
+
202
+ ### Lucide icon path data (chevron-left, chevron-right, x) (ISC)
203
+
204
+ Source: https://lucide.dev. Used in `src/components/icons.tsx`.
205
+
206
+ ```
207
+ ISC License
208
+
209
+ Copyright (c) for portions of Lucide are held by Cole Bemis 2013-2022 as part of Feather (MIT). All other copyright (c) for Lucide are held by Lucide Contributors 2022.
210
+
211
+ Permission to use, copy, modify, and/or distribute this software for any
212
+ purpose with or without fee is hereby granted, provided that the above
213
+ copyright notice and this permission notice appear in all copies.
214
+
215
+ THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
216
+ WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
217
+ MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
218
+ ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
219
+ WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
220
+ ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
221
+ OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
222
+ ```
223
+
199
224
  ## License texts
200
225
 
201
226
  ### Apache License 2.0 (Apache-2.0)
@@ -2709,7 +2734,7 @@ DEALINGS IN THE SOFTWARE.
2709
2734
 
2710
2735
  ### MIT License (MIT)
2711
2736
 
2712
- Used by: bytes@1.11.1
2737
+ Used by: bytes@1.12.1
2713
2738
 
2714
2739
  ```
2715
2740
  Copyright (c) 2018 Carl Lerche
package/lib/handoff.css CHANGED
@@ -11,7 +11,18 @@
11
11
  var(--sift-ink, var(--foreground, #111827))
12
12
  );
13
13
  cursor: pointer;
14
- font: 500 11px/1.2 var(--sift-font, Inter, ui-sans-serif, system-ui, sans-serif);
14
+ font: 500 11px/1.2
15
+ var(
16
+ --sift-font,
17
+ Inter,
18
+ "Helvetica Neue",
19
+ "Segoe UI",
20
+ Helvetica,
21
+ Arial,
22
+ ui-sans-serif,
23
+ system-ui,
24
+ sans-serif
25
+ );
15
26
  padding: 3px 7px;
16
27
  white-space: nowrap;
17
28
  transition: