@nteract/sift 0.6.0 → 0.7.0-nightly.42413490472
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +11 -2
- package/docs/embedding-guide.md +417 -10
- package/lib/THIRD_PARTY_NOTICES.md +29 -4
- package/lib/handoff.css +12 -1
- package/lib/index.js +2847 -2052
- package/lib/protocol-B_bRY_76.js +885 -0
- package/lib/style.css +1 -1
- package/lib/types/packages/sift/src/auto-width.d.ts +9 -3
- package/lib/types/packages/sift/src/components/icons.d.ts +16 -0
- package/lib/types/packages/sift/src/file-data.d.ts +34 -5
- package/lib/types/packages/sift/src/filter-schema.d.ts +11 -0
- package/lib/types/packages/sift/src/index.d.ts +11 -4
- package/lib/types/packages/sift/src/parquet-file.d.ts +52 -5
- package/lib/types/packages/sift/src/permutation.d.ts +24 -0
- package/lib/types/packages/sift/src/predicate.d.ts +56 -3
- package/lib/types/packages/sift/src/query.d.ts +100 -0
- package/lib/types/packages/sift/src/react.d.ts +108 -6
- package/lib/types/packages/sift/src/table.d.ts +168 -2
- package/lib/types/packages/sift/src/wasm-summaries.d.ts +19 -1
- package/lib/types/packages/sift/src/windowed-resident.d.ts +20 -0
- package/lib/types/packages/sift/src/windowed.d.ts +116 -0
- package/lib/types/packages/sift/src/worker/client.d.ts +87 -0
- package/lib/types/packages/sift/src/worker/entry.d.ts +11 -0
- package/lib/types/packages/sift/src/worker/factory.d.ts +11 -0
- package/lib/types/packages/sift/src/worker/host.d.ts +31 -0
- package/lib/types/packages/sift/src/worker/protocol.d.ts +191 -0
- package/lib/wasm/sift_wasm.d.ts +67 -3
- package/lib/wasm/sift_wasm.js +109 -3
- package/lib/wasm/sift_wasm_bg.wasm +0 -0
- package/lib/worker.js +277 -0
- package/package.json +9 -6
package/README.md
CHANGED
|
@@ -53,8 +53,17 @@ const load = useCallback(() => loadTableFile(bytes), [bytes]) // Parquet or Arro
|
|
|
53
53
|
|
|
54
54
|
Files load whole or fail; defaults are 16 MiB, 100,000 rows, and 256 columns.
|
|
55
55
|
Parquet can also be read by byte range through a `ByteSource` (footer first,
|
|
56
|
-
then one row group at a time), which drops the 16 MiB input cap
|
|
57
|
-
|
|
56
|
+
then one row group at a time), which drops the 16 MiB input cap. With
|
|
57
|
+
`overflow: 'truncate'` it shows the leading rows that fit the row and memory
|
|
58
|
+
budget instead of failing, and `autoLoad` pauses at a soft budget with a
|
|
59
|
+
"Load more" button; see the embedding guide. Every summary labels what
|
|
60
|
+
its numbers cover (`scope`): histograms are over loaded rows, while range and
|
|
61
|
+
null count come from Parquet footer statistics for the whole file when present.
|
|
62
|
+
Data behind a database or scan service plugs in through a `query-session`
|
|
63
|
+
source: the host implements `TableQuerySession`, Sift shows a bounded head and
|
|
64
|
+
applies brushes to the whole dataset on request; see the embedding guide.
|
|
65
|
+
`worker: true` on a byte-range Parquet source decodes in the `@nteract/sift/worker`
|
|
66
|
+
data worker under an allocation cap, once the host registers `setWorkerFactory`.
|
|
58
67
|
CSV and TSV go through [`@nteract/sift-csv`](https://www.npmjs.com/package/@nteract/sift-csv), a separate WASM
|
|
59
68
|
module that returns Arrow IPC bytes for `loadTableFile`, so apps that only open
|
|
60
69
|
Parquet or Arrow do not carry the CSV parser.
|
package/docs/embedding-guide.md
CHANGED
|
@@ -106,7 +106,8 @@ large. If you load untrusted files, do it in a Worker you can terminate.
|
|
|
106
106
|
|
|
107
107
|
This API loads a complete file. Virtual scrolling bounds rendered DOM, not the
|
|
108
108
|
dataset: sorting, filtering, and summaries cover every accepted row. There is no
|
|
109
|
-
paging or sampled preview.
|
|
109
|
+
paging or sampled preview. Byte-range Parquet can load a bounded leading
|
|
110
|
+
prefix instead; see [Bounded previews](#bounded-previews).
|
|
110
111
|
|
|
111
112
|
Lists, structs, and maps display as text. Known image structures have dedicated
|
|
112
113
|
rendering.
|
|
@@ -163,14 +164,368 @@ memory while it decodes. Files with column chunks past 4 GiB are rejected at
|
|
|
163
164
|
open; the WASM reader cannot address them.
|
|
164
165
|
|
|
165
166
|
If a row group fails after the table has mounted (a read error, the decoded
|
|
166
|
-
limit), the rows already loaded stay on screen
|
|
167
|
-
|
|
167
|
+
limit), the rows already loaded stay on screen and usable. Sift does not show
|
|
168
|
+
its blocking error box. Instead its footer reads "Stopped loading: <message> ·
|
|
169
|
+
Showing loaded rows" and the status indicator turns to a warning mark whose
|
|
170
|
+
tooltip names the failure. `onLoadError` still fires, so the host can log it.
|
|
171
|
+
A failure before mount shows the error box as usual. To tell the two apart,
|
|
172
|
+
check whether the `engine-mounted` milestone arrived for this load. The next
|
|
173
|
+
source to load clears the notice. A Parquet `url` streams through the same
|
|
174
|
+
loader and behaves the same way.
|
|
175
|
+
|
|
176
|
+
### Bounded previews
|
|
177
|
+
|
|
178
|
+
To open files of any size and show the leading rows, set `overflow: 'truncate'`.
|
|
179
|
+
Rows and decoded bytes become a budget instead of an admission check:
|
|
180
|
+
|
|
181
|
+
```tsx
|
|
182
|
+
<SiftTable
|
|
183
|
+
source={{
|
|
184
|
+
kind: 'byte-source',
|
|
185
|
+
source,
|
|
186
|
+
limits: { maxRows: 1_000_000, maxDecodedBytes: 512 * 1024 * 1024 },
|
|
187
|
+
overflow: 'truncate',
|
|
188
|
+
}}
|
|
189
|
+
onLoadMilestone={(m) => {
|
|
190
|
+
if (m.phase === 'streaming-complete' && m.truncated) {
|
|
191
|
+
console.log(`Showing first ${m.rowCount} of ${m.totalRowCount} rows`)
|
|
192
|
+
}
|
|
193
|
+
}}
|
|
194
|
+
/>
|
|
195
|
+
|
|
196
|
+
// loadTableFile accepts the same option for a ByteSource.
|
|
197
|
+
const data = await loadTableFile(source, { maxRows: 100_000, overflow: 'truncate' })
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
- The footer is not rejected for its total row count. `maxColumns` and the
|
|
201
|
+
footer-size check still reject, and so does a file where one row's
|
|
202
|
+
fixed-width columns alone exceed `maxDecodedBytes`.
|
|
203
|
+
- Row groups decode in 256-row batches. Loading stops at exactly `maxRows`,
|
|
204
|
+
at the rows whose fixed-width values fit what is left of the budget, or
|
|
205
|
+
before the first batch that would take retained memory past
|
|
206
|
+
`maxDecodedBytes`. Once the budget is spent, later row groups are never
|
|
207
|
+
read.
|
|
208
|
+
- A row group whose compressed column chunks exceed `maxDecodedBytes` fails
|
|
209
|
+
before it is fetched, and gaps between chunks are fetched only while the
|
|
210
|
+
request stays within that size. If the budget cannot hold even the first
|
|
211
|
+
batch, the load fails: there is nothing to preview.
|
|
212
|
+
- String and binary view columns are copied out of the decoded pages, so
|
|
213
|
+
retained memory is what the budget counts.
|
|
214
|
+
- The `streaming-complete` milestone carries `rowCount` (loaded rows),
|
|
215
|
+
`totalRowCount` (footer rows), and `truncated`. When truncated, Sift's footer
|
|
216
|
+
shows "Showing first N of M rows" and the status tooltip reads "Preview
|
|
217
|
+
loaded". Sorting, filtering, and summaries cover only the loaded rows.
|
|
218
|
+
|
|
219
|
+
The default for `byte-source` and `loadTableFile`, `overflow: 'error'`, keeps
|
|
220
|
+
the behavior described above; a Parquet `url` source defaults to `'truncate'`
|
|
221
|
+
(see below). Only streaming Parquet honors `overflow`; in-memory buffers and
|
|
222
|
+
Arrow IPC still fail on overflow.
|
|
223
|
+
|
|
224
|
+
The budget is a loading guardrail, like the other limits, not isolation from
|
|
225
|
+
hostile files. Page decompression and one decoder batch allocate before a
|
|
226
|
+
batch is measured. Load untrusted files in a Worker you can terminate.
|
|
227
|
+
|
|
228
|
+
### Load more on demand
|
|
229
|
+
|
|
230
|
+
A truncating load can also stop early on purpose. `autoLoad` sets a soft budget
|
|
231
|
+
below `limits`: Sift loads row groups until the committed rows or decoded bytes
|
|
232
|
+
reach it, then pauses with the rows loaded so far and lets the user ask for
|
|
233
|
+
more.
|
|
234
|
+
|
|
235
|
+
```tsx
|
|
236
|
+
<SiftTable
|
|
237
|
+
source={{
|
|
238
|
+
kind: 'byte-source',
|
|
239
|
+
source,
|
|
240
|
+
limits: { maxRows: 5_000_000, maxDecodedBytes: 1024 * 1024 * 1024 },
|
|
241
|
+
overflow: 'truncate',
|
|
242
|
+
autoLoad: { rows: 100_000, decodedBytes: 64 * 1024 * 1024, trigger: 'scroll' },
|
|
243
|
+
}}
|
|
244
|
+
/>
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
- `autoLoad` needs `overflow: 'truncate'`; with `'error'` the source is
|
|
248
|
+
rejected before anything is read. It applies to `byte-source` (including
|
|
249
|
+
`worker: true`, where the worker pauses and waits) and to Parquet behind a
|
|
250
|
+
`url` source. `rows` and `decodedBytes` are each optional; the pause
|
|
251
|
+
comes at whichever is reached first.
|
|
252
|
+
- At a pause the table is at rest: the status indicator settles, casts are
|
|
253
|
+
allowed again, and the footer reads "Showing first N of M rows" with a
|
|
254
|
+
`Load more` button (`.sift-load-more`). Each press extends the soft budget by
|
|
255
|
+
the same `rows` and `decodedBytes` and loads until it is reached again, the
|
|
256
|
+
hard budget is spent, or the file ends. While a resume is in flight the button
|
|
257
|
+
reads "Loading…" and is disabled. With `trigger: 'scroll'`, scrolling within
|
|
258
|
+
the overscan of the last loaded row also resumes; the button stays.
|
|
259
|
+
- Milestones: a pause emits `streaming-complete` with `truncated: true` and
|
|
260
|
+
`resumable: true`, so hosts that clear their loading state on
|
|
261
|
+
`streaming-complete` keep doing so. A resume emits `streaming-resumed` with
|
|
262
|
+
`rowCount`, then the usual `chunk-appended` milestones, then
|
|
263
|
+
`streaming-complete` again: `resumable: true` for another pause, or
|
|
264
|
+
`resumable: false` once the file finished or the hard budget cut it, with
|
|
265
|
+
`truncated` meaning what it means for any truncating load.
|
|
266
|
+
- `TableData.coverage` reports `truncated: true` and `resumable: true` while
|
|
267
|
+
paused, and `resumable: false` once the load finished. Summaries, sort, and
|
|
268
|
+
filters cover the loaded rows throughout.
|
|
269
|
+
- Unmounting or changing the source while paused closes the footer and aborts a
|
|
270
|
+
resume in flight, as for any streaming load.
|
|
168
271
|
|
|
169
272
|
Only Parquet can be read this way. A `ByteSource` with `format: 'arrow-ipc'` is
|
|
170
273
|
read whole and goes through the bytes path with its `maxBytes` check.
|
|
171
274
|
|
|
172
275
|
`onLoadError` fires for `loader`, `url`, `byte-source`, and manifest sources
|
|
173
|
-
when a load fails, with the same `Error` Sift
|
|
276
|
+
when a load fails, with the same `Error` Sift shows. For streaming Parquet it
|
|
277
|
+
may arrive after rows mounted; see above.
|
|
278
|
+
|
|
279
|
+
### What the numbers cover
|
|
280
|
+
|
|
281
|
+
A table loaded from a file carries three pieces of provenance:
|
|
282
|
+
|
|
283
|
+
- `TableData.coverage` says how much of the dataset is resident: `loadedRows`,
|
|
284
|
+
`totalRows` (with its own scope, precision, and source), the file's
|
|
285
|
+
`rowGroups`, and `truncated`. `loadTableFile` sets it on every path, so a
|
|
286
|
+
caller of `loadTableFile(source, { overflow: 'truncate' })` reads
|
|
287
|
+
`data.coverage.truncated` to learn the result was cut. Absent means every row
|
|
288
|
+
is resident.
|
|
289
|
+
- `TableData.datasetStats` holds one `ColumnFileStats` per column, read from
|
|
290
|
+
the Parquet footer at open and never changed: a whole-file null count when
|
|
291
|
+
every row group recorded one, min/max when every row group had usable bounds,
|
|
292
|
+
and flags for inexact (truncated string) bounds and infinite values. Columns
|
|
293
|
+
without usable footer statistics are `null`.
|
|
294
|
+
- Each non-null `ColumnSummary` has `scope`, which labels every metric
|
|
295
|
+
separately. `scope.distribution` covers the histogram bins, category bars,
|
|
296
|
+
and true/false counts; it is always computed over loaded rows and says
|
|
297
|
+
`dataset` only when every row is resident. `scope.range` and
|
|
298
|
+
`scope.nullCount` come from footer metadata when the footer has them, and
|
|
299
|
+
then describe the whole file; otherwise they describe loaded rows.
|
|
300
|
+
`scope.distinctCount` is always over loaded rows. A cast column ignores its
|
|
301
|
+
footer entry, since that entry describes the stored type.
|
|
302
|
+
|
|
303
|
+
The header renders these as text. The histogram and its brush domain stay over
|
|
304
|
+
loaded rows. When the file has more rows than are loaded, a numeric or timestamp
|
|
305
|
+
header adds a line with the whole-file bounds ("0 – 100 · whole file", with "≈"
|
|
306
|
+
when the bounds are inexact), the null percentage names its scope ("20% null ·
|
|
307
|
+
whole file" or "20% null · loaded rows"), the category popover keeps its "Search
|
|
308
|
+
N values…" and "All" controls and adds a note "Values from loaded rows · N of M"
|
|
309
|
+
that both controls reference through `aria-describedby`, and a boolean column keeps its loaded-rows bar
|
|
310
|
+
and shows whole-file nulls on a separate line. Captions carry `data-scope` and
|
|
311
|
+
`data-precision` attributes. `TableEngineState.loadedCount` reports resident
|
|
312
|
+
rows alongside `totalCount`, which keeps its current meaning.
|
|
313
|
+
|
|
314
|
+
## Query sessions
|
|
315
|
+
|
|
316
|
+
When the data lives behind a database, a scan service, or anything that can
|
|
317
|
+
answer a filter without shipping the whole table, the host implements
|
|
318
|
+
`TableQuerySession` and hands it to a `query-session` source. Sift sends a
|
|
319
|
+
`TableQuery` and receives one bounded Arrow IPC stream; it loads that stream
|
|
320
|
+
through the same resident path as a file and records that the rows already
|
|
321
|
+
satisfy the query. It does not filter them again, and the footer and pills
|
|
322
|
+
describe the committed query rather than a local one.
|
|
323
|
+
|
|
324
|
+
```tsx
|
|
325
|
+
import type { TableQuerySession } from '@nteract/sift'
|
|
326
|
+
|
|
327
|
+
const session: TableQuerySession = {
|
|
328
|
+
capabilities: {
|
|
329
|
+
filters: { price: ['between'], category: ['in', 'not_in', 'is_null'] },
|
|
330
|
+
sort: [],
|
|
331
|
+
casts: [],
|
|
332
|
+
counts: ['exact'],
|
|
333
|
+
compound: ['or'], // for Sift's category exclusion, or(is_null, not_in)
|
|
334
|
+
},
|
|
335
|
+
async query(q, ctx) {
|
|
336
|
+
// Compile q.filters to your dialect with bound parameters, run it with
|
|
337
|
+
// ctx.signal and ctx.budget, and return one uncompressed IPC stream.
|
|
338
|
+
return { ipc, returnedRows, matchedCount: { kind: 'exact', value }, continuation: { kind: 'more' } }
|
|
339
|
+
},
|
|
340
|
+
async count(q, mode, ctx) { /* COUNT(*) or an estimate */ },
|
|
341
|
+
async close() { /* release the connection */ },
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
<SiftTable source={{ kind: 'query-session', session, headRows: 10_000 }} />
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
The contract (`src/query.ts`):
|
|
348
|
+
|
|
349
|
+
- `TableQuery` is `casts`, `filters` (top-level AND of `FilterPredicate`), and
|
|
350
|
+
`sort`. The vocabulary is the same name-addressed predicate tree that
|
|
351
|
+
`engineStateToExplorerState` produces; SQL and pandas text are exports, not the
|
|
352
|
+
wire format. The host owns the SQL dialect, the trusted base relation, the
|
|
353
|
+
connection, credentials, and placement. Sift never receives SQL text.
|
|
354
|
+
- `capabilities` says what the provider can run: leaf ops per column, sortable
|
|
355
|
+
columns, casts, which count kinds it offers, and which compound predicates it
|
|
356
|
+
accepts. `supportsQuery(caps, q)` checks a query against it and returns the
|
|
357
|
+
reason when something is missing. A provider still validates the whole query
|
|
358
|
+
and refuses what it cannot run; it never narrows a result silently.
|
|
359
|
+
- `query` takes a `window` of `{ mode: 'head', limit }` and returns a
|
|
360
|
+
`QueryPage`: the IPC stream (schema-only when empty), `returnedRows`,
|
|
361
|
+
`matchedCount` (a `Count`: exact, estimate, lower bound, or unknown, always for
|
|
362
|
+
the query, never the base relation), and a `continuation` that says whether
|
|
363
|
+
more rows matched. `RequestContext` carries a `requestId`, an `AbortSignal`,
|
|
364
|
+
and a byte budget. Sift aborts a request when a newer query supersedes it or
|
|
365
|
+
the source unmounts; a late answer is dropped and its store freed.
|
|
366
|
+
- `count` is separate so first paint never waits on `COUNT(*)`. Sift asks for a
|
|
367
|
+
`'cheap'` count when the page's `matchedCount` is `unknown` and the provider
|
|
368
|
+
lists a count kind.
|
|
369
|
+
|
|
370
|
+
What the user sees:
|
|
371
|
+
|
|
372
|
+
- The footer's primary line is the whole-dataset count with the shown rows:
|
|
373
|
+
"N whole-dataset matches · K rows shown", "≈N whole-dataset matches ·
|
|
374
|
+
Estimated · K rows shown", "≥N …", or "Counting whole-dataset matches… · K
|
|
375
|
+
rows shown" while the count is pending. A query with no filters says "rows in
|
|
376
|
+
whole dataset" instead of "matches".
|
|
377
|
+
- Committed filters render as pills with `.sift-filter-pill--committed` and the
|
|
378
|
+
title "Applied to the whole dataset". Clearing one does not touch the resident
|
|
379
|
+
rows; it runs the reduced query against the source. Removing a filter must
|
|
380
|
+
query the source, not the already-filtered result, or the rows it hid could
|
|
381
|
+
never come back.
|
|
382
|
+
- Brushes and category clicks stay local, over the shown rows, exactly as for a
|
|
383
|
+
truncated file: the footer adds "J of K shown rows · Filters use shown rows"
|
|
384
|
+
and the summaries recompute over the resident rows. When local filters exist,
|
|
385
|
+
the footer offers "Apply filters to whole dataset" (`.sift-apply-query`). It
|
|
386
|
+
commits the committed filters plus the local ones as one `TableQuery` against
|
|
387
|
+
the source and replaces the resident rows with the new head; it does not layer
|
|
388
|
+
on the previous result or union rows. While the query runs the button reads
|
|
389
|
+
"Querying…" and the old rows stay visible. If the merged query is outside
|
|
390
|
+
`capabilities`, the button is disabled with the reason as its title.
|
|
391
|
+
- `onQueryStateChange` reports each generation (`pending`, `committedQuery`,
|
|
392
|
+
`resultCount`); `onChange` keeps reporting the local state, and
|
|
393
|
+
`TableEngineState` gains `committedQuery` and `resultCount`.
|
|
394
|
+
`engineStateToExplorerState` carries them as `committedQuery` and `counts`;
|
|
395
|
+
its existing fields keep meaning resident rows.
|
|
396
|
+
|
|
397
|
+
What stays local in this version: sort (the head is sorted in the browser, so
|
|
398
|
+
it orders the shown rows, not the whole result), casts (they apply to the
|
|
399
|
+
resident rows and are not part of `committedQuery`), and summaries (computed
|
|
400
|
+
over the shown rows; the interface has no summary request yet). `limits` bounds
|
|
401
|
+
decoded memory and columns per result; `headRows` is the row limit, 10,000 by
|
|
402
|
+
default. Only the head window exists: there is no paging or windowing past it.
|
|
403
|
+
|
|
404
|
+
For the imperative API the same ownership is available directly:
|
|
405
|
+
`engine.replaceData(data, { committedQuery })` swaps in rows that satisfy a
|
|
406
|
+
query without re-applying local filters or sort, `engine.setResultCount(count)`
|
|
407
|
+
sets the footer count, and `onCommittedQueryChange` in `createTable` options
|
|
408
|
+
receives the reduced query when a committed pill is cleared.
|
|
409
|
+
|
|
410
|
+
## Windowed data (experimental)
|
|
411
|
+
|
|
412
|
+
`createTable` also accepts a `WindowedTableData`: a source whose cells,
|
|
413
|
+
permutation, filtered summaries, and image bytes arrive asynchronously. The
|
|
414
|
+
data side owns the filter-and-sort permutation and answers `setQuery` with the
|
|
415
|
+
filtered count and summaries, `requestWindow(viewStart, viewEnd, generation)`
|
|
416
|
+
with the data rows and cells for a span of view positions, and `requestImage`
|
|
417
|
+
with one cell's bytes; it pushes `rows-appended` and `streaming-done` through
|
|
418
|
+
`subscribe`. The engine shows skeleton placeholders (`.sift-cell-skeleton`,
|
|
419
|
+
rows carry `.sift-row-placeholder`, the grid `aria-busy="true"`) for view rows
|
|
420
|
+
whose cells have not landed, drops any answer tagged with a stale generation,
|
|
421
|
+
and reports `pending: true` in `TableEngineState` while a query is in flight.
|
|
422
|
+
Geometry stays dense: `rowCount` is a resident row count. `head` holds cells
|
|
423
|
+
for data rows `[0, headRows)` in identity order, whatever the current query; the
|
|
424
|
+
engine paints from it only when no sort or filter is active, and ignores a head
|
|
425
|
+
whose `dataRows` are not `0, 1, 2, …`. A data side that rewrites cell strings
|
|
426
|
+
(cast, undo) refreshes `head` the same way.
|
|
427
|
+
|
|
428
|
+
What is painted stays painted. A `rows-appended` push under the identity
|
|
429
|
+
permutation extends the map in place and calls `onChange` with the new
|
|
430
|
+
`totalCount` and `filteredCount`; under a sort or filter it re-queries, and
|
|
431
|
+
the answer is held until the window covering the visible span arrives, then
|
|
432
|
+
both commit in one frame under the scroll anchor (`onChange` fires when the
|
|
433
|
+
answer arrives). The scroll anchor is the row under the header's bottom edge:
|
|
434
|
+
landing windows, appends, and commits keep that row at the same screen
|
|
435
|
+
position; `scrollTop` changes only by row height measured above it. A user's
|
|
436
|
+
own sort or filter shows placeholders at once (the viewport returns to the
|
|
437
|
+
top), and a refused query puts the delivered cells back. A
|
|
438
|
+
`{ type: "error", fatal: true }` event
|
|
439
|
+
sets `TableEngineState.dataFailed`: the rows on screen stay, and sort, filter,
|
|
440
|
+
and cast controls are disabled with the title "Data source stopped; table tools
|
|
441
|
+
are unavailable".
|
|
442
|
+
|
|
443
|
+
`createResidentWindowedData(tableData)` adapts a loaded `TableData` to this
|
|
444
|
+
shape on the main thread. It exists to exercise the mode and is the contract a
|
|
445
|
+
worker-backed data side will implement over a `MessagePort`; it is not faster
|
|
446
|
+
than resident mode. `byte-source` and Parquet `url` sources take
|
|
447
|
+
`windowed: true` to mount through it, with the same milestones as today. Off
|
|
448
|
+
by default. Lists render as text in this mode, and casts and summaries go
|
|
449
|
+
through the data side.
|
|
450
|
+
|
|
451
|
+
## Running the data side in a worker (experimental)
|
|
452
|
+
|
|
453
|
+
`worker: true` on a `byte-source` moves decoding and the table store into a
|
|
454
|
+
dedicated worker. The worker is the resident windowed adapter behind a
|
|
455
|
+
`MessagePort`: it owns the WASM module, the Parquet file, the permutation, and
|
|
456
|
+
the summaries, and answers the engine's windows over messages. The main thread
|
|
457
|
+
keeps the engine, the host's `ByteSource`, and the rows it has already been
|
|
458
|
+
shown.
|
|
459
|
+
|
|
460
|
+
The library ships the worker bundle as `@nteract/sift/worker` but not the
|
|
461
|
+
spawn. Register one before mounting:
|
|
462
|
+
|
|
463
|
+
```ts
|
|
464
|
+
import { setWorkerFactory } from '@nteract/sift'
|
|
465
|
+
|
|
466
|
+
// Same-origin module worker (Desktop, a browser page you control).
|
|
467
|
+
setWorkerFactory(
|
|
468
|
+
() => new Worker(new URL('@nteract/sift/worker', import.meta.url), { type: 'module' }),
|
|
469
|
+
)
|
|
470
|
+
|
|
471
|
+
<SiftTable source={{ kind: 'byte-source', source, overflow: 'truncate', worker: true }} />
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
With no factory the source fails at validation with a visible error. Sift does
|
|
475
|
+
not fall back to the main thread: a file that needs the worker's containment
|
|
476
|
+
must not run without it. A factory that throws (a CSP or sandbox
|
|
477
|
+
`SecurityError`, for example) takes the same path as any other load error:
|
|
478
|
+
Sift's error state shows "Worker factory threw: …" and `onLoadError` fires; it
|
|
479
|
+
does not reach your error boundary. The notebook output frame has an opaque origin and
|
|
480
|
+
cannot load a same-origin module worker; a blob-URL worker build for it is a
|
|
481
|
+
follow-up and that host is not covered yet.
|
|
482
|
+
|
|
483
|
+
Reads are relayed: the worker asks for a byte range, the main thread calls
|
|
484
|
+
`source.read` and transfers the copy. `setWasmUrl` state is forwarded in the
|
|
485
|
+
`open` message, so configure it before mounting as today.
|
|
486
|
+
|
|
487
|
+
What the worker contains: WASM traps, decoder hangs, and retained memory. A trap
|
|
488
|
+
(including the allocation cap below) ends that worker: Sift terminates it,
|
|
489
|
+
keeps the rows already loaded, shows "Stopped loading: row group N exceeded the
|
|
490
|
+
worker memory limit … · Showing loaded rows" in the footer, and fires
|
|
491
|
+
`onLoadError`. Rows not yet shown stay as placeholders; a respawn with
|
|
492
|
+
row-group quarantine is a follow-up. What it does not contain: the renderer
|
|
493
|
+
process's memory (a dedicated worker lives in it) and the host's own I/O. The
|
|
494
|
+
allocation cap is what bounds memory: the worker calls `set_allocation_cap`
|
|
495
|
+
with `allocationCap` (default `2.5 × maxDecodedBytes`) before loading, so a
|
|
496
|
+
decompression bomb traps in Rust before the process pays for it. Between row
|
|
497
|
+
groups the host also compares live heap bytes against the cap and stops a load
|
|
498
|
+
that has already exceeded it.
|
|
499
|
+
|
|
500
|
+
After a trap the table keeps its rows, but sort, filter, and cast are disabled
|
|
501
|
+
(`getState().dataFailed` is `true`). A host calling `createWorkerWindowedData`
|
|
502
|
+
directly gets a `WorkerDataError` with `code: "terminated"` from every pending
|
|
503
|
+
and later request; other codes are `factory`, `open`, `aborted`, and `closed`
|
|
504
|
+
(after `dispose()`). A failed or aborted open terminates its worker.
|
|
505
|
+
|
|
506
|
+
A failure that is not a trap, such as a `source.read` rejection for a later row
|
|
507
|
+
group, stops loading without ending the worker. The footer shows "Stopped
|
|
508
|
+
loading: <message> · Showing loaded rows", `onLoadError` fires, and the rows
|
|
509
|
+
already loaded keep their sort, filter, and cast tools.
|
|
510
|
+
|
|
511
|
+
With `overflow: 'truncate'`, a load the budget stops early shows the same
|
|
512
|
+
"Showing first N of M rows · Table tools use shown rows" notice as the
|
|
513
|
+
main-thread loader, and the `streaming-complete` milestone carries `rowCount`,
|
|
514
|
+
`totalRowCount`, and `truncated`.
|
|
515
|
+
|
|
516
|
+
`autoLoad` works the same as on the main thread: the worker pauses at the soft
|
|
517
|
+
budget and waits, the footer offers `Load more`, and `trigger: 'scroll'`
|
|
518
|
+
resumes near the end. The milestones match (`streaming-complete` with
|
|
519
|
+
`resumable: true`, then `streaming-resumed`).
|
|
520
|
+
|
|
521
|
+
A worker that never starts (its script is refused, as with a `blob:` URL under
|
|
522
|
+
a CSP without `worker-src blob:`) fails the open with "Sift worker failed to
|
|
523
|
+
start; the host's Content-Security-Policy may block this worker URL (…)"
|
|
524
|
+
(`WorkerDataError` code `open`). A worker that dies after answering reports
|
|
525
|
+
"Sift worker terminated: …".
|
|
526
|
+
|
|
527
|
+
Not in this version: `url` and manifest sources, a request-latency watchdog,
|
|
528
|
+
and respawn.
|
|
174
529
|
|
|
175
530
|
## Lifecycle and host controls
|
|
176
531
|
|
|
@@ -192,11 +547,31 @@ responsibilities. Parquet can use a view-only mode.
|
|
|
192
547
|
|
|
193
548
|
For already hosted data, `<SiftTable url="/data.arrow" />` and
|
|
194
549
|
`<SiftTable url="/data.parquet" />` remain available. The URL/manifest paths have
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
550
|
+
their own loading behavior. A Parquet URL is downloaded whole, then streams row
|
|
551
|
+
groups through the same loader as `byte-source`, with the same budget and
|
|
552
|
+
preview footer. Its defaults are `URL_PARQUET_LIMITS` (1,000,000 rows, 512 MiB
|
|
553
|
+
of decoded memory, 256 columns) and `overflow: 'truncate'`, so a file past the
|
|
554
|
+
budget shows its leading rows and the footer reads "Showing first N of M rows".
|
|
555
|
+
The `streaming-complete` milestone carries `rowCount`, `totalRowCount`, and
|
|
556
|
+
`truncated`, and the mounted data carries `coverage`, as for `byte-source`.
|
|
557
|
+
Override either on the source:
|
|
558
|
+
|
|
559
|
+
```tsx
|
|
560
|
+
<SiftTable
|
|
561
|
+
source={{
|
|
562
|
+
kind: 'url',
|
|
563
|
+
url: '/data.parquet',
|
|
564
|
+
limits: { maxRows: 5_000_000, maxDecodedBytes: 2 * 1024 * 1024 * 1024 },
|
|
565
|
+
overflow: 'error', // fail instead of truncating
|
|
566
|
+
}}
|
|
567
|
+
/>
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
The download itself is not bounded; the budget bounds decoded memory. Arrow IPC
|
|
571
|
+
over a URL is read whole and ignores `limits` and `overflow`. A schema-only
|
|
572
|
+
Parquet URL mounts an empty table. A sampled Arrow manifest must include its
|
|
573
|
+
`summary.sampled` and included/total row metadata; its sorting, filtering, and
|
|
574
|
+
summaries cover only the supplied rows.
|
|
200
575
|
|
|
201
576
|
## Interaction and appearance
|
|
202
577
|
|
|
@@ -234,11 +609,17 @@ that are portaled to `<body>`:
|
|
|
234
609
|
--sift-color-accent: #2563eb; /* histogram bars, selection, focus ring */
|
|
235
610
|
--sift-color-true: #22c55e; /* boolean true */
|
|
236
611
|
--sift-color-false: #ef4444; /* boolean false */
|
|
237
|
-
--sift-font-family: Inter,
|
|
612
|
+
--sift-font-family: Inter, "Helvetica Neue", "Segoe UI", Arial, sans-serif;
|
|
238
613
|
--sift-radius-base: 0.5rem;
|
|
239
614
|
}
|
|
240
615
|
```
|
|
241
616
|
|
|
617
|
+
Row heights are computed from text measured in the cell's font, read from the
|
|
618
|
+
DOM when the table mounts. Set `--sift-font-family` before creating the table.
|
|
619
|
+
Lead with a named family such as `Inter` or `"Helvetica Neue"`: on macOS,
|
|
620
|
+
`system-ui` measures a little differently in canvas than it paints, so an
|
|
621
|
+
occasional row can come out a line short or tall.
|
|
622
|
+
|
|
242
623
|
Every other color (row stripes, scrollbars, badges, skeletons) is derived from
|
|
243
624
|
these with `color-mix()`, so they follow light and dark automatically.
|
|
244
625
|
|
|
@@ -256,3 +637,29 @@ Sift. The JavaScript in `lib/` is first-party; every npm dependency is
|
|
|
256
637
|
installed by the host under its own license. Building and
|
|
257
638
|
verifying the package from source is covered in `packages/sift/AGENTS.md` in
|
|
258
639
|
the nteract repository.
|
|
640
|
+
|
|
641
|
+
## Verified in an Electron harness
|
|
642
|
+
|
|
643
|
+
The worker path (see "Running the data side in a worker") has been run in
|
|
644
|
+
Electron 45 under Kilo Desktop's renderer policy: a custom standard scheme
|
|
645
|
+
registered `secure` with fetch and CORS privileges, `script-src 'self'
|
|
646
|
+
'wasm-unsafe-eval'` with no `worker-src`, and Parquet read over a ranged IPC
|
|
647
|
+
`ByteSource`. What an embedder in that setup needs to know:
|
|
648
|
+
|
|
649
|
+
- `new Worker(new URL('@nteract/sift/worker', import.meta.url), { type: 'module' })`
|
|
650
|
+
resolves under Vite and runs with no `worker-src` and no CSP warnings. Set
|
|
651
|
+
`worker: { format: 'es' }`; the worker bundle code-splits.
|
|
652
|
+
- The worker needs `'wasm-unsafe-eval'`. Its script response carries your
|
|
653
|
+
CSP, and it instantiates the WASM itself.
|
|
654
|
+
- Serve `.wasm` as `application/wasm` and `.js` as `text/javascript`.
|
|
655
|
+
- A `blob:` worker blocked by CSP does not throw from `new Worker`; Chromium
|
|
656
|
+
fires `error` instead, and Sift reports "Sift worker failed to start; the
|
|
657
|
+
host's Content-Security-Policy may block this worker URL (…)" rather than
|
|
658
|
+
"Worker factory threw: …".
|
|
659
|
+
- `autoLoad` pauses and resumes the worker. On both loaders, resumed appends
|
|
660
|
+
keep the row under the header in place; `scrollTop` may move to do it.
|
|
661
|
+
- `allocationCap` defaults to 2.5 × `maxDecodedBytes`. Decoding a row group
|
|
662
|
+
briefly needs about twice the retained heap, so a 2× cap can trap on the last
|
|
663
|
+
row group that fits a truncating budget instead of ending in a clean preview.
|
|
664
|
+
|
|
665
|
+
The harness is `apps/sift-electron-harness` in the nteract repository.
|
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
# Third-party notices
|
|
2
2
|
|
|
3
3
|
This package ships WebAssembly compiled from the Rust crates below, under
|
|
4
|
-
the licenses listed. The JavaScript in `lib/` is first-party
|
|
5
|
-
dependencies are installed separately and
|
|
4
|
+
the licenses listed. The JavaScript in `lib/` is first-party apart from the
|
|
5
|
+
vendored assets listed; its npm dependencies are installed separately and
|
|
6
|
+
carry their own licenses.
|
|
6
7
|
|
|
7
8
|
Generated by `cargo about` from `about.toml` for sift-wasm.
|
|
8
9
|
|
|
@@ -31,7 +32,7 @@ Generated by `cargo about` from `about.toml` for sift-wasm.
|
|
|
31
32
|
| bitflags | 2.11.1 | MIT OR Apache-2.0 | https://github.com/bitflags/bitflags |
|
|
32
33
|
| bumpalo | 3.20.2 | MIT OR Apache-2.0 | https://github.com/fitzgen/bumpalo |
|
|
33
34
|
| byteorder | 1.5.0 | Unlicense OR MIT | https://github.com/BurntSushi/byteorder |
|
|
34
|
-
| bytes | 1.
|
|
35
|
+
| bytes | 1.12.1 | MIT | https://github.com/tokio-rs/bytes |
|
|
35
36
|
| cfg-if | 1.0.4 | MIT OR Apache-2.0 | https://github.com/rust-lang/cfg-if |
|
|
36
37
|
| chrono | 0.4.44 | MIT OR Apache-2.0 | https://github.com/chronotope/chrono |
|
|
37
38
|
| chrono-tz | 0.10.4 | MIT OR Apache-2.0 | https://github.com/chronotope/chrono-tz |
|
|
@@ -196,6 +197,30 @@ its NOTICE file:
|
|
|
196
197
|
(c) Copyright [2014-2015] Hewlett-Packard Development Company, L.P
|
|
197
198
|
```
|
|
198
199
|
|
|
200
|
+
## Vendored assets
|
|
201
|
+
|
|
202
|
+
### Lucide icon path data (chevron-left, chevron-right, x) (ISC)
|
|
203
|
+
|
|
204
|
+
Source: https://lucide.dev. Used in `src/components/icons.tsx`.
|
|
205
|
+
|
|
206
|
+
```
|
|
207
|
+
ISC License
|
|
208
|
+
|
|
209
|
+
Copyright (c) for portions of Lucide are held by Cole Bemis 2013-2022 as part of Feather (MIT). All other copyright (c) for Lucide are held by Lucide Contributors 2022.
|
|
210
|
+
|
|
211
|
+
Permission to use, copy, modify, and/or distribute this software for any
|
|
212
|
+
purpose with or without fee is hereby granted, provided that the above
|
|
213
|
+
copyright notice and this permission notice appear in all copies.
|
|
214
|
+
|
|
215
|
+
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
|
|
216
|
+
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
|
217
|
+
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
|
|
218
|
+
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
|
219
|
+
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
|
|
220
|
+
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
|
|
221
|
+
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
|
222
|
+
```
|
|
223
|
+
|
|
199
224
|
## License texts
|
|
200
225
|
|
|
201
226
|
### Apache License 2.0 (Apache-2.0)
|
|
@@ -2709,7 +2734,7 @@ DEALINGS IN THE SOFTWARE.
|
|
|
2709
2734
|
|
|
2710
2735
|
### MIT License (MIT)
|
|
2711
2736
|
|
|
2712
|
-
Used by: bytes@1.
|
|
2737
|
+
Used by: bytes@1.12.1
|
|
2713
2738
|
|
|
2714
2739
|
```
|
|
2715
2740
|
Copyright (c) 2018 Carl Lerche
|
package/lib/handoff.css
CHANGED
|
@@ -11,7 +11,18 @@
|
|
|
11
11
|
var(--sift-ink, var(--foreground, #111827))
|
|
12
12
|
);
|
|
13
13
|
cursor: pointer;
|
|
14
|
-
font: 500 11px/1.2
|
|
14
|
+
font: 500 11px/1.2
|
|
15
|
+
var(
|
|
16
|
+
--sift-font,
|
|
17
|
+
Inter,
|
|
18
|
+
"Helvetica Neue",
|
|
19
|
+
"Segoe UI",
|
|
20
|
+
Helvetica,
|
|
21
|
+
Arial,
|
|
22
|
+
ui-sans-serif,
|
|
23
|
+
system-ui,
|
|
24
|
+
sans-serif
|
|
25
|
+
);
|
|
15
26
|
padding: 3px 7px;
|
|
16
27
|
white-space: nowrap;
|
|
17
28
|
transition:
|