@gmod/tabix 3.8.0 → 3.8.2

package/README.md

@@ -40,6 +40,23 @@ const custom = new TabixIndexedFile({
 })
 ```
 
+Over HTTP, swapping in
+[`@gmod/range-cache-filehandle`](https://github.com/GMOD/range-cache-filehandle)
+is usually worth it: a query reads the index and then a scattered set of BGZF
+blocks, and a byte-range cache coalesces those into one request per contiguous
+run and serves an overlapping query from memory.
+
+```typescript
+import { RemoteFileWithRangeCache } from '@gmod/range-cache-filehandle'
+
+const cached = new TabixIndexedFile({
+  filehandle: new RemoteFileWithRangeCache('https://example.com/file.vcf.gz'),
+  tbiFilehandle: new RemoteFileWithRangeCache(
+    'https://example.com/file.vcf.gz.tbi',
+  ),
+})
+```
+
 ### getLines
 
 Fetches lines overlapping a region. `start`/`end` are 0-based half-open
@@ -73,19 +90,19 @@ await file.getLines('chr1', 200, 300, {
 })
 ```
 
-`onProgress` ticks once per compressed block, including instant ticks for cache
-hits, and `totalBytes` is known up front from the index — enough for a
-determinate progress bar.
+`onProgress` ticks once per chunk — the run of BGZF blocks the index resolves a
+query to — including instant ticks for chunks already cached, and the index
+supplies `totalBytes` up front, which is enough for a determinate progress bar.
 
 Notes:
 
-- Meta/comment lines are skipped
+- The scan skips meta/comment lines
 - Line strings have no trailing whitespace
 - Pass `undefined` for `end` to read to the end of the contig
 - A `refName` that is not in the index yields no lines and no error, so a
   `chr1`/`1` naming mismatch looks like an empty region. Check against
-  [`getReferenceSequenceNames`](#getreferencesequencenamesopts-promisestring) if
-  a query comes back unexpectedly empty
+  [`getReferenceSequenceNames`](docs/api.md#getreferencesequencenamesopts-promisestring)
+  if a query comes back unexpectedly empty
 - `start > end` throws a `TypeError`; `start === end` returns without reading
 
 ### Without NPM (CDN)
@@ -98,93 +115,77 @@ See [example/index.html](example/index.html) for a working demo. It fetches the
 VCF over HTTP, so serve the directory (e.g. `npx serve example`) rather than
 opening the file directly.
 
-## API
-
-### `new TabixIndexedFile(args)`
-
-| Arg                       | Type                 | Description                                                                                                                                                                                                                                   |
-| ------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `path`                    | `string?`            | Local file path                                                                                                                                                                                                                               |
-| `url`                     | `string?`            | Remote URL                                                                                                                                                                                                                                    |
-| `filehandle`              | `GenericFilehandle?` | Custom filehandle (from [generic-filehandle2](https://github.com/GMOD/generic-filehandle2))                                                                                                                                                   |
-| `tbiPath`                 | `string?`            | TBI index path (defaults to `path + '.tbi'`)                                                                                                                                                                                                  |
-| `tbiUrl`                  | `string?`            | TBI index URL                                                                                                                                                                                                                                 |
-| `tbiFilehandle`           | `GenericFilehandle?` | TBI index filehandle                                                                                                                                                                                                                          |
-| `csiPath`                 | `string?`            | CSI index path                                                                                                                                                                                                                                |
-| `csiUrl`                  | `string?`            | CSI index URL                                                                                                                                                                                                                                 |
-| `csiFilehandle`           | `GenericFilehandle?` | CSI index filehandle                                                                                                                                                                                                                          |
-| `chunkCacheSize`          | `number?`            | Chunk LRU cache budget, in _decompressed_ bytes (default 1 GiB). A retention bound, not a bound on peak memory. Size it to hold several queries: below one query's working set the hit rate drops to zero while the memory is retained anyway |
-| `chunkCacheIdleTimeoutMs` | `number?`            | Drop a cached chunk once nothing has read it for this long (default 3 minutes, `0` disables). The only thing that lowers the cache while nothing is happening, and what makes the budget above a peak rather than a resting level             |
-
-### `getLines(refName, start, end, opts)`
-
-Calls the line callback for each line overlapping `[start, end)`. `start`
-defaults to `0` and `end` to the end of the contig when `undefined`. `opts` is
-either the callback itself or an object:
-
-| Option         | Type                                                     | Description                             |
-| -------------- | -------------------------------------------------------- | --------------------------------------- |
-| `lineCallback` | `(line, fileOffset, start, end) => void`                 | Required                                |
-| `signal`       | `AbortSignal?`                                           | Aborts the in-flight reads              |
-| `onProgress`   | `(bytesDownloaded: number, totalBytes?: number) => void` | Called as compressed blocks are fetched |
-
-### `getHeader(opts?): Promise<string>`
-
-Returns all comment/meta lines before the first data line as a string, matching
-what `tabix -H` prints. A header row that is not commented is therefore not
-included, even when the index counted it as a line to skip — see
-`getSkippedLines`.
+## How a query flows
 
-### `getHeaderBuffer(opts?): Promise<Uint8Array>`
+`getLines` turns a region into BGZF chunks through the index and decompresses
+each one in wasm — index reads included, since `.tbi` and `.csi` are bgzipped
+too. The rest is ordinary JS: it matches lines as bytes and decodes only the
+ones you asked for. [docs/dataflow.md](docs/dataflow.md) has the diagram and
+walks it through.
 
-Returns the header as raw bytes.
+The file then holds on to those decompressed chunks, so overlapping and adjacent
+queries reuse them instead of inflating again — up to 1GB per file, dropped
+after three idle minutes. A consumer holding one file per track should bound
+them together with a shared `chunkCacheBudget` rather than shrinking each file's
+own ceiling: [docs/caching.md](docs/caching.md).
 
-### `getSkippedLines(opts?): Promise<string[]>`
+## Decompressing on a worker pool
 
-Returns the leading lines the index says to skip (`tabix -S N`), or `[]` when it
-records none. This is where a file whose header row is not commented keeps it,
-which PLINK `.ld`, bedGraph and BED deflines routinely are.
+BGZF blocks inflate independently, so that decompression can spread across
+threads.
 
-Separate from `getHeader` because htslib treats the two differently: a line is
-not data when the index's skip count covers it **or** it starts with the meta
-character, but `tabix -H` prints only the latter.
-
-### `getHeaderLines(opts?): Promise<string[]>`
-
-Returns the file's header lines however the file keeps them: the commented block
-when there is one, and otherwise the rows the index counted. Empty lines are
-dropped.
-
-This is usually the one you want. Reading `getHeader` alone cannot tell a file
-that has no header from one whose header is not commented — both come back as
-the empty string — so callers fall back to an assumed column layout and quietly
-mis-name columns. Both forms are parsed from a single read of the file's leading
-blocks, which is also shared with `getHeader` and `getSkippedLines`.
-
-### `getReferenceSequenceNames(opts?): Promise<string[]>`
-
-Returns reference sequence names in index order.
-
-### `lineCount(refName, opts?): Promise<number>`
-
-Returns the number of data lines on the given reference, or `-1` if the
-reference is not in the index.
-
-### `bytesForRegions(regions, opts?): Promise<number>`
-
-Estimates the compressed byte size of index chunks covering the given regions.
-Useful for deciding whether a request is too large before calling `getLines`.
+```typescript
+import { getSharedWorkerPool } from '@gmod/bgzf-filehandle'
 
-## Contributing
+const file = new TabixIndexedFile({
+  url: 'https://example.com/yourfile.vcf.gz',
+  // the pending promise is fine — it is awaited at the point of use
+  bgzfWorkerPool: getSharedWorkerPool(),
+})
+```
 
-See [CONTRIBUTING.md](CONTRIBUTING.md) for development and release steps.
+Safe to pass unconditionally: `getSharedWorkerPool()` returns `undefined` under
+node, or anywhere the host forbids Workers, which keeps the in-process path. No
+cross-origin isolation needed. tabix-js never creates a pool on its own — the
+thread budget belongs to the consumer.
+
+**Worth about 1.4x here, against the 1.95x a BAM reader reports.** Measured in
+jbrowse-components on `test/data/1kg.chr1.subset.vcf.gz` — 213MB of 1000
+Genomes, headless Chrome, real HTTP, four workers, arms interleaved, both
+returning the same record count: **1.34-1.46x** across five window sizes and a
+twelve-step pan.
+
+The decompression itself moves **1.83x**. What holds the end-to-end figure below
+that is a **28% floor of per-line byte scanning and string decoding**, which no
+worker count reaches — and that floor is at its worst on multi-sample VCF, whose
+records carry a genotype field per sample and run to ~60KB a line. A format with
+narrower lines sits closer to BAM. If you want more than ~1.5x on a multi-sample
+VCF, the scan is what is left to attack, not the decompression.
+
+Worker counts, lifecycle and benchmarks:
+[bgzf-filehandle's worker pool docs](https://github.com/GMOD/bgzf-filehandle/blob/main/docs/worker-pool.md);
+the end-to-end numbers above, and how to confirm a pool is really engaging in
+production rather than quietly falling back, are in jbrowse-components'
+[BGZF_WORKER_POOL.md](https://github.com/GMOD/jbrowse-components/blob/main/agent-docs/reference/BGZF_WORKER_POOL.md).
+
+## Docs
+
+- [docs/api.md](docs/api.md) — every constructor arg and method
+- [docs/dataflow.md](docs/dataflow.md) — a query end to end, diagrammed
+- [docs/optimizations.md](docs/optimizations.md) — why each step of that path
+  looks the way it does, and what measured it
+- [docs/caching.md](docs/caching.md) — sizing the decompressed-chunk cache, and
+  bounding many files together
+- [agent-docs/adr/](agent-docs/adr/) — the measurements behind those decisions
+- [agent-docs/TODO.md](agent-docs/TODO.md) — what is worth doing next, and what
+  has to be measured before it
+- [CONTRIBUTING.md](CONTRIBUTING.md) — development and release steps
 
 ## Academic Use
 
-This package was written with funding from the [NHGRI](http://genome.gov) as
-part of the [JBrowse](http://jbrowse.org) project. If you use it in an academic
-project that you publish, please cite the most recent JBrowse paper, which will
-be linked from [jbrowse.org](http://jbrowse.org).
+Written with [NHGRI](http://genome.gov) funding as part of
+[JBrowse](http://jbrowse.org). If you use this in a publication, please cite the
+most recent JBrowse paper at [jbrowse.org](http://jbrowse.org).
 
 ## License