@jbrowse/core 5.0.0-beta.1 → 5.0.0-beta.11

package/esm/util/bgzfWorkerPool.d.ts

@@ -1,78 +1,24 @@
 import type { BgzfWorkerPool } from '@gmod/bgzf-filehandle';
 /**
- * The BGZF inflate pool the indexed-format libraries decompress their chunks
- * on, instead of decompressing them on the thread that asked.
- *
- * BGZF decompression is 70-90% of a cold query (`@gmod/bam` ADR 0003) and BGZF
- * blocks are independently inflatable, so spreading them across workers is the
- * largest remaining lever in the read path. Measured 1.95x end to end on the
- * BAM side: a 22-view pan / zoom out / pan back over 1000x long-read data, real
- * HTTP, headless Chrome, 4 workers, with both arms returning the same 38,246
- * records.
- *
- * ## One pool per JS context, like the cache budgets
- *
- * Same scope and same reason as `decompressedBytesBudget` — see
- * {@link ./cacheBudgets.ts} — so one per RPC worker plus one on the main
- * thread. That is the scope with spare cores: a track's queries are sticky to a
- * single RPC worker, so without a pool every chunk of every query inflates on
- * that one thread while the rest of the machine idles. `getSharedWorkerPool()`
- * memoizes per context, so every adapter in a worker shares its four workers
- * rather than each opening its own.
- *
- * ## Call it, don't hoist it
- *
- * This is a function and not an exported constant on purpose: it spawns workers
- * the first time it is called, and a module-level call would spawn them on
- * import, in every context that pulls in anything from `util`, whether or not a
- * bgzip-backed track is ever opened. Call it where a file is constructed.
- *
- * ## The dynamic import is about bundle weight, and only that
- *
- * `@gmod/bgzf-filehandle` inlines its worker bundle as a base64 string, and
- * `getSharedWorkerPool` is what reaches it, so a **static** import pins that
- * blob into the initial bundle of every entry point that reaches this helper.
- * Measured with esbuild, splitting and minifying, on the import alone:
- *
- * ```
- * static    entry 54.7kb  (23.4kb gzipped) — the blob is in the initial bundle
- * dynamic   entry 114b    (141b  gzipped) + 123.9kb in a split chunk
- * ```
- *
- * So ~23kb gzipped off every bundle that can reach a bgzip adapter, deferred to
- * a chunk that is only fetched once such a track is actually opened. Core's
- * `util/unzip.ts` and `fetchAndMaybeUnzip.ts` dodge it the same way. It costs
- * nothing to do, because the libraries want a promise anyway — their callers'
- * `configure()` is synchronous and the pool is only available asynchronously.
- *
- * **It is not about test speed**, though it has been described that way. That
- * claim was checked here and did not reproduce: three jbrowse-web variant
- * suites ran 13.887s static against 13.856s dynamic, warm cache, and a 29s
- * reading that started this off turned out to be a cold jest transform cache in
- * a fresh worktree. Don't reintroduce a static import on the grounds that the
- * jest argument was wrong — the bundle argument above is the real one, and it
- * stands on its own.
- *
- * The `catch` is deliberate, but for a plain reason rather than a specific
- * failure: a pool is an optimization, `@gmod/bam` and `@gmod/tabix` both await
- * this, and anything going wrong here has to degrade to inflating in process
- * rather than break every read. A documented CJS-interop throw under jest was
- * the original motivation; that no longer reproduces (core's own suite, the
- * BamAdapter suites and jbrowse-web's Alignments suite all pass with the catch
- * removed), so keep it for the invariant, not the anecdote.
- *
- * ## Safe to pass unconditionally
- *
- * Returns a promise, and the libraries await it at the point of use rather than
- * at construction — which is what lets a synchronous `configure()` hand the
- * pending pool straight over. It resolves to `undefined` wherever Workers
- * cannot be created (node, jest), and `@gmod/bam` and `@gmod/tabix` then take
- * their in-process path, so no caller needs an environment check and the test
- * suites are unaffected.
- *
- * Needs no cross-origin isolation: each worker's range is transferred to it as
- * an ArrayBuffer rather than shared, so this works on an ordinary page. That
- * was the point of dropping SharedArrayBuffer in `@gmod/bgzf-filehandle` 6.4.0
- * — measured, the shared transport was at parity at best.
+ * The BGZF inflate pool the indexed-format libraries decompress large chunks on,
+ * one per JS context like `decompressedBytesBudget`: one per RPC worker plus one
+ * on the main thread. BGZF decompression is 70-90% of a cold query (`@gmod/bam`
+ * ADR 0003), and its blocks inflate independently.
+ *
+ * Holding it costs nothing. `@gmod/bgzf-filehandle` 6.7 starts the workers only
+ * when a chunk of at least `POOL_MIN_DECOMPRESSED_BYTES` (0.4MB inflated)
+ * arrives, inflates in process while they boot, and inflates in process for good
+ * if one of them fails, so no read waits on a worker. What it is worth per
+ * format, and why GFF3 and GTF go without it: reference/BGZF_WORKER_POOL.md.
+ *
+ * A function over a dynamic import because the library inlines its worker bundle
+ * as a base64 string, and a static import pins it into every entry point that
+ * can reach this helper: measured with esbuild, 23.4kb gzipped against 141b. Test
+ * speed is not a reason either way; three jbrowse-web variant suites ran 13.887s
+ * static against 13.856s dynamic.
+ *
+ * Resolves to `undefined` wherever Workers cannot be created (node, jest), which
+ * keeps `@gmod/bam` and `@gmod/tabix` on their in-process path, and the `catch`
+ * keeps anything else going wrong here on it too.
  */
 export declare function sharedBgzfWorkerPool(): Promise<BgzfWorkerPool | undefined>;

package/esm/util/bgzfWorkerPool.js

@@ -1,78 +1,24 @@
 /**
- * The BGZF inflate pool the indexed-format libraries decompress their chunks
- * on, instead of decompressing them on the thread that asked.
- *
- * BGZF decompression is 70-90% of a cold query (`@gmod/bam` ADR 0003) and BGZF
- * blocks are independently inflatable, so spreading them across workers is the
- * largest remaining lever in the read path. Measured 1.95x end to end on the
- * BAM side: a 22-view pan / zoom out / pan back over 1000x long-read data, real
- * HTTP, headless Chrome, 4 workers, with both arms returning the same 38,246
- * records.
- *
- * ## One pool per JS context, like the cache budgets
- *
- * Same scope and same reason as `decompressedBytesBudget` — see
- * {@link ./cacheBudgets.ts} — so one per RPC worker plus one on the main
- * thread. That is the scope with spare cores: a track's queries are sticky to a
- * single RPC worker, so without a pool every chunk of every query inflates on
- * that one thread while the rest of the machine idles. `getSharedWorkerPool()`
- * memoizes per context, so every adapter in a worker shares its four workers
- * rather than each opening its own.
- *
- * ## Call it, don't hoist it
- *
- * This is a function and not an exported constant on purpose: it spawns workers
- * the first time it is called, and a module-level call would spawn them on
- * import, in every context that pulls in anything from `util`, whether or not a
- * bgzip-backed track is ever opened. Call it where a file is constructed.
- *
- * ## The dynamic import is about bundle weight, and only that
- *
- * `@gmod/bgzf-filehandle` inlines its worker bundle as a base64 string, and
- * `getSharedWorkerPool` is what reaches it, so a **static** import pins that
- * blob into the initial bundle of every entry point that reaches this helper.
- * Measured with esbuild, splitting and minifying, on the import alone:
- *
- * ```
- * static    entry 54.7kb  (23.4kb gzipped) — the blob is in the initial bundle
- * dynamic   entry 114b    (141b  gzipped) + 123.9kb in a split chunk
- * ```
- *
- * So ~23kb gzipped off every bundle that can reach a bgzip adapter, deferred to
- * a chunk that is only fetched once such a track is actually opened. Core's
- * `util/unzip.ts` and `fetchAndMaybeUnzip.ts` dodge it the same way. It costs
- * nothing to do, because the libraries want a promise anyway — their callers'
- * `configure()` is synchronous and the pool is only available asynchronously.
- *
- * **It is not about test speed**, though it has been described that way. That
- * claim was checked here and did not reproduce: three jbrowse-web variant
- * suites ran 13.887s static against 13.856s dynamic, warm cache, and a 29s
- * reading that started this off turned out to be a cold jest transform cache in
- * a fresh worktree. Don't reintroduce a static import on the grounds that the
- * jest argument was wrong — the bundle argument above is the real one, and it
- * stands on its own.
- *
- * The `catch` is deliberate, but for a plain reason rather than a specific
- * failure: a pool is an optimization, `@gmod/bam` and `@gmod/tabix` both await
- * this, and anything going wrong here has to degrade to inflating in process
- * rather than break every read. A documented CJS-interop throw under jest was
- * the original motivation; that no longer reproduces (core's own suite, the
- * BamAdapter suites and jbrowse-web's Alignments suite all pass with the catch
- * removed), so keep it for the invariant, not the anecdote.
- *
- * ## Safe to pass unconditionally
- *
- * Returns a promise, and the libraries await it at the point of use rather than
- * at construction — which is what lets a synchronous `configure()` hand the
- * pending pool straight over. It resolves to `undefined` wherever Workers
- * cannot be created (node, jest), and `@gmod/bam` and `@gmod/tabix` then take
- * their in-process path, so no caller needs an environment check and the test
- * suites are unaffected.
- *
- * Needs no cross-origin isolation: each worker's range is transferred to it as
- * an ArrayBuffer rather than shared, so this works on an ordinary page. That
- * was the point of dropping SharedArrayBuffer in `@gmod/bgzf-filehandle` 6.4.0
- * — measured, the shared transport was at parity at best.
+ * The BGZF inflate pool the indexed-format libraries decompress large chunks on,
+ * one per JS context like `decompressedBytesBudget`: one per RPC worker plus one
+ * on the main thread. BGZF decompression is 70-90% of a cold query (`@gmod/bam`
+ * ADR 0003), and its blocks inflate independently.
+ *
+ * Holding it costs nothing. `@gmod/bgzf-filehandle` 6.7 starts the workers only
+ * when a chunk of at least `POOL_MIN_DECOMPRESSED_BYTES` (0.4MB inflated)
+ * arrives, inflates in process while they boot, and inflates in process for good
+ * if one of them fails, so no read waits on a worker. What it is worth per
+ * format, and why GFF3 and GTF go without it: reference/BGZF_WORKER_POOL.md.
+ *
+ * A function over a dynamic import because the library inlines its worker bundle
+ * as a base64 string, and a static import pins it into every entry point that
+ * can reach this helper: measured with esbuild, 23.4kb gzipped against 141b. Test
+ * speed is not a reason either way; three jbrowse-web variant suites ran 13.887s
+ * static against 13.856s dynamic.
+ *
+ * Resolves to `undefined` wherever Workers cannot be created (node, jest), which
+ * keeps `@gmod/bam` and `@gmod/tabix` on their in-process path, and the `catch`
+ * keeps anything else going wrong here on it too.
  */
 export function sharedBgzfWorkerPool() {
     return import('@gmod/bgzf-filehandle')

package/esm/util/binnedAggregate.d.ts

@@ -0,0 +1,17 @@
+import type { AggregateStep, BinStep } from './markEncodingTypes.ts';
+import type { Staged } from './stepTables.ts';
+/**
+ * Whether an interval `bin` and the `aggregate` behind it run as one kernel:
+ * the aggregate groups by the bin's two edges, and every field it reads is a
+ * plain name, so a piece's value is its row's or one the bin wrote.
+ */
+export declare function fusesBinAggregate(bin: BinStep, agg: AggregateStep): boolean;
+/**
+ * An interval `bin` and the `aggregate` grouped by its edges as one walk: each
+ * row's bins accumulate straight into their group's sums, with no piece table,
+ * the way `binRawRegion` bins a BigWig's raw section. The groups come out in
+ * the order the pieces would have met them and each sum adds its pieces in
+ * the pieces' order, so the answer is the two steps' to the bit. Undefined
+ * where the bins spread too thinly over the region for a dense index.
+ */
+export declare function binnedAggregate({ table, bounds }: Staged, bin: BinStep, agg: AggregateStep): Staged | undefined;

package/esm/util/binnedAggregate.js

@@ -0,0 +1,275 @@
+// The interval `bin` and the `aggregate` behind it as one kernel, which
+// `runSteps` in featureTransforms.ts reaches by relative path only.
+import { readerOf } from './featureTable.js';
+import { isPlainFieldRef } from './fieldReader.js';
+import { isJexl } from './jexlStrings.js';
+import { BIN_OVERLAP_FIELD, DEFAULT_BIN_AS, OpSums, binSize, groupKey, intervalsOf, isWeighted, madeGroups, stepNumberReader, } from './stepTables.js';
+/**
+ * The bin's edges and `overlap` a piece carries, where an `aggregate` reads a
+ * field off a piece: the row's own field where the bin wrote none. A later
+ * entry of the bin's written fields shadows an earlier one of the same name,
+ * as the piece table's columns do.
+ */
+const FROM_ROW = 0;
+const FROM_BIN_START = 1;
+const FROM_BIN_END = 2;
+const FROM_OVERLAP = 3;
+function pieceSource(field, [asStart, asEnd]) {
+    return field === BIN_OVERLAP_FIELD
+        ? FROM_OVERLAP
+        : field === asEnd
+            ? FROM_BIN_END
+            : field === asStart
+                ? FROM_BIN_START
+                : FROM_ROW;
+}
+function pick(source, row, lo, hi, overlap) {
+    return source === FROM_ROW
+        ? row
+        : source === FROM_BIN_START
+            ? lo
+            : source === FROM_BIN_END
+                ? hi
+                : overlap;
+}
+/**
+ * Whether an interval `bin` and the `aggregate` behind it run as one kernel:
+ * the aggregate groups by the bin's two edges, and every field it reads is a
+ * plain name, so a piece's value is its row's or one the bin wrote.
+ */
+export function fusesBinAggregate(bin, agg) {
+    const as = bin.as ?? DEFAULT_BIN_AS;
+    const groupby = agg.groupby ?? [];
+    const plain = (field) => field === undefined || (isPlainFieldRef(field) && !isJexl(field));
+    return (bin.fields !== undefined &&
+        as[0] !== as[1] &&
+        groupby.length === 2 &&
+        as.every(edge => groupby.includes(edge)) &&
+        as.every(plain) &&
+        agg.ops.every(op => plain(op.field) && plain(op.weight)));
+}
+// Beyond this many bins over the pieces a dense bin index costs more than the
+// pieces would, and the two steps run as written.
+const SPARSE_BINS_PER_PIECE = 2;
+const SPARSE_BINS_SLACK = 65536;
+/**
+ * An interval `bin` and the `aggregate` grouped by its edges as one walk: each
+ * row's bins accumulate straight into their group's sums, with no piece table,
+ * the way `binRawRegion` bins a BigWig's raw section. The groups come out in
+ * the order the pieces would have met them and each sum adds its pieces in
+ * the pieces' order, so the answer is the two steps' to the bit. Undefined
+ * where the bins spread too thinly over the region for a dense index.
+ */
+export function binnedAggregate({ table, bounds }, bin, agg) {
+    const size = binSize(bin);
+    const as = bin.as ?? DEFAULT_BIN_AS;
+    const iv = intervalsOf(table, bin.fields, size, bounds);
+    const { lowBin, highBin } = iv;
+    const sections = bounds.length - 1;
+    // Each section's bins from its lowest, at `offset[s]` in one index.
+    const offset = new Float64Array(sections + 1);
+    for (let s = 0; s < sections; s++) {
+        const span = highBin[s] - lowBin[s] + 1;
+        offset[s + 1] = offset[s] + (span > 0 ? span : 0);
+    }
+    const spans = offset[sections];
+    if (spans > SPARSE_BINS_PER_PIECE * iv.pieces + SPARSE_BINS_SLACK) {
+        return undefined;
+    }
+    const capacity = Math.min(spans, iv.pieces);
+    const groupOfBin = new Int32Array(spans).fill(-1);
+    const groupRow = new Uint32Array(capacity);
+    const groupBin = new Float64Array(capacity);
+    const groupSection = new Uint32Array(capacity);
+    const start = new Float64Array(capacity);
+    const end = new Float64Array(capacity);
+    const startSource = pieceSource('start', as);
+    const endSource = pieceSource('end', as);
+    // A group's extent is its bin's where the pieces' start and end are the
+    // bin's edges, the least and greatest of its pieces' otherwise.
+    const binExtent = startSource === FROM_BIN_START && endSource === FROM_BIN_END;
+    const readRowStart = readerOf(table.column('start'));
+    const readRowEnd = readerOf(table.column('end'));
+    const groups = numberGroups({ iv, bounds, size, lowBin, offset, groupOfBin }, { groupRow, groupBin, groupSection, start, end }, binExtent
+        ? undefined
+        : { startSource, endSource, readRowStart, readRowEnd });
+    const walk = { iv, bounds, size, lowBin, offset, groupOfBin };
+    const ops = agg.ops.map(op => {
+        const sums = new OpSums(op, groups);
+        const valueSource = op.op === 'count' || op.field === undefined
+            ? undefined
+            : pieceSource(op.field, as);
+        const weightSource = op.weight !== undefined && isWeighted(op)
+            ? pieceSource(op.weight, as)
+            : undefined;
+        const rowNumbers = (field, source) => field !== undefined && source === FROM_ROW
+            ? stepNumberReader(table, field, 'an aggregate')
+            : undefined;
+        const readValue = rowNumbers(op.field, valueSource);
+        const readWeight = rowNumbers(op.weight, weightSource);
+        if ((valueSource === undefined || readValue) &&
+            (weightSource === undefined ||
+                readWeight ||
+                weightSource === FROM_OVERLAP)) {
+            addByRow(walk, sums, readValue, readWeight, weightSource === FROM_OVERLAP);
+        }
+        else {
+            addByPiece(walk, sums, valueSource, weightSource, readValue, readWeight);
+        }
+        return sums.column(groups);
+    });
+    // A field's value on each group's first piece.
+    const each = (field) => {
+        const source = pieceSource(field, as);
+        const read = readerOf(table.column(field));
+        const out = new Array(groups);
+        for (let g = 0; g < groups; g++) {
+            const i = groupRow[g];
+            const lo = groupBin[g] * size;
+            const hi = lo + size;
+            out[g] =
+                source === FROM_ROW
+                    ? read(i)
+                    : pick(source, 0, lo, hi, Math.min(iv.end[i], hi) - Math.max(iv.start[i], lo));
+        }
+        return out;
+    };
+    return madeGroups({
+        groups,
+        keys: (agg.groupby ?? []).map((field) => [
+            field,
+            each(field).map(groupKey),
+        ]),
+        ops,
+        step: agg,
+        refNames: each('refName'),
+        start: start.slice(0, groups),
+        end: end.slice(0, groups),
+        sectionOf: groupSection,
+        sections: bounds.length,
+    });
+}
+/**
+ * Each bin's group, numbered in the order the pieces meet them, with the
+ * first piece's row, the bin and the section, and each group's extent: its
+ * bin's, or the least start and greatest end its pieces read where `extent`
+ * says where they read them. Answers how many groups there are.
+ */
+function numberGroups({ iv, bounds, size, lowBin, offset, groupOfBin }, out, extent) {
+    const { groupRow, groupBin, groupSection, start, end } = out;
+    let groups = 0;
+    for (let s = 0; s + 1 < bounds.length; s++) {
+        const base = offset[s] - lowBin[s];
+        for (let i = bounds[s]; i < bounds[s + 1]; i++) {
+            const first = iv.first[i];
+            if (Number.isNaN(first)) {
+                continue;
+            }
+            const last = iv.last[i];
+            for (let b = first; b <= last; b++) {
+                let g = groupOfBin[base + b];
+                if (g < 0) {
+                    g = groups++;
+                    groupOfBin[base + b] = g;
+                    groupRow[g] = i;
+                    groupBin[g] = b;
+                    groupSection[g] = s;
+                    start[g] = b * size;
+                    end[g] = b * size + size;
+                }
+            }
+        }
+    }
+    if (extent) {
+        start.fill(Infinity);
+        end.fill(-Infinity);
+        const { startSource, endSource, readRowStart, readRowEnd } = extent;
+        for (let s = 0; s + 1 < bounds.length; s++) {
+            const base = offset[s] - lowBin[s];
+            for (let i = bounds[s]; i < bounds[s + 1]; i++) {
+                const first = iv.first[i];
+                if (Number.isNaN(first)) {
+                    continue;
+                }
+                const ownStart = readRowStart(i);
+                const ownEnd = readRowEnd(i);
+                for (let b = first; b <= iv.last[i]; b++) {
+                    const g = groupOfBin[base + b];
+                    const lo = b * size;
+                    const hi = lo + size;
+                    const overlap = Math.min(iv.end[i], hi) - Math.max(iv.start[i], lo);
+                    start[g] = Math.min(start[g], pick(startSource, ownStart, lo, hi, overlap));
+                    end[g] = Math.max(end[g], pick(endSource, ownEnd, lo, hi, overlap));
+                }
+            }
+        }
+    }
+    return groups;
+}
+/**
+ * One op's sums where its value is its row's and its weight its row's, its
+ * piece's overlap or none: a row whose value or weight is not a number adds
+ * nothing to any bin, so it is skipped whole. A piece's overlap is always a
+ * number, since the row's ends are.
+ */
+function addByRow({ iv, bounds, size, lowBin, offset, groupOfBin }, { sum, weight, min, max, op }, readValue, readWeight, byOverlap) {
+    const readsValue = op !== 'count';
+    for (let s = 0; s + 1 < bounds.length; s++) {
+        const base = offset[s] - lowBin[s];
+        for (let i = bounds[s]; i < bounds[s + 1]; i++) {
+            const first = iv.first[i];
+            const v = readValue ? readValue(i) : Number.NaN;
+            const rowWeight = readWeight ? readWeight(i) : 1;
+            if (Number.isNaN(first) ||
+                (readsValue && !Number.isFinite(v)) ||
+                !Number.isFinite(rowWeight)) {
+                continue;
+            }
+            const rowS = iv.start[i];
+            const rowE = iv.end[i];
+            const last = iv.last[i];
+            for (let b = first; b <= last; b++) {
+                const g = groupOfBin[base + b];
+                const lo = b * size;
+                const w = byOverlap
+                    ? Math.min(rowE, lo + size) - Math.max(rowS, lo)
+                    : rowWeight;
+                weight[g] += w;
+                if (readsValue) {
+                    sum[g] += v * w;
+                    if (v < min[g]) {
+                        min[g] = v;
+                    }
+                    if (v > max[g]) {
+                        max[g] = v;
+                    }
+                }
+            }
+        }
+    }
+}
+/** One op's sums where its value or weight is a field the bin wrote. */
+function addByPiece({ iv, bounds, size, lowBin, offset, groupOfBin }, sums, valueSource, weightSource, readValue, readWeight) {
+    for (let s = 0; s + 1 < bounds.length; s++) {
+        const base = offset[s] - lowBin[s];
+        for (let i = bounds[s]; i < bounds[s + 1]; i++) {
+            const first = iv.first[i];
+            if (Number.isNaN(first)) {
+                continue;
+            }
+            const rowV = readValue ? readValue(i) : Number.NaN;
+            const rowW = readWeight ? readWeight(i) : 1;
+            const last = iv.last[i];
+            for (let b = first; b <= last; b++) {
+                const lo = b * size;
+                const hi = lo + size;
+                const overlap = Math.min(iv.end[i], hi) - Math.max(iv.start[i], lo);
+                sums.add(groupOfBin[base + b], valueSource === undefined
+                    ? Number.NaN
+                    : pick(valueSource, rowV, lo, hi, overlap), weightSource === undefined
+                    ? 1
+                    : pick(weightSource, rowW, lo, hi, overlap));
+            }
+        }
+    }
+}

package/esm/util/cacheBudgets.d.ts

@@ -1,10 +1,10 @@
 import { SharedBudget } from '@gmod/shared-read-cache';
 /**
- * Ceilings on what the indexed-format libraries retain, shared by every adapter
- * in this JS context — so one per RPC worker, plus one on the main thread,
- * which is the scope that actually runs out of memory.
+ * The ceiling on what the indexed-format libraries retain, shared by every
+ * adapter in this JS context — so one per RPC worker, plus one on the main
+ * thread, which is the scope that actually runs out of memory.
  *
- * ## Why these are not per file
+ * ## Why this is not per file
  *
  * `@gmod/bam`, `@gmod/tabix` and `@gmod/cram` each take a per-file budget,
  * sized so that a single track panning never falls off the cliff where a cache
@@ -29,24 +29,27 @@ import { SharedBudget } from '@gmod/shared-read-cache';
  * at hand their space to the one being panned, so the active track keeps a
  * whole working set however many are open.
  *
- * ## The numbers are not new
+ * ## The number is not new
  *
- * Both are the libraries' own per-file defaults, moved from per-file to
- * per-context. That makes this a strict tightening — retention can only fall —
- * and avoids inventing a number that would need its own justification. The
- * budget still has to clear one query's working set or the cliff returns, and
- * on that workload 512 MB shared across three tracks already cost 43 refills
- * where 1 GB cost 4.
+ * 1 GB is the libraries' own per-file default (`DEFAULT_MAX_CACHE_BYTES` in
+ * both `@gmod/bam` and `@gmod/cram`), moved from per-file to per-context. That
+ * makes this a strict tightening — retention can only fall — and avoids
+ * inventing a number that would need its own justification. With one file open
+ * the two ceilings coincide; from the second file on, this one is the one that
+ * binds. The budget still has to clear one query's working set or the cliff
+ * returns, and on that workload 512 MB shared across three tracks already cost
+ * 43 refills where 1 GB cost 4.
  *
- * ## Two budgets, because units cannot mix
+ * ## One budget, because every member weighs bytes
  *
  * `SharedBudget.total` is a sum over its members, so they must all weigh in the
- * same unit. `@gmod/bam` and `@gmod/tabix` weigh decompressed bytes;
- * `@gmod/cram` weighs decoded records, because a decoded record has no cheap
- * size. A single budget spanning them would add records to bytes and bound
- * neither. `sizeOf` is opaque, so nothing can catch that at runtime — hence two
- * budgets named for their units.
+ * same unit, and `sizeOf` is opaque, so nothing could catch a mismatch at
+ * runtime. `@gmod/bam` and `@gmod/tabix` weigh decompressed bytes, and since 14
+ * `@gmod/cram` weighs a decoded slice by `DecodedSlice.byteLength` — its typed
+ * columns exactly and its strings by estimate, within 1–8% of the measured heap
+ * (its ADR 0013). Before that a CRAM slice weighed its record count, which is
+ * why this used to be two budgets named for their units: a million records
+ * could not be added to bytes, and never bound long-read data either, where a
+ * 6 MB slice counted as 37.
  */
 export declare const decompressedBytesBudget: SharedBudget;
-/** See {@link decompressedBytesBudget}. Records, not bytes — CRAM only. */
-export declare const decodedRecordsBudget: SharedBudget;

package/esm/util/cacheBudgets.js

@@ -1,10 +1,10 @@
 import { SharedBudget } from '@gmod/shared-read-cache';
 /**
- * Ceilings on what the indexed-format libraries retain, shared by every adapter
- * in this JS context — so one per RPC worker, plus one on the main thread,
- * which is the scope that actually runs out of memory.
+ * The ceiling on what the indexed-format libraries retain, shared by every
+ * adapter in this JS context — so one per RPC worker, plus one on the main
+ * thread, which is the scope that actually runs out of memory.
  *
- * ## Why these are not per file
+ * ## Why this is not per file
  *
  * `@gmod/bam`, `@gmod/tabix` and `@gmod/cram` each take a per-file budget,
  * sized so that a single track panning never falls off the cliff where a cache
@@ -29,24 +29,27 @@ import { SharedBudget } from '@gmod/shared-read-cache';
  * at hand their space to the one being panned, so the active track keeps a
  * whole working set however many are open.
  *
- * ## The numbers are not new
+ * ## The number is not new
  *
- * Both are the libraries' own per-file defaults, moved from per-file to
- * per-context. That makes this a strict tightening — retention can only fall —
- * and avoids inventing a number that would need its own justification. The
- * budget still has to clear one query's working set or the cliff returns, and
- * on that workload 512 MB shared across three tracks already cost 43 refills
- * where 1 GB cost 4.
+ * 1 GB is the libraries' own per-file default (`DEFAULT_MAX_CACHE_BYTES` in
+ * both `@gmod/bam` and `@gmod/cram`), moved from per-file to per-context. That
+ * makes this a strict tightening — retention can only fall — and avoids
+ * inventing a number that would need its own justification. With one file open
+ * the two ceilings coincide; from the second file on, this one is the one that
+ * binds. The budget still has to clear one query's working set or the cliff
+ * returns, and on that workload 512 MB shared across three tracks already cost
+ * 43 refills where 1 GB cost 4.
  *
- * ## Two budgets, because units cannot mix
+ * ## One budget, because every member weighs bytes
  *
  * `SharedBudget.total` is a sum over its members, so they must all weigh in the
- * same unit. `@gmod/bam` and `@gmod/tabix` weigh decompressed bytes;
- * `@gmod/cram` weighs decoded records, because a decoded record has no cheap
- * size. A single budget spanning them would add records to bytes and bound
- * neither. `sizeOf` is opaque, so nothing can catch that at runtime — hence two
- * budgets named for their units.
+ * same unit, and `sizeOf` is opaque, so nothing could catch a mismatch at
+ * runtime. `@gmod/bam` and `@gmod/tabix` weigh decompressed bytes, and since 14
+ * `@gmod/cram` weighs a decoded slice by `DecodedSlice.byteLength` — its typed
+ * columns exactly and its strings by estimate, within 1–8% of the measured heap
+ * (its ADR 0013). Before that a CRAM slice weighed its record count, which is
+ * why this used to be two budgets named for their units: a million records
+ * could not be added to bytes, and never bound long-read data either, where a
+ * 6 MB slice counted as 37.
  */
 export const decompressedBytesBudget = new SharedBudget(1024 * 2 ** 20);
-/** See {@link decompressedBytesBudget}. Records, not bytes — CRAM only. */
-export const decodedRecordsBudget = new SharedBudget(1_000_000);