qvdjs 0.9.4 → 0.10.1

package/README.md

@@ -30,9 +30,10 @@ It parses the binary QVD format into a JavaScript object structure and back agai
   callbacks for long writes. Files it produces open in Qlik Sense and QlikView.
 - **Refusing rather than crashing.** A load too large for the process throws a catchable `QvdValidationError`
   naming the limit it hit and a row count that would fit, instead of a `FATAL ERROR: Reached heap limit` that no
-  `try`/`catch` can intercept. The budget comes from the V8 heap ceiling and any container memory limit — the
-  two things that actually kill a process — so it neither refuses work the machine can do nor waves through work
-  it cannot. See [QVD File Size Limitations](#qvd-file-size-limitations).
+  `try`/`catch` can intercept. The estimate accounts for the rows and columns being materialised, not just the
+  symbol table, and the budget comes from the V8 heap ceiling and any container memory limit — the two things
+  that actually kill a process. The suggested row count is checked against the same estimate before being
+  offered, so following it works. See [QVD File Size Limitations](#qvd-file-size-limitations).
 - **Corrupt files are detected, not silently misread.** Truncated index tables, missing header delimiters and
   out-of-range offsets raise typed errors rather than returning short or fabricated data.
 - **Path traversal protection by default.** File access is confined to the working directory unless you widen it,
@@ -134,6 +135,43 @@ console.log(df.head(5));
 The above example loads the _qvdjs_ library and parses an example QVD file. A QVD file is typically loaded using the static
 `QvdDataFrame.fromQvd` function of the `QvdDataFrame` class itself. After loading the file's content, numerous methods and properties are available to work with the parsed data.
 
+### Three ways to open a file
+
+`fromQvd` is the general one, and often not the one you want. A QVD is an XML header, then a
+symbol table of every distinct value, then a bit-packed index table of one code per cell — so how
+far into the file a read has to go is what separates these, and all three take the same options.
+
+| You want                                | Call                              | How far it reads                                                       |
+| --------------------------------------- | --------------------------------- | ---------------------------------------------------------------------- |
+| Rows, to index, iterate or write back    | `QvdDataFrame.fromQvd(path)`      | Everything, and materialises every row                                 |
+| A few columns of a large file            | `QvdColumnTable.fromQvd(path)`    | Everything, but stops before building rows — 141 MiB against 385 MiB   |
+| Only the schema: names, row count, types | `QvdDataFrame.readMetadata(path)` | The header alone. Constant cost, whatever the file's size              |
+
+```javascript
+import {QvdDataFrame, QvdColumnTable} from 'qvdjs';
+
+// What is in this file? Costs the same whether it is 20 KB or 20 GB.
+const {columns, rowCount} = await QvdDataFrame.readMetadata('sales.qvd');
+console.log(`${rowCount} rows x ${columns.length} columns`);
+
+// Sum one column without ever building a row.
+const table = await QvdColumnTable.fromQvd('sales.qvd');
+let total = 0;
+for (const value of table.column('amount')) {
+  if (typeof value === 'number') total += value;
+}
+
+// Rows, when you want rows.
+const df = await QvdDataFrame.fromQvd('sales.qvd');
+console.log(df.head(5));
+```
+
+Reaching for `fromQvd` when you wanted one of the other two is the common mistake, and
+`fromQvd(path, {maxRows: 0})` is not a substitute for `readMetadata`: it loads no rows but still
+reads and parses the whole symbol table, which grows with the data. See
+[QvdColumnTable](#qvdcolumntable) and the [API Documentation](#api-documentation) for the
+details and the measurements behind the table above.
+
 ### Lazy Loading
 
 For large QVD files, you can load only a specific number of rows to improve performance and reduce memory usage. The library implements **lazy loading** - it reads only the necessary portions of the file from disk, not the entire file.
@@ -287,11 +325,12 @@ The maximum QVD file size you can handle with qvdjs depends on several factors a
 
 - **Node.js Memory Limits**: By default, Node.js limits heap memory to approximately 4GB (varies by architecture and Node.js version). The library **automatically detects your configured heap size** and scales its safety limits accordingly. You can increase heap size using the `--max-old-space-size` flag (e.g., `node --max-old-space-size=16384 script.js` for 16GB), and qvdjs will automatically allow larger files.
 
-  For larger heap configurations, you can also adjust the `memorySafetyFactor` option (default 0.3 = 30%) to make more efficient use of available memory:
+  You can also adjust the `memorySafetyFactor` option (default 0.8 = 80% of the budget) to trade
+  headroom for reach:
 
   ```javascript
   const df = await QvdDataFrame.fromQvd('large-file.qvd', {
-    memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
+    memorySafetyFactor: 0.9, // Use 90% of the budget instead of the default 80%
   });
   ```
 
@@ -326,8 +365,32 @@ The maximum QVD file size you can handle with qvdjs depends on several factors a
 - **Operation Type**: Reading typically uses less memory than writing, especially when using lazy loading (`maxRows` option). Writing requires building complete symbol and index tables in memory.
 
 - **Practical Guidance**:
-  - For typical business data with moderate cardinality, files up to 1-2GB usually work well with default Node.js settings
-  - High-cardinality data (unique values in most rows) may limit you to smaller files (hundreds of MB)
+  - **Count cells, not megabytes.** What exhausts the heap on a full load is one array per row plus
+    one slot per cell, so a file's size on disk predicts very little. On a default 4 GB heap the
+    guard currently accepts roughly:
+
+    | Columns | Rows   | Cells |
+    | ------- | ------ | ----- |
+    | 5       | 30.5 M | 153 M |
+    | 10      | 22.5 M | 225 M |
+    | 20      | 14.7 M | 295 M |
+    | 40      | 8.7 M  | 349 M |
+
+    These are about 4× what they were before 2026-09-11, for two reasons: the bitwise decode
+    (#134) cut what a read actually needs by roughly 2.4×, and the guard's model was recalibrated
+    against that — it had been over-estimating a real read by 4.8×. The guard still errs high,
+    admitting a load only at 1.3–2× the heap it measurably needs, because under-estimating means
+    an uncatchable abort while over-estimating means a catchable refusal. Doubling the heap with
+    `--max-old-space-size` roughly doubles these numbers.
+
+  - **A columnar read is not bounded by any of this.** `QvdColumnTable.fromQvd()` stores codes in
+    typed arrays, which live outside the V8 heap, so the heap ceiling barely applies: the 38 MB
+    taxi fixture reads columnar in a 15 MB heap, where the same file as rows needs 367 MB. If you
+    are hitting the table above, reading the columns you need is usually the answer rather than a
+    bigger heap.
+
+  - High-cardinality data (unique values in most rows) also loads the symbol table, so the ceiling
+    drops further — subtract about 90 MB of heap per million distinct values
   - **With increased heap** (e.g., 16GB+), you can handle proportionally larger files by adjusting `memorySafetyFactor`
   - Use lazy loading (`maxRows` option) when possible to reduce memory footprint when reading
   - Monitor memory usage with tools like `process.memoryUsage()` for your specific use cases
@@ -432,7 +495,21 @@ These optimizations can reduce write times by 80-90% for large datasets (100K+ r
 
 ### Working with Metadata
 
-The library provides full access to QVD file and field metadata:
+To inspect a file's schema without reading its data, use `readMetadata`. It stops at the XML
+header, so it costs the same for a 40 MB file as for a 40 GB one:
+
+```javascript
+import {QvdDataFrame} from 'qvdjs';
+
+const {columns, rowCount, fields} = await QvdDataFrame.readMetadata('path/to/file.qvd');
+
+console.log(`${rowCount} rows x ${columns.length} columns`);
+```
+
+Note that `fromQvd(path, {maxRows: 0})` is not equivalent — it loads no rows, but still reads and
+parses the whole symbol table.
+
+For metadata alongside the data, every accessor below is available on a loaded frame:
 
 ```javascript
 import {QvdDataFrame} from 'qvdjs';
@@ -577,13 +654,13 @@ try {
 
 Honest boundaries rather than an issue list — these are the ones that change what you can do.
 
-| Limitation                                                                   | What it means in practice                                                                                                                                                                                                                  | Tracked as                                                |
-| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
-| **Full loads stop at 2 GiB**                                                 | Use `maxRows` above that; the lazy path reads in chunks and has no such ceiling. Failure is a raw Node `RangeError`, not a `QvdError`.                                                                                                     | [#122](https://github.com/ptarmiganlabs/qvdjs/issues/122) |
-| **The memory estimate is based on symbol table size, not on rows × columns** | A file with a modest symbol table but very many rows can pass every check and then exhaust the heap while materialising rows. Loading tens of millions of rows, size the heap for the result rather than for the file.                     | [#121](https://github.com/ptarmiganlabs/qvdjs/issues/121) |
-| **Dual values are not round-tripped as pairs**                               | Qlik stores dates and timestamps as a number plus a display string. A data frame exposes one of the two, so a read-modify-write does not reproduce the original dual — a `BirthDate` field comes back as `24205`, not as a formatted date. | [#138](https://github.com/ptarmiganlabs/qvdjs/issues/138) |
-| **Writes are not atomic**                                                    | `toQvd()` writes in place. A failure part-way through, or two writers targeting one path, leaves the previous file damaged rather than intact. Write to a temporary path and rename if that matters.                                       | [#129](https://github.com/ptarmiganlabs/qvdjs/issues/129) |
-| **Writer input is not validated**                                            | Unsupported value types can produce a raw `TypeError`, or a file that is written but does not read back as intended. Feed `fromDict()` numbers, strings and `null`.                                                                        | [#130](https://github.com/ptarmiganlabs/qvdjs/issues/130) |
+| Limitation                                       | What it means in practice                                                                                                                                                                                                                                                                                                                  | Tracked as                                                |
+| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
+| **Full loads stop at 2 GiB**                     | Use `maxRows` above that; the lazy path reads in chunks and has no such ceiling. Failure is a raw Node `RangeError`, not a `QvdError`.                                                                                                                                                                                                     | [#122](https://github.com/ptarmiganlabs/qvdjs/issues/122) |
+| **A full load materialises every row in memory** | Rows are stored as one JavaScript array per row with a boxed value per cell, measured at roughly 11 bytes per cell. That, not file size, sets the ceiling in the table above. `maxRows` reads a prefix, and `QvdColumnTable` avoids row materialisation entirely — but there is still no streaming or chunked read of an arbitrary window. | [#140](https://github.com/ptarmiganlabs/qvdjs/issues/140) |
+| **Dual values are not round-tripped as pairs**   | Qlik stores dates and timestamps as a number plus a display string. A data frame exposes one of the two, so a read-modify-write does not reproduce the original dual — a `BirthDate` field comes back as `24205`, not as a formatted date.                                                                                                 | [#138](https://github.com/ptarmiganlabs/qvdjs/issues/138) |
+| **Writes are not atomic**                        | `toQvd()` writes in place. A failure part-way through, or two writers targeting one path, leaves the previous file damaged rather than intact. Write to a temporary path and rename if that matters.                                                                                                                                       | [#129](https://github.com/ptarmiganlabs/qvdjs/issues/129) |
+| **Writer input is not validated**                | Unsupported value types can produce a raw `TypeError`, or a file that is written but does not read back as intended. Feed `fromDict()` numbers, strings and `null`.                                                                                                                                                                        | [#130](https://github.com/ptarmiganlabs/qvdjs/issues/130) |
 
 Numeric-looking strings are currently coerced to numbers on read, which is being reconsidered as a breaking
 change in [#120](https://github.com/ptarmiganlabs/qvdjs/issues/120). Empty and whitespace-only strings are _not_
@@ -722,7 +799,7 @@ to a `QvdDataFrame` instance.
 - `options` (object, optional): Loading options
   - `maxRows` (number, optional): Maximum number of rows to load. If not specified, all rows are loaded. This is useful for loading only a subset of data from large QVD files to improve performance and reduce memory usage.
   - `allowedDir` (string, optional): Base directory for file access validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
-  - `memorySafetyFactor` (number, optional): Fraction (0.0-1.0) of the memory budget a load may use. Default is 0.3 (30%). The budget is the smaller of the V8 heap limit and any container memory limit; see [QVD File Size Limitations](#qvd-file-size-limitations). Increase this (e.g., to 0.5 or 0.7) when running with a larger heap via `--max-old-space-size`. **`0` disables the memory check entirely.**
+  - `memorySafetyFactor` (number, optional): Fraction (0.0-1.0) of the memory budget a load may use. Default is 0.8 (80%). The budget is the smaller of the V8 heap limit and any container memory limit; see [QVD File Size Limitations](#qvd-file-size-limitations). Increase this (e.g., to 0.5 or 0.7) when running with a larger heap via `--max-old-space-size`. **`0` disables the memory check entirely.**
   - `symbolFilteringThreshold` (number, optional): Symbol table size, in bytes, above which a lazy load switches to the two-pass filtering path. Defaults to 50 MB, the point where the extra analysis pass pays for itself. Lower it to use filtering on smaller files, raise it to keep the simpler single-pass read for longer.
 
 **Example:**
@@ -741,7 +818,7 @@ const dfSecure = await QvdDataFrame.fromQvd('reports/sales.qvd', {
 
 // Load with increased memory usage for large heap configurations
 const dfLarge = await QvdDataFrame.fromQvd('large-file.qvd', {
-  memorySafetyFactor: 0.5, // Use 50% of the budget instead of the default 30%
+  memorySafetyFactor: 0.9, // Use 90% of the budget instead of the default 80%
 });
 
 // Manage memory yourself: skip the check entirely
@@ -753,6 +830,124 @@ console.log(preview.loadStats.symbolFiltering); // true when the two-pass path r
 console.log(preview.loadStats.symbolsKept); // how many symbols it kept
 ```
 
+#### `static readMetadata(path: string, options?: object): Promise<object>`
+
+Reads a QVD file's schema and header metadata without reading its data. The cost is the same
+whatever the file's size, because it stops at the XML header and never touches the symbol or
+index tables.
+
+This is not the same as `fromQvd(path, {maxRows: 0})`. That loads no rows but still reads and
+parses the entire symbol table — 0.4 MB on a 38 MB file, but 15 MB on a high-cardinality one, and
+it grows with the data. Measured on a 200,000-row file where every value is distinct,
+`readMetadata` is **352× faster**, and unlike `{maxRows: 0}` it does not get slower as the file
+grows.
+
+**Parameters:**
+
+- `path` (string): The path to the QVD file.
+- `options` (object, optional):
+  - `allowedDir` (string, optional): Base directory for file access validation, applied exactly as
+    it is for `fromQvd`.
+
+**Returns** a plain object — deliberately not a `QvdDataFrame`, since one with `data: []` would be
+indistinguishable from an empty file at the call site:
+
+| Property       | Type       | Description                                                                         |
+| -------------- | ---------- | ----------------------------------------------------------------------------------- |
+| `columns`      | `string[]` | Field names, in file order.                                                         |
+| `rowCount`     | `number`   | Rows the **file** declares. Nothing was loaded; this is not a count of rows read.   |
+| `columnCount`  | `number`   | Number of fields.                                                                   |
+| `fields`       | `object[]` | Per-field metadata, same shape and order as `getFieldMetadata()` on a loaded frame. |
+| `fileMetadata` | `object`   | Same shape as the `fileMetadata` accessor on a loaded frame.                        |
+| `metadata`     | `object`   | The raw `QvdTableHeader`, as `metadata` gives it.                                   |
+
+**Example:**
+
+```javascript
+// What is in this file, without reading any of it
+const {columns, rowCount, fields} = await QvdDataFrame.readMetadata('sales.qvd');
+
+console.log(`${rowCount} rows x ${columns.length} columns`);
+
+for (const field of fields) {
+  console.log(`${field.fieldName}: ${field.noOfSymbols} distinct values`);
+}
+
+// Decide whether it is worth loading at all
+if (rowCount < 1_000_000) {
+  const df = await QvdDataFrame.fromQvd('sales.qvd');
+}
+```
+
+### QvdColumnTable
+
+`QvdColumnTable` reads a QVD **as columns instead of rows**. It uses the same decoder and the
+same symbol resolution as `QvdDataFrame.fromQvd` — it simply stops before building rows, and
+keeps what the decoder already produced: one `Int32Array` of codes per field, and one resolved
+value per _distinct_ symbol.
+
+Measured on the bundled 1.7 M × 20 taxi fixture, each in its own process so only one
+representation is alive:
+
+|                          | Live memory | Sum one column |
+| ------------------------ | ----------- | -------------- |
+| `QvdDataFrame.fromQvd`   | 385 MiB     | 26.8 ms        |
+| `QvdColumnTable.fromQvd` | **141 MiB** | **3.6 ms**     |
+
+Use it when you want to scan or aggregate a few columns of a large file. Use `QvdDataFrame` when
+you want rows. It does not convert to a data frame, deliberately — holding both representations
+is the one configuration in which this costs more than it saves, and re-reading a file as rows
+costs no more than reading it as rows always did.
+
+```javascript
+import {QvdColumnTable} from 'qvdjs';
+
+const table = await QvdColumnTable.fromQvd('trips.qvd');
+const fare = table.column('fare');
+
+// The fastest scan: both sides are contiguous typed arrays and the dictionary fits in cache.
+const codes = fare.codes;
+const values = fare.numericSymbols();
+let total = 0;
+
+for (let row = 0; row < codes.length; row++) {
+  const code = codes[row];
+  if (code >= 0) {
+    const value = values[code];
+    if (!Number.isNaN(value)) total += value;
+  }
+}
+
+// Or, more simply
+for (const value of fare) {
+  /* number | string | null */
+}
+console.log(fare.at(0), fare.toArray().length);
+```
+
+**`QvdColumnTable`** — `static fromQvd(path, options)` (same options as `QvdDataFrame.fromQvd`),
+`column(name)`, `columns`, `rowCount`, `shape`, `metadata`, `loadStats`.
+
+**`QvdColumn`** — what `column(name)` returns, frozen:
+
+| Member                     | Type                 | Description                                                                                                      |
+| -------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| `name`                     | `string`             | The field name.                                                                                                  |
+| `length`                   | `number`             | Rows in the column.                                                                                              |
+| `codes`                    | `Int32Array`         | One stored index per row. **Negative means NULL.** The table's own array — treat as read-only.                   |
+| `symbols`                  | `ReadonlyArray<any>` | The distinct values, indexed by the codes. One entry per distinct value, not per row.                            |
+| `at(row)`                  | `any`                | The value of one row, or `null`.                                                                                 |
+| `[Symbol.iterator]`        |                      | Iterates values without materialising the column.                                                                |
+| `toArray()`                | `Array<any>`         | Lossless; cell-for-cell what `data[row][column]` holds. Caller-owned.                                            |
+| `numericSymbols()`         | `Float64Array`       | The **dictionary** as numbers, NaN for non-numeric. One entry per distinct value — 17 KB for a 1.7 M-row column. |
+| `toFloat64Array(options?)` | `Float64Array`       | One number **per row**. Lossy, so it throws on a non-numeric value unless you pass `{onNonNumeric: 'nan'}`.      |
+
+`toFloat64Array` refuses by default rather than writing NaN because on real QVDs the lossy case
+is common, not exceptional: in the bundled taxi fixture **no column is strictly numeric** —
+`dropoff_census_tract` is 43 % empty strings, and four columns are 100 % strings. Silently
+turning two fifths of a column into NaN would erase Qlik's distinction between a blank and a
+number. `numericSymbols()` is the cheap conversion; `toFloat64Array()` is the expensive one.
+
 #### `static fromDict(dict: object): Promise<QvdDataFrame>`
 
 The static method `QvdDataFrame.fromDict` constructs a data frame from a dictionary. The dictionary must contain the columns and