qvdjs 0.9.1 → 0.9.3

package/README.md

@@ -16,10 +16,31 @@ The library works with real-world QVD files and maintains compatibility with Qli
 
 ---
 
-The _qvdjs_ library provides a simple API for reading/writing Qlik View Data (QVD) files in JavaScript.
-
-Using this library, it is possible to parse the binary QVD file format and convert it to a JavaScript object
-structure and vice versa. The library is written to be used in a Node.js environment exclusively.
+The _qvdjs_ library provides a simple API for reading and writing Qlik View Data (QVD) files in JavaScript.
+It parses the binary QVD format into a JavaScript object structure and back again, and is written for Node.js
+(20.10 or newer) exclusively.
+
+**What it does well:**
+
+- **Large files without loading them.** Pass `maxRows` and the library reads only the header, the symbols those
+  rows actually reference, and that slice of the index table. On a file whose symbol table is large because its
+  fields hold mostly unique values, this is the difference between minutes and under a second — and it is the
+  only way to read a file above 2 GiB at all. See [Lazy Loading](#lazy-loading).
+- **Writing at real row counts.** `toQvd()` handles hundreds of thousands of rows, with optional progress
+  callbacks for long writes. Files it produces open in Qlik Sense and QlikView.
+- **Refusing rather than crashing.** A load too large for the process throws a catchable `QvdValidationError`
+  naming the limit it hit and a row count that would fit, instead of a `FATAL ERROR: Reached heap limit` that no
+  `try`/`catch` can intercept. The budget comes from the V8 heap ceiling and any container memory limit — the
+  two things that actually kill a process — so it neither refuses work the machine can do nor waves through work
+  it cannot. See [QVD File Size Limitations](#qvd-file-size-limitations).
+- **Corrupt files are detected, not silently misread.** Truncated index tables, missing header delimiters and
+  out-of-range offsets raise typed errors rather than returning short or fabricated data.
+- **Path traversal protection by default.** File access is confined to the working directory unless you widen it,
+  and containment is decided by the filesystem — symlinks are resolved, so a link inside the allowed directory
+  pointing outside it is refused. See [Security Considerations](#security-considerations).
+
+**What it does not do yet** is worth knowing before you start: see
+[Known limitations](#known-limitations).
 
 ---
 
@@ -31,10 +52,13 @@ structure and vice versa. The library is written to be used in a Node.js environ
       - [Important: Symbol Table and High-Cardinality Fields](#important-symbol-table-and-high-cardinality-fields)
     - [Performance Optimizations](#performance-optimizations)
     - [QVD File Size Limitations](#qvd-file-size-limitations)
+      - [The 2 GiB boundary](#the-2-gib-boundary)
       - [Why Safety Limits Exist](#why-safety-limits-exist)
-    - [Progress Tracking for Large File Writes](#progress-tracking-for-large-file-writes)
+    - [Writing QVD files](#writing-qvd-files)
+      - [Progress tracking](#progress-tracking)
     - [Working with Metadata](#working-with-metadata)
     - [Security Considerations](#security-considerations)
+    - [Known limitations](#known-limitations)
   - [QVD File Format](#qvd-file-format)
     - [XML Header](#xml-header)
     - [Symbol Table](#symbol-table)
@@ -131,8 +155,11 @@ console.log(df.shape); // [1000, numberOfColumns]
 
 The QVD format stores data in two parts:
 
-1. **Symbol table**: Contains ALL unique values for ALL fields (must be fully loaded)
-2. **Index table**: Contains row-by-row indices into the symbol table (can be partially loaded with `maxRows`)
+1. **Symbol table**: Contains every unique value for every field. It has to be _scanned_ in full, because a
+   symbol's length is only known once the previous one has been read — but with `maxRows` the library parses
+   only the symbols the requested rows actually reference and steps over the rest. Scanning is cheap; parsing
+   is what costs memory.
+2. **Index table**: Contains row-by-row indices into the symbol table (only the requested rows are read)
 
 ⚠️ **Performance Impact of High-Cardinality Fields:**
 
@@ -142,12 +169,23 @@ If your QVD file contains fields with many unique values (high cardinality), suc
 - Timestamps with millisecond precision
 - Unique text fields
 
-The symbol table can become very large and **must be read completely** even when using `maxRows`. This means:
+The symbol table then becomes very large, and has to be scanned end to end even when using `maxRows`. Parsing
+is skipped for symbols the requested rows do not use, so the cost is I/O rather than memory:
 
 - **Small symbol table** (fields with reusable values): Fast loading regardless of file size
-  - Example: A 500MB file with only 100KB symbol table loads in ~100ms for 5000 rows ✅
-- **Large symbol table** (fields with unique values per row): Slow loading even with `maxRows`
-  - Example: A 500MB file with 266MB symbol table takes ~6 seconds for any row count ❌
+- **Large symbol table** (fields with unique values per row): Slower, because the scan is proportional to the
+  symbol table's size — but memory stays proportional to the rows you asked for, not to the file
+
+You can see what happened on any load through `loadStats` (see [QvdDataFrame](#qvddataframe)):
+
+```javascript
+const df = await QvdDataFrame.fromQvd('large.qvd', {maxRows: 1000});
+console.log(df.loadStats);
+// { symbolTableBytes: 451704, totalRows: 1705805, rowsLoaded: 1000,
+//   symbolFiltering: true, symbolsKept: 3898 }
+//
+// 3,898 symbols parsed for 1,000 rows, out of a file holding 1.7 million.
+```
 
 To check your QVD's symbol table size, look at the `NoOfSymbols` in field metadata - values close to the total row count indicate high cardinality.
 
@@ -170,24 +208,28 @@ When you specify `maxRows`, the library:
 2. Parses **only** those symbols from the symbol table
 3. Skips parsing unused symbols entirely (not just filtering after parsing)
 
-**Performance Impact:**
+This matters most on files whose symbol table is large because most values are unique — exactly the files where
+a naive `maxRows` would still pay for the whole table. Compared to parsing every symbol regardless of `maxRows`,
+loading a few thousand rows from a multi-million-row file has been measured at roughly an order of magnitude
+faster and an order of magnitude smaller in peak memory.
 
-Compared to parsing all symbols regardless of `maxRows`:
-
-- **~10x faster load times** - Loading 5,000 rows from a 20M row file: 8.3 seconds → 0.9 seconds
-- **~10x less memory usage** - Same operation: 1,616 MB → 180 MB
-- **Native or better efficiency** - Memory overhead reduced from 6.0x to 0.7x of raw data size
-
-**Real-World Example:**
+The saving is visible on any file through `loadStats`. On a bundled 1.7-million-row fixture, asking for 1,000
+rows parses 3,898 symbols:
 
 ```javascript
-// File: orders_20m.qvd (20M rows, 266MB symbol table)
-const df = await QvdDataFrame.fromQvd('orders_20m.qvd', {maxRows: 5000});
-// Load time: ~0.9 seconds
-// Memory usage: ~180 MB
-// Only 17,228 out of millions of symbols parsed
+const df = await QvdDataFrame.fromQvd('chicago_taxi_rides_2016_01.qvd', {
+  maxRows: 1000,
+  symbolFilteringThreshold: 0, // this fixture is well under the 50 MB default
+});
+console.log(df.loadStats);
+// { symbolTableBytes: 451704, totalRows: 1705805, rowsLoaded: 1000,
+//   symbolFiltering: true, symbolsKept: 3898 }
 ```
 
+The optimisation engages automatically once the symbol table passes `symbolFilteringThreshold`, which defaults
+to 50 MB — the point where the extra analysis pass pays for itself. The example lowers it so the path can be
+demonstrated on a small bundled fixture; on a real high-cardinality file it engages on its own.
+
 **Key Benefits:**
 
 - Much faster previews of large QVD files
@@ -199,6 +241,28 @@ This optimization is particularly effective for files with many unique values (h
 
 ### QVD File Size Limitations
 
+#### The 2 GiB boundary
+
+There is one hard limit worth knowing before anything else: **a full load cannot read a file larger than
+2 GiB**, because Node caps `fs.readFile` there. It fails with a raw Node error rather than a `QvdError`:
+
+```
+RangeError: File size (2362232013) is greater than 2 GiB
+```
+
+**`maxRows` reads past that boundary**, because the lazy path reads the file in chunks instead:
+
+```javascript
+// A 2.2 GiB file
+await QvdDataFrame.fromQvd(huge, {}); // ❌ RangeError, above
+await QvdDataFrame.fromQvd(huge, {maxRows: 1000}); // ✅ works
+```
+
+So above 2 GiB, `maxRows` is not an optimisation — it is the only way in. Removing the full-load ceiling is
+tracked in [#122](https://github.com/ptarmiganlabs/qvdjs/issues/122).
+
+Below that boundary, what limits you is memory rather than file size, which is the rest of this section.
+
 **Simple Explanation:**
 
 Node.js has memory limits that affect how large QVD files you can work with. By default, Node.js can use up to about 4GB of memory. This means if you try to load a very large QVD file, you might run out of memory and get an error. Think of it like trying to open a very large document on a computer with limited RAM - if the document is too big, it won't open.
@@ -231,6 +295,26 @@ The maximum QVD file size you can handle with qvdjs depends on several factors a
   });
   ```
 
+  **What the budget is measured against.** The V8 heap ceiling, and any container memory limit reported by
+  `process.constrainedMemory()`. Those are the two limits that actually terminate a process — exceeding the heap
+  is a fatal, uncatchable V8 error, and exceeding a cgroup limit is a SIGKILL that arrives as exit 137 with no
+  JavaScript error at all.
+
+  What the operating system reports as _free_ memory is deliberately **not** part of it. It is recorded in the
+  error context for diagnostics and ignored for the decision, because it does not describe a wall the process
+  can hit — a machine with virtual memory gets slower, not fatal — and because the figure is not dependable:
+  on macOS it counts only free and speculative pages, excluding the file-cache pages the OS reclaims on demand,
+  and readings on one idle machine varied nine-fold within minutes. Budgeting from it meant the same file loaded
+  or was refused depending on when you asked.
+
+  **Turning the check off.** `memorySafetyFactor: 0` disables it entirely, for callers who would rather manage
+  memory themselves, or run on a runtime whose limits cannot be measured — Bun reports its current heap as its
+  heap limit, so the ceiling it advertises is meaningless.
+
+  ```javascript
+  const df = await QvdDataFrame.fromQvd('large-file.qvd', {memorySafetyFactor: 0});
+  ```
+
   To phrase it differently: There is no magic here - viewing a 40 GB QVD on a laptop with 24 GB RAM will not work. That laptop may in fact struggle with QVD files larger than 4-6 GB depending on data characteristics - or happily work with 10+ GB files if the data is very friendly.
 
 - **Data Characteristics**: The actual memory consumption depends _heavily_ on what's inside your QVD:
@@ -284,7 +368,24 @@ try {
 
 For more technical details about the memory safety system, including specific thresholds and the four-tier protection model, see [docs/DYNAMIC_SAFETY_LIMITS.md](docs/DYNAMIC_SAFETY_LIMITS.md). For practical examples, see [docs/examples/heap-scaling-example.md](docs/examples/heap-scaling-example.md).
 
-### Progress Tracking for Large File Writes
+### Writing QVD files
+
+`toQvd()` builds the symbol and index tables in memory and writes the file in one pass. Two things are worth
+knowing about it.
+
+**Row count is no longer a barrier.** Writing used to fail outright above roughly 122,000 rows with a
+`RangeError: Maximum call stack size exceeded`, because the bit width for each column was derived with
+`Math.max(...oneArgumentPerRow)`. It is now derived from the symbol count in constant time, and writes of
+hundreds of thousands of rows are routine. The regression suite round-trips a 200,000-row data frame and a
+150,000-row slice of real Qlik output on every CI run, across Linux, Windows and macOS.
+
+**Memory scales with the data, not with the file you are writing.** A write holds the whole data frame plus its
+symbol and index tables, so it needs more memory than reading the equivalent file. There is no lazy equivalent
+of `maxRows` for writing — if the data does not fit in the heap, split it across several QVDs.
+
+Writes are not atomic; see [Known limitations](#known-limitations).
+
+#### Progress tracking
 
 When writing large QVD files, the `toQvd()` operation can take significant time. The library provides optional progress callbacks to track the write operation in real-time:
 
@@ -411,7 +512,25 @@ await df.toQvd('processed/sales-filtered.qvd', {
 - **Null Byte Protection**: Detects and blocks null byte injection attempts
 - **Default CWD Restriction**: By default, file operations are restricted to the current working directory (CWD) and its subdirectories to prevent path traversal attacks
 - **Custom Directory Restriction**: Optional `allowedDir` parameter allows you to specify a different base directory
-- **Security Errors**: Throws `QvdSecurityError` with detailed context when security violations are detected
+- **Symlinks are resolved**: Containment is decided by the filesystem, not by comparing strings. Both paths are
+  resolved through symlinks and compared by device and inode, so a link _inside_ `allowedDir` that points
+  outside it is refused — for writes as well as reads. A string comparison sees only the link's own name, still
+  under `allowedDir`, and lets it through; that meant a link planted in an upload directory could be used to
+  read any file the process could read, and to overwrite and truncate any file it could write.
+- **Case is handled as the filesystem handles it**: comparing by inode means `Qvd` and `qvd` are the same
+  directory on a case-insensitive volume and different directories on a case-sensitive one, without guessing
+  from `process.platform`. macOS supports both.
+- **Security Errors**: Throws `QvdSecurityError` with detailed context when security violations are detected.
+  The context includes a `check` field saying whether the filesystem or the fallback string comparison refused,
+  so a rejection of a path that looks contained is traceable to a symlink or a case difference.
+
+**A note on `allowedDir` values:** `null`, `undefined` and `''` all fall back to the current working directory
+rather than meaning "no restriction" — callers routinely produce those from optional config or a JSON round
+trip, and silently dropping the sandbox there would be a security hole. To permit an entire volume, pass its
+root explicitly (`'/'` on POSIX, `'C:\\'` on Windows).
+
+**Not covered:** this is a check on a path, so it remains open to a symlink swapped in between the check and the
+open. Closing that would require opening with `O_NOFOLLOW` and verifying the descriptor.
 
 **Best Practices:**
 
@@ -454,6 +573,22 @@ try {
 }
 ```
 
+### Known limitations
+
+Honest boundaries rather than an issue list — these are the ones that change what you can do.
+
+| Limitation                                                                   | What it means in practice                                                                                                                                                                                                                  | Tracked as                                                |
+| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
+| **Full loads stop at 2 GiB**                                                 | Use `maxRows` above that; the lazy path reads in chunks and has no such ceiling. Failure is a raw Node `RangeError`, not a `QvdError`.                                                                                                     | [#122](https://github.com/ptarmiganlabs/qvdjs/issues/122) |
+| **The memory estimate is based on symbol table size, not on rows × columns** | A file with a modest symbol table but very many rows can pass every check and then exhaust the heap while materialising rows. Loading tens of millions of rows, size the heap for the result rather than for the file.                     | [#121](https://github.com/ptarmiganlabs/qvdjs/issues/121) |
+| **Dual values are not round-tripped as pairs**                               | Qlik stores dates and timestamps as a number plus a display string. A data frame exposes one of the two, so a read-modify-write does not reproduce the original dual — a `BirthDate` field comes back as `24205`, not as a formatted date. | [#138](https://github.com/ptarmiganlabs/qvdjs/issues/138) |
+| **Writes are not atomic**                                                    | `toQvd()` writes in place. A failure part-way through, or two writers targeting one path, leaves the previous file damaged rather than intact. Write to a temporary path and rename if that matters.                                       | [#129](https://github.com/ptarmiganlabs/qvdjs/issues/129) |
+| **Writer input is not validated**                                            | Unsupported value types can produce a raw `TypeError`, or a file that is written but does not read back as intended. Feed `fromDict()` numbers, strings and `null`.                                                                        | [#130](https://github.com/ptarmiganlabs/qvdjs/issues/130) |
+
+Numeric-looking strings are currently coerced to numbers on read, which is being reconsidered as a breaking
+change in [#120](https://github.com/ptarmiganlabs/qvdjs/issues/120). Empty and whitespace-only strings are _not_
+coerced — that was a defect, and is fixed.
+
 ## QVD File Format
 
 The QVD file format is a binary file format that is used by QlikView to store data. The format is proprietary. However,
@@ -567,13 +702,14 @@ Empty QVDs are fully supported for both reading and writing, maintaining compati
 The `QvdDataFrame` class represents the data frame stored inside of a finally parsed QVD file. It provides a high-level
 abstraction access to the QVD file content. This includes meta information as well as access to the actual data records.
 
-| Property       | Type       | Description                                                                                                        |
-| -------------- | ---------- | ------------------------------------------------------------------------------------------------------------------ |
-| `shape`        | `number[]` | The shape of the data table. The first element is the number of rows, the second element is the number of columns. |
-| `data`         | `any[][]`  | The actual data records of the QVD file. The first dimension represents the single rows.                           |
-| `columns`      | `string[]` | The names of the fields that are contained in the QVD file.                                                        |
-| `metadata`     | `object`   | The complete metadata object from the QVD file header, or null if not loaded from a QVD file.                      |
-| `fileMetadata` | `object`   | File-level metadata from the QVD header (qvBuildNo, tableName, createUtcTime, etc.).                               |
+| Property       | Type       | Description                                                                                                                                        |
+| -------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `shape`        | `number[]` | The shape of the data table. The first element is the number of rows, the second element is the number of columns.                                 |
+| `data`         | `any[][]`  | The actual data records of the QVD file. The first dimension represents the single rows.                                                           |
+| `columns`      | `string[]` | The names of the fields that are contained in the QVD file.                                                                                        |
+| `metadata`     | `object`   | The complete metadata object from the QVD file header, or null if not loaded from a QVD file.                                                      |
+| `fileMetadata` | `object`   | File-level metadata from the QVD header (qvBuildNo, tableName, createUtcTime, etc.).                                                               |
+| `loadStats`    | `object`   | What the read did: `symbolTableBytes`, `totalRows`, `rowsLoaded`, `symbolFiltering`, `symbolsKept`. `null` unless the frame came from `fromQvd()`. |
 
 #### `static fromQvd(path: string, options?: object): Promise<QvdDataFrame>`
 
@@ -586,7 +722,8 @@ to a `QvdDataFrame` instance.
 - `options` (object, optional): Loading options
   - `maxRows` (number, optional): Maximum number of rows to load. If not specified, all rows are loaded. This is useful for loading only a subset of data from large QVD files to improve performance and reduce memory usage.
   - `allowedDir` (string, optional): Base directory for file access validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
-  - `memorySafetyFactor` (number, optional): Memory safety factor (0.0-1.0). Default is 0.3 (30%). Determines what percentage of available memory (or V8 heap limit, whichever is smaller) can be used. Increase this (e.g., to 0.5 or 0.7) when running with larger heap sizes via `--max-old-space-size` to allow processing of proportionally larger files.
+  - `memorySafetyFactor` (number, optional): Fraction (0.0-1.0) of the memory budget a load may use. Default is 0.3 (30%). The budget is the smaller of the V8 heap limit and any container memory limit; see [QVD File Size Limitations](#qvd-file-size-limitations). Increase this (e.g., to 0.5 or 0.7) when running with a larger heap via `--max-old-space-size`. **`0` disables the memory check entirely.**
+  - `symbolFilteringThreshold` (number, optional): Symbol table size, in bytes, above which a lazy load switches to the two-pass filtering path. Defaults to 50 MB, the point where the extra analysis pass pays for itself. Lower it to use filtering on smaller files, raise it to keep the simpler single-pass read for longer.
 
 **Example:**
 
@@ -604,8 +741,16 @@ const dfSecure = await QvdDataFrame.fromQvd('reports/sales.qvd', {
 
 // Load with increased memory usage for large heap configurations
 const dfLarge = await QvdDataFrame.fromQvd('large-file.qvd', {
-  memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
+  memorySafetyFactor: 0.5, // Use 50% of the budget instead of the default 30%
 });
+
+// Manage memory yourself: skip the check entirely
+const dfUnchecked = await QvdDataFrame.fromQvd('large-file.qvd', {memorySafetyFactor: 0});
+
+// Inspect what the read actually did
+const preview = await QvdDataFrame.fromQvd('large-file.qvd', {maxRows: 500});
+console.log(preview.loadStats.symbolFiltering); // true when the two-pass path ran
+console.log(preview.loadStats.symbolsKept); // how many symbols it kept
 ```
 
 #### `static fromDict(dict: object): Promise<QvdDataFrame>`