qvdjs 0.10.1 → 1.0.0

package/README.md

@@ -1,158 +1,63 @@
+<!-- Served from the npm tarball via jsDelivr rather than by relative path: this
+     repository is private, and npmjs.com resolves relative README images against
+     raw.githubusercontent.com, where they 404. See CLAUDE.md, "Logo assets". -->
+<p align="center">
+  <img src="https://cdn.jsdelivr.net/npm/qvdjs/img/logo/qvdjs_logo-512.png" alt="qvdjs logo" width="200" />
+</p>
+
 # qvdjs
 
-> Utility library for reading/writing Qlik Sense and QlikView (QVD) files in JavaScript/Node.js
-
-## ⚠️ Important Disclaimer
-
-**This library is based on reverse engineering of the QVD file format**, as the format is not publicly documented by Qlik. While extensive effort has been made to understand and implement the format correctly, there may be incorrect assumptions or interpretations of the file structure.
-
-**Comprehensive testing has been performed** to ensure that QVD files created or modified by qvdjs are valid and can be loaded by Qlik Sense and QlikView without errors. However, users should:
-
-- Test thoroughly with their specific use cases
-- Validate output files in their Qlik environment
-- Report any issues or inconsistencies discovered
-
-The library works with real-world QVD files and maintains compatibility with Qlik products, but it is an independent, community-driven implementation.
-
----
-
-The _qvdjs_ library provides a simple API for reading and writing Qlik View Data (QVD) files in JavaScript.
-It parses the binary QVD format into a JavaScript object structure and back again, and is written for Node.js
-(20.10 or newer) exclusively.
-
-**What it does well:**
-
-- **Large files without loading them.** Pass `maxRows` and the library reads only the header, the symbols those
-  rows actually reference, and that slice of the index table. On a file whose symbol table is large because its
-  fields hold mostly unique values, this is the difference between minutes and under a second — and it is the
-  only way to read a file above 2 GiB at all. See [Lazy Loading](#lazy-loading).
-- **Writing at real row counts.** `toQvd()` handles hundreds of thousands of rows, with optional progress
-  callbacks for long writes. Files it produces open in Qlik Sense and QlikView.
-- **Refusing rather than crashing.** A load too large for the process throws a catchable `QvdValidationError`
-  naming the limit it hit and a row count that would fit, instead of a `FATAL ERROR: Reached heap limit` that no
-  `try`/`catch` can intercept. The estimate accounts for the rows and columns being materialised, not just the
-  symbol table, and the budget comes from the V8 heap ceiling and any container memory limit — the two things
-  that actually kill a process. The suggested row count is checked against the same estimate before being
-  offered, so following it works. See [QVD File Size Limitations](#qvd-file-size-limitations).
-- **Corrupt files are detected, not silently misread.** Truncated index tables, missing header delimiters and
-  out-of-range offsets raise typed errors rather than returning short or fabricated data.
-- **Path traversal protection by default.** File access is confined to the working directory unless you widen it,
-  and containment is decided by the filesystem — symlinks are resolved, so a link inside the allowed directory
-  pointing outside it is refused. See [Security Considerations](#security-considerations).
-
-**What it does not do yet** is worth knowing before you start: see
-[Known limitations](#known-limitations).
-
----
-
-- [qvdjs](#qvdjs)
-  - [⚠️ Important Disclaimer](#️-important-disclaimer)
-  - [Install](#install)
-  - [Usage](#usage)
-    - [Lazy Loading](#lazy-loading)
-      - [Important: Symbol Table and High-Cardinality Fields](#important-symbol-table-and-high-cardinality-fields)
-    - [Performance Optimizations](#performance-optimizations)
-    - [QVD File Size Limitations](#qvd-file-size-limitations)
-      - [The 2 GiB boundary](#the-2-gib-boundary)
-      - [Why Safety Limits Exist](#why-safety-limits-exist)
-    - [Writing QVD files](#writing-qvd-files)
-      - [Progress tracking](#progress-tracking)
-    - [Working with Metadata](#working-with-metadata)
-    - [Security Considerations](#security-considerations)
-    - [Known limitations](#known-limitations)
-  - [QVD File Format](#qvd-file-format)
-    - [XML Header](#xml-header)
-    - [Symbol Table](#symbol-table)
-    - [Index Table](#index-table)
-    - [Empty QVD Files](#empty-qvd-files)
-  - [API Documentation](#api-documentation)
-    - [QvdDataFrame](#qvddataframe)
-      - [`static fromQvd(path: string, options?: object): Promise<QvdDataFrame>`](#static-fromqvdpath-string-options-object-promiseqvddataframe)
-      - [`static fromDict(dict: object): Promise<QvdDataFrame>`](#static-fromdictdict-object-promiseqvddataframe)
-      - [`head(n: number): QvdDataFrame`](#headn-number-qvddataframe)
-      - [`tail(n: number): QvdDataFrame`](#tailn-number-qvddataframe)
-      - [`rows(...args: number): QvdDataFrame`](#rowsargs-number-qvddataframe)
-      - [`at(row: number, column: string): any`](#atrow-number-column-string-any)
-      - [`select(...args: string): QvdDataFrame`](#selectargs-string-qvddataframe)
-      - [`toDict(): Promise<object>`](#todict-promiseobject)
-      - [`toQvd(path: string, options?: object): Promise<void>`](#toqvdpath-string-options-object-promisevoid)
-      - [`getFieldMetadata(fieldName: string): object | null`](#getfieldmetadatafieldname-string-object--null)
-      - [`getAllFieldMetadata(): object[]`](#getallfieldmetadata-object)
-      - [`setFileMetadata(metadata: object): void`](#setfilemetadatametadata-object-void)
-      - [`setFieldMetadata(fieldName: string, metadata: object): void`](#setfieldmetadatafieldname-string-metadata-object-void)
-  - [Documentation](#documentation)
-    - [For Users](#for-users)
-    - [For Contributors](#for-contributors)
-    - [Quick Links by Task](#quick-links-by-task)
-  - [Testing](#testing)
-    - [Running Tests](#running-tests)
-  - [Contributing](#contributing)
-  - [Contributors](#contributors)
-
----
-
-## Install
-
-_qvdjs_ is a Node.js module available through [npm](https://www.npmjs.com/). The recommended way to install and maintain _qvdjs_ as a dependency is through the Node.js Package Manager (NPM).
-Before installing this library, download and install Node.js.
-
-You can get _qvdjs_ using the following command:
+> Read and write Qlik Sense and QlikView (QVD) files from Node.js
 
 ```bash
-npm install qvdjs --save
+npm install qvdjs
 ```
 
-**Module Format Support:**
-This library is published as a **dual ESM/CJS package**, providing full compatibility with both modern ES modules and traditional CommonJS environments:
-
-- ✅ **ESM (ES Modules)**: Native `import` statements in Node.js and modern bundlers
-- ✅ **CommonJS**: Traditional `require()` for compatibility with older Node.js projects
-- ✅ **Bundlers**: Works with Webpack, Vite, Rollup, esbuild, and other modern build tools
-
-**Usage Examples:**
-
 ```javascript
-// ESM (ES Modules) - Modern Node.js and TypeScript
 import {QvdDataFrame} from 'qvdjs';
 
-// CommonJS - Traditional Node.js
-const {QvdDataFrame} = require('qvdjs');
+const df = await QvdDataFrame.fromQvd('sales.qvd');
+console.log(df.shape); // [ 1705805, 20 ]
+console.log(df.head(5));
 ```
 
-The package automatically provides the correct format based on your project's configuration.
+No Qlik installation, no ODBC driver, no running engine — just the file. Node 20.10 or newer,
+published as a dual ESM/CommonJS package, MIT licensed, one runtime dependency.
 
-## Usage
+📖 **[Full documentation at qvdjs.ptarmiganlabs.com](https://qvdjs.ptarmiganlabs.com)** — guides,
+API reference, the QVD format explained, measured performance, and troubleshooting.
 
-Below is a quick example how to use _qvdjs_.
+## ⚠️ This library is based on reverse engineering
 
-```javascript
-import {QvdDataFrame} from 'qvdjs';
+Qlik does not publish the QVD format. Everything qvdjs knows about it was worked out by reading
+files Qlik produced, and validated by feeding its own output back to Qlik Sense and QlikView.
 
-const df = await QvdDataFrame.fromQvd('path/to/file.qvd');
-console.log(df.head(5));
-```
-
-The above example loads the _qvdjs_ library and parses an example QVD file. A QVD file is typically loaded using the static
-`QvdDataFrame.fromQvd` function of the `QvdDataFrame` class itself. After loading the file's content, numerous methods and properties are available to work with the parsed data.
+That works well enough to be useful, and is not the same as being exact. **Test with your own
+files, and validate output in your own Qlik environment** before relying on it. Reading any bundled
+Qlik file and writing it back reproduces its symbol table byte for byte, whichever way it was read; a
+value you change is written as Qlik stores that kind of value. Where that boundary lies is the first
+thing worth reading:
+[What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v0.10/overview/what-a-round-trip-preserves/).
 
-### Three ways to open a file
+## Four ways to open a file
 
-`fromQvd` is the general one, and often not the one you want. A QVD is an XML header, then a
-symbol table of every distinct value, then a bit-packed index table of one code per cell — so how
-far into the file a read has to go is what separates these, and all three take the same options.
+`fromQvd` is the general one, and often not the one you want. A QVD is an XML header, then a symbol
+table holding every distinct value, then a bit-packed index table of one code per cell. Two things
+separate the four calls: how far into the file a read has to go, and how much of it the read then
+holds.
 
-| You want                                | Call                              | How far it reads                                                       |
-| --------------------------------------- | --------------------------------- | ---------------------------------------------------------------------- |
-| Rows, to index, iterate or write back    | `QvdDataFrame.fromQvd(path)`      | Everything, and materialises every row                                 |
-| A few columns of a large file            | `QvdColumnTable.fromQvd(path)`    | Everything, but stops before building rows — 141 MiB against 385 MiB   |
-| Only the schema: names, row count, types | `QvdDataFrame.readMetadata(path)` | The header alone. Constant cost, whatever the file's size              |
+| You want                                 | Call                                      | How far it reads, and what it holds                                  |
+| ---------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------- |
+| Rows, to index, iterate or write back    | `QvdDataFrame.fromQvd(path)`              | Everything, and materialises every row                               |
+| Rows from a file too large to hold       | `QvdDataFrame.iterate(path, {chunkSize})` | Everything, but holds two chunks — a 96 MB heap against 512 MB       |
+| A few columns of a large file            | `QvdColumnTable.fromQvd(path)`            | Everything, but stops before building rows — 141 MiB against 385 MiB |
+| Only the schema: names, row count, types | `QvdDataFrame.readMetadata(path)`         | The header alone. Constant cost, whatever the file's size            |
 
 ```javascript
 import {QvdDataFrame, QvdColumnTable} from 'qvdjs';
 
 // What is in this file? Costs the same whether it is 20 KB or 20 GB.
 const {columns, rowCount} = await QvdDataFrame.readMetadata('sales.qvd');
-console.log(`${rowCount} rows x ${columns.length} columns`);
 
 // Sum one column without ever building a row.
 const table = await QvdColumnTable.fromQvd('sales.qvd');
@@ -161,1044 +66,192 @@ for (const value of table.column('amount')) {
   if (typeof value === 'number') total += value;
 }
 
+// Every row of a file that will not fit, a chunk at a time.
+for await (const chunk of QvdDataFrame.iterate('sales.qvd', {chunkSize: 50_000})) {
+  process(chunk.data);
+}
+
 // Rows, when you want rows.
 const df = await QvdDataFrame.fromQvd('sales.qvd');
-console.log(df.head(5));
-```
-
-Reaching for `fromQvd` when you wanted one of the other two is the common mistake, and
-`fromQvd(path, {maxRows: 0})` is not a substitute for `readMetadata`: it loads no rows but still
-reads and parses the whole symbol table, which grows with the data. See
-[QvdColumnTable](#qvdcolumntable) and the [API Documentation](#api-documentation) for the
-details and the measurements behind the table above.
-
-### Lazy Loading
-
-For large QVD files, you can load only a specific number of rows to improve performance and reduce memory usage. The library implements **lazy loading** - it reads only the necessary portions of the file from disk, not the entire file.
-
-```javascript
-import {QvdDataFrame} from 'qvdjs';
-
-// Load only the first 1000 rows
-const df = await QvdDataFrame.fromQvd('path/to/file.qvd', {maxRows: 1000});
-console.log(df.shape); // [1000, numberOfColumns]
-```
-
-**How it works:**
-
-- The library reads only the header, symbol table, and the first N rows from the index table
-- This provides significant memory savings and faster loading times for large files
-
-#### Important: Symbol Table and High-Cardinality Fields
-
-The QVD format stores data in two parts:
-
-1. **Symbol table**: Contains every unique value for every field. It has to be _scanned_ in full, because a
-   symbol's length is only known once the previous one has been read — but with `maxRows` the library parses
-   only the symbols the requested rows actually reference and steps over the rest. Scanning is cheap; parsing
-   is what costs memory.
-2. **Index table**: Contains row-by-row indices into the symbol table (only the requested rows are read)
-
-⚠️ **Performance Impact of High-Cardinality Fields:**
-
-If your QVD file contains fields with many unique values (high cardinality), such as:
-
-- Unique IDs (OrderID, TransactionID, UUID)
-- Timestamps with millisecond precision
-- Unique text fields
-
-The symbol table then becomes very large, and has to be scanned end to end even when using `maxRows`. Parsing
-is skipped for symbols the requested rows do not use, so the cost is I/O rather than memory:
-
-- **Small symbol table** (fields with reusable values): Fast loading regardless of file size
-- **Large symbol table** (fields with unique values per row): Slower, because the scan is proportional to the
-  symbol table's size — but memory stays proportional to the rows you asked for, not to the file
-
-You can see what happened on any load through `loadStats` (see [QvdDataFrame](#qvddataframe)):
-
-```javascript
-const df = await QvdDataFrame.fromQvd('large.qvd', {maxRows: 1000});
-console.log(df.loadStats);
-// { symbolTableBytes: 451704, totalRows: 1705805, rowsLoaded: 1000,
-//   symbolFiltering: true, symbolsKept: 3898 }
-//
-// 3,898 symbols parsed for 1,000 rows, out of a file holding 1.7 million.
-```
-
-To check your QVD's symbol table size, look at the `NoOfSymbols` in field metadata - values close to the total row count indicate high cardinality.
-
-**When lazy loading works best:**
-
-- Previewing data from very large QVD files without loading the entire file into memory
-- Files where most fields have reusable values (low cardinality)
-- Data exploration and schema inspection of large datasets
-- Faster loading times when you only need a subset of the data
-
-### Performance Optimizations
-
-The library includes intelligent symbol table parsing that dramatically improves performance when loading partial data with `maxRows`:
-
-**Smart Symbol Loading:**
-
-When you specify `maxRows`, the library:
-
-1. Analyzes which symbols are actually needed for the requested rows
-2. Parses **only** those symbols from the symbol table
-3. Skips parsing unused symbols entirely (not just filtering after parsing)
-
-This matters most on files whose symbol table is large because most values are unique — exactly the files where
-a naive `maxRows` would still pay for the whole table. Compared to parsing every symbol regardless of `maxRows`,
-loading a few thousand rows from a multi-million-row file has been measured at roughly an order of magnitude
-faster and an order of magnitude smaller in peak memory.
-
-The saving is visible on any file through `loadStats`. On a bundled 1.7-million-row fixture, asking for 1,000
-rows parses 3,898 symbols:
-
-```javascript
-const df = await QvdDataFrame.fromQvd('chicago_taxi_rides_2016_01.qvd', {
-  maxRows: 1000,
-  symbolFilteringThreshold: 0, // this fixture is well under the 50 MB default
-});
-console.log(df.loadStats);
-// { symbolTableBytes: 451704, totalRows: 1705805, rowsLoaded: 1000,
-//   symbolFiltering: true, symbolsKept: 3898 }
-```
-
-The optimisation engages automatically once the symbol table passes `symbolFilteringThreshold`, which defaults
-to 50 MB — the point where the extra analysis pass pays for itself. The example lowers it so the path can be
-demonstrated on a small bundled fixture; on a real high-cardinality file it engages on its own.
-
-**Key Benefits:**
-
-- Much faster previews of large QVD files
-- Lower memory footprint for data exploration
-- Efficient handling of files with large symbol tables
-- Automatic optimization - no configuration needed
-
-This optimization is particularly effective for files with many unique values (high cardinality) where the symbol table is large but you only need to preview a small portion of the data.
-
-### QVD File Size Limitations
-
-#### The 2 GiB boundary
-
-There is one hard limit worth knowing before anything else: **a full load cannot read a file larger than
-2 GiB**, because Node caps `fs.readFile` there. It fails with a raw Node error rather than a `QvdError`:
-
-```
-RangeError: File size (2362232013) is greater than 2 GiB
-```
-
-**`maxRows` reads past that boundary**, because the lazy path reads the file in chunks instead:
-
-```javascript
-// A 2.2 GiB file
-await QvdDataFrame.fromQvd(huge, {}); // ❌ RangeError, above
-await QvdDataFrame.fromQvd(huge, {maxRows: 1000}); // ✅ works
 ```
 
-So above 2 GiB, `maxRows` is not an optimisation — it is the only way in. Removing the full-load ceiling is
-tracked in [#122](https://github.com/ptarmiganlabs/qvdjs/issues/122).
+Reaching for `fromQvd` when you wanted one of the other three is the common mistake, and
+`fromQvd(path, {maxRows: 0})` is **not** a substitute for `readMetadata`: it loads no rows but still
+parses the whole symbol table, which grows with the data.
 
-Below that boundary, what limits you is memory rather than file size, which is the rest of this section.
+→ [Choosing an entry point](https://qvdjs.ptarmiganlabs.com/v0.10/getting-started/)
 
-**Simple Explanation:**
+## What a cell holds
 
-Node.js has memory limits that affect how large QVD files you can work with. By default, Node.js can use up to about 4GB of memory. This means if you try to load a very large QVD file, you might run out of memory and get an error. Think of it like trying to open a very large document on a computer with limited RAM - if the document is too big, it won't open.
+| The file stores                                                     | The cell holds                                          |
+| ------------------------------------------------------------------- | ------------------------------------------------------- |
+| A number                                                            | A `number`                                              |
+| A string                                                            | A `string`, never parsed - `'007'` stays `'007'`        |
+| A dual: a number with the text Qlik displays, such as a date's text | Its `number`. The text is kept, and `textAt` returns it |
+| NULL                                                                | `null`                                                  |
 
-**What happens when files are too large:**
-
-- **When opening large files**: If a QVD file exceeds available memory, Node.js will throw an out-of-memory error (typically "JavaScript heap out of memory" or "FATAL ERROR: Reached heap limit"). The process will crash before completing the file load.
-- **When saving large files**: Writing very large QVD files can similarly exhaust memory during symbol table and index table construction, causing the same out-of-memory errors before the file is written to disk.
-- **Performance degradation**: Even before running out of memory completely, you may notice significant slowdowns, high memory usage, and system swapping as files approach memory limits.
-
-The good news is that Node.js memory limits can be increased (though there will always be some limit), and the actual file size you can handle depends on your data.
-
-The most common question at this point is usually:
-
-> "How large of a QVD file can I work with using qvdjs?"
-
-There is unfortunately no simple answer to this question, as it very much depends on the characteristics of what data is inside the QVD file. See below for more details.
-
-**Technical Details:**
-
-The maximum QVD file size you can handle with qvdjs depends on several factors and there is no single fixed limit:
-
-- **Node.js Memory Limits**: By default, Node.js limits heap memory to approximately 4GB (varies by architecture and Node.js version). The library **automatically detects your configured heap size** and scales its safety limits accordingly. You can increase heap size using the `--max-old-space-size` flag (e.g., `node --max-old-space-size=16384 script.js` for 16GB), and qvdjs will automatically allow larger files.
-
-  You can also adjust the `memorySafetyFactor` option (default 0.8 = 80% of the budget) to trade
-  headroom for reach:
-
-  ```javascript
-  const df = await QvdDataFrame.fromQvd('large-file.qvd', {
-    memorySafetyFactor: 0.9, // Use 90% of the budget instead of the default 80%
-  });
-  ```
-
-  **What the budget is measured against.** The V8 heap ceiling, and any container memory limit reported by
-  `process.constrainedMemory()`. Those are the two limits that actually terminate a process — exceeding the heap
-  is a fatal, uncatchable V8 error, and exceeding a cgroup limit is a SIGKILL that arrives as exit 137 with no
-  JavaScript error at all.
-
-  What the operating system reports as _free_ memory is deliberately **not** part of it. It is recorded in the
-  error context for diagnostics and ignored for the decision, because it does not describe a wall the process
-  can hit — a machine with virtual memory gets slower, not fatal — and because the figure is not dependable:
-  on macOS it counts only free and speculative pages, excluding the file-cache pages the OS reclaims on demand,
-  and readings on one idle machine varied nine-fold within minutes. Budgeting from it meant the same file loaded
-  or was refused depending on when you asked.
-
-  **Turning the check off.** `memorySafetyFactor: 0` disables it entirely, for callers who would rather manage
-  memory themselves, or run on a runtime whose limits cannot be measured — Bun reports its current heap as its
-  heap limit, so the ceiling it advertises is meaningless.
-
-  ```javascript
-  const df = await QvdDataFrame.fromQvd('large-file.qvd', {memorySafetyFactor: 0});
-  ```
-
-  To phrase it differently: There is no magic here - viewing a 40 GB QVD on a laptop with 24 GB RAM will not work. That laptop may in fact struggle with QVD files larger than 4-6 GB depending on data characteristics - or happily work with 10+ GB files if the data is very friendly.
-
-- **Data Characteristics**: The actual memory consumption depends _heavily_ on what's inside your QVD:
-  - **Field Cardinality**: Files with high-cardinality fields (many unique values per field, like unique IDs or timestamps) require more memory for symbol tables. This is usually the biggest factor, at least if there are many rows too.
-  - **Number of Rows**: More rows mean larger index tables in memory
-  - **Number of Fields**: More columns increase overall memory requirements
-  - **Data Types**: String data generally uses more memory than numeric data
-
-- **Operation Type**: Reading typically uses less memory than writing, especially when using lazy loading (`maxRows` option). Writing requires building complete symbol and index tables in memory.
-
-- **Practical Guidance**:
-  - **Count cells, not megabytes.** What exhausts the heap on a full load is one array per row plus
-    one slot per cell, so a file's size on disk predicts very little. On a default 4 GB heap the
-    guard currently accepts roughly:
-
-    | Columns | Rows   | Cells |
-    | ------- | ------ | ----- |
-    | 5       | 30.5 M | 153 M |
-    | 10      | 22.5 M | 225 M |
-    | 20      | 14.7 M | 295 M |
-    | 40      | 8.7 M  | 349 M |
-
-    These are about 4× what they were before 2026-09-11, for two reasons: the bitwise decode
-    (#134) cut what a read actually needs by roughly 2.4×, and the guard's model was recalibrated
-    against that — it had been over-estimating a real read by 4.8×. The guard still errs high,
-    admitting a load only at 1.3–2× the heap it measurably needs, because under-estimating means
-    an uncatchable abort while over-estimating means a catchable refusal. Doubling the heap with
-    `--max-old-space-size` roughly doubles these numbers.
-
-  - **A columnar read is not bounded by any of this.** `QvdColumnTable.fromQvd()` stores codes in
-    typed arrays, which live outside the V8 heap, so the heap ceiling barely applies: the 38 MB
-    taxi fixture reads columnar in a 15 MB heap, where the same file as rows needs 367 MB. If you
-    are hitting the table above, reading the columns you need is usually the answer rather than a
-    bigger heap.
-
-  - High-cardinality data (unique values in most rows) also loads the symbol table, so the ceiling
-    drops further — subtract about 90 MB of heap per million distinct values
-  - **With increased heap** (e.g., 16GB+), you can handle proportionally larger files by adjusting `memorySafetyFactor`
-  - Use lazy loading (`maxRows` option) when possible to reduce memory footprint when reading
-  - Monitor memory usage with tools like `process.memoryUsage()` for your specific use cases
-  - Consider processing large datasets in chunks or using streaming approaches if you hit memory limits. Clever things can be done by doing multiple passes over the file instead of loading everything at once.
-
-If you consistently work with very large QVD files, consider increasing Node.js memory limits (with matching `memorySafetyFactor` adjustment) or splitting your data into multiple smaller QVD files.
-
-#### Why Safety Limits Exist
-
-The library implements **dynamic safety limits** to prevent catastrophic crashes. Without these limits:
-
-- **Your application will crash hard** - Node.js terminates with `FATAL ERROR: Reached heap limit` when attempting to load files that are too large
-- **No error handling is possible** - JavaScript try-catch blocks cannot intercept out-of-memory (OOM) crashes at the V8 engine level
-- **The entire process dies** - Not just the QVD operation, but your entire application terminates ungracefully
-
-The safety limits **prevent these crashes** by checking available memory _before_ attempting to load files, throwing graceful `QvdValidationError` exceptions that you can catch and handle. While the multi-tier safety system may seem complex, it ensures your application stays running and provides helpful error messages with recommendations (like using `maxRows` parameter or increasing heap size) instead of cryptic fatal errors.
-
-**Example without safety limits:**
+`{duals: 'text'}` reads a dual as its text instead, and `{duals: 'both'}` as a frozen `QvdDual` holding
+both halves. `{coerceNumericStrings: true}` reads a numeric-looking string as a number. Whichever way a
+frame was read, it writes back the symbols it was read from.
 
 ```javascript
-// Process crashes with no chance to recover
-const df = await QvdDataFrame.fromQvd('huge-file.qvd'); // 💥 FATAL ERROR
-// Your application is now terminated
-```
+import {QvdDataFrame, qlikSerialToDate} from 'qvdjs';
 
-**Example with safety limits:**
-
-```javascript
-try {
-  const df = await QvdDataFrame.fromQvd('huge-file.qvd');
-} catch (error) {
-  if (error.name === 'QvdValidationError') {
-    console.log('File too large, trying with maxRows:', error.context.recommendedMaxRows);
-    // ✅ Your application continues running
-  }
-}
+const df = await QvdDataFrame.fromQvd('rides.qvd');
+df.data[0][0]; // 42382.260416666664 - the date is its number, as Qlik sums and sorts it
+df.textAt(0, 'started'); // '2016-01-13 06:15:00' - the text Qlik displays
+qlikSerialToDate(df.data[0][0]).toISOString(); // '2016-01-13T06:15:00.000Z'
 ```
 
-For more technical details about the memory safety system, including specific thresholds and the four-tier protection model, see [docs/DYNAMIC_SAFETY_LIMITS.md](docs/DYNAMIC_SAFETY_LIMITS.md). For practical examples, see [docs/examples/heap-scaling-example.md](docs/examples/heap-scaling-example.md).
-
-### Writing QVD files
-
-`toQvd()` builds the symbol and index tables in memory and writes the file in one pass. Two things are worth
-knowing about it.
-
-**Row count is no longer a barrier.** Writing used to fail outright above roughly 122,000 rows with a
-`RangeError: Maximum call stack size exceeded`, because the bit width for each column was derived with
-`Math.max(...oneArgumentPerRow)`. It is now derived from the symbol count in constant time, and writes of
-hundreds of thousands of rows are routine. The regression suite round-trips a 200,000-row data frame and a
-150,000-row slice of real Qlik output on every CI run, across Linux, Windows and macOS.
-
-**Memory scales with the data, not with the file you are writing.** A write holds the whole data frame plus its
-symbol and index tables, so it needs more memory than reading the equivalent file. There is no lazy equivalent
-of `maxRows` for writing — if the data does not fit in the heap, split it across several QVDs.
-
-Writes are not atomic; see [Known limitations](#known-limitations).
-
-#### Progress tracking
-
-When writing large QVD files, the `toQvd()` operation can take significant time. The library provides optional progress callbacks to track the write operation in real-time:
+## Writing
 
 ```javascript
-import {QvdDataFrame} from 'qvdjs';
-
 const df = await QvdDataFrame.fromDict({
-  columns: ['ID', 'Name', 'Value'],
-  data: largeDataArray, // e.g., 100,000+ rows
+  columns: ['Region', 'Year', 'Amount'],
+  data: [
+    ['North', 2025, 1200.0],
+    ['South', 2025, 980.5],
+  ],
 });
 
-await df.toQvd('output.qvd', {
-  onProgress: (progress) => {
-    console.log(`${progress.stage}: ${progress.percent}% complete`);
-  },
+await df.toQvd('out.qvd', {
+  onProgress: ({stage, percent}) => console.log(`${stage}: ${percent}%`),
 });
 ```
 
-**Progress Object Properties:**
-
-The `progress` parameter passed to the `onProgress` callback contains the following properties:
-
-- `stage` (string): Current operation stage being performed
-- `current` (number): Current progress value (e.g., rows processed, columns completed)
-- `total` (number): Total progress value for the current stage
-- `percent` (number): Progress percentage (0-100) calculated as `(current / total) * 100`
-
-**Progress stages:**
-
-- `symbol-table`: Building unique value tables for each column
-- `index-table`: Processing all data rows and creating index mappings
-- `header`: Generating XML header metadata
-- `write`: Writing data to disk
-
-**Performance optimizations:**
+Files it produces open in Qlik Sense and QlikView. A number is written as a pure number - an integer
+within 32 bits as an integer, any other number as a double - a string as a string, and `null` as
+NULL. A dual value - a `QvdDual`, or an object with exactly the keys `number` and `text` - is written
+as a dual, which is how a number gets the text Qlik displays. As in Qlik, a field holds one text per
+number, the first written. Anything else is refused with a `QvdValidationError`.
 
-The library uses optimized algorithms for large dataset processing:
+→ [Writing a QVD](https://qvdjs.ptarmiganlabs.com/v0.10/guides/)
 
-- **Single-pass symbol table building**: All columns are processed in one pass through the data (previously required one pass per column)
-- **Map-based lookups**: O(1) symbol index lookups instead of O(n) findIndex operations
-- **Reduced algorithmic complexity**: From O(n×m×s) to O(n×m) where n=rows, m=columns, s=symbols
+## Reading part of a file
 
-These optimizations can reduce write times by 80-90% for large datasets (100K+ rows), compared to earlier versions of the library.
-
-### Working with Metadata
-
-To inspect a file's schema without reading its data, use `readMetadata`. It stops at the XML
-header, so it costs the same for a 40 MB file as for a 40 GB one:
-
-```javascript
-import {QvdDataFrame} from 'qvdjs';
-
-const {columns, rowCount, fields} = await QvdDataFrame.readMetadata('path/to/file.qvd');
-
-console.log(`${rowCount} rows x ${columns.length} columns`);
-```
-
-Note that `fromQvd(path, {maxRows: 0})` is not equivalent — it loads no rows, but still reads and
-parses the whole symbol table.
-
-For metadata alongside the data, every accessor below is available on a loaded frame:
+The three calls that read data share the same options — `offset`, `limit` (or `maxRows`), `fields`,
+`duals`, `coerceNumericStrings`, `onProgress` and `signal` — because they are three answers about the
+same file rather than three features.
 
 ```javascript
-import {QvdDataFrame} from 'qvdjs';
-
-// Load QVD file
-const df = await QvdDataFrame.fromQvd('path/to/file.qvd');
-
-// Access file-level metadata
-console.log(df.fileMetadata.tableName);
-console.log(df.fileMetadata.createUtcTime);
-console.log(df.fileMetadata.noOfRecords);
-
-// Access field-level metadata
-const fieldMeta = df.getFieldMetadata('ProductKey');
-console.log(fieldMeta.comment);
-console.log(fieldMeta.numberFormat);
-console.log(fieldMeta.tags);
-
-// Get all field metadata
-const allFields = df.getAllFieldMetadata();
-allFields.forEach((field) => {
-  console.log(`${field.fieldName}: ${field.noOfSymbols} symbols`);
-});
-
-// Modify metadata (only modifiable properties can be changed)
-df.setFileMetadata({
-  tableName: 'UpdatedProducts',
-  comment: 'Modified product data',
-});
-
-df.setFieldMetadata('ProductKey', {
-  comment: 'Primary key for products',
-  tags: {String: ['$key', '$numeric']},
+// A window, and only the columns you need.
+const df = await QvdDataFrame.fromQvd('large.qvd', {
+  offset: 1_000_000,
+  limit: 1000,
+  fields: ['Region', 'Amount'],
 });
 
-// Metadata is preserved when writing
-await df.toQvd('path/to/output.qvd');
+console.log(df.loadStats);
+// { symbolTableBytes, totalRows, rowsLoaded, offset, symbolFiltering, symbolsKept }
 ```
 
-### Security Considerations
-
-The library includes built-in protection against path traversal attacks.
-
-**Default Security Behavior:**
-
-By default, all file operations are restricted to the **current working directory (CWD)** and its subdirectories. This means:
-
-- ✅ Files within CWD can be accessed: `./data/file.qvd` or `data/file.qvd`
-- ❌ Files outside CWD are blocked: `/etc/passwd` or `../../../sensitive.qvd`
-- ❌ Path traversal attempts are detected and blocked
-
-This default behavior protects against path traversal attacks without requiring additional configuration.
+**Above 2 GiB a full load cannot work at all** — Node caps `fs.readFile` there and it fails with a
+raw `RangeError`. A windowed read is not an optimisation above that boundary, it is the only way in.
 
-**Custom Directory Restriction:**
-
-You can explicitly specify a different base directory using the `allowedDir` option:
+Below it the limit is memory rather than file size, and a read too large to fit throws a **catchable**
+`QvdValidationError` carrying a row count that would have fitted, instead of a fatal
+`Reached heap limit` that no `try`/`catch` can intercept:
 
 ```javascript
-import {QvdDataFrame} from 'qvdjs';
-
-// Restrict reading to a specific directory
-const allowedDataDir = '/var/data/qvd-files';
-const df = await QvdDataFrame.fromQvd('reports/sales.qvd', {
-  allowedDir: allowedDataDir,
-});
-
-// Restrict writing to a specific directory
-const allowedOutputDir = '/var/output';
-await df.toQvd('processed/sales-filtered.qvd', {
-  allowedDir: allowedOutputDir,
-});
-```
-
-**Security Features:**
-
-- **Path Normalization**: All paths are automatically normalized using `path.resolve()` to eliminate `..` and `.` segments
-- **Null Byte Protection**: Detects and blocks null byte injection attempts
-- **Default CWD Restriction**: By default, file operations are restricted to the current working directory (CWD) and its subdirectories to prevent path traversal attacks
-- **Custom Directory Restriction**: Optional `allowedDir` parameter allows you to specify a different base directory
-- **Symlinks are resolved**: Containment is decided by the filesystem, not by comparing strings. Both paths are
-  resolved through symlinks and compared by device and inode, so a link _inside_ `allowedDir` that points
-  outside it is refused — for writes as well as reads. A string comparison sees only the link's own name, still
-  under `allowedDir`, and lets it through; that meant a link planted in an upload directory could be used to
-  read any file the process could read, and to overwrite and truncate any file it could write.
-- **Case is handled as the filesystem handles it**: comparing by inode means `Qvd` and `qvd` are the same
-  directory on a case-insensitive volume and different directories on a case-sensitive one, without guessing
-  from `process.platform`. macOS supports both.
-- **Security Errors**: Throws `QvdSecurityError` with detailed context when security violations are detected.
-  The context includes a `check` field saying whether the filesystem or the fallback string comparison refused,
-  so a rejection of a path that looks contained is traceable to a symlink or a case difference.
-
-**A note on `allowedDir` values:** `null`, `undefined` and `''` all fall back to the current working directory
-rather than meaning "no restriction" — callers routinely produce those from optional config or a JSON round
-trip, and silently dropping the sandbox there would be a security hole. To permit an entire volume, pass its
-root explicitly (`'/'` on POSIX, `'C:\\'` on Windows).
-
-**Not covered:** this is a check on a path, so it remains open to a symlink swapped in between the check and the
-open. Closing that would require opening with `O_NOFOLLOW` and verifying the descriptor.
-
-**Best Practices:**
-
-1. **Understand default security**: Files are restricted to CWD by default - no additional configuration needed for basic protection
-2. **Use explicit `allowedDir` in production**: When your application's CWD differs from your data directory, specify an explicit `allowedDir` for user-provided file paths
-3. **Validate user input**: Even with built-in protections, validate and sanitize any user-provided paths
-4. **Principle of least privilege**: Use the most restrictive `allowedDir` possible for your use case
-5. **Monitor security errors**: Log and monitor `QvdSecurityError` exceptions as they may indicate attack attempts
-
-**Examples:**
-
-```javascript
-import {QvdDataFrame, QvdSecurityError} from 'qvdjs';
-
-// Example 1: Default behavior (restricted to CWD)
-// This is safe by default - no path traversal possible
-try {
-  const df = await QvdDataFrame.fromQvd('data/file.qvd'); // ✅ Works (within CWD)
-  const df2 = await QvdDataFrame.fromQvd('../../../etc/passwd'); // ❌ Throws QvdSecurityError
-} catch (error) {
-  if (error instanceof QvdSecurityError) {
-    console.error('Path traversal blocked:', error.message);
-  }
-}
-
-// Example 2: Custom allowedDir for specific use cases
 try {
-  const df = await QvdDataFrame.fromQvd(userProvidedPath, {
-    allowedDir: '/safe/directory',
-  });
-  // Process data...
+  const df = await QvdDataFrame.fromQvd('huge.qvd');
 } catch (error) {
-  if (error instanceof QvdSecurityError) {
-    console.error('Security violation detected:', error.message);
-    console.error('Context:', error.context);
-    // Log security incident
-  } else {
-    throw error;
+  if (error.name === 'QvdValidationError') {
+    await QvdDataFrame.fromQvd('huge.qvd', {maxRows: error.context.recommendedMaxRows});
   }
 }
 ```
 
-### Known limitations
-
-Honest boundaries rather than an issue list — these are the ones that change what you can do.
-
-| Limitation                                       | What it means in practice                                                                                                                                                                                                                                                                                                                  | Tracked as                                                |
-| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
-| **Full loads stop at 2 GiB**                     | Use `maxRows` above that; the lazy path reads in chunks and has no such ceiling. Failure is a raw Node `RangeError`, not a `QvdError`.                                                                                                                                                                                                     | [#122](https://github.com/ptarmiganlabs/qvdjs/issues/122) |
-| **A full load materialises every row in memory** | Rows are stored as one JavaScript array per row with a boxed value per cell, measured at roughly 11 bytes per cell. That, not file size, sets the ceiling in the table above. `maxRows` reads a prefix, and `QvdColumnTable` avoids row materialisation entirely — but there is still no streaming or chunked read of an arbitrary window. | [#140](https://github.com/ptarmiganlabs/qvdjs/issues/140) |
-| **Dual values are not round-tripped as pairs**   | Qlik stores dates and timestamps as a number plus a display string. A data frame exposes one of the two, so a read-modify-write does not reproduce the original dual — a `BirthDate` field comes back as `24205`, not as a formatted date.                                                                                                 | [#138](https://github.com/ptarmiganlabs/qvdjs/issues/138) |
-| **Writes are not atomic**                        | `toQvd()` writes in place. A failure part-way through, or two writers targeting one path, leaves the previous file damaged rather than intact. Write to a temporary path and rename if that matters.                                                                                                                                       | [#129](https://github.com/ptarmiganlabs/qvdjs/issues/129) |
-| **Writer input is not validated**                | Unsupported value types can produce a raw `TypeError`, or a file that is written but does not read back as intended. Feed `fromDict()` numbers, strings and `null`.                                                                                                                                                                        | [#130](https://github.com/ptarmiganlabs/qvdjs/issues/130) |
-
-Numeric-looking strings are currently coerced to numbers on read, which is being reconsidered as a breaking
-change in [#120](https://github.com/ptarmiganlabs/qvdjs/issues/120). Empty and whitespace-only strings are _not_
-coerced — that was a defect, and is fixed.
-
-## QVD File Format
-
-The QVD file format is a binary file format that is used by QlikView to store data. The format is proprietary. However,
-the format is well documented and can be parsed without the need of a QlikView installation. In fact, a QVD file consists
-of three parts: a XML header, and two binary parts, the symbol and the index table. The XML header contains meta information
-about the QVD file, such as the number of data records and the names of the fields. The symbol table contains the actual
-distinct values of the fields. The index table contains the actual data records. The index table is a list of indices
-which point to values in the symbol table.
-
-### XML Header
-
-The XML header contains meta information about the QVD file. The header is always located at the beginning of the file and
-is in human readable text format. The header contains information about the number of data records, the names of the fields,
-and the data types of the fields.
-
-### Symbol Table
-
-The symbol table contains the distinct/unique values of the fields and is located directly after the XML header. The order
-of columns in the symbol table corresponds to the order of the fields in the XML header. The length and offset of the
-symbol sections of each column are also stored in the XML header.
-
-**Important**: The offset values in the XML header are **relative to the start of the symbol table section**, not absolute file positions. For example, if a field has `Offset=1143`, this means its symbol data starts 1143 bytes after the symbol table section begins (which itself starts immediately after the XML header ends).
-
-Each symbol section consists of the unique symbols of the
-respective column. The type of a single symbol is determined by a type byte prefixed to the respective symbol value. The
-following type of symbols are supported:
-
-| Code | Type         | Description                                                                                   |
-| ---- | ------------ | --------------------------------------------------------------------------------------------- |
-| 1    | Integer      | signed 4-byte integer (little endian)                                                         |
-| 2    | Float        | signed 8-byte IEEE floating point number (little endian)                                      |
-| 4    | String       | null terminated string                                                                        |
-| 5    | Dual Integer | signed 4-byte integer (little endian) followed by a null terminated string                    |
-| 6    | Dual Float   | signed 8-byte IEEE floating point number (little endian) followed by a null terminated string |
-
-### Index Table
-
-After the symbol table, the index table follows. The index table contains the actual data records. The index table contains
-binary indices that refrences to the values of each row in the symbol table. The order of the columns in the index table
-corresponds to the order of the fields in the XML header. Hence, the index table does not contain the actual values of a
-data record, but only the indices that point to the values in the symbol table.
-
-### Empty QVD Files
-
-QVD files can be empty in the sense that they contain zero data rows while still maintaining valid field definitions and metadata. This is a valid use case in Qlik applications where table structures need to be preserved even when no data is available.
-
-**Characteristics of Empty QVD Files:**
-
-- **NoOfRecords**: Set to `0` in the XML header
-- **RecordByteSize**: Set to `1` (following Qlik Sense's convention)
-- **Fields**: Field definitions are present and complete with metadata
-- **Symbol Table**: Each field has zero symbols (`NoOfSymbols=0`, `Offset=0`, `Length=0`)
-- **Index Table**: Empty with `Length=0`
-- **BitWidth**: Can vary per field (typically `0` or `8`)
-
-**Example Empty QVD XML Header:**
-
-```xml
-<QvdTableHeader>
-  <TableName>EmptyTable</TableName>
-  <Fields>
-    <QvdFieldHeader>
-      <FieldName>Country</FieldName>
-      <BitOffset>0</BitOffset>
-      <BitWidth>0</BitWidth>
-      <Bias>0</Bias>
-      <NoOfSymbols>0</NoOfSymbols>
-      <Offset>0</Offset>
-      <Length>0</Length>
-    </QvdFieldHeader>
-  </Fields>
-  <RecordByteSize>1</RecordByteSize>
-  <NoOfRecords>0</NoOfRecords>
-  <Offset>0</Offset>
-  <Length>0</Length>
-</QvdTableHeader>
-```
-
-**Working with Empty QVDs:**
-
-```javascript
-import {QvdDataFrame} from 'qvdjs';
-
-// Create an empty QVD with field structure but no data
-const emptyDf = new QvdDataFrame(
-  [], // No data rows
-  ['Country', 'Year', 'Sales'], // Column definitions
-);
-
-// Set metadata
-emptyDf.setFileMetadata({
-  tableName: 'EmptyTable',
-  comment: 'Template table structure',
-});
-
-// Write empty QVD (compatible with Qlik Sense)
-await emptyDf.toQvd('empty.qvd');
-
-// Read it back
-const loadedDf = await QvdDataFrame.fromQvd('empty.qvd');
-console.log(loadedDf.shape); // [0, 3] - zero rows, three columns
-console.log(loadedDf.columns); // ['Country', 'Year', 'Sales']
-```
-
-Empty QVDs are fully supported for both reading and writing, maintaining compatibility with files created by Qlik Sense and QlikView.
-
-## API Documentation
-
-### QvdDataFrame
-
-The `QvdDataFrame` class represents the data frame stored inside of a finally parsed QVD file. It provides a high-level
-abstraction access to the QVD file content. This includes meta information as well as access to the actual data records.
-
-| Property       | Type       | Description                                                                                                                                        |
-| -------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `shape`        | `number[]` | The shape of the data table. The first element is the number of rows, the second element is the number of columns.                                 |
-| `data`         | `any[][]`  | The actual data records of the QVD file. The first dimension represents the single rows.                                                           |
-| `columns`      | `string[]` | The names of the fields that are contained in the QVD file.                                                                                        |
-| `metadata`     | `object`   | The complete metadata object from the QVD file header, or null if not loaded from a QVD file.                                                      |
-| `fileMetadata` | `object`   | File-level metadata from the QVD header (qvBuildNo, tableName, createUtcTime, etc.).                                                               |
-| `loadStats`    | `object`   | What the read did: `symbolTableBytes`, `totalRows`, `rowsLoaded`, `symbolFiltering`, `symbolsKept`. `null` unless the frame came from `fromQvd()`. |
-
-#### `static fromQvd(path: string, options?: object): Promise<QvdDataFrame>`
-
-The static method `QvdDataFrame.fromQvd` loads a QVD file from the given path and parses it. The method returns a promise that resolves
-to a `QvdDataFrame` instance.
-
-**Parameters:**
-
-- `path` (string): The path to the QVD file.
-- `options` (object, optional): Loading options
-  - `maxRows` (number, optional): Maximum number of rows to load. If not specified, all rows are loaded. This is useful for loading only a subset of data from large QVD files to improve performance and reduce memory usage.
-  - `allowedDir` (string, optional): Base directory for file access validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
-  - `memorySafetyFactor` (number, optional): Fraction (0.0-1.0) of the memory budget a load may use. Default is 0.8 (80%). The budget is the smaller of the V8 heap limit and any container memory limit; see [QVD File Size Limitations](#qvd-file-size-limitations). Increase this (e.g., to 0.5 or 0.7) when running with a larger heap via `--max-old-space-size`. **`0` disables the memory check entirely.**
-  - `symbolFilteringThreshold` (number, optional): Symbol table size, in bytes, above which a lazy load switches to the two-pass filtering path. Defaults to 50 MB, the point where the extra analysis pass pays for itself. Lower it to use filtering on smaller files, raise it to keep the simpler single-pass read for longer.
-
-**Example:**
-
-```javascript
-// Load all rows (default behavior)
-const df = await QvdDataFrame.fromQvd('path/to/file.qvd');
-
-// Load only the first 1000 rows
-const dfLazy = await QvdDataFrame.fromQvd('path/to/file.qvd', {maxRows: 1000});
-
-// Load with security restriction (recommended for production)
-const dfSecure = await QvdDataFrame.fromQvd('reports/sales.qvd', {
-  allowedDir: '/var/data/qvd-files',
-});
-
-// Load with increased memory usage for large heap configurations
-const dfLarge = await QvdDataFrame.fromQvd('large-file.qvd', {
-  memorySafetyFactor: 0.9, // Use 90% of the budget instead of the default 80%
-});
-
-// Manage memory yourself: skip the check entirely
-const dfUnchecked = await QvdDataFrame.fromQvd('large-file.qvd', {memorySafetyFactor: 0});
-
-// Inspect what the read actually did
-const preview = await QvdDataFrame.fromQvd('large-file.qvd', {maxRows: 500});
-console.log(preview.loadStats.symbolFiltering); // true when the two-pass path ran
-console.log(preview.loadStats.symbolsKept); // how many symbols it kept
-```
-
-#### `static readMetadata(path: string, options?: object): Promise<object>`
-
-Reads a QVD file's schema and header metadata without reading its data. The cost is the same
-whatever the file's size, because it stops at the XML header and never touches the symbol or
-index tables.
-
-This is not the same as `fromQvd(path, {maxRows: 0})`. That loads no rows but still reads and
-parses the entire symbol table — 0.4 MB on a 38 MB file, but 15 MB on a high-cardinality one, and
-it grows with the data. Measured on a 200,000-row file where every value is distinct,
-`readMetadata` is **352× faster**, and unlike `{maxRows: 0}` it does not get slower as the file
-grows.
-
-**Parameters:**
-
-- `path` (string): The path to the QVD file.
-- `options` (object, optional):
-  - `allowedDir` (string, optional): Base directory for file access validation, applied exactly as
-    it is for `fromQvd`.
-
-**Returns** a plain object — deliberately not a `QvdDataFrame`, since one with `data: []` would be
-indistinguishable from an empty file at the call site:
-
-| Property       | Type       | Description                                                                         |
-| -------------- | ---------- | ----------------------------------------------------------------------------------- |
-| `columns`      | `string[]` | Field names, in file order.                                                         |
-| `rowCount`     | `number`   | Rows the **file** declares. Nothing was loaded; this is not a count of rows read.   |
-| `columnCount`  | `number`   | Number of fields.                                                                   |
-| `fields`       | `object[]` | Per-field metadata, same shape and order as `getFieldMetadata()` on a loaded frame. |
-| `fileMetadata` | `object`   | Same shape as the `fileMetadata` accessor on a loaded frame.                        |
-| `metadata`     | `object`   | The raw `QvdTableHeader`, as `metadata` gives it.                                   |
-
-**Example:**
-
-```javascript
-// What is in this file, without reading any of it
-const {columns, rowCount, fields} = await QvdDataFrame.readMetadata('sales.qvd');
-
-console.log(`${rowCount} rows x ${columns.length} columns`);
-
-for (const field of fields) {
-  console.log(`${field.fieldName}: ${field.noOfSymbols} distinct values`);
-}
-
-// Decide whether it is worth loading at all
-if (rowCount < 1_000_000) {
-  const df = await QvdDataFrame.fromQvd('sales.qvd');
-}
-```
-
-### QvdColumnTable
+When it is `iterate()` that overflowed, the context carries `recommendedChunkSize` as well, so a
+caller cannot mistake one recommendation for the other.
 
-`QvdColumnTable` reads a QVD **as columns instead of rows**. It uses the same decoder and the
-same symbol resolution as `QvdDataFrame.fromQvd` — it simply stops before building rows, and
-keeps what the decoder already produced: one `Int32Array` of codes per field, and one resolved
-value per _distinct_ symbol.
+→ [Memory and file size limits](https://qvdjs.ptarmiganlabs.com/v0.10/overview/memory-and-file-size-limits/)
 
-Measured on the bundled 1.7 M × 20 taxi fixture, each in its own process so only one
-representation is alive:
+## Metadata
 
-|                          | Live memory | Sum one column |
-| ------------------------ | ----------- | -------------- |
-| `QvdDataFrame.fromQvd`   | 385 MiB     | 26.8 ms        |
-| `QvdColumnTable.fromQvd` | **141 MiB** | **3.6 ms**     |
-
-Use it when you want to scan or aggregate a few columns of a large file. Use `QvdDataFrame` when
-you want rows. It does not convert to a data frame, deliberately — holding both representations
-is the one configuration in which this costs more than it saves, and re-reading a file as rows
-costs no more than reading it as rows always did.
+Table name, comments, number formats and field tags survive a read, can be changed on any frame, and
+are written back out - less a tag or number format the written values contradict, such as `$numeric`
+on a field that now holds text.
 
 ```javascript
-import {QvdColumnTable} from 'qvdjs';
-
-const table = await QvdColumnTable.fromQvd('trips.qvd');
-const fare = table.column('fare');
+const df = await QvdDataFrame.fromQvd('products.qvd');
+df.fileMetadata.tableName; // 'Products'
+df.getFieldMetadata('ProductKey'); // { comment, numberFormat, tags, ... }
 
-// The fastest scan: both sides are contiguous typed arrays and the dictionary fits in cache.
-const codes = fare.codes;
-const values = fare.numericSymbols();
-let total = 0;
-
-for (let row = 0; row < codes.length; row++) {
-  const code = codes[row];
-  if (code >= 0) {
-    const value = values[code];
-    if (!Number.isNaN(value)) total += value;
-  }
-}
-
-// Or, more simply
-for (const value of fare) {
-  /* number | string | null */
-}
-console.log(fare.at(0), fare.toArray().length);
+df.setFileMetadata({tableName: 'UpdatedProducts'});
+await df.toQvd('products-v2.qvd');
 ```
 
-**`QvdColumnTable`** — `static fromQvd(path, options)` (same options as `QvdDataFrame.fromQvd`),
-`column(name)`, `columns`, `rowCount`, `shape`, `metadata`, `loadStats`.
-
-**`QvdColumn`** — what `column(name)` returns, frozen:
-
-| Member                     | Type                 | Description                                                                                                      |
-| -------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------- |
-| `name`                     | `string`             | The field name.                                                                                                  |
-| `length`                   | `number`             | Rows in the column.                                                                                              |
-| `codes`                    | `Int32Array`         | One stored index per row. **Negative means NULL.** The table's own array — treat as read-only.                   |
-| `symbols`                  | `ReadonlyArray<any>` | The distinct values, indexed by the codes. One entry per distinct value, not per row.                            |
-| `at(row)`                  | `any`                | The value of one row, or `null`.                                                                                 |
-| `[Symbol.iterator]`        |                      | Iterates values without materialising the column.                                                                |
-| `toArray()`                | `Array<any>`         | Lossless; cell-for-cell what `data[row][column]` holds. Caller-owned.                                            |
-| `numericSymbols()`         | `Float64Array`       | The **dictionary** as numbers, NaN for non-numeric. One entry per distinct value — 17 KB for a 1.7 M-row column. |
-| `toFloat64Array(options?)` | `Float64Array`       | One number **per row**. Lossy, so it throws on a non-numeric value unless you pass `{onNonNumeric: 'nan'}`.      |
-
-`toFloat64Array` refuses by default rather than writing NaN because on real QVDs the lossy case
-is common, not exceptional: in the bundled taxi fixture **no column is strictly numeric** —
-`dropoff_census_tract` is 43 % empty strings, and four columns are 100 % strings. Silently
-turning two fifths of a column into NaN would erase Qlik's distinction between a blank and a
-number. `numericSymbols()` is the cheap conversion; `toFloat64Array()` is the expensive one.
-
-#### `static fromDict(dict: object): Promise<QvdDataFrame>`
-
-The static method `QvdDataFrame.fromDict` constructs a data frame from a dictionary. The dictionary must contain the columns and
-the actual data as properties. The columns property is an array of strings that contains the names of the fields in the QVD file.
-The data property is an array of arrays that contains the actual data records. The order of the values in the inner arrays
-corresponds to the order of the fields in the QVD file.
-
-#### `head(n: number): QvdDataFrame`
-
-The method `head` returns the first `n` rows of the data frame.
-
-#### `tail(n: number): QvdDataFrame`
-
-The method `tail` returns the last `n` rows of the data frame.
-
-#### `rows(...args: number): QvdDataFrame`
-
-The method `rows` returns a new data frame that contains only the specified rows.
-
-#### `at(row: number, column: string): any`
-
-The method `at` returns the value at the specified row and column.
-
-#### `select(...args: string): QvdDataFrame`
+→ [Metadata reference](https://qvdjs.ptarmiganlabs.com/v0.10/reference/)
 
-The method `select` returns a new data frame that contains only the specified columns.
+## File paths are sandboxed
 
-#### `toDict(): Promise<object>`
-
-The method `toDict` returns the data frame as a dictionary. The dictionary contains the columns and the
-actual data as properties. The columns property is an array of strings that contains the names of the
-fields in the QVD file. The data property is an array of arrays that contains the actual data records.
-The order of the values in the inner arrays corresponds to the order of the fields in the QVD file.
-
-#### `toQvd(path: string, options?: object): Promise<void>`
-
-The method `toQvd` writes the data frame to a QVD file at the specified path.
-
-**Parameters:**
-
-- `path` (string): The path where the QVD file should be written.
-- `options` (object, optional): Writing options
-  - `allowedDir` (string, optional): Base directory for file write validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
-  - `onProgress` (function, optional): Progress callback function for tracking write operations. Receives progress updates during symbol table building, index table building, header generation, and data writing.
-
-**Progress Callback:**
-
-The `onProgress` callback receives an object with the following properties:
-
-- `stage` (string): Current operation stage - `'symbol-table'`, `'index-table'`, `'header'`, or `'write'`
-- `current` (number): Current progress value (e.g., rows processed, columns completed)
-- `total` (number): Total progress value
-- `percent` (number): Progress percentage (0-100)
-
-This is particularly useful for large QVD files where write operations can take significant time.
-
-**Examples:**
+Every path is checked against an allowed directory before it is opened. **Leaving the option out
+does not mean "anywhere"** — it means the current working directory.
 
 ```javascript
-// Write to file (default behavior)
-await df.toQvd('output/data.qvd');
-
-// Write with security restriction (recommended for production)
-await df.toQvd('processed/data.qvd', {
-  allowedDir: '/var/output/qvd-files',
-});
-
-// Write with progress tracking for large files
-await df.toQvd('large-output.qvd', {
-  onProgress: (progress) => {
-    console.log(`${progress.stage}: ${progress.percent}% (${progress.current}/${progress.total})`);
-  },
-});
-
-// Write with detailed progress bar
-await df.toQvd('data.qvd', {
-  onProgress: (progress) => {
-    const bar = '█'.repeat(Math.floor(progress.percent / 2)) + '░'.repeat(50 - Math.floor(progress.percent / 2));
-    process.stdout.write(`\r[${progress.stage}] ${bar} ${progress.percent}%`);
-    if (progress.current === progress.total) console.log(' ✓');
-  },
-});
+await QvdDataFrame.fromQvd('sales.qvd', {allowedDir: '/var/data/qvd'});
+await QvdDataFrame.fromQvd(anyPath, {allowedDir: '/'}); // deliberately unrestricted
 ```
 
-#### `getFieldMetadata(fieldName: string): object | null`
-
-The method `getFieldMetadata` returns the metadata for a specific field/column from the QVD header. Returns null if the field is not found or metadata is not available.
+Containment is decided by the filesystem — both paths are resolved through symlinks and compared by
+device and inode — so a link inside the allowed directory pointing out of it is refused.
 
-The returned object contains:
+→ [Path security](https://qvdjs.ptarmiganlabs.com/v0.10/overview/path-security/)
 
-- `fieldName`: Name of the field
-- `bitOffset`: Bit offset in the index table
-- `bitWidth`: Bit width in the index table
-- `bias`: Bias value for index calculation
-- `noOfSymbols`: Number of unique symbols/values
-- `offset`: Byte offset in the symbol table
-- `length`: Byte length in the symbol table
-- `comment`: Field comment (modifiable)
-- `numberFormat`: Number format settings (modifiable)
-- `tags`: Field tags (modifiable)
+## Errors
 
-Note: Properties like `offset`, `length`, `bitOffset`, `bitWidth`, `bias`, and `noOfSymbols` are immutable and relate to internal data storage.
+`QvdError` and five subclasses, each carrying a `code` and a `context` object:
+`QvdValidationError` (`QVD_VALIDATION_ERROR`), `QvdCorruptedError` (`QVD_CORRUPTED_ERROR`),
+`QvdParseError` (`QVD_PARSE_ERROR`), `QvdIOError` (`QVD_IO_ERROR`) and `QvdSecurityError`
+(`QVD_SECURITY_ERROR`). Nothing currently throws `QvdIOError`; a filesystem failure surfaces as the
+underlying Node error.
 
-#### `getAllFieldMetadata(): object[]`
+`error.name` is always the class name, in the CommonJS and ES module builds alike, so it works across
+a module boundary where `instanceof` may not.
 
-The method `getAllFieldMetadata` returns an array of metadata objects for all fields in the data frame. Each object has the same structure as returned by `getFieldMetadata`.
+→ [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v0.10/troubleshooting/)
 
-#### `setFileMetadata(metadata: object): void`
+## Known limitations
 
-The method `setFileMetadata` allows modifying file-level metadata. Only modifiable properties are updated; immutable properties related to data storage are ignored.
-
-Modifiable properties:
-
-- `qvBuildNo`: QlikView build number
-- `creatorDoc`: Document GUID that created the QVD
-- `createUtcTime`: Creation timestamp
-- `sourceCreateUtcTime`: Source creation timestamp
-- `sourceFileUtcTime`: Source file timestamp
-- `sourceFileSize`: Source file size
-- `staleUtcTime`: Stale timestamp
-- `tableName`: Table name
-- `compression`: Compression method
-- `comment`: Table comment
-- `encryptionInfo`: Encryption information
-- `tableTags`: Table tags
-- `profilingData`: Profiling data
-- `lineage`: Data lineage information
-
-Immutable properties (cannot be modified):
-
-- `noOfRecords`: Number of records
-- `recordByteSize`: Record byte size
-- `offset`: Byte offset in file
-- `length`: Byte length in file
-
-#### `setFieldMetadata(fieldName: string, metadata: object): void`
-
-The method `setFieldMetadata` allows modifying field-level metadata for a specific field. Only modifiable properties are updated; immutable properties related to data storage are ignored.
-
-Modifiable properties:
-
-- `comment`: Field comment/description
-- `numberFormat`: Number format settings (Type, nDec, UseThou, Fmt, Dec, Thou)
-- `tags`: Field tags (typically used for field classification)
+Honest boundaries rather than an issue list — these are the ones that change what you can do.
 
-Immutable properties (cannot be modified):
+| Limitation                                                    | What it means in practice                                                                                                                                                                                                                                                                                                                   |
+| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Full loads stop at 2 GiB**                                  | Use a windowed read or `iterate()` above that. Failure is a raw Node `RangeError`, not a `QvdError`.                                                                                                                                                                                                                                        |
+| **The symbol table is always parsed in full**                 | `{offset, limit}`, `iterate()` and `QvdColumnTable` all avoid materialising rows, but none is constant-memory in the size of a high-cardinality file.                                                                                                                                                                                       |
+| **Writes are not atomic**                                     | `toQvd()` writes in place. A failure part-way through leaves the previous file damaged. Write to a temporary path and rename if that matters.                                                                                                                                                                                               |
+| **The writer takes numbers, strings, dual values and `null`** | Anything else - a `Date`, a boolean, an array, a `QvdSymbol` - is refused with a `QvdValidationError` naming the field and the row. Convert first: a `Date` to `new QvdDual(dateToQlikSerial(date), text)`, the serial and the text Qlik shows, which is how Qlik stores a date; a boolean to `-1` and `0`, as a Qlik comparison stores it. |
 
-- `offset`: Byte offset in symbol table
-- `length`: Byte length in symbol table
-- `bitOffset`: Bit offset in index table
-- `bitWidth`: Bit width in index table
-- `bias`: Bias value
-- `noOfSymbols`: Number of symbols
+→ [What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v0.10/overview/what-a-round-trip-preserves/)
 
 ## Documentation
 
-Comprehensive documentation is available to help you understand, use, and contribute to qvdjs:
-
-### For Users
-
-- **[README.md](./README.md)** (this file) - Quick start guide and API reference
-- **[QVD_FORMAT.md](./QVD_FORMAT.md)** - Complete QVD file format specification
-  - Binary structure details
-  - Symbol type encodings
-  - Bit packing algorithms
-  - Example file breakdowns
-
-### For Contributors
-
-- **[CONTRIBUTING.md](./CONTRIBUTING.md)** - How to contribute to the project
-  - Development setup
-  - Code style guidelines
-  - Commit message conventions
-  - Pull request process
-  - Bug reporting and feature requests
-
-- **[DEVELOPMENT.md](./DEVELOPMENT.md)** - Technical development guide
-  - Architecture overview
-  - Implementation details
-  - Design patterns
-  - Performance considerations
-  - Error handling strategies
-  - Debugging tips
-
-- **[ARCHITECTURE.md](./ARCHITECTURE.md)** - High-level architecture
-  - Component diagrams
-  - Class relationships
-  - Data flow visualization
-  - Design decisions and rationale
-  - Extension points
-  - Future considerations
-
-- **[docs/TESTING.md](./docs/TESTING.md)** - Comprehensive testing guide
-  - Testing philosophy and strategy
-  - How to write unit and integration tests
-  - Performance testing guidelines
-  - Coverage targets
-  - Multi-platform testing setup
-
-### Quick Links by Task
-
-| I want to...            | See...                                                                    |
-| ----------------------- | ------------------------------------------------------------------------- |
-| Use the library         | [README.md](./README.md) - Usage section                                  |
-| Understand QVD format   | [QVD_FORMAT.md](./QVD_FORMAT.md)                                          |
-| Report a bug            | [CONTRIBUTING.md](./CONTRIBUTING.md#reporting-bugs)                       |
-| Suggest a feature       | [CONTRIBUTING.md](./CONTRIBUTING.md#suggesting-features)                  |
-| Contribute code         | [CONTRIBUTING.md](./CONTRIBUTING.md) + [DEVELOPMENT.md](./DEVELOPMENT.md) |
-| Understand architecture | [ARCHITECTURE.md](./ARCHITECTURE.md)                                      |
-| Write tests             | [docs/TESTING.md](./docs/TESTING.md)                                      |
-| Debug an issue          | [DEVELOPMENT.md](./DEVELOPMENT.md#debugging-tips)                         |
-
-## Testing
-
-qvdjs has comprehensive test coverage with automated multi-platform testing. For detailed information about the testing infrastructure, including:
-
-- Test architecture and coverage breakdown
-- Multi-platform support (Windows, macOS, Linux)
-- Security testing approach
-- Self-hosted runner setup
-- Performance benchmarking
-
-See the **[Testing Documentation](./docs/README.md)** in the `docs/` directory.
-
-Quick links:
-
-- **[Testing Summary](./docs/TESTING_SUMMARY.md)** - Executive overview and quick start
-- **[Complete Design](./docs/MULTI_PLATFORM_TEST_DESIGN.md)** - Full technical specification
-
-### Running Tests
-
-```bash
-# Run all tests
-npm test
-
-# Run tests with coverage
-npm run coverage
+**[qvdjs.ptarmiganlabs.com](https://qvdjs.ptarmiganlabs.com)** is the documentation site, and it is
+where everything above is covered properly:
 
-# Run specific test file
-npm test -- __tests__/reader.test.js
-```
-
-Current test coverage: **91.7%** across 95+ tests covering unit, integration, and error handling scenarios.
+|                                                                           |                                                                   |
+| ------------------------------------------------------------------------- | ----------------------------------------------------------------- |
+| [Getting started](https://qvdjs.ptarmiganlabs.com/v0.10/getting-started/) | Install, choose an entry point, first read and write              |
+| [Guides](https://qvdjs.ptarmiganlabs.com/v0.10/guides/)                   | One page per task, runnable sample code                           |
+| [Concepts](https://qvdjs.ptarmiganlabs.com/v0.10/concepts/)               | The QVD format, symbols and duals, bit stuffing, the memory model |
+| [Reference](https://qvdjs.ptarmiganlabs.com/v0.10/reference/)             | Every class, method, option and error                             |
+| [Performance](https://qvdjs.ptarmiganlabs.com/v0.10/performance/)         | Measured baselines, and how to read them                          |
+| [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v0.10/troubleshooting/) | What each failure means, and what to do about it                  |
 
-## Contributing
+Benchmarks run weekly and publish to
+[ptarmiganlabs.github.io/qvdjs](https://ptarmiganlabs.github.io/qvdjs/).
 
-We welcome contributions! When contributing to this project, please follow the [Conventional Commits](https://www.conventionalcommits.org/) specification for commit messages. This enables automatic version management and changelog generation.
+## Contributors
 
-For detailed information about the release process, see [RELEASING.md](docs/RELEASING.md).
+- [Göran Sander](https://github.com/mountaindude) and [Ptarmigan Labs](https://github.com/ptarmiganlabs) —
+  creator of qvdjs. Added lazy loading, columnar and chunked reads, header-only metadata reads,
+  exposed QVD header metadata, ESM/CJS support, memory safety limits, security hardening,
+  multi-platform testing and the automated release process.
+- [Constantin Müller](https://mueller-constantin.de) — author of the original
+  [qvd4js](https://github.com/MuellerConstantin/qvd4js) library, from which qvdjs inherited initial
+  versions of the core read and write functions.
 
-## Contributors
+## Licence
 
-- [Göran Sander](https://github.com/mountaindude) and [Ptarmigan Labs](https://github.com/ptarmiganlabs) - Creator of qvdjs. Added features to qvd4js library (see below), including improved error handling, expose metadata from XML headers, lazy loading of symbol and index tables, ESM/CJS support, multi-platform testing, TypeScript typings, security hardening, bug fixes, automated npm release process and more
-- [Constantin Müller](https://mueller-constantin.de) - Author of the original qvd4js library, from which qvdjs inherited initial versions of the core functions for reading and writing QVD files.
+[MIT](https://opensource.org/licenses/MIT) © Ptarmigan Labs