qvdjs 0.6.2 → 0.8.0

package/README.md

@@ -1,6 +1,6 @@
 # qvdjs
 
-> Utility library for reading/writing Qlik View Data (QVD) files in JavaScript.
+> Utility library for reading/writing Qlik Sense and QlikView (QVD) files in JavaScript/Node.js
 
 ## ⚠️ Important Disclaimer
 
@@ -28,6 +28,10 @@ structure and vice versa. The library is written to be used in a Node.js environ
   - [Install](#install)
   - [Usage](#usage)
     - [Lazy Loading](#lazy-loading)
+      - [Important: Symbol Table and High-Cardinality Fields](#important-symbol-table-and-high-cardinality-fields)
+    - [Performance Optimizations](#performance-optimizations)
+    - [QVD File Size Limitations](#qvd-file-size-limitations)
+      - [Why Safety Limits Exist](#why-safety-limits-exist)
     - [Progress Tracking for Large File Writes](#progress-tracking-for-large-file-writes)
     - [Working with Metadata](#working-with-metadata)
     - [Security Considerations](#security-considerations)
@@ -123,12 +127,162 @@ console.log(df.shape); // [1000, numberOfColumns]
 - The library reads only the header, symbol table, and the first N rows from the index table
 - This provides significant memory savings and faster loading times for large files
 
-This is particularly useful for:
+#### Important: Symbol Table and High-Cardinality Fields
+
+The QVD format stores data in two parts:
+
+1. **Symbol table**: Contains ALL unique values for ALL fields (must be fully loaded)
+2. **Index table**: Contains row-by-row indices into the symbol table (can be partially loaded with `maxRows`)
+
+⚠️ **Performance Impact of High-Cardinality Fields:**
+
+If your QVD file contains fields with many unique values (high cardinality), such as:
+
+- Unique IDs (OrderID, TransactionID, UUID)
+- Timestamps with millisecond precision
+- Unique text fields
+
+The symbol table can become very large and **must be read completely** even when using `maxRows`. This means:
+
+- **Small symbol table** (fields with reusable values): Fast loading regardless of file size
+  - Example: A 500MB file with only 100KB symbol table loads in ~100ms for 5000 rows ✅
+- **Large symbol table** (fields with unique values per row): Slow loading even with `maxRows`
+  - Example: A 500MB file with 266MB symbol table takes ~6 seconds for any row count ❌
+
+To check your QVD's symbol table size, look at the `NoOfSymbols` in field metadata - values close to the total row count indicate high cardinality.
+
+**When lazy loading works best:**
 
 - Previewing data from very large QVD files without loading the entire file into memory
-- Reducing memory consumption when working with large files
-- Faster loading times when you only need a subset of the data
+- Files where most fields have reusable values (low cardinality)
 - Data exploration and schema inspection of large datasets
+- Faster loading times when you only need a subset of the data
+
+### Performance Optimizations
+
+The library includes intelligent symbol table parsing that dramatically improves performance when loading partial data with `maxRows`:
+
+**Smart Symbol Loading:**
+
+When you specify `maxRows`, the library:
+
+1. Analyzes which symbols are actually needed for the requested rows
+2. Parses **only** those symbols from the symbol table
+3. Skips parsing unused symbols entirely (not just filtering after parsing)
+
+**Performance Impact:**
+
+Compared to parsing all symbols regardless of `maxRows`:
+
+- **~10x faster load times** - Loading 5,000 rows from a 20M row file: 8.3 seconds → 0.9 seconds
+- **~10x less memory usage** - Same operation: 1,616 MB → 180 MB
+- **Native or better efficiency** - Memory overhead reduced from 6.0x to 0.7x of raw data size
+
+**Real-World Example:**
+
+```javascript
+// File: orders_20m.qvd (20M rows, 266MB symbol table)
+const df = await QvdDataFrame.fromQvd('orders_20m.qvd', {maxRows: 5000});
+// Load time: ~0.9 seconds
+// Memory usage: ~180 MB
+// Only 17,228 out of millions of symbols parsed
+```
+
+**Key Benefits:**
+
+- Much faster previews of large QVD files
+- Lower memory footprint for data exploration
+- Efficient handling of files with large symbol tables
+- Automatic optimization - no configuration needed
+
+This optimization is particularly effective for files with many unique values (high cardinality) where the symbol table is large but you only need to preview a small portion of the data.
+
+### QVD File Size Limitations
+
+**Simple Explanation:**
+
+Node.js has memory limits that affect how large QVD files you can work with. By default, Node.js can use up to about 4GB of memory. This means if you try to load a very large QVD file, you might run out of memory and get an error. Think of it like trying to open a very large document on a computer with limited RAM - if the document is too big, it won't open.
+
+**What happens when files are too large:**
+
+- **When opening large files**: If a QVD file exceeds available memory, Node.js will throw an out-of-memory error (typically "JavaScript heap out of memory" or "FATAL ERROR: Reached heap limit"). The process will crash before completing the file load.
+- **When saving large files**: Writing very large QVD files can similarly exhaust memory during symbol table and index table construction, causing the same out-of-memory errors before the file is written to disk.
+- **Performance degradation**: Even before running out of memory completely, you may notice significant slowdowns, high memory usage, and system swapping as files approach memory limits.
+
+The good news is that Node.js memory limits can be increased (though there will always be some limit), and the actual file size you can handle depends on your data.
+
+The most common question at this point is usually:
+
+> "How large of a QVD file can I work with using qvdjs?"
+
+There is unfortunately no simple answer to this question, as it very much depends on the characteristics of what data is inside the QVD file. See below for more details.
+
+**Technical Details:**
+
+The maximum QVD file size you can handle with qvdjs depends on several factors and there is no single fixed limit:
+
+- **Node.js Memory Limits**: By default, Node.js limits heap memory to approximately 4GB (varies by architecture and Node.js version). The library **automatically detects your configured heap size** and scales its safety limits accordingly. You can increase heap size using the `--max-old-space-size` flag (e.g., `node --max-old-space-size=16384 script.js` for 16GB), and qvdjs will automatically allow larger files.
+
+  For larger heap configurations, you can also adjust the `memorySafetyFactor` option (default 0.3 = 30%) to make more efficient use of available memory:
+
+  ```javascript
+  const df = await QvdDataFrame.fromQvd('large-file.qvd', {
+    memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
+  });
+  ```
+
+  To phrase it differently: There is no magic here - viewing a 40 GB QVD on a laptop with 24 GB RAM will not work. That laptop may in fact struggle with QVD files larger than 4-6 GB depending on data characteristics - or happily work with 10+ GB files if the data is very friendly.
+
+- **Data Characteristics**: The actual memory consumption depends _heavily_ on what's inside your QVD:
+  - **Field Cardinality**: Files with high-cardinality fields (many unique values per field, like unique IDs or timestamps) require more memory for symbol tables. This is usually the biggest factor, at least if there are many rows too.
+  - **Number of Rows**: More rows mean larger index tables in memory
+  - **Number of Fields**: More columns increase overall memory requirements
+  - **Data Types**: String data generally uses more memory than numeric data
+
+- **Operation Type**: Reading typically uses less memory than writing, especially when using lazy loading (`maxRows` option). Writing requires building complete symbol and index tables in memory.
+
+- **Practical Guidance**:
+  - For typical business data with moderate cardinality, files up to 1-2GB usually work well with default Node.js settings
+  - High-cardinality data (unique values in most rows) may limit you to smaller files (hundreds of MB)
+  - **With increased heap** (e.g., 16GB+), you can handle proportionally larger files by adjusting `memorySafetyFactor`
+  - Use lazy loading (`maxRows` option) when possible to reduce memory footprint when reading
+  - Monitor memory usage with tools like `process.memoryUsage()` for your specific use cases
+  - Consider processing large datasets in chunks or using streaming approaches if you hit memory limits. Clever things can be done by doing multiple passes over the file instead of loading everything at once.
+
+If you consistently work with very large QVD files, consider increasing Node.js memory limits (with matching `memorySafetyFactor` adjustment) or splitting your data into multiple smaller QVD files.
+
+#### Why Safety Limits Exist
+
+The library implements **dynamic safety limits** to prevent catastrophic crashes. Without these limits:
+
+- **Your application will crash hard** - Node.js terminates with `FATAL ERROR: Reached heap limit` when attempting to load files that are too large
+- **No error handling is possible** - JavaScript try-catch blocks cannot intercept out-of-memory (OOM) crashes at the V8 engine level
+- **The entire process dies** - Not just the QVD operation, but your entire application terminates ungracefully
+
+The safety limits **prevent these crashes** by checking available memory _before_ attempting to load files, throwing graceful `QvdValidationError` exceptions that you can catch and handle. While the multi-tier safety system may seem complex, it ensures your application stays running and provides helpful error messages with recommendations (like using `maxRows` parameter or increasing heap size) instead of cryptic fatal errors.
+
+**Example without safety limits:**
+
+```javascript
+// Process crashes with no chance to recover
+const df = await QvdDataFrame.fromQvd('huge-file.qvd'); // 💥 FATAL ERROR
+// Your application is now terminated
+```
+
+**Example with safety limits:**
+
+```javascript
+try {
+  const df = await QvdDataFrame.fromQvd('huge-file.qvd');
+} catch (error) {
+  if (error.name === 'QvdValidationError') {
+    console.log('File too large, trying with maxRows:', error.context.recommendedMaxRows);
+    // ✅ Your application continues running
+  }
+}
+```
+
+For more technical details about the memory safety system, including specific thresholds and the four-tier protection model, see [docs/DYNAMIC_SAFETY_LIMITS.md](docs/DYNAMIC_SAFETY_LIMITS.md). For practical examples, see [docs/examples/heap-scaling-example.md](docs/examples/heap-scaling-example.md).
 
 ### Progress Tracking for Large File Writes
 
@@ -173,7 +327,7 @@ The library uses optimized algorithms for large dataset processing:
 - **Map-based lookups**: O(1) symbol index lookups instead of O(n) findIndex operations
 - **Reduced algorithmic complexity**: From O(n×m×s) to O(n×m) where n=rows, m=columns, s=symbols
 
-These optimizations can reduce write times by 80-90% for large datasets (100K+ rows).
+These optimizations can reduce write times by 80-90% for large datasets (100K+ rows), compared to earlier versions of the library.
 
 ### Working with Metadata
 
@@ -319,7 +473,11 @@ and the data types of the fields.
 
 The symbol table contains the distinct/unique values of the fields and is located directly after the XML header. The order
 of columns in the symbol table corresponds to the order of the fields in the XML header. The length and offset of the
-symbol sections of each column are also stored in the XML header. Each symbol section consist of the unique symbols of the
+symbol sections of each column are also stored in the XML header.
+
+**Important**: The offset values in the XML header are **relative to the start of the symbol table section**, not absolute file positions. For example, if a field has `Offset=1143`, this means its symbol data starts 1143 bytes after the symbol table section begins (which itself starts immediately after the XML header ends).
+
+Each symbol section consists of the unique symbols of the
 respective column. The type of a single symbol is determined by a type byte prefixed to the respective symbol value. The
 following type of symbols are supported:
 
@@ -428,6 +586,7 @@ to a `QvdDataFrame` instance.
 - `options` (object, optional): Loading options
   - `maxRows` (number, optional): Maximum number of rows to load. If not specified, all rows are loaded. This is useful for loading only a subset of data from large QVD files to improve performance and reduce memory usage.
   - `allowedDir` (string, optional): Base directory for file access validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
+  - `memorySafetyFactor` (number, optional): Memory safety factor (0.0-1.0). Default is 0.3 (30%). Determines what percentage of available memory (or V8 heap limit, whichever is smaller) can be used. Increase this (e.g., to 0.5 or 0.7) when running with larger heap sizes via `--max-old-space-size` to allow processing of proportionally larger files.
 
 **Example:**
 
@@ -442,6 +601,11 @@ const dfLazy = await QvdDataFrame.fromQvd('path/to/file.qvd', {maxRows: 1000});
 const dfSecure = await QvdDataFrame.fromQvd('reports/sales.qvd', {
   allowedDir: '/var/data/qvd-files',
 });
+
+// Load with increased memory usage for large heap configurations
+const dfLarge = await QvdDataFrame.fromQvd('large-file.qvd', {
+  memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
+});
 ```
 
 #### `static fromDict(dict: object): Promise<QvdDataFrame>`