qvdjs 0.6.2 → 0.7.0

package/README.md

@@ -1,6 +1,6 @@
 # qvdjs
 
-> Utility library for reading/writing Qlik View Data (QVD) files in JavaScript.
+> Utility library for reading/writing Qlik Sense and QlikView (QVD) files in JavaScript/Node.js
 
 ## ⚠️ Important Disclaimer
 
@@ -28,6 +28,9 @@ structure and vice versa. The library is written to be used in a Node.js environ
   - [Install](#install)
   - [Usage](#usage)
     - [Lazy Loading](#lazy-loading)
+      - [Important: Symbol Table and High-Cardinality Fields](#important-symbol-table-and-high-cardinality-fields)
+    - [Performance Optimizations](#performance-optimizations)
+    - [QVD File Size Limitations](#qvd-file-size-limitations)
     - [Progress Tracking for Large File Writes](#progress-tracking-for-large-file-writes)
     - [Working with Metadata](#working-with-metadata)
     - [Security Considerations](#security-considerations)
@@ -123,12 +126,120 @@ console.log(df.shape); // [1000, numberOfColumns]
 - The library reads only the header, symbol table, and the first N rows from the index table
 - This provides significant memory savings and faster loading times for large files
 
-This is particularly useful for:
+#### Important: Symbol Table and High-Cardinality Fields
+
+The QVD format stores data in two parts:
+
+1. **Symbol table**: Contains ALL unique values for ALL fields (must be fully loaded)
+2. **Index table**: Contains row-by-row indices into the symbol table (can be partially loaded with `maxRows`)
+
+⚠️ **Performance Impact of High-Cardinality Fields:**
+
+If your QVD file contains fields with many unique values (high cardinality), such as:
+
+- Unique IDs (OrderID, TransactionID, UUID)
+- Timestamps with millisecond precision
+- Unique text fields
+
+The symbol table can become very large and **must be read completely** even when using `maxRows`. This means:
+
+- **Small symbol table** (fields with reusable values): Fast loading regardless of file size
+  - Example: A 500MB file with only 100KB symbol table loads in ~100ms for 5000 rows ✅
+- **Large symbol table** (fields with unique values per row): Slow loading even with `maxRows`
+  - Example: A 500MB file with 266MB symbol table takes ~6 seconds for any row count ❌
+
+To check your QVD's symbol table size, look at the `NoOfSymbols` in field metadata - values close to the total row count indicate high cardinality.
+
+**When lazy loading works best:**
 
 - Previewing data from very large QVD files without loading the entire file into memory
-- Reducing memory consumption when working with large files
-- Faster loading times when you only need a subset of the data
+- Files where most fields have reusable values (low cardinality)
 - Data exploration and schema inspection of large datasets
+- Faster loading times when you only need a subset of the data
+
+### Performance Optimizations
+
+The library includes intelligent symbol table parsing that dramatically improves performance when loading partial data with `maxRows`:
+
+**Smart Symbol Loading:**
+
+When you specify `maxRows`, the library:
+
+1. Analyzes which symbols are actually needed for the requested rows
+2. Parses **only** those symbols from the symbol table
+3. Skips parsing unused symbols entirely (not just filtering after parsing)
+
+**Performance Impact:**
+
+Compared to parsing all symbols regardless of `maxRows`:
+
+- **~10x faster load times** - Loading 5,000 rows from a 20M row file: 8.3 seconds → 0.9 seconds
+- **~10x less memory usage** - Same operation: 1,616 MB → 180 MB
+- **Native or better efficiency** - Memory overhead reduced from 6.0x to 0.7x of raw data size
+
+**Real-World Example:**
+
+```javascript
+// File: orders_20m.qvd (20M rows, 266MB symbol table)
+const df = await QvdDataFrame.fromQvd('orders_20m.qvd', {maxRows: 5000});
+// Load time: ~0.9 seconds
+// Memory usage: ~180 MB
+// Only 17,228 out of millions of symbols parsed
+```
+
+**Key Benefits:**
+
+- Much faster previews of large QVD files
+- Lower memory footprint for data exploration
+- Efficient handling of files with large symbol tables
+- Automatic optimization - no configuration needed
+
+This optimization is particularly effective for files with many unique values (high cardinality) where the symbol table is large but you only need to preview a small portion of the data.
+
+### QVD File Size Limitations
+
+**Simple Explanation:**
+
+Node.js has memory limits that affect how large QVD files you can work with. By default, Node.js can use up to about 4GB of memory. This means if you try to load a very large QVD file, you might run out of memory and get an error. Think of it like trying to open a very large document on a computer with limited RAM - if the document is too big, it won't open.
+
+**What happens when files are too large:**
+
+- **When opening large files**: If a QVD file exceeds available memory, Node.js will throw an out-of-memory error (typically "JavaScript heap out of memory" or "FATAL ERROR: Reached heap limit"). The process will crash before completing the file load.
+- **When saving large files**: Writing very large QVD files can similarly exhaust memory during symbol table and index table construction, causing the same out-of-memory errors before the file is written to disk.
+- **Performance degradation**: Even before running out of memory completely, you may notice significant slowdowns, high memory usage, and system swapping as files approach memory limits.
+
+The good news is that Node.js memory limits can be increased (though there will always be some limit), and the actual file size you can handle depends on your data.
+
+The most common question at this point is usually:
+
+> "How large of a QVD file can I work with using qvdjs?"
+
+There is unfortunately no simple answer to this question, as it very much depends on the characteristics of what data is inside the QVD file. See below for more details.
+
+**Technical Details:**
+
+The maximum QVD file size you can handle with qvdjs depends on several factors and there is no single fixed limit:
+
+- **Node.js Memory Limits**: By default, Node.js limits heap memory to approximately 4GB (varies by architecture and Node.js version). You can increase this using the `--max-old-space-size` flag (e.g., `node --max-old-space-size=8192 script.js` for 8GB), but physical RAM and system architecture will ultimately constrain you.
+  To phrase it differently: There is no magic here - viewing a 40 GB QVD on a laptop with 24 GB RAM will not work.
+  That laptop may in fact struggle with QVD files larger than 4-6 GB depending on data characteristics - or happily work with 10+ GB files if the data is very friendly.
+
+- **Data Characteristics**: The actual memory consumption depends _heavily_ on what's inside your QVD:
+  - **Field Cardinality**: Files with high-cardinality fields (many unique values per field, like unique IDs or timestamps) require more memory for symbol tables. This is usually the biggest factor, at least if there are many rows too.
+  - **Number of Rows**: More rows mean larger index tables in memory
+  - **Number of Fields**: More columns increase overall memory requirements
+  - **Data Types**: String data generally uses more memory than numeric data
+
+- **Operation Type**: Reading typically uses less memory than writing, especially when using lazy loading (`maxRows` option). Writing requires building complete symbol and index tables in memory.
+
+- **Practical Guidance**:
+  - For typical business data with moderate cardinality, files up to 1-2GB usually work well with default Node.js settings
+  - High-cardinality data (unique values in most rows) may limit you to smaller files (hundreds of MB)
+  - Use lazy loading (`maxRows` option) when possible to reduce memory footprint when reading
+  - Monitor memory usage with tools like `process.memoryUsage()` for your specific use cases
+  - Consider processing large datasets in chunks or using streaming approaches if you hit memory limits. Clever things can be done by doing multiple passes over the file instead of loading everything at once.
+
+If you consistently work with very large QVD files, consider increasing Node.js memory limits or splitting your data into multiple smaller QVD files.
 
 ### Progress Tracking for Large File Writes
 
@@ -173,7 +284,7 @@ The library uses optimized algorithms for large dataset processing:
 - **Map-based lookups**: O(1) symbol index lookups instead of O(n) findIndex operations
 - **Reduced algorithmic complexity**: From O(n×m×s) to O(n×m) where n=rows, m=columns, s=symbols
 
-These optimizations can reduce write times by 80-90% for large datasets (100K+ rows).
+These optimizations can reduce write times by 80-90% for large datasets (100K+ rows), compared to earlier versions of the library.
 
 ### Working with Metadata
 
@@ -319,7 +430,11 @@ and the data types of the fields.
 
 The symbol table contains the distinct/unique values of the fields and is located directly after the XML header. The order
 of columns in the symbol table corresponds to the order of the fields in the XML header. The length and offset of the
-symbol sections of each column are also stored in the XML header. Each symbol section consist of the unique symbols of the
+symbol sections of each column are also stored in the XML header.
+
+**Important**: The offset values in the XML header are **relative to the start of the symbol table section**, not absolute file positions. For example, if a field has `Offset=1143`, this means its symbol data starts 1143 bytes after the symbol table section begins (which itself starts immediately after the XML header ends).
+
+Each symbol section consists of the unique symbols of the
 respective column. The type of a single symbol is determined by a type byte prefixed to the respective symbol value. The
 following type of symbols are supported: