qvdjs 0.7.0 → 0.9.0

package/README.md

@@ -31,6 +31,7 @@ structure and vice versa. The library is written to be used in a Node.js environ
       - [Important: Symbol Table and High-Cardinality Fields](#important-symbol-table-and-high-cardinality-fields)
     - [Performance Optimizations](#performance-optimizations)
     - [QVD File Size Limitations](#qvd-file-size-limitations)
+      - [Why Safety Limits Exist](#why-safety-limits-exist)
     - [Progress Tracking for Large File Writes](#progress-tracking-for-large-file-writes)
     - [Working with Metadata](#working-with-metadata)
     - [Security Considerations](#security-considerations)
@@ -220,9 +221,17 @@ There is unfortunately no simple answer to this question, as it very much depend
 
 The maximum QVD file size you can handle with qvdjs depends on several factors and there is no single fixed limit:
 
-- **Node.js Memory Limits**: By default, Node.js limits heap memory to approximately 4GB (varies by architecture and Node.js version). You can increase this using the `--max-old-space-size` flag (e.g., `node --max-old-space-size=8192 script.js` for 8GB), but physical RAM and system architecture will ultimately constrain you.
-  To phrase it differently: There is no magic here - viewing a 40 GB QVD on a laptop with 24 GB RAM will not work.
-  That laptop may in fact struggle with QVD files larger than 4-6 GB depending on data characteristics - or happily work with 10+ GB files if the data is very friendly.
+- **Node.js Memory Limits**: By default, Node.js limits heap memory to approximately 4GB (varies by architecture and Node.js version). The library **automatically detects your configured heap size** and scales its safety limits accordingly. You can increase heap size using the `--max-old-space-size` flag (e.g., `node --max-old-space-size=16384 script.js` for 16GB), and qvdjs will automatically allow larger files.
+
+  For larger heap configurations, you can also adjust the `memorySafetyFactor` option (default 0.3 = 30%) to make more efficient use of available memory:
+
+  ```javascript
+  const df = await QvdDataFrame.fromQvd('large-file.qvd', {
+    memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
+  });
+  ```
+
+  To phrase it differently: There is no magic here - viewing a 40 GB QVD on a laptop with 24 GB RAM will not work. That laptop may in fact struggle with QVD files larger than 4-6 GB depending on data characteristics - or happily work with 10+ GB files if the data is very friendly.
 
 - **Data Characteristics**: The actual memory consumption depends _heavily_ on what's inside your QVD:
   - **Field Cardinality**: Files with high-cardinality fields (many unique values per field, like unique IDs or timestamps) require more memory for symbol tables. This is usually the biggest factor, at least if there are many rows too.
@@ -235,11 +244,45 @@ The maximum QVD file size you can handle with qvdjs depends on several factors a
 - **Practical Guidance**:
   - For typical business data with moderate cardinality, files up to 1-2GB usually work well with default Node.js settings
   - High-cardinality data (unique values in most rows) may limit you to smaller files (hundreds of MB)
+  - **With increased heap** (e.g., 16GB+), you can handle proportionally larger files by adjusting `memorySafetyFactor`
   - Use lazy loading (`maxRows` option) when possible to reduce memory footprint when reading
   - Monitor memory usage with tools like `process.memoryUsage()` for your specific use cases
   - Consider processing large datasets in chunks or using streaming approaches if you hit memory limits. Clever things can be done by doing multiple passes over the file instead of loading everything at once.
 
-If you consistently work with very large QVD files, consider increasing Node.js memory limits or splitting your data into multiple smaller QVD files.
+If you consistently work with very large QVD files, consider increasing Node.js memory limits (with matching `memorySafetyFactor` adjustment) or splitting your data into multiple smaller QVD files.
+
+#### Why Safety Limits Exist
+
+The library implements **dynamic safety limits** to prevent catastrophic crashes. Without these limits:
+
+- **Your application will crash hard** - Node.js terminates with `FATAL ERROR: Reached heap limit` when attempting to load files that are too large
+- **No error handling is possible** - JavaScript try-catch blocks cannot intercept out-of-memory (OOM) crashes at the V8 engine level
+- **The entire process dies** - Not just the QVD operation, but your entire application terminates ungracefully
+
+The safety limits **prevent these crashes** by checking available memory _before_ attempting to load files, throwing graceful `QvdValidationError` exceptions that you can catch and handle. While the multi-tier safety system may seem complex, it ensures your application stays running and provides helpful error messages with recommendations (like using `maxRows` parameter or increasing heap size) instead of cryptic fatal errors.
+
+**Example without safety limits:**
+
+```javascript
+// Process crashes with no chance to recover
+const df = await QvdDataFrame.fromQvd('huge-file.qvd'); // 💥 FATAL ERROR
+// Your application is now terminated
+```
+
+**Example with safety limits:**
+
+```javascript
+try {
+  const df = await QvdDataFrame.fromQvd('huge-file.qvd');
+} catch (error) {
+  if (error.name === 'QvdValidationError') {
+    console.log('File too large, trying with maxRows:', error.context.recommendedMaxRows);
+    // ✅ Your application continues running
+  }
+}
+```
+
+For more technical details about the memory safety system, including specific thresholds and the four-tier protection model, see [docs/DYNAMIC_SAFETY_LIMITS.md](docs/DYNAMIC_SAFETY_LIMITS.md). For practical examples, see [docs/examples/heap-scaling-example.md](docs/examples/heap-scaling-example.md).
 
 ### Progress Tracking for Large File Writes
 
@@ -543,6 +586,7 @@ to a `QvdDataFrame` instance.
 - `options` (object, optional): Loading options
   - `maxRows` (number, optional): Maximum number of rows to load. If not specified, all rows are loaded. This is useful for loading only a subset of data from large QVD files to improve performance and reduce memory usage.
   - `allowedDir` (string, optional): Base directory for file access validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
+  - `memorySafetyFactor` (number, optional): Memory safety factor (0.0-1.0). Default is 0.3 (30%). Determines what percentage of available memory (or V8 heap limit, whichever is smaller) can be used. Increase this (e.g., to 0.5 or 0.7) when running with larger heap sizes via `--max-old-space-size` to allow processing of proportionally larger files.
 
 **Example:**
 
@@ -557,6 +601,11 @@ const dfLazy = await QvdDataFrame.fromQvd('path/to/file.qvd', {maxRows: 1000});
 const dfSecure = await QvdDataFrame.fromQvd('reports/sales.qvd', {
   allowedDir: '/var/data/qvd-files',
 });
+
+// Load with increased memory usage for large heap configurations
+const dfLarge = await QvdDataFrame.fromQvd('large-file.qvd', {
+  memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
+});
 ```
 
 #### `static fromDict(dict: object): Promise<QvdDataFrame>`

package/dist/index.cjs

@@ -6,6 +6,7 @@ var crypto = require('crypto');
 var xml2 = require('xml2js');
 var assert = require('assert');
 var os = require('os');
+var v8 = require('v8');
 
 function _interopDefault (e) { return e && e.__esModule ? e : { default: e }; }
 
@@ -15,6 +16,7 @@ var crypto__default = /*#__PURE__*/_interopDefault(crypto);
 var xml2__default = /*#__PURE__*/_interopDefault(xml2);
 var assert__default = /*#__PURE__*/_interopDefault(assert);
 var os__default = /*#__PURE__*/_interopDefault(os);
+var v8__default = /*#__PURE__*/_interopDefault(v8);
 
 var __defProp = Object.defineProperty;
 var __getOwnPropNames = Object.getOwnPropertyNames;
@@ -658,6 +660,9 @@ var init_bitUtils = __esm({
   "src/util/bitUtils.js"() {
   }
 });
+function getHeapLimit() {
+  return v8__default.default.getHeapStatistics().heap_size_limit;
+}
 function estimateMemoryUsage(symbolTableSize, maxRows, totalRows) {
   const FULL_PARSE_OVERHEAD = 6;
   const MINIMAL_OVERHEAD = 0.01;
@@ -670,11 +675,15 @@ function estimateMemoryUsage(symbolTableSize, maxRows, totalRows) {
   const skippedSymbolsMemory = symbolTableSize * (1 - symbolPercentage) * MINIMAL_OVERHEAD;
   return keptSymbolsMemory + skippedSymbolsMemory;
 }
-function validateMemoryAvailability(symbolTableSize, maxRows, totalRows, filePath) {
+function validateMemoryAvailability(symbolTableSize, maxRows, totalRows, filePath, safetyFactor = 0.3) {
+  if (typeof safetyFactor !== "number" || safetyFactor < 0 || safetyFactor > 1) {
+    throw new exports.QvdValidationError("safetyFactor must be a number between 0.0 and 1.0", { safetyFactor });
+  }
   const availableMemory = os__default.default.freemem();
+  const heapLimit = getHeapLimit();
   const estimatedMemory = estimateMemoryUsage(symbolTableSize, maxRows, totalRows);
-  const MEMORY_SAFETY_FACTOR = 0.3;
-  const maxAllowedMemory = availableMemory * MEMORY_SAFETY_FACTOR;
+  const effectiveLimit = Math.min(availableMemory, heapLimit);
+  const maxAllowedMemory = effectiveLimit * safetyFactor;
   if (estimatedMemory > maxAllowedMemory) {
     const safeSymbolPercentage = maxAllowedMemory / (symbolTableSize * 6);
     const safeRowPercentage = Math.pow(safeSymbolPercentage, 2);
@@ -682,14 +691,20 @@ function validateMemoryAvailability(symbolTableSize, maxRows, totalRows, filePat
     const sizeMB = Math.round(symbolTableSize / 1024 / 1024);
     const estimatedMB = Math.round(estimatedMemory / 1024 / 1024);
     const availableMB = Math.round(maxAllowedMemory / 1024 / 1024);
+    const heapLimitMB = Math.round(heapLimit / 1024 / 1024);
+    const availableRamMB = Math.round(availableMemory / 1024 / 1024);
+    const limitingFactor = heapLimit < availableMemory ? "V8 heap limit" : "available RAM";
     throw new exports.QvdValidationError(
-      `Insufficient memory to load file safely. Symbol table: ${sizeMB}MB, Estimated memory needed: ${estimatedMB}MB, Available: ${availableMB}MB. Try loading fewer rows using the maxRows parameter (recommended: ${recommendedMaxRows.toLocaleString()} rows or less).`,
+      `Insufficient memory to load file safely. Symbol table: ${sizeMB}MB, Estimated memory needed: ${estimatedMB}MB, Available: ${availableMB}MB (limited by ${limitingFactor}: ${heapLimitMB}MB heap / ${availableRamMB}MB RAM). Try loading fewer rows using the maxRows parameter (recommended: ${recommendedMaxRows.toLocaleString()} rows or less).`,
       {
         file: filePath,
         symbolTableSize,
         symbolTableSizeMB: sizeMB,
         estimatedMemoryMB: estimatedMB,
         availableMemoryMB: availableMB,
+        heapLimitMB,
+        availableRamMB,
+        limitingFactor,
         totalRows,
         maxRows,
         recommendedMaxRows
@@ -698,13 +713,15 @@ function validateMemoryAvailability(symbolTableSize, maxRows, totalRows, filePat
   }
 }
 function warnLargeSymbolTable(symbolTableSize, maxRows, totalRows) {
-  const LARGE_SYMBOL_TABLE_WARNING = 500 * 1024 * 1024;
+  const heapLimit = getHeapLimit();
+  const LARGE_SYMBOL_TABLE_WARNING = heapLimit * 0.125;
   if (symbolTableSize > LARGE_SYMBOL_TABLE_WARNING && (maxRows === null || maxRows >= totalRows)) {
     const sizeMB = Math.round(symbolTableSize / 1024 / 1024);
     const estimatedMemory = estimateMemoryUsage(symbolTableSize, maxRows, totalRows);
     const estimatedMB = Math.round(estimatedMemory / 1024 / 1024);
+    const warnMB = Math.round(LARGE_SYMBOL_TABLE_WARNING / 1024 / 1024);
     console.warn(
-      `\u26A0\uFE0F  Large symbol table detected (${sizeMB}MB). Loading all ${totalRows.toLocaleString()} rows will use ~${estimatedMB}MB RAM. Consider using the maxRows parameter for better performance and lower memory usage.`
+      `\u26A0\uFE0F  Large symbol table detected (${sizeMB}MB > ${warnMB}MB threshold). Loading all ${totalRows.toLocaleString()} rows will use ~${estimatedMB}MB RAM. Consider using the maxRows parameter for better performance and lower memory usage.`
     );
   }
 }
@@ -716,35 +733,43 @@ var init_memoryUtils = __esm({
 
 // src/util/validationUtils.js
 function validateSymbolTableSizeEarly(symbolTableLength, filePath) {
-  const MAX_SYMBOL_TABLE_SIZE = 500 * 1024 * 1024;
+  const heapLimit = getHeapLimit();
+  const MAX_SYMBOL_TABLE_SIZE = heapLimit * 0.125;
   if (symbolTableLength > MAX_SYMBOL_TABLE_SIZE) {
     const sizeMB = Math.round(symbolTableLength / 1024 / 1024);
     const maxMB = Math.round(MAX_SYMBOL_TABLE_SIZE / 1024 / 1024);
+    const heapMB = Math.round(heapLimit / 1024 / 1024);
     throw new exports.QvdValidationError(
-      `Symbol table too large (${sizeMB}MB exceeds ${maxMB}MB limit). This QVD file contains extremely high-cardinality fields. Even with symbol filtering, this file cannot be loaded efficiently. Consider: (1) using a subset of the data, (2) aggregating high-cardinality fields, or (3) using a different data format.`,
+      `Symbol table too large (${sizeMB}MB exceeds ${maxMB}MB limit for lazy loading). This QVD file contains extremely high-cardinality fields. Limit scales with heap size (current: ${heapMB}MB, limit: 12.5% = ${maxMB}MB). Consider: (1) loading the full file without maxRows, (2) increasing heap size with --max-old-space-size, or (3) aggregating high-cardinality fields.`,
       {
         file: filePath,
         symbolTableSize: symbolTableLength,
         symbolTableSizeMB: sizeMB,
         maxAllowed: MAX_SYMBOL_TABLE_SIZE,
-        maxAllowedMB: maxMB
+        maxAllowedMB: maxMB,
+        heapLimitMB: heapMB,
+        limitPercentage: 12.5
       }
     );
   }
 }
 function validateSymbolTableSize(symbolTableLength, filePath, totalRows) {
-  const ABSOLUTE_MAX_SYMBOL_TABLE = 2 * 1024 * 1024 * 1024;
+  const heapLimit = getHeapLimit();
+  const ABSOLUTE_MAX_SYMBOL_TABLE = heapLimit * 0.5;
   if (symbolTableLength > ABSOLUTE_MAX_SYMBOL_TABLE) {
     const sizeMB = Math.round(symbolTableLength / 1024 / 1024);
     const maxMB = Math.round(ABSOLUTE_MAX_SYMBOL_TABLE / 1024 / 1024);
+    const heapMB = Math.round(heapLimit / 1024 / 1024);
     throw new exports.QvdValidationError(
-      `Symbol table exceeds absolute maximum size (${sizeMB}MB > ${maxMB}MB). This QVD file has pathological cardinality (likely a data modeling issue). Even with Phase 2.5 optimization, files this large cannot be processed safely. Consider: (1) aggregating high-cardinality fields, (2) splitting the data, or (3) using a different format.`,
+      `Symbol table exceeds absolute maximum size (${sizeMB}MB > ${maxMB}MB). This QVD file has pathological cardinality (likely a data modeling issue). Limit scales with heap size (current: ${heapMB}MB, limit: 50% = ${maxMB}MB). Consider: (1) increasing heap size with --max-old-space-size, (2) aggregating high-cardinality fields, (3) splitting the data, or (4) using a different format.`,
       {
         file: filePath,
         symbolTableSize: symbolTableLength,
         symbolTableSizeMB: sizeMB,
         maxAllowed: ABSOLUTE_MAX_SYMBOL_TABLE,
         maxAllowedMB: maxMB,
+        heapLimitMB: heapMB,
+        limitPercentage: 50,
         totalRows
       }
     );
@@ -881,6 +906,7 @@ function validateFieldBitMetadata(field, recordSize, filePath) {
 var init_validationUtils = __esm({
   "src/util/validationUtils.js"() {
     init_QvdErrors();
+    init_memoryUtils();
   }
 });
 
@@ -1163,10 +1189,14 @@ var init_QvdFileReader = __esm({
        * @param {Object} [options={}] Options for the reader.
        * @param {string} [options.allowedDir] Optional allowed directory path. If provided, the file
        *   path must be within this directory. Defaults to current working directory.
+       * @param {number} [options.memorySafetyFactor=0.3] Memory safety factor (0.0-1.0). Determines
+       *   what percentage of available memory (or V8 heap limit, whichever is smaller) can be used
+       *   for loading QVD files. Default is 0.3 (30%). Increase for larger heap configurations.
        */
       constructor(filePath, options = {}) {
-        const { allowedDir } = options;
+        const { allowedDir, memorySafetyFactor = 0.3 } = options;
         this._path = validatePath(filePath, allowedDir);
+        this._memorySafetyFactor = memorySafetyFactor;
         this._buffer = null;
         this._headerOffset = null;
         this._symbolTableOffset = null;
@@ -1390,7 +1420,7 @@ var init_QvdFileReader = __esm({
         const symbolTableSize = symbolBuffer.length;
         const totalRows = parseInt(this._header["QvdTableHeader"]["NoOfRecords"], 10);
         validateSymbolTableSize(symbolTableSize, this._path, totalRows);
-        validateMemoryAvailability(symbolTableSize, maxRows, totalRows, this._path);
+        validateMemoryAvailability(symbolTableSize, maxRows, totalRows, this._path, this._memorySafetyFactor);
         warnLargeSymbolTable(symbolTableSize, maxRows, totalRows);
         if (!Array.isArray(fields)) {
           fields = [fields];
@@ -1971,11 +2001,17 @@ var init_QvdDataFrame = __esm({
        * @param {Object} [options] Optional loading options.
        * @param {number|null} [options.maxRows] The maximum number of rows to load. If not specified, all rows are loaded.
        * @param {string} [options.allowedDir] Optional allowed directory path. If provided, the file path must be within this directory.
+       * @param {number} [options.memorySafetyFactor=0.3] Memory safety factor (0.0-1.0). Determines what percentage
+       *   of available memory (or V8 heap limit, whichever is smaller) can be used. Default is 0.3 (30%).
+       *   Increase this (e.g., to 0.5 or 0.7) when running with larger heap sizes via --max-old-space-size.
        * @return {Promise<QvdDataFrame>} The data frame of the QVD file.
        */
       static async fromQvd(path3, options = {}) {
         const { QvdFileReader: QvdFileReader2 } = await Promise.resolve().then(() => (init_QvdFileReader(), QvdFileReader_exports));
-        const readerOptions = { allowedDir: options.allowedDir };
+        const readerOptions = {
+          allowedDir: options.allowedDir,
+          memorySafetyFactor: options.memorySafetyFactor
+        };
         return await new QvdFileReader2(path3, readerOptions).load(options.maxRows !== void 0 ? options.maxRows : null);
       }
       /**