qvdjs 2.0.0 → 2.0.1

package/README.md

@@ -38,7 +38,7 @@ files, and validate output in your own Qlik environment** before relying on it.
 Qlik file and writing it back reproduces its symbol table byte for byte, whichever way it was read; a
 value you change is written as Qlik stores that kind of value. Where that boundary lies is the first
 thing worth reading:
-[What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v1.0/overview/what-a-round-trip-preserves/).
+[What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v2.0/overview/what-a-round-trip-preserves/).
 
 ## Checked against Qlik Sense
 
@@ -65,7 +65,7 @@ Two differences are by design: Qlik often gives a field more bits in the index t
 where qvdjs uses the fewest, and qvdjs adds no tags or number formats of its own - Qlik works tags out
 when it loads a file. QVD is also QlikView's format, but QlikView is not part of the comparison.
 
-→ [Checked against Qlik Sense](https://qvdjs.ptarmiganlabs.com/v1.0/overview/checked-against-qlik-sense/)
+→ [Checked against Qlik Sense](https://qvdjs.ptarmiganlabs.com/v2.0/overview/checked-against-qlik-sense/)
 
 ## Four ways to open a file
 
@@ -107,7 +107,7 @@ Reaching for `fromQvd` when you wanted one of the other three is the common mist
 `fromQvd(path, {maxRows: 0})` is **not** a substitute for `readMetadata`: it loads no rows but still
 parses the whole symbol table, which grows with the data.
 
-→ [Choosing an entry point](https://qvdjs.ptarmiganlabs.com/v1.0/getting-started/)
+→ [Choosing an entry point](https://qvdjs.ptarmiganlabs.com/v2.0/getting-started/)
 
 ## What a cell holds
 
@@ -171,7 +171,7 @@ NULL. A dual value - a `QvdDual`, or an object with exactly the keys `number` an
 as a dual, which is how a number gets the text Qlik displays. As in Qlik, a field holds one text per
 number, the first written. Anything else is refused with a `QvdValidationError`.
 
-→ [Writing a QVD](https://qvdjs.ptarmiganlabs.com/v1.0/guides/)
+→ [Writing a QVD](https://qvdjs.ptarmiganlabs.com/v2.0/guides/)
 
 ## Reading part of a file
 
@@ -211,7 +211,7 @@ try {
 When it is `iterate()` that overflowed, the context carries `recommendedChunkSize` as well, so a
 caller cannot mistake one recommendation for the other.
 
-→ [Memory and file size limits](https://qvdjs.ptarmiganlabs.com/v1.0/overview/memory-and-file-size-limits/)
+→ [Memory and file size limits](https://qvdjs.ptarmiganlabs.com/v2.0/overview/memory-and-file-size-limits/)
 
 ## Metadata
 
@@ -228,7 +228,7 @@ df.setFileMetadata({tableName: 'UpdatedProducts'});
 await df.toQvd('products-v2.qvd');
 ```
 
-→ [Metadata reference](https://qvdjs.ptarmiganlabs.com/v1.0/reference/)
+→ [Metadata reference](https://qvdjs.ptarmiganlabs.com/v2.0/reference/)
 
 ## File paths are sandboxed
 
@@ -252,7 +252,7 @@ is a different file at that name. Either is a `QvdSecurityError` whose `context.
 swapped for a symlink in the moment between the check and the open — is the one gap left, so keep
 the allowed directory's own subdirectories out of untrusted hands.
 
-→ [Path security](https://qvdjs.ptarmiganlabs.com/v1.0/overview/path-security/)
+→ [Path security](https://qvdjs.ptarmiganlabs.com/v2.0/overview/path-security/)
 
 ## Errors
 
@@ -273,7 +273,7 @@ without reporting an error; `error.context.filePosition` says where the file sto
 `error.name` is always the class name, in the CommonJS and ES module builds alike, so it works across
 a module boundary where `instanceof` may not.
 
-→ [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v1.0/troubleshooting/)
+→ [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v2.0/troubleshooting/)
 
 ## Known limitations
 
@@ -286,7 +286,7 @@ Honest boundaries rather than an issue list — these are the ones that change w
 | **A killed process can leave a temporary file**               | A write builds `<name>.qvdjs-<hex>.tmp` beside the QVD and removes it however the write ends — unless the process is killed outright, or the removal is itself refused, in which case the error names the file in `context.temporaryFile`. The QVD is untouched either way, and such a file can be deleted. `{atomic: false}` writes none when it rewrites a QVD that exists, and damages the QVD instead; a QVD that does not exist yet is renamed into place in either mode. |
 | **The writer takes numbers, strings, dual values and `null`** | Anything else - a `Date`, a boolean, an array, a `QvdSymbol` - is refused with a `QvdValidationError` naming the field and the row. Convert first: a `Date` to `new QvdDual(dateToQlikSerial(date), text)`, the serial and the text Qlik shows, which is how Qlik stores a date; a boolean to `-1` and `0`, as a Qlik comparison stores it.                                                                                                                                    |
 
-→ [What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v1.0/overview/what-a-round-trip-preserves/)
+→ [What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v2.0/overview/what-a-round-trip-preserves/)
 
 ## Documentation
 
@@ -295,13 +295,13 @@ where everything above is covered properly:
 
 |                                                                                                         |                                                                   |
 | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
-| [Getting started](https://qvdjs.ptarmiganlabs.com/v1.0/getting-started/)                                | Install, choose an entry point, first read and write              |
-| [Guides](https://qvdjs.ptarmiganlabs.com/v1.0/guides/)                                                  | One page per task, runnable sample code                           |
-| [Concepts](https://qvdjs.ptarmiganlabs.com/v1.0/concepts/)                                              | The QVD format, symbols and duals, bit stuffing, the memory model |
-| [Reference](https://qvdjs.ptarmiganlabs.com/v1.0/reference/)                                            | Every class, method, option and error                             |
-| [Performance](https://qvdjs.ptarmiganlabs.com/v1.0/performance/)                                        | Measured baselines, and how to read them                          |
-| [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v1.0/troubleshooting/)                                | What each failure means, and what to do about it                  |
-| [Checked against Qlik Sense](https://qvdjs.ptarmiganlabs.com/v1.0/overview/checked-against-qlik-sense/) | How qvdjs's output is compared with Qlik Sense's                  |
+| [Getting started](https://qvdjs.ptarmiganlabs.com/v2.0/getting-started/)                                | Install, choose an entry point, first read and write              |
+| [Guides](https://qvdjs.ptarmiganlabs.com/v2.0/guides/)                                                  | One page per task, runnable sample code                           |
+| [Concepts](https://qvdjs.ptarmiganlabs.com/v2.0/concepts/)                                              | The QVD format, symbols and duals, bit stuffing, the memory model |
+| [Reference](https://qvdjs.ptarmiganlabs.com/v2.0/reference/)                                            | Every class, method, option and error                             |
+| [Performance](https://qvdjs.ptarmiganlabs.com/v2.0/performance/)                                        | Measured baselines, and how to read them                          |
+| [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v2.0/troubleshooting/)                                | What each failure means, and what to do about it                  |
+| [Checked against Qlik Sense](https://qvdjs.ptarmiganlabs.com/v2.0/overview/checked-against-qlik-sense/) | How qvdjs's output is compared with Qlik Sense's                  |
 
 Benchmarks run weekly and publish to
 [ptarmiganlabs.github.io/qvdjs](https://ptarmiganlabs.github.io/qvdjs/).

package/dist/index.cjs

@@ -1098,8 +1098,12 @@ function fieldGeometry(bitOffset, bitWidth) {
     byteCount: bitWidth === 0 ? 0 : shift + bitWidth + 7 >>> 3
   };
 }
-function decodeIndexColumn(buffer, recordSize, rowCount, bitOffset, bitWidth, bias, out) {
+function decodeIndexColumn(buffer, recordSize, rowCount, bitOffset, bitWidth, bias, out, bounds = null) {
+  const symbolCount = bounds === null ? Infinity : bounds.symbolCount;
   if (bitWidth === 0) {
+    if (bounds !== null && rowCount > 0 && bias >= symbolCount) {
+      refuse2(bounds, 0, bias, bias);
+    }
     out.fill(bias, 0, rowCount);
     return out;
   }
@@ -1113,10 +1117,25 @@ function decodeIndexColumn(buffer, recordSize, rowCount, bitOffset, bitWidth, bi
     if (byteCount > 2) acc += buffer[base + 2] * 65536;
     if (byteCount > 3) acc += buffer[base + 3] * 16777216;
     if (byteCount > 4) acc += buffer[base + 4] * 4294967296;
-    out[row] = Math.floor(acc / divisor) % modulus + bias;
+    const index = Math.floor(acc / divisor) % modulus + bias;
+    if ((index >= symbolCount || index < 0 && index !== bias) && bounds !== null) {
+      refuse2(bounds, row, index, bias);
+    }
+    out[row] = index;
   }
   return out;
 }
+function refuse2(bounds, row, index, bias) {
+  throw new exports.QvdCorruptedError("Symbol index out of range", {
+    field: bounds.field,
+    row: bounds.firstRow + row,
+    symbolIndex: index,
+    symbolCount: bounds.symbolCount,
+    bias,
+    file: bounds.file,
+    stage: "parseIndexTable"
+  });
+}
 function writeBitField(buffer, recordBase, geometry, value) {
   const { byteStart, shift, byteCount } = geometry;
   if (byteCount === 0) {
@@ -1133,10 +1152,12 @@ function writeBitField(buffer, recordBase, geometry, value) {
 var MAX_BIT_WIDTH, POW2;
 var init_bitUtils = __esm({
   "src/util/bitUtils.js"() {
+    init_QvdErrors();
     MAX_BIT_WIDTH = 31;
     POW2 = Array.from({ length: 41 }, (_, exponent) => 2 ** exponent);
     __name(fieldGeometry, "fieldGeometry");
     __name(decodeIndexColumn, "decodeIndexColumn");
+    __name(refuse2, "refuse");
     __name(writeBitField, "writeBitField");
   }
 });
@@ -2570,6 +2591,15 @@ function validateFieldBitMetadata(field, recordSize, filePath) {
       stage: "parseIndexTable"
     });
   }
+  if (bias < -2147483648 || bias + 2 ** bitWidth - 1 > 2 ** 31 - 1) {
+    throw new exports.QvdCorruptedError("Bias out of range", {
+      field: field["FieldName"],
+      bias,
+      bitWidth,
+      file: filePath,
+      stage: "parseIndexTable"
+    });
+  }
   const recordSizeInBits = recordSize * 8;
   if (bitOffset + bitWidth > recordSizeInBits) {
     throw new exports.QvdCorruptedError("Bit field extends beyond record size", {
@@ -2697,7 +2727,10 @@ function parseFieldSymbols(symbolBuffer, start, end, keep, fieldName, filePath)
   }
   return { numbers, texts };
 }
-var MAX_TEXT_BYTES;
+function countFieldSymbols(symbolBuffer, start, end, fieldName, filePath) {
+  return parseFieldSymbols(symbolBuffer, start, end, DECODE_NOTHING, fieldName, filePath).numbers.length;
+}
+var MAX_TEXT_BYTES, DECODE_NOTHING;
 var init_symbolParser = __esm({
   "src/util/symbolParser.js"() {
     init_QvdErrors();
@@ -2705,6 +2738,8 @@ var init_symbolParser = __esm({
     __name(textEnd, "textEnd");
     __name(overflow, "overflow");
     __name(parseFieldSymbols, "parseFieldSymbols");
+    DECODE_NOTHING = /* @__PURE__ */ new Set();
+    __name(countFieldSymbols, "countFieldSymbols");
   }
 });
 
@@ -3218,7 +3253,7 @@ async function* chunksFrom(handle, chunkSize, failed) {
     position += bytesRead;
   }
 }
-var MAX_HEADER_SIZE, READ_CHUNK_SIZE, ANALYSIS_SLICE_ROWS; exports.QvdFileReader = void 0;
+var MAX_HEADER_SIZE, READ_CHUNK_SIZE, ANALYSIS_SLICE_ROWS, COUNT_SYMBOLS_PAST; exports.QvdFileReader = void 0;
 var init_QvdFileReader = __esm({
   "src/QvdFileReader.js"() {
     init_QvdDataFrame();
@@ -3236,6 +3271,7 @@ var init_QvdFileReader = __esm({
     MAX_HEADER_SIZE = 16 * 1024 * 1024;
     READ_CHUNK_SIZE = 512 * 1024 * 1024;
     ANALYSIS_SLICE_ROWS = 65536;
+    COUNT_SYMBOLS_PAST = 65536;
     __name(chunksFrom, "chunksFrom");
     exports.QvdFileReader = class {
       static {
@@ -3637,6 +3673,7 @@ var init_QvdFileReader = __esm({
         }
         const headerEndIndex = headerDelimiterIndex + HEADER_DELIMITER.length;
         const headerBuffer = this._buffer.subarray(headerBeginIndex, headerEndIndex);
+        this._fieldBitMetadataValidated = false;
         this._header = await xml2__default.default.parseStringPromise(headerBuffer.toString(), { explicitArray: false });
         if (!this._header) {
           throw new exports.QvdParseError("The XML header could not be parsed.", {
@@ -3682,8 +3719,8 @@ var init_QvdFileReader = __esm({
        * @param {QvdRowWindow} window The rows of interest, as file row indices.
        * @param {string} stage Stage name for any error raised here.
        * @return {{fields: Array<any>, recordSize: number, totalRows: number, rowsToLoad: number,
-       *   indexBuffer: Buffer}} The record geometry. `indexBuffer` starts at the window's first
-       *   record, so the decoder always counts from zero.
+       *   indexBuffer: Buffer, firstRow: number}} The record geometry. `indexBuffer` starts at the
+       *   window's first record, file row `firstRow`, so the decoder always counts from zero.
        * @private
        */
       _planIndexTable(window, stage) {
@@ -3729,7 +3766,7 @@ var init_QvdFileReader = __esm({
           rowsToLoad === 0 || recordSize === 0 || Math.floor(indexBuffer.length / recordSize) >= rowsToLoad,
           `The index table holds ${Math.floor(indexBuffer.length / (recordSize || 1))} whole records but ${rowsToLoad} were validated as present.`
         );
-        return { fields, recordSize, totalRows, rowsToLoad, indexBuffer };
+        return { fields, recordSize, totalRows, rowsToLoad, indexBuffer, firstRow };
       }
       /**
        * Analyzes the index table to determine which symbols are actually needed.
@@ -3756,6 +3793,9 @@ var init_QvdFileReader = __esm({
           const bitOffset = parseInt(field["BitOffset"], 10);
           const bitWidth = parseInt(field["BitWidth"], 10);
           const bias = parseInt(field["Bias"], 10);
+          const length = parseInt(field["Length"], 10);
+          let indexLimit = Number.isSafeInteger(length) && length >= 0 ? Math.ceil(length / 2) : Infinity;
+          let counted = false;
           for (let first = 0; first < rowsToLoad; first += sliceRows) {
             const count = Math.min(sliceRows, rowsToLoad - first);
             decodeIndexColumn(
@@ -3768,15 +3808,53 @@ var init_QvdFileReader = __esm({
               column
             );
             for (let row = 0; row < count; row++) {
-              if (column[row] >= 0) {
+              if (column[row] >= 0 && column[row] < indexLimit) {
                 needed.add(column[row]);
               }
             }
+            if (!counted && needed.size > COUNT_SYMBOLS_PAST) {
+              counted = true;
+              indexLimit = Math.min(indexLimit, this._countFieldSymbols(field));
+              for (const index of needed) {
+                if (index >= indexLimit) {
+                  needed.delete(index);
+                }
+              }
+            }
           }
           this._emitProgress("symbol-analysis", position + 1, fields.length);
         });
         return symbolUsage;
       }
+      /**
+       * How many symbols a field holds, for the symbol-usage pass, which runs before the symbols are parsed.
+       *
+       * The count is `countFieldSymbols`, the parse itself told to decode nothing, so it is the count
+       * `_parseSymbolTable` will produce and `_parseIndexTable` will check against. The field's area is
+       * validated first, as `_parseSymbolTable` would, so a damaged `Offset` or `Length` is reported the
+       * same way wherever it is met.
+       *
+       * @param {any} field The field's header.
+       * @return {number} Its symbols.
+       * @throws {QvdCorruptedError} If the area is not inside the symbol table, or a symbol runs past it.
+       * @private
+       */
+      _countFieldSymbols(field) {
+        assert3__default.default(
+          this._buffer && this._symbolTableOffset && this._indexTableOffset,
+          "The QVD file has not been read before its symbols were counted."
+        );
+        const symbolBuffer = this._buffer.subarray(this._symbolTableOffset, this._indexTableOffset);
+        validateFieldMetadata(field, symbolBuffer.length, this._path);
+        const offset = parseInt(field["Offset"], 10);
+        return countFieldSymbols(
+          symbolBuffer,
+          offset,
+          offset + parseInt(field["Length"], 10),
+          field["FieldName"],
+          this._path
+        );
+      }
       /**
        * Parses the symbol table of the QVD file. This method is part of the parsing process
        * and should not be called directly.
@@ -3862,12 +3940,19 @@ var init_QvdFileReader = __esm({
        * `base += recordSize`, so decoding rows k to k+n is a question of where the buffer slice starts
        * and how many iterations run. Nothing about the decoder changed to support it.
        *
+       * Every index is checked as it is decoded. One that addresses neither a symbol of its field nor
+       * NULL throws a `QvdCorruptedError` naming the file row (#125). Here and not where rows or columns
+       * are built, because every read decodes through this method and a columnar read hands its codes
+       * straight to the caller. Rows outside the window are not decoded, so they are not checked.
+       *
        * @param {QvdRowWindow} window The rows to decode.
+       * @throws {QvdCorruptedError} If an index in the window addresses neither a symbol nor NULL.
        */
       async _parseIndexTable(window) {
-        const { fields, recordSize, rowsToLoad, indexBuffer } = this._planIndexTable(window, "parseIndexTable");
-        this._rowsDecoded = rowsToLoad;
-        this._indexColumns = fields.map((field, position) => {
+        const { fields, recordSize, rowsToLoad, indexBuffer, firstRow } = this._planIndexTable(window, "parseIndexTable");
+        assert3__default.default(this._symbolTable, "The QVD file symbol table has not been parsed.");
+        const symbolTable = this._symbolTable;
+        const columns = fields.map((field, position) => {
           this._throwIfAborted();
           const column = decodeIndexColumn(
             indexBuffer,
@@ -3876,11 +3961,14 @@ var init_QvdFileReader = __esm({
             parseInt(field["BitOffset"], 10),
             parseInt(field["BitWidth"], 10),
             parseInt(field["Bias"], 10),
-            new Int32Array(rowsToLoad)
+            new Int32Array(rowsToLoad),
+            { symbolCount: symbolTable[position].numbers.length, field: field["FieldName"], file: this._path, firstRow }
           );
           this._emitProgress("index-table", position + 1, fields.length);
           return column;
         });
+        this._indexColumns = columns;
+        this._rowsDecoded = rowsToLoad;
       }
       /**
        * Reads the file's schema and header metadata, without touching the symbol or index tables.
@@ -4009,7 +4097,8 @@ var init_QvdFileReader = __esm({
        * `chunkSize * 2`, and why the heap it needs is twice what one chunk suggests.
        *
        * A window covering no rows yields nothing at all, rather than one empty frame - so
-       * `for await` over an exhausted offset does nothing, which is what a paging loop wants.
+       * `for await` over an exhausted offset does nothing, which is what a paging loop wants. Its header
+       * is still checked, as every read of a file's rows checks it.
        *
        * @param {number|null|{offset?: number, limit?: number|null, maxRows?: number|null}} window
        *   The rows to cover, in the same spellings `load()` accepts.
@@ -4027,6 +4116,10 @@ var init_QvdFileReader = __esm({
         const liveRows = { rows: chunkSize * 2, perChunk: 2 };
         const rows = normaliseWindow(window, this._path);
         const prepared = await this._prepare(rows, liveRows);
+        if (prepared.rowsAvailable === 0) {
+          this._planIndexTable({ offset: prepared.offset, limit: 0 }, "parseIndexTable");
+          return;
+        }
         for (let done = 0; done < prepared.rowsAvailable; done += chunkSize) {
           this._throwIfAborted();
           const count = Math.min(chunkSize, prepared.rowsAvailable - done);