qvdjs 2.0.5 → 2.1.0

package/README.md

@@ -80,6 +80,7 @@ holds.
 | Rows from a file too large to hold       | `QvdDataFrame.iterate(path, {chunkSize})` | Everything, but holds two chunks — a 96 MB heap against 512 MB       |
 | A few columns of a large file            | `QvdColumnTable.fromQvd(path)`            | Everything, but stops before building rows — 141 MiB against 385 MiB |
 | Only the schema: names, row count, types | `QvdDataFrame.readMetadata(path)`         | The header alone. Constant cost, whatever the file's size            |
+| To know whether a read will fit first    | `QvdDataFrame.checkRead(path, options)`   | The header alone. Answers rather than reads; a read it approves is not refused for memory |
 
 ```javascript
 import {QvdDataFrame, QvdColumnTable} from 'qvdjs';
@@ -87,6 +88,9 @@ import {QvdDataFrame, QvdColumnTable} from 'qvdjs';
 // What is in this file? Costs the same whether it is 20 KB or 20 GB.
 const {columns, rowCount} = await QvdDataFrame.readMetadata('sales.qvd');
 
+// Will reading it fit? Also the header alone, and every suggestion it gives has been checked.
+const {fits, suggestions} = await QvdDataFrame.checkRead('sales.qvd');
+
 // Sum one column without ever building a row.
 const table = await QvdColumnTable.fromQvd('sales.qvd');
 let total = 0;
@@ -191,12 +195,13 @@ console.log(df.loadStats);
 // { symbolTableBytes, totalRows, rowsLoaded, offset, symbolFiltering, symbolsKept }
 ```
 
-**Above 2 GiB a full load cannot work at all** — Node caps `fs.readFile` there and it fails with a
-raw `RangeError`. A windowed read is not an optimisation above that boundary, it is the only way in.
+**The limit is memory, not file size.** A read holds the symbols of the fields it reads and what it
+builds — the rows, or the columns — and reads the records a slice at a time rather than holding the file.
+So a QVD of any size can be read whole when its result fits, and `iterate()` over one holds those symbols
+and two chunks of rows. A read with `fields` never reads the other fields' symbols at all.
 
-Below it the limit is memory rather than file size, and a read too large to fit throws a **catchable**
-`QvdValidationError` carrying a row count that would have fitted, instead of a fatal
-`Reached heap limit` that no `try`/`catch` can intercept:
+A read too large to fit throws a **catchable** `QvdValidationError` carrying a row count that would have
+fitted, instead of a fatal `Reached heap limit` that no `try`/`catch` can intercept:
 
 ```javascript
 try {
@@ -281,7 +286,6 @@ Honest boundaries rather than an issue list — these are the ones that change w
 
 | Limitation                                                    | What it means in practice                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| **Full loads stop at 2 GiB**                                  | Use a windowed read or `iterate()` above that. Failure is a raw Node `RangeError`, not a `QvdError`.                                                                                                                                                                                                                                                                                                                                                                           |
 | **The symbol table is always parsed in full**                 | `{offset, limit}`, `iterate()` and `QvdColumnTable` all avoid materialising rows, but none is constant-memory in the size of a high-cardinality file.                                                                                                                                                                                                                                                                                                                          |
 | **A killed process can leave a temporary file**               | A write builds `<name>.qvdjs-<hex>.tmp` beside the QVD and removes it however the write ends — unless the process is killed outright, or the removal is itself refused, in which case the error names the file in `context.temporaryFile`. The QVD is untouched either way, and such a file can be deleted. `{atomic: false}` writes none when it rewrites a QVD that exists, and damages the QVD instead; a QVD that does not exist yet is renamed into place in either mode. |
 | **The writer takes numbers, strings, dual values and `null`** | Anything else - a `Date`, a boolean, an array, a `QvdSymbol` - is refused with a `QvdValidationError` naming the field and the row. Convert first: a `Date` to `new QvdDual(dateToQlikSerial(date), text)`, the serial and the text Qlik shows, which is how Qlik stores a date; a boolean to `-1` and `0`, as a Qlik comparison stores it.                                                                                                                                    |