qvdjs 1.0.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +2 -1
- package/README.md +102 -32
- package/dist/index.cjs +639 -176
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +635 -172
- package/dist/index.js.map +1 -1
- package/package.json +10 -3
package/LICENSE
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
MIT License
|
|
2
2
|
|
|
3
|
-
Copyright (c)
|
|
3
|
+
Copyright (c) 2024 Constantin Müller
|
|
4
|
+
Copyright (c) 2025 Ptarmigan Labs AB
|
|
4
5
|
|
|
5
6
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
7
|
of this software and associated documentation files (the "Software"), to deal
|
package/README.md
CHANGED
|
@@ -21,7 +21,7 @@ console.log(df.shape); // [ 1705805, 20 ]
|
|
|
21
21
|
console.log(df.head(5));
|
|
22
22
|
```
|
|
23
23
|
|
|
24
|
-
No Qlik installation, no ODBC driver, no running engine — just the file. Node
|
|
24
|
+
No Qlik installation, no ODBC driver, no running engine — just the file. Node 22 or newer,
|
|
25
25
|
published as a dual ESM/CommonJS package, MIT licensed, one runtime dependency.
|
|
26
26
|
|
|
27
27
|
📖 **[Full documentation at qvdjs.ptarmiganlabs.com](https://qvdjs.ptarmiganlabs.com)** — guides,
|
|
@@ -30,14 +30,42 @@ API reference, the QVD format explained, measured performance, and troubleshooti
|
|
|
30
30
|
## ⚠️ This library is based on reverse engineering
|
|
31
31
|
|
|
32
32
|
Qlik does not publish the QVD format. Everything qvdjs knows about it was worked out by reading
|
|
33
|
-
files Qlik produced, and
|
|
33
|
+
files Qlik produced, and what it writes is compared with what Qlik Sense writes from the same data -
|
|
34
|
+
[below](#checked-against-qlik-sense).
|
|
34
35
|
|
|
35
36
|
That works well enough to be useful, and is not the same as being exact. **Test with your own
|
|
36
37
|
files, and validate output in your own Qlik environment** before relying on it. Reading any bundled
|
|
37
38
|
Qlik file and writing it back reproduces its symbol table byte for byte, whichever way it was read; a
|
|
38
39
|
value you change is written as Qlik stores that kind of value. Where that boundary lies is the first
|
|
39
40
|
thing worth reading:
|
|
40
|
-
[What a round trip preserves](https://qvdjs.ptarmiganlabs.com/
|
|
41
|
+
[What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v1.0/overview/what-a-round-trip-preserves/).
|
|
42
|
+
|
|
43
|
+
## Checked against Qlik Sense
|
|
44
|
+
|
|
45
|
+
qvdjs is compared with Qlik Sense itself, not only with its own reading of the format. 36 fields in
|
|
46
|
+
7 datasets - integers, doubles, text (non-ASCII, empty and number-like included), booleans, dates,
|
|
47
|
+
timestamps and NULLs - are defined twice, as Qlik load-script expressions and as the JavaScript
|
|
48
|
+
values a caller would write. Qlik Sense and qvdjs each write them to a QVD, and three things are
|
|
49
|
+
checked:
|
|
50
|
+
|
|
51
|
+
- **qvdjs writes the content Qlik writes** - the same symbols, of the same types, with the same text,
|
|
52
|
+
on the same rows.
|
|
53
|
+
- **Given Qlik's tags and number formats, qvdjs writes them unchanged** - the bit layout, below, is
|
|
54
|
+
the only part of a field's header left to differ.
|
|
55
|
+
- **Qlik reads qvdjs's file as it reads its own** - loaded into Qlik and stored again, with `Text()`,
|
|
56
|
+
`Num()`, `IsNum()`, `IsText()`, `IsNull()` and `Year()` the same on every row.
|
|
57
|
+
|
|
58
|
+
The live comparison runs on demand against a Qlik Sense server; its last run, against Qlik Sense
|
|
59
|
+
Enterprise on Windows build 50699, passed for all 36 fields. Qlik's files from that run are kept in
|
|
60
|
+
the test suite, so **every CI run compares qvdjs's output with them again** - on Linux, Windows and
|
|
61
|
+
macOS, under Node 22, 24 and 26 - along with reference files Qlik wrote for booleans, dates and
|
|
62
|
+
timestamps.
|
|
63
|
+
|
|
64
|
+
Two differences are by design: Qlik often gives a field more bits in the index than its values need,
|
|
65
|
+
where qvdjs uses the fewest, and qvdjs adds no tags or number formats of its own - Qlik works tags out
|
|
66
|
+
when it loads a file. QVD is also QlikView's format, but QlikView is not part of the comparison.
|
|
67
|
+
|
|
68
|
+
→ [Checked against Qlik Sense](https://qvdjs.ptarmiganlabs.com/v1.0/overview/checked-against-qlik-sense/)
|
|
41
69
|
|
|
42
70
|
## Four ways to open a file
|
|
43
71
|
|
|
@@ -79,7 +107,7 @@ Reaching for `fromQvd` when you wanted one of the other three is the common mist
|
|
|
79
107
|
`fromQvd(path, {maxRows: 0})` is **not** a substitute for `readMetadata`: it loads no rows but still
|
|
80
108
|
parses the whole symbol table, which grows with the data.
|
|
81
109
|
|
|
82
|
-
→ [Choosing an entry point](https://qvdjs.ptarmiganlabs.com/
|
|
110
|
+
→ [Choosing an entry point](https://qvdjs.ptarmiganlabs.com/v1.0/getting-started/)
|
|
83
111
|
|
|
84
112
|
## What a cell holds
|
|
85
113
|
|
|
@@ -119,13 +147,31 @@ await df.toQvd('out.qvd', {
|
|
|
119
147
|
});
|
|
120
148
|
```
|
|
121
149
|
|
|
122
|
-
|
|
150
|
+
The QVD is built beside the destination under a temporary name and renamed over it, so a write that
|
|
151
|
+
fails — a full disk, a killed process, an error from the disk — leaves the previous file exactly as it
|
|
152
|
+
was, and anything reading the path meanwhile gets the old file or the new one and never a part of
|
|
153
|
+
either. `toQvd()` resolves once the contents have reached the disk. The rename that puts the file in
|
|
154
|
+
place is not flushed, so a power loss moments after the call can still lose the replacement, or a QVD
|
|
155
|
+
that did not exist before — what it cannot leave is a damaged one.
|
|
156
|
+
|
|
157
|
+
Two options turn those off, each on its own:
|
|
158
|
+
|
|
159
|
+
| Option | What `false` does |
|
|
160
|
+
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
161
|
+
| `atomic: false` | Rewrites the file where it stands, as versions before 2.0.0 did. It keeps the file's identity — hard links, its owner, permissions set on the file itself — and needs neither room for two copies at once nor permission to create files in the directory. It also destroys the previous file as the write begins. A QVD that does not exist yet is renamed into place in either mode, since there is nothing to rewrite. |
|
|
162
|
+
| `fsync: false` | Resolves as soon as the operating system has accepted the bytes, without waiting for the disk. Faster, and a power loss just after the write can then leave a damaged file rather than the old one. |
|
|
163
|
+
|
|
164
|
+
A destination that exists and is not a regular file — a device such as `/dev/null`, or a FIFO — is
|
|
165
|
+
written in place whatever you ask for, because replacing it would replace the device itself.
|
|
166
|
+
|
|
167
|
+
Files it produces load in Qlik Sense as Qlik's own do - [checked](#checked-against-qlik-sense) for
|
|
168
|
+
each kind of value it writes. A number is written as a pure number - an integer
|
|
123
169
|
within 32 bits as an integer, any other number as a double - a string as a string, and `null` as
|
|
124
170
|
NULL. A dual value - a `QvdDual`, or an object with exactly the keys `number` and `text` - is written
|
|
125
171
|
as a dual, which is how a number gets the text Qlik displays. As in Qlik, a field holds one text per
|
|
126
172
|
number, the first written. Anything else is refused with a `QvdValidationError`.
|
|
127
173
|
|
|
128
|
-
→ [Writing a QVD](https://qvdjs.ptarmiganlabs.com/
|
|
174
|
+
→ [Writing a QVD](https://qvdjs.ptarmiganlabs.com/v1.0/guides/)
|
|
129
175
|
|
|
130
176
|
## Reading part of a file
|
|
131
177
|
|
|
@@ -165,7 +211,7 @@ try {
|
|
|
165
211
|
When it is `iterate()` that overflowed, the context carries `recommendedChunkSize` as well, so a
|
|
166
212
|
caller cannot mistake one recommendation for the other.
|
|
167
213
|
|
|
168
|
-
→ [Memory and file size limits](https://qvdjs.ptarmiganlabs.com/
|
|
214
|
+
→ [Memory and file size limits](https://qvdjs.ptarmiganlabs.com/v1.0/overview/memory-and-file-size-limits/)
|
|
169
215
|
|
|
170
216
|
## Metadata
|
|
171
217
|
|
|
@@ -182,7 +228,7 @@ df.setFileMetadata({tableName: 'UpdatedProducts'});
|
|
|
182
228
|
await df.toQvd('products-v2.qvd');
|
|
183
229
|
```
|
|
184
230
|
|
|
185
|
-
→ [Metadata reference](https://qvdjs.ptarmiganlabs.com/
|
|
231
|
+
→ [Metadata reference](https://qvdjs.ptarmiganlabs.com/v1.0/reference/)
|
|
186
232
|
|
|
187
233
|
## File paths are sandboxed
|
|
188
234
|
|
|
@@ -195,63 +241,87 @@ await QvdDataFrame.fromQvd(anyPath, {allowedDir: '/'}); // deliberately unrestri
|
|
|
195
241
|
```
|
|
196
242
|
|
|
197
243
|
Containment is decided by the filesystem — both paths are resolved through symlinks and compared by
|
|
198
|
-
device and inode — so a link inside the allowed directory pointing out of it is refused
|
|
244
|
+
device and inode — so a link inside the allowed directory pointing out of it is refused, whether or
|
|
245
|
+
not the file it points at exists.
|
|
199
246
|
|
|
200
|
-
|
|
247
|
+
A check is only as good as the open that follows it, so the open is held to what was checked. The
|
|
248
|
+
path is checked again immediately before the file is opened, and the file opened is the one the
|
|
249
|
+
check approved: a symlink swapped in at its name afterwards is refused rather than followed, and so
|
|
250
|
+
is a different file at that name. Either is a `QvdSecurityError` whose `context.reason` is
|
|
251
|
+
`'changed_after_check'`. What cannot be covered from Node — a _directory_ further up the path
|
|
252
|
+
swapped for a symlink in the moment between the check and the open — is the one gap left, so keep
|
|
253
|
+
the allowed directory's own subdirectories out of untrusted hands.
|
|
254
|
+
|
|
255
|
+
→ [Path security](https://qvdjs.ptarmiganlabs.com/v1.0/overview/path-security/)
|
|
201
256
|
|
|
202
257
|
## Errors
|
|
203
258
|
|
|
204
259
|
`QvdError` and five subclasses, each carrying a `code` and a `context` object:
|
|
205
260
|
`QvdValidationError` (`QVD_VALIDATION_ERROR`), `QvdCorruptedError` (`QVD_CORRUPTED_ERROR`),
|
|
206
261
|
`QvdParseError` (`QVD_PARSE_ERROR`), `QvdIOError` (`QVD_IO_ERROR`) and `QvdSecurityError`
|
|
207
|
-
(`QVD_SECURITY_ERROR`).
|
|
208
|
-
|
|
262
|
+
(`QVD_SECURITY_ERROR`). A file-system failure — a missing file, a directory, a permission error, a
|
|
263
|
+
full disk, a file another program has locked on Windows — is a `QvdIOError`: the system code is
|
|
264
|
+
`error.context.code` (`ENOENT`, `EACCES`, `ENOSPC`, …) and Node's own error is `error.cause`. On
|
|
265
|
+
Windows an `EPERM` or `EBUSY` from `error.context.operation === 'rename'` means the destination is open
|
|
266
|
+
in another program — Qlik Sense reading it, most often — and the file it holds is unchanged. A
|
|
267
|
+
`QvdSecurityError` says why in `error.context.reason`: `'outside_allowed_directory'`, `'null_byte'`,
|
|
268
|
+
or `'changed_after_check'` for a file that was not the one the check approved by the time it was
|
|
269
|
+
opened. The one
|
|
270
|
+
`QvdIOError` without `error.context.code` or `error.cause` is a write that the system accepted none of
|
|
271
|
+
without reporting an error; `error.context.filePosition` says where the file stops.
|
|
209
272
|
|
|
210
273
|
`error.name` is always the class name, in the CommonJS and ES module builds alike, so it works across
|
|
211
274
|
a module boundary where `instanceof` may not.
|
|
212
275
|
|
|
213
|
-
→ [Troubleshooting](https://qvdjs.ptarmiganlabs.com/
|
|
276
|
+
→ [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v1.0/troubleshooting/)
|
|
214
277
|
|
|
215
278
|
## Known limitations
|
|
216
279
|
|
|
217
280
|
Honest boundaries rather than an issue list — these are the ones that change what you can do.
|
|
218
281
|
|
|
219
|
-
| Limitation | What it means in practice
|
|
220
|
-
| ------------------------------------------------------------- |
|
|
221
|
-
| **Full loads stop at 2 GiB** | Use a windowed read or `iterate()` above that. Failure is a raw Node `RangeError`, not a `QvdError`.
|
|
222
|
-
| **The symbol table is always parsed in full** | `{offset, limit}`, `iterate()` and `QvdColumnTable` all avoid materialising rows, but none is constant-memory in the size of a high-cardinality file.
|
|
223
|
-
| **
|
|
224
|
-
| **The writer takes numbers, strings, dual values and `null`** | Anything else - a `Date`, a boolean, an array, a `QvdSymbol` - is refused with a `QvdValidationError` naming the field and the row. Convert first: a `Date` to `new QvdDual(dateToQlikSerial(date), text)`, the serial and the text Qlik shows, which is how Qlik stores a date; a boolean to `-1` and `0`, as a Qlik comparison stores it.
|
|
282
|
+
| Limitation | What it means in practice |
|
|
283
|
+
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
284
|
+
| **Full loads stop at 2 GiB** | Use a windowed read or `iterate()` above that. Failure is a raw Node `RangeError`, not a `QvdError`. |
|
|
285
|
+
| **The symbol table is always parsed in full** | `{offset, limit}`, `iterate()` and `QvdColumnTable` all avoid materialising rows, but none is constant-memory in the size of a high-cardinality file. |
|
|
286
|
+
| **A killed process can leave a temporary file** | A write builds `<name>.qvdjs-<hex>.tmp` beside the QVD and removes it however the write ends — unless the process is killed outright, or the removal is itself refused, in which case the error names the file in `context.temporaryFile`. The QVD is untouched either way, and such a file can be deleted. `{atomic: false}` writes none when it rewrites a QVD that exists, and damages the QVD instead; a QVD that does not exist yet is renamed into place in either mode. |
|
|
287
|
+
| **The writer takes numbers, strings, dual values and `null`** | Anything else - a `Date`, a boolean, an array, a `QvdSymbol` - is refused with a `QvdValidationError` naming the field and the row. Convert first: a `Date` to `new QvdDual(dateToQlikSerial(date), text)`, the serial and the text Qlik shows, which is how Qlik stores a date; a boolean to `-1` and `0`, as a Qlik comparison stores it. |
|
|
225
288
|
|
|
226
|
-
→ [What a round trip preserves](https://qvdjs.ptarmiganlabs.com/
|
|
289
|
+
→ [What a round trip preserves](https://qvdjs.ptarmiganlabs.com/v1.0/overview/what-a-round-trip-preserves/)
|
|
227
290
|
|
|
228
291
|
## Documentation
|
|
229
292
|
|
|
230
293
|
**[qvdjs.ptarmiganlabs.com](https://qvdjs.ptarmiganlabs.com)** is the documentation site, and it is
|
|
231
294
|
where everything above is covered properly:
|
|
232
295
|
|
|
233
|
-
|
|
|
234
|
-
|
|
|
235
|
-
| [Getting started](https://qvdjs.ptarmiganlabs.com/
|
|
236
|
-
| [Guides](https://qvdjs.ptarmiganlabs.com/
|
|
237
|
-
| [Concepts](https://qvdjs.ptarmiganlabs.com/
|
|
238
|
-
| [Reference](https://qvdjs.ptarmiganlabs.com/
|
|
239
|
-
| [Performance](https://qvdjs.ptarmiganlabs.com/
|
|
240
|
-
| [Troubleshooting](https://qvdjs.ptarmiganlabs.com/
|
|
296
|
+
| | |
|
|
297
|
+
| ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
|
298
|
+
| [Getting started](https://qvdjs.ptarmiganlabs.com/v1.0/getting-started/) | Install, choose an entry point, first read and write |
|
|
299
|
+
| [Guides](https://qvdjs.ptarmiganlabs.com/v1.0/guides/) | One page per task, runnable sample code |
|
|
300
|
+
| [Concepts](https://qvdjs.ptarmiganlabs.com/v1.0/concepts/) | The QVD format, symbols and duals, bit stuffing, the memory model |
|
|
301
|
+
| [Reference](https://qvdjs.ptarmiganlabs.com/v1.0/reference/) | Every class, method, option and error |
|
|
302
|
+
| [Performance](https://qvdjs.ptarmiganlabs.com/v1.0/performance/) | Measured baselines, and how to read them |
|
|
303
|
+
| [Troubleshooting](https://qvdjs.ptarmiganlabs.com/v1.0/troubleshooting/) | What each failure means, and what to do about it |
|
|
304
|
+
| [Checked against Qlik Sense](https://qvdjs.ptarmiganlabs.com/v1.0/overview/checked-against-qlik-sense/) | How qvdjs's output is compared with Qlik Sense's |
|
|
241
305
|
|
|
242
306
|
Benchmarks run weekly and publish to
|
|
243
307
|
[ptarmiganlabs.github.io/qvdjs](https://ptarmiganlabs.github.io/qvdjs/).
|
|
244
308
|
|
|
309
|
+
## Reporting bugs
|
|
310
|
+
|
|
311
|
+
Email [bugs@ptarmiganlabs.com](mailto:bugs@ptarmiganlabs.com). Include the qvdjs and Node versions,
|
|
312
|
+
the call that failed, and the whole error, with its `code` and `context`.
|
|
313
|
+
|
|
245
314
|
## Contributors
|
|
246
315
|
|
|
247
316
|
- [Göran Sander](https://github.com/mountaindude) and [Ptarmigan Labs](https://github.com/ptarmiganlabs) —
|
|
248
317
|
creator of qvdjs. Added lazy loading, columnar and chunked reads, header-only metadata reads,
|
|
249
318
|
exposed QVD header metadata, ESM/CJS support, memory safety limits, security hardening,
|
|
250
319
|
multi-platform testing and the automated release process.
|
|
251
|
-
- [Constantin Müller](https://mueller-constantin.de) — author of
|
|
252
|
-
[qvd4js](https://github.com/MuellerConstantin/qvd4js)
|
|
253
|
-
|
|
320
|
+
- [Constantin Müller](https://mueller-constantin.de) — author of
|
|
321
|
+
[qvd4js](https://github.com/MuellerConstantin/qvd4js), whose code qvdjs started from. Parts of it are
|
|
322
|
+
still in qvdjs, used under qvd4js's MIT licence.
|
|
254
323
|
|
|
255
324
|
## Licence
|
|
256
325
|
|
|
257
|
-
[MIT](https://opensource.org/licenses/MIT) © Ptarmigan Labs
|
|
326
|
+
[MIT](https://opensource.org/licenses/MIT) © Ptarmigan Labs AB, and © Constantin Müller for the code that
|
|
327
|
+
comes from qvd4js. The full text is in the `LICENSE` file shipped with the package.
|