simple-zstd 2.0.0-0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/.github/workflows/ci.yml +46 -0
  2. package/.github/workflows/release.yml +45 -0
  3. package/.prettierignore +5 -0
  4. package/.prettierrc +8 -0
  5. package/.release-it.json +28 -0
  6. package/CHANGELOG.md +26 -0
  7. package/README.md +657 -110
  8. package/dist/src/buffer-writable.d.ts +12 -0
  9. package/dist/src/buffer-writable.js +41 -0
  10. package/dist/src/index.d.ts +49 -0
  11. package/dist/src/index.js +430 -0
  12. package/dist/src/peek-transform.d.ts +16 -0
  13. package/dist/src/peek-transform.js +145 -0
  14. package/dist/src/process-duplex.d.ts +11 -0
  15. package/dist/src/process-duplex.js +157 -0
  16. package/dist/src/process-queue.d.ts +8 -0
  17. package/dist/src/process-queue.js +94 -0
  18. package/dist/src/types.d.ts +34 -0
  19. package/dist/src/types.js +3 -0
  20. package/eslint.config.js +49 -0
  21. package/package.json +29 -21
  22. package/src/buffer-writable.ts +30 -0
  23. package/src/index.ts +472 -0
  24. package/src/is-zst.d.ts +5 -0
  25. package/src/peek-transform.ts +153 -0
  26. package/src/process-duplex.ts +164 -0
  27. package/src/process-queue.ts +97 -0
  28. package/src/types.ts +35 -0
  29. package/tsconfig.json +110 -0
  30. package/.eslintrc.js +0 -19
  31. package/.nyc_output/88144029-aa5a-40fb-a072-f586e59e4a38.json +0 -1
  32. package/.nyc_output/cb2a47ed-9ada-42b3-863b-b8437ca49715.json +0 -1
  33. package/.nyc_output/processinfo/88144029-aa5a-40fb-a072-f586e59e4a38.json +0 -1
  34. package/.nyc_output/processinfo/cb2a47ed-9ada-42b3-863b-b8437ca49715.json +0 -1
  35. package/.nyc_output/processinfo/index.json +0 -1
  36. package/.travis.yml +0 -9
  37. package/buffer-writable.js +0 -23
  38. package/coverage/base.css +0 -224
  39. package/coverage/block-navigation.js +0 -87
  40. package/coverage/buffer-writable.js.html +0 -154
  41. package/coverage/favicon.png +0 -0
  42. package/coverage/index.html +0 -146
  43. package/coverage/index.js.html +0 -910
  44. package/coverage/oven.js.html +0 -244
  45. package/coverage/prettify.css +0 -1
  46. package/coverage/prettify.js +0 -2
  47. package/coverage/sort-arrow-sprite.png +0 -0
  48. package/coverage/sorter.js +0 -196
  49. package/index.js +0 -295
  50. package/process-queue.js +0 -74
package/README.md CHANGED
@@ -1,183 +1,730 @@
1
1
  # simple-zstd
2
2
 
3
3
  [![Build Status](https://travis-ci.org/Stieneee/simple-zstd.svg?branch=master)](https://travis-ci.org/Stieneee/simple-zstd)
4
- [![Package Size Size](https://badgen.net/badge/packagephobia/install/simple-zstd)](https://packagephobia.now.sh/result?p=simple-zstd)
5
4
  [![License](https://badgen.net/badge/license/MIT/blue)](https://choosealicense.com/licenses/mit/)
6
5
 
7
- Node.js interface to system installed zstandard (zstd).
6
+ Node.js interface to system-installed Zstandard (zstd) with TypeScript support.
8
7
 
9
- ## "Simple"-ZSTD
8
+ ## Overview
10
9
 
11
- The package name is inspired by another simple-git which is a wrapper around the git binary installed on the system.
12
- In summary this package, like simple-git, attempts to provide a lightwieght wrapper around the system installed ZSTD binary.
13
- This provides a more stable package in comparison to a package that builds against a library at the cost of having to manage child process.
10
+ simple-zstd is a lightweight wrapper around the system-installed zstd binary, inspired by simple-git's approach of wrapping system binaries rather than building against native libraries. This provides a more stable and portable solution at the cost of requiring zstd to be installed on the system.
14
11
 
15
- A few additional features have made there way into version 2 including a class that will attempt to pre-start child processes before they are need for the most latenacy concerned applications.
12
+ ### Features
16
13
 
17
- Regardless of whether you are performing a compression or decompression or using a stream or buffer interface this package will spawn an instance of zstd to handle the action.
18
- When the action is completed the child process will be killed.
19
- A dicitionary parameter on all functions supports both buffer and path definintons.
20
- If you provide the dictionary as a buffer this package will create a tmp file using the tmp (tmp-promise) package.
14
+ - **TypeScript Support**: Full TypeScript definitions and modern ES modules
15
+ - **Multiple Interfaces**: Static functions, buffer methods, and class-based API with process pooling
16
+ - **Promise-Based**: All operations return promises for modern async/await patterns
17
+ - **Stream & Buffer**: Support for both streaming and buffer-based compression/decompression
18
+ - **Smart Decompression**: Automatic detection and passthrough of non-compressed data
19
+ - **Dictionary Support**: Use compression dictionaries via Buffer or file path
20
+ - **Process Pooling**: Pre-spawn child processes for latency-sensitive applications
21
+ - **Node.js 18+**: Built on modern Node.js features
21
22
 
22
- All functions return a promise.
23
- This promise will return the stream or a buffer depending on the function type called.
23
+ ## Requirements
24
24
 
25
- Finally the decompression methods provide a "maybe" functionality.
26
- If a buffer or stream is passed that is not a zstd compressed byte stream the byte stream is not altered.
25
+ - **Node.js**: >= 18.0.0
26
+ - **zstd**: Must be installed and available on system PATH
27
27
 
28
- ## Dependencies
28
+ ## Installation
29
29
 
30
- ZSTD must be installed on the system.
31
- It will work in both Linux and Windows environments assuming the zstd(.exe) can be found on the path.
30
+ ### Install zstd
32
31
 
33
- Example:
32
+ **Ubuntu/Debian:**
34
33
 
35
- `sudo apt install zstd`
34
+ ```bash
35
+ sudo apt install zstd
36
+ ```
36
37
 
37
- ## Installation
38
+ **macOS:**
39
+
40
+ ```bash
41
+ brew install zstd
42
+ ```
43
+
44
+ **Windows:**
45
+
46
+ ```bash
47
+ choco install zstd
48
+ # or download from: https://github.com/facebook/zstd/releases
49
+ ```
50
+
51
+ ### Install simple-zstd
52
+
53
+ ```bash
54
+ npm install simple-zstd
55
+ ```
56
+
57
+ ## API Reference
58
+
59
+ ### Types
60
+
61
+ ```typescript
62
+ import { Duplex } from "node:stream";
63
+ import { SpawnOptions } from "node:child_process";
64
+ import { DuplexOptions } from "node:stream";
65
+
66
+ interface ZSTDOpts {
67
+ dictionary?: Buffer | { path: string }; // Compression dictionary
68
+ zstdOptions?: string[]; // CLI args to pass to zstd (e.g., ['--ultra'])
69
+ spawnOptions?: SpawnOptions; // Node.js child_process spawn options
70
+ streamOptions?: DuplexOptions; // Node.js stream options
71
+ }
72
+
73
+ interface PoolOpts {
74
+ compressQueueSize?: number; // Number of pre-spawned compression processes
75
+ decompressQueueSize?: number; // Number of pre-spawned decompression processes
76
+ compressQueue?: {
77
+ compLevel?: number;
78
+ dictionary?: Buffer | { path: string };
79
+ zstdOptions?: string[];
80
+ spawnOptions?: SpawnOptions;
81
+ streamOptions?: DuplexOptions;
82
+ };
83
+ decompressQueue?: {
84
+ dictionary?: Buffer | { path: string };
85
+ zstdOptions?: string[];
86
+ spawnOptions?: SpawnOptions;
87
+ streamOptions?: DuplexOptions;
88
+ };
89
+ }
90
+ ```
91
+
92
+ ### Static Functions
93
+
94
+ ```typescript
95
+ // Compression
96
+ compress(compLevel: number, opts?: ZSTDOpts): Promise<Duplex>
97
+ compressBuffer(buffer: Buffer, compLevel: number, opts?: ZSTDOpts): Promise<Buffer>
98
+
99
+ // Decompression (with automatic passthrough for non-compressed data)
100
+ decompress(opts?: ZSTDOpts): Promise<Duplex>
101
+ decompressBuffer(buffer: Buffer, opts?: ZSTDOpts): Promise<Buffer>
102
+ ```
103
+
104
+ ### Class: SimpleZSTD
38
105
 
39
- `npm i simple-zstd`
106
+ The `SimpleZSTD` class provides process pooling for better performance when performing many compression/decompression operations.
107
+
108
+ ```typescript
109
+ class SimpleZSTD {
110
+ // Static factory method (recommended)
111
+ static create(poolOptions?: PoolOpts): Promise<SimpleZSTD>;
112
+
113
+ // Instance methods
114
+ compress(compLevel?: number): Promise<Duplex>;
115
+ compressBuffer(buffer: Buffer, compLevel?: number): Promise<Buffer>;
116
+ decompress(): Promise<Duplex>;
117
+ decompressBuffer(buffer: Buffer): Promise<Buffer>;
118
+ destroy(): void;
119
+
120
+ // Statistics
121
+ get queueStats(): {
122
+ compress: { hits: number; misses: number };
123
+ decompress: { hits: number; misses: number };
124
+ };
125
+ }
126
+ ```
127
+
128
+ **Note:** Use the static `create()` method instead of the constructor. The constructor is private to ensure proper async initialization.
40
129
 
41
130
  ## Usage
42
131
 
43
- The static functions provide the most basic interface for usage
132
+ ### Example 1: Stream Interface (TypeScript)
44
133
 
45
- ```javascript
46
- const {compress, decompress, compressBuffer, decompressBuffer} = require('../index');
134
+ ```typescript
135
+ import fs from "node:fs";
136
+ import { pipeline } from "node:stream/promises";
137
+ import { compress, decompress } from "simple-zstd";
47
138
 
48
- compLevel = 3; // ZSTD Compression Level
49
- spawnOptions = {} // node:child_process spawnOptions object - adjust the spwan options of the ZSTD process
50
- streamOptions = {} // node:stream streamOptions - adjust the stream options
51
- zstdOptions = [] // Array of Options to pass to the zstd process e.g. ['--ultra']
52
- dictionary = Buffer || {path} // Supply an optional dictionary buffer or path to dictionary file
139
+ async function copyFile() {
140
+ const c = await compress(3); // Compression level 3
141
+ const d = await decompress();
142
+
143
+ await pipeline(
144
+ fs.createReadStream("example.txt"),
145
+ c,
146
+ d,
147
+ fs.createWriteStream("example_copy.txt")
148
+ );
53
149
 
54
- // Static Functions
55
- function compress(compLevel, spawnOptions, streamOptions, zstdOptions, dictionary) // returns a promise that resolves to a stream
56
- function compressBuffer(buffer, compLevel, spawnOptions, streamOptions, zstdOptions, dictionary) // returns a promise that resolves to a buffer
57
- function decompress(spawnOptions, streamOptions, zstdOptions, dictionary) // returns a promise that resolves to a stream
58
- function decompressBuffer(buffer, spawnOptions, streamOptions, zstdOptions, dictionary) // returns a promise that resolves to a buffer
150
+ console.log("File compressed and decompressed!");
151
+ }
152
+
153
+ copyFile().catch(console.error);
59
154
  ```
60
155
 
61
- The SimpleZSTD class allows an the settings to be preset for a pool of child process.
62
- The function names are the same on the class however the options can not be changed from the class constructor.
156
+ ### Example 2: Buffer Interface (TypeScript)
63
157
 
64
- ### Example - Stream Interface
158
+ ```typescript
159
+ import { compressBuffer, decompressBuffer } from "simple-zstd";
65
160
 
66
- This simple example reads a file, compressed, decopressed and writes the file as a copy.
67
- Running this example will spawn two instances of zstd as child processes.
161
+ async function processBuffer() {
162
+ const buffer = Buffer.from("this is a test");
163
+
164
+ // Compress with level 3
165
+ const compressed = await compressBuffer(buffer, 3);
166
+ console.log(
167
+ `Original: ${buffer.length} bytes, Compressed: ${compressed.length} bytes`
168
+ );
169
+
170
+ // Decompress
171
+ const decompressed = await decompressBuffer(compressed);
172
+ console.log(decompressed.toString()); // "this is a test"
173
+ }
174
+
175
+ processBuffer().catch(console.error);
176
+ ```
177
+
178
+ ### Example 3: CommonJS Usage
68
179
 
69
180
  ```javascript
70
- const fs = require('fs');
71
- const { pipeline } = require('node:stream');
72
- const {compress, decompress} = require('simple-zstd');
181
+ const fs = require("fs");
182
+ const { pipeline } = require("node:stream/promises");
183
+ const { compress, decompress } = require("simple-zstd");
73
184
 
74
185
  async function copyFile() {
75
186
  const c = await compress(3);
76
187
  const d = await decompress();
77
188
 
78
189
  await pipeline(
79
- fs.createReadStream('example.txt'),
190
+ fs.createReadStream("example.txt"),
80
191
  c,
81
192
  d,
82
- fs.createWriteStream('example_copy.txt'),
83
- () => {
84
- console.log('The file has been compressed and decompressed to example_copy.txt')
85
- }
193
+ fs.createWriteStream("example_copy.txt")
86
194
  );
195
+
196
+ console.log("File compressed and decompressed!");
87
197
  }
88
198
 
89
- copyFile();
199
+ copyFile().catch(console.error);
200
+ ```
90
201
 
202
+ ### Example 4: Class Interface with Process Pooling
203
+
204
+ The `SimpleZSTD` class pre-spawns zstd processes for lower latency. This is ideal for high-throughput scenarios.
205
+
206
+ ```typescript
207
+ import fs from "node:fs";
208
+ import { pipeline } from "node:stream/promises";
209
+ import { SimpleZSTD } from "simple-zstd";
210
+
211
+ async function processMultipleFiles() {
212
+ // Create instance with process pools using static factory method
213
+ const zstd = await SimpleZSTD.create({
214
+ compressQueueSize: 2, // Pre-spawn 2 compression processes
215
+ decompressQueueSize: 2, // Pre-spawn 2 decompression processes
216
+ compressQueue: {
217
+ compLevel: 3, // Default compression level for pool
218
+ },
219
+ });
220
+
221
+ try {
222
+ // Process first file with pool default (level 3)
223
+ const c1 = await zstd.compress();
224
+ const d1 = await zstd.decompress();
225
+
226
+ await pipeline(
227
+ fs.createReadStream("file1.txt"),
228
+ c1,
229
+ d1,
230
+ fs.createWriteStream("file1_copy.txt")
231
+ );
232
+
233
+ console.log("File 1 processed!");
234
+
235
+ // Process second file with custom compression level (bypasses pool)
236
+ const c2 = await zstd.compress(19); // Override with level 19
237
+ const d2 = await zstd.decompress();
238
+
239
+ await pipeline(
240
+ fs.createReadStream("file2.txt"),
241
+ c2,
242
+ d2,
243
+ fs.createWriteStream("file2_copy.txt")
244
+ );
245
+
246
+ console.log("File 2 processed!");
247
+
248
+ // Check pool statistics
249
+ console.log("Pool stats:", zstd.queueStats);
250
+ // Example output: { compress: { hits: 1, misses: 1 }, decompress: { hits: 2, misses: 0 } }
251
+ // Note: compress shows 1 miss because we used custom level for file2
252
+ } finally {
253
+ // Clean up all child processes
254
+ zstd.destroy();
255
+ }
256
+ }
257
+
258
+ processMultipleFiles().catch(console.error);
91
259
  ```
92
260
 
93
- ### Example - Buffer Interface
261
+ ### Example 5: Using Compression Dictionaries
94
262
 
95
- This example demonstrates the use of the buffer interface
263
+ ```typescript
264
+ import fs from "node:fs";
265
+ import { compressBuffer, decompressBuffer, SimpleZSTD } from "simple-zstd";
96
266
 
97
- ```javascript
98
- const {compressBuffer, decompressBufer} = require('simple-zstd');
267
+ // Static functions with dictionaries
268
+ async function useDictionaryStatic() {
269
+ const dictionary = fs.readFileSync("my-dictionary.zstd");
270
+ const data = Buffer.from("Sample text to compress");
99
271
 
100
- async function printString() {
101
- const buffer = Buffer.from('this is a test');
272
+ // Compress with dictionary
273
+ const compressed = await compressBuffer(data, 3, { dictionary });
102
274
 
103
- const compressed = await compressBuffer(buffer, 3);
104
- const decompressed = await decompressBuffer(compressed);
275
+ // Decompress with same dictionary
276
+ const decompressed = await decompressBuffer(compressed, { dictionary });
277
+
278
+ console.log(decompressed.toString()); // "Sample text to compress"
279
+ }
105
280
 
106
- console.log(decompressed.toString()); // this is a test
281
+ // Class with dictionaries (supports Buffer or file path)
282
+ async function useDictionaryClass() {
283
+ const dictionary = fs.readFileSync("my-dictionary.zstd");
284
+
285
+ const zstd = await SimpleZSTD.create({
286
+ compressQueue: {
287
+ compLevel: 3,
288
+ dictionary, // Can be Buffer or { path: '/path/to/dict' }
289
+ },
290
+ decompressQueue: {
291
+ dictionary, // Same dictionary for decompression
292
+ },
293
+ });
294
+
295
+ try {
296
+ const data = Buffer.from("Sample text to compress");
297
+ const compressed = await zstd.compressBuffer(data);
298
+ const decompressed = await zstd.decompressBuffer(compressed);
299
+ console.log(decompressed.toString()); // "Sample text to compress"
300
+ } finally {
301
+ zstd.destroy();
302
+ }
107
303
  }
108
304
 
109
- printString();
305
+ useDictionaryStatic().catch(console.error);
110
306
  ```
111
307
 
112
- ### Example - Class Interface
308
+ **Dictionary Caching:** When using dictionary Buffers with static functions, simple-zstd automatically caches the temporary dictionary files using SHA-256 hashing. This means:
309
+ - ✅ Multiple calls with the **same dictionary Buffer** reuse the same temp file
310
+ - ✅ No performance penalty for repeated operations with dictionaries
311
+ - ✅ Automatic cleanup when the dictionary is no longer in use
312
+ - ✅ **Fixes exponential slowdown** when compressing thousands of items with dictionaries
113
313
 
114
- This example demonstrates the use of the class interface.
115
- Once a class instance is created it can be used to with either the stream or buffer interface.
116
- Options can not be changed once the class is instantiated.
117
- This due to the fact that the class will start to spawn child processes with settings passed to the constructor.
118
- Calling ```destroy()``` will kill all child processes.
314
+ ```typescript
315
+ const dict = fs.readFileSync('my-dict.zstd');
119
316
 
120
- ```javascript
121
- const fs = require('fs');
122
- const { pipeline } = require('node:stream');
123
- const { SimpleZSTD } = require('simple-zstd');
124
-
125
- poolOptions = {
126
- compressQueue: {
127
- targetSize: 1 // this determines how many instances of zstd are spawned into the qeueue. leaving it empty will result child processes to bring spawned as needed.
128
- compLevel,
129
- spawnOptions,
130
- streamOptions,
131
- zstdOptions
132
- },
133
- decompressQueue: {
134
- targetSize: 1 // there is a queue for both compression and decompression
135
- spawnOptions,
136
- streamOptions,
137
- zstdOptions
138
- },
317
+ // These 1000 operations will only create ONE temp file total
318
+ for (let i = 0; i < 1000; i++) {
319
+ await compressBuffer(data[i], 3, { dictionary: dict });
139
320
  }
321
+ // Temp file is automatically cleaned up when no longer referenced
322
+ ```
140
323
 
141
- async function copyFile() {
142
- const i = new SimpleZSTD(poolOptions, dictionary); // poolOptions set options for the compressions and decompressiong porcess pools. dictionary is optional and set for both pools.
324
+ ### Example 6: Smart Decompression (Auto-detect)
143
325
 
144
- const c = await i.compress(3); // this will pull a instance from the queue
145
- const d = await i.decompress();
326
+ The decompression functions automatically detect if data is zstd-compressed and pass through uncompressed data unchanged.
146
327
 
147
- pipeline(
148
- fs.createReadStream('example.txt'),
149
- c,
150
- d,
151
- brake(200000),
152
- fs.createWriteStream('example_copy.txt'),
153
- () => {
154
- console.log('The file has been compressed and decompressed to example_copy.txt')
155
- }
156
- );
328
+ ```typescript
329
+ import { decompressBuffer } from "simple-zstd";
330
+
331
+ async function smartDecompress() {
332
+ const plainText = Buffer.from("not compressed");
333
+ const result = await decompressBuffer(plainText);
334
+
335
+ // Non-compressed data passes through unchanged
336
+ console.log(result.toString()); // "not compressed"
157
337
  }
158
338
 
159
- copyFile();
339
+ smartDecompress().catch(console.error);
160
340
  ```
161
341
 
162
- ## zstdOptions
342
+ ## Advanced Options
163
343
 
164
- This interface allows you to pass any command line option to the zstd process.
165
-
166
- ```javascript
167
- const c2 = await compress(22, {}, {}, ['--ultra']);
344
+ ### Custom zstd Options
345
+
346
+ Pass any command-line option to the zstd process via `zstdOptions`:
347
+
348
+ ```typescript
349
+ import { compress } from "simple-zstd";
350
+
351
+ // Use ultra compression (level 22)
352
+ const stream = await compress(22, {
353
+ zstdOptions: ["--ultra"],
354
+ });
355
+
356
+ // Multiple options
357
+ const stream2 = await compress(19, {
358
+ zstdOptions: ["--ultra", "--long"],
359
+ });
360
+ ```
361
+
362
+ ### Spawn Options
363
+
364
+ Control the child process spawn behavior:
365
+
366
+ ```typescript
367
+ import { compress } from "simple-zstd";
368
+
369
+ const stream = await compress(3, {
370
+ spawnOptions: {
371
+ cwd: "/custom/working/directory",
372
+ env: { ...process.env, CUSTOM_VAR: "value" },
373
+ },
374
+ });
375
+ ```
376
+
377
+ ### Stream Options
378
+
379
+ Customize the Duplex stream behavior:
380
+
381
+ ```typescript
382
+ import { compress } from "simple-zstd";
383
+
384
+ const stream = await compress(3, {
385
+ streamOptions: {
386
+ highWaterMark: 64 * 1024, // 64KB buffer
387
+ },
388
+ });
389
+ ```
390
+
391
+ ### Stream Events
392
+
393
+ All compression and decompression streams emit the following events:
394
+
395
+ ```typescript
396
+ import { compress } from "simple-zstd";
397
+
398
+ const stream = await compress(3);
399
+
400
+ // Standard Duplex stream events
401
+ stream.on("data", (chunk: Buffer) => {
402
+ console.log("Received chunk:", chunk.length, "bytes");
403
+ });
404
+
405
+ stream.on("end", () => {
406
+ console.log("Stream finished");
407
+ });
408
+
409
+ stream.on("error", (err: Error) => {
410
+ console.error("Stream error:", err);
411
+ });
412
+
413
+ // zstd-specific events
414
+ stream.on("stderr", (message: string) => {
415
+ // zstd process stderr output
416
+ console.warn("zstd stderr:", message);
417
+ });
418
+
419
+ stream.on("exit", (code: number, signal: NodeJS.Signals | null) => {
420
+ // zstd process exit event
421
+ console.log("zstd process exited with code:", code);
422
+ });
168
423
  ```
169
424
 
170
- ## Debug
425
+ **Event Reference:**
426
+
427
+ - `data` - Emitted when compressed/decompressed data is available
428
+ - `end` - Emitted when the stream has finished processing
429
+ - `error` - Emitted on stream errors or if zstd exits with non-zero code
430
+ - `stderr` - Emitted when the zstd process writes to stderr (warnings, debug info)
431
+ - `exit` - Emitted when the underlying zstd process exits
171
432
 
172
- The package supports the debug package.
173
- Setting the DEBUD Environment variable will cause the debug messages to be printed to the console.
174
- This will presents debug information for both zstd spawns and the child process queue.
433
+ ## Debugging
434
+
435
+ Enable debug output using the `DEBUG` environment variable:
175
436
 
176
437
  ```bash
177
- DEBUG=SimpleZSTD,SimpleZSTDQueue node example.js
438
+ # Debug simple-zstd operations
439
+ DEBUG=SimpleZSTD node app.js
440
+
441
+ # Debug process queue
442
+ DEBUG=SimpleZSTDQueue node app.js
178
443
 
444
+ # Debug both
445
+ DEBUG=SimpleZSTD,SimpleZSTDQueue node app.js
179
446
  ```
180
447
 
448
+ ## Migrating to v2
449
+
450
+ Version 2.0 is a complete rewrite with TypeScript support and a modernized API. Here's what you need to know:
451
+
452
+ ### Breaking Changes
453
+
454
+ #### 1. Node.js Version Requirement
455
+
456
+ **v1:** No explicit requirement
457
+ **v2:** Requires Node.js >= 18.0.0
458
+
459
+ ```bash
460
+ # Check your Node version
461
+ node --version # Should be v18.0.0 or higher
462
+ ```
463
+
464
+ #### 2. Function Names Changed
465
+
466
+ | v1 Function | v2 Function |
467
+ | ----------------------- | -------------------------------------------- |
468
+ | `ZSTDCompress(level)` | `compress(level, opts?)` |
469
+ | `ZSTDDecompress()` | `decompress(opts?)` |
470
+ | `ZSTDDecompressMaybe()` | `decompress(opts?)` _(built-in auto-detect)_ |
471
+
472
+ **v1 Code:**
473
+
474
+ ```javascript
475
+ const { ZSTDCompress, ZSTDDecompress } = require("simple-zstd");
476
+
477
+ const compressStream = ZSTDCompress(3);
478
+ const decompressStream = ZSTDDecompress();
479
+ ```
480
+
481
+ **v2 Code:**
482
+
483
+ ```javascript
484
+ const { compress, decompress } = require("simple-zstd");
485
+
486
+ const compressStream = await compress(3);
487
+ const decompressStream = await decompress();
488
+ ```
489
+
490
+ #### 3. All Functions Now Return Promises
491
+
492
+ **v1:** Functions returned streams synchronously
493
+ **v2:** Functions return `Promise<Duplex>` and must be awaited
494
+
495
+ **v1 Code:**
496
+
497
+ ```javascript
498
+ fs.createReadStream("file.txt")
499
+ .pipe(ZSTDCompress(3))
500
+ .pipe(ZSTDDecompress())
501
+ .pipe(fs.createWriteStream("output.txt"));
502
+ ```
503
+
504
+ **v2 Code:**
505
+
506
+ ```javascript
507
+ const c = await compress(3);
508
+ const d = await decompress();
509
+
510
+ await pipeline(
511
+ fs.createReadStream("file.txt"),
512
+ c,
513
+ d,
514
+ fs.createWriteStream("output.txt")
515
+ );
516
+ ```
517
+
518
+ #### 4. "Maybe" Functionality Now Built-in
519
+
520
+ **v1:** Had separate `ZSTDDecompressMaybe()` function
521
+ **v2:** All decompression functions auto-detect and pass through non-compressed data
522
+
523
+ **v1 Code:**
524
+
525
+ ```javascript
526
+ const { ZSTDDecompressMaybe } = require("simple-zstd");
527
+
528
+ stream.pipe(ZSTDDecompressMaybe()).pipe(output);
529
+ ```
530
+
531
+ **v2 Code:**
532
+
533
+ ```javascript
534
+ const { decompress } = require("simple-zstd");
535
+
536
+ const d = await decompress(); // Automatically detects compressed data
537
+ pipeline(stream, d, output);
538
+ ```
539
+
540
+ #### 5. Options Structure Changed
541
+
542
+ **v1:** Limited options as separate parameters
543
+ **v2:** Unified options object with TypeScript types
544
+
545
+ **v1 Code:**
546
+
547
+ ```javascript
548
+ // v1 had limited customization
549
+ ZSTDCompress(3, streamOptions);
550
+ ```
551
+
552
+ **v2 Code:**
553
+
554
+ ```javascript
555
+ await compress(3, {
556
+ dictionary: Buffer.from("..."),
557
+ zstdOptions: ["--ultra"],
558
+ spawnOptions: { cwd: "/tmp" },
559
+ streamOptions: { highWaterMark: 64 * 1024 },
560
+ });
561
+ ```
562
+
563
+ ### New Features in v2
564
+
565
+ #### TypeScript Support
566
+
567
+ ```typescript
568
+ import { compress, decompress, SimpleZSTD } from "simple-zstd";
569
+ import type { ZSTDOpts, PoolOpts } from "simple-zstd";
570
+ ```
571
+
572
+ #### Buffer Interface
573
+
574
+ New convenience methods for working with buffers directly:
575
+
576
+ ```typescript
577
+ import { compressBuffer, decompressBuffer } from "simple-zstd";
578
+
579
+ const compressed = await compressBuffer(Buffer.from("data"), 3);
580
+ const decompressed = await decompressBuffer(compressed);
581
+ ```
582
+
583
+ #### Process Pooling with SimpleZSTD Class
584
+
585
+ Pre-spawn processes for better performance with async factory method:
586
+
587
+ ```typescript
588
+ import { SimpleZSTD } from "simple-zstd";
589
+
590
+ const zstd = await SimpleZSTD.create({
591
+ compressQueueSize: 2,
592
+ decompressQueueSize: 2,
593
+ compressQueue: {
594
+ compLevel: 3,
595
+ dictionary: Buffer.from("..."), // Optional
596
+ },
597
+ });
598
+
599
+ // Use pooled processes
600
+ const stream = await zstd.compress();
601
+
602
+ // Or override compression level for specific operations
603
+ const stream2 = await zstd.compress(19);
604
+
605
+ // Clean up when done
606
+ zstd.destroy();
607
+ ```
608
+
609
+ #### Dictionary Support
610
+
611
+ Full support for compression dictionaries:
612
+
613
+ ```typescript
614
+ const dictionary = fs.readFileSync("dict.zstd");
615
+
616
+ await compress(3, { dictionary });
617
+ await compressBuffer(data, 3, { dictionary });
618
+ ```
619
+
620
+ ### Migration Examples
621
+
622
+ #### Simple Stream Compression
623
+
624
+ **v1:**
625
+
626
+ ```javascript
627
+ const { ZSTDCompress, ZSTDDecompress } = require("simple-zstd");
628
+
629
+ fs.createReadStream("input.txt")
630
+ .pipe(ZSTDCompress(3))
631
+ .pipe(fs.createWriteStream("output.zst"));
632
+ ```
633
+
634
+ **v2:**
635
+
636
+ ```javascript
637
+ const { compress } = require("simple-zstd");
638
+ const { pipeline } = require("node:stream/promises");
639
+
640
+ const c = await compress(3);
641
+ await pipeline(
642
+ fs.createReadStream("input.txt"),
643
+ c,
644
+ fs.createWriteStream("output.zst")
645
+ );
646
+ ```
647
+
648
+ #### Error Handling
649
+
650
+ **v1:**
651
+
652
+ ```javascript
653
+ ZSTDCompress(3).on("error", (err) => console.error(err));
654
+ ```
655
+
656
+ **v2:**
657
+
658
+ ```javascript
659
+ try {
660
+ const c = await compress(3);
661
+ c.on("error", (err) => console.error(err));
662
+ } catch (err) {
663
+ console.error("Failed to create stream:", err);
664
+ }
665
+ ```
666
+
667
+ #### With Custom Options
668
+
669
+ **v1:**
670
+
671
+ ```javascript
672
+ // Limited options in v1
673
+ const stream = ZSTDCompress(3, { highWaterMark: 64 * 1024 });
674
+ ```
675
+
676
+ **v2:**
677
+
678
+ ```javascript
679
+ const stream = await compress(3, {
680
+ streamOptions: { highWaterMark: 64 * 1024 },
681
+ zstdOptions: ["--ultra"],
682
+ });
683
+ ```
684
+
685
+ ### Upgrade Checklist
686
+
687
+ - [ ] Update Node.js to >= 18.0.0
688
+ - [ ] Replace `ZSTDCompress` with `compress`
689
+ - [ ] Replace `ZSTDDecompress` with `decompress`
690
+ - [ ] Replace `ZSTDDecompressMaybe` with `decompress` (same function)
691
+ - [ ] Add `await` to all compression/decompression calls
692
+ - [ ] Update imports to use new function names
693
+ - [ ] Consider using buffer methods (`compressBuffer`/`decompressBuffer`) for simpler use cases
694
+ - [ ] Consider using `SimpleZSTD` class for high-throughput scenarios
695
+ - [ ] Update error handling for async/await pattern
696
+ - [ ] Update tests to handle promises
697
+
698
+ ## Performance Benchmarks
699
+
700
+ This package has been benchmarked against other zstd packages.
701
+ At this time is appears to be the fastest package for processing large files.
702
+
703
+ [Benchmark Tests](https://github.com/Stieneee/node-compression-test)
704
+
705
+ ## Performance Considerations
706
+
707
+ This package spawns a child process for each compression or decompression operation. While this provides excellent performance for large files, child process creation overhead can become a bottleneck when processing many small files rapidly.
708
+
709
+ **Solution:** Use the `SimpleZSTD` class with process pooling for high-throughput scenarios:
710
+
711
+ ```typescript
712
+ const zstd = await SimpleZSTD.create({
713
+ compressQueueSize: 4, // Pre-spawn 4 compression processes
714
+ decompressQueueSize: 4, // Pre-spawn 4 decompression processes
715
+ });
716
+
717
+ // Reuse pooled processes for multiple operations
718
+ for (const file of files) {
719
+ const stream = await zstd.compress();
720
+ // ... process file ...
721
+ }
722
+
723
+ zstd.destroy(); // Clean up when done
724
+ ```
725
+
726
+ Process pooling significantly reduces latency by reusing existing child processes instead of spawning new ones for each operation.
727
+
181
728
  ## Contributing
182
729
 
183
730
  Pull requests are welcome.
@@ -186,7 +733,7 @@ Pull requests are welcome.
186
733
 
187
734
  MIT License
188
735
 
189
- Copyright (c) 2022 Tyler Stiene
736
+ Copyright (c) 2025 Tyler Stiene
190
737
 
191
738
  Permission is hereby granted, free of charge, to any person obtaining a copy
192
739
  of this software and associated documentation files (the "Software"), to deal